Radiologists pan study critical of computer-aided detection

The radiology community has for the most part dissented from the findings of a study reported in April showing that computer-aided detection may actually do more harm than good in screening mammography. While critical of the study's design and conclusions, however, prominent radiologists agree that problems with misuse of the technology are worrisome.

"Nobody knows how widespread misuse is. CAD is not perfect. For CAD to be maximally effective, it has to be used properly," said Dr. Carol Lee, chair of the American College of Radiology breast imaging commission.

The retrospective study by Dr. Joshua J. Fenton and colleagues at the University of California, Davis Health System and the Breast Cancer Surveillance Consortium found that CAD results in more false positives, recalls, and biopsies (NEJM 2007;356;14:1399-1409).

Researchers analyzed performance at seven of 43 consortium facilities that had adopted CAD. The detection rate of invasive cancers dropped by 12% to 2.63 cases per 1000, while the detection rate of ductal carcinoma in situ increased by 34% to 1.57 cases per 1000. The number of biopsies was up for CAD adopters by nearly 20%, and, compared with 36 non-CAD centers in the consortium, overall accuracy was lower.

CAD is meant to act as a second reader that picks up additional cancers missed by radiologists. The study, however, was not designed to evaluate whether the technology reduced the number of overlooked cancers, said Dr. Daniel Kopans, a professor of radiology at Harvard Medical School. Nor did it assess whether CAD helped find cancers at earlier stages.

Among other criticisms, CAD users point out that the software used in the study was an older version and radiologists did not have enough time to reach optimal performance. In early stages of CAD implementation, users show a greater tendency to call patients back. Radiologists in the CAD user group had from two to 25 months' experience with the technology (mean time: 18 months).

"This study was set up to fail," said Dr. Stamatia Destounis, a diagnostic radiologist at the Elizabeth Wende Breast Clinic in Rochester, NY.

Also, greater detection of DCIS was viewed as a potential disadvantage because these cases might not progress into invasive cancers.

"In our clinical practice, we don't ignore microcalcifications. We biopsy DCIS. We feel good that we have found DCIS early and prevent it from becoming cancer down the line," Destounis said.

Dr. Carl D'Orsi, vice chair of the ACR breast imaging commission, expressed similar sentiments.

"Statisticians, nonimagers pooh-pooh the discovery of DCIS," he said. "The problem is, we don't know from imaging which types of DCIS have the potential to kill and which ones will remain quiescent. We don't have a crystal ball."

Some of the results are actually not far out of line with other papers in the literature, which indicate an increase in sensitivity of 2% to 20% and higher recall rates, said Robert Nishikawa, Ph.D., an associate professor in the radiology department at the University of Chicago.

Nishikawa considers the most controversial aspect of the study the pooling of data from seven facilities that used CAD with results from 36 consortium centers that did not. The radiologists in the CAD user group were generally more experienced mammographers and were more likely to be doing double reading, and the patient populations were vastly different. There were about 430,000 mammograms in the non-CAD group versus only 30,000 in the CAD facilities.

The researchers indicated that they adjusted for these differences, but it is unclear how, Nishikawa said.

These data were used to create an area under the ROC (receiver operator characteristics) curve, which forms the basis of the study's central conclusion of lower overall accuracy for CAD users compared with non-CAD users. This finding contradicts all other clinical data, Nishikawa said. The curve is not consistent with typical curves in screening mammography.

"I don't know how this ROC curve was generated, but it looks incorrect to me," he said.

REAL WORLD RESULTS?

Study lead author Fenton argues that some previous positive CAD studies might reflect optimal, by-the-book use of the technology.

"The advantage of our study is that it shows how CAD is being used by radiologists in the community. That is what ultimately will affect the population," said Fenton, an assistant professor of family and community medicine at UC Davis.

Although CAD is meant to be a second reader, helping to increase the number of cancers detected, some radiologists may be finding suspicious areas and deciding against a recall based on the second, CAD, reading.

"The technology may be leading radiologists to find certain cancers but miss others that are not marked. The aim of our study is to improve practice," Fenton said.

Anecdotal reports indicate that some radiologists are using CAD inappropriately as the primary, rather than secondary, reader, D'Orsi said.

Every radiologist, regardless of skill level, will sometimes fail to see a breast cancer.

"I believe that CAD development needs to be encouraged. Computers will get more powerful, and we need all the help that we can get to build on the major impact that screening mammography has had in reducing the mortality rate by 25% since 1990," Kopans said. "On the other hand, radiologists need to know that CAD is no substitute for their careful review and they should first read mammograms without CAD before they read them with CAD, since CAD does not mark all cancers that are visible on mammograms."

Ms. Hayes is feature editor for Diagnostic Imaging.