CAD vendors seek balance between sensitivity and false-positive findings

Vendors and academics look for solutions

Computer-aided diagnosis (CAD) systems can help radiologists find cancers they might otherwise miss, especially small, early breast and lung cancers. But the complex algorithms that can identify these tiny lesions also have a drawback-high rates of false positives. CAD vendors know they must strike a delicate balance in developing systems sensitive enough to help radiologists but not so sensitive that they highlight false positives and, consequently, degrade productivity.

"As you increase specificity, you trade off sensitivity," said Peter Farrell, director of marketing at CADx, the Canadian developer of Second Look, which is being groomed as an aid in detecting breast cancer. "We've found that if you push to lower the FP rate to one mark per case, the sensitivity drops by 10%."

Vendors, therefore, are struggling to determine how many FPs are too many. Researchers in the Netherlands are studying CAD practice patterns in an effort to get a handle on this figure. They are looking at the number of false positives radiologists will tolerate before they give up on using the system as their criterion. The Dutch team believes that radiologists will tolerate FP rates of 0.2 per image before losing interest.

"In (mammography) screening practice, it appears that the quality of prompts plays a very important role," said lead researcher Dr. Nico Karssemeijer of the University of Nijmegen. "When the specificity of prompts is too low, radiologists tend to pay little attention to them."

No single answer may fit all cases. Dr. H. Michael Yeh, president and CEO of Deus Technologies, thinks the issue of FPs is somewhat different when screening for cancer in the breast rather than the lung, which is the focus of the Rockville, MD, company's RapidScreen RS-2000.

"Our FP rate is about three per image on average and we continue to work on reducing that," he said. "But CAD can so greatly improve a doctor's ability to detect stage I lung cancers that the FP issue is not such a concern. Only about 15% of lung cancers are found at stage I without CAD, but one clinical trial showed that with CAD, the detection rate improved to 75%."

Lung cancer screening programs tend to include only heavy smokers, and their lungs often contain scarring or other evidence of lung disease, which reduces specificity with CAD. At Georgetown University Medical Center, Dr. Matthew Freedman has been studying acceptable FP levels among radiologists using RapidScreen.

"In the clinical trials, you will see as many as five FPs per image," said Freedman, an associate professor of radiology at Georgetown. "But the radiologists like the system. They say, 'yes, we'd prefer fewer false positives.' But having at least five per image did not cause them to say they'd stop using the system."

Deus has found one way to reduce false positives. The more images included in the database used to "train" the CAD system, the fewer FPs, Yeh said. The company is constantly increasing its database to refine and improve the system. Yeh expects to see major reductions in FPs within at least five years.

Researchers are also investigating other approaches for improving specificity without affecting sensitivity. At the University of Chicago, a group led by Dr. Qiang Li, a research scientist in the radiology department, has developed a multiple-templates matching technique to reduce FPs in lung cancer CAD software. A CAD system under development at the University of Michigan in Ann Arbor also employs a multiple-templates matching technique to both reduce FPs and improve overall CAD performance. Their system is designed to assist the detection of pulmonary nodules using spiral CT. They were able to decrease the average number of FPs per study from 3.6 to 1.74 without lowering overall sensitivity.

CAD vendors believe that while the FP incidence can be diminished if not eliminated, the systems are still highly valuable in clinical practice. False negatives-missed cancers-are of far greater concern than false positives.

"The whole purpose of CAD is to reduce false negatives," Yeh said. "What's really important is not the machine's sensitivity but what happens when the radiologist uses the machine. Together they can find about 75% of early cancers, which is better than either one alone."