Software Criteria For Defining Marks As True, False

Israeli and U.S. researchers have found that computer-aided detection systems can boost the accuracy of chest radiography for lung cancer, especially for procedures performed by inexperienced readers.

Although chest CT represents the de facto imaging standard for lung cancer diagnosis, many hospitals in the U.S. and most around the developing world still rely on cheaper digital or analog chest radiography systems. The use of a CAD device for detection of lung nodules on digital chest radiographs improves sensitivity, particularly of less experienced readers, and can function as a second reader, according to Dr. Dorith Shaham, a radiologist at the Hadassah-Hebrew University Medical Center in Jerusalem.

Shaham's retrospective study in­volved 77 digital chest radiographs from three Israeli health facilities. The DR samples were correlated with chest CT by two independent expert readers who established the "ground truth" for the presence and location of masses and lung nodules greater than 5 mm. Three radiologists with various levels of expertise-a resident, a general radiologist, and a thoracic radiologist-subsequently read these cases randomly and marked apparent lung masses and nodules greater than 5 mm without CT correlation. Readers relied on a prototype detection system that generated CAD marks for all DR images.

The investigators found that the CAD device increased the detection rate for nodules of all readers but most significantly for the resident. Results were presented at the 2007 RSNA meeting.

Researchers established the ground truth based on 46 findings from 36 digital radiographs. Thirty-nine findings were nodules 5 mm or larger. Using the CAD system, the general radiologist detected 22 nodules compared with 20 nodules without CAD. The thoracic radiologist detected 23 nodules compared with 20 nodules without CAD.

The detection rate for the resident, on the other hand, increased by 10.3% and was statistically significant (p

"The increase (for the resident) was inversely proportional to the expertise of the reader," Shaham said.

In a separate study, researchers at the University of Chicago sought to establish a digital radiography system's true rate of detection versus the "accidental" markings that can be attributed to this technology when used in lung cancer detection.

Lead investigator Dr. Feng Li, Ph.D., and colleagues retrospectively applied CAD to 34 digital chest radiographs with malignant nodules not mentioned in their respective reports. The researchers defined CAD markings as true detections only when their center was within the area of a lesion boundary. Detection was deemed accidental if the center point of the CAD marking was not within the lesion boundary, even if the lesion was located completely or partially within the5-cm circle provided by the marking.

The investigators found the reported sensitivity of the CAD system can vary by as much as 100% depending on the precise definition of true versus false marks. The same criterion could yield smaller, though substantial, specificity.

CAD found a total of 132 marks, with a sensitivity of only 26% and an average of 3.6 false positives per radiograph. If the criterion was expanded to include lesions located completely within the circles, the sensitivity for cancers marked increased to 41% with 3.4 false-positive marks per image. If the criterion was again expanded to include lesions located at least partially within the circles, the sensitivity increased to 53% with 3.3 false-positive marks per image. Specificity changed only marginally as more of the lesions farther away from the center of the markings were included.

"In evaluating CAD systems, it is important to understand how their apparent accuracy can be influenced by the specific criteria that are used to determine sensitivity and specificity," Li said.
-H.A. Abella