NCI consortium proposes standardized lung interpretation for CAD "truth" assessment

A major issue facing researchers who develop computer-aided diagnosis systems for early detection of lung cancer is deciding what in a CT image truly represents a lung nodule. An effort sponsored by the National Cancer Institute could be the key to getting around this problem.

One of the main goals of the NCI's Lung Image Database Consortium (LIDC) initiative was to develop consensus guidelines for the creation of a database of CT lung images as a research resource for the development and evaluation of CAD methods. The LIDC has come up with a two-step process that can help provide a robust, standardized way of establishing "truth" for use in lung nodule CAD studies, said Dr. Samuel G. Armato, an associate professor of radiology at the University of Chicago. He presented findings of an LIDC study Tuesday at the 2007 RSNA meeting.

Armato and his LIDC colleagues looked into what CAD specialists refer to as "ground truth." The study contained two parts: First, researchers analyzed variable definitions of "truth" that could be created from combined lung CT reads by experienced thoracic radiologists. Next, they analyzed the variability in the performance of other experienced thoracic radiologists based on these definitions of truth.

Four thoracic radiologists reviewed 25 thoracic CT scans and marked lesions they considered to be nodules 3 mm or larger. The study then retrospectively created sets of "true" nodules from different combinations of two of the four radiologists. The nodule-detection performance of the other two radiologists was evaluated based on this two-radiologist assessment of true nodules.

The investigators found substantial variability across radiologists for lung nodule identification. They also noted substantial variability in the truth sets that may be created from different combinations of radiologists and in the nodule-detection performance of radiologists compared against those truth sets.

The most important aspect of the study is that it showed that the definition of truth can be highly variable depending upon who provides that truth and what process is used to gather that truth, according to Armato. CAD researchers need to understand the truth they are using to evaluate their systems and the potential limitations. The LIDC two-step process for evaluating lung CT scans (which was not explicitly investigated in this study) is expected to help make the truth assessment more robust, Armato said in an interview with Diagnostic Imaging.

"What it comes down to is that what we as CAD researchers consider truth is not necessarily 'the' truth, and that there are variabilities in that truth. That variability in truth will impact the reported performances of CAD systems. That can be important for research and for FDA approval of commercial systems now and in the future," he said.