“Appropriate threshold settings need to be identified for clinical application. It is important to note that this result was based on visual assessment as ground truth, a common but imperfect method for ILA evaluation. … Determination of the best model or threshold should be based on more reliable clinical outcomes, such as survival,” wrote lead study author Akinori Hata, M.D., who is affiliated with the Center for Pulmonary Functional Imaging in the Department of Radiology at Brigham and Women’s Hospital and Harvard Medical School in Boston, and colleagues.
Three Key Takeaways
1. Machine learning efficacy. The study showed that a machine learning model combining a three-label method for section inference and a two-label method with a random forest (RF) classifier achieved an 87 percent area under the curve (AUC) and 90 percent specificity for predicting ILAs. However, the sensitivity was lower (64 percent).
2. Model comparisons. There was no significant difference in AUC performance between the support vector machine (SVM), RF, and convolutional neural network (CNN) classifiers with researchers noting that limited data set sizes may have impacted the performance of CNN models.
3. Threshold and clinical outcome relevance. Researchers highlighted the need for appropriate threshold settings and emphasized that models should ideally be evaluated based on reliable clinical outcomes like survival rather than purely visual assessments.
The study authors also noted no significant differences in AUCs between SVM, RF and CNN classifiers with respect to models with two-label section inference and three-label case inference (ranging between 80 to 83 percent), those with three-label section inference and two-label case inference (86 to 87 percent), and models with three-label section inference and three-label case inference (ranging between 85 to 86 percent).
“While RF and SVM classifiers use the probability of each section directly as a feature, a CNN classifier generates new features through convolution and other operations. This process requires a large number of cases to generate effective features, so the limited sample size in this study might have resulted in the lower performance of the CNN models compared with the RF and SVM models,” noted Hata and colleagues.
(Editor’s note: For related content, see “FDA Clears CT-Based AI Software for Assessing Interstitial Lung Disease,” “FDA Clears AI-Powered CT Assessment Tool for Lung Fibrosis” and “Can Deep Learning Bolster CT Detection and Classification of Usual Interstitial Pneumonia?”)
In regard to study limitations, the authors conceded the lack of external validation and clinical outcome evaluation. They also noted the study’s inclusion of section images 2 < 2.5 mm, and contrast-enhanced CT for 75 percent of the reviewed exams, two factors that can affect the detection of ILA, according to the researchers.
