Pertinent Insights on Evaluating the Value of AI Models in Radiology, Part 2

When evaluating adjunctive AI models for assisting with detection of a given condition, David Larson, M.D., said, in a recent interview with Diagnostic Imaging, that radiologists should consider the prevalence of that condition and its impact upon the AI model.

Employing a structured pre-deployment method to evaluate AI models in a prospective study, recently published in the American Journal of Roentgenology, Dr. Larson and colleagues assessed the value of multiple Aidoc AI triage models.

While the adjunctive AI model for aortic dissection offered an 85.4 percent sensitivity and a 98.9 percent specificity, the researchers noted the low prevalence rate of the condition (0.3 percent) contributed to a 21.1 percent positive predictive value (PPV) and .03 gain-to-pain ratio GPR, a metric to evaluate the tradeoff between enhanced detection and increased false positives with AI.

“I don't know if people really realize yet how impactful prevalence is to the performance of these AI models, and actually to radiologists as well. … How often was the entity present when you ordered the study? (This) has a strong relationship to overall value of imaging. (In regard to) this concept of high value imaging versus low value imaging, I think that in an unexpected way, our AI monitoring metrics may help contribute to that, to better understand where the high value versus low value ordering behavior is coming from,” explained Dr. Larson, the lead study author and executive vice chair of the Department of Radiology at the Stanford University School of Medicine and medical director of performance improvement at Stanford Health Care.

The researchers also discussed the potential with AI foundation models and agentic AI.

Jason Poff, M.D., a coauthor of the aforementioned study, said that adjunctive AI may do a first pass at assessing measurements for a follow-up cancer study. He suggested that agentic AI may enable “crosstalk” between different elements of the operating system so any subsequent revisions he makes to the AI measurements on the viewer are automatically captured in the radiology report.

“I think that the AI agents will help us in terms of allowing one piece of our software, like the reporting system, to speak in a bidirectional way with the viewer, and with the electronic medical record and all of the pieces that need to be more tightly integrated so we can really capture the efficiency gains,” posited Dr. Poff, the director of clinical AI for Radiology Partners.