How do incorrect suggestions from adjunctive AI affect radiologist accuracy and reading behavior when assessing screening mammograms?
For a retrospective multi-reader study, recently published in Radiology, researchers compared the use of adjunctive AI and unassisted interpretation in a test set of 60 mammograms. The cases in the test set included 14 cases with false negatives and 14 cases with false-positive findings, according to the study.
The researchers found that unassisted radiologists had a 32 percent higher sensitivity rate than those who utilized AI in cases involving false-negative (FN) AI suggestions (71 percent vs. 39 percent). The study authors added that those who employed adjunctive AI had reduced fixation on the images in these cases in contrast to unassisted radiologists (0.44 fixations per second in comparison to 0.47 fixations per second).
“These findings suggest that in FN AI cases, reduced search effort and fewer pauses occurred when the AI failed to highlight the correct area, leading to undetected lesions,” noted lead study author Adnan G. Taib, BMBS, MRCS, who is affiliated with Translational Medical Sciences at the School of Medicine at the University of Nottingham in Nottingham, U.K., and colleagues.
For the cases involving false positive (FP) suggestions from AI, researchers also noted reduced image fixation from reviewers using AI versus unassisted radiologists (54 second vs. 56 seconds). However, in these cases, the median specificity with adjunctive AI was 18 percent higher than that of unassisted radiologist assessment (39 percent vs. 21 percent), according to the study authors.
“This finding indicates a quicker, more superficial review of FP prompts when reading with AI. Importantly, changes in visual search were observed only with false AI suggestions, whereas true AI suggestions did not alter fixation behavior,” added Taib and colleagues.
Three Key Takeaways
• Don't let AI silence override your search pattern. A "clean" AI read may be the highest-risk scenario for missed cancers. It can suppress your normal visual search effort. Maintain full diagnostic scrutiny even when AI doesn't flag a finding rather than treating the absence of a prompt as a negative result in itself.
• False-positive AI prompts are comparatively low-risk to your judgment. Radiologists in this study handled incorrect "positive" AI flags well, maintaining good specificity despite reduced fixation time, indicating that skepticism toward AI-flagged findings is reasonably well-calibrated.
• Automation bias is a real, measurable phenomenon. Given this risk, routine local performance audits of AI systems (especially after software/device updates) are advisable, as recommended in the accompanying editorial, to catch drift in AI accuracy before it erodes radiologist vigilance.
In an accompanying editorial, Paola Clauser, MD, PhD, noted the contrasting findings in which experienced radiologists exercised good judgment with respect to the false positive prompts from AI but tended to be caught up in automation bias in cases involving false-negative findings with adjunctive AI.
“When accustomed to a good performance of the AI system in clinical practice, even experienced radiologists might overlook possible changes in the performance associated with software updates of the mammography device or the AI system. … Considering the risk of automation bias, it is likely that the most reliable solution would be to establish regular evaluation of the diagnostic performance of the clinically used AI systems on a separate dataset representative of the population for each single institution,” advised Dr. Clauser, an associate professor and head of female imaging with the Department of Biomedical Imaging and Image-Guided Therapy at the Medical University of Vienna in Vienna, Austria.
(Editor’s note: For related content, see “Mammography News: FDA Clears New Functionalities for DeepHealth’s AI-Powered Breast Suite,” “Mammography Study Shows Dynamic Changes of AI Risk Scores Years Before Breast Cancer Diagnosis” and “Mammography Study Shows Impact of AI-Powered Slab Reconstruction with DBT.”)
In regard to study limitations, the study authors acknowledged a small number of cases with true-negative AI findings, the use of one vendor’s AI software and limited exposure to using adjunctive AI among the reviewing radiologists who did have significant experience with mammography interpretation.
