The researchers also found a consistently higher level of agreement between AI model evaluations and standard-of-care diagnoses in contrast to radiologist interpretations. For the PRIME cohort, the study authors pointed out a 15.6 percent higher agreement with the AI assessments in comparison to unassisted radiologist evaluation (83 percent vs. 67.4 percent). Similarly, researchers noted a 14.9 percent higher agreement for AI evaluation in contrast to radiologist assessment for patients in the IP1-PROSTAGRAM group (87.7 percent vs. 72.8 percent).
“ … The SCARLET-1 study demonstrates that AI achieved diagnostic performance comparable to standard-of-care radiology readings for detection of Gleason grade group ≥2 prostate cancer across diverse populations and clinical settings. These findings support a potential role for AI in extending prostate MRI interpretation, particularly where subspecialty expertise is limited,” noted lead study author Anindo Saha, PhD, who is affiliated with the Diagnostic Image Analysis Group at Radboud University Medical Center in Nijmegen, the Netherlands, and colleagues.
Why Local Validation and Monitoring of AI Remains Essential
However, when using a threshold of lower sensitivity (73-76 percent) and higher specificity (88-90 percent) for a subset of patients with histologically-verified PCa, the researchers found conflicting results in the population-based screening cohorts.
Three Key Takeaways
• Performance is strong in external validation, but it is comparable to radiologists rather than superior to them. PI-CAI-2B reached AUCs of 84 to 89 percent for detecting Gleason grade group ≥2 cancer across one diagnostic cohort and two screening cohorts. Its agreement with standard-of-care diagnoses was about 15 percent higher than unassisted reads (83 percent vs. 67.4 percent in the PRIME cohort, and 87.7 percent vs. 72.8 percent in the IP1-PROSTAGRAM cohort). The authors’ own conclusion is that it matches standard-of-care readings. Its most realistic near-term role is extending prostate MRI interpretation where subspecialty expertise is limited.
• Results in screening populations depend on the setting so trade-offs at a fixed threshold may differ by site. At a high-specificity operating point, the AI found approximately 26 percent more high-grade cancers in the STHLM3-MRI cohort, but it also more than doubled unnecessary biopsies. In the IP1-PROSTAGRAM cohort, the AI model cut unnecessary biopsies by more than 19 percent while detecting a similar number of clinically significant cancers. Practices should not assume the published thresholds will produce the same results in their patient population.
• Local validation and ongoing monitoring support safe adoption. Performance can shift as patient populations and case mix change. The authors recommend site-level validation and continued monitoring, particularly in rapidly changing populations or where AI tools are deployed beyond the settings in which they were originally validated.
While the AI model detected over 26 percent more high-grade cases of PCa for patients in the STHLM3-MRI cohort, there was also more than double the number of unnecessary biopsies with the AI model in this group. However, in the IP1-PROSTAGRAM cohort, the researchers found that AI facilitated over 19 percent fewer unnecessary biopsies while offering comparable detection of Gleason grade group > 2 cancer in comparison to unassisted radiologist interpretation.
“Over time, shifts in case mix may affect performance in ways that are not immediately apparent. Local validation and performance monitoring are therefore essential, particularly in rapidly changing populations, for systems receiving regulatory clearance based on strong performance in narrow validation settings, and for those considered for broader use before their oncologic safety and generalizability have been fully characterized,” emphasized Saha and colleagues.
(Editor’s note: For related content, see “Prostate MRI Study Links Lower ACD Values on DWI to Higher Long-Term Risk of Distant Metastases,” “FDA Clears Emerging AI Software Application for Prostate MRI” and “A Closer Look at 3D AI Mapping with Prostate MRI.”)
In regard to study limitations, the authors conceded small cohort sizes for subset analyses and acknowledged that the lack of biopsy in the majority of negative MRI cases precluded a full comparison of false-negative rates between AI and unassisted interpretation of prostate MRI.
