Adjunctive artificial intelligence (AI) may significantly bolster detection of ischemic stroke on non-contrast computed tomography (NCCT) but the authors of a new meta-analysis cautioned that more research with adequate external validation is necessary prior to clinical adoption.
For the meta-analysis, recently published in Academic Radiology, researchers reviewed data from 38 studies and a total of 7,612 participants, including 6,179 individuals who had ischemic stroke.
Pooled internal validation data demonstrated a 91.2 percent sensitivity and 96 percent specificity for AI. However, only three of the reviewed studies (a total of six AI models) offered external validation data, which revealed a 31.4 percent decrease in sensitivity (59.8 percent) in contrast to the reported internal validation findings, according to the meta-analysis authors.
“Our study reveals that AI has an acceptable performance in detecting IS in NCCT in internal validation, although significant heterogeneity was observed in the meta-analysis. However, the generalizability and practical applicability of AI in real-world clinical settings remain limited due to insufficient external validation,” wrote lead meta-analysis author Wenzhuo Shen, MMed, who is affiliated with the Department of Radiology and Nuclear Medicine at Xuanwu Hospital and Capital Medical University in Beijing, China, and colleagues.
Overall, the researchers noted the adjunctive AI provided over 35 percent higher sensitivity (83.7 percent) than unassisted clinician assessment in internal validation (44.1 percent) and external validation cohorts (46.1 percent). The meta-analysis authors also noted comparable specificity between adjunctive AI (86.7 percent) and unassisted clinician evaluation (85.5 percent for internal validation and 83.6 percent for external validation).
The study authors pointed out that deep learning models offered 15 percent higher sensitivity (92.6 percent vs. 79.6 percent) and 6.7 percent higher specificity than machine learning approaches (96.7 percent vs. 90 percent).
Three Key Takeaways
- Adjunctive AI substantially improves stroke detection versus clinicians alone. Researchers noted pooled sensitivity gains of >35 percent in both internal and external validation while maintaining comparable specificity.
- Generalizability remains limited. External validation revealed markedly lower sensitivity (59.8 percent vs. 91.2 percent in internal validation) and many studies had high risk of bias.
- Deep learning models outperform machine learning approaches. Researchers noted higher sensitivity and specificity for deep learning models but also pointed out that the performance of adjunctive AI overall may decline for small infarcts or early stroke onset, underscoring the need for larger multicenter datasets and robust external validation.
However, the researchers conceded that 16 of the reviewed studies had a high degree of bias. An analysis of studies with low bias and external validation showed that adjunctive AI provided over 43 percent higher pooled sensitivity than clinician assessment of ischemic stroke (89.2 percent vs. 46.1 percent) and higher pooled specificity (88.2 percent vs. 83.6 percent). The meta-analysis authors acknowledged, however, that these low-bias study findings were based on six trials for the unassisted clinician assessments and four trials for adjunctive AI evaluation.
The researchers also noted studies demonstrating reduced sensitivity with adjunctive AI for lesions smaller than 4.8 cm2 and infarction size between 0.5 and 1.5 cm2 as well as reduced capability during early stroke onset times.
“It is essential to construct large-scale, multicenter NCCT databases for (ischemic stroke) to train models, improve their performance and generalization, and develop models capable of rapid and accurate lesion detection based on NCCT,” added Shen and colleagues.
In addition to the aforementioned high risk of bias in 16 of the reviewed studies, the authors said another limitation of the meta-analysis was a lack of reporting of area under the curve (AUC) values in the majority of the studies.
