system existed on their planes, immediately putting them at a disadvantage.
Within radiology, notification and training can be critical in systems that are intended to be transparent to users, such as AI-based image reconstruction algorithms, Mongan said.
“If people are uninformed about AI in their workflow, the likelihood that failure of the system will be detected decreases, and the risk associated with failures increases,” he explained.
4. Include a way to disable closed-loop systems. Such systems, including the planes’ safety system, can introduce added risk because they initiate actions without any human involvement. Currently, most radiology AI tools aren’t closed loop systems as they provide triage, prioritization, or diagnostic decision support. But, this doesn’t preclude that possibility for the future.
If any closed-loop AI tools are developed, Mongan said, designers should create a type of fail-safe mechanism that can be activated to side-step potentially harmful situations. Boeing’s planes didn’t have this type of measure.
“To mitigate this additional risk, closed-loop systems should clearly alert users when they are initiating actions, systems should accompany the alerts with a simple and rapid mechanism for disabling the system,” he said, “and the system should remain disabled long enough for the failure to be addressed.”
5. Don’t rely on regulation alone. The airline industry has been stringently regulated by the U.S. Federal Aviation Administration since the 1980s, but even with these guidelines and measure in place, an analysis of these crashes revealed that Boeing underreported and mischaracterized the performance and risks of its safety system.
This could be significant in radiology because the aviation industry’s regulatory environment has served as a model for the U.S. Food & Drug Administration’s (FDA) approach to regulating AI in healthcare. Rather than certifying individual AI tools, the FDA has opted to certify software developers, streamlining application reviews in a process called Pre-Cert.
Consequently, Mongan said, regulation is no guarantee of proper performance and or that critical information will always be shared.
“Regulation is necessary, but may not be sufficient to protect patient safety,” he advised, “particularly when subject to the conflicts of interest inherent in delegated regulatory review.”
Although, in hindsight, these mistakes might appear as though they should have been evident, they still easily occurred, he said. With this knowledge, radiology is armed to potentially prevent negative impacts on patient care.
“In retrospect, these errors may seem obvious, by they occurred in a mature field with a strong safety culture, and similar failures could easily recur in the developing area of AI in radiology,” Mongan said. “We have the opportunity to learn from these failures now, before there is widespread clinical implementation of AI in radiology. If we miss this chance, our future patients will be needlessly at risk for harm from the same mistakes that brought down these planes.”