Paragon Health Institute Proposes Safety Framework for AI Medical Devices
Paragon Health Institute published a research paper May 20 proposing a new approach to one of the central safety problems in AI-enabled medical devices: the gap between how well these systems perform in testing versus how they perform on real patients.
AI medical devices often work reliably during development but behave unpredictably when deployed to patients whose medical images or data differ from the training set. This performance gap, called generalization uncertainty, has no settled solution in current regulatory frameworks.
The Cost Problem With Current Approaches
Existing remedies carry significant drawbacks. High-cost consultations required to validate device performance create barriers for rural hospitals and safety-net providers that lack resources well-financed health systems have. Risk assessments tied to broad demographic categories also fail to account for individual patient variation.
The policy challenge is clear: regulators must avoid mandating solutions that sound safe but actually slow adoption of life-saving technology without meaningfully improving outcomes.
Digital Similarity Analysis: A Voluntary Framework
The paper proposes Digital Similarity Analysis (DSA), a voluntary system manufacturers could implement to flag when a patient's medical data falls outside the range the device was trained on. When DSA alerts a physician to an outlier case, the physician can choose to forgo the device, request additional validation of its output, or use it with reduced confidence.
DSA would not eliminate generalization uncertainty. Instead, it gives clinicians actionable information at the point of care-before a device is used on an atypical patient.
Trade-offs and Broader Implications
The framework preserves manufacturers' training data confidentiality, a critical competitive asset in AI development. It also shifts the conversation about algorithmic bias from population-level demographics to individual patient characteristics, potentially improving safety across demographic groups.
Paragon Health Institute, a nonprofit research organization founded in 2021, does not accept industry funding. The institute has published related work on AI for healthcare, including papers on postmarket surveillance and regulatory guidelines for AI innovation in medicine.
Your membership also unlocks:
