Health AI Is Accelerating. Who Sets the Guardrails?
AI is moving into clinics, wards, and patient portals. The upside is huge. The risks are real: bias, uneven performance across sites, new burdens on clinicians, and patient harm if things go wrong.
I. Glenn Cohen, faculty director of Harvard Law School's Petrie-Flom Center for Health Law, Biotechnology, and Bioethics, argues for a middle path: set serious standards without grinding progress to a halt. The question isn't whether to regulate, but how.
Where regulation is non-negotiable
For any medium- to high-risk use, oversight is table stakes - either strong internal governance or external rules. Today, most checks are internal, and approaches vary widely by hospital. That unevenness creates gaps and costs that many systems can't absorb.
Consumer-facing tools are a different story. Mental health chatbots or triage helpers operating outside hospital walls don't get internal review at all. That's a clearer case for external oversight.
Speed vs. safety: the race dynamic
AI can scale in weeks. That creates a "race dynamic" where teams sprint to deploy before funds, competitors, or countries beat them. Ethics and safety get sidelined fast.
Most medical AI will never see a federal or state regulator. Running everything through the FDA's drug or device pathways would be too slow and too expensive. Yet these systems can influence millions of decisions and behave differently across sites due to staffing, workflows, and training. That blurs the line between product approval and the practice of medicine - a long-standing FDA boundary.
For context on the FDA's current approach to AI/ML in medical devices, see the agency's overview: FDA: AI/ML for Software as a Medical Device.
Accreditation as a lever
The Joint Commission and the Coalition for Health AI (CHAI) released implementation guidance that puts most responsibility on individual facilities. If the Joint Commission ties AI standards to accreditation, hospitals will have to pay attention - Medicare and Medicaid billing depend on it.
Some recommendations are tough but directionally right: disclose when AI directly impacts care, obtain consent where appropriate, and continuously test and monitor performance. Frequency should match clinical risk.
That isn't cheap. It demands multidisciplinary committees and ongoing checks for accuracy, errors, adverse events, and equity across populations. Many hospitals will face a hard choice: build this capability or sit out AI adoption.
Read more about CHAI's work here: Coalition for Health AI.
The small-hospital squeeze
Vetting a complex model and its real-world rollout can run $300,000 to $500,000, according to leaders at large systems. Repeating near-identical evaluations across dozens of hospitals wastes money and time.
The result: a have/have-not split. Big academic centers adopt AI and improve care; community hospitals fall behind. That's especially painful because lower-resource settings may benefit the most. If models are trained on data from everyone but only help patients in a few regions, that's an ethical failure.
Assurance labs: a possible bridge
One federal idea on the table: "assurance labs." Private organizations, in partnership with government, would test algorithms against agreed standards so hospitals can rely on their reports. The prior administration supported the problem framing but not that approach; details on an alternative path are still unclear.
What leaders can do now
- Inventory and risk-tier your AI. Catalog every tool touching clinical or operational decisions. Classify by patient risk and data sensitivity.
- Stand up an AI oversight committee. Clinical leaders, nursing, IT, quality/safety, equity, legal, and frontline staff. Give it decision rights.
- Set disclosure and consent rules. Tell patients when AI meaningfully impacts their care. Obtain consent where it changes risk or standard workflows.
- Monitor what matters. Track calibration drift, subgroup performance, error types, adverse events, and user feedback. Match frequency to risk.
- Demand more from vendors. Require model cards, training data provenance, bias testing, validation results by subgroup, update logs, and a post-deployment monitoring plan.
- Pilot with gates. Start small, compare against clinician performance and standard of care, then scale. Define clear kill-switch criteria.
- Share evaluation work. Join regional consortia, HIEs, or group-purchasing collaboratives to pool testing and monitoring.
- Budget realistically. Account for implementation, governance, and monitoring - not just license fees. Consider shared services to lower costs.
- Prepare clinicians. Offer training on AI limits, bias, and safe use. Focus on decision support, not decision replacement.
If you're building workforce skills for safe AI adoption, explore practical learning paths by role: Complete AI Training: Courses by Job.
The mindset: careful optimism
Cohen is clear-eyed about the trade-offs but optimistic: "I speak about legal and ethical issues in this space, but I'm an optimist about this. I think that, in 10 years, the world will be significantly better off because of medical artificial intelligence."
That future depends on aligning incentives so smaller systems can participate, pressure-testing systems in the settings where they'll be used, and sharing the evaluation burden instead of duplicating it. Do that, and AI can lift care quality and access - not just in a few hubs, but everywhere patients show up for help.
Your membership also unlocks:
