Trump and Kennedy seek to relax safeguards for AI healthcare tools
The Trump administration is moving to weaken federal requirements for AI-powered healthcare software, removing rules that mandate user testing and transparency about how algorithms work. The changes would affect electronic health records systems that clinicians rely on daily to document patient care and make treatment decisions.
The proposed rules from the Office of the National Coordinator for Health IT would eliminate requirements for "user-centered design" testing - where developers test products on actual doctors and nurses - and scrap transparency standards for AI tools. The administration argues removing these rules will spur innovation and competition in a market dominated by a handful of vendors.
But the move troubles patient safety researchers, hospital administrators, and some healthcare software developers who say the safeguards exist for good reason.
The real-world problem
Paul Boyer, a psychotherapist at Kaiser Permanente in Oakland, California, is using one of these AI tools daily. His hospital deployed Abridge, software that automatically summarizes patient visits to save clinicians time on paperwork. The technology works better in some settings than others.
For Boyer, the AI scribe falls short. It misses clinical nuance and emotional tone - critical details in mental health where how something is said matters more than what is said. Boyer and his colleagues spend time correcting the software's errors rather than seeing it as a time-saver.
A study published in April in the Journal of the American Medical Association found that doctors using AI scribes saved more than half an hour of work daily after a year. But the same research shows clinicians must still catch and fix mistakes - work that takes time.
A Veterans Health Administration study comparing 11 AI scribes found the software performed worse than humans across five test scenarios. The authors noted the tools omitted information that could affect follow-up care.
What regulators want to remove
Raj Ratwani, a researcher at MedStar Health specializing in how people interact with technology, said "there is currently no safeguard in place" to vet scribe software at the federal level. The proposed rules would make that gap wider.
The administration is eliminating requirements for:
- User testing with doctors and nurses before products go live
- Companies reporting test results to federal regulators
- AI "model cards" - simple tools that let clinicians see what data trained an AI system and how it works
- Transparency standards about AI use in medical records
Regulators argue these rules burden innovation. Ryan Howells, a consultant for digital health companies, said federal regulations are "the single biggest inhibitor to true clinical innovation."
Ratwani counters that poor design creates real risks. A cluttered medication list, for example, could cause a doctor to select the wrong drug. User testing catches these problems before deployment.
Hospital systems push back
The American Hospital Association warned that removing transparency requirements is dangerous. "Hospitals and health systems have been challenged by the black box nature of certain AI tools and how the algorithms are developed," the association said in a comment letter. "Transparency is even more critical" as more AI tools flood the market.
The American College of Physicians offered similar concerns, saying lack of clarity about how AI works "could undermine clinician trust, increase liability expense, and erode the patient-physician relationship."
Even some healthcare software developers are divided. Leigh Burchell, vice president for policy at Altera Digital Health, said her industry group had "a lot of different perspectives" on the rules - unusual for a trade association. But most agreed companies should disclose what data AI relies on and how it makes recommendations.
Abridge, the scribe company, backs the proposed rules as "necessary modernization" that accommodates how fast AI is evolving.
The practical concern
Boyer said he can ignore his AI scribe for now. But he worries Kaiser will schedule more patients based on the expected time savings, forcing him to spend more time correcting errors.
"When I am correcting that note, I feel like this is too much work," Boyer said. "This is taking up time that I need to not be spending on correcting an AI tool."
A Kaiser Permanente spokesperson said the company does not require clinicians to use AI, though it remains deployed across the system.
The proposed rules are still in the regulatory process. Public comment closed in February.
For more on how AI is being deployed in clinical settings, see our coverage of AI for Healthcare and Generative AI and LLM applications.
Your membership also unlocks:
