Trump administration proposes removing AI safeguards from healthcare software
The Trump administration is moving to eliminate federal requirements that healthcare AI tools be tested on real users and that their decision-making processes be transparent to doctors and nurses. The proposed rules from the Office of the National Coordinator for Health IT would weaken standards for electronic health records, the central documentation system for patient care.
The shift comes as hospitals nationwide deploy AI scribes - software that automatically summarizes patient visits and generates clinical notes. While some doctors report time savings, others say the tools introduce errors that require manual correction, raising questions about whether regulators are moving in the right direction.
Current problems with AI scribes
Paul Boyer, a psychotherapist at Kaiser Permanente in Oakland, California, uses Abridge, a leading AI scribe vendor. He finds the software "not super useful." It misses clinical nuance and emotional tone - critical details in mental health where how something is said matters more than what is said.
Boyer spends time correcting the computer-written notes. A 2022 Veterans Health Administration study comparing 11 AI scribes found they performed worse than humans across five simulated scenarios, with particular problems omitting information that could affect follow-up care.
Yet the technology is spreading. A study of five hospitals published in April in the Journal of the American Medical Association found doctors who used AI scribes most saved more than half an hour daily.
What safeguards are being removed
The proposed rules eliminate two key requirements. First, companies would no longer need to conduct user-centered design testing with actual doctors and nurses before deploying products. Second, the administration is scrapping a Biden-era plan for AI transparency "model cards" - tools that let clinicians examine the data used to train AI systems with a mouse click.
The administration argues that removing requirements will spur innovation and competition. The electronic health record market is highly consolidated: Epic and Oracle Health control more than 70% of the hospital market, according to a 2022 study.
Abridge's general counsel, Tim Hwang, said the company "broadly supports" the rules as a "necessary modernization" that accommodates how fast AI is evolving.
Hospital and physician concerns
Even some in the healthcare and software industries worry the changes go too far. The American Hospital Association cautioned that hospitals have long struggled with the "black box nature" of certain AI tools and that transparency becomes more critical as more AI products enter the market.
Raj Ratwani, a researcher specializing in how people interact with technology at MedStar Health, said there is "currently no safeguard in place" to vet scribe software at the federal level. Poor record design can have direct clinical consequences. A cluttered medication list might cause a doctor to select the wrong drug.
The American College of Physicians warned that removing transparency requirements could "undermine clinician trust, increase liability expense, and erode the patient-physician relationship."
Even EHR developers aren't unified. Leigh Burchell, vice president for policy at Altera Digital Health, said the industry had "a lot of different perspectives" on the transparency model cards issue - unusual disagreement for the group.
The practical impact
Boyer worries that his employer will redesign his schedule around the expected time savings from AI scribes, scheduling more patients while he still needs to spend time correcting software errors. "When I am correcting that note, I feel like this is too much work," he said. "This is definitely making this worse, and this is taking up time that I need to not be spending on correcting an AI tool."
Kaiser Permanente said it does not require clinicians to use the AI scribe. The company declined to comment further.
The proposed rules are still moving through the regulatory process. Public comment closed in February. HHS' health IT office declined to comment, citing the ongoing review.
Learn more about AI for Healthcare and how Generative AI and LLM systems are being deployed in clinical settings.
Your membership also unlocks:
