Utah's AI Prescription Bot Faces Safety Questions Despite State Backing
Utah became the first state in January to let an AI system autonomously handle routine prescription refills for patients with chronic conditions. The pilot aims to reduce delays that prevent people from taking their medications consistently.
Within weeks, researchers published a report showing they could trick the same company's AI into recommending dangerous doses and unsafe medical guidance. The finding has exposed a central tension in healthcare AI: how to deploy these systems safely when the underlying technology remains vulnerable to manipulation.
What the security researchers found
Mindgard AI, a London-based cybersecurity firm, tested Doctronic's chatbot system in January. The researchers showed they could extract the AI's hidden instructions and then feed it false information to generate unsafe outputs.
In one example, they cited a fabricated regulatory body and fake guidance. The AI then said it would triple the standard dose of Oxycontin. The researchers also got the system to produce instructions for illegal drugs.
The vulnerability stems from how large language models work. These systems are designed to be helpful and cannot truly verify whether information is real or fake. Once the researchers revealed the AI's knowledge cutoff date (June 2024), they fed it "new guidance" from made-up authorities released after that date. The system accepted it.
Peter Garraghan, Mindgard's founder, said this type of manipulation is a fundamental flaw in large language models across the industry. But he acknowledged the stakes are higher in healthcare than in other domains.
Doctronic and Utah's response
Doctronic's co-CEOs said Mindgard didn't test the actual system deployed in Utah. The production model works differently: it can only renew medications already in a patient's records, checks dosages against external databases, and automatically escalates unusual behavior to a physician.
Matt Pavelle, co-CEO, said the Utah system cannot authorize new prescriptions, renew controlled substances, or modify treatment plans. It also uses a restricted formulary of 190 medications. Even if the chatbot could be tricked into saying it would increase a dose, the underlying code prevents it from actually doing so.
"It is absolutely impossible for the chatbot to change the rest of the code to modify a prescription," Pavelle said.
Utah's Office of AI Policy said it was aware of these risks before launching the pilot. The program includes layered safeguards, physician oversight, and real-time monitoring through what Doctronic calls a "guardian" system - an additional AI layer that detects risky behavior.
The real-world problem the pilot addresses
About half of people with heart disease or diabetes don't stick to their medication plans. This leads to preventable complications and higher costs. An estimated 125,000 preventable deaths occur annually in the U.S. due to medication nonadherence.
Much of this stems from friction in the system - delays in getting refills, paperwork requirements, and administrative burdens on clinicians. Automating routine refills for stable patients could remove a barrier without requiring clinical judgment.
Dr. Thomas Savage, an internal medicine physician at Doctronic, said his team reviews every patient interaction to ensure the system works as intended. He sees the tool as addressing a real clinical problem when deployed with proper constraints.
"There are a lot of tasks that physicians do where we just need to find the contained box that is appropriate for using these technologies," Savage said.
The unanswered question
Mindgard's report raises a policy question that neither side fully resolves: whether developers, providers, and regulators are exercising sufficient diligence as they deploy medication systems without a human in the loop.
Doctronic and Utah maintain their safeguards are sufficient. Both organizations say they see no reason to slow the rollout until real-world evidence suggests otherwise.
Garraghan noted that while prompt-manipulation vulnerabilities are known across the industry, healthcare is a domain where such risks matter more than in other applications. The question now is whether the additional safeguards Doctronic built around its AI are enough to prevent harm in practice.
For healthcare professionals managing patient care, understanding how these systems work - and their actual limitations - will matter as more AI tools enter clinical workflows.
Learn more about AI for Healthcare and Generative AI and LLM vulnerabilities to stay informed on how these technologies are being deployed in your field.
Your membership also unlocks:
