Healthcare leaders at AXS26 in Las Vegas called for stronger safeguards around artificial intelligence tools that influence patient care, including independent validation, ongoing monitoring for inaccurate or biased results, and clear requirements for human oversight. The push comes as hospitals and managed care organizations adopt AI for tasks ranging from clinical documentation to prior authorization, often with less regulatory scrutiny than the food served in hospital cafeterias.
"I often tell lawmakers that there is more regulation of the food in the hospital cafeteria than of AI tools used in the hospital," said Shawn Griffin, MD, president and CEO of URAC.
Validation gaps in clinical AI
Dr. Griffin said hospitals have no guarantee that an AI transcription product accurately documents a patient-clinician encounter, or that an AI-generated sepsis risk score has been adequately validated. He argued that some claims about AI's uniqueness compared with previous technologies are overblown.
For example, some contend that agentic AI generates new knowledge based on inputs it receives. More accurately, agentic AI can independently plan and carry out multistep tasks, including using tools, making decisions, and adjusting its approach. Generating "new knowledge" is not its defining feature. Conventional generative AI, in contrast, primarily produces content in response to prompts.
"Sometimes agentic AI is just a glorified phone tree," Dr. Griffin said, in which the AI software takes forking paths based on initial inputs. He added: "I am not against healthcare technology. I just want to make sure it is safe before we deploy it."
In September 2025, URAC launched a new accreditation program to recognize excellence in AI healthcare development and usage. The organization will announce its first recipients soon. Pharmacists should ask how an AI product was trained and tested, whether it performs reliably in their patient population, and how its recommendations will be reviewed before incorporating it into clinical or operational workflows, Dr. Griffin said.
AI in managed care pharmacy
Mitzi Wasik, PharmD, MBA, BCPS, senior vice president for practice advancement and member experience at AMCP, highlighted potential benefits of AI in core managed care areas such as medication adherence. AI tools can predict which patients are most likely to stop taking medications and help pharmacists tailor interventions like financial support or text reminders.
"A patient could have an adherence score of 90, but the pill bottles are in their medicine cabinet and the medicine is unused," Dr. Wasik said. "Even though we've had machine learning and predictive analytics to predict when patients will not take medications, AI has allowed this to be a lot more powerful."
AI tools can also analyze free-text notes in patient records to help case managers focus on overcoming roadblocks to adherence, such as cost or duplicative medications. Previous analytic models could sort out adherence barriers retrospectively at a population level. Today's tools may spot barriers before they emerge.
AI software can automate routine aspects of prior authorization, speeding up time to therapy for approved medications. This connects directly to revenue cycle work, where AI for Medical Billers is becoming a practical skill set. Dr. Wasik stressed that "a human has to stay involved" with prior authorization denials facilitated by AI. That could include ensuring the denial was not in error, reaching out to patients, and finding alternative medications that may meet patient needs.
Bias and equity risks
Dr. Wasik also warned that AI could inadvertently enable healthcare disparities. Many healthcare AI tools are trained on data that do not represent a region or nation's population. "There's bias in that data, which could deepen existing health inequities," she said.
For professionals evaluating AI adoption in clinical or operational settings, understanding both the capabilities and limitations of these tools is increasingly part of the job. Training programs focused on AI for Healthcare can help teams assess vendor claims and implement oversight processes that match the risks involved.
Why this matters for healthcare professionals
If you work in healthcare, the message from AXS26 is direct: ask vendors how their AI was trained, on what data, and how its outputs will be monitored. Whether you're a pharmacist reviewing adherence predictions, a case manager acting on AI-flagged patient notes, or a clinician whose documentation passes through AI transcription, the burden of safety checks currently falls on you. Accreditation programs like URAC's may eventually shift some of that burden, but until then, documented validation and human review are your responsibility.
Your membership also unlocks:
