Healthcare Systems Need Clear Accountability Structures to Deploy AI Safely
Health systems rolling out clinical AI tools must embed accountability into their workflows before deployment, not after. A model's technical accuracy means little if no one knows who responds when it flags a risk or what happens when it fails.
That's the core argument from Azizi Seixas, an associate professor of psychiatry and behavioral sciences at the University of Miami Miller School of Medicine. In recent work on AI for Healthcare, he and colleagues have outlined a framework called SAFE-standards, accountability, fit to workflow, and evaluation-that treats AI governance as operational infrastructure rather than abstract policy.
The Infrastructure Problem
Most healthcare organizations treat AI deployment like a software purchase: buy the tool, run a pilot, measure results. This approach fails because it ignores the systems around the model.
"The model is only one part of the safety story," Seixas said. "The infrastructure around the model determines whether that model becomes useful, or if it should be ignored, or if it's just too dangerous."
The SAFE framework addresses four elements:
- Standards: What evidence must a tool meet before touching patient care?
- Accountability: Who owns performance, response, and failure? This must be assigned before deployment.
- Fit to workflow: Where does the AI appear in clinical processes? Who sees it and acts on it?
- Evaluation: Who monitors performance drift, bias, overrides, near misses, and outcomes over time?
Without these structures, a technically sound model can create operational chaos. If an AI tool identifies a high-risk patient, the response chain must be explicit: Does a nurse respond? A case manager? A physician? If that's unclear, the model is unsafe regardless of accuracy.
Governance Reduces Uncertainty
Traditional AI governance frameworks-and newer ones like FAIR AI-all point to the same conclusion: accountability requires practical structures, not principles on paper.
Pre-deployment review, clear role assignments, continuous monitoring, and the ability to retrain or retire a model when performance changes are not optional. They're the foundation of safe clinical AI.
"A model without governance is really a guess with authority," Seixas said. "Governance reduces the uncertainty in AI implementation. Workflow reduces confusion, and monitoring reduces harm."
What Patients Need to Know
Transparency must be meaningful, not exhaustive. Patients don't need to understand algorithms. They need clarity about where AI is used and what it does.
Seixas outlined a framework called CLEAR for patient communication:
- Context: Where is AI being used-outpatient, inpatient, at home?
- Limits: What can the tool do and what can't it do?
- Escalation: When does a human take over?
- Accountability: Who remains responsible throughout?
- Rights: What can patients ask, challenge, or refuse?
If AI helps draft a patient message that a clinician reviews before sending, patients don't need to see the algorithm. They should understand that AI assisted and a clinician remained accountable.
"Transparency is not about exposing complexity," Seixas said. "It's about preserving trust."
Real-World Accountability: The Mayo Clinic Model
Mayo Clinic treats AI as an enterprise capability requiring governance and validation, not a one-time technology purchase. Their approach emphasizes structured review, clinical assurance, implementation discipline, and continuous evaluation.
Most organizations fail here. One department buys a tool, another tests it, and nobody owns what happens after deployment. Mayo does the opposite: it assigns institutional responsibility for standards, committees, monitoring, and the authority to pause or retire models when evidence changes.
"Governance must be operational," Seixas said. "It has to live in workflows and approvals and quality review and executive accountability."
The Core Message for Leaders
AI for Executives & Strategy in healthcare requires a shift in thinking. AI will not scale because it's powerful. It will scale because it becomes trustworthy.
That means operationalizing accountability in clinical workflows. The protagonist of your system should never be AI. The protagonists are the workflows and the patients. AI is the tool.
"Safety means that the right model for the right patient in the right workflow with the right oversight is always the right choice," Seixas said.
Your membership also unlocks:
