An AI Sepsis Tool Failed Worse Than a Coin Flip. Here's Why Trust Matters in Healthcare
A hospital system deployed an AI tool designed to flag sepsis risk early. The system couldn't tell the difference between high-risk and low-risk patients. Its predictions were no better than random chance.
This wasn't a minor technical glitch. The failure cost the organization time, money, and something harder to rebuild: clinician trust.
Doctors won't use AI tools they don't trust. Patients won't follow recommendations from systems that have proven unreliable. When adoption stalls, the millions spent on development become sunk costs that derail entire digital strategies.
Where AI Breaks Down in Clinical Settings
AI hallucinations-confident-sounding but incorrect outputs-happen across industries. In healthcare, the stakes are different. A hallucination that produces a wrong diagnosis, triggers a false alert, or recommends incorrect treatment can harm a patient.
Data quality poses a separate risk. Training data can contain hidden problems:
- Sources may be unreliable or poorly validated
- The data may not represent the actual patient population using the tool
- Regional and contextual differences may be missing
When errors surface in clinical practice-say, a patient spots mistakes in AI-generated clinical notes-trust erodes fast. Often one or two visible failures are enough. Patients may stop following medical advice. The doctor-patient relationship, which depends on trust, weakens. Effective care becomes harder.
Governance Isn't Optional
Healthcare already operates under heavy regulation. Adding AI multiplies the complexity. Regulatory frameworks for AI are still developing and often lag behind the technology itself.
Healthcare organizations can't wait for regulators to catch up. They need to build trust into their AI systems through governance.
This means establishing a governance committee that is:
- Active - monitoring AI results over time and watching for hallucinations and data inconsistencies
- Watchful - checking that workflows still include adequate human input and aren't becoming over-reliant on automation
- Proactive - setting guardrails and policies before tools launch, not after failures occur
- Ongoing - treating governance as continuous, not a one-time project
Many organizations have created a chief AI officer role-someone dedicated solely to overseeing and maintaining AI tools as they operate. This person runs the governance committee and ensures it stays focused.
Why Governance Fails in Practice
Setting up effective governance sounds straightforward. In reality, organizations face several obstacles:
- Workflows are decentralized or unstructured, making unified governance difficult
- AI adoption moves faster than oversight can keep pace
- Different departments adopt AI independently, creating a shadow IT problem
Shadow IT is particularly acute in healthcare AI. Traditional tech upgrades-like new imaging systems-require major organization-wide projects. Equipment must be purchased, installed, and staff trained. Everyone knows what's happening.
AI tools have lower barriers to entry. A group of doctors dissatisfied with a hospital-approved AI scribe can adopt an alternative on their own. When this happens across multiple departments, monitoring becomes nearly impossible.
The Cost of Skipping Governance
The sepsis AI tool had merit. If doctors could identify high-risk patients early, they could intervene before conditions worsened. Better patient outcomes were possible.
Development outpaced governance. Problems that could have been caught and fixed weren't. The tool failed to deliver on its promise.
That failure has lasting consequences. Clinicians burned by one failed AI system are unlikely to trust the next one. Patients will be skeptical too. Healthcare organizations rolling out AI tools have limited chances to earn that trust.
For more on implementing AI effectively in healthcare settings, see our guide to AI for Healthcare. Understanding Generative AI and LLM issues like hallucinations is also essential for anyone overseeing these deployments.
Your membership also unlocks:
