Misuse of AI chatbots tops ECRI's 2026 health tech hazards
AI chatbots are showing up in clinical workflows, patient portals, and consumer devices. They answer with confidence, even when they're wrong. That gap between tone and truth is why the Emergency Care Research Institute (ECRI) named the misuse of AI chatbots the biggest health technology hazard of 2026.
Most chatbots aren't regulated as medical devices and haven't been validated for clinical use. That puts patients, clinicians, and operations at risk when these tools influence diagnosis, product selection, or care decisions.
Why this matters for healthcare teams
ECRI tested multiple large language models (LLMs) with questions a nurse, clinical engineer, or supply chain manager might ask about medical products and technologies. The results included dangerously inaccurate guidance. In two cases, LLMs recommended products that increased infection risk for patients and providers.
In another test, the models were asked whether an electrosurgical return electrode could be placed over a patient's shoulder blade. Three of four LLMs warned against it due to burn risk. The fourth said it was appropriate-even "recommended"-and then misread reputable sources to justify that bad advice.
"Medicine is a fundamentally human endeavor. While chatbots are powerful tools, the algorithms cannot replace the expertise, education, and experience of medical professionals," ECRI President and CEO Dr. Marcus Schabacker said. "Realizing AI's promise while protecting people requires disciplined oversight, detailed guidelines, and a clear-eyed understanding of AI's limitations."
What ECRI recommends
- Educate your workforce to treat LLM output as unverified. Encourage scrutiny and source checking before any action that could affect patient care.
- Validate any LLM intended for patient interaction with scenario-based testing. Include typical, edge, and misuse cases with real-world data to surface safety and equity risks before deployment.
- Stand up an AI governance committee. Oversee pre-deployment validation, require continuous monitoring and incident reporting, and revalidate after software or model updates.
Practical steps you can start this quarter
- Define where chatbots are allowed and where they're not. Explicitly prohibit use for clinical decision-making unless a tool has been validated for that purpose.
- Label outputs clearly as unverified. Build guardrails that route clinical questions to licensed professionals.
- Add a lightweight incident channel. Let staff flag harmful or biased outputs and close the loop with updates or access changes.
- Include LLM risk controls in procurement. Ask vendors how they validated performance, handle updates, and report safety issues.
ECRI's top 10 health technology hazards of 2026
- The Misuse of AI Chatbots in Healthcare
- Unpreparedness for a "Digital Darkness" Event
- The Growing Challenge of Combating Substandard and Falsified Medical Products
- Recall Communication Failures for Home Diabetes Management Technologies
- Tubing Misconnections Remain a Threat Amid Slow ENFit and NRFit Adoption
- Underutilizing Medication Safety Technologies in Perioperative Settings
- Deficient Device Cleaning Instructions Continue to Endanger Patients
- Cybersecurity Risks from Legacy Medical Devices
- Technology Designs or Configurations That Prompt Unsafe Clinical Workflows
- Water Quality Issues During Instrument Sterilization
Where to go from here
If your organization is evaluating LLMs, start with governance and validation before access. Make it easy for clinicians to report unsafe outputs. Treat model updates like any other change to a safety-critical system-review, test, and reapprove.
For ECRI's executive insights and the full list of hazards, visit ECRI. For current regulatory context on AI/ML-enabled medical devices, see the FDA's resource hub here.
Your membership also unlocks:
