Misuse of AI chatbots tops ECRI's 2026 health tech hazards: what healthcare leaders should do now
AI chatbots have reached the front desk, the clinic, and the patient portal. ECRI's 2026 Top 10 Health Technology Hazards puts misuse of AI chatbots at #1 - a clear signal that healthcare is out over its skis with tools that sound expert but aren't regulated or validated for clinical use.
Large language model (LLM) chatbots like ChatGPT, Claude, Copilot, Gemini, and Grok generate fluent answers by predicting the next word, not by reasoning about physiology or patient context. They speak with confidence even when they're wrong. More than 40 million people reportedly use ChatGPT daily for health information, and many ask clinical questions. That mix can fuel misinformation, risky decisions, and delayed care.
Why this matters
ECRI experts have seen chatbots suggest incorrect diagnoses, unnecessary testing, and questionable products - and even "invent" anatomy while sounding authoritative. In one case, a bot said placing an electrosurgical return electrode over the shoulder blade was acceptable. It isn't. Following that advice could cause burns.
As care access tightens due to higher costs or closures, more patients may lean on chatbots as a stand-in for clinicians. Bias in training data can also skew responses, deepening disparities. As ECRI notes, AI reflects the data it learns from - biases included.
Bottom line: These tools can assist with communication and admin work, but they require guardrails. Clinical judgment is nonnegotiable.
Practical guardrails for clinicians
- Never accept clinical recommendations from a chatbot without independent verification from trusted sources or specialists.
- Use chatbots for low-risk tasks: drafting patient education to be reviewed, summarizing general literature, or creating after-visit communication for clinician sign-off.
- Avoid using chatbots for diagnosis, triage, treatment plans, device placement, dosing, or urgent decision support.
- Cross-check content against current guidelines or institutional protocols, especially for high-acuity or high-variability conditions.
- Watch for confident but vague statements, fabricated citations, or made-up anatomy and contraindications.
- Keep PHI out of systems not approved by your organization. Use enterprise versions with BAA and logging when possible.
- Label patient-facing content created with AI and include clear instructions to contact a clinician for medical concerns.
- Document when AI drafts contributed to communications, and ensure a licensed clinician reviews and finalizes.
Governance moves to implement this quarter
- Stand up an AI governance committee spanning clinical, safety, risk, IT, legal, and patient advocacy.
- Inventory every AI and automation tool in use. Classify by risk: low (admin), medium (education), high (clinical decision influence).
- Define approved use cases, red lines, and human-in-the-loop rules. Publish quick-reference guides for staff.
- Procure only tools with clear model provenance, update cadence, safety testing, audit logs, and incident response commitments.
- Train clinicians on limitations, prompt hygiene, bias pitfalls, and verification workflows. Recurrent refreshers matter.
- Implement auditing: random sample reviews, outcome tracking, bias checks on key cohorts, and error reporting channels.
- Set PHI and cybersecurity controls: access management, data retention limits, and vendor BAAs.
- Create a patient communication standard: disclaimers, escalation pathways, and clear guidance for urgent symptoms.
What ECRI is saying
"Medicine is a fundamentally human endeavor. While chatbots are powerful tools, the algorithms cannot replace the expertise, education, and experience of medical professionals," said Marcus Schabacker, MD, PhD, ECRI's president and CEO. The organization urges disciplined oversight, clear guidelines, and a realistic view of AI's limits.
ECRI will host a webcast on January 28 to outline risks and safer practices. For the executive brief and membership details, visit ECRI.
The Top 10 Health Technology Hazards for 2026
- Misuse of AI chatbots in healthcare
- Unpreparedness for a "digital darkness" event, or a sudden loss of access to electronic systems and patient information
- Substandard and falsified medical products
- Recall communication failures for home diabetes management technologies
- Misconnections of syringes or tubing to patient lines, particularly amid slow ENFit and NRFit adoption
- Underutilizing medication safety technologies in perioperative settings
- Inadequate device cleaning instructions
- Cybersecurity risks from legacy medical devices
- Health technology implementations that prompt unsafe clinical workflows
- Poor water quality during instrument sterilization
If you're leading safety, here's your next step
Pick two high-impact workflows where chatbots are creeping in - patient education and staff communication are common - and build a tight review loop around them. Measure error rates, equity impacts, and turnaround time. Expand only when you can show clinical and safety gains.
If your teams need structured upskilling on safe, verifiable AI use, see AI courses by job for role-based options.
Your membership also unlocks:
