Included Health publishes framework for clinically governed AI in healthcare

Included Health published a peer-reviewed framework for deploying patient-facing generative AI in healthcare, detailing a clinician-in-the-loop model that reduced unnecessary handoffs for standard-risk questions by 65% during a seven-week pilot. The paper, which appears in the July 2026 issue of NEJM Catalyst Innovations in Care Delivery, arrives as more patients turn to public AI tools for medical answers - often without clinical safeguards.

The model is built around four core principles: cross-functional governance with clinical oversight, proactive risk analysis and pre-launch testing, a three-tier risk classification system, and continuous clinician review of interactions. During the pilot, the company deployed the AI assistant to half its patient population and manually reviewed every clinical interaction daily.

A three-tier system for risk

The framework classifies patient questions into three categories: standard, high-risk, and emergency. Standard-risk queries - routine questions about symptoms, medications, or care instructions - receive AI-generated guidance. Higher-risk situations escalate to qualified clinicians. The structure embeds safety directly into the workflow, with clear escalation paths for situations the AI is not designed to handle.

"At Included Health, we believe healthcare AI should be held to a higher standard than general-purpose tools," said Ami Parekh, MD, chief health officer at Included Health. "People deserve timely support, but they also need a model they can trust to provide accurate, safe guidance."

What the pilot found

Clinical reviewers audited all interactions daily over the seven-week period and found the approach maintained a high level of safety. The 65% reduction in unnecessary handoffs meant more patients received immediate, clinically appropriate answers without waiting for a human clinician. Patient experience scores remained on par with the comparison group.

The paper adds to a growing body of work on AI for Healthcare that emphasizes building risk recognition and clear communication of limitations into products from day one. Ankoor Shah, MD, vice president of clinical excellence at Included Health, said the framework "offers a practical blueprint for how healthcare organizations can deploy patient-facing AI with safety, transparency, and human oversight built in from the start."

Why this matters for healthcare professionals

For clinicians and healthcare administrators, the Included Health framework provides a concrete reference point for evaluating or building AI tools internally. The three-tier risk model and mandatory daily clinical review move beyond abstract governance principles into operational specifics - specifying what pre-launch testing looks like, how interactions get audited, and when escalation should kick in. As health systems face pressure to adopt AI quickly, the paper's emphasis on stress-testing guardrails before launch and reviewing 100% of interactions afterward offers a clear starting point for teams building their own governance structures.