Healthcare leaders reject autonomous AI. Hybrid intelligence wins on accuracy and trust
Hospitals and health systems are clear: AI should work with clinicians, not replace them. A new November 2025 market survey from Reaction Data, commissioned by Carta Healthcare, shows strong preference for "hybrid intelligence" - AI paired with clinical oversight - for safer, more reliable outcomes.
The message is consistent across findings. Fully autonomous, black-box AI is viewed as risky and insufficient for high-stakes clinical use. Human validation and clinician involvement are seen as essential for accuracy, safety, and adoption.
Key findings that matter for your organization
- 62.5% flagged misinterpretation of data as the top risk when AI operates without human oversight.
- Only 12.5% said autonomous AI has delivered the most value so far.
- 75% rely on human validation to ensure AI outputs are clinically relevant and trustworthy.
- 75% rated clinician involvement in AI design and deployment as "critically important."
- 50% said AI should augment human decision-making, not replace tasks outright.
As one leader put it: "good AI is not good enough." According to Brent Dover, CEO of Carta Healthcare, health systems want a force multiplier - AI for speed and scale, clinicians for judgment and care - integrated into existing workflows with clear validation steps.
Why this is the practical path forward
Clinical environments demand transparency, accountability, and the ability to audit how an output was produced. Hybrid intelligence answers that with embedded checkpoints, measurable quality, and clear ownership. It de-risks deployment while building trust with clinicians and patients.
Action plan: Build hybrid intelligence into your workflows
- Define oversight: Establish a cross-functional governance group (clinical, quality, data science, IT, compliance) with clear decision rights and escalation paths.
- Put clinicians in the loop: Require expert review for model outputs that touch patient care, registry submissions, coding, or quality reporting.
- Demand transparency: Select models and vendors that provide audit trails, versioning, rationale summaries, and drift monitoring.
- Measure what matters: Track accuracy, precision/recall, calibration, turnaround time, and clinician override rates. Tie metrics to safety and cost outcomes.
- Integrate, don't bolt on: Fit AI into existing EHR and registry workflows. Reduce clicks, minimize alert fatigue, and clarify "who does what" at each step.
- Set safety nets: Define high-risk scenarios that always require human review. Log all overrides and feed them back into continuous improvement.
- Protect data quality: Standardize abstraction rules, vocabularies, and data lineage. Poor inputs multiply downstream errors.
- Negotiate vendor guarantees: Require human validation options, quality SLAs, PHI controls/BAA, security attestations, and indemnification for model failures.
Where hybrid intelligence delivers fast wins
- Clinical data abstraction and registry submissions
- Quality measurement and reporting
- Care gap identification and documentation support
- Throughput and capacity planning with human QA review
Policy watch
Keep your governance aligned with external guidance. The FDA's perspective on AI/ML in Software as a Medical Device can inform lifecycle controls and change management. See the agency's resources here: FDA AI/ML-enabled medical devices.
Survey methodology
Reaction Data conducted a national online survey in November 2025 targeting healthcare IT and health system leaders. Respondents opted in based on role and expertise; only qualified responses were included.
About Carta Healthcare
Carta Healthcare provides AI-powered clinical data abstraction solutions rooted in a hybrid intelligence approach: advanced AI paired with expert clinicians. The company focuses on accurate, actionable data that helps reduce costs and improve efficiency for hospitals and health systems. Learn more at carta.healthcare.
Your membership also unlocks:
