Clinician Input Required: Human-Centered AI That Works in Real-World Care

Make Healthcare AI Human-Centered: Require Clinician Input

AI will improve care only if it works for the people who deliver it. That means building, testing, and deploying tools with clinicians at the table-then holding those tools accountable to real outcomes across diverse patient populations.

Three priorities should guide AI in healthcare: embed clinician involvement, use representative data with human-centered design, and maintain continuous oversight through post-market evaluation and outcomes-based contracting. These steps keep safety, equity, and effectiveness front and center.

Why this matters now

Health systems are adopting AI across triage, diagnostics, and workflow support. The FDA has already authorized many AI-enabled medical devices, especially in radiology, cardiology, and pathology. Without clinician input and representative data, adoption stalls, bias creeps in, and patient risk rises.

Federal directives emphasize safety, transparency, interoperability, and outcomes. This is the moment to hardwire clinician expertise into funding, standards, and evaluation to ensure AI delivers in real clinical environments.

• FDA: AI/ML-enabled medical devices
• ONC: TEFCA interoperability framework

Core problems to fix

• Workflow misfit and algorithm aversion: If tools don't match clinical practice, clinicians ignore them. Trust erodes fast after visible errors, even when average performance is strong.
• Interoperability gaps: AI that can't read and write to EHRs, imaging systems, and labs won't scale. Fragmented data leads to inefficiency and mistakes.
• Clinician deskilling: Prolonged reliance on automation can degrade skills, with performance dropping when AI is absent or wrong-especially in high-stakes settings.
• Embedded bias: Unrepresentative training data leads to uneven performance across patient groups, undermining equity and safety.

What good looks like

• Clinicians define the clinical tasks, usability needs, and handoffs between AI and human judgment.
• Models are validated on diverse populations with interpretable outputs and clear uncertainty signals.
• Integration is seamless: alerts at the right time, in the right place, with the right data.
• Continuous monitoring detects drift, bias, and safety issues-and triggers timely updates.
• Funding ties to clinical impact through outcomes-based contracting, not just tool delivery.

Plan of action

• 1) Require clinician involvement for AI used in care: ONC and FDA should issue guidance and tie eligibility to clinical roles and checkpoints across four stages: defining tasks and UI, validating interpretability and performance across populations, piloting in real workflows, and reviewing safety and bias metrics.
• 2) Incentivize with outcomes-based contracting: HHS, CMS, and AHRQ should fund projects that embed clinicians on design teams, run formal feedback loops, and demonstrate measurable gains (diagnostic accuracy, workflow efficiency, safety events avoided, equity metrics improved).
• 3) Set interoperability standards: ONC should extend guidelines to ensure AI tools can exchange data across EHRs and clinical systems using vendor-neutral APIs, FHIR/USCDI, and TEFCA-aligned participation-while protecting privacy.
• 4) Establish post-market surveillance: FDA and AHRQ should require ongoing, privacy-protected monitoring with bias audits, safety reporting, version tracking, and clinician/patient feedback loops to maintain accuracy and equity over time.

Clinician input without overload

• Use time-boxed checkpoints at key milestones instead of all-day committees.
• Stand up paid clinical advisory panels with clear decision rights.
• Pilot in live workflows with short in-EHR feedback prompts and weekly synthesis.
• Train clinical champions who can translate between bedside needs and data science.

Post-market surveillance that works

• Track diagnostic accuracy, error rates, override rates, near misses, turnaround time, and clinician satisfaction.
• Run scheduled bias audits across age, sex, race/ethnicity, language, disability, and social risk-publish summaries.
• Set drift alerts and performance thresholds that trigger review, rollback, or retraining.
• Maintain transparent versioning, audit logs, and change notes accessible to clinical leaders.

Interoperability must-haves

• Read/write via FHIR APIs and support USCDI data elements.
• Link to identity, attribution, and consent services across networks.
• Operate with vendor-neutral interfaces; no hard locks to a single EHR.
• Honor TEFCA participation for cross-organization data exchange and governance.

Where outcomes-based contracting fits

Outcomes-based contracting ties payment to real patient impact. It keeps everyone focused on results, not features.

• Example targets: improved sensitivity/specificity for defined conditions, time-to-diagnosis reduction, fewer avoidable admissions, improved throughput, documented equity improvements.
• Contract levers: holdbacks until targets are met, shared-savings for outperforming baselines, penalties or remediation plans for bias or drift.

Risk tiers matter

• High-stakes (oncology, cardiology, critical care): require clinician validation before action, strict interpretability, and rigorous surveillance.
• Lower-risk (minor conditions, administrative tasks): allow lighter oversight while maintaining feedback loops and safety monitoring.

Equity guardrails

• Use representative data and document known gaps.
• Set fairness metrics with thresholds and corrective actions.
• Enable patient and clinician reporting for suspected disparities.

What this delivers

• Safer decisions and fewer errors through clinician-validated workflows.
• Faster adoption because tools fit how care is delivered.
• Better outcomes and lower waste by tying funding to measurable impact.
• Stronger trust through transparency, bias audits, and continuous learning.

If your teams are building AI skills for clinical, quality, and informatics roles, explore curated learning paths by job to accelerate safe adoption: AI courses by job.

The path is straightforward: require clinician input, use representative data, ensure interoperability, monitor performance, and pay for outcomes. Do that, and AI becomes a reliable partner in care-improving safety, equity, and effectiveness where it counts most.