Human Review Emerges as Critical Check on AI Bias in Clinical Settings
A new study from UC Davis researchers shows that keeping clinicians and other experts in the decision-making loop significantly reduces bias in AI for Healthcare systems. The work, published in Social Science and Medicine, tested an approach where interdisciplinary panels reviewed how AI models reached their conclusions before those systems were used in patient care.
The findings matter because AI tools now read medical images, predict patient risks, and monitor conditions remotely across U.S. health systems. But these systems can fail when trained on incomplete or skewed data - producing results that sound plausible while missing critical context.
How Bias Enters AI Medical Systems
AI models learn patterns from historical data. If that data reflects existing healthcare disparities - say, fewer diagnostic tests for certain patient populations - the AI will encode those gaps as if they were medical facts.
Without human oversight, AI systems may produce outputs that are incomplete, biased, or unsafe, said Courtney Lyles, director of the UC Davis Center for Healthcare Policy and Research and lead author of the study.
The problem runs deeper than bad data. Lyles said clinicians and researchers must understand the social and structural forces that shape health data itself. An AI pattern might reflect how patients interact with medical devices, differences in how the data was collected, or structural inequities - not actual clinical differences.
The Panel Approach: Diverse Expertise Catches What Algorithms Miss
The UC Davis study brought together experts from medicine, epidemiology, behavioral science, engineering, and data science. The team also included community members and patient advocates. Each group reviewed the same AI model outputs through their own lens.
When the AI highlighted a pattern, the panel asked three key questions:
- Could this pattern come from differences in how the dataset was collected?
- Is this result tied to how patients use medical devices?
- Does this reflect a social or structural issue rather than a medical one?
This process exposed what researchers call "shortcut features" - patterns that look meaningful but actually reflect bias baked into the data.
Patient advocates and community members brought lived experience that traditional experts often miss. Their input helped ensure the AI tools would actually serve the populations they were designed for.
Explainable AI as a Foundation
The study relied on explainable AI (XAI) tools that show why a model made specific predictions. Without this transparency, human review becomes guesswork.
XAI peels back the AI's logic so clinicians and researchers can see which factors the model weighted most heavily. That visibility is what makes human oversight possible.
Real-World Application at UC Davis Health
UC Davis Health has embedded this approach into clinical practice. The health system established an AI governance committee that reviews new models before deployment. A separate team led by population health director Reshma Gupta developed a formal process for identifying and addressing potential bias when building AI systems that predict patient readmission risk.
The health system also piloted an AI Scribe program in 2024. The tool generated clinical notes from audio recordings of patient visits, saving physicians transcription time. A pilot study published in the Journal of Medical Informatics Research found the AI-generated notes were free from significant errors 94.7% of the time.
But even at that accuracy rate, clinicians had to review every note. Those reviews caught and fixed the small percentage of errors - a reminder that human oversight remains essential.
Building Teams Before Building Systems
Lyles said health systems can implement this approach by assembling interdisciplinary teams before developing or deploying AI systems. The team should include data scientists, clinicians, patient advocates, and domain experts relevant to the specific application.
This structure improves both technical accuracy and trust between the groups that must work together: data scientists, clinicians, patients, and communities.
Private companies developing health tech are increasingly receptive to this model. They want academic expertise that is practical and moves them closer to real-world implementation. Public-private partnerships like UC S.O.L.V.E Health Tech, which connects UC researchers with digital health companies, create structured spaces for this collaboration.
The message is straightforward: as AI becomes routine in clinical care, algorithms alone are not enough. Human judgment, informed by diverse expertise and supported by explainable AI tools, is the mechanism that catches bias before it reaches patients.
Your membership also unlocks:
