LLMs as public-facing medical assistants: strong on tests, weak with real users
A large UK study found that large language models (LLMs) acting as AI medical assistants did not reliably help the public identify conditions or decide when to seek care. The gap between benchmark performance and real-world safety was clear: high scores didn't translate into better decisions by lay users.
For healthcare teams considering AI triage or patient self-care tools, the message is direct: technical accuracy alone is not a safety strategy.
Why these assistants draw attention
Healthcare is stretched. Staffing shortages and rising demand create interest in tools that can extend access. LLMs now post near-perfect results on medical-exam-style benchmarks, so expectations are high.
But exams don't test messy reality. Patients share partial details, interpret advice through personal filters, and face anxiety, bias, and time pressure. That's the fault line the study probed.
What the study tested
- Participants: 1,298 UK adults, randomized.
- Tasks: One of ten doctor-designed scenarios; identify possible conditions and choose a disposition from "stay home" to "call an ambulance."
- Arms: GPT-4o, Llama 3, Command R+, or a control group using any source they preferred.
When models were tested alone, they correctly identified the underlying condition in 94.9% of cases and the disposition in 56.3% on average. But when real users interacted with the same tools, performance collapsed: users identified relevant conditions in fewer than 34.5% of cases and chose the correct disposition in fewer than 44.2%-no better than the control group.
Why? Participants often gave incomplete information, missed prompts, or misread the responses. The underlying models could produce the right answer, but the human-system interaction broke down.
What this means for clinical practice
- High benchmark scores do not predict safer decisions by the public.
- Unsupervised use may give false reassurance and fail to reduce risk.
- Human-centered design and rigorous user testing are non-negotiable before deployment at scale.
Practical steps for safer deployment
- Structure the intake: Use guided questions and forced-choice inputs for red flags (e.g., chest pain, dyspnea, neuro deficits, sepsis signs) before any open-ended chat.
- Make safety the default: If critical data are missing, escalate disposition rather than "wait and see." Surface clear thresholds for self-care vs. urgent vs. emergency.
- Use plain language: Target short, direct sentences and concrete next steps. Include "why this matters" to build appropriate caution without panic.
- Demand clarifying follow-ups: The assistant should ask targeted questions when uncertainty is high, not proceed on thin inputs.
- Localize pathways: Map outputs to local services (urgent care, GP, pharmacy, mental health lines). Reduce the gap between advice and action.
- Design for misunderstanding: Add teach-back checks ("Here's what I think you said… Is that right?"). Offer examples of concerning symptoms users might have missed.
- Guardrails and audit: Block speculative diagnoses when confidence is low; log near-miss and adverse-signal events; review transcripts for quality and equity.
- Measure human outcomes, not model scores: Track disposition accuracy, time to appropriate care, avoidable ED visits, and user comprehension.
- Keep clinicians in the loop for higher-risk flows: Use AI as a copilot or pre-triage assistant with clear handoffs to licensed staff.
What to test before scaling
- Comprehension gain: Do lay users actually understand the condition and the next step better than with usual sources?
- Disposition accuracy and sensitivity: Especially for time-critical conditions.
- Equity: Performance across age, literacy, language, disability, and socioeconomic groups.
- Behavior change: Do users follow-through on the recommended action?
- Safety signals: False reassurance rates, delayed care, near misses.
Where LLMs may fit now
- Clinician-facing summarization and drafting (with review).
- Structured patient messaging templates that prompt for key symptoms and red flags.
- Educational handouts written at clear reading levels, localized to services.
Public-facing triage without strong interaction design and testing is not ready. Use cases that keep clinicians in the loop and reduce low-value work are a safer starting point.
Policy and governance pointers
- Require user-centered evidence (not just benchmarks) before approval and procurement.
- Mandate ongoing post-deployment monitoring and clear intended-use labeling.
- Align with national guidance on self-care and urgent care access to avoid conflicting advice.
For UK teams, see trusted resources such as NHS 111 and the NICE Evidence Standards Framework for Digital Health Technologies.
Bottom line
LLMs can know the right answer and still fail the real test: helping a worried person make a safe choice. Until systems prove they improve lay decision-making across diverse users, keep them supervised, structured, and measured.
Reference
Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine (2026). DOI: 10.1038/s41591-025-04074-y
If your team is building or governing clinical AI, you may find practical upskilling resources here: AI courses by job.
Your membership also unlocks:
