AI news ·
New medical AI benchmark exposes gap between single-task accuracy and coherent patient care in nasopharyngeal carcinoma
Even the strongest medical AI models failed to keep diagnoses and care plans consistent for the same patient, with the top multimodal models reaching at most 23% strict CarePath success.
A new clinical benchmark called NPCBench reveals that even the strongest medical AI models struggle to maintain consistent, accurate decisions across an entire patient care pathway. The evaluation, focused on nasopharyngeal carcinoma, tested 30 models and found that none could reliably link correct diagnoses with correct care plans for the same patient over time.
Current medical benchmarks typically test isolated skills - answering a factual question, interpreting a single scan, or solving a static case vignette. These tests score individual answers but reveal little about whether a model can integrate subspecialty knowledge with multimodal evidence and track a patient's condition throughout a course of care. NPCBench was designed to close that gap.
How the benchmark mirrors clinical training
NPCBench structures its evaluation around an apprenticeship model, mirroring how specialists actually train. It moves from guideline and foundational knowledge to case planning, longitudinal review, CarePath integration, and open-ended consultation. The benchmark contains 4,777 component and consultation items alongside a 100-patient CarePath cohort with 417 scored diagnostic and care tasks.
The episode-level evaluation tests whether diagnostic and care answers are jointly correct within a care pathway. By linking stepwise tasks to patient-level care trajectories, NPCBench evaluates whether models ground decisions in visual evidence, interpret changes in patient state, and meet clinical requirements during consultation.
A composition gap in model performance
Across 30 evaluated models, the researchers observed what they describe as a composition gap. Strong performance on individual tasks did not translate into jointly correct diagnostic and care answers for the same patient. Among 21 multimodal models, strict CarePath success reached at most 23%.
Paired controls and physician calibration further examined evidence grounding, longitudinal revision, and consultation judging. The results point to a structural weakness: models can perform well on decontextualized questions but falter when required to maintain temporally coherent reasoning across a patient's care journey.
Why this matters for healthcare and research professionals
For insurers and health systems evaluating AI tools, NPCBench provides a testbed that moves beyond static question-answering. It measures whether a model's clinical reasoning holds together over time - a capability directly relevant to prior authorization, care pathway compliance, and clinical decision support. A model that answers individual questions correctly but produces contradictory care plans for the same patient introduces risk that static benchmarks won't catch.
For research teams building or procuring medical AI, the benchmark offers a framework for assessing evidence-based, temporally coherent reasoning in oncology. The low strict success rates - even among multimodal models - suggest that current architectures need substantial work before they can reliably support longitudinal clinical workflows. Professionals working with AI for Healthcare Courses or AI Research Courses may find the benchmark's apprenticeship-oriented design relevant to their own evaluation pipelines.