Harvard Study: AI Diagnosed Emergency Room Patients More Accurately Than Doctors
OpenAI's o1 model identified correct or near-correct diagnoses in 67 percent of emergency room cases, outperforming human physicians who achieved 50-55 percent accuracy. The findings, published in Science by researchers at Harvard Medical School and Beth Israel Deaconess Medical Center, suggest AI for Healthcare may soon assist doctors in real-world clinical settings.
The study tested 76 patients admitted to an emergency room. Two internal medicine physicians diagnosed each patient alongside Generative AI and LLM models. Independent physicians evaluated all diagnoses without knowing their source.
How the Test Was Structured
At the initial triage point-when minimal information exists beyond vital signs and basic demographics-AI models outpaced human doctors. The o1 model's accuracy rose to 82 percent when additional patient data became available, compared to 70-79 percent for physicians.
In a separate test involving 46 human doctors and the two AI systems analyzing five clinical cases, the models scored 89 percent versus 34 percent for doctors using conventional search tools.
Researchers presented the same patient data available in electronic medical records at the time of diagnosis. No information was pre-processed or cleaned for the AI systems.
What Doctors Are Saying
Ewen Harrison, co-director of the University of Edinburgh's centre for medical informatics, said the systems "are starting to look like useful second-opinion tools for clinicians, particularly when it is important to consider a wider range of possible diagnoses."
Kristen Panthagani, an emergency physician, offered a different perspective. "If we're going to compare AI tools to physicians' clinical ability, we should start by comparing them to physicians who actually practice that specialty," Panthagani said. She noted that an ER doctor's primary goal is identifying life-threatening conditions, not guessing a final diagnosis.
Real Limitations Remain
The study evaluated only text-based patient records. It did not assess how AI models interpret visual cues like a patient's appearance or level of distress-factors physicians routinely use.
Current AI models are prone to hallucinations and errors. No formal accountability framework exists for AI-assisted diagnoses, and the study did not evaluate performance with elderly patients or non-English speakers.
Doctors also worry about deferring to AI without independent thinking. Most patients still expect human physicians to guide them through critical decisions.
Adoption Is Already Underway
Nearly one in five US physicians already use AI to assist diagnosis, according to the American Medical Association. In the UK, 16 percent of doctors use AI tools daily for clinical decision-making, a Royal College of Physicians survey found.
Arjun Manrai, a lead author of the Harvard study, said the findings do not mean AI replaces doctors. "I think it does mean that we're witnessing a really profound change in technology that will reshape medicine," he said. Co-author Adam Rodman expects a "triadic care model" where doctors, patients, and AI systems work together.
Your membership also unlocks:
