A commercial radiology AI algorithm finds more brain aneurysms than radiologists working alone, but its value shifts sharply depending on where the patient is being treated, according to a Neiman Health Policy study published Monday in the Journal of the American College of Radiology. The prospective study of nearly 4,000 CT angiography scans found AI boosted aneurysm detection by 39% overall - a gain driven almost entirely by emergency and inpatient cases, while outpatient use produced more false alarms than correct catches.
Intracranial aneurysms affect roughly 3% to 4% of the population. Missing one can be catastrophic. Yet spotting these ballooning blood vessels on CT scans remains difficult, even for experienced radiologists dealing with fatigue or hunting for small, subtle lesions. Deep learning tools have been promoted as a solution, but evidence from real clinical environments has been thin.
The study, conducted at Northwell Health in New Hyde Park, New York, ran Aidoc's FDA-cleared algorithm in "shadow mode" - processing scans alongside radiologists without influencing their decisions. Of the 3,954 consecutive scans collected in late 2023, about 5% were positive for aneurysm. Radiologists and the AI agreed in more than 96% of cases. When disagreements arose, independent neuroradiologists reviewed the images to establish ground truth.
Where AI delivered - and where it didn't
AI identified 55 true-positive aneurysms that human readers missed, corresponding to the 39% relative increase in detection. Radiologists, however, were more precise: when they called a case positive, they were right 93% of the time, compared to 78% for the algorithm. Both were similarly effective at ruling out aneurysms when none existed.
The performance gap across care settings was stark. In inpatient settings, the algorithm found 18 additional aneurysms while generating only 7 false positives. Emergency department results were also favorable. Outpatient scans told a different story - just four additional detections, with more false positives than true ones.
"A likely explanation is that higher-acuity inpatient and emergency settings involve more clinically complex examinations, creating additional opportunities for AI to provide value by serving as a complementary detection tool alongside radiologist interpretation," said Matthew Barish, MD, Northwell Health's vice chair of radiology informatics.
Radiologists caught what AI missed, too
The relationship was not one-sided. Radiologists identified 30 true-positive aneurysms that the AI overlooked. And 46 of the algorithm's solo findings turned out to be false positives. The authors stressed that AI functioned best as a second reader, not a replacement - augmenting human judgment rather than substituting for it.
Elizabeth Rula, PhD, executive director of the ACR-backed policy institute, said the results point to a need for ongoing scrutiny after deployment. "The findings demonstrate why healthcare organizations should evaluate AI based on how it improves physician performance and patient care in real-world use, not solely on results achieved in its original testing environment," she said. "This study shows that AI can deliver meaningful clinical value by helping radiologists find additional aneurysms while also revealing important differences in performance across care settings."
Why this matters for healthcare professionals
The study makes clear that an algorithm's label claims tell only part of the story. A tool cleared by regulators can still behave differently on your outpatient floor than in your emergency department. For radiology departments and hospital systems investing in AI for healthcare, the takeaway is straightforward: vendor performance metrics from controlled testing environments do not predict how the software will perform on your patient mix. Post-implementation monitoring - tracking true positives, false positives, and setting-specific accuracy - is not optional. It is the only way to know whether the tool is helping or adding noise.
Your membership also unlocks:
