AI news ·
Benchmark finds most AI models score below zero on factual accuracy
Only three large language models scored above zero on a new factual accuracy benchmark, meaning most still get more answers wrong than right. The top model reached just 4.8 on a -100 to 100 scale, while the lowest hallucination rate was 4%.
Only three large language models managed to score above zero on a rigorous new factual accuracy benchmark, revealing how even the most advanced AI systems still produce more wrong answers than right ones across many economically important domains. The AA-Omniscience v1.1 benchmark, developed by Artificial Analysis, tested 3,000 questions drawn from authoritative academic and industry sources across six domains: business, humanities and social sciences, science, engineering and mathematics, health, law, and software engineering.
How the benchmark measures knowledge reliability
The AA-Omniscience Index scores models on a scale from -100 to 100. A score of zero means a model answers questions correctly as often as it gets them wrong. The metric rewards accurate answers, penalizes hallucinations, and imposes no penalty when a model refuses to answer or admits it does not know. This design explicitly tests whether models can recognize gaps in their own knowledge rather than guessing.
According to the accompanying paper published on arXiv, "Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero." The researchers, Declan Jackson, William Keating, George Cameron, and Micah Hill-Smith, said these results "reveal persistent factuality and calibration weaknesses across frontier models."
Performance varies sharply by domain
No single research lab dominated all categories. Models from three different labs led across the six domains tested. The paper's authors said this variability "suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important."
On hallucination rate - defined as the proportion of incorrect answers out of all non-correct responses - Gemini 4 Argon scored lowest at 4%, followed by Gemini 4 Carbon at 11% and Command A+ at 14%. A low hallucination rate means the model tends to stay silent rather than fabricate an answer when uncertain.
Accuracy scores, which measure the proportion of correctly answered questions regardless of whether the model attempts a response, were led by Claude Fable 5.1 at 68%, with several configurations of Claude models occupying the top spots.
Question design and domain coverage
The benchmark's questions were generated automatically using an LLM-based question generation agent to ensure unambiguity and factual precision. Topics span 42 economically relevant subdomains, from JavaScript and neurology to microeconomics. For example, one software engineering question asks: "In ECMAScript 2025, what JavaScript string literal does RegExp.escape(' ') return?" A health question probes: "In focal seizures, which specific anatomical region of the insular cortex is implicated by electrical stimulation studies in the generation of gustatory auras?"
The v1.1 release trims the question set to 3,000 from the original 6,000 used in the v1 dataset described in the paper. All evaluations are conducted independently by Artificial Analysis.
Why this matters for science and research professionals
For researchers and technical professionals who rely on AI tools for literature review, data interpretation, or domain-specific queries, the benchmark offers a concrete way to assess which models are least likely to fabricate information in your field. A model that scores well on general benchmarks may still hallucinate frequently in law, health, or engineering. The domain-level breakdowns make it possible to select a model based on factual reliability in the specific subject area you work in, rather than relying on aggregate performance scores that obscure these differences. Professionals building internal knowledge tools or research assistants should test models against domain-relevant benchmarks like this one before deployment.