AI news ·
Sakana AI's multi-layered review system catches 73% of core-claim errors in planted-contradiction benchmark
Sakana AI's multi-agent review system caught 73.43% of core-claim contradictions deliberately planted in academic papers, while the best alternative managed just 14.81%.
Sakana AI published a paper in TMLR that tests AI reviewers on a harder task than matching human scores: finding errors someone deliberately inserted into a paper. Their Multi-Layered Review (MLR) system caught 73.43% of core-claim contradictions when run across four reviews, while the best baseline system managed 14.81%. The team also released a Contradiction Benchmark of 1,164 planted errors across 257 papers from five academic venues.
The benchmark and MLR system both run on off-the-shelf API models - Claude Sonnet 4 and Claude Haiku 3.5 - with no GPU usage or fine-tuning. A single review costs about $0.47 and uses 189,062 input tokens, roughly half the tokens of the next best system. The code is available on request.
How the review system works
MLR uses three agents working across different Claude models. An Appendix Agent running Haiku 3.5 summarizes experiments and implementation details. An optional Literature Review Agent running Sonnet 4 searches the web to place the paper in prior work. The main Review Agent, also on Sonnet 4, runs a three-pass prompt chain inspired by Keshav's Three-Pass Approach.
Pass 1 produces a high-level outline. Pass 2 reads in detail and flags weaknesses, assumptions, and gaps. Pass 3 merges all agent outputs into a structured review with Strengths, Weaknesses, Questions, a Recommendation, a Score, and a To-Do list. The PDF is passed directly so figures and equations are preserved.
What the benchmark measures
The team collected 257 CC-licensed papers from ACL, AISTATS, CVPR, ICML 2025, and NeurIPS 2024. Gemini 2.5 Pro built a knowledge graph of each paper's claims, evidence, and methods. Node distance from a main claim set the severity - distance 0 hits a core claim, while larger distances hit supporting details. GPT-4.1 then rewrote one node per distance into a contradiction, creating 1,164 data points.
An o3 judge scored each review 10 times. On clean papers it reached 99.9% accuracy. It showed 86.8% sensitivity on manually confirmed catches, so the reported scores may be conservative.
Where MLR succeeds and where it still fails
MLR led every baseline on the benchmark. With four reviews, it caught 73.43% of distance-0 contradictions and 40.95% overall. A single MLR review still caught 60.79% of core-claim errors. The gap between MLR and baselines grew as error severity increased.
An ablation separated model choice from system design. Swapping GPT-4.1 for Claude Sonnet 4 inside the LLM-Review baseline lifted distance-0 detection from 14.56% to 35.40%. MLR's three-pass design added roughly 25 more points on top of that. Accuracy falls as node distance grows, which supports the severity scoring approach.
On real retracted papers from WithdrarXiv-Check, a set of 211 papers, the gains shrink. MLR scored 26.07% on "similar" matches and 16.11% on "exact" matches. The strongest baselines scored 18.48% and 9.00%, respectively. Hidden prompt injection still sways every AI reviewer tested.
On score alignment with humans, MLR reached a Pearson correlation of 0.586 on ICLR 2025 submissions, compared with a human-to-human reference of 0.742. On ICML 2025, the AI Reviewer edged it out, 0.439 versus 0.429. The authors frame MLR as a complementary perspective: the system stresses validity and experiments, while humans weigh clarity and novelty more heavily.
Why this matters for researchers and reviewers
The paper shifts the evaluation standard for AI review tools from "does it mimic a human?" to "can it catch a planted mistake?" That is a harder and more useful question. For teams building research agents or automated review pipelines, the findings point to two practical levers: model choice and multi-pass design each contribute large, separable gains in error detection. The cost profile - under fifty cents per review on API models - makes the approach accessible without dedicated infrastructure. Professionals working in AI Research Courses or developing internal review tools for grant proposals, regulatory submissions, or clinical research can take a concrete lesson: structured, multi-agent reading catches errors that single-pass systems miss, but real-world retracted-paper detection remains a hard problem no current system has solved.