AI news ·
GdpX benchmark finds top AI models score just 12% on frontier tasks
SemiAnalysis's GDPx benchmark tests frontier AI on 11 high-value domains, and the best model scores just 12%. Finance and legal tasks are so hard nearly every model scores zero.
SemiAnalysis has released GDPx, a benchmark that tests frontier AI models on eleven high-value domains, and the best model scores just 12%. The benchmark covers chip design, cybersecurity, biology, finance, legal, quant trading, recursive self-improvement, hardware design, materials science, alignment, and software engineering. It deliberately skips industries like government, real estate, and insurance.
The authors argue that measuring AI progress by share of GDP or occupational taxonomies misses where the money will actually go. They expect the first trillion dollars of AI revenue to concentrate in sectors that can deploy frontier intelligence at high return, not spread evenly across the economy.
How the models rank
Excluding cyber, biology, and alignment - the three domains where safety classifiers sometimes block responses - the ranking shifts. OpenAI and Anthropic dominate the top. SpaceXAI, Meta, and open-weight models trail meaningfully behind.
Opus 5.5 takes the best all-around spot, winning quant trading, hardware design, and chip design. Gemini 4 Carbon is the strongest biology model, which the authors attribute to weaker safety classifiers, and edges out the software engineering win. GPT-6 Astra wins convincingly for recursive self-improvement and materials science.
Finance and legal tasks are so hard that nearly every model scores zero. "Our finance and legal tasks are so challenging that basically every model scores exactly 0," the authors write. The benchmark also shows wide variation within single models across domains.
On cost versus performance, Luna and DeepSeek compete at the low end, Opus wins the high end, and OpenAI claims the middle. The authors flag Gemini 4 Carbon as the model to watch.
Why every task ships with a working environment
The benchmark was built with 14 specialized data partners, one or more per domain. SemiAnalysis says it approached more than 30 RL environment companies and dropped many, including prominent ones, that failed its quality checks on the first pass. When multiple partners wanted exclusivity in a domain, the team picked whichever produced better tasks.
Each task drops the model into a simulated workplace rather than a prompt. A finance task from partner Living Context seeds a private equity deal - Bannock Capital buying 62% of Haverly Snack Company for $880 million plus an earnout - across roughly 4,000 files. The model has to find the right documents in a 2,000-file deal room, the way a human associate would. Cyber tasks run live services built around real zero-days. Materials science tasks put models inside simulated laboratories with battery cyclers and electron microscopes.
The authors single out OpenAI's GDPval, released in September 2025, as outdated. They say its tasks are non-interactive, spoon-feed context in the prompt, and rely on noisy rubric graders. Human expert agreement on GDPval's gold subset was 71%. "It's not OpenAI's fault that a benchmark created before the rise of agents does a poor job evaluating them today," the authors write.
The default time limit per task is eight hours. Models run in their native harness where one exists, such as Claude Code or Codex, and in OpenCode otherwise. The headline score is pass@1 across three attempts, with all domains weighted equally.
What the tasks reveal about model limits
In hardware design, no model achieved a verified full pass on a public task to design an injection-molded compressor housing. Opus and Astra produced rebuildable housings and wrote their own verification routines. Grok and GLM struggled with basic assembly interfaces - one GLM run misplaced its housing by more than 11 millimeters, and a Grok run cut four screw holes in the wrong direction.
In finance, no model solved the public quality-of-earnings task across 25 attempts. The best single run passed 8 of 11 checks. Common errors included misreading a waived insurance premium adjustment, counting a banker's success fee in net debt, and keeping compensation for a family member who logged no hours.
In recursive self-improvement, all four tested models failed to trace a training bug to a harness instruction that told the model to stop exploring. The bug cut mean training reward by three times compared with clean rollouts.
For teams tracking which models to deploy, the results suggest evaluating candidates on the specific tasks they'll face rather than on general scores. Those building agent workflows can find related training through AI Agent Courses.
Why this matters for your team
If you work in finance, legal, healthcare, or insurance, the benchmark's main finding is that frontier models still fall short on realistic professional work. Finance and legal tasks score near zero. A model that performs well on a general leaderboard may fail on the document-heavy, judgment-intensive work your staff does every day.
For IT and product teams, the lesson is about evaluation design. Tasks that hand the model all the context in a prompt no longer separate strong models from weak ones. The gap shows up when a model has to search a messy environment, weigh conflicting sources, and recognize second-order consequences. That's the capability worth testing before you commit to a vendor or a workflow.