AI news ·
FrontierPhysics benchmark finds no AI agent passes more than 27% of research tasks
No AI model passed more than 46 of 168 rollouts on a new physics benchmark requiring literature review, experiments, and manuscript writing. Evaluators found the core failure was insufficiently rigorous reasoning and limited expert intuition, not missing knowledge.
AI agents still can't produce publishable physics research. That's the central finding from FrontierPhysics, a new benchmark built by 42 physicists that tests whether large language models can handle the full scientific process - from literature review to manuscript writing - across 56 original research tasks.
The benchmark covers nine domains spanning theoretical, computational, and experimental physics. Each task demands that an agent review existing literature, form a research plan, run experiments, and deliver structured results and a scientific manuscript. Performance is measured with deterministic verifiers and expert-authored rubrics that check both correctness and research quality.
How the benchmark works
Unlike existing physics evaluations that rely on fixed questions or isolated sub-tasks, FrontierPhysics tasks are drawn directly from the contributing physicists' own research. This means agents face open-ended problems without pre-packaged solutions. The evaluation combines automated verification with human expert judgment on whether the resulting work meets a publishable standard.
Researchers tested 22 models across 26 configurations. The result: no configuration passed more than 46 of its 168 rollouts. Even the strongest frontier models fell short across the board.
Where agents stumble
Manual review of agent trajectories revealed a clear pattern. The primary weaknesses were not gaps in physics knowledge or execution errors. Instead, the evaluators flagged "insufficiently rigorous reasoning and limited expert intuition" as the main failure modes.
This finding challenges the assumption that scaling up factual knowledge and tool-use capabilities will close the gap between AI assistants and working scientists. The models knew relevant physics. They could run calculations. They struggled to think like a physicist when the path forward wasn't obvious.
What this means for AI in research
FrontierPhysics arrives at a moment when AI tools are being woven into research workflows across disciplines. The benchmark offers a concrete measure of what current systems can and cannot do in a setting that mirrors real scientific work - not just answering questions, but formulating them and carrying through to a finished product.
The tasks span computational, theoretical, and experimental physics, making the results relevant across different modes of scientific work. For organizations investing in AI for research, the 46-out-of-168 ceiling provides a sobering data point on current capabilities. For those exploring how to build these skills, AI for Scientists Courses offer structured paths to understand where the technology stands today.
Why this matters for executives and research leaders
For anyone making resource decisions about AI in R&D, FrontierPhysics offers a clear signal: current frontier models cannot independently produce publishable physics research. The bottleneck is not knowledge retrieval or computation, but the kind of expert reasoning that experienced scientists develop over years. Investment in AI research tools should account for this gap - the technology can accelerate parts of the workflow, but the benchmark suggests it is not a replacement for trained scientific judgment.