FrontierPhysics benchmark: AI agents struggle with end-to-end physics research
A new paper introduces FrontierPhysics, a benchmark built to test whether AI agents can do physics research from start to finish. The authors say agents play an increasingly important role in physics research, but existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness.
FrontierPhysics has 56 tasks made by 42 physicists, each based on the physicist's own research. The tasks cover nine domains across theoretical, computational, and experimental physics. Each task asks an agent to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and a scientific manuscript.
Grading combines two things: deterministic verifiers and expert-authored rubrics. Together they are meant to judge whether the agent's research is correct and publishable, not just whether a final answer matches.
The results are sobering. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. The authors also reviewed agent trajectories by hand. They report that frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors. In other words, the agents mostly know the physics and can run the work; what falls short is the rigor and judgment a working researcher brings.
Key facts
- FrontierPhysics has 56 tasks written by 42 physicists, each based on their own research, across nine domains of theoretical, computational, and experimental physics.
- Each task covers the full workflow: literature review, research plan, research and experiments, structured outputs and a scientific manuscript.
- Grading combines deterministic verifiers with expert-authored rubrics to judge whether the research is correct and publishable.
- Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts.
- Manual review of trajectories points to insufficiently rigorous reasoning and limited expert intuition as the main weaknesses, not missing physics knowledge or execution errors.
Why it matters
Most physics benchmarks test fixed questions, isolated sub-tasks, or reproduction of a paper where the solution is given. FrontierPhysics instead asks agents to do the whole job: read the literature, plan, run the research, and write it up. It also checks research quality alongside correctness. The headline result is that even the best configuration passes no more than 46 of its 168 rollouts, which gives a concrete measure of how far agents are from independent physics research.
Who it affects
Physicists who are considering AI agents as research assistants get a picture built from tasks that 42 of their peers wrote from their own work. Developers of agents and models get a harder, more realistic target than question-answering sets. Benchmark designers get an example of pairing deterministic verifiers with expert rubrics.
How to use it
The abstract describes an evaluation setup rather than a product. Teams building research agents can use the framing: judge an agent on whether its manuscript-level output is correct and publishable, not only on final answers. The authors' trajectory review suggests focusing on rigorous reasoning and expert intuition rather than on adding physics knowledge. The source mentions no link to code or a dataset release.
How solid is it
The source is the paper's abstract, so the claims are the authors' own. The numbers are specific: 56 tasks, 42 physicists, nine domains, 22 models in 26 configurations, and a ceiling of 46 of 168 rollouts for any configuration. The diagnosis of weaknesses comes from manual review of agent trajectories. The abstract names no specific models, gives no per-model scores, and does not say which configuration scored highest.
Risks and caveats
The abstract does not say what a rollout is, how many rollouts each task gets, or how the pass threshold is defined, so the 46 of 168 figure should be read as a ceiling and not as a general success rate. It does not name the nine domains. Because it gives no model names or per-model results, it is not possible to say from this text how individual systems compare. The explanation that rigor and intuition are the weak points rests on manual review of trajectories.
“frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors”
— FrontierPhysics paper abstract