Study finds AI research agents act as optimizers, not innovators
A new evaluation looks past the final score AI agents get on a task and instead asks how they got there. The authors tested seven frontier models on 36 long-horizon tasks drawn from AI research and development work, using a framework built around rule-based metrics that characterize a run's behavior along three dimensions: Solution Framing, Execution, and Feedback Control. Alongside this, controlled comparisons check whether an agent's accumulated experience, both within a single task and carried over across different tasks, actually helps it make better decisions later, or misleads it.
The motivation is that a final score alone does not show where an agent gained or lost ground during a run, and does not reveal whether experience improves later decisions at all. Two agents can land on the same final outcome by very different routes, and a score-only view cannot tell them apart.
The results: current agents behave more like engineering optimizers than fully autonomous researchers. They can formulate and implement practical solutions, but their performance varies substantially from run to run. Their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis found that observed performance is shaped by several factors: process bottlenecks that differ even when final outcomes look similar, experience reuse that can help or mislead an agent's subsequent decisions, and harness design choices that affect how stable performance is across runs.
The authors present these findings as pointing to concrete directions for improving model training, inference-time strategies, how agents manage accumulated experience, and how evaluation harnesses are designed.
Key facts
- Seven frontier models were evaluated on 36 long-horizon AI research and development tasks using a new framework.
- The framework scores within-run behavior on three dimensions: Solution Framing, Execution, and Feedback Control, plus controlled tests of experience reuse within and across tasks.
- Agents act as engineering optimizers rather than autonomous researchers: they produce workable solutions, but performance varies substantially across runs.
- Their strongest solutions mainly adapt or combine established techniques; genuine methodological novelty is rare.
- Similar final outcomes can hide different process bottlenecks, and reused experience can help or mislead an agent's later decisions; harness design affects performance stability.
Why it matters
Benchmarks for AI agents typically report only a final score, which hides where a run gained or lost ground and whether an agent's earlier experience actually improved its later decisions. As agents take on longer, more open-ended research and development work, understanding the process behind the score matters as much as the score itself, especially if agents are eventually expected to contribute genuine methodological progress rather than just optimize within existing designs.
Who it affects
Teams building or evaluating AI agents for extended technical work: those designing agent harnesses, labs benchmarking frontier models on research-style tasks, and anyone deciding whether an agent can be trusted to run a long, multi-step R&D project with limited supervision.
How to use it
The paper's framework is something other evaluators can apply directly: rule-based metrics tracking a run along Solution Framing, Execution and Feedback Control, plus controlled comparisons showing whether an agent's experience, within a task or carried across tasks, helps or hurts its next decision. Teams assessing agents for long-horizon work can use this structure instead of relying on a single end score.
How solid is it
The evaluation covers seven frontier models across 36 long-horizon tasks, a reasonably broad test bed for this kind of study. Not available from the material reviewed: which seven models were tested, the numeric scores or pass rates behind the qualitative findings, who wrote the paper or which institutions they are affiliated with, and when the work was published. The three behavior dimensions are named but not individually defined or scored in the text, so how each one was measured cannot be checked here.
Risks and caveats
The central conclusion, that agents mostly adapt or recombine known techniques rather than produce genuine methodological novelty, is a qualitative judgment from the study rather than a number that can be independently checked from what is available. Because the specific models, scores and authorship are not given in the material reviewed, readers who want to verify how a particular model performed, or judge the study's rigor for themselves, will need to consult the full paper directly.
“current agents operate more like engineering optimizers than fully autonomous researchers”
— the study's authors