Evo-Bench tests whether models can evolve their own agent harness

Researchers introduce Evo-Bench, described as the first benchmark built to evaluate a language model's intrinsic ability to autonomously evolve its own agent operating harness, the software scaffolding an agent runs inside, rather than being measured only on solving fixed tasks. The paper argues that existing evaluations fail to isolate harness improvements from the strength of the underlying base model, allow task-specific overfitting, and fail to capture the kind of long-horizon iterative research that real harness development involves. To get around these problems, Evo-Bench uses what the authors call a harness-guided construction framework: it applies auxiliary-task evolution to find tasks that are genuinely sensitive to changes in the harness itself, then applies sensitivity-aware stratified splitting so results generalize across the different task suites rather than reflecting one narrow set. The benchmark spans three agent domains: Search, Office and General. The authors ran evaluations across nine frontier and open-weight models (none named in the source). Top models achieved absolute gains reaching 16.6 points, described as closely approaching state-of-the-art human-engineered baselines built by people rather than by the models themselves. Performance varied by domain: autonomous harness evolution outperformed the artificial (human-built) harness on General tasks and excelled on Search tasks, but struggled on Office tasks, which the authors say demand highly specific processing workflows. The analysis also surfaced what the authors call critical temporal anomalies, including early saturation, where a model's improvement curve appears to plateau earlier than expected. Separately, the authors report that the harnesses models synthesize during this process act as highly transferable reasoning structures, and consistently improve performance when applied to other, different policy models.

Key facts

Evo-Bench is presented as the first benchmark for evaluating models' intrinsic ability to autonomously evolve their own agent harness.
It covers three agent domains: Search, Office and General, built via a harness-guided construction framework combining auxiliary-task evolution with sensitivity-aware stratified splitting.
Across nine frontier and open-weight models tested, top models reached absolute gains of 16.6 points, closely approaching human-engineered baselines.
Autonomous evolution beat the human-built harness on General tasks and excelled on Search tasks, but struggled on Office tasks needing highly specific workflows.
The analysis found temporal anomalies including early saturation, and found that synthesized harnesses transfer well, boosting performance across different policy models.

Why it matters

Most agent benchmarks measure whether a model can solve a fixed task with a fixed harness. Evo-Bench targets a different, newer capability: whether a model can autonomously rewrite and improve the harness it operates in, the tooling and workflow scaffolding around it, rather than just the task itself. The authors say prior evaluations could not cleanly separate this skill from raw base-model strength, and were vulnerable to task-specific overfitting or to missing the long, iterative nature of real harness development. Evo-Bench is built specifically to isolate and measure that capability.

Who it affects

The benchmark is aimed at researchers and engineers building autonomous agents and the frameworks (harnesses) they run in, particularly anyone evaluating whether an agent can improve its own operating setup rather than needing a human to hand-engineer it. The three tested domains, Search, Office and General, point to agents used for information retrieval, office-style document or workflow tasks, and general-purpose assistance.

How to use it

Evo-Bench itself is a benchmark rather than a product: it is constructed with a harness-guided framework that uses auxiliary-task evolution to identify tasks genuinely sensitive to harness changes, followed by sensitivity-aware stratified splitting to keep results generalizable across the different task suites. The source text does not name the specific nine models evaluated, give per-domain numeric scores beyond the 16.6-point top-line figure, or state a publication venue or timeline.

How solid is it

The authors report evaluations across nine frontier and open-weight models. Top models achieved absolute gains reaching 16.6 points, which the paper describes as closely approaching state-of-the-art human-engineered baselines. Results differed by domain: autonomous evolution outperformed the artificial harness on General tasks and excelled on Search tasks, but struggled on Office tasks that demand highly specific processing workflows. The authors also report that harnesses synthesized by the models act as highly transferable reasoning structures, consistently boosting performance when applied to other, different policy models.

Risks and caveats

The analysis surfaced what the authors call critical temporal anomalies, including early saturation, without explaining in the source text what causes it or how it should be interpreted. The Office-task weakness suggests autonomous harness evolution is not uniformly reliable and can lag a human-built harness in workflows that require highly specific processing. The abstract names no authors, institutions, publication venue or timeline, and gives no breakdown of scores by domain beyond the single 16.6-point top-line figure.