HarnessDev benchmark finds LLM-built agent harnesses trail humans on code, match them on writing
As AI agents move from research prototypes into deployed tools, how well they perform increasingly depends on infrastructure that sits outside the model itself: the prompts, tools, control loop and scaffolding collectively called the agent harness. Changing that harness while leaving a model's weights untouched can substantially alter task performance, yet most agent evaluations report performance under one harness chosen by the people running the benchmark, and rarely test whether a model can build that harness itself.
The authors introduce HarnessDev, a benchmark that shifts what gets measured from a model's task outputs to the runnable infrastructure it produces. HarnessDev has two stages. In Creation, a model starts from a minimal seed and a small number of example cases and has to build a complete, working execution system from scratch. In Evolution, it starts instead from the harness it just created and revises it iteratively, using feedback from actually running it, with the explicit goal of improving its own benchmark performance. Every resulting harness is then scored on two axes: capability, meaning task success on held-out benchmarks, and efficiency, meaning how many execution tokens it costs to run.
The reported Creation results span six creator LLMs tested across four domains and five downstream benchmarks, totaling 2,207 unique downstream instances, with the evaluation tasks themselves withheld from the models during development.
The reported findings are mixed rather than uniformly poor. Harnesses that the LLMs build on their own remain substantially behind mature, human-engineered reference harnesses on coding tasks and on search-and-research tasks. But on writing and on machine-learning experimentation, the self-built harnesses match or even exceed those same human references, though how much they cost to run varies widely from one generated harness to another. The Evolution stage, where a model iteratively edits its own harness using execution feedback, does produce some performance gains, but the authors describe them as unstable and only partially transferable to held-out tasks the harness was not evolved against. A further experiment that fixed which model actually executes the harness found that these gains depend strongly on that specific runtime model, so an improvement measured with one model running the harness does not reliably carry over when a different model runs the same harness.
Key facts
- HarnessDev evaluates an LLM not on the tasks it completes, but on whether it can build (Creation) and then iteratively improve (Evolution) the execution harness that runs it as an agent.
- In Creation, a model builds a full execution system from a minimal seed and a handful of example cases; in Evolution, it revises its own harness using feedback from actually running it, aiming to raise its own benchmark performance.
- Reported Creation results cover six creator LLMs across four domains and five downstream benchmarks, totaling 2,207 unique downstream instances, with the grading tasks withheld from development.
- LLM-built harnesses trail mature human-engineered references on code and on search-and-research tasks, but match or exceed them on writing and machine-learning experimentation, with execution cost varying widely.
- Evolution's gains are unstable, transfer only partially to held-out tasks, and depend strongly on which specific model executes the harness, so improvements do not reliably transfer across models.
Why it matters
As agents move from research prototypes into deployed tools, how well they perform increasingly depends on the execution infrastructure wrapped around the model, the prompts, tools, control loop and scaffolding collectively called the agent harness, and changing that harness while leaving a model's weights untouched can substantially change task performance. Most agent evaluations sidestep this: they report a model's performance under one harness chosen by the people running the benchmark, leaving a model's own ability to build or improve that harness essentially untested. HarnessDev targets exactly that gap, treating the harness itself, not just the model's answers inside it, as the thing being evaluated.
Who it affects
This is aimed at people building agent products and at people designing agent evaluations: it shows that scoring models under one fixed, hand-built harness can hide real differences in how well each model would do at constructing its own tooling and control loop from scratch. It also matters to anyone hoping a model can both build and later improve its own scaffolding unattended, since the reported gains from self-revision do not reliably carry over once a different model is put in charge of running the same harness.
How to use it
As a benchmark, HarnessDev works by handing a model a minimal seed and a handful of example cases and asking it to build a complete, runnable agent execution system from that (Creation), then letting it revise that same system using feedback from running it, with the explicit goal of raising its own benchmark performance (Evolution). Each resulting harness is scored on capability, meaning task success on held-out benchmarks, and on efficiency, meaning execution-token cost. The practical takeaway for anyone building agent systems is uneven: do not assume a self-built harness will match careful human engineering on coding or on search-and-research work, where the reported gap is substantial, but a self-built harness may well match or beat a human-engineered one on writing or on machine-learning-experimentation work. And because Evolution's gains track the specific model that produced them, swapping in a different model to run an already-tuned harness is not a safe assumption.
How solid is it
The source reads as a paper abstract: it names no individual authors, institution, publication venue or date, and it gives no numeric accuracy, success-rate or execution-token-cost figures for the reported comparisons, only qualitative language such as "substantially behind" and "matching or exceeding." On breadth, the study is not thin: six creator LLMs across four domains, five downstream benchmarks and 2,207 unique downstream instances, with the grading tasks kept hidden from the models during development, a reasonable safeguard against a model simply being built around what it will be tested on. Without hard numbers, though, there is no way from this text alone to judge how large the reported gaps and matches actually are.
Risks and caveats
The results come from the researchers who built HarnessDev testing their own benchmark, the usual caveat for a newly introduced evaluation. The paper's own headline limitation is that Evolution's gains are unstable and only partially transfer to held-out tasks, and a separate experiment found those gains depend strongly on which specific model executes the harness, so an improvement is not guaranteed to survive a change of runtime model. Execution cost also varies widely across the generated harnesses, meaning a harness that performs well is not automatically a cheap one to run.
“Changing this harness while holding model weights fixed can substantially alter task performance.”
— the HarnessDev paper