Glonce

HarnessOpt-Bench benchmarks LLMs at optimizing agent harnesses

researchThe Glonce editorsHugging Face🔥58Glonce rating58Impact65Novelty78Relevance100Surprise52Players42Credibility82
HarnessOpt-Bench benchmarks LLMs at optimizing agent harnesses

Researchers introduce HarnessOpt-Bench, a benchmark for automated harness optimization: the task of an LLM iteratively editing an agent's harness (its prompts, tools, control flow, memory, and orchestration code) to improve its performance, guided only by evaluation feedback. In the benchmark's protocol, an optimizer, itself an LLM paired with a coding harness, is given a target agent's seed harness, graded evaluation feedback, and a fixed budget of target-evaluation calls. The optimizer edits the harness and nominates a final candidate. That candidate is then scored by its normalized gain over the seed harness on a held-out test partition that stays inaccessible throughout the search process, so the optimizer cannot see the exact data it will ultimately be judged on. A trusted execution environment enforces this evaluation boundary, meters how much of the target agent's resources are used, and preserves every candidate version for audit. The study evaluates 5 frontier LLMs as optimizers, testing each one both under a shared coding harness and under its own native harness, across 4 downstream tasks, for a total of 111 scored runs. The results show that which optimizer model is used separates performance more than which coding harness it runs through, that native harnesses are not consistently better than the shared one, and that gains vary substantially depending on the task and the starting seed harness. The abstract does not name the specific models tested, give per-model or per-task scores, or state a release date.

Key facts

  • HarnessOpt-Bench evaluates 5 frontier LLMs as optimizers across 4 downstream tasks, totaling 111 scored runs.
  • An optimizer (an LLM plus a coding harness) edits a target agent's seed harness under graded feedback and a fixed evaluation budget, then nominates a final candidate.
  • Candidates are scored by normalized gain over the seed harness on a held-out test partition kept inaccessible during search.
  • Optimizer models separate more than the coding harnesses they act through, and native harnesses are not consistently superior to a shared one.
  • Gains vary substantially across tasks and seed regimes.

Why it matters

As LLMs get deployed inside agentic systems, what those systems can do depends not just on the model's weights but on the harness wrapped around it: the prompts, tools, control flow, memory, and orchestration code. Automating the improvement of that harness is both a promising way to make agentic systems better and a demanding test of what AI systems can do on their own. Until now, there was no shared protocol for measuring how well frontier LLMs perform at this specific task.

Who it affects

The benchmark targets researchers and engineers who build agentic systems and need a way to compare which LLM, used as an automated harness optimizer, actually improves an agent's prompts, tools, and control flow rather than just producing plausible-looking edits.

How to use it

The protocol gives an optimizer, an LLM paired with a coding harness, a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. The optimizer edits the harness and nominates one final candidate. Scoring uses that candidate's normalized gain over the seed on a held-out test partition the optimizer cannot access during search. A trusted execution environment enforces this boundary, meters the target agent's resource use, and keeps every candidate version for later audit.

How solid is it

The evaluation spans 5 frontier LLMs acting as optimizers, tested both under one shared coding harness and under each model's own native harness, across 4 downstream tasks, for 111 scored runs in total. The source text reports only qualitative findings from this run; it gives no per-model or per-task numeric scores.

Risks and caveats

The abstract does not name which 5 frontier models were tested, does not identify the paper's authors or their institutions, and gives no release date or publication venue for the benchmark. Without concrete model names or scores, the findings should be read as directional rather than as a leaderboard.

“These results establish harness optimization as a measurable and discriminative capability with large space for improvement.”

— the authors