AutoSaddler automates agent harness tuning for up to 10pp gains
LLM agents tend to fail on long-horizon tasks because small local mistakes compound across an extended interaction into a full task failure. External harnesses, the prompts, tool configurations, and control logic wrapped around a model, can substantially improve robustness, but building one is a manual, expensive process that means searching a large space of possible prompts, tools, and control flows by hand.
A new paper proposes AutoSaddler, a framework that automates that search. It treats harness improvement as an offline learning problem: the system iteratively updates the harness using failure signals drawn from mini-batches of past agent runs. The approach has three parts working together: diagnosing why a run failed from its trace, generating structured patches that edit the harness as if it were code, and validating each candidate update before it is kept, rather than accepting any change that merely looks plausible.
The authors tested AutoSaddler on three agent benchmarks: GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0. Against the corresponding base harness on each, AutoSaddler-optimized harnesses raised performance by 9.0 percentage points on GAIA2, 9.6 percentage points on SWE-Bench Pro, and 10.0 percentage points on Terminal-Bench 2.0.
Ablation studies isolated three factors behind those gains. Deep debugging of a failure beats shallow reflection on it. Targeted, scoped modifications to the harness beat unconstrained editing. And selecting updates for how well they generalize beats selecting them for fixing one specific trajectory. The paper's abstract does not name authors or institutions, give a release timeline, or say whether code or a trained model will be made available, and it does not specify the mini-batch size used for updates or describe the base harnesses beyond calling them 'corresponding'.
Key facts
- AutoSaddler automatically optimizes LLM agent harnesses by treating harness design as an offline learning problem driven by failure signals from mini-batches of agent runs
- It combines three steps: diagnosing failures from execution traces, generating structured patches that treat the harness as code, and validating candidate updates before keeping them
- On GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, AutoSaddler improved agent performance over the base harness by 9.0, 9.6, and 10.0 percentage points respectively
- Ablation studies found three ingredients drive effective optimization: deep debugging over shallow reflection, targeted edits over unconstrained editing, and generalization-aware selection over trajectory-specific repair
- The paper's abstract names no authors or institutions and gives no release date or code/model availability for AutoSaddler
Why it matters
LLM agents remain unreliable on tasks that unfold over many steps, since small local errors compound into overall failure. Wrapping a model in a well-designed harness, the prompts, tool access, and control logic around it, substantially improves robustness, but hand-designing that harness means manually searching a large space of prompt variants, tool configurations, and control flows. AutoSaddler turns that manual search into an automated, iterative optimization loop driven by the agent's own failure traces.
Who it affects
The work targets people building or operating agent harnesses for long-horizon tasks: general web and tool-use agents (GAIA2), coding agents (SWE-Bench Pro), and terminal-operating agents (Terminal-Bench 2.0). Anyone maintaining a hand-tuned harness for one of these workloads is the direct audience for an automated alternative.
How to use it
AutoSaddler works by repeatedly diagnosing why an agent's run failed from its execution trace, generating a structured patch to the harness itself (treated as editable code) in response, and validating each candidate update before adopting it, using mini-batches of failure signals rather than single trajectories. The abstract gives no release date and does not say whether code or a trained artifact will be published, so there is currently no stated way to use AutoSaddler outside the paper's own experiments.
How solid is it
The claimed gains, 9.0, 9.6, and 10.0 percentage points over the respective base harness, are consistent in size across three different benchmark types (general agent tasks, software engineering, and terminal operations), and the ablation studies back the method's design choices with controlled comparisons rather than a single end-to-end number. That said, the abstract does not identify the authors or their institution, does not describe the base harnesses being compared against beyond calling them 'corresponding', and does not state the mini-batch size used, all of which limits how easily the results can be checked or reproduced from the abstract alone.
Risks and caveats
Key details are missing from the abstract: no authors or institutions are named, there is no release date, and it is unclear whether code or a trained model will ever be made available. The size of the failure-signal mini-batches and the exact makeup of the base harnesses used for comparison are also unstated. More generally, a system that patches its own harness from past failures risks overfitting to the failure modes it has already seen, which is the problem the paper's generalization-aware selection step is specifically meant to guard against, according to its own ablations.
“harness design remains a manual and expensive process that requires searching over a large space of prompts, tool configurations, and control logic”
— AutoSaddler paper abstract