Glonce

StateM lifts GPT-5.6 to 95.3% accuracy on Terminal-Bench 2.1

agentsThe Glonce editorsHugging Face🔥59Glonce rating59Impact65Novelty70Relevance95Surprise45Players50Credibility75
StateM lifts GPT-5.6 to 95.3% accuracy on Terminal-Bench 2.1

Long-horizon agents can fail even when the underlying model is capable of solving each individual step: they lose track of mutable state, fail to reuse lessons from earlier runs, skip known procedures, or stop prematurely. Rather than retrain the model, the authors bet on scaling the harness around it. Their runtime, StateM, organizes execution around durable states, phase-local context, checked transitions, recoverable runbooks and versioned procedural practices that both agents and users can inspect, leaving model weights untouched.

On Terminal-Bench 2.1, StateM raises GPT-5.5 xhigh from an 83.1% reference to 92.1%, ahead of GPT-5.6 Sol Ultra's own 91.9%. The runbook then transfers unchanged to GPT-5.6: with GPT-5.6 Sol xhigh, StateM reaches 95.3% raw accuracy across 445 trials, succeeding on all 89 tasks at least once. A frozen version of the profile also raises GPT-5.6 Luna from 76.7% to 85.4%, above the 84.9% Sol xhigh reference score.

The authors then test whether the same runtime, runbook structure and golden rules generalize to a different model family. Under $38 of adaptation work raises DeepSeek-V4 Flash from 82.7% to 88.1% under standard timeouts, and to 89.1% on an 88-task common core; extending only the one remaining latency-sensitive task matches the paper's own reported 88.8% GPT-5.6 Sol max result. They report the DeepSeek run's final-score API usage at about $15, against $574.68 for the GPT reference run, with total DeepSeek expenditure across the project at $52.22.

On a separate benchmark, BusinessBench, family-specific runbooks built on development sets produce held-out gains of 0.55 macro and 1.34 micro points overall, rising to 10.04 points for two families where the runbook's mechanism specifically matches the task. The authors frame this as showing that concrete rules generalize when tasks share execution structure, while the broader control methodology applies more widely: StateM turns selected postmortem findings into persistent, executable preconditions and practices, converting learned controls into enforceable stateful controls. Code is published at github.com/henryqin1997/statem.

Key facts

  • StateM is a runtime layer wrapped around an agent, not a new model: it adds durable states, phase-local context, checked transitions and versioned runbooks without changing model weights.
  • On Terminal-Bench 2.1, StateM raises GPT-5.5 xhigh from an 83.1% reference to 92.1%, and GPT-5.6 Sol xhigh to 95.3% raw accuracy across 445 trials, succeeding on all 89 tasks at least once.
  • The same runbook, adapted to DeepSeek-V4 Flash for under $38, raises its score from 82.7% to 88.1% (89.1% on an 88-task common core), against $574.68 of API usage for the GPT reference run.
  • On BusinessBench, family-specific runbooks add 0.55 macro and 1.34 micro points held-out overall, rising to 10.04 points for two mechanism-matched families.
  • Code is published at github.com/henryqin1997/statem.

Why it matters

The result argues that how an agent's execution is structured around a model can matter as much as the model itself. Without touching model weights, wrapping the same durable-state, checked-transition and versioned-runbook harness around several different frontier and non-frontier models produces accuracy gains on each, from single digits up to just over ten points, and the runbook built for one GPT-5.6 variant transferred unchanged to another. That points at harness design as a lever that is cheaper to iterate on than retraining.

Who it affects

Teams building or evaluating long-horizon coding and task agents on GPT-5.5, GPT-5.6 (the Sol and Luna variants the paper tests), or DeepSeek-V4 Flash, and anyone benchmarking agents on Terminal-Bench 2.1 or the paper's own BusinessBench.

How to use it

The code is open and published at github.com/henryqin1997/statem. The source does not state a price, license terms or usage limits for StateM itself beyond the benchmark API costs reported for the experiments, so none are claimed here.

How solid is it

The claims come from the paper's own abstract and title; no independent replication or third-party review is mentioned, and discussion activity so far is modest (181 points but only 2 comments on the discussion page). The headline figures are internally consistent (445 trials, 89 tasks, matching percentage pairs for each model), but the abstract gives no mechanism detail for how the runtime features individually contribute to the gains beyond naming them.

Risks and caveats

The title and abstract name no human authors or institutional affiliation, and no publication date is given; only the code repository is named. Several cost figures in the source paper are missing their dollar sign and have to be read from context (only its title explicitly states "$15"), and no description of what BusinessBench actually measures is provided beyond its name.

“We bet on harness scaling to improve the execution system around an agent without changing its model weights.”

— the authors