Glonce

StopLoss monitor aborts failing coding-agent runs early, cutting SWE-bench cost

researchThe Glonce editorsHugging Face🔥58Glonce rating58Impact45Novelty60Relevance95Surprise30Players15Credibility75
StopLoss monitor aborts failing coding-agent runs early, cutting SWE-bench cost

Coding agents often keep working on runs that will end up failing verification. Asking a frontier model to watch for this is expensive in its own right. A new Hugging Face papers entry introduces StopLoss, a small external monitor built to recognise failing runs cheaply and end them early. The name comes from the stop-loss order in finance: cut the run when it is likely to fail, and save the resources it would have burned.

The method uses outcomes that already exist. The final result of a completed trajectory, as judged by a verifier, labels the earlier local windows of that run. With those labels, a fine-tuned language model learns to score the most recent tool steps in a single forward pass. A threshold calibrated on successful runs then turns the score into an abort decision, and the executor itself does not change.

StopLoss was trained on GPT-family SWE-rebench trajectories. It was evaluated on SWE-bench Verified and a SWE-bench Pro subset, with GPT and Claude executors. The comparison set was six general-purpose judges reading identical windows.

The authors report that outcome supervision lifts failure prediction from near chance to a useful signal at the first check, at one to two orders of magnitude lower cost per check than any tested judge. In money terms, relative to completing every task, StopLoss lowers cost per retained solve by 4.5% on Verified while preserving 99% of successful runs. On the costlier Pro subset the reduction reaches up to 31%, and with a Qwen3-8B backbone it reaches 48%. In all four settings, StopLoss has a lower cost per retained solve than every judge policy. The authors conclude that outcome-trained small monitors can make failure recognition cheap enough for early stopping to pay for itself.

Key facts

  • StopLoss is a small external monitor that learns from completed coding-agent trajectories and verifier outcomes to predict failure and abort a run early.
  • On SWE-bench Verified it lowers cost per retained solve by 4.5% relative to completing every task, preserving 99% of successful runs.
  • On the costlier SWE-bench Pro subset the reduction is up to 31%, and it reaches 48% with a Qwen3-8B backbone.
  • At the first check it costs one to two orders of magnitude less per check than any of six general-purpose judges tested.
  • In all four settings its cost per retained solve is below that of every judge policy.

Why it matters

Agents that keep grinding on doomed runs waste compute, and the obvious fix, a frontier model that watches the run, adds its own bill. StopLoss tests a cheaper route: a small model trained on outcomes that teams already collect. The framing is financial. Like a stop-loss order, the monitor cuts a run when it looks likely to fail, so the question is whether the check costs less than the run it saves. The authors say it does, in all four settings they tested, against six general-purpose judges.

Who it affects

The work is aimed at people who run coding agents against verifiable tasks and pay for long trajectories. It was tested with GPT and Claude executors on SWE-bench Verified and a SWE-bench Pro subset. Researchers working on agent monitoring and cost-aware evaluation are the other audience, since the paper compares a trained small monitor against general-purpose judges.

How to use it

The design is meant to sit beside an existing agent. The monitor scores the most recent tool steps in one forward pass, and a threshold calibrated on successful runs decides whether to abort, without changing the executor. Reproducing the approach needs completed trajectories with verifier outcomes to train on. No code, model or dataset release is mentioned.

How solid is it

This is a preprint abstract, and the numbers are the authors' own. The headline saving on Verified is modest: 4.5% relative to completing every task, with 99% of successful runs preserved. Larger figures come with qualifiers: up to 31% on the costlier Pro subset, and 48% with a Qwen3-8B backbone. The 48% figure is not tied to a specific benchmark or to the four settings explicitly. The share of successful runs preserved on the Pro subset is not stated, and no absolute dollar costs are given. The source does not say which Claude or GPT models served as executors, nor which six judges were compared.

Risks and caveats

Training used GPT-family SWE-rebench trajectories, so how well the monitor carries over to other agents, tasks or benchmarks is not shown here. Aborting a run always risks losing a solve that would have succeeded; the threshold is calibrated on successful runs to limit that, and 99% were preserved on Verified. The savings on Verified are small, and the larger numbers come from the costlier subset or a specific backbone. The 'near chance to a useful signal' wording describes the first check, and the source does not quantify it further.

“Outcome-trained small monitors can make failure recognition cheap enough for early stopping to pay for itself.”

— StopLoss paper abstract, Hugging Face papers