Glonce

LoopArena tests models as controllers for coding agents

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact70Novelty78Relevance95Surprise45Players40Credibility85
LoopArena tests models as controllers for coding agents

Researchers have introduced LoopArena, a benchmark for measuring how well a language model can act as the "Controller" that steers a separate, fixed coding agent, the "Worker," through a long-running coding task. The idea responds to a gap in how coding agents are usually judged: looking only at whether a single end-to-end run succeeded cannot tell you whether the outcome reflects the quality of the loop's guidance or the coding agent's own ability to do the work. A loop can go wrong in several ways even when the underlying coding agent is capable: it can trust a stale progress note, skip verification it should have run, spend its budget on the wrong subtask, or stop before the work is actually safe to submit. LoopArena is built to separate those two factors by fixing the Worker and varying only the Controller. After each round of the Worker's coding, the Controller receives a structured summary of what happened and then instructs the Worker on what to do or check next, or decides the task is done. The benchmark scores this control ability in three settings of increasing scope and cost. Type I scores next-step "Loop Contract" selection using execution-validated questions, without running the Worker at evaluation time at all. Type II executes repeated control over a chosen slice of a full task. Type III runs the paired full task starting from its original state. On the full-task Type III setting, the best Strict Success Rate observed across the Controllers tested is 24.69%, which the authors say leaves substantial room for improvement in long-horizon loop control. Across the Controllers evaluated, the average paired reduction in estimated inference cost is 64.4%. The cheaper Type II setting also tracks the full-scale results well: its ordering of Controllers correlates strongly with the ordering produced by the benchmark's main Core criterion, with a Spearman's rho of 0.9747. The team has released the benchmark data and evaluation code on GitHub at AMAP-ML/LoopArena.

Key facts

  • LoopArena is a benchmark that isolates a model's skill as a "Controller" directing a separate, fixed coding agent ("Worker") through a long-running task, rather than judging the coding agent's raw ability.
  • It evaluates control in three settings: Type I (execution-validated next-step selection without running the Worker), Type II (repeated control over a slice of a task), and Type III (the paired full task from its original state).
  • On full tasks (Type III), the best observed Strict Success Rate across Controllers is 24.69%.
  • The average paired reduction in estimated inference cost across Controllers is 64.4%.
  • The cheaper Type II setting produces an ordering of Controllers that closely matches the benchmark's main Core criterion ranking, with a Spearman's rho of 0.9747, and the benchmark data and evaluation code are released at github.com/AMAP-ML/LoopArena.

Why it matters

Loop engineering, designing the harness that monitors a coding agent's progress, assigns it work, runs checks, and decides what happens next, is becoming its own discipline separate from prompt writing. Until now, judging a loop meant judging the whole run end to end, which conflates two different things: whether the loop gave good instructions and whether the coding agent could execute them. LoopArena is built specifically to pull those two apart by holding the coding agent (the Worker) fixed and testing only the quality of the controlling model's decisions.

Who it affects

The benchmark is aimed at teams and researchers building or choosing the automation loops that sit on top of coding agents, and at model providers whose systems might be deployed as such Controllers. Anyone assembling a long-running coding-agent pipeline, where a model has to track progress, decide what to verify, and judge when a task is actually done, is the intended audience.

How to use it

The benchmark data and evaluation code are released publicly on GitHub at AMAP-ML/LoopArena. Because it runs a fixed Worker against a swappable Controller, it can be used to score a candidate model's ability to direct a coding agent across the three evaluation settings without needing to build a full end-to-end task suite from scratch.

How solid is it

The paper reports concrete, execution-based results rather than only proposing a method: a 24.69% best Strict Success Rate on full tasks, an average 64.4% paired reduction in estimated inference cost across the Controllers tested, and a strong Spearman correlation (0.9747) between the cheaper Type II setting and the benchmark's main Core criterion ranking, which suggests Type II can serve as a reasonably faithful, lower-cost proxy for full Type III evaluation.

Risks and caveats

The released text does not say which specific models were tested as Controllers or which one achieved the 24.69% figure, nor does it define exactly what "Strict Success Rate" or the "main Core criterion" measure beyond their names. The headline result itself is a caution: even the best Controller tested succeeds strictly on full tasks well under a quarter of the time, meaning current models remain unreliable at directing long-running coding work.

“Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task.”

— the LoopArena paper