Glonce

Terminal-Bench audit: only 78 of 125 all-fail tasks are genuinely hard

researchThe Glonce editorsarXiv🔥75Glonce rating75Impact65Novelty78Relevance100Surprise58Players25Credibility92

A new paper examines what a zero pass rate on an agentic coding benchmark actually proves. The authors study a frozen production record of Terminal-Bench 3 and Frontier-Bench 0.1 covering 1,081 pull requests, 639 scored tasks, 28,801 trials and $105,933 in logged agent spend. Their starting observation is that a task no model solves is not automatically a hard task: the same zero pass rate can come from a genuine capability gap, but also from missing context, a broken reference solution, an infrastructure failure, or a verifier that can be bypassed. To separate these cases, the authors take the 125 tasks in the record with no honest pass and run them through an ordered validity screen combining task artifacts, reference-solution runs, empty-solution controls, adversarial trials, agent trajectories, telemetry and review records. The result: only 78 of the 125 tasks survive as certified-unsolved candidates. Of the remainder, 14 tasks have broken oracles (faulty grading logic), 8 are dominated by infrastructure failures, 4 are only passable through verifier bypasses, and 21 have solvability that the available evidence cannot certify either way. The paper stresses that even the certified-unsolved label is narrow: it means the authored solution route passed, infrastructure did not dominate the failures, no strict bypass was observed, and every evaluated agent failed on that task. It does not prove the task is intrinsically hard, that the verifier is complete, or that agents failed at the capability the task was meant to test. The authors also analyze rejected submissions and passing tasks to show that pass rate alone cannot explain why a task is difficult. Their conclusion is that frontier benchmarks should report the evidence behind their all-fail tasks before treating those tasks as capability claims about AI systems.

Key facts

  • The study audits a frozen Terminal-Bench 3 / Frontier-Bench 0.1 record of 1,081 pull requests, 639 scored tasks, 28,801 trials and $105,933 in logged agent spend.
  • Of 125 tasks with no honest pass, an ordered validity screen certifies only 78 as genuinely unsolved.
  • The rest break down as 14 tasks with broken oracles, 8 dominated by infrastructure failures, 4 passable only via verifier bypasses, and 21 whose solvability is uncertified.
  • Even a 'certified-unsolved' label only means the authored solution passed, infrastructure did not dominate, no bypass was observed, and all evaluated agents failed; it does not prove the task is intrinsically hard.
  • The authors argue frontier benchmarks should publish the evidence behind their all-fail tasks before using them as capability claims about AI models.

Why it matters

Benchmark scores are widely used as proof of what AI models can and cannot do, and an all-fail task is often read as evidence of a real capability limit. This study shows that assumption can be wrong: in the record examined, less than two-thirds of nominally unsolved tasks held up under scrutiny, with the rest traceable to broken grading logic, infrastructure problems or exploitable verifiers rather than genuine model failure.

Who it affects

The findings matter most to benchmark designers and maintainers who build and publish agentic coding evaluations like Terminal-Bench, and to researchers and companies who cite all-fail benchmark tasks as evidence about frontier model capability.

How to use it

The paper's ordered validity screen, which combines task artifacts, reference-solution runs, empty-solution controls, adversarial trials, agent trajectories, telemetry and review records, offers a concrete method benchmark builders can apply to their own all-fail tasks before citing them as capability evidence.

How solid is it

The analysis is grounded in a large, itemized production record (1,081 pull requests, 639 tasks, 28,801 trials, over $105,000 in agent spend) and an explicit, multi-stage validity screen with reported category counts for every one of the 125 all-fail tasks, which makes the methodology and its results auditable.

Risks and caveats

The authors themselves caution that their 'certified-unsolved' label is narrow: it confirms the authored solution route worked, infrastructure was not the dominant failure cause, no strict bypass was seen, and every evaluated agent failed, but it does not prove the task is intrinsically hard, that the verifier is complete, or that agents failed at the intended capability. Twenty-one of the 125 tasks remain uncertified either way.

“The same zero pass rate can come from a real capability gap, but it can also come from missing context, a broken reference solution, infrastructure failure, or a verifier that can be bypassed.”

— the paper's authors