Glonce

CalibForge synthesizes 5,431 calibrated tasks to train terminal agents

researchThe Glonce editorsHugging Face🔥77Glonce rating77Impact62Novelty68Relevance92Surprise42Players22Credibility82
CalibForge synthesizes 5,431 calibrated tasks to train terminal agents

Training terminal agents needs tasks that are executable and verifiable, but that alone does not guarantee a task is pitched at the right difficulty for a given solver. A paper describes CalibForge, an autonomous terminal-task synthesis system that revises candidate tasks using verified solver behavior, a process the authors call adversarial solver calibration.

CalibForge combines two strategies. Multi-solver calibration targets disagreement within a heterogeneous pool of solvers. Contrastive solver calibration targets a designated strong-pass, weak-fail relation, meaning a stronger solver succeeds on a task while a weaker one fails it. Both strategies aim at what the paper calls a solver-relative learnable zone, anchored in demonstrated solvability rather than just executability.

Using CalibForge, the researchers built 5,431 calibrated terminal tasks. Their ablations show that both calibration strategies produce more effective supervision than authoring and validation alone, and more effective than ordinary single-solver feedback. Models trained on the full collection of tasks scored 32.58% and 47.57% on Terminal-Bench 2.0, two figures the source reports side by side without further distinguishing what separates them. The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo.

The authors argue these results support solver-relative learnability as a practical target for constructing effective and transferable agent training data.

Key facts

  • CalibForge is an autonomous system that revises candidate terminal-agent tasks using verified solver behavior, a process the paper calls adversarial solver calibration.
  • It combines two strategies: multi-solver calibration, which targets disagreement across a heterogeneous solver pool, and contrastive calibration, which targets a designated strong-pass, weak-fail relation.
  • Using CalibForge, the researchers constructed 5,431 calibrated terminal tasks.
  • Models trained on the full task collection scored 32.58% and 47.57% on Terminal-Bench 2.0.
  • The largest gains over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo.

Why it matters

Training data for terminal agents needs tasks that are not just solvable and verifiable but calibrated to the right difficulty for a given solver. Executable validation only proves a task can be completed; it does not show how the task behaves relative to a solver's actual capability. CalibForge addresses that gap by using solver behavior itself, rather than authoring judgment alone, to revise candidate tasks toward a learnable zone.

Who it affects

Teams that build or train terminal and coding agents, and anyone who evaluates such agents against Terminal-Bench 2.0, SWE-bench Pro, or Doc2Repo. The source text does not name the authors or their institution, so who built the system is not established here.

How to use it

CalibForge is presented as a research method, not a released product; the source gives no pricing, license, or API. It describes the calibration technique and reports that applying it produced 5,431 calibrated terminal tasks, which were then used to train models evaluated on the three benchmarks named above.

How solid is it

The claims rest on the paper's own ablations, which compare multi-solver and contrastive calibration against authoring-and-validation-only and ordinary single-solver feedback baselines, and report gains across three separate benchmarks. The source does not give author names, an institution, a publication date, or the identity of the corresponding base model used for the improvement figures, which limits how far the results can be checked from the text alone.

Risks and caveats

The text does not explain what distinguishes the two reported Terminal-Bench 2.0 scores, 32.58% and 47.57%, from each other, nor does it identify the specific solvers that made up the heterogeneous pool, nor does it compare task count or quality against any competing task-synthesis system. The reported gains are self-reported by the paper's own ablations and have not been independently verified here.