SWE-Duel arena has coding agents plant and fix each other's bugs
The paper starts from a complaint about static coding benchmarks. They are fixed at release, their tasks leak into training data over time, and stronger agents eventually saturate them. The authors argue that a benchmark should instead be self-scaling: tasks and oracles are created at evaluation time, so no content exists in advance to leak, and difficulty is set by the participating agents rather than fixed at release.
Their answer is SWE-Duel, an arena in which two coding agents evaluate each other over mirrored Red-Blue turns. In each turn the Red agent works on a real repository across three context-isolated sessions. It proposes and implements a new feature with feature tests, then embeds a subtle bug in that feature along with sealed bug-triggering tests.
The resulting pull request is admitted only if two conditions hold. First, it must pass the feature tests and the repository's existing tests while failing exactly the bug tests. Second, a fresh session of the same agent must be able to repair the bug blind, which rules out unsolvable tasks. The Blue agent then reviews the pull request without being told a bug exists. It wins only if its revision passes all three test suites (feature tests, existing tests and bug tests), and then the two agents swap roles.
The authors ran a tournament among 11 model-harness participants on 15 repositories in five languages for 1,500 turns. The leaders win in different ways. GPT-5.5 with Codex CLI ranks first as the strongest solver, repairing 98.3% of the defects it reviews. Claude Opus 4.8 with mini-swe-agent ranks second as the strongest task author, with a 27.3% Red win rate.
One further result concerns difficulty. Within one model family, tasks written by the newest version defeat a fixed-reference solver far more often than those of older versions (42.9% vs. 4.0%). The authors present this as preliminary evidence that task difficulty scales with task authoring capability.
Key facts
- SWE-Duel is an arena where two coding agents evaluate each other over mirrored Red-Blue turns on real repositories, with tasks and oracles created at evaluation time.
- Red adds a feature and hides a subtle bug with sealed tests; the pull request is admitted only if it fails exactly the bug tests and a fresh session of the same agent can fix it blind.
- Blue reviews the pull request without being told a bug exists and wins only if its revision passes all three test suites; then the roles swap.
- The tournament had 11 model-harness participants, 15 repositories in five languages and 1,500 turns. GPT-5.5 with Codex CLI ranked first as solver (98.3% of reviewed defects repaired); Claude Opus 4.8 with mini-swe-agent ranked second as task author (27.3% Red win rate).
- Within one model family, tasks from the newest version beat a fixed-reference solver 42.9% of the time versus 4.0% for older versions, which the authors call preliminary evidence that difficulty scales with authoring capability.
Why it matters
Static coding benchmarks age badly: their tasks leak into training data and strong agents eventually saturate them. SWE-Duel tries to remove both problems by creating tasks and oracles at evaluation time, so there is nothing fixed to leak, and by letting the participating agents set the difficulty. It also measures two skills at once, writing hard tasks (Red) and repairing hidden defects (Blue), instead of only solving.
Who it affects
Mainly people who build or evaluate coding agents and the benchmarks used to compare them. The tournament results also touch the vendors behind the named entrants: GPT-5.5 with Codex CLI led as solver and Claude Opus 4.8 with mini-swe-agent led as task author, with a 27.3% Red win rate.
How to use it
This is a preprint describing an evaluation design, not a product. The mechanics are spelled out in the abstract: three context-isolated Red sessions per turn, an admission check on every pull request, a blind review by Blue, then a role swap. No code, dataset or leaderboard link is mentioned.
How solid is it
The abstract reports concrete figures from a 1,500-turn tournament across 11 participants and 15 repositories, and the admission checks are specified clearly. The authors themselves call the difficulty-scaling result preliminary evidence. The abstract names no authors or institutions, and the 11 participants other than the top two, their rankings and their scores are not listed.
Risks and caveats
The 98.3% repair rate belongs to GPT-5.5 with Codex CLI reviewing defects as Blue; the 27.3% Red win rate belongs to Claude Opus 4.8 with mini-swe-agent. They are different metrics for different roles and should not be merged into one score. The 42.9% vs. 4.0% comparison comes from one model family against a fixed-reference solver, and neither the family nor the solver is named. The 15 repositories and the five languages are also not named.
“We argue that a benchmark should instead aim to be self-scaling”
— SWE-Duel paper abstract