SWE-CoWork benchmark: top coding agents finish under 8% of assignments

SWE-CoWork is a benchmark that tests coding agents less as solitary programmers and more as employees. It has 101 development assignments across 101 real repositories, 11 programming languages and 12 application domains. Each assignment places the agent in a simulated engineering organization with 26 colleagues, who have distinct responsibilities, bounded knowledge and controlled collaboration behavior.
The agent starts from a brief handoff. From there it must recover requirements that are spread across colleagues and workplace records, reconcile conflicting evidence, and implement changes. Those changes are judged by private executable tests.
The authors evaluate eleven models and report a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests, yet complete fewer than 8% of assignments.
Giving the agents complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra. Complete task success stays unchanged. So missing information is not the whole problem.
A requirement-level trajectory analysis shows failures at several stages of delivery. Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs, but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed. Implementations remain incomplete even after requirements are acquired. And 31% of requirements reported as completed fail their tests.
The authors conclude that sustained requirement tracking, from clarification through implementation, verification and handoff, is a key challenge for reliable autonomous software engineering.
Key facts
- SWE-CoWork has 101 development assignments across 101 real repositories, 11 programming languages and 12 application domains, each set in a simulated organization with 26 colleagues.
- Of eleven models evaluated, the two leading ones satisfy approximately 63% of fail-to-pass tests but complete fewer than 8% of assignments.
- Supplying complete requirements upfront lifts F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, but complete task success does not change.
- Claude-Opus-5.5 sends a mean of 93% of first questions about queried cards to the right owners, yet obtains only 64% of colleague-held requirements.
- 31% of requirements that agents reported as completed fail their tests.
Why it matters
Most coding benchmarks hand the agent a defined task. SWE-CoWork instead makes the agent work out what to build by asking colleagues and reading workplace records, then verify and hand off its work. In that setting the results show a wide split: about 63% of fail-to-pass tests satisfied by the two leading models, but fewer than 8% of assignments fully completed. Partial credit on tests overstates how much real delivery agents manage.
Who it affects
Teams building or buying coding agents for use inside organizations, where requirements sit with several people and in records rather than in one prompt. Researchers who design agent evaluations are also affected, since the paper argues that test pass rates alone miss failures in clarification, tracking and handoff.
How to use it
The practical lesson from the paper is where to look when an agent underdelivers: whether it asked the right people, whether it followed up on deferred questions, whether it finished implementation after getting requirements, and whether its self-reported completions hold up under tests. The abstract does not say whether the benchmark, code or data are released.
How solid is it
This account rests on the paper's abstract, which reports the headline numbers but not per-model scores for the eleven models beyond the figures quoted. The abstract does not name the two leading models. The 93% and 64% figures are given only for Claude-Opus-5.5, and the abstract does not state whether Claude-Opus-5.5 is one of the two leading models. The 16-point gain is stated for Seed Evolving and GPT-6 Astra only. The results come from the authors' own simulated organization, not from real workplaces.
Risks and caveats
The 63% and 8% figures describe the two leading models, not all eleven. The 16-point F2P gain is in percentage points, not a relative improvement. The abstract does not name authors or institutions, and it does not say whether the 16-point gain applies to models other than the two named. Colleagues in the benchmark are simulated, with controlled collaboration behavior, so how closely they resemble real coworkers is not something the abstract settles.
“Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests.”
— SWE-CoWork paper abstract