Co-Design Bench: best coding agent GPT-6 Astra fully solves only 26.4% of tasks
Researchers from Alibaba and several Asian universities have built Co-Design Bench, a new test for coding agents drawn from real user change requests. The agents must rework existing web apps based on those requests, sometimes using images, audio recordings, videos or documents as input.
According to the paper, GPT-6 Astra, the best of the ten models tested, meets about 70 percent of a task's requirements on average. But the result is fully correct in only 26.4 percent of tasks. In most cases the agent handles the bulk of the work yet leaves at least one detail unfinished, so anyone using these tools almost always has to clean up by hand.
The authors justify the new test by pointing to the saturation of existing benchmarks. When models can run, check and fix their own code, they score above 99 percent on those tests, and differences between models can barely be measured. Co-Design Bench instead uses real requests from actual coding sessions. The researchers favored cases where users had to rephrase their request several times or roll back changes, so the test holds mainly tasks that already gave AI agents trouble in practice. Tasks range from small tweaks, such as pulling an amount from a receipt and updating every dependent total, to full-scale project overhauls.
Grading is strict. One AI judge tests the finished app in a browser, then a second judge re-checks every positive verdict and looks for mock-ups or hardcoded results. Points go only to features that actually work in the app.
Behind GPT-6 Astra, Claude Opus 5 and Qwen3.8-Max take the next spots, and Grok 4.6 is last. On larger overhauls the much smaller Qwen3.8-27B climbs to second place, but the authors do not derive a general advantage of that model from the result. Performance drops across all models as a task combines more types of changes, including layout, design, interaction and content, and such intertwined requirements are the norm in real projects. More compute does not fix this: GLM-5.2 uses nearly six times as many tokens as GPT-6 Astra on smaller changes and still scores significantly worse.
In a separate test the researchers ran the same models in different harnesses, the programs through which an agent reads, writes and executes code. Qwen3.8-27B scored 75 or just 56 percent on larger overhauls, depending on the environment. Which of two models came out ahead also depended on the environment. The authors therefore recommend always reporting model and working environment together; for companies picking a coding agent, that means testing the two as a pair.
The article adds outside context. A METR study already showed how far benchmark results and real-world usability can diverge: the actual acceptance rate of AI-generated code by project leads was roughly 24 percentage points below scores on SWE-bench Verified. Flaws in test environments can also let agents pass tasks without solving them. The effect of the working environment is not clear-cut, though: a recent study on coding agents found barely any measurable benefit from more elaborate harnesses, even though they consumed up to three times as many tokens.
Key facts
- Co-Design Bench, built by researchers from Alibaba and several Asian universities, tests coding agents on real user change requests to existing web apps, some with images, audio, video or documents as input.
- According to the paper, the best of ten models, GPT-6 Astra, meets about 70 percent of a task's requirements on average but is fully correct on only 26.4 percent of tasks.
- Existing coding benchmarks are saturated: models that can run, check and fix their own code score above 99 percent on them, according to the authors.
- Results vary with the harness: Qwen3.8-27B scored 75 or 56 percent on larger overhauls depending on the environment, and the authors recommend reporting model and environment together.
- Spending more tokens does not help: GLM-5.2 used nearly six times as many tokens as GPT-6 Astra on smaller changes and still scored significantly worse.
Why it matters
Older coding benchmarks have grown too easy: with the ability to run, check and fix their own code, models score above 99 percent, and the differences between them can barely be measured. Co-Design Bench is built to bring the gap back into view. It uses requests from real coding sessions, favors cases that gave AI agents trouble in practice, and grades strictly. Under those conditions the best model, GPT-6 Astra, delivers a fully correct result on only 26.4 percent of tasks, even though it meets about 70 percent of requirements on average. The pattern is that agents do most of the work and miss a detail.
Who it affects
Teams and companies that pick or use coding agents are the main audience. The source says that anyone using these tools almost always has to clean up by hand, and that companies choosing a coding agent should test the model and its harness as a pair. Benchmark and leaderboard makers are also addressed: the authors argue that leaderboards listing only the model tell only part of the story.
How to use it
The practical takeaway from the authors is to report and evaluate model and working environment together, since the same model can score very differently in different harnesses (Qwen3.8-27B: 75 or 56 percent on larger overhauls). For a buying decision, that means testing the specific model and harness combination you plan to use rather than relying on a model-only ranking. The source does not describe how to obtain or run the benchmark.
How solid is it
The figures come from a paper by researchers from Alibaba and several Asian universities, as reported by The Decoder. Grading uses two AI judges: one tests the app in a browser, and a second re-checks every positive verdict and looks for mock-ups or hardcoded results. The source itself notes that the AI judges can make mistakes. It does not name the judge models or the harnesses compared, and it gives no scores for Claude Opus 5, Qwen3.8-Max or Grok 4.6. The METR comparison is earlier, separate work cited for context.
Risks and caveats
The benchmark evaluates one fixed request per task; follow-up questions or mid-task corrections from users are not part of the test, so it does not capture how people iterate with an agent in practice. The AI judges can err. The strong showing of the much smaller Qwen3.8-27B on larger overhauls is not read by the authors as a general advantage of that model. The effect of the environment is also not clear-cut: a recent study found barely any measurable benefit from more elaborate harnesses, even though they consumed up to three times as many tokens. The study is not named in the source.