Co-Design Bench tests coding agents on 273 real multimodal app edits
Co-Design Bench is a new benchmark that asks whether coding agents can turn real users' multimodal requests into working revisions of an existing application. The authors call it reference-free and build it from 273 tasks, curated from 37,463 deduplicated queries drawn from 26,314 real user coding sessions.
The benchmark has two tracks. Multimodal Editing (ME) covers localized edits that may come with image, audio, video or document inputs. Repository Editing (RE) covers coordinated revisions across a repository. Tasks span three capability dimensions: Grounded Understanding, UI Transformation and Functional Implementation. Each task pairs a normalized request and a runnable pre-edit application with hidden, expert-authored acceptance criteria; there are 2,331 such criteria across the benchmark.
Grading uses what the authors call a judge-then-refute protocol. Criteria are graded using source and browser evidence. Positive verdicts are then challenged by a separate LLM verifier. Visual and interaction claims are audited without gold code or target screenshots.
The headline result comes from running ten model endpoints under a shared agentic harness. The strongest endpoint passes an average of 69.9% of criteria per task, yet fully satisfies only 26.4% of tasks. In other words, it gets most individual requirements right but rarely gets every requirement of a task right at once. The authors also report that all ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two.
The authors present Co-Design Bench as a testbed for studying the gap between partial requirement satisfaction and complete task success under agentic execution, for real-user application revisions.
Key facts
- Co-Design Bench has 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions, with 2,331 hidden expert-authored acceptance criteria.
- Two tracks: Multimodal Editing (localized edits with optional image, audio, video or document inputs) and Repository Editing (coordinated repository revisions).
- Under a shared agentic harness, the strongest of ten model endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks.
- All ten endpoints score lower criterion completion on repository edits spanning three or four change types than on those spanning one or two.
- Grading uses a judge-then-refute protocol: evidence-based grading, then a separate LLM verifier challenges positive verdicts.
Why it matters
Many coding benchmarks start from curated problems. This one starts from real user sessions and asks for working revisions of runnable applications, including requests that come with images, audio, video or documents. The central finding is the gap between partial and full success: 69.9% of criteria per task for the best endpoint, but only 26.4% of tasks fully satisfied. A user who asks for a change usually needs all of it, so the second number is the one that matches the experience of using an agent.
Who it affects
People building or evaluating coding agents are the main audience, since the benchmark targets agentic execution on application edits. It also bears on users who hand agents mixed requests such as a screenshot plus a description, and on teams who ask agents to make coordinated changes across a repository.
How to use it
The authors offer Co-Design Bench as a testbed for studying the gap between partial requirement satisfaction and complete task success. Its two tracks and three capability dimensions (Grounded Understanding, UI Transformation, Functional Implementation) let evaluators look at where an agent falls short. No link to released code or data is stated.
How solid is it
The numbers are the authors' own results on their own benchmark, as given in the paper's abstract. The grading design has safeguards: criteria are hidden and expert-authored, and a separate LLM verifier challenges positive verdicts. The 69.9% figure is an average of per-task criterion pass rates for the best endpoint only; no figures for the other nine endpoints are given. The abstract does not name the strongest model endpoint or any of the ten endpoints evaluated, and the source does not say which agentic harness was used.
Risks and caveats
Without the endpoint names, readers cannot tell which models the 26.4% describes. No numbers are given for the gap between one-or-two and three-or-four change-type repository edits, only the direction. Grading relies partly on an LLM verifier, so the results depend on how well that protocol works. The benchmark has 273 tasks, so it is a focused sample of real sessions rather than the full set of 26,314.