ProgramDistill: GPT-6 Astra beats Claude Opus 5 on undocumented-feature benchmark
Researchers introduced ProgramDistill, a benchmark that evaluates coding agents on features they have to infer by interacting with fully functional reference web applications, rather than being told what to build through issues or instructions. The authors note that coding agents are typically evaluated against desired behavior spelled out in writing, but in practical web development an agent may instead need to infer behavior from working software and reproduce it in an incomplete version of the application.
To build the benchmark, the team created a pipeline called mine-craft-patch, which factorizes applications into features at different levels of granularity, each tied to a replayable behavior that can be executed via its gold patch. Running this pipeline discovered 1,975 replay-verified behaviors across 26 applications and constructed 4,063 tasks without any human intervention.
Nine frontier coding agents were then evaluated on the resulting tasks. On cumulative workflows in full-application reconstruction, GPT-6 Astra achieved 49.2% success and Claude Opus 5 achieved 28.8%. In partial-application reconstruction, success rates fell as restoration depth increased from 1 to 8: in one case from 100% down to 64.0%, and in another from 96% down to 32%. The source text does not restate by name which of the two agents corresponds to which of these two result pairs.
The authors present ProgramDistill as a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and as a natural basis for future curriculum-based training.
Key facts
- ProgramDistill evaluates coding agents on features discovered by interacting with fully functional reference web apps, not from issues or instructions.
- The mine-craft-patch pipeline discovered 1,975 replay-verified behaviors across 26 applications and built 4,063 tasks without human intervention.
- Among nine frontier coding agents tested, GPT-6 Astra reached 49.2% success on cumulative workflows in full-application reconstruction, versus 28.8% for Claude Opus 5.
- In partial-application reconstruction, success dropped as restoration depth rose from 1 to 8: from 100% to 64.0% in one case and from 96% to 32% in another.
- The authors present ProgramDistill as a scalable, difficulty-controlled benchmark for diagnosing coding agents and a basis for future curriculum-based training.
Why it matters
Most coding-agent benchmarks hand the agent a written spec: an issue, a ticket, a set of instructions. ProgramDistill instead measures whether an agent can figure out what a feature is supposed to do just by using a working reference application and then reproduce that behavior in an incomplete copy of the app. That is closer to a common real situation in web development, where the working software is the closest thing to a specification, and it targets a gap the authors say issue-based benchmarks do not test.
Who it affects
Teams building and evaluating coding agents, including the makers of the nine frontier agents tested, among them GPT-6 Astra and Claude Opus 5. It also matters to researchers who need a way to measure how well an agent handles undocumented, partially specified work rather than clearly stated tickets.
How to use it
ProgramDistill is a benchmark and task set, not a product: 4,063 tasks built automatically from 26 applications via the mine-craft-patch pipeline, with no human curation involved in constructing them. The source gives no pricing or licensing terms. The authors describe it as usable both for evaluating and diagnosing current coding agents and, going forward, as a basis for curriculum-based training.
How solid is it
The described pipeline is systematic: it discovered 1,975 replay-verified behaviors across the 26 applications and turned them into 4,063 tasks, and each behavior is checked against a gold patch rather than judged subjectively. Results are reported for nine frontier agents, with GPT-6 Astra and Claude Opus 5 named explicitly and specific success percentages given for both full- and partial-application reconstruction. The available text names no human authors, institutions, publication venue or release date, and gives no comparison to prior benchmarks beyond the nine agents evaluated here.
Risks and caveats
The text does not explain what specifically defines a 'restoration depth' level beyond the depth values 1 through 8, so the practical meaning of that difficulty axis is not fully spelled out. It also does not restate by name which of GPT-6 Astra or Claude Opus 5 produced the 100% to 64.0% partial-reconstruction result versus the 96% to 32% result, so that pairing should be read as unspecified rather than assumed from the order results are listed in elsewhere.