ProgramBench Vetted tightens Meta's binary reverse-engineering benchmark

ProgramBench, released by Meta in May 2026, tests whether a coding agent can reconstruct a program's source code purely from a runnable but unreadable compiled binary plus its usage documentation. Each of its 200 tasks is a real open source CLI project, ranging from small utilities up to FFmpeg, SQLite and the PHP interpreter. At evaluation time the agent gets a shell, no internet access, a thousand steps and six hours; it can execute the sealed reference binary but has no access to the original source, git history or repository tests. A behavioral test suite, itself written by an agent that probed the original repository and binary, grades the result, and a task counts as resolved only when every test in that suite passes.

Vetto.ai has now released ProgramBench Vetted, a 50-task set built on the same premise but reworked to close weaknesses found while auditing the original dataset. The audit turned up concrete exploits: on the cmatrix task, a response that just echoes the documentation clears 79.5% of the test suite without implementing any of the program's real behavior; on the XZ task, an eight-line wrapper around the XZ executable that is already installed in the task environment earns 77.3% of the active reward; and the FFmpeg task depends on reference inputs that are not actually present. ProgramBench Vetted's construction pipeline is built to catch cases like these before a task ships.

Building each task couples a cleanroom reconstruction environment with a behavioral test suite. Continuous review by researchers seeds specialized "Agents as Judges," which investigate a candidate task, test its behavior and produce scored reports, and "Agents as Doctors," which use that evidence to make a constrained repair before the task is checked again by the same judge. Deterministic gates cover properties that can be measured exactly, and separate adversarial agents actively try to exploit each task or its reward. Five frontier models independently explore each candidate through as many as 15 test generation trials; a generated check must pass against the real ("gold") binary and fail against a dummy implementation before it can count toward the score, and deduplication then removes redundant or subsumed checks. When an automated review loop hits its round limit, a human reviewer inspects the artifacts and can reject the task, repair and approve it, or hand it back to the relevant doctor agent with explicit instructions; those findings become stronger criteria for later runs.

On the resulting leaderboard, GPT 5.5 leads ProgramBench Vetted by mean reward at 85.2% but fully resolves only 2% of tasks, while Claude Opus 5 posts a lower mean reward of 81.9% yet fully resolves 14%, the highest rate among the 17 models evaluated; six of those 17 models resolve at least one task outright. The team calibrated task difficulty so most tasks sit in a middle band rather than clustering at the extremes: 46 of the 50 tasks have a mean fractional test score, across the full model panel, between 20% and 80%. They caution that because the two benchmarks were evaluated on materially different, only partly overlapping model sets, the score-distribution comparison does not show that either version is intrinsically harder than the other.

Key facts

ProgramBench Vetted contains 50 tasks, versus 200 in Meta's original ProgramBench (released May 2026); each requires an agent to rebuild a CLI program's source from a compiled binary within a thousand steps and six hours, with no internet access.
Audits of the original benchmark found real exploits: a documentation echo clears 79.5% of the cmatrix test suite, and an eight-line wrapper around a pre-installed XZ binary earns 77.3% of active reward, in both cases without reconstructing the program's real behavior.
The vetted pipeline layers "Agents as Judges" and "Agents as Doctors" with deterministic gates and adversarial agents, and escalates unresolved cases to human reviewers who can reject, repair, or send a task back for another fix.
Five frontier models generate up to 15 test-generation trials per candidate task, and deduplication removes redundant checks before any test can count toward the reward.
On the new leaderboard, GPT 5.5 leads by mean reward (85.2%) but fully resolves only 2% of tasks; Claude Opus 5 trails on mean reward (81.9%) but leads full resolution at 14%, and only 6 of 17 evaluated models resolve any task outright.

Why it matters

Reconstructing a program from its compiled binary is one of the few coding-benchmark formats built around very long-horizon work: an agent has to build and revise a model of the program across an extended trajectory, which makes it useful for studying long-context coordination. It also suits reinforcement learning from verifiable rewards, since many deterministic behavioral checks provide dense reward signal that an agent has to integrate into one coherent implementation. Because the tasks come from public repositories, models may have encountered the code during pretraining; the authors argue this does not mean a model can retrieve or reconstruct the implementation from memory when it only has the compiled binary's observed behavior to work from, drawing an analogy to the Reversal Curse. A benchmark like this is only useful if its scores reflect the skill it claims to measure, which is why so much of the vetted version's effort goes into closing exploits: a benchmark that can be gamed for a high score stops measuring reverse-engineering ability at all.

Who it affects

The direct audience is teams building and evaluating coding agents: frontier labs whose models sit on the leaderboard, with GPT 5.5 and Claude Opus 5 both named, and anyone using program-reconstruction tasks as a reinforcement-learning-from-verifiable-rewards training environment, where a gameable test suite would let a model farm reward without learning the underlying skill. It also matters to people building or auditing benchmarks generally: the specific exploits found in the original ProgramBench, a documentation echo, a wrapper around a preinstalled binary, missing reference inputs, are generic failure modes that can turn up in any behavioral-test benchmark, not just this one.

How to use it

The benchmark itself defines its own usage: an agent is handed a sealed container holding only the compiled binary and its usage documentation, with source, git history and repository tests stripped out. It gets a shell, no internet access, a thousand steps and six hours to rebuild a working codebase, and its reconstruction is graded by the fraction of held-out behavioral tests it passes; a task counts as resolved only once every test passes. The available text does not say where or how to obtain the ProgramBench Vetted task set itself.

How solid is it

The methodology described is layered rather than a single review pass. Automated "Agents as Judges" score each candidate task and produce evidence; paired "Agents as Doctors" repair fixable problems and resend the task to the same judge; deterministic gates check properties that can be measured exactly, such as whether a generated test passes against the real binary and fails against a dummy one; and separate adversarial agents actively try to break or exploit each task. Five frontier models each run up to 15 independent test-generation trials per candidate, and deduplication removes tests that cover the same or a subsumed behavior. When an automated loop hits its retry limit, the case escalates to a human reviewer who can reject, repair and approve, or send it back to a doctor agent with explicit instructions, and recurring diagnoses get turned into new checks for future runs. On difficulty calibration, the team ran multiple models against each candidate and studied the resulting distribution of partial credit, revising tests, documentation or environment whenever a score did not reflect the behavior it was meant to measure; the result is that 46 of the 50 final tasks land with a mean fractional score between 20% and 80% rather than clustering at the extremes. The team is explicit that this rigor does not let them claim ProgramBench Vetted is simply harder or easier than the original: the two leaderboards evaluate materially different, only partly overlapping sets of models, so the direction of any change is mixed rather than uniform.

Risks and caveats

The captured source text cuts off mid sentence near the end, so whatever conclusion followed the difficulty-distribution discussion is not available here. No individual researcher is named anywhere in the text, only "Meta" for the original benchmark and an unnamed "we" for the team publishing the vetted version. Even after all the added scrutiny, the benchmark stays difficult: the leading model by mean reward, GPT 5.5, still fully resolves only 2% of the 50 tasks, and across all 17 evaluated models, only six resolve so much as a single task outright, a reminder that most of the reported reward is partial credit rather than a complete working reconstruction.

“Yet prior exposure does not imply that a model can retrieve or reconstruct the implementation from memory when asked to move from observed behavior back to code.”

— ProgramBench Vetted announcement