WhatWorkedBench tests whether AI research agents actually understand their own experiments
Researchers introduced WhatWorkedBench, a benchmark built to measure what they call experimental understanding: how accurately an AI research agent can predict the effect of changing one component of an experiment after it has been given a limited budget to run experiments itself. In the benchmark, an agent inspects code, chooses which measurements to run, and then submits a response surface, a table predicting the score for every possible configuration of component settings. To grade those predictions, the benchmark relies on exhaustive CPU execution: every component is changed in turn while all the others are held fixed, producing reference effects that show what changing each component actually does to the outcome.
The benchmark spans 36 tasks drawn from 30 data sources across 8 workflow types, with 1,248 configuration records capturing combinations of component changes. Its core evaluation set combines 4,206 numerical-control records across all eight families with 108 agent episodes across the original six workflow types.
On the results side: when agents are given eight new measurements, a method the authors call pair-effect ridge selects the actual optimal configuration on 15 of 22 data sources, and on three of those sources it keeps every effect-size error within 10% of the score's range. Fitting a Gaussian process to the same observations the agents already gathered raises effect recovery, meaning accuracy relative to the true effect magnitude, from 0.632 to 0.698 in the original Flash cohort, and from 0.621 to 0.720 in an additional cohort. On six completed beat-detection and graph submissions, the same approach raises family-macro recovery from 0.303 to 0.455. On six workflows that each had six binary options, given 20 new measurements, encoding code equivalences (recognizing configurations that behave identically) raises Gaussian-process recovery from 0.248 to 0.462.
The authors say WhatWorkedBench is meant to support further research on experimental agents, adaptive experimental design, numerical inference, and better use of program structure by AI systems.
Key facts
- WhatWorkedBench covers 36 tasks from 30 data sources across 8 workflow types, with 1,248 configuration records of component changes
- The core evaluation combines 4,206 numerical-control records across all eight families with 108 agent episodes across the original six workflow types
- Given eight new measurements, pair-effect ridge selects the true optimal configuration on 15 of 22 data sources, keeping error within 10% of score range on three of them
- Fitting a Gaussian process to the same agent observations raises effect recovery from 0.632 to 0.698 (Flash cohort) and from 0.621 to 0.720 (an additional cohort)
- Encoding code equivalences raises Gaussian-process recovery from 0.248 to 0.462 on six workflows with six binary options at 20 new measurements
Why it matters
AI research agents are increasingly asked to run experiments and tune configurations on their own, but that only works if the agent actually understands how a given change affects the outcome, rather than just guessing well after the fact. WhatWorkedBench isolates that specific skill by grading an agent's predicted effects against reference effects obtained from exhaustive, brute-force execution.
Who it affects
The benchmark is aimed at people building and evaluating AI research agents, and more broadly at researchers working on adaptive experimental design, numerical inference, and how agents can use the structure of code to reason about experiments.
How to use it
WhatWorkedBench is an evaluation setup rather than a product: an agent inspects code, picks a limited number of measurements to run, and submits a response surface predicting scores across configurations, which is then checked against ground-truth effects computed by exhaustively running every configuration on CPU. The source gives no pricing, release, or access details for the benchmark itself.
How solid is it
The account is built entirely on the paper's own reported figures: task counts, record counts, and recovery scores across several cohorts and methods. The text does not name the authors, their institution, a publication date, or the specific identities of the AI models tested beyond labels like the 'Flash cohort', so the results should be read as the authors' own reporting rather than independently confirmed findings.
Risks and caveats
Even the best-performing methods described leave substantial room for error: effect recovery scores such as 0.462 or 0.72 are well short of perfect prediction, meaning agents still frequently misjudge how much a given change actually moves the outcome. The benchmark's coverage is also bounded, 36 tasks across 30 sources and 8 workflow types, and the source does not specify what the 'original six' workflow types or the 'eight families' in the core evaluation actually consist of.
“Exhaustive CPU execution supplies reference effects for changing each component while holding the others fixed.”
— WhatWorkedBench paper