AutoWorldModel-Bench tests whether Codex-5.4 and Claude Opus 4.6 can do open-ended research
Researchers introduce AutoWorldModel-Bench, a closed-loop benchmark built to test whether frontier coding agents can act as autonomous researchers rather than execute a pre-specified engineering task. The authors frame world modeling as an unsettled field: architecture, training objective, and state representation interact in complex ways, and no single recipe dominates across environments. That makes it a testbed where the improvement direction is not specified in advance, unlike the engineering-to-spec tasks that dominate current agent benchmarks.
The benchmark spans eight game environments, each exposed through a unified structured-state representation: ground-truth entity state extracted from the game and consumed through a shared tensor format. This isolates dynamics modeling from perception and keeps each run to minutes rather than longer training cycles, so iteration is fast. In each session an agent receives a starter world model and a fixed compute budget, then works autonomously to improve it, with no target metric or fixed recipe handed to it in advance.
Across 64 sessions, two frontier coding agents, Codex-5.4 and Claude Opus 4.6, were evaluated together. They improved their starter world model in 63 of the 64 sessions. In 91% of sessions, the edit that won was not a routine hyperparameter tweak: it was a non-trivial research-style change, meaning a new training objective, a new state representation, a new rollout procedure, or an architectural change. The authors present the results as evidence that frontier coding agents can already be evaluated on open-ended research problems, not only on tasks where the solution path is already specified.
Key facts
- AutoWorldModel-Bench is a closed-loop benchmark in which frontier coding agents autonomously improve a provided world-model starter under a fixed compute budget.
- The benchmark spans eight game environments under a unified structured-state representation that isolates dynamics modeling from perception.
- Across 64 sessions, Codex-5.4 and Claude Opus 4.6 together improved their starter world model in 63 of the 64 sessions.
- In 91% of sessions the winning edit was a non-trivial research-style modification, such as a new objective, representation, rollout procedure, or architectural change, rather than a hyperparameter tweak.
- The authors offer the benchmark as a setting for evaluating coding agents on open-ended research rather than engineering-to-spec problems.
Why it matters
Most agent benchmarks hand an agent a fixed specification and score whether it hits it. AutoWorldModel-Bench does the opposite: it gives coding agents a starter world model and a compute budget, and leaves the improvement direction open, closer to what a human researcher does than what an engineer executing a ticket does. The 91% figure is the headline result: when Codex-5.4 and Claude Opus 4.6 found a winning edit, it was almost always a genuine research move, a new objective, representation, rollout procedure, or architecture, rather than a parameter sweep. That is offered as evidence that current coding agents can already operate past the point of executing a known recipe.
Who it affects
The benchmark speaks most directly to teams building and evaluating coding agents, including the labs behind Codex and Claude, and to researchers who design agent benchmarks and want a testbed that scores open-ended improvement rather than spec compliance. It also bears on the world-modeling research community, since the benchmark's structured-state format and eight game environments are offered as a shared evaluation surface for that subfield.
How to use it
The source gives no code availability, dataset availability, or release date for AutoWorldModel-Bench, so there is nothing yet to point to for running it directly.
How solid is it
The result rests on 64 sessions across eight game environments, with a unified structured-state representation meant to keep runs to minutes and to separate dynamics modeling from perception, both choices aimed at making iteration and comparison cheap. Two frontier agents were evaluated, Codex-5.4 and Claude Opus 4.6, and the source reports their combined outcome (63 of 64 sessions improved, 91% non-trivial edits) rather than breaking the numbers out per agent, so it is not stated whether the two performed similarly or how the 64 sessions split between them. The source also does not define what counts as an 'improvement', no specific metric or scoring criterion is given.
Risks and caveats
Several details needed to judge the result closely are missing from the source: no value or unit is given for the fixed compute budget the agents operate under, the names of the eight game environments are not listed, and no authors or institutions are named. Without a stated improvement metric, it is hard to independently judge how meaningful the 63-of-64 success rate is, and without a compute-budget figure, it is hard to judge how much room the agents actually had to experiment.