Embodied First Exam benchmark: best coding agent reaches 34% on robot tasks
The Embodied First Exam is a benchmark and evaluation framework for coding agents that control robots. The authors start from the observation that early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots. Existing robotics benchmarks, they say, are built around visuomotor policy learning, which leaves it open whether coding agents can reliably control diverse robots under physical-world constraints.
The framework is described as unified, comprehensive and agent-native. Coding agents choose their own control approach: direct control, code to control, or train a policy to control. The authors hope it serves as a first exam for coding agents before they enter the physical world.
The benchmark holds 1,156 tasks in 35 suites on 47 robots. Nine of the suites are new and go beyond pick-and-place to spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control and sim-to-sim engineering.
Each task can be run in up to three evaluation settings. The Standard Setting mimics the real world: the agent receives only sensor observations and, by default, cannot reset the environment. Its Realtime variant keeps the environment running while the agent reasons. The Privileged Setting gives the agent access to the full simulator state and unlimited resets, so it can search for an oracle solution for training purposes. Success rates in the Privileged Setting serve as an approximate upper bound.
The headline result: on the 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, achieves a success rate of 34% in the Standard Setting. The authors say this leaves much room for improvement. To see where agents fail, they label the stage at which failed attempts go wrong and compare failures across agents, settings and stage categories.
Key facts
- The Embodied First Exam has 1,156 tasks in 35 suites on 47 robots, including 9 new suites that go beyond pick-and-place.
- The new suites cover spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control and sim-to-sim engineering.
- On the 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, reaches a 34% success rate in the Standard Setting.
- Each task can run in up to three settings: Standard (sensor observations only, no reset by default), its Realtime variant, and Privileged (full simulator state, unlimited resets).
- Coding agents choose their own control approach: direct control, code to control, or train a policy to control.
Why it matters
Existing robotics benchmarks are built around visuomotor policy learning. The authors argue this leaves open whether coding agents can reliably control diverse robots under physical-world constraints, even as early successes hint at a paradigm shift toward agent-controlled robots. This benchmark is designed around the agent: it lets the agent pick its own way of controlling a robot, rather than fixing a policy-learning setup. A best result of 34% in the Standard Setting gives a first reference point for how far coding agents are from reliable robot control.
Who it affects
Teams building or evaluating coding agents, such as the Codex-based agent that tops the results, are the direct audience. So are robotics researchers who work with simulated robots and want to test agent-driven control instead of trained visuomotor policies. The benchmark spans 47 robots and tasks from spatial puzzles to multi-robot interaction and whole-body control.
How to use it
The framework offers up to three settings per task. The Standard Setting, with sensor observations only and no reset by default, is the one that mimics the real world and is where the headline 34% was measured. The Realtime variant keeps the environment running while the agent reasons. The Privileged Setting gives full simulator state and unlimited resets, and its success rates serve as an approximate upper bound. No code, dataset or leaderboard link is given in the source text.
How solid is it
This is a paper abstract, so the figures are the authors' own claims. The abstract names no authors or institutions. The headline number is narrow: 34% is the best of eight agents, on the 456 core tasks, in the Standard Setting. The names of the other seven agents and their success rates are not given. Success rates in the Privileged Setting and the Realtime variant are not given either.
Risks and caveats
The 34% figure should not be read as a score on all 1,156 tasks; it covers the 456 core tasks. It is not stated how those core tasks were selected out of the 1,156. No breakdown of failure stages or failure causes is given; only that failures are labeled by stage. The source does not say whether the benchmark is simulation-only beyond mentioning the simulator state and sim-to-sim engineering; no real-robot results are reported.
“We hope it serves as a first exam for coding agents before they enter the physical world”
— From the paper's abstract