Glonce

GPT-6 Astra leads new computer-use benchmark but fully passes just 2.8%

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact70Novelty65Relevance100Surprise55Players60Credibility85
GPT-6 Astra leads new computer-use benchmark but fully passes just 2.8%

Researchers have introduced RecreationWorld, a training and evaluation framework for what they call hybrid computer-use agents (CUAs): agents that decide for themselves when to click through a graphical interface, when to write and run code, and when to check their own output, rather than being limited to one mode. The framework is built around a single task: given a running reference application, an agent must figure out how it behaves and build a faithful copy of it, with no prescribed steps for doing so. RecreationWorld supplies reproducible environments on five platforms: Ubuntu, macOS, Windows, Android and Web, along with one harness that gives agents both native GUI control and coding tools. The running reference application doubles as an oracle: it is used to generate hidden behavioral tests, which in turn produce rewards tied to whether the agent's rebuilt version actually executes correctly, not just whether it looks right.

Training agents on trajectories collected this way carried over: models trained on RecreationWorld trajectories improved on five separate out-of-distribution coding and hybrid computer-use benchmarks, and checked their own rendered output more often than before, which the researchers read as evidence the skill transfers beyond the recreation task itself.

To measure where things stand, the researchers built a held-out benchmark, RecreationBench, comprising 250 tasks spread across different domains and platforms. Each task's grading combines programmatic and visual assertions tied back to the reference application, covering outcomes at several depths of interaction; every assertion is checked against the reference itself and reviewed by a human before the suite is frozen for automatic scoring. On this benchmark, GPT-6 Astra comes out on top with 58.1% overall, but that aggregate score conceals a much narrower result: the model passes every programmatic test on only 2.8% of tasks. The researchers trace part of the gap to a consistent pattern: agents reproduce the static structure of an interface more reliably than they reproduce its interactions and computed outputs, and the applications they build stay smaller and more monolithic than the originals they are copying.

Key facts

  • RecreationWorld provides reproducible environments across five platforms: Ubuntu, macOS, Windows, Android and Web, plus a single harness combining native GUI control with coding tools.
  • Agents must recreate a running reference application with no prescribed workflow; the reference itself acts as an oracle for hidden behavioral tests, producing execution-grounded rewards.
  • Models trained on RecreationWorld trajectories improved across five separate out-of-distribution coding and hybrid computer-use benchmarks, and verified their own rendered output more often.
  • On the held-out RecreationBench (250 tasks), GPT-6 Astra scores highest overall at 58.1%, but passes every programmatic test on just 2.8% of tasks.
  • Agents reproduce a static interface's structure more reliably than its interactions and computed outputs, and the applications they generate remain smaller and more monolithic than the references.

Why it matters

Computer-use agents have developed along two separate tracks: interacting with a graphical interface, and writing software through code and the command line. The researchers argue real digital work needs both, interleaved rather than run one after the other. RecreationWorld is built to train and test agents that make that choice themselves, using the task of faithfully recreating a real, running application as the signal for whether they got it right.

Who it affects

The work is aimed at people building or evaluating computer-use and coding agents, the kind that operate a desktop, browser or mobile interface and then have to check that what they built or changed actually works. It also targets benchmark design itself: RecreationBench is meant to replace static rubrics with a harness that has an executable reference to check answers against, across five platforms at once.

How to use it

The researchers state they are releasing the benchmark, the environments and the test suites. RecreationBench's 250 tasks are graded by assertions, both programmatic and visual, that are tied to the reference application and cover outcomes at multiple depths of interaction; each assertion is checked against the reference and reviewed by a human before the suite is frozen, so scoring afterward runs automatically.

How solid is it

The grading is reference-grounded rather than judged by a rubric alone: every test in RecreationBench is validated against the actual running reference application and by a human reviewer before the suite is frozen for automatic scoring. That design lets the benchmark check whether a rebuilt application behaves like the original, not only whether it looks like it, which is a stricter bar than most interface-recreation tests apply.

Risks and caveats

The headline number is easy to misread on its own. GPT-6 Astra's 58.1% overall is an aggregate that gives partial credit, while the stricter measure, passing every programmatic test on a task, succeeds only 2.8% of the time. The researchers' own qualitative finding explains part of the gap: agents copy an interface's static structure reliably but struggle with its interactions and computed outputs, and the applications they produce stay smaller and more monolithic than the references they are copying. No other model's score, no author list, and no release timeline appear in the source, so this result is best read as evidence of a specific weak point in current agents rather than as a comparative leaderboard.