Microsoft's Echoverse pushes a 9B agent within 14 points of GPT-5.4

Microsoft Research describes Echoverse, a pipeline for building synthetic training environments for computer-use agents, the software that clicks, types and navigates real applications on a user's behalf. The team built twelve training worlds: ten deep domain worlds covering communication, technical work, regulated records, community, media and travel, plus two capability worlds that each drill a single hard UI control, such as date pickers and nested filters, across many different layouts. Trained on all twelve worlds, a 9B parameter model nearly doubled its baseline score, from 36.5% to 67.1%, landing within fourteen points of GPT-5.4 on the team's evaluation.

The core problem Echoverse addresses is that the most valuable automation targets, email, banking, health records and internal cloud consoles, sit behind logins and cannot be safely used for repeated training: every action writes to a real account, there is no reset, and the true internal state stays hidden behind the screen. Open, login-free websites are not a substitute either, because they keep changing (redesigns, rolling dates, anti-bot throttling), so a task means something different on every run. Echoverse instead rebuilds each target as a synthetic world, a FastAPI and SQLite backend behind a React interface, where the database is owned by the researchers: the state is real and changes for real, but it is safe to break, quick to reset and graded directly from the data rather than from a screenshot.

Microsoft frames the project around three levers, none of which is simply adding more environments. Depth means a world reproduces an application's actual behavior: permissions, error handling and state that stays coherent across screens and users, for example a cancelled meeting clearing both calendars. Capability targeting means building narrow worlds around one interaction a model keeps failing, rather than another full site. Co-evolution means the loop treats environment-building and model-training as one process: run the model, find where it fails, sharpen the world, its tasks and its verifier at that failure point, retrain, repeat, so the benchmark itself keeps getting harder instead of saturating.

An ablation supports the depth argument directly: trained on shallow and deep builds of the same sites, the model regressed on the shallow versions and improved on the deep ones, meaning superficial clones actively hurt performance rather than being merely useless.

Every task in an Echoverse world carries a verifier grounded in the live database rather than in human labels: a read is checked against the stored value with some tolerance (the source gives $288 passing for a stored $287.62), a write is checked as a real before-and-after database diff, so an agent claiming it closed a ticket fails unless the underlying row actually flipped. This grounded grading is also what supplied the supervised fine-tuning data: GPT-5.4 attempted each generated task, and only trajectories a verifier confirmed against ground truth were kept for training. The report adds that reinforcement learning against the same grounded verifier, used directly as the reward signal, lifted held-out performance further and taught the agent to reach the goal in fewer steps than imitation learning alone.

Among the ten domain worlds, EchoStay is seeded from real InsideAirbnb data (listings, hosts, reviews and amenities) and models a booking flow spanning roughly 87 routes and 23 tables; EchoForum sits on a public corpus of 2.55 million real comments; EchoMail, EchoCare and EchoBank cover mail, health records and banking respectively, domains with no comparable open dataset, so their state was generated under constraints tight enough to keep it dense and internally consistent rather than a handful of placeholder rows. Tasks in these domains often run five to twenty actions deep and only count as finished once the underlying database state has actually changed.

Microsoft is releasing four of the twelve worlds, along with their code, data and grounded graders, on GitHub (microsoft/Echoverse) and Hugging Face, alongside a technical report, to let outside researchers build on the same approach to high-fidelity computer-use training environments.

Key facts

• Twelve training worlds in total: ten deep domain worlds (communication, technical work, regulated records, community, media, travel) and two capability worlds targeting hard UI controls like date pickers and nested filters.
• A 9B model trained on all twelve worlds nearly doubled its score, from 36.5% to 67.1%, coming within fourteen points of GPT-5.4.
• Depth ablation: the same model regressed on shallow clones of a site but improved on deep, behaviorally faithful versions of the same site.
• Verification is grounded in the live database: reads are checked against stored values, writes against a real before/after diff, not against screenshots or human labels.
• EchoStay is seeded from real InsideAirbnb data across about 87 routes and 23 tables; EchoForum draws on 2.55 million real forum comments; four of the twelve worlds ship publicly on GitHub and Hugging Face with code, data and graders.

Why it matters

Computer-use agents can only learn from consequences that are real, and the applications worth automating (email, banking, health records, internal cloud and ML consoles) are closed systems where you cannot safely train against the live version. Echoverse's answer is to rebuild those systems as synthetic worlds the researchers own end to end, and to keep improving the world alongside the model rather than treating environment-building as a one-off setup step. The reported jump, a 9B model nearly doubling its score and closing most of the gap to GPT-5.4, is offered as evidence that this co-evolving loop, not simply adding more environments, is what currently limits computer-use agent training.

Who it affects

Researchers and engineering teams building or evaluating computer-use agents, and anyone trying to automate work inside closed, login-gated systems such as corporate mail, internal admin consoles, banking or health-record software, where public benchmarks built on open websites do not transfer well.

How to use it

Microsoft is releasing four of the twelve Echoverse worlds publicly, with their code, data and grounded graders, on GitHub under microsoft/Echoverse and on Hugging Face, alongside a technical report describing the full pipeline. This is a research release with no pricing or product tier attached; the value is the reusable environments and the method for building more of them, not a hosted service.

How solid is it

The claims come from Microsoft Research's own blog post and technical report, with training worlds, code, data and evaluation numbers published on GitHub and Hugging Face rather than only asserted in prose. The headline result, 36.5% to 67.1% for a 9B model against a GPT-5.4 reference point, is Microsoft's own internal evaluation; the post does not describe independent replication by outside labs.

Risks and caveats

Synthetic worlds trade some surface realism for control and reproducibility, an explicit tradeoff the post itself acknowledges. The reported numbers and the depth ablation come from a single organization's internal experiments rather than a third-party benchmark, so the exact size of the improvement should be read with that in mind even though the underlying worlds and code are open for others to check. The database-grounded verifier is designed to resist an agent gaming appearances, but it is still Microsoft's own grading pipeline being used to evaluate Microsoft's own training method.

“A screenshot can show what an interface looks like, but only a working world shows what an action caused.”

— Microsoft Research, Echoverse blog post