WM^MLLM uses MLLMs to build persistent, editable worlds from one image

WM^MLLM uses MLLMs to build persistent, editable worlds from one image

A paper on Hugging Face presents WM^MLLM, which its authors describe as the first interactive world model built on multimodal large language models (MLLMs). It generates a persistent, editable world from a single image.

The starting point is a weakness the authors see in video world models. These keep the world in the context of a generator, so when a user revisits a place it is synthesized again, and earlier edits can drift or disappear.

WM^MLLM moves the world state out of every generator. The state is a panorama with a revision history, and the MLLMs write to it. They turn each request into operations on this state, select where a local edit is written, and check every candidate before it becomes the next state. In this design, image generators only propose views, and video only presents them.

Users can walk to new places and return, edit objects through text or a box, place subjects, and change lighting, weather or season. The authors say that by construction a revisit at the same revision reproduces the earlier view exactly, a local edit leaves every panorama pixel outside its region unchanged, and each request starts from the result of the previous one.

The reported results are as follows. Against six world model baselines on MBench-A, WM^MLLM has the highest mean on all four environment-recovery metrics; on the Epipolar metric it scores 91.5, while the baselines range from 0.1 to 74.0. It also completes the most camera round trips: 85.3%, against at most 50.9% for the baselines. On CompBench it has the best value in all 15 columns against 15 baselines.

Two further checks target the design itself. On 112 local edits judged blind, selecting the region to write after generation raises completion from 34.8% to 83.0%. On 258 WBench and MBench-A items, all 46 revisits recover their earlier views, and all 97 local edits that pass the check leave panorama pixels outside their region unchanged.

Key facts

  • WM^MLLM keeps the world state outside the generators, as a panorama with a revision history that MLLMs write to, so revisits and earlier edits are not regenerated and do not drift.
  • On MBench-A against six world model baselines, it has the highest mean on all four environment-recovery metrics (Epipolar 91.5 versus 0.1 to 74.0) and completes 85.3% of camera round trips versus at most 50.9%.
  • It has the best value in all 15 CompBench columns against 15 baselines.
  • On 112 blind-judged local edits, selecting the region to write after generation raises completion from 34.8% to 83.0%.
  • On 258 WBench and MBench-A items, all 46 revisits recover their earlier views and all 97 local edits that pass the check leave pixels outside their region unchanged.

Why it matters

Video world models hold the world in a generator's context, so a place that is visited again is synthesized again and earlier edits can drift or vanish. WM^MLLM answers this by making the world a stored panorama with a revision history, rather than something the generator has to remember. The authors say image generators only propose views and video only presents them, while MLLMs do the writing. They also claim it is the first interactive world model built on MLLMs, which is their own claim.

Who it affects

The paper speaks to researchers working on interactive world models and on persistent, editable scene generation from a single image. The interactions it describes are walking to new places and returning, editing objects through text or a box, placing subjects, and changing lighting, weather or season.

How to use it

No code, demo or release availability is stated, so there is nothing to try yet. What the paper describes is the interaction model: start from a single image, then walk, return, edit objects through text or a box, place subjects, or change lighting, weather or season. Each request starts from the result of the previous one.

How solid is it

The reported margins are large, but they come from the authors' own evaluation. On MBench-A, WM^MLLM has the highest mean on all four environment-recovery metrics against six baselines and completes 85.3% of camera round trips versus at most 50.9%. It has the best value in all 15 CompBench columns against 15 baselines. The region-selection ablation (34.8% to 83.0% completion) rests on 112 local edits judged blind. The first-of-its-kind claim is the authors' own; no independent verification is mentioned. The six MBench-A baselines and the 15 CompBench baselines are not named.

Risks and caveats

The exact-revisit and unchanged-pixel guarantees are design properties, stated as holding by construction. The empirical check covers 258 WBench and MBench-A items: all 46 revisits recover their earlier views, and all 97 local edits that pass the check leave pixels outside their region unchanged. Edits that fail the check are not counted in the 97. The abstract names no authors or institutions, does not say which MLLMs, image generators or video models are used, and gives no latency or compute cost figures. How the blind judging was done is not described.

“Image generators only propose views, and video only presents them.”

— WM^MLLM paper abstract