ReWorld separates control from memory in interactive world models

An interactive world model has to do three things at once: follow the user's actions instantly, remember places it has already shown, and stream video in real time. The paper argues this creates a structural tension, since control works best with a short attention horizon while memory needs an effectively unbounded one. ReWorld's answer is to stop trying to satisfy both with one mechanism and instead separate them during training, then bound them together at inference.

The architecture uses mixed per-head attention windows: most attention heads are confined to the recent past, handling control, while a small set of global heads attends over the entire history to handle memory. Random head routing during training prevents either capability from locking onto specific heads, and random chunk dropping exposes the model to sparse histories so it stays in distribution when parts of a session are skipped. At inference, the entire past is held under a fixed budget through a bounded key-value cache backed by a pose-indexed landmark bank: the model retrieves whichever landmarks are nearest the camera's current pose rather than scanning the full history.

Training data comes from a metric-scale-aligned data engine that combines eight sources, including Unreal-rendered fly-throughs, game roaming footage, and real-world video, all normalized onto one physical action scale so that the same key press moves the camera the same distance regardless of source. Palindrome trajectories, where a path is retraced back to its start, supply the revisit evidence the memory mechanism needs to learn from.

For deployment, distribution-matching distillation confined to a LoRA adapter compresses sampling to four steps. A single backbone then serves two modes from that: a high-fidelity multi-step mode and a real-time interactive one, both streaming 704x1280 video across photorealistic, game-style, and stylized worlds.

The paper evaluates ReWorld against six recent interactive world models on a three-axis protocol covering action following, long-horizon recall, and video quality. It reports the best control fidelity, with an 11.95-degree rotation error, the best camera-motion consistency, and the best generation quality of the group. On minute-long out-and-back rollouts, 64 seconds and 384 latents, where the camera travels away from and back to its starting point, ReWorld's fixed 12-chunk cache still regenerates the starting view correctly. The paper contrasts this with sliding-window approaches, which have long since evicted that evidence at this rollout length, and full key-value attention, which runs out of memory before reaching it.

Key facts

ReWorld trains control (short attention horizon) and memory (unbounded horizon) as separate mechanisms via mixed per-head attention windows and random head routing, then bounds both at inference with a fixed key-value cache backed by a pose-indexed landmark bank.
Its data engine aligns eight sources, Unreal-rendered fly-throughs, game roaming, and real-world footage, on one physical action scale, and uses palindrome trajectories to teach the model to recognize revisited locations.
Distribution-matching distillation confined to a LoRA adapter compresses sampling to four steps, letting one backbone stream 704x1280 video in both a high-fidelity multi-step mode and a real-time interactive mode.
Against six recent interactive world models, ReWorld reports the best control fidelity (11.95-degree rotation error), the best camera-motion consistency, and the best generation quality.
On 64-second, 384-latent out-and-back rollouts, ReWorld's fixed 12-chunk cache still regenerates the starting view, at a length where sliding-window methods have evicted the evidence and full key-value attention runs out of memory.

Why it matters

Interactive world models sit at the core of anything that lets a user steer a generated video in real time, from game-style environments to embodied-AI training grounds, and they have faced a structural tradeoff: tight, responsive control tends to use a short attention window, while remembering a place well enough to return to it later needs a long one. Most systems pick one and suffer on the other axis, either drifting out of control or forgetting where they have been. ReWorld's contribution is architectural: it trains the two capabilities apart, using different attention head groups for local control and global memory, then reconciles them at inference through a fixed-budget cache indexed by camera pose rather than by simple recency. That separation is what lets it hold a bounded memory footprint without discarding the specific evidence needed for a long-horizon recall.

Who it affects

The paper targets people building interactive video generators and world models: researchers working on generative game engines, real-time video simulators, and the kind of interactive environments used to train or evaluate embodied and robotic agents. Anyone whose system needs to keep a session running for more than a few seconds while still letting a user revisit earlier ground faces exactly the control-versus-memory tension the paper describes, and ReWorld is offered as one architectural answer to it.

How to use it

ReWorld is presented as a research architecture rather than a shipped product, so the paper does not state pricing, licensing, or availability, and none should be assumed. Practically, its distillation step compresses sampling to four steps via a LoRA adapter, and the resulting backbone runs in two modes: a slower, high-fidelity multi-step mode and a real-time interactive mode, both streaming 704x1280 video across photorealistic, game-style, and stylized settings. The bounded, pose-indexed cache is the part most directly reusable elsewhere: it lets a system hold a fixed memory budget while still retrieving the landmarks relevant to wherever the camera currently is, rather than a plain sliding window over recent frames.

How solid is it

The paper tests ReWorld against six recent interactive world models on a three-axis protocol covering action following, long-horizon recall, and video quality, and reports it ahead on all three: the best control fidelity (11.95-degree rotation error), the best camera-motion consistency, and the best generation quality. The long-horizon claim is backed by a concrete stress test, a 64-second out-and-back rollout of 384 latents, where the fixed 12-chunk cache still regenerates the correct starting view at a length the paper says defeats both sliding-window caching and full key-value attention. These are the paper's own reported results, and the source text does not name the six baseline models it is compared against, nor does it give training compute, model size, or dataset size beyond the count of eight source types, so independent verification of the comparison is not possible from what is available here.

Risks and caveats

The source text gives no author names, institution, or affiliation, and no publication or release date, so the provenance of the work cannot be confirmed here. It also omits any comparison of ReWorld's own inference cost or latency against the six baseline models, reporting only fidelity, consistency, and quality rankings, which leaves open whether the architecture's gains come with a speed or compute tradeoff. All performance numbers are self-reported by the paper rather than independently reproduced, and 704x1280 is a moderate streaming resolution, so the fidelity claims should be read as relative to the six unnamed comparison models rather than as an absolute quality bar.

“An interactive world model must follow the user's actions, remember the places it has shown, and stream in real time.”

— ReWorld paper