PlayWorld benchmark finds video world models unreliable over long horizons
Video world models generate future video frames conditioned on the current scene and a user's actions, and recent systems have gotten good at producing long, consistent, controllable video sequences. Comparing these models fairly has been hard, though: a human tester typically judges a world model by giving it a long-horizon goal, such as turning a full 360 degrees to check whether the environment stays consistent, or walking into water to see whether the ripples look realistic. The problem is that different models may need entirely different numbers and types of actions to reach the same goal, so evaluating them all on one fixed, scripted action sequence does not produce a fair comparison. To get around this, the researchers introduce PlayWorld, a benchmark that uses multi-modal 'Agent Players' rather than a fixed script: these agents interact with each world model toward a stated objective, the way a human player would, and the benchmark judges whether the objective was actually achieved. PlayWorld provides 171 such scenarios, each paired with a specified long-horizon objective. Model performance is scored along four core dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution, alongside basic video quality and controllability metrics. Running experiments across nine state-of-the-art world models, the researchers found that current models remain unreliable on long-horizon interactive objectives, particularly at maintaining spatial consistency and at correctly evolving persistent state over time. Code and data for the benchmark are released on GitHub at kxding/PlayWorld.
Key facts
- PlayWorld is a benchmark of 171 scenarios, each with a specified long-horizon objective, for testing video world models.
- Instead of a fixed, scripted action sequence, the benchmark uses multi-modal 'Agent Players' that interact with each model toward the objective, since the number and type of actions needed varies by model.
- Models are scored on four core dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution, plus basic video quality and controllability metrics.
- Across nine state-of-the-art world models tested, current models remain unreliable on long-horizon interactive objectives, particularly in maintaining spatial consistency and persistent state evolution.
- Code and data for PlayWorld are released on GitHub at kxding/PlayWorld.
Why it matters
Fairly comparing video world models has been an open problem: the same long-horizon goal, such as checking whether an environment stays consistent after a full turn or whether water ripples look right, can take a completely different sequence of actions in different models. A fixed, scripted evaluation cannot capture that, so PlayWorld replaces the script with agent players pursuing a stated objective, giving each model room to reach the goal its own way while still being judged on whether it got there.
Who it affects
The benchmark targets researchers and engineers building or evaluating video world models and interactive video generation systems, since it offers a standardized way to compare models that behave very differently on the same task rather than a tool aimed at end users.
How to use it
Code and data for PlayWorld are available on GitHub at kxding/PlayWorld; the source text gives no pricing, licensing, or access-restriction details.
How solid is it
The benchmark covers 171 scenarios and was run across nine state-of-the-art world models on four core dimensions plus basic video-quality and controllability metrics, which is a reasonably broad test set. The abstract does not name individual authors, institutions, the nine models tested, or numeric scores per model or dimension, so the specific size of the gap between models cannot be assessed from the source alone.
Risks and caveats
The four evaluation dimensions and the objectives behind the 171 scenarios are not defined in detail here, and using AI 'Agent Players' rather than human testers to judge whether an objective was met introduces its own evaluation method that is not independently verified in the source. No per-model results are disclosed, only the aggregate finding that current models are unreliable on long-horizon tasks.
“current models remain unreliable on long-horizon interactive objectives, particularly in maintaining spatial consistency and persistent state evolution”
— PlayWorld paper (arXiv 2608.13552)