Orbia benchmark tests whether video world models keep a consistent 3D scene
Video generators are increasingly used as interactive worlds for exploration and simulation. The authors of a new paper argue that realistic, visually appealing frames do not necessarily depict a consistent 3D world, and that existing benchmarks leave this underexplored: they either assess selected geometric properties or use vision-language models to judge memory and consistency.
Their answer is Orbia, which they present as the first benchmark dedicated to a systematic and comprehensive evaluation of 3D consistency in interactive world models. It works through controlled exploration and revisit: a model explores a scene and then returns to places it has already shown. The evaluation is organised into three dimensions.
The first is input preservation. Registered RGB-D references and annotated anchors support measurements of appearance, geometry and spatial stability. The second is generated-content persistence. Here the benchmark compares revisits to regions that were absent from the input image with their first generated observations, to test whether models preserve what they create. The third is 3D self-consistency, tested through cross-view depth and feature correspondence, regional scale agreement, and held-out Gaussian rendering, which together check whether generated views describe the same 3D scene. Minute-long exploration and revisits are used to test long-term consistency.
The construction and annotation pipeline combines public datasets, self-captured scenes and simulation. The result is 720 instances across indoor and outdoor environments and six exploration-and-revisit protocols. Evaluation spans 27 text-, action- and camera-conditioned models, run through native-interface adapters. The authors say Orbia gives a systematic basis for diagnosing 3D inconsistency and developing more persistent video world models.
Key facts
- Orbia is presented as the first benchmark dedicated to systematic, comprehensive evaluation of 3D consistency in interactive world models.
- It has 720 instances across indoor and outdoor environments and six exploration-and-revisit protocols, built from public datasets, self-captured scenes and simulation.
- Evaluation covers three dimensions: input preservation, generated-content persistence and 3D self-consistency, plus minute-long exploration and revisits for long-term consistency.
- 27 text-, action- and camera-conditioned models are evaluated through native-interface adapters.
- The authors argue that realistic, visually appealing frames do not necessarily depict a consistent 3D world.
Why it matters
Video generators are being used as interactive worlds for exploration and simulation, and that use depends on the world staying put. The authors point out that frames can look realistic without depicting a consistent 3D world. They say existing benchmarks either check selected geometric properties or rely on vision-language models to judge memory and consistency, which leaves systematic 3D evaluation underexplored. Orbia is offered as the first benchmark aimed squarely at that gap.
Who it affects
Mainly researchers building and comparing video world models, including text-, action- and camera-conditioned ones, since Orbia is meant to diagnose where 3D inconsistency arises and to guide more persistent models. Anyone who plans to use generated video as an explorable environment or simulator has a stake in the same question.
How to use it
The benchmark is organised so a model is run through exploration and revisit scenarios, then scored on three dimensions. Input preservation measures appearance, geometry and spatial stability against RGB-D references and annotated anchors. Generated-content persistence checks whether revisited regions match what the model first generated. 3D self-consistency uses cross-view depth and feature correspondence, regional scale agreement and held-out Gaussian rendering. Models are evaluated through native-interface adapters. No release date, code, dataset link, or license is stated.
How solid is it
This account rests on the paper's abstract, so the claims are the authors' own, including that Orbia is the first benchmark of its kind. The scale is concrete: 720 instances, six protocols and 27 models. The abstract reports no evaluation results: no model rankings, scores, or named models. It does not say which of the 27 models were tested or which performed best or worst, and no authors or institutions are named.
Risks and caveats
Without results, it is not yet clear what Orbia shows about any particular model, or how well its metrics separate real 3D consistency from visual plausibility. The benchmark's value depends on the construction pipeline, which mixes public datasets, self-captured scenes and simulation, and on whether its measures hold up under independent use. The first-of-its-kind claim is the authors' framing.
“realistic and visually appealing frames do not necessarily depict a consistent 3D world”
— Orbia paper abstract