Video-Point Model forecasts multi-view video and 3D point trajectories together
The paper starts from a simple requirement: reliable multi-view forecasting needs videos from different cameras to describe the same evolving 3D scene. To meet it, the authors introduce Video-Point Model (VPM), a world model that takes multi-view video histories and a language instruction as input. From these it jointly forecasts two things: future videos from arbitrary viewpoints, and the 3D trajectories of persistent scene points.
VPM is built by extending a pretrained world model in two ways. Camera-conditioned multi-view processing integrates complementary observations across cameras, and a trajectory head predicts how physical points move through time. Each trajectory tracks a single point across all views. Supervising these trajectories alongside video generation therefore anchors the appearance changes seen in every camera to one shared description of 3D motion.
To support training and evaluation, the authors build two resources. VPM-Data is a simulation corpus that pairs synchronized multi-view videos with persistent point trajectories. VP-WorldBench is a forecasting benchmark spanning simulated and real environments.
The authors then extend VPM to robot control by adding robot actions as a token modality in the generator. Action tokens share the point-supervised backbone and are denoised jointly with future videos, which grounds the policy in the same 3D-aware representation.
On the results, the authors report that on VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart. In simulated and real-robot experiments, it also raises the manipulation success of the action-enabled model. The text of the paper summary gives no figures for these gains.
Key facts
- Video-Point Model (VPM) conditions on multi-view video histories plus a language instruction and jointly forecasts future videos from arbitrary viewpoints and 3D trajectories of persistent scene points.
- It extends a pretrained world model with camera-conditioned multi-view processing and a trajectory head that predicts how physical points move over time.
- The authors build VPM-Data, a simulation corpus of synchronized multi-view videos with point trajectories, and VP-WorldBench, a forecasting benchmark covering simulated and real environments.
- Adding robot actions as a token modality, denoised jointly with future videos, turns VPM into a robot policy that shares the point-supervised backbone.
- Against a video-only counterpart, point supervision improves video fidelity, 3D motion accuracy and manipulation success in simulated and real-robot tests, according to the authors.
Why it matters
World models that generate video from several cameras can drift apart, with each view telling a slightly different story about the scene. VPM tackles this by forcing the model to also predict where individual 3D points go, since one trajectory tracks one point across every view. That ties the cameras to a shared account of 3D motion. The same backbone is then reused for robot control, linking video forecasting to action prediction.
Who it affects
The work is aimed at researchers building world models, 4D generation systems and video-conditioned robot policies. The two new resources, VPM-Data and VP-WorldBench, are of direct interest to anyone evaluating multi-view forecasting in simulated and real settings.
How to use it
The summary mentions no code, weights or dataset release, so there is nothing to run yet. The practical takeaway is the recipe: add a point-trajectory head and multi-view camera conditioning to a pretrained world model, and optionally add robot actions as extra tokens denoised alongside future video.
How solid is it
This is the authors' own description of their method and results, from a paper summary on Hugging Face. The reported comparison is against a video-only counterpart, and the evidence covers VP-WorldBench, existing 4D generation benchmarks, and simulated and real-robot experiments. No numerical results are given in the text, so the size of the improvements cannot be judged from it.
Risks and caveats
The only baseline mentioned is a video-only version of the same model; no comparison with other published models is stated. The training corpus, VPM-Data, is simulated. The pretrained base world model is not named, and no model size, compute, benchmark names beyond VP-WorldBench or robot platforms are given. The claims should be treated as unverified until the full paper's numbers are checked.