RynnWorld-Latent trains one robot video world model on human, robot and simulated data
The paper starts from a practical problem. Robots from different makers use incompatible action spaces, and human videos come with no native control labels, so it is hard to learn controllable world dynamics from all that interaction footage together.
The proposed answer is RynnWorld-Latent, a cross-embodiment video world model. It uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. According to the authors, this representation allows joint pretraining on human interaction videos, robot demonstrations and simulation rollouts without requiring a common native action space.
The model is built on a pretrained video transformer. It combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. Pretraining covers 23 datasets with approximately 553.5M frames in total. After that, the model is adapted to target robots through post-training.
The framework supports two ways of generating training data. In digital teleoperation, the target robot's commands are used to generate action-video demonstrations. In latent-action synthesis, motion from a source video is transferred to an edited initial frame, and the resulting rollout is paired with retargeted pseudo robot actions.
On results, the authors report improved video prediction metrics on public benchmarks and on two real-robot platforms. They also describe qualitative action following: when different action sequences start from the same initial observation, the model's output follows the action given. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus. Both data-generation routes improve real-world policy success rates over training on real demonstrations alone, in the evaluated settings. The authors conclude that these results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.
Key facts
- RynnWorld-Latent is a cross-embodiment video world model that conditions on the 3D-grounded latent actions introduced by RynnLAM.
- It is pretrained on 23 datasets with approximately 553.5M frames of human videos, robot demonstrations and simulation rollouts, then post-trained for target robots.
- Two data-generation routes are supported: digital teleoperation, and latent-action synthesis that pairs transferred motion with retargeted pseudo robot actions.
- The authors report better video prediction on public benchmarks and two real-robot platforms, and better policy success on LIBERO, LIBERO-Plus and in real-world settings.
Why it matters
Robot learning has plenty of video but little shared labelling. Each robot has its own action space, and human videos have no control labels at all. The paper's idea is to put all of it behind one latent action interface, so a single world model can learn from human footage, robot demonstrations and simulation together. If it holds up, data that was hard to combine becomes usable in one pretraining run.
Who it affects
Robotics researchers working on world models and policy learning, especially those who need to move across different robot bodies. Teams that have a small set of real demonstrations for a target robot are the most direct audience, since the paper adds synthetic data to those demonstrations.
How to use it
The abstract describes a workflow rather than a product. Pretrain on mixed data through latent actions, adapt to the target robot with post-training, then optionally add synthetic data from digital teleoperation or latent-action synthesis. Mid-training on that synthetic data is what the authors link to higher policy success on LIBERO and LIBERO-Plus.
How solid is it
This is a preprint abstract, and the claims are the authors' own. It gives no metric values, success rates or baselines, so the size of the improvements cannot be judged from it. The authors name no model size, no underlying video transformer, and no real-robot platforms. The wording on the conclusion is measured: the results "support" universal latent actions as a scalable interface.
Risks and caveats
The real-world gains are stated only for the evaluated settings, so they should not be read as general. Action following is described qualitatively. The 23 datasets are not listed, and the mix of human, robot and simulated data is not given. The abstract mentions no code, weights or release plans.