Glonce

WorldBridge splits video and action learning to help robots generalize across tasks

researchThe Glonce editorsHugging Face🔥55Glonce rating55Impact40Novelty55Relevance95Surprise25Players10Credibility70
WorldBridge splits video and action learning to help robots generalize across tasks

The paper studies how robots built on World Action Models (WAMs) can handle tasks they were not trained on. It splits the problem into two capabilities. Video generalization is predicting how a new task will visually unfold. Action generalization is turning those predictions into executable behavior through a video-to-action bridge. To see what each contributes, the authors vary the video and action supervision given for tasks outside an initial training set.

The central experiment is on RoboTwin 2.0. Training with only in-distribution video-action supervision gives a 25.07% success rate on out-of-distribution tasks. Adding videos of those tasks, without their action labels, raises success to 55.90%. Adding just one action-labeled demonstration per task lifts it further to 70.70%. That is a gain of 30.83 percentage points from unlabeled video and another 14.80 points from the single demonstrations. The authors suggest that broader video coverage enables task transfer by leveraging existing inverse-dynamics capabilities, and that even sparse action supervision strengthens those capabilities.

Guided by this finding, they introduce WorldBridge. It is a WAM with separate video and action experts, connected through structured attention and trained with joint flow matching. It includes visual memory at two temporal scales and action forcing. It also comes with a unified data processing pipeline that standardizes and filters heterogeneous robot trajectories and egocentric hand-derived supervision.

Across several manipulation benchmarks, the authors say WorldBridge achieves state-of-the-art performance among the compared methods. The figures they cite are 96.23% on the RoboTwin 2.0 full data setting, 80.9% on the RoboTwin 2.0 clean to random setting, 82.8% on RoboCasa GR1 TableTop, and 57.6% on DOMINO Level-1. They conclude that video prediction and action grounding play complementary roles in building robots that generalize across tasks.

Key facts

  • The paper separates video generalization (predicting how a new task unfolds) from action generalization (turning that prediction into behavior via a video-to-action bridge).
  • On RoboTwin 2.0, out-of-distribution success is 25.07% with in-distribution video-action data only, 55.90% after adding action-free videos of the new tasks, and 70.70% with one action-labeled demonstration per task added.
  • WorldBridge uses separate video and action experts linked by structured attention, trained with joint flow matching, plus visual memory at two temporal scales and action forcing.
  • Reported results: 96.23% on RoboTwin 2.0 full data, 80.9% on clean to random, 82.8% on RoboCasa GR1 TableTop, 57.6% on DOMINO Level-1.

Why it matters

Broad task generalization is described as a central challenge in robot learning. The paper's useful contribution is a decomposition: it shows, on RoboTwin 2.0, that most of the lift on unseen tasks came from videos without action labels (25.07% to 55.90%), and that a single labeled demonstration per task added another step (to 70.70%). If the pattern holds more widely, video is a cheaper kind of data to broaden coverage than action-labeled demonstrations, though the paper only claims that the findings suggest this.

Who it affects

Robot-learning researchers working on World Action Models and manipulation benchmarks are the direct audience. The data pipeline, which standardizes and filters heterogeneous robot trajectories and egocentric hand-derived supervision, is also relevant to anyone assembling mixed training data for such models.

How to use it

This is a research preprint, not a product. The practical takeaway the authors draw is about data: collect broader video coverage of new tasks, then add even sparse action supervision to strengthen the video-to-action link. The source does not mention code, a model release, or model size.

How solid is it

The experimental setup is clear and the three-step comparison on RoboTwin 2.0 is stated with exact success rates. The state-of-the-art claim is scoped by the authors themselves to the compared methods. The abstract does not name the metric behind the 96.23%, 80.9%, 82.8% and 57.6% figures or which baselines were compared, and it does not give the size of the improvement over the previous best method on any benchmark.

Risks and caveats

All results cited are on benchmarks (RoboTwin 2.0, RoboCasa GR1 TableTop, DOMINO); no real-robot experiments are mentioned. The abstract does not say how many out-of-distribution tasks were tested or how many in-distribution tasks were used for training, so the strength of the 25.07% to 70.70% progression is hard to judge from the summary alone. The explanation about inverse-dynamics capabilities is presented as a suggestion, not a proven mechanism.