AFFECT training objective makes robot world models track action effects

AFFECT training objective makes robot world models track action effects

World models predict how an environment will change under a given action, so that the consequences of actions can be estimated without running them in the real world. The authors of this preprint argue that, even as these models produce increasingly realistic videos, they often fail to capture how small changes in an action affect future observations. That matters for robotic policy evaluation and model-based planning, where nearly identical actions can lead to qualitatively different outcomes.

To address this, the authors propose AFFECT, an auxiliary training objective for learning the distribution of visual changes induced by action perturbations. The method works in two steps. First, temporally correlated noise is injected into the actions of a policy trained on task-agnostic human teleoperation. Then the world model is trained so that its generated videos respond to this noise as the recorded videos do. This does not require repeated executions from the same initial state.

The authors report that across OGBench simulation and real-world DROID and ALOHA-2 setups, AFFECT makes world-model rollouts more faithful to real outcomes. On DROID, the Pearson correlation between policy success rates in the world model and on the real robot rises from 0.47 to 0.79. Alongside that, FVD (lower is better) falls by up to 23%.

The gains carry over to planning. Selecting actions with AFFECT-trained world models improves task success by up to 8.3%. The text does not say whether that figure is in percentage points or relative terms.

Key facts

  • AFFECT is an auxiliary training objective that teaches a world model the distribution of visual changes caused by action perturbations.
  • The method injects temporally correlated noise into the actions of a policy trained on task-agnostic human teleoperation, then trains the model to respond to that noise as the recorded videos do.
  • On DROID, the Pearson correlation between policy success rates in the world model and on the real robot rises from 0.47 to 0.79, with up to 23% lower FVD.
  • Selecting actions with AFFECT-trained world models improves task success by up to 8.3% in planning.
  • Results are reported on OGBench simulation and on real-world DROID and ALOHA-2 setups.

Why it matters

Realistic-looking video is not the same as a model that responds correctly to the action it was given. The authors say current world models often miss how small action changes affect what comes next, which limits their use for robotic policy evaluation and model-based planning. AFFECT targets that gap directly, by training the model on how visual outcomes shift when actions are perturbed.

Who it affects

Robotics researchers who use world models to evaluate policies without running them on a physical robot, and teams doing model-based planning. The reported setups are OGBench, DROID and ALOHA-2.

How to use it

AFFECT is an auxiliary objective, so it is added to the training of a world model rather than replacing it. The recipe in the abstract: run a policy trained on task-agnostic human teleoperation, inject temporally correlated noise into its actions, and train the world model so its generated videos respond to that noise as the recorded videos do. No repeated executions from the same initial state are needed. The text mentions no code, model or dataset release.

How solid is it

This is an abstract-level account of a preprint listed on Hugging Face Papers. The headline numbers are specific: correlation from 0.47 to 0.79 on DROID, up to 23% lower FVD, and up to 8.3% better task success. The text does not name baseline models, and does not say how many tasks, policies or trials the success rates and correlations rest on.

Risks and caveats

Two of the three figures are stated only as "up to", so typical gains may be smaller. The 8.3% task-success figure comes without a baseline value, and the text does not say whether it is percentage points or a relative gain. The 23% FVD reduction has no per-setup breakdown across OGBench, DROID and ALOHA-2. The correlation result is reported for DROID only.