PAWBench exposes a probability gap in video world models

Video generation models are increasingly described as world models, systems meant to simulate how a scene could unfold rather than just produce a convincing clip. The paper behind PAWBench starts from a simple observation: many physical processes can play out in more than one valid way, so a genuine world model should not just produce a single plausible sequence of events. It should reproduce the full distribution of possible outcomes, with the correct probability attached to each one, given the same starting observation and action. The authors call this requirement probabilistic alignment. The problem is that existing evaluation methods do not test for it: they judge whether one generated video looks individually plausible, but never check whether generating the same scenario many times over recovers the correct distribution of outcomes.

To answer the resulting question, how far current video generators are from probabilistically aligned world modeling, the researchers formalize probabilistic alignment as a distributional criterion and introduce two new pieces. PAWBench is a benchmark that treats video generators as stochastic samplers of world dynamics rather than single-answer predictors. PAWEval is the companion protocol: it runs a scenario through a generator many times and converts those repeated rollouts into an empirical distribution over the physical behaviors that actually occurred, which can then be compared against a reference distribution for that scenario.

Applied across 50 scenarios and eleven current video generation systems, which the text does not name individually, the result is that no model consistently matched the reference probabilities while also recovering the full range of valid behaviors. In other words, a system that reproduced some outcomes with roughly correct odds often missed others entirely, or clustered its generations around one dominant outcome instead of spreading across the true range of possibilities.

Having established that gap, the researchers also test three possible levers for closing it: language prompts, initial noise sampling, and model training, checking whether any single one can reshape a model's predictive distribution toward the correct one. The text does not state what that test found. The authors present PAWBench and PAWEval as a foundation for future work aimed at closing the gap they measured, rather than as a solution to it.

Key facts

PAWBench evaluates video generation models as world models by testing whether repeated generations reproduce the correct probability distribution of outcomes, not just one plausible video, a property the researchers term probabilistic alignment.
PAWEval is the companion protocol: it converts repeated rollouts of the same scenario into an empirical distribution over possible physical behaviors, which can then be checked against a reference distribution.
Across 50 scenarios and eleven current video generation systems, no model consistently matched the reference probabilities while also recovering the full range of valid behaviors.
The researchers separately test three possible levers: language prompts, initial noise sampling, and model training, to see whether any can reshape a model's predictive distribution, though the text does not report the outcome of that test.
The authors frame PAWBench and PAWEval as a foundation for future work moving video generators toward probabilistic alignment, not as a finished fix.

Why it matters

Video generation systems are increasingly described as world models, meaning they are expected to simulate how a scene could evolve rather than just produce a convincing clip. The paper's starting point is that many physical processes can unfold in more than one valid way, so a genuine world model needs to reproduce the correct probability of each possible outcome, not merely generate a single plausible one. The authors call this requirement probabilistic alignment. Existing evaluation methods do not test for it: they judge whether one generated video looks individually plausible, but never check whether repeating the same scenario recovers the correct distribution of outcomes. PAWBench and PAWEval exist to fill that specific evaluation gap, giving researchers an outcome-level way to check probabilistic alignment rather than just single-sample plausibility.

Who it affects

The direct audience is researchers who build or evaluate video generation models marketed as world models, since PAWBench and PAWEval give them a concrete way to check whether their systems reproduce correct outcome probabilities rather than one convincing clip. It also affects the makers of the eleven systems that were tested, though the text does not name which ones, so no individual system's result can be reported here. More broadly, it affects anyone who would rely on a video generator's output to represent the range of things that could plausibly happen next, rather than a single guess, since PAWBench is a direct check on whether that expectation actually holds.

How to use it

PAWBench works by running a video generator repeatedly on the same starting scenario, and PAWEval turns those repeated rollouts into an empirical distribution over the physical behaviors that occurred, which is then compared against a reference distribution for that scenario. That is the mechanism a research team would apply to check its own model. The text does not explain how the reference distributions themselves are established, and it gives no information on code or dataset availability, licensing, or cost, so it is not possible to say here whether PAWBench and PAWEval are openly released or how a team would go about accessing them.

How solid is it

The evidence comes from testing 50 scenarios across eleven current video generation systems. The central finding, that no model consistently matched the reference probabilities while also recovering the full range of valid behaviors, is presented as a clear result rather than a preliminary observation. But the text does not name the eleven systems, does not give individual scores or a ranking between them, and does not identify the paper's authors or institutions, so the claim cannot be checked against specific numbers or attributed to a specific research group here.

Risks and caveats

Because no individual system scores or names are given, it is impossible to tell from the text alone whether the gap is spread evenly across all eleven systems or driven by a few weak performers dragging the average down. The researchers say they tested whether language prompts, initial noise sampling, or model training can reshape a model's predictive distribution, but the text does not report what that test found, so whether any of those three levers actually helps remains an open question. There is also no venue, publication date, or code and dataset release mentioned, which limits how easily the benchmark can be checked or adopted right now.

“no model consistently matches the reference probabilities while recovering the range of valid behaviors”

— the paper's authors