V-JEPA-LM teaches video-language models to read motion and speed

The paper starts from a problem with video-language models built on world-model encoders such as V-JEPA. These encoders learn how scenes change (direction, speed, order, state) from video alone. To answer questions, they are joined to a language model trained on captions and question-answer pairs. The authors argue that this training wastes what the encoder knows: each caption or question names only a few attributes of a clip, so the language model learns to retrieve those attributes rather than to read the clip, and the dynamics the encoder carries go unused. Their evidence is a gap on the same items: a linear probe reads speed from frozen V-JEPA features at 87%, while the video-language model answers at chance.
The authors then show where the dynamics information sits in a frozen encoder. For predictive encoders such as V-JEPA it lives in a subspace shared across videos. For frame-wise, text-aligned encoders such as SigLIP it is not a shared component.
Their fix is counterfactual supervision. Pixel transformations turn public videos into answer-balanced variants. The answers are set by the transformation, and the variants differ only in the change, so the model cannot get by on a clip's static attributes and has to read the dynamics.
The resulting models are called V-JEPA-LM and range from 1B to 8B parameters, built on Qwen3 and Llama. They are trained in a single stage on 4.53M public samples with 32 visual tokens per frame. The paper contrasts this with open video LLMs, which it describes as using several stages, 9M to 172M samples and 64 to 256 tokens per frame on average. The authors report that V-JEPA-LM leads every size class in average score across eleven video benchmarks. Their 4B model scores 72.1, against 61.1 for the best open 4B and 64.8 for the best open 8B model.
A controlled study crossing encoder and supervision supports the authors' summary that the encoder decides what can be read and the supervision decides what is read. At a matched budget, counterfactual supervision adds 4.5 points on average, and 38 on the speed questions of natural benchmarks. Code and models are released at github.com/kkt94/Llama-V-JEPA-LM.
Key facts
- On the same items, a linear probe reads speed from frozen V-JEPA features at 87% while the video-language model answers at chance, which the authors take as proof that the language model leaves the encoder's dynamics unused.
- Counterfactual supervision uses pixel transformations of public videos to make answer-balanced variants whose answers are set by the transformation and differ only in the change.
- V-JEPA-LM (1B to 8B, Qwen3 and Llama) trains in a single stage on 4.53M public samples with 32 visual tokens per frame.
- The authors report the lead in every size class in average score across eleven video benchmarks; the 4B model scores 72.1 against 61.1 for the best open 4B and 64.8 for the best open 8B.
- At a matched budget, counterfactual supervision adds 4.5 points on average and 38 on the speed questions of natural benchmarks. Code and models are released.
Why it matters
The paper targets a specific weakness: video-language models can sit on top of an encoder that tracks speed, direction and order, yet answer questions about them at chance. The authors trace this to captions and questions that mention only a few attributes of a clip, which teaches the language model to look attributes up rather than read the video. They claim a fix that needs a single training stage and 4.53M public samples, against the several stages and 9M to 172M samples they describe for open video LLMs. If the results hold, the encoder choice and the training signal both matter for motion understanding, and the data requirement is lower than the paper says is typical.
Who it affects
Researchers and engineers building open video-language models, especially those pairing a language model with a video encoder. The paper contrasts predictive encoders such as V-JEPA with frame-wise, text-aligned encoders such as SigLIP, so teams choosing between the two are the most direct audience. The models come in sizes from 1B to 8B on Qwen3 and Llama.
How to use it
Code and models are released at https://github.com/kkt94/Llama-V-JEPA-LM. The V-JEPA-LM models span 1B to 8B parameters. The recipe the paper describes is to take public videos, apply pixel transformations that yield answer-balanced variants, and train the language model in a single stage on the resulting samples, with 32 visual tokens per frame. The source gives no licence or price information.
How solid is it
This is an abstract-level source from a Hugging Face papers page, and the numbers are the authors' own. The headline comparison is internal to the paper: 72.1 for the 4B model against 61.1 for the best open 4B and 64.8 for the best open 8B model. The 64.8 figure is compared with the 4B V-JEPA-LM score, and the abstract gives no V-JEPA-LM 8B score. The eleven benchmarks and the competitor models are not named in the abstract. The abstract does not say whether the results are independently reproduced or peer reviewed. The controlled encoder and supervision study, with a matched budget, is the stronger part of the evidence for the causal claim.
Risks and caveats
The claim of leading every size class rests on one average over eleven benchmarks, so it says little about any single task. The 38 gain on speed questions has no unit stated in the source. The "on average" qualifier on the competitors' token counts is the source's own wording, so the efficiency comparison with open video LLMs is approximate. The abstract also gives no compute cost, training time or hardware, so the cost comparison covers data size and token count only.