ARC-Bench finds frozen JEPA world models rank actions almost backwards
Reward free world models built on frozen JEPA style encoders plan by comparing latent embeddings: an action counts as good if the world model predicts it lands closer, in latent space, to the goal embedding than the alternatives do. That only works if latent closeness actually tracks true cost, meaning that sorting candidate actions by predicted latent distance should agree with sorting them by how good they really are. The authors introduce ARC-Bench, a no leak, fixed candidate protocol built to test this assumption directly, and run it against official released JEPA world model checkpoints across navigation tasks and manipulation style control tasks.
The assumption fails, in the authors' words, severely and structurally. On the official manipulation audits, the action the model scores as best is almost always suboptimal, not merely wrong on occasion. The same ranking inversion shows up in the maze navigation domains as well. To check whether the visual backbone was the culprit, the authors ran a controlled extension swapping the DINOv2 encoder for video pretrained V-JEPA 1 and V-JEPA 2 encoders at ViT-L and ViT-G scale; the ranking defect persisted across all of them. They also ran provenance checks, undertraining checks, matched budget backbone controls, and metric circularity controls, and none of these explains the failure away as a trivial artifact.
The paper's second contribution is explaining why this defect went unnoticed until now: closed loop replanning masks it. A planner that replans frequently keeps re-scoring actions at every step, so even a world model that ranks the first candidate set badly can stumble into a workable action on a later attempt, and the episode still ends in success. When the authors cut the replanning frequency, success collapses in both a navigation domain and a manipulation domain. Using a first-plan diagnostic in PointMaze, they further show that the episodes rescued specifically by frequent replanning are enriched for cases with severe first-plan ranking failures, meaning the successes that make these systems look competent are disproportionately the ones where the initial plan was bad and only survived because the planner kept correcting it. The authors conclude that closed loop success rates systematically overstate how rankable these frozen latent representations actually are, and they position ARC-Bench and this masking mechanism as tools for anyone building methods that adapt, amortize around, or replan on top of latent space planners without first auditing whether the released JEPA world model can rank actions at all.
Key facts
- ARC-Bench is a new no-leak, fixed-candidate protocol for testing whether frozen JEPA-style world models correctly rank candidate actions by latent distance.
- Applied to official released JEPA-WM checkpoints, the top-scored action is almost always suboptimal on manipulation audits, with the same inversion appearing in maze navigation domains.
- The defect persists when the DINOv2 backbone is swapped for video-pretrained V-JEPA 1 and V-JEPA 2 encoders at ViT-L and ViT-G scale.
- Provenance, undertraining, matched-budget backbone, and metric-circularity controls rule out trivial explanations for the failure.
- Reducing replanning frequency collapses success rates in navigation and manipulation domains, showing that frequent closed-loop replanning has been masking the broken ranking rather than fixing it.
Why it matters
A large family of reward-free robotics and navigation methods relies on the unstated premise that a frozen JEPA-style world model's latent space is action-rankable: that predicting an action's future embedding and comparing its distance to a goal embedding tells you which action is actually better. This paper is a direct audit of that premise on official released checkpoints, and it finds the premise false, not marginally but structurally, across both navigation and manipulation control.
Who it affects
The finding targets anyone building or evaluating planners on top of frozen JEPA-style world models, including the released JEPA-WM checkpoints tested here and the V-JEPA 1 and V-JEPA 2 encoder family used in the backbone extension. It is also relevant to researchers evaluating such systems by closed-loop success rate alone, since the paper shows that metric can hide the exact defect ARC-Bench is built to expose.
How to use it
The paper offers ARC-Bench as a no-leak, fixed-candidate audit protocol that other researchers can apply to their own frozen JEPA-style world models to check action-rankability directly, rather than trusting a high closed-loop success rate as a proxy for it. The paper does not mention a code or dataset release, a price, or a license for ARC-Bench.
How solid is it
The result rests on official released JEPA-WM checkpoints, not the authors' own retrained models, which reduces the chance the failure is an artifact of a poorly trained system. The authors also ran a matched-budget backbone extension across three different encoder families (DINOv2, V-JEPA 1, V-JEPA 2) at two model scales, plus dedicated controls for provenance, undertraining, and metric circularity, all of which point the same direction. The text gives no quantitative failure rate or specific benchmark scores, describing severity only qualitatively as candidates being 'almost always' suboptimal, and it names no authors, institutions, or publication venue.
Risks and caveats
Because closed-loop replanning systematically masks the ranking defect, any system evaluated primarily on closed-loop success rate may be hiding the same failure this paper documents in JEPA-WM checkpoints. The source text does not quantify how much worse a first-plan ranking is than random, nor does it state which specific manipulation tasks or benchmarks were audited beyond the PointMaze first-plan diagnostic, so the generality of the severity beyond the tested checkpoints is not established by the text alone.