RoboFollow benchmark shows embodied agents barely follow instructions
Researchers introduce RoboFollow, a diagnostic benchmark built to check whether embodied agents (robot policies that take visual and language input) genuinely follow instructions, or merely appear to because of how their test scenes are designed. The authors argue that today's high success rates are misleading: they trace the illusion to what they call low scene entropy, a structural flaw where a visual scene admits only one valid task. When that happens, language becomes redundant, since the robot can guess the right action from the scene alone, and a policy can score well while barely using the instruction at all.
RoboFollow is built on three design principles. First, high scene entropy: each training scene supports multiple kinematically distinct task branches, so vision alone is not enough and the policy must actually rely on language to pick the right one. Second, a hierarchical diagnostic protocol with four levels, L0 through L3, which progressively perturb the visual layout and the semantics of instructions, testing whether equivalent instructions produce consistent robot behavior and different instructions produce distinguishably different behavior, across spatial relations, object attributes, trajectory constraints and logic. Third, confound-controlled diagnosis: the researchers simplify the objects being manipulated, restrict actions to the policy's trained repertoire, and report separate Intent and Execution scores so that failures of comprehension can be told apart from failures of motor control.
The team evaluated nine vision-language-action (VLA) and world-action-model (WAM) policies on RoboFollow. The pattern that emerged: strong performance at the easiest level, L0, where policies did attain it, did not reliably carry over to the harder L1 through L3 levels under the authors' fine-tuning setup. They then tested several representative fixes meant to strengthen language grounding, including using stronger vision-language model backbones, co-training with question-answering data, a method called LangForce, and Classifier-Free Guidance. None of these mitigations closed the gap between L0 and the higher diagnostic levels. The authors conclude that RoboFollow exposes genuine instruction following as a critical bottleneck in embodied agents that has been largely overlooked by standard success-rate benchmarks. Code and the dataset used for the evaluation are released on GitHub and Hugging Face.
Key facts
- RoboFollow is a diagnostic benchmark testing whether embodied robot policies genuinely follow language instructions rather than exploiting scenes that admit only one valid task (low scene entropy)
- The benchmark uses a four-level protocol (L0-L3) that progressively perturbs visual layout and instruction semantics, plus separate Intent and Execution scores to isolate comprehension from motor skill
- Nine VLA and WAM policies were evaluated; strong L0 performance did not reliably transfer to the harder L1-L3 levels
- Four mitigations were tested, stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance, and none closed the gap
- Code and dataset are publicly released on GitHub and Hugging Face
Why it matters
Robot policies are typically judged by end-to-end success rates, but the paper argues those numbers can be inflated by a hidden shortcut: if a test scene only supports one plausible task, a policy can succeed without ever really parsing the instruction. RoboFollow is built specifically to strip that shortcut out and measure whether language is doing any real work.
Who it affects
Researchers and engineers building or evaluating embodied agents, vision-language-action (VLA) and world-action-model (WAM) policies, who rely on benchmark success rates to judge progress in instruction following.
How to use it
The RoboFollow code and dataset are publicly available, at the project's GitHub repository and as a dataset on Hugging Face, for researchers to evaluate their own policies against the four-level diagnostic protocol and the Intent/Execution scoring split.
How solid is it
The evaluation covers nine VLA and WAM policies against a purpose-built four-level protocol with confound controls (simplified objects, restricted action repertoire, separate comprehension and execution scoring), and the authors tested four distinct mitigation strategies rather than just one. The source text does not give specific numeric accuracy figures for the L0-to-L1-L3 gap, nor does it name the authors or their institution.
Risks and caveats
The finding that mitigations like stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance all failed to close the gap is specific to the authors' fine-tuning setup, and the paper does not report exact accuracy numbers for how large the L0-to-L1-L3 drop is, only that it is not reliably closed.
“When a visual scene admits only one valid task, language becomes redundant and a policy can score highly while barely using it.”
— the paper's authors