ZeroVLN navigates without fine-tuning its VLM brain on navigation data

ZeroVLN navigates without fine-tuning its VLM brain on navigation data

A paper listed on Hugging Face Papers introduces ZeroVLN, a framework for vision-language navigation (VLN). The authors start from how the field usually works. Representative VLN methods pair a vision-language model (VLM) "brain" with a learnable action head, then fine-tune that brain on navigation-domain data so it predicts geometric goals. The authors say this adaptation can substantially alter the base VLM's weights and degrade its general visual capabilities.

ZeroVLN is their alternative: a slow-fast navigation framework with zero navigation-domain supervised fine-tuning (SFT) of the VLM brain. Instead of teaching the brain to predict geometric goals, the authors give it a semantic tool-calling interface. Through that interface the brain passes its movement decisions to an action expert in natural language.

Training runs in two steps. First, the action expert is adapted to carry out these movement requests. Then the brain's movement decisions and the action expert's continuous execution are improved together through closed-loop reinforcement learning (RL).

The authors report that, trained on the same navigation episodes, ZeroVLN substantially outperforms representative SFT-based hierarchical VLN methods on standard VLN-CE (continuous environments) benchmarks. They also say it induces only minor changes to the brain's weights and largely preserves the brain's general capabilities. To their knowledge, ZeroVLN is the first slow-fast VLN framework with zero navigation-domain SFT of the VLM brain, and the first to jointly optimize its slow and fast policies through closed-loop agentic RL in VLN-CE.

The text gives no benchmark scores, so the size of the improvement is not quantified.

Key facts

  • ZeroVLN is a slow-fast VLN framework in which the VLM brain gets zero navigation-domain supervised fine-tuning.
  • The brain conveys movement decisions to an action expert in natural language through a semantic tool-calling interface, rather than predicting geometric goals.
  • Training adapts the action expert first, then jointly improves the brain's decisions and the expert's continuous execution with closed-loop RL.
  • The authors say it substantially outperforms representative SFT-based hierarchical VLN methods on standard VLN-CE benchmarks when trained on the same navigation episodes.
  • The authors say the brain's weights change only slightly and its general capabilities are largely preserved.

Why it matters

The usual recipe for VLN fine-tunes the VLM brain on navigation data, and the authors argue this can substantially alter its weights and degrade general visual capabilities. ZeroVLN tries to get navigation without that cost, by leaving the brain unfine-tuned for navigation and moving the learning into the action expert and a closed-loop RL stage. The authors also claim two firsts: the first slow-fast VLN framework with zero navigation-domain SFT of the VLM brain, and the first to jointly optimize slow and fast policies through closed-loop agentic RL in VLN-CE.

Who it affects

The work is aimed at researchers building vision-language navigation systems, especially in continuous environments (VLN-CE), and at anyone who wants a VLM to keep its general visual abilities while it is used for embodied tasks. The text names no authors, institutions or deployments.

How to use it

This is a research result, not a product. The framework's recipe is the part the text describes: give the VLM brain a natural-language tool-calling interface to an action expert, adapt the expert to the movement requests, then run joint closed-loop RL on both. No code, weights or release plans are mentioned.

How solid is it

The claims come from the authors' own abstract on Hugging Face Papers, so they are self-reported. No benchmark scores, success rates or percentage improvements are given; "substantially outperforms" is not quantified. The base VLM, the model sizes and the specific baseline methods are not named, and "minor changes to the brain's weights" is not quantified either. The "first" claims are qualified by the authors with "to our knowledge".

Risks and caveats

Without numbers, the size of the gain over SFT-based hierarchical methods cannot be judged. No real-robot or real-world deployment is mentioned, and the evidence described is limited to standard VLN-CE benchmarks. No amount of training data, compute or training time is given, so the cost of the RL stage is unknown. The comparison is described as using the same navigation episodes, and no measured general-capability scores are given to back the preservation claim.

“To our knowledge, ZeroVLN is the first slow-fast VLN framework with zero navigation-domain SFT of the VLM brain”

— ZeroVLN authors, paper abstract