Glonce

Vision-RL2 cuts MLLM visual tokens 4x without losing accuracy

researchThe Glonce editorsHugging Face🔥70Glonce rating70Impact65Novelty70Relevance95Surprise50Players25Credibility88
Vision-RL2 cuts MLLM visual tokens 4x without losing accuracy

Multimodal large language models (MLLMs) usually get better at fine-grained visual perception by processing images at higher resolution, but that inflates the number of visual tokens and drives up both vision-encoding and prefilling cost. A team behind a new paper argues this tradeoff is unnecessary because the two steps involved in fine-grained perception, finding the region of interest (RoI) and then reading its content, do not need the same amount of resolution. In a controlled diagnostic, the authors found that localization tolerates roughly 3 to 4 times stronger token compression than recognition does. That gap is the basis for their approach: localize coarsely first, then spend resolution only on the region that matters.

The paper reviews two existing ways to do that localization and explains why both fall short. Decoding region coordinates directly with the MLLM can be trained end to end from the final answers, but it requires a full model pass for every query and depends on the model's grounding ability. The alternative, a lightweight proposal network distilled from the model's own attention, is cheap to run but simply inherits whatever noise is already present in the attention maps it learns from. Neither can be supervised well because the chosen region reaches the final answer through a discrete choice, which blocks the answer's accuracy from directly training the network that picked the region.

The authors' fix is Vision-RL2, a region-level reinforcement learning method for training the proposal network. It treats each coherent region of the image as an action, and a frozen MLLM reader scores every candidate region by measuring how removing it changes the likelihood of the correct answer. Two complementary objectives work off that score: a subtractive one that suppresses regions that mislead the model, and an additive one that recovers regions carrying evidence the model would otherwise miss. Only the proposal network is updated, and the method needs no manual region annotations, no sampled responses and no reasoning traces. The sharper region proposals that result then feed a sparse encoding scheme that concentrates resolution on the evidence and drops background tokens.

Tested across six fine-grained perception benchmarks and four different MLLM backbones, Vision-RL2 improved accuracy over the base model at every visual-token budget tested, and matched or beat the base model's accuracy at its largest token budget while using about 4 times fewer visual tokens. The authors have released their code on GitHub.

Key facts

  • Vision-RL2 is a region-level reinforcement learning method that trains a lightweight proposal network to localize the region of interest before an MLLM reads it in detail.
  • A controlled diagnostic found that localization tolerates roughly 3 to 4 times stronger token compression than recognition does, which is why the method separates the two steps.
  • A frozen MLLM reader scores each candidate region by how removing it changes answer likelihood; subtractive and additive objectives then suppress distracting regions and recover missing evidence.
  • Training needs no manual region annotations, sampled responses or reasoning trajectories, only the proposal network is updated.
  • Across six benchmarks and four MLLM backbones, the method improves accuracy at every token budget and matches the base model's best-budget accuracy with about 4 times fewer visual tokens.

Why it matters

MLLMs typically buy fine-grained visual accuracy by processing images at higher resolution, which multiplies the number of visual tokens and the compute needed to encode and prefill them. The paper's core insight, that localizing a region of interest tolerates far more token compression than recognizing its content does, gives a principled reason to split the two steps instead of running both at full resolution. Training the localizer with region-level reinforcement learning, rather than distilling it from noisy attention maps or decoding coordinates end to end, removes a dependency on the model's own grounding ability and lets the proposal network be judged directly by how much each region actually helps the answer.

Who it affects

The method targets researchers and engineers building or fine-tuning multimodal models for tasks that depend on fine-grained visual perception, such as reading small text or details embedded in an image, where high resolution has been the default way to preserve accuracy. Anyone constrained by the compute cost of long visual-token sequences in MLLM inference is the intended audience.

How to use it

The authors have published their code at github.com/YuHengsss/VisionRL2. The proposal network is trained with reinforcement learning against a frozen MLLM reader, without manual region annotations, sampled responses or reasoning trajectories, which lowers the setup cost of adopting it compared with methods that need labeled regions or full end-to-end decoding.

How solid is it

The evaluation spans six fine-grained perception benchmarks and four different MLLM backbones, and Vision-RL2 improved accuracy over the base model at every token budget tested, surpassing the base model's largest-budget accuracy while using about 4 times fewer visual tokens. That consistency across multiple backbones and benchmarks is a reasonably broad test, and the code being public allows independent verification.

Risks and caveats

The source text does not give specific accuracy figures, such as percentage points, for the reported improvements, only the token-budget comparisons, so the size of the gain cannot be quoted more precisely. It also does not disclose details of the training data, the compute used, or the size of the frozen MLLM reader that scores proposals during training, all of which affect how easily the results generalize or reproduce.