Glonce

LOWN lets diffusion VLMs skip the image at most decoding steps

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact45Novelty60Relevance100Surprise30Players10Credibility75
LOWN lets diffusion VLMs skip the image at most decoding steps

Diffusion vision-language models (DVLMs) generate text by unmasking tokens over many denoising steps, and at every step they process their entire input. The authors of a new paper note that visual tokens make up 89-97% of that input on the benchmarks they study, so most of the decoding computation goes into processing the same image again and again.

Existing visual token pruning methods ease the load by removing some of the visual tokens. The authors say the drawback is twofold: the removed information is lost, and the remaining image is still processed at every step.

To see how often the image is really needed, the authors compared LLaDA-V's predictions with and without the image at every decoding step. Removing the image changed the token being unmasked at only 22.8% of steps on average, and at as few as 7.9% of steps.

From that observation they built LOWN, a decoding method that decides at every step whether to process the image and how many tokens to unmask. LOWN first predicts the masked tokens without the image. It processes the image only when that image-free prediction disagrees with the last image-conditioned prediction. When the two agree, the earlier prediction is certified for reuse, and the number of certified tokens sets how many tokens are unmasked in that step.

The authors also observe that the need for the image differs across benchmarks, which is why they say an adaptive decision is needed rather than a fixed schedule.

Without training or a tuned threshold, LOWN retains 99.6% of LLaDA-V's score on ten benchmarks at 35.3% of the decoding FLOPs on average. The authors add that it generalizes across LLaDA-based DVLMs, composes orthogonally with visual token pruning, and runs under key-value caching for further efficiency gains. Code and a project page are available at https://hyeon9698.github.io/LOWN/.

Key facts

  • Visual tokens make up 89-97% of a DVLM's input on the benchmarks studied, yet the model reprocesses them at every denoising step.
  • On LLaDA-V, removing the image changes the token being unmasked at only 22.8% of steps on average, and at as few as 7.9%.
  • LOWN predicts masked tokens without the image and processes the image only when that prediction disagrees with the last image-conditioned one.
  • Across ten benchmarks, LOWN retains 99.6% of LLaDA-V's score at 35.3% of the decoding FLOPs on average, with no training or tuned threshold.
  • The authors say it generalizes across LLaDA-based DVLMs and combines with visual token pruning and key-value caching.

Why it matters

Diffusion language models revisit their whole input at every denoising step, and for vision-language versions the image dominates that input (89-97% of tokens on the studied benchmarks). LOWN attacks the redundancy directly: instead of shrinking the image, it skips processing it when an image-free prediction already agrees with the last image-conditioned one. The reported result is 35.3% of the decoding FLOPs on average for 99.6% of LLaDA-V's score.

Who it affects

Mainly researchers and engineers working on diffusion vision-language models, especially LLaDA-based ones, where the authors report that the method generalizes. It is also relevant to anyone working on visual token pruning or key-value caching for these models, since the authors say LOWN composes with both.

How to use it

LOWN is a decoding method, so it needs no training and no tuned threshold according to the authors. Code and a project page are available at https://hyeon9698.github.io/LOWN/. The authors say it can be combined with visual token pruning and runs under key-value caching for further efficiency gains.

How solid is it

The headline figures are the authors' own: 99.6% of LLaDA-V's score and 35.3% of decoding FLOPs on average, over ten benchmarks. Both are relative to LLaDA-V's own numbers. The text gives them for LLaDA-V; the per-benchmark results and the names of the ten benchmarks are not given in the text. The efficiency claim is stated in FLOPs, and no wall-clock speedup or latency figure is given. The size of the extra gain from pruning or key-value caching is not quantified.

Risks and caveats

The savings are measured in decoding FLOPs, not in latency, so real-world speedups are not established by the text. The authors note the need for the image varies by benchmark, so the share of steps that can skip it will not be the same everywhere. The 22.8% and 7.9% step-level figures are for LLaDA-V, and the abstract does not say what they are for other models. The generalization and composability claims are stated without numbers.