Glonce

MAP method prunes visual tokens in LLaVA-NeXT-7B for 3.09x speedup

researchThe Glonce editorsarXiv🔥74Glonce rating74Impact68Novelty72Relevance98Surprise58Players38Credibility88

Multimodal large language models (MLLMs) process images as long sequences of visual tokens, and that token count is a major cost driver at inference time. One family of speedups prunes the visual tokens, keeping only the ones the model needs to answer a given question, and recent work has shown that text-to-vision attention from a middle language-model layer is a good signal for which tokens matter. The catch, as this paper's authors point out, is that the standard version of this idea picks one fixed middle layer for every input. Their own analysis found that the layer whose attention best tracks the question varies substantially from sample to sample, so a fixed layer is not the right choice for every case. There is a second problem too: to read attention from the right middle layer at all, the model first has to run the full set of visual tokens through several language-model layers, by which point most of the computation the pruning was supposed to save has already been spent.

The authors propose Middle-layer Attention Prediction (MAP) to address both issues. It uses a step they call Question Contrastive Teacher Selection: for a given sample, it contrasts attention computed under the original question against attention under a reference question to pick a sample-specific teacher layer, rather than committing to one fixed layer in advance. The attention from that selected layer is then distilled into a lightweight predictor, which learns to estimate visual token importance directly from multi-modal input features, without needing to run the full model first. At inference time, MAP combines these predicted importance scores with a diversity criterion and prunes the visual tokens before the first language-model layer even runs. Because pruning decisions come from the predictor rather than from computed attention maps, MAP does not need attention maps at inference and stays compatible with other inference-acceleration techniques used alongside it.

The authors tested MAP on LLaVA-NeXT-7B across ten benchmarks. It retained 97.5% of the unpruned model's performance while keeping only 5.56% of the visual tokens, and delivered a 3.09x end-to-end speedup.

Key facts

  • MAP (Middle-layer Attention Prediction) prunes visual tokens in multimodal LLMs by predicting, per sample, which middle layer's attention best signals token importance, instead of using one fixed layer for every input.
  • Question Contrastive Teacher Selection picks a sample-specific teacher layer by contrasting attention under the original question against a reference question.
  • That teacher layer's attention is distilled into a lightweight predictor that estimates visual token importance from multi-modal input features, then combined with a diversity criterion to prune tokens before the first language-model layer runs.
  • On LLaVA-NeXT-7B across ten benchmarks, MAP retains 97.5% of the unpruned model's performance while keeping only 5.56% of the visual tokens.
  • The approach yields a 3.09x end-to-end speedup and needs no attention maps at inference, so it stays compatible with other inference-acceleration techniques.

Why it matters

Visual token pruning is one of the main levers for cutting MLLM inference cost, but prior attention-guided methods pick a single predefined middle layer for reading importance signals. The authors' analysis found the best layer varies sample by sample, and reading attention from any middle layer at all requires first running visual tokens through several language-model layers, burning much of the compute the pruning was meant to save. MAP's contribution is to predict the useful attention signal directly from input features, before any of those layers run, and to select the right teacher layer per sample rather than fixing it in advance.

Who it affects

The paper targets builders and users of multimodal large language models who care about inference cost and latency, specifically systems that process many visual tokens per image. The evaluation is on LLaVA-NeXT-7B; the text does not report results on other model sizes or architectures.

How to use it

MAP is described as a method distilled into a lightweight predictor rather than a hosted product: it estimates visual token importance from multi-modal input features, prunes tokens before the first language-model layer, and requires no attention maps at inference time, which the authors say keeps it compatible with other inference-acceleration techniques already in use. No pricing, license or release details are given in the text.

How solid is it

The headline results, 97.5% of unpruned performance retained, 5.56% of visual tokens kept, and a 3.09x end-to-end speedup, are reported as averages across ten benchmarks on a single model, LLaVA-NeXT-7B. The text does not give a per-benchmark breakdown of those figures, does not name or quantify comparisons against other fixed-layer pruning baselines, and does not report whether the results generalize to other MLLM architectures or model sizes.

Risks and caveats

The source text does not give author names, institutional affiliations, or a publication or submission date. It reports only MAP's own retained-performance, token-fraction and speedup numbers, with no baseline comparison figures against competing pruning methods, and no breakdown by individual benchmark, so how MAP stacks up against alternatives on any single benchmark is not stated.

“MAP requires no attention maps for pruning and remains compatible with existing inference acceleration techniques.”

— the paper