Glonce

HNSW vector index speeds up Gemma 3 270M decoding by up to 82%

researchThe Glonce editorsarXiv🔥71Glonce rating71Impact65Novelty70Relevance95Surprise50Players45Credibility80

The paper targets a specific bottleneck in autoregressive decoding: the output embedding matrix that projects a model's hidden state onto every token in the vocabulary. For compact LLMs with large multilingual vocabularies, this dense projection has to touch the entire vocabulary matrix at every decoding step, and the authors identify it as a major source of memory bandwidth pressure, especially in the batch-size-one, latency-sensitive setting typical of running a small model locally.

Their fix is to reformulate the combination of output projection and top-k token selection as a maximum inner product search over token embeddings, then serve that search with an HNSW-based approximate nearest neighbor index instead of a dense matrix multiply. Rather than computing scores for the full vocabulary, the resulting output head retrieves only a small candidate set of high-scoring tokens. To stay compatible with existing decoding code, those retrieved logits are scattered back into a sparse tensor shaped like the full vocabulary, so the method can be integrated into existing decoding pipelines without restructuring them.

The authors tested the approach on CPU inference with three model families: Gemma 3, Llama 3.2, and Qwen 3. The one quantified result given is for Gemma 3 270M, where end-to-end batch-size-one decoding throughput improves by up to 82%. Generation quality was checked under AlpacaEval and, according to the authors, is preserved. No throughput figures for Llama 3.2 or Qwen 3, and no AlpacaEval score, are given in the abstract; the claim for those models is qualitative. The paper's conclusion is that approximate retrieval is a practical alternative to dense output projections specifically for latency-sensitive, small-batch decoding.

Key facts

  • The method reformulates output projection plus top-k token selection as a maximum inner product search, replacing the dense vocabulary matrix with an HNSW-based vector index.
  • Only a small candidate set of high-scoring tokens is retrieved per step, then scattered into a sparse full-vocabulary tensor so it drops into existing decoding pipelines.
  • On CPU inference, end-to-end batch-size-one decoding throughput improves by up to 82% for Gemma 3 270M.
  • The technique was also tested on Llama 3.2 and Qwen 3, though no throughput numbers for those two are given.
  • Generation quality under AlpacaEval evaluation is reported as preserved, without a stated score.

Why it matters

Small LLMs with large multilingual vocabularies spend a disproportionate share of decoding time and memory bandwidth on the output projection, the step that turns a hidden state into a probability over every possible next token. That cost hits hardest exactly where compact models are meant to shine: single-request, latency-sensitive inference on CPU rather than large-batch serving on accelerators. By recasting that projection as a nearest-neighbor search and answering it with an approximate index instead of a full dense multiply, the paper attacks the bottleneck directly rather than shrinking the model or the vocabulary.

Who it affects

The result is aimed at anyone running small LLMs in batch-size-one settings, such as on-device or CPU-bound inference, where memory bandwidth rather than raw compute limits decoding speed. It is most relevant for models with large multilingual vocabularies, since that is where the dense output projection is heaviest.

How to use it

The method is described as integrating into existing decoding pipelines: retrieved logits from the HNSW index are scattered into a sparse tensor shaped like the full vocabulary, so downstream sampling and decoding logic does not need to change. The abstract does not state whether code or a reference implementation has been released.

How solid is it

The evidence given is CPU-inference throughput on three model families, Gemma 3, Llama 3.2, and Qwen 3, with the single quantified figure, up to 82% throughput improvement, reported only for Gemma 3 270M. Quality preservation is checked with AlpacaEval, but no score is given, only a claim that quality holds. No comparable numbers are provided for Llama 3.2 or Qwen 3, and no absolute latency, throughput, or memory-bandwidth figures accompany the relative 82% number.

Risks and caveats

HNSW is an approximate nearest-neighbor method, so the retrieved candidate set is not guaranteed to match the exact top-k tokens a dense projection would produce; the paper's quality claim rests on aggregate AlpacaEval behavior rather than a token-level accuracy check. The abstract gives no figures for index build time, memory footprint, or index size relative to the dense projection it replaces, and the 82% gain is demonstrated for one model size on CPU only, so it should not be assumed to generalize to larger models, GPU inference, or batched serving without further evidence.