Glonce

Random Attention matches top KV cache evictor with 32-43% higher throughput

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact62Novelty72Relevance98Surprise68Players48Credibility88
Random Attention matches top KV cache evictor with 32-43% higher throughput

Large language models that reason through long chains of thought run into a memory bottleneck: the KV cache, which stores keys and values for every token the model has processed, grows with the length of the reasoning trace. Existing methods for shrinking that cache all follow the same recipe: assign each cached token a score estimating how much it will matter later, then keep only the top-scoring tokens and evict the rest.

A new paper tests that recipe directly and finds the scoring step contributes almost nothing. Its authors introduce Random Attention, a method that keeps the original prompt intact but evicts everything else from the KV cache uniformly at random within each attention head, computing no score of any kind. Tested across four models and six reasoning tasks, Random Attention matches the strongest prior scoring-based evictor's performance while delivering 32-43% higher throughput than it when deployed in vLLM.

Controlled experiments trace that result to two mechanisms. First, the prompt turns out to be the fragile part of the cache: most of the performance gap between different scoring methods comes down to whether their particular signal happened to preserve the prompt, something Random Attention preserves by design rather than by scoring. Second, the reasoning trace protects itself from random eviction through built-in redundancy at two levels: in the text itself, since the model keeps restating what it still needs as it reasons, and across attention heads, since each head holds its own copy of the trace. Once the prompt is safe, a random draw across the rest of the cache keeps enough copies of what the model still needs, so no scoring signal is required to choose them. The authors publish their code on GitHub under the SalesforceAIResearch account.

Key facts

  • Random Attention preserves the original prompt in the KV cache, then evicts everything else uniformly at random within each attention head, computing no importance score at all.
  • Across four models and six reasoning tasks, it matches the performance of the strongest prior scoring-based evictor.
  • It delivers 32-43% higher throughput than that evictor when deployed in vLLM.
  • Most of the performance gap between scoring methods traces back to whether they happen to preserve the prompt, which the paper identifies as the fragile part of the cache.
  • The reasoning trace survives random eviction because of redundancy at two levels: it is restated in the text as the model works, and duplicated across separate attention heads.

Why it matters

Reasoning models generate long chains of thought, and every token of that reasoning has to sit in the KV cache during generation, making the cache the main memory and throughput bottleneck for serving these models at scale. The field has responded by building increasingly elaborate scoring methods to decide which cached tokens survive. This paper's result, that a method computing no score at all matches the best scorer while running 32-43% faster in vLLM, suggests much of that engineering effort was aimed at a problem a far simpler and cheaper method already solves.

Who it affects

Teams that build or operate inference for long chain-of-thought reasoning models, particularly on serving stacks like vLLM, are the direct audience: they gain a way to cut memory pressure and raise throughput without an accuracy cost or the overhead of running a scoring model. Researchers working on KV cache compression are affected too, since the paper's controlled experiments question the premise behind an entire line of scoring-based eviction work.

How to use it

The authors publish an implementation on GitHub under the SalesforceAIResearch account, so the method can be tried directly rather than reimplemented from the paper's description alone. The technique itself needs no training or tuning: keep the prompt tokens, then evict the rest of the cache with a uniform random draw inside each attention head. Because there is no score to compute, the eviction policy adds none of the compute cost that scoring-based methods carry, which is itself part of where the throughput gain comes from.

How solid is it

The result rests on evaluation across four separate models and six reasoning tasks, backed by controlled experiments that isolate why random eviction works rather than just reporting a benchmark table: one experiment attributes most of the scoring methods' advantage to prompt preservation, another confirms redundancy in the reasoning trace at both the text and attention-head level. The text does not name the four models, the six tasks, or the specific prior evictor used as the comparison point, and it reports only the relative 32-43% throughput figure rather than absolute tokens-per-second numbers, so the result cannot yet be checked against a reader's own model or workload without consulting the full paper.

Risks and caveats

The method works because the paper's own analysis shows the prompt is protected by design and the reasoning trace is redundant enough to survive random cuts; a task whose reasoning is less repetitive, or a pipeline that does not special-case the prompt, might not see the same result. The text also does not describe the deployment setup behind the 32-43% figure, including hardware, batch size or sequence length, so that gain may not transfer identically to every serving configuration.

“We show that the selection signal contributes almost nothing.”

— the paper's authors