LowRAM paper compresses KV cache by predicting values from keys
A paper listed on Hugging Face Papers proposes LowRAM, a method for shrinking the KV cache used in autoregressive decoding. The problem it starts from is familiar: the cache's memory footprint grows linearly with context length, and that caps batch size and throughput.
The authors say existing post-training methods exploit redundancy across tokens, layers and feature dimensions, or lower numerical precision. They identify a complementary axis: within each attention head, values are largely predictable from their corresponding keys.
They exploit this with what they call value cache amortisation. At test time it fits a lightweight per-head predictor, and it stores residuals only for the tokens the predictor reconstructs worst. The authors report that this gives 4x near-lossless compression of the value cache. The predictor can be initialised by meta-learning, which substantially reduces prefill overhead, or fitted from scratch in settings that allow a one-off cost, such as multi-query inference.
LowRAM pairs this value amortisation with cross-layer low-rank key compression, giving an asymmetric pipeline: keys and values are compressed by different mechanisms. The authors say it stays close to lossless across three model families. Because it reconstructs only a selected subset of KV pairs at each decoding step, they say LowRAM can keep decoding overhead minimal and reach up to 5x higher peak decode throughput at 4x compression.
LowRAM also composes with quantisation and eviction. In that combination, on long contexts, the authors report 26.9x compression at 94.5% of baseline performance. Their conclusion is that treating keys and values as structurally distinct enables efficient, aggressive and composable KV cache compression.
Key facts
- LowRAM rests on the observation that, within each attention head, values are largely predictable from their corresponding keys.
- Value cache amortisation fits a lightweight per-head predictor at test time and stores residuals only for the worst-reconstructed tokens, giving 4x near-lossless value compression.
- Paired with cross-layer low-rank key compression, LowRAM is reported to stay close to lossless across three model families and reach up to 5x higher peak decode throughput at 4x compression.
- Combined with quantisation and eviction, it reaches 26.9x compression at 94.5% of baseline performance on long contexts.
Why it matters
The KV cache is the memory bottleneck of long-context decoding: it grows linearly with context length and limits batch size and throughput. Prior post-training methods work on tokens, layers, feature dimensions or numerical precision. LowRAM adds a different axis, the predictability of values from keys inside each head, and treats keys and values as structurally distinct rather than compressing them alike. If the reported numbers hold, the cache shrinks much further without a large quality cost.
Who it affects
The work is aimed at anyone serving or running autoregressive models where the KV cache limits batch size or throughput, especially at long context lengths. The authors single out multi-query inference as a setting where a one-off predictor fitting cost is acceptable.
How to use it
This is a method described in a paper, not a product. The predictor can be initialised by meta-learning to cut prefill overhead, or fitted from scratch where a one-off cost is acceptable. LowRAM is designed to compose with quantisation and eviction. No code release, model release or licence is mentioned in the source.
How solid is it
The evidence here is the paper's own abstract, so every figure is the authors' self-reported result. The headline numbers are 4x near-lossless value compression, up to 5x higher peak decode throughput at 4x compression, and 26.9x compression at 94.5% of baseline performance on long contexts. The abstract names no authors or institutions. The three model families are not named, nor are model sizes. The benchmarks, tasks and metric behind "performance" and "long contexts" are not named. No comparison with named competing KV cache compression methods is given.
Risks and caveats
The baseline for the "up to 5x" throughput figure is not stated, and "up to" marks a best case. The 94.5% figure is performance relative to baseline, not an absolute score. No numbers are given for prefill overhead, for the key compression ratio alone, or for the quality loss at 4x compression beyond "near-lossless" and "close to lossless". Fitting the predictor at test time costs something up front, which the authors say meta-learned initialisation reduces.
“These results show that treating keys and values as structurally distinct enables efficient, aggressive, and composable KV cache compression.”
— LowRAM paper abstract