Glonce

Fathom speeds long-context KV cache decoding 1.67x with per-query reads

researchThe Glonce editorsHugging Face🔥72Glonce rating72Impact65Novelty75Relevance95Surprise50Players35Credibility80
Fathom speeds long-context KV cache decoding 1.67x with per-query reads

When agentic sessions run to a million tokens, with many sessions resident in host memory at once, the KV cache and the index that ranks it live in host memory, and the scan that ranks all n keys for a top-k decoding step becomes the traffic that bounds decoding speed.

Fathom is a key scan built for that setting: each query decides how many bits of each key channel to read, instead of every key being read at one fixed depth. The 4-bit K cache is stored channel-major as bit planes, so a prefix of t planes is exactly that channel's t-bit quantizer. A query spends its bit budget by reverse water-filling over the variance-weighted importance of its channels, reading some channels deeper than others.

At one million tokens on Qwen3-8B, a decode step with Fathom is 1.67x faster in GPU time than with the 136-bit scans of Double Sparsity, Loki, and SparQ r=32. Given the same GPU time as SparQ's 68-bit read (r=16), Fathom reads 18% fewer bytes and produces lower attention error on six of seven model and context settings.

On RULER-style tasks, every per-token scan tested, Fathom included, matches exact top-k decoding. On real coding-agent sessions, Fathom reaches the step agreement of the most accurate 136-bit scan while reading only 92 bits.

The method reads from the 4-bit K copy that a quantized serving stack already holds, and it is not faster when the ranking index is resident in GPU memory rather than offloaded to host memory.

Key facts

  • Fathom lets each query decide how many bits of each key channel to read from an offloaded, 4-bit quantized KV cache, rather than reading every key at a fixed bit depth.
  • At one million tokens on Qwen3-8B, Fathom's decode step is 1.67x faster in GPU time than the 136-bit scans of Double Sparsity, Loki, and SparQ r=32.
  • At the same GPU time as SparQ's 68-bit read (r=16), Fathom reads 18% fewer bytes while producing lower attention error on six of seven model and context settings.
  • On real coding-agent sessions, Fathom matches the step agreement of the most accurate 136-bit scan while reading only 92 bits.
  • The method gives no speedup when the ranking index is resident in GPU memory rather than offloaded to host memory.

Why it matters

Agentic sessions that run to a million tokens, with many sessions resident in host memory at once, push the KV cache and its ranking index out of GPU memory. The scan that ranks all n keys for a top-k decoding step then becomes the traffic that limits how fast decoding can go. Fathom targets that bottleneck by changing how many bits of each key get read during the scan, instead of reading every key at one fixed depth.

Who it affects

The method targets long-context, agentic inference where the KV cache is offloaded to host memory and served from a quantized store, the setting in which scan traffic becomes the bottleneck. It is built on and measured against other sparse-decoding scans: Double Sparsity, Loki, and SparQ.

How to use it

Fathom reads from the 4-bit K cache that a quantized serving stack already holds, stored channel-major as bit planes, so a prefix of t planes is exactly that channel's t-bit quantizer. Each query spends a bit budget by reverse water-filling over the variance-weighted importance of its channels, deciding on the fly how deep to read each channel instead of using one fixed depth for the whole cache.

How solid is it

The reported numbers come from tests on Qwen3-8B at one million tokens, where a decode step runs 1.67x faster in GPU time than the 136-bit scans of Double Sparsity, Loki, and SparQ r=32, and at SparQ's 68-bit read (r=16) Fathom reads 18% fewer bytes with lower attention error on six of seven model and context settings. On RULER-style tasks, every per-token scan tested, Fathom included, matches exact top-k decoding, and on real coding-agent sessions Fathom reaches the step agreement of the most accurate 136-bit scan while reading only 92 bits.

Risks and caveats

The source text names no authors or institutions and gives no publication date, no GPU model for the timing figures, and no description of what the RULER-style tasks or coding-agent sessions consist of beyond their names. The method also has a stated limit: it gives no speedup when the ranking index sits in GPU memory instead of being offloaded, so the gains apply specifically to the offloaded, host-memory setting.