Glonce

KVBoost cuts LLM time-to-first-token 4.49x with chunk-level KV cache reuse

researchThe Glonce editorsarXiv🔥73Glonce rating73Impact68Novelty72Relevance100Surprise52Players32Credibility82

Researchers present KVBoost, a chunk-level key-value (KV) cache reuse system for HuggingFace-compatible decoder LLMs. Standard prefix-caching systems already avoid recomputing KV tensors for repeated content, but only when that content sits in a shared leading prefix; if the same text appears elsewhere in the prompt, no reuse happens and it gets recomputed from scratch. KVBoost removes that positional restriction with a dual-hash keying scheme that separates positional identity (a prefix hash) from content identity (a content hash), so both exact and approximate matches can be found regardless of where the content falls in the prompt. Because chunks are cached independently, stitching them back together can introduce attention boundary errors at the seams. KVBoost addresses this with two repair strategies: SelectiveRecompute, which re-encodes the boundary regions between chunks, and CacheBlendRecompute, which runs a probe pass to identify high-deviation tokens and recomputes only those. The system also applies asymmetric KV quantization (int8/int4), adaptive chunk boundary splitting, and importance-weighted eviction to stay within a fixed memory budget. The authors evaluated KVBoost on Qwen/Qwen2.5-3B across 1,000 bug-localization samples. Time-to-first-token dropped 4.49x, from 639.1 ms to 142.4 ms, and KVBoost separately beat prefix caching by 16% on the same metric. Accuracy held at 99.2% versus 99.1% for the comparison baseline, which the authors describe as no loss. They frame KVBoost as a practical, memory-bounded acceleration layer that works with RoPE-based models without any architectural changes.

Key facts

  • KVBoost is a chunk-level KV cache reuse system that lets HuggingFace-compatible decoder LLMs reuse cached content regardless of its position in the prompt, unlike prefix caching, which only reuses a shared leading prefix.
  • A dual-hash keying scheme separates positional identity (prefix hash) from content identity (content hash) to support both exact and approximate cache matches.
  • Two repair strategies, SelectiveRecompute and CacheBlendRecompute, fix attention boundary errors that arise from stitching together independently cached chunks.
  • On Qwen/Qwen2.5-3B over 1,000 bug-localization samples, KVBoost cut time-to-first-token 4.49x (639.1 ms to 142.4 ms) and beat prefix caching by 16% on the same metric, with accuracy at 99.2% versus 99.1% for the baseline.
  • KVBoost also uses asymmetric int8/int4 KV quantization, adaptive chunk boundary splitting, and importance-weighted eviction to operate within a fixed memory budget.

Why it matters

LLM inference pays a real, repeated cost: every request that shares content with a previous one, a system prompt, a retrieved document, boilerplate context, still has its key-value tensors recomputed unless that content happens to sit in a shared leading prefix. Prefix caching only catches the leading-prefix case. Once the shared content moves to the middle or end of a prompt, or appears in a different order across requests, no reuse occurs and the model pays full prefill cost again. KVBoost's dual-hash scheme is built to catch reuse in exactly those cases prefix caching misses, by keying content independently of its position.

Who it affects

Anyone serving HuggingFace-compatible decoder LLMs where requests share content at varying positions, retrieval-augmented pipelines with repeated retrieved chunks, coding assistants and bug-localization tools like the one KVBoost was tested on, or any workload with reusable context that does not sit in a clean shared prefix. It targets RoPE-based models specifically, since the positional handling is built around that scheme.

How to use it

The authors describe KVBoost as a memory-bounded inference acceleration layer that works with RoPE-based models without any architectural modification, implying it can sit alongside an existing model rather than requiring retraining or redesign. It combines the chunk-reuse mechanism with asymmetric int8/int4 KV quantization, adaptive splitting of chunk boundaries, and importance-weighted eviction to keep memory use within a fixed budget. The source gives no pricing, licensing terms, or code release, so nothing can be said here about availability.

How solid is it

The results come from a single evaluation: Qwen/Qwen2.5-3B on 1,000 bug-localization samples. Time-to-first-token fell 4.49x against a comparison baseline (639.1 ms to 142.4 ms), and KVBoost separately beat prefix caching by 16% on the same metric, a distinct comparison from the 4.49x figure. Accuracy was 99.2% against 99.1% for the baseline. The source does not specify exactly what that baseline is, whether no caching at all or full recomputation, beyond identifying it as the comparison point behind the 4.49x and accuracy figures. No authors, institutions, hardware details, or code release are named in the text.

Risks and caveats

The need for two separate repair strategies to fix attention boundary errors shows that splitting the KV cache into independently stored chunks is not free: stitching them back together can corrupt attention unless boundaries are recomputed, which itself adds work. The evaluation covers one 3B model and one task type, bug localization, so how the approach generalizes to larger models or other workloads is untested here. The only named comparison system is prefix caching in general; no other existing cache-reuse method is benchmarked. With no code or artifact release mentioned, the results cannot currently be independently verified or reproduced.