Glonce

OasisKV boosts LLM inference throughput with lookahead KV cache prefetching

infrastructureThe Glonce editorsHugging Face🔥71Glonce rating71Impact70Novelty70Relevance92Surprise50Players35Credibility82
OasisKV boosts LLM inference throughput with lookahead KV cache prefetching

LLM inference serving is increasingly bottlenecked by memory rather than compute: as long-context and long-form reasoning workloads spread, the key-value (KV) cache dominates both memory footprint and memory traffic during token generation (decode), and high-bandwidth memory (HBM) capacity has become the scarce, costly resource that limits batch size and system throughput.

The authors present OasisKV, a memory-centric LLM inference system built on vLLM that decouples full KV-cache storage from HBM during decoding. The design exploits the fact that decode-time attention is naturally sparse: OasisKV keeps only the KV entries for the most relevant tokens in HBM for attention computation, while the rest are staged in higher-capacity memory tiers such as host or remote memory. It predicts which tokens will matter next using lookahead tokens drafted by speculative decoding, then runs a background attention pipeline that identifies the important KV blocks and prefetches them into HBM ahead of the next decode step.

In evaluation, the lookahead prediction stays accurate enough that, under a 2,048-token KV budget, OasisKV's accuracy is within 0.7 points of full attention. That sparsity converts into throughput: OasisKV reaches 1.69x the throughput of dense vLLM on a reasoning workload at a cost of 0.1 points of accuracy, and up to 2.1x on multi-GPU long-context serving. Under prefill-decode disaggregation, OasisKV reaches about 2x dense throughput while admitting each request with 6.5x to 9.7x less KV data and holding 2.2x to 2.6x less decode-node host memory than a full KV transfer.

Key facts

  • OasisKV decouples KV cache storage from HBM during LLM decode, keeping only the most relevant tokens' KV entries in HBM and staging the rest in host or remote memory.
  • It predicts important future tokens using lookahead tokens from speculative decoding, then prefetches the matching KV blocks via a background attention pipeline before the next decode step.
  • Under a 2,048-token KV budget, accuracy stays within 0.7 points of full attention.
  • Throughput gains: 1.69x over dense vLLM on a reasoning workload at 0.1 points of accuracy loss, and up to 2.1x on multi-GPU long-context serving.
  • Under prefill-decode disaggregation, OasisKV reaches about 2x dense throughput while using 6.5x to 9.7x less KV per request and 2.2x to 2.6x less decode-node host memory than full KV transfer.

Why it matters

LLM inference is increasingly memory-bound rather than compute-bound as context windows and reasoning chains grow, and the KV cache is the dominant consumer of that memory. HBM capacity limits how many requests a server can batch together and, in turn, overall throughput. OasisKV attacks this bottleneck directly by moving cold KV data off HBM without much accuracy loss, which raises the ceiling on batch size and throughput for long-context serving.

Who it affects

Teams operating LLM inference infrastructure, especially for long-context or long-form reasoning workloads where the KV cache is the binding constraint, and anyone running vLLM-based serving stacks or evaluating prefill-decode disaggregated architectures, since OasisKV is implemented directly on vLLM.

How to use it

OasisKV is presented as a system built on top of vLLM rather than a standalone product; the source text does not state a release, licence, or code availability, so there is nothing to install from what's described here. Its relevance is architectural: it shows how HBM can be combined with host or remote memory tiers for KV cache management during decode.

How solid is it

The results come from the authors' own evaluation on vLLM: 1.69x throughput on a reasoning workload at 0.1 points of accuracy loss, up to 2.1x on multi-GPU long-context serving, about 2x under prefill-decode disaggregation, and 6.5x to 9.7x less KV admitted per request. The source text does not name the specific model(s) tested, the reasoning benchmark, the GPU hardware, or give author affiliations, a publication venue, or a date, so independent verification of the setup is not possible from what is available here.

Risks and caveats

The reported numbers are the authors' own benchmarks rather than independently reproduced results, and the source does not specify which LLM, benchmark, or hardware produced them, making it hard to judge how the gains generalize beyond the tested setup. The approach also depends on speculative decoding's lookahead tokens being accurate predictors of future attention importance; how that holds up outside the tested settings is not addressed in the text.