Glonce

Topology-aware routing cuts LLM KV cache transfer latency up to 18x

infrastructureThe Glonce editorsarXiv🔥73Glonce rating73Impact70Novelty75Relevance95Surprise50Players30Credibility75

In disaggregated LLM inference, the prefill and decode stages run on separate GPU pools, so the KV cache has to be moved between them for every request. For a 70B parameter model, that transfer is 2.6 GB per request, and at production scale the aggregate bandwidth needed exceeds 100 GB/s. A new paper argues that existing disaggregated-serving systems, including DistServe, Splitwise, and Mooncake, all move this data over uniform RDMA and ignore a huge gap in the hardware underneath: bandwidth between two GPUs can vary by 72x depending on their physical relationship, from 900 GB/s over NVLink within a domain, to 50 GB/s over InfiniBand across nodes, down to 12.5 GB/s over TCP across data centers. Treating all of that as one uniform transport wastes the fast paths and gets throttled by the slow ones.

The paper proposes a topology-aware transfer orchestrator that discovers the interconnect hierarchy at startup and picks the best transport for each transfer rather than treating the network as uniform. Three mechanisms do the work together. First, pipelined layer-by-layer transfer sends KV cache data as it is produced, overlapping transmission with ongoing prefill computation and hiding 60 to 85 percent of the transfer latency behind compute that is happening anyway. Second, for Mixture-of-Experts models, NVLink domain-aware placement co-optimizes where experts are dispatched with where the KV cache already sits, keeping transfers inside the fast NVLink domain where possible. Third, the design adds CXL 3.0 memory expanders as a shared overflow tier, which the authors say provide 6x the capacity of NVMe storage at 86x lower latency, giving the system somewhere fast to spill to when GPU memory runs out.

The authors are upfront that they did not run a full empirical evaluation. That would require multi-node clusters with heterogeneous interconnects plus CXL 3.0 hardware, which the paper says is beyond academic resources and not yet available even in GPU clouds. Instead, the paper presents analytical bandwidth models, working implementations of the individual components, and a projected analysis across three architectures, estimating a 3 to 18x reduction in transfer latency compared with uniform RDMA. The text does not name the three architectures, identify authors or an institution, or give a publication date; the 3 to 18x figure should be read as a projection from the models, not a measured result on running hardware.

Key facts

  • For a 70B parameter model, KV cache transfer between prefill and decode GPU pools is 2.6 GB per request, exceeding 100 GB/s aggregate bandwidth at production scale.
  • Bandwidth between two GPUs varies by up to 72x depending on physical relationship: 900 GB/s via NVLink within a domain, 50 GB/s via InfiniBand across nodes, 12.5 GB/s via TCP across data centers.
  • Existing disaggregated-serving systems, including DistServe, Splitwise, and Mooncake, all use uniform RDMA and ignore this bandwidth gap, according to the paper.
  • The proposed orchestrator combines pipelined layer-by-layer transfer (hiding 60 to 85 percent of latency behind compute), NVLink domain-aware placement for Mixture-of-Experts models, and a CXL 3.0 overflow tier offering 6x the capacity of NVMe at 86x lower latency.
  • Projected analysis across three unnamed architectures shows a 3 to 18x reduction in transfer latency versus uniform RDMA; the authors state a full empirical evaluation was not performed because it requires multi-node, heterogeneous-interconnect, CXL 3.0 hardware not yet available even in GPU clouds.

Why it matters

Disaggregated inference, splitting prefill and decode across separate GPU pools, is becoming common precisely because it lets each stage scale independently. But every request still has to carry its KV cache across that split, and the paper's central claim is that current systems handle this badly: they push all transfers through uniform RDMA even though the underlying interconnects differ in bandwidth by up to 72x, from 900 GB/s NVLink links down to 12.5 GB/s TCP across data centers. Named systems DistServe, Splitwise, and Mooncake are all described as making this mistake. If the analysis holds, it identifies a concrete, fixable inefficiency in how large-scale LLM serving infrastructure is built today, rather than a hardware or model limitation.

Who it affects

The audience is teams that operate or design large-scale disaggregated LLM inference systems, cloud providers and GPU infrastructure operators running prefill/decode-split serving, and researchers or engineers working on Mixture-of-Experts deployment, since one of the three mechanisms is specifically about MoE expert placement. It is not consumer-facing and names no specific model family, company, or product beyond the generic '70B model' example used for the size calculation.

How to use it

The paper describes component implementations rather than a shipping product: a topology discovery step at startup, pipelined layer-by-layer KV cache transfer that overlaps with prefill computation, NVLink-domain-aware expert and cache placement for MoE models, and a CXL 3.0 memory tier used as shared overflow storage. No pricing, license, or release details are given, and the text does not say whether any of this code has been made available; adoption would depend on operating the kind of heterogeneous, multi-node interconnect topology and CXL 3.0 hardware the design targets.

How solid is it

The paper itself flags the limits of its own evidence: it presents analytical bandwidth models and working component implementations, but not a full empirical evaluation, because that would need multi-node clusters with heterogeneous interconnects and CXL 3.0 hardware the authors say is beyond academic resources and not yet available in GPU clouds. The headline 3 to 18x latency reduction is a projected analysis across three architectures, not a benchmark measured on running systems, and the three architectures are not identified in the text.

Risks and caveats

The central result rests on models and projections rather than measurement, so the real-world gain could differ once tested on actual hardware. CXL 3.0 memory expanders, one of the three mechanisms, are described as not yet available in GPU clouds, which limits how much of the design can be deployed today. The source text gives no author names, institution, or publication date, and does not identify the three architectures used for the projected 3 to 18x figure.