Cloudflare quantizes Kimi and GLM caches for 41% faster inference

Cloudflare's Workers AI runs inference for Moonshot's Kimi K-series and Z.ai's GLM on GPUs in its own data centers. Both are large, long-context mixture-of-experts models that are hard to serve efficiently because they strain GPU memory. On top of the prefill/decode separation Cloudflare has already described elsewhere, the company lays out three further techniques it uses to fit these models into memory and keep them fast: quantizing the KV cache, compressing the model weights, and protecting the shared cache those denser deployments create. All experiments and production traffic run on SGLang, an open-source inference serving framework Cloudflare says offers the best performance in the market and that it works closely with the SGLang team to upstream patches and features.

The first technique is KV cache quantization. As a model generates text it stores the attention keys and values for every prior token in a KV cache, which is usually what fills GPU memory first on long-context models. Cloudflare stores this cache in 8-bit floating point (FP8, e4m3) instead of the default 16-bit (BF16), halving its size. On Kimi K2.6 this raises the context that fits in memory from roughly 686,000 tokens to about 1.37 million, twice as much. At any single concurrency level BF16 is a few percent faster per token, but it runs out of cache at 32 concurrent requests and cannot admit a 33rd, while FP8 keeps going to 64 concurrent requests and reaches a peak of 2,192 tokens per second, about 41% higher than BF16's peak, at roughly 30% less cost per token. Because Cloudflare runs prefill and decode as separate pools, it applies FP8 only where it helps: prefill is compute-bound, so it stays in BF16, while decode, which is memory-bound, gets FP8. Across Cloudflare's evaluation suite, FP8 and BF16 caches produce indistinguishable answers.

The second technique is weight compression. For GLM 5.2, Cloudflare compresses the weights from 8-bit floating point down to 4-bit integers (INT4). The checkpoint shrinks from 705 GB to 421 GB, about 40%, and per-GPU memory across an 8-way tensor-parallel deployment drops from roughly 88 GB to 52 GB, freeing room for around 1.18 million tokens of additional KV cache on the same hardware. Since generating each token means streaming weights out of GPU memory, smaller weights speed up the memory-bandwidth-bound decode phase, especially at low concurrency. Prefill behaves the opposite way: it is compute-bound, and INT4 weights first have to be expanded back out before multiplication, which makes prefill slower rather than faster. GLM sustains about 10,160 tokens per second of prefill in FP8 versus 8,660 in INT4, so Cloudflare again splits the work: INT4 for decode, FP8 for prefill. Accuracy stays within 0.8 points of the FP8 model across every benchmark Cloudflare runs.

The third technique addresses a side effect of the first two: packing far more requests onto one GPU's shared memory means hundreds of requests are reading and writing pages of the same physical KV cache, and at Cloudflare's request volumes even a one-in-a-billion bookkeeping mistake would surface regularly. Cloudflare built a KV cache integrity check: every physical cache page gets a tag that changes whenever the page is reallocated, and the server records which pages and tags each request expects. Before supported decode operations read from the cache, those mappings are checked, and if anything does not match, the affected request is aborted rather than allowed to return data from the wrong page. Measured on a mid-sized production model in a two-prefill, two-decode configuration with 8,192-token inputs and 1,000-token outputs, the cost is under 1% on both throughput and tail latency, and even the upper bound of the 95% confidence interval stays near 1%. Cloudflare kept this cheap by running the check as a separate batch validation rather than fusing it into the attention kernel, which would have created a race between GPU thread groups; the default path uses a no-op tracker with no measurable overhead for deployments that do not enable checking.

Cloudflare says it is expanding FP8 KV caches across more of its fleet, validating NVFP4 weights on Nvidia's Blackwell GPU architecture, and working toward making integrity checks something it can leave on everywhere at negligible cost.

Key facts

FP8 KV cache quantization (replacing the default BF16) raises Kimi K2.6's max context held in memory from about 686,000 to about 1.37 million tokens, lets it serve 64 concurrent requests instead of 32, and reaches 2,192 tokens per second peak decode throughput, about 41% higher than BF16's peak, at roughly 30% less cost per token.
INT4 weight compression on GLM 5.2 shrinks the checkpoint from 705 GB to 421 GB (about 40%) and per-GPU memory from about 88 GB to 52 GB across an 8-way tensor-parallel deployment, freeing room for about 1.18 million more tokens of KV cache; accuracy stays within 0.8 points of the FP8 baseline.
Because prefill is compute-bound and decode is memory-bound, Cloudflare runs each phase in whichever precision wins: BF16/FP8 for prefill, FP8 cache and INT4 weights for decode; GLM prefill runs at 10,160 tokens per second in FP8 versus 8,660 in INT4.
A new KV cache integrity check tags every physical cache page and aborts any request whose expected page and tag mapping does not match, protecting requests that now share GPU memory at much higher density; measured cost is under 1% on throughput and tail latency.
All experiments and production traffic run on SGLang, an open-source inference serving framework Cloudflare says offers the best performance in the market and works closely with the SGLang team to upstream patches and features.

Why it matters

Serving large, long-context open mixture-of-experts models like Kimi and GLM in production is bottlenecked by GPU memory, not raw compute. Cloudflare's three stacked techniques, cache quantization, weight compression and integrity checking, let it pack substantially more concurrent requests onto the same hardware while holding accuracy steady, which directly lowers the cost of running frontier-scale open models at volume.

Who it affects

Cloudflare Workers AI customers running inference on Kimi or GLM benefit directly through more capacity and lower cost per token. The techniques generalize to any inference provider serving large long-context mixture-of-experts models on memory-constrained GPU fleets, and to end users who see faster responses as a result.

How to use it

These are backend infrastructure changes inside Cloudflare's own serving stack rather than something a developer configures directly. The one exposed control is the integrity check, which Cloudflare says is enabled per deployment; its default path uses a no-op tracker, so deployments that do not need it pay no overhead.

How solid is it

The figures come from Cloudflare's own benchmarks on a disaggregated H200 deployment for the cache and weight measurements, and a two-prefill, two-decode configuration with stated input and output token counts for the integrity-check cost, including a reported 95% confidence interval. All testing runs on SGLang. These are Cloudflare's self-reported production and evaluation-suite numbers, not independently audited figures.

Risks and caveats

The post is unsigned, carries a publication date of 2026-08-03, and does not name the internal 'evaluation suite' used to confirm accuracy is unaffected, so the claims cannot be independently checked against a named benchmark. As a vendor's own account of its infrastructure, it has an incentive to present the results favorably. No dollar pricing, customer names or case studies are given, and no timeline is set for expanding FP8 caches fleet-wide or completing NVFP4 validation on Blackwell.

“If anything doesn't match, the affected request is aborted rather than allowed to return data from the wrong page.”

— Cloudflare, Workers AI blog