Glonce

Cloudflare reports GLM-5.3 on Workers AI leaked other customers' data in 155 responses

securityThe Glonce editorsHacker News🔥78Glonce rating78Impact70Novelty65Relevance95Surprise70Players70Credibility90
Cloudflare reports GLM-5.3 on Workers AI leaked other customers' data in 155 responses

On September 8, 2026, security researcher Gadi Evron from Knostic told Cloudflare, through its HackerOne program, that very long prompts sent to the GLM-5.3 model on Workers AI (@cf/zai-org/glm-5.3) sometimes returned text belonging to other users' sessions. Cloudflare reproduced the problem the next morning, took the model offline within 40 minutes of declaring an incident, deployed a corrected configuration and a patched inference engine, and restored the model on September 9. The exposure ran from August 27 to September 9. In total, 155 responses were affected, all to prompts longer than 681,280 tokens, and a small number of customers received them. Cloudflare says no other model hosted on Workers AI was affected.

After excluding its own testing and the researcher's, Cloudflare says the requests it identified are consistent in timing and volume with ordinary long-context use, not with someone repeatedly exploiting the bug. It states it found no evidence of deliberate exploitation. The report was published alongside a post from SGLang, the open-source inference engine Cloudflare uses, as a coordinated disclosure.

The cause was two overlapping issues. The first was a misconfiguration. Cloudflare deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on its prefill GPUs had room for only 681,280 tokens. SGLang sizes that pool at startup from whatever GPU memory remains after loading the model weights, and nothing forces it to match the context length. Cloudflare knew about the mismatch before launch. It expected the engine to return an error for any prompt it could not process, and its long-context tests returned successful responses, so it did not treat the gap as a problem. Its pre-launch testing checked only HTTP response codes and engine performance, not model output and coherence.

The second was an SGLang bug in disaggregated prefill/decode mode, where prefill GPUs process the prompt and hand the KV cache over RDMA (via Mooncake) to decode GPUs. When a prompt was too long for the pool, SGLang's scheduler marked it as rejected and swapped in a one-token placeholder. On a single node, that worked and the client got an error. In disaggregated mode, the admission queues checked whether a request was finished, not whether it was rejected, so a rejected request looked live. Prefill ran the placeholder, sent one page of KV cache and signalled that the transfer was complete. Decode, which had reserved 8 pages in the source's example, then attended over all 8. Only page P0 had been written by the new request; pages P1 to P7 still held earlier requests' prompts and partial generations. The client got HTTP 200 instead of HTTP 400, with a coherent reply that arrived faster than an 8-page prefill should take. KV pages are not zeroed between requests, which Cloudflare says is common among inference engines and necessary for best performance, so the engine itself must track page ownership.

Cloudflare's fixes were twofold. It changed the GLM 5.3 configuration to match its GLM 5.2 one, freeing GPU memory so the prefill KV pool now holds about 2.04 million tokens, roughly twice the context length; a valid prompt now always fits and the rejection path is not reached. It also patched the engine so a request carrying a pending rejection is retired at the disaggregation admission queue with its original error, before any bootstrap, allocation or KV transfer. Cloudflare proposed this upstream as #38704, which also has the decode node verify that the number of tokens transferred matches the number it reserved. A fix was already on SGLang main in #36988, merged September 6 for a related failure path but not yet released, and maintainers also merged #38935, which rejects such requests earlier, following Cloudflare's approach. Both changes were validated against the researcher's reproduction script and Cloudflare's own canary-marker tests before the model returned.

The same bug had been reported to SGLang by another user as issue #30233 on July 6, 2026, which Cloudflare noticed during the investigation. The patch shipped in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22, and Cloudflare recommends all operators update to v0.5.22 or later. Cloudflare has applied the engine fix and hardening to every model it serves with SGLang, and its deployment checks require them for every new deployment. It now also checks KV pool size against context length directly from the running engine before any disaggregated deployment change ships.

Cloudflare places this bug in a class of KV cache memory-safety bugs it has worked on for several months, where a page is freed, reused or read while something else still has a claim on it. One cause is page lifetime within a single node; earlier fixes of that kind include #24226, #30899 and #30937. The other is a race across nodes: when a request is aborted while prefill is still writing KV cache to decode over RDMA, writes already sent cannot be recalled, and if decode frees the pages at once, late writes can land in another request's cache. Cloudflare has contributed proposed fixes (#31357, #38961) and a reproduction script (#32564). In both cases the result is corrupted output, or one user's data in another user's response, with no error.

Key facts

  • From August 27 to September 9, 2026, 155 GLM-5.3 responses on Workers AI, all to prompts longer than 681,280 tokens, contained data from other customers' requests.
  • Cause one: GLM 5.3 was deployed with a 1,048,576-token context window, but the prefill KV cache pool held only 681,280 tokens. Cloudflare knew of the mismatch before launch.
  • Cause two: in SGLang's disaggregated prefill/decode mode, a rejected over-length prompt was treated as live, so decode read stale KV pages from earlier requests and returned HTTP 200.
  • Cloudflare took the model offline within 40 minutes of declaring an incident, restored it on September 9 and found no evidence of deliberate exploitation; no other Workers AI model was affected.
  • Cloudflare recommends SGLang operators update to v0.5.22 or later; the first patch shipped in v0.5.20.

Why it matters

This is a cross-tenant leak in a hosted LLM, caused by KV cache memory pages that were reused without being cleared. Cloudflare says the pages are not zeroed between requests, which is common among inference engines and needed for best performance, so the engine has to track who owns each page. The incident shows how that duty can fail quietly: the affected requests got HTTP 200 and coherent answers, not errors. Cloudflare also says this bug is one of a class of KV cache memory-safety bugs it has been working on with SGLang maintainers for several months, including a separate cross-node race on aborted requests that can likewise put one user's data in another user's response.

Who it affects

Customers who sent prompts longer than 681,280 tokens to @cf/zai-org/glm-5.3 on Workers AI between August 27 and September 9, 2026, and the other customers whose requests made shortly before theirs ended up in those responses. Cloudflare says a small number of customers received the 155 affected responses. No other model on Workers AI was affected, because every other disaggregated deployment has KV pools larger than its context length on both prefill and decode. Operators who run SGLang themselves, especially in disaggregated prefill/decode mode, are also in scope.

How to use it

Operators who depend on SGLang should update to v0.5.22 or later, as Cloudflare strongly recommends; the patch first shipped in v0.5.20 with follow-ups in v0.5.21 and v0.5.22. Cloudflare's own lessons are practical: compare the KV pool size (printed in SGLang's startup logs and exposed as a metric) with the model's context length from the running engine before any disaggregated deployment change, rather than inferring it from response codes, and test model output and coherence at long contexts, not only HTTP status and performance. A pool sized at roughly twice the context length, as Cloudflare now runs for GLM 5.3, means a valid prompt always fits.

How solid is it

This is a first-party post-incident report from the operator, published as a coordinated disclosure alongside a post from SGLang, with technical detail down to the code path, version numbers and issue and PR numbers. The researcher is named as Gadi Evron from Knostic. The finding of no deliberate exploitation is Cloudflare's own conclusion from excluding its testing and the researcher's and looking at timing and volume of the remaining requests. The report as available to us is cut off before its timeline table, so the detailed response timeline is not covered here.

Risks and caveats

The report describes a second, related risk: a cross-node race on aborted requests, with proposed fixes (#31357, #38961) and a reproduction script (#32564) from Cloudflare. Cloudflare admits a testing gap: it knew about the pool and context mismatch before launch, assumed the engine would handle it, and validated only HTTP response codes and performance. The report does not say what kind of data leaked in the 155 responses. The same bug had been reported to SGLang as issue #30233 on July 6, 2026, well before the exposure window, and Cloudflare notes it had been flagging this type of issue with SGLang since v0.5.10.

“We found no evidence of deliberate exploitation.”

— Cloudflare, post-incident report