Cloudflare Says a KV-Cache Bug Leaked Other Users' Data in 155 GLM-5.3 Responses

TL;DR

Cloudflare says that between August 27 and September 9, 155 responses from the GLM-5.3 endpoint on Workers AI contained data from other customers' requests. Two faults overlapped: a prefill KV cache pool that held 681,280 tokens behind a context window advertised at 1,048,576, and a bug in SGLang's disaggregated prefill/decode path that let a rejected prompt through as a one-token stub and returned HTTP 200. SGLang's coordinated post discloses a second, wider race in the same code path that affected default prefill/decode deployments from the v0.4.x line until v0.5.21. If you run SGLang with Mooncake, NIXL or MORI, upgrade and measure your pool.

What leaked, and to whom

The incident report, written by Cloudflare's Kevin Flansburg and Michelle Chen and published on October 26 alongside a post from the SGLang team, is specific about scope. The affected endpoint is @cf/zai-org/glm-5.3, Z.ai's agentic coding model, served with a 1,048,576-token context window. The exposure ran from the model's launch on August 27 until Cloudflare pulled it on September 9. In those 13 days, 155 responses carried fragments of other users' sessions, all of them replies to prompts longer than 681,280 tokens.

Cloudflare says no other Workers AI model was affected, and that the affected requests look like ordinary long-context traffic rather than someone hammering the bug. The leak runs one way: the user who sent the oversized prompt received someone else's cached context in their reply. If you sent a normal prompt to GLM-5.3 in that window, the risk is that a slice of your context landed in a stranger's output.

Issue one: a pool 35% too small

SGLang does not let you set the size of the prefill KV cache pool. It computes it at startup from whatever GPU memory is left after the weights load. The model's context length is a separate configuration value, and as Cloudflare puts it, "nothing forces the two to agree, and it is our job to check that they do." On the GLM-5.3 prefill nodes the pool came to 681,280 tokens while the advertised context was 1,048,576.

The uncomfortable part of the write-up is that Cloudflare knew. "We were aware of this mismatch before launch," the post says. The team expected an error for any prompt the pool could not hold, and long-context tests came back with clean status codes, so they shipped. Their own diagnosis: "we only validated HTTP response codes and engine performance at long contexts, without validating model output and coherence."

The fix was mundane. Cloudflare copied its GLM-5.2 configuration, which freed enough GPU memory for the prefill pool to hold about 2.04 million tokens, roughly twice the context window, so a valid prompt can no longer reach the rejection path.

Issue two: a rejection that was not a rejection

On its own, the undersized pool should have produced errors, and on a single node it does. When a prompt exceeds the pool, SGLang's scheduler calls Req.set_finish_with_abort, which stores an error for the client and swaps the prompt for a one-token placeholder. A single-node server sends the error back and moves on.

Workers AI does not run single nodes for this model. It runs SGLang in a disaggregated configuration: a pool of prefill GPUs processes the prompt and builds the KV cache, then ships it over RDMA to a separate pool of decode GPUs using Mooncake. The admission queues on that path checked whether a request was finished, not whether it had been rejected. A rejected request still looked live, so it kept going.

The sequence Cloudflare reconstructs, using an eight-page example:

• The router sends the original prompt length to the decode side, which reserves eight KV pages for it.
• Prefill admits the rejected request and runs a forward pass on the one-token placeholder.
• One page of KV is transferred to decode, followed by a "transfer complete" signal.
• Decode admits the handoff through the same flawed check and attends over all eight pages. Only page zero belongs to this request; pages one through seven hold whatever earlier requests left there.
• The client gets HTTP 200, a prompt_tokens count that matches what it sent, and a coherent reply built partly from other people's context. The serving stack's equivalent of a straight face.

Picture a hotel that books you eight rooms, hands over eight keys, and only turns over room one. The other seven still hold the last guests' luggage, and the receipt says every room is yours. SGLang, like most serving engines, does not zero KV pages between requests, because the next owner is expected to overwrite them.

The bug was not new. Another user had filed it as SGLang issue #30233 on July 6, seven weeks before the GLM-5.3 launch, with a root-cause walkthrough that matches Cloudflare's almost line for line. Cloudflare found it while investigating its own incident.

The response timeline

Gadi Evron of Knostic filed the report through Cloudflare's HackerOne program at 23:11 UTC on September 8. Cloudflare began verifying at 08:00 the next morning, declared an incident at 13:22, and reproduced the leak with two independent accounts within 25 minutes. At 14:00 the model went private and started returning HTTP 400. The model was public again at 19:40, after a full rebuild with the engine patch. Cloudflare's patch, which retires any request carrying a pending rejection at the admission queue before any bootstrap or KV transfer, went upstream as PR #38704; the maintainers merged their own version of the same idea as PR #38935 on September 16.

SGLang's half of the disclosure is bigger than Cloudflare's

The SGLang post, from the maintainers at RadixArk, covers the admission bug and then goes further. In prefill/decode serving, KV cache moves between nodes as asynchronous one-sided RDMA writes through Mooncake, NIXL or MORI. When a request aborted mid-transfer, the decode side freed its slots while writes were still in flight. The allocator handed those slots to a new request, the late writes landed, and that request's cache was silently corrupted, with no error and no log line. Launched RDMA work cannot be cancelled, and longer contexts widen the window. SGLang's post says operators have no reliable way to tell after the fact whether this already happened on their hardware.

An RDMA write is a letter already in the mail. You cannot recall it, so the only safe move is to keep the mailbox empty until you know it has arrived. That is the fix: deferred decode-side KV release, which holds the slots until every prefill rank acknowledges the abort or a 30-second timeout fires. It shipped as opt-in, then became the default in v0.5.21 (PR #41023, merged September 24).

The affected range is wide: default disaggregated configurations with an RDMA transfer backend from the v0.4.x line in early 2025 until v0.5.21, or v0.5.22 for MORI. Single-instance deployments without cross-engine transfer are not affected. Exploitation is probabilistic: long prompts and mid-transfer disconnects raise the odds, but an attacker cannot pick the victim. The fixes are not backported, so the upgrade is the fix. SGLang measured the cost of the default-on hold at -0.38% throughput on two H200s, inside a confidence interval that spans zero.

What to do if you run SGLang

Both posts end in operator checklists, and they agree. Read the KV pool size from the engine's startup log on every prefill and decode node and compare it to your configured context length, after any change to weights, quantization, parallelism or memory fraction; if the pool is smaller, lower the context length. Upgrade to v0.5.22 or later, which carries #36988 and #38935 plus the deferred-release defaults. Run a router that pairs aborts across both halves, such as sgl-router in PD mode, and bound Mooncake transfer batches so one call cannot outlast the release timeout. Turn on --disaggregation-enable-kv-checksum and --kv-canary where your architecture supports them, and watch sglang:decode_deferred_kv_release_total: a healthy deployment is mostly drained, and a timeout-heavy ratio means your ack path is broken.

On v0.5.19 or v0.5.20 with no upgrade window, SGLang says to set SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1 on both sides, and on v0.5.20 also cap the transfer batch and set the release timeout to 60 to 120 seconds. It adds that the knobs do not replace an upgrade. A public reproduction gist from maintainer Shangming Cai lets you check your own deployment.

The check that would have caught it was switched off

On August 3, Cloudflare published a post describing a page-tagging scheme for exactly this failure: every physical KV page carries a tag that changes when the page is reallocated, each request records the tags it expects, and the mapping is checked before decode reads the cache. A mismatch aborts the request instead of returning the wrong page, at a measured cost under 1% in throughput and tail latency. Cloudflare tested it on GLM-5.2 in mid-August and disabled it shortly afterwards because of reliability issues. GLM-5.3 launched on August 27 without it. Reliability, it turns out, is a word with more than one meaning. Cloudflare says it is building a production version with SGLang and plans to formally model its RDMA KV transfer protocol.

The Hacker News thread split three ways: praise for the root-cause analysis, alarm at "we were aware of this mismatch before launch," and an objection to naming GLM-5.3 in the title. Flansburg replied that the post names the Workers AI endpoint because customers need to know which one was affected, and never claims a fault in the model.

Caveats

The 155 count, the token threshold and the claim that no other model was affected all come from Cloudflare's own investigation. Other operators running disaggregated serving on the affected versions have no equivalent accounting, and SGLang says none is possible after the fact. Evron's own report on the discovery had not been published when the two posts went live. The under-1% page-tag overhead comes from an August benchmark on a mid-sized model at concurrency 1 to 8, not a 1M-context coding workload.

Key Takeaways

• 155 responses from Cloudflare's GLM-5.3 Workers AI endpoint contained other customers' data between August 27 and September 9, all to prompts over 681,280 tokens; Cloudflare found no evidence of deliberate exploitation.
• Two faults overlapped: a prefill KV pool 35% smaller than the context window, which Cloudflare knew about and tested only by status code, and an SGLang admission bug that let a rejected prompt reach decode and read seven stale pages.
• The admission bug had been filed as SGLang issue #30233 on July 6, seven weeks before launch; the fix merged as PR #38935 on September 16.
• SGLang's coordinated post discloses a separate aborted-transfer race affecting default Mooncake, NIXL and MORI disaggregated deployments from v0.4.x until v0.5.21 (v0.5.22 for MORI); fixes are not backported.
• Operators should read KV pool size against context length from the running engine, upgrade to v0.5.22, run a router that pairs aborts, and enable the checksum and canary checks.
• Cloudflare had a page-tag integrity check built and measured in August, then disabled it two weeks before the launch that needed it.

Sources: Cloudflare incident report: cross-request KV cache exposure in the GLM-5.3 Workers AI model, SGLang security update: improving request isolation (LMSYS), SGLang issue #30233, SGLang PR #38935, SGLang PR #38704, SGLang PR #41023, SGLang v0.5.22 release, Cloudflare, Smaller, faster, safer (August 2026), Workers AI GLM-5.3 model page, Hacker News discussion