Cloudflare's GLM-5.3 KV Cache Exposure: 155 Cross-Tenant Responses and the SGLang Fix Disaggregated Inference Operators Should Apply
Cloudflare says 155 GLM-5.3 responses on Workers AI carried other tenants' data after a KV pool misconfiguration met an SGLang rejection-path bug. What broke, the v0.5.22 fix, and the checks disaggregated inference operators should run.
Cloudflare has disclosed a cross-tenant data exposure in its Workers AI hosting of GLM-5.3, Z.ai’s model for agentic coding workflows. Between August 27 and September 9, 2026, a small number of customers calling @cf/zai-org/glm-5.3 received responses containing data from other customers’ requests made to the same model shortly before theirs. In total, 155 responses were affected, every one a prompt longer than 681,280 tokens, and Cloudflare found no evidence of deliberate exploitation. The model was patched and restored on September 9; the full write-up went public on October 26 alongside a companion post from the SGLang maintainers at RadixArk.
For anyone running or renting hosted inference for coding agents, the takeaway is blunt: in disaggregated prefill/decode serving, the KV cache pool is a tenant isolation boundary, and nothing in the default stack enforces it for you.
Two failures had to line up
A silent config mismatch. GLM 5.3 launched with its full 1,048,576-token context window, but the prefill GPUs’ KV pool held only 681,280 tokens. Pool size is not a configuration value: SGLang, the LMSYS-hosted open-source inference engine, computes it at startup from GPU memory left after the weights load, and nothing forces agreement with the advertised context length. Cloudflare knew of the gap and expected the engine to error out on over-length prompts; pre-launch tests showed clean status codes and streams, so the team shipped without validating output coherence.
A rejection path that leaked. When a prompt exceeded the pool, SGLang’s scheduler rejected it, recorded the client error, and swapped in a one-token placeholder. On one node that ends in a clean HTTP 400. In disaggregated mode, admission queues checked whether a request was finished, not whether it was rejected, so the rejected request read as live. Prefill ran a trivial forward pass, sent one KV page to decode, and signaled transfer complete. Decode, holding reservations for the full prompt, attended over stale pages still holding earlier requests’ data. The client got HTTP 200, a coherent reply, and fragments of someone else’s session.
The numbers and the clock
| Fact | Value |
|---|---|
| Affected model | @cf/zai-org/glm-5.3 on Workers AI |
| Exposure window | Aug 27 to Sep 9, 2026 |
| Affected responses | 155, all prompts over 681,280 tokens |
| Context window vs. KV pool | 1,048,576 vs. 681,280 tokens |
| Report to restore | About 20.5 hours |
Gadi Evron of Knostic reported it through HackerOne on September 8. Cloudflare declared an incident at 13:22 UTC on September 9, reproduced the leak with two accounts within 25 minutes, set the model private at 14:00, rolled corrected configuration to every prefill node by 14:52, and restored public access at 19:40, validated against the researcher’s script and canary tests.
The fix, on both sides
Configuration: the prefill KV pool now holds about 2.04 million tokens, roughly twice the context length, so valid prompts never reach the rejection path.
Engine: a request carrying a pending rejection is retired at the disaggregation admission queue, with its original error, before any bootstrap, allocation, or KV transfer. Cloudflare proposed this as #38704; SGLang merged the related #36988 on September 6 and #38935 to reject at admission, the same behavior another user had reported as issue #30233 back in July. Fixes shipped in v0.5.20, with follow-ups in v0.5.21 and v0.5.22 (October 19). GLM 5.3 was the only Workers AI model affected; other disaggregated deployments run pools larger than context length.
What operators should do now
Cloudflare’s advice for anyone running disaggregated SGLang:
- Measure, don’t assume. Compare KV pool to context length on every prefill and decode node, from the startup log, after any change to weights, quantization, parallelism, or memory fraction. If the pool is smaller, lower the context length; don’t rely on a proxy to enforce it.
- Upgrade to SGLang v0.5.22 or later so over-length requests are rejected before handoff regardless of pool size.
- Run deferred KV release on both roles and bound Mooncake transfer batches.
- Enable corruption detection where supported (
--disaggregation-enable-kv-checksum,--kv-canary) and confirm in logs that verification runs. - Stop unnecessary aborts: let requests proceed to the first token rather than disconnecting mid-transfer.
Why it matters beyond one model
API keys and quotas worked as intended; the leak sat a layer below, in shared GPU memory management, so vendor reviews of hosted model endpoints should ask how KV page ownership is tracked. Long-context agents hit the tripwire first: only prompts beyond 681,280 tokens triggered it, and the outputs looked coherent. The same caching pattern exists in vLLM and local servers, so self-hosters should add the pool-versus-context check to their deploy checklist; our llama-server flags runbook covers the local equivalent. Cloudflare’s longer-term answer is per-page KV tags checked before every decode read, prototyped at under 1% throughput cost, plus formal verification of the RDMA transfer protocol.
GLM-5.3 remains a leading agentic coding model, also on Amazon Bedrock; the lesson is about the serving layer. Background: the GLM-5.3 release and Workers AI.
Sources: Cloudflare Blog · SGLang on GitHub · Workers AI docs
