Skip to content

Cloudflare: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and remediation with SGLang

Summary

On 2026-08-27, Cloudflare launched the GLM-5.3 text-generation model (@cf/zai-org/glm-5.3) on Workers AI configured with a 1,048,576-token context window, while the KV-cache pool on its prefill GPUs could hold only 681,280 tokens — a mismatch Cloudflare knew about but believed was handled gracefully. A security researcher (Gadi Evron, Knostic) reported via HackerOne on 2026-09-08 that very long prompts to the model sometimes returned other users' session data. The root cause was a two-part failure: (1) a misconfiguration (prefill KV pool smaller than the advertised context length) that let over-length requests reach a rejection path, and (2) a bug in SGLang's disaggregated prefill/decode mode where a rejected over-length request was not aborted — instead it ran a one-token placeholder through prefill, signalled "transfer complete" to the decode node, and the decode node then generated a reply while attending over stale KV-cache pages still holding earlier requests' prompts and generations. 155 responses (all to prompts > 681,280 tokens) leaked cross-tenant data between 08-27 and 09-09; Cloudflare took the model offline within 40 minutes of declaring an incident, restored it the same day with a corrected config + a patched engine, and published this as a coordinated disclosure with the SGLang maintainers (RadixArk). GLM 5.3 was the only affected model.

Key takeaways

  1. The pool size is not a config value — and nothing forces it to agree with the advertised context length. SGLang sizes the KV pool at startup from whatever GPU memory remains after loading weights; the model's context length is a separate setting. "Nothing forces the two to agree, and it is our job to check that they do." On GLM 5.3's prefill nodes the pool came to 681,280 tokens vs a 1,048,576-token context. Any change to quantization, tensor parallelism, CUDA-graph sizes, or model version silently moves the pool size — making this a fragile invariant to rely on. (Source: sources/2026-10-26-cloudflare-cross-request-kv-cache-exposure-glm-53-sglang)

  2. The testing gap was validating HTTP status, not output coherence. Pre-launch long-context tests saw 200 OK + a streaming body with no error frames, so Cloudflare assumed the engine handled the over-length case gracefully. It did not — it returned coherent but contaminated output. Remediation: "check KV pool size against context length directly, from the running engine, before any disaggregated deployment change ships, rather than inferring it from response codes."

  3. The bug: a rejected request reported itself as live. SGLang's rejection path (Req.set_finish_with_abort) recorded the error in to_finish (not finished_reason) and swapped the prompt for a one-token placeholder so it could drain through the batch machinery. On a single node this worked. In disaggregated prefill/decode mode, the admission queues on both prefill and decode checked req.finished() (still False for a rejected request), so the rejected request was admitted on both sides. Decode had reserved pages for the full prompt length (e.g. 8 pages); prefill wrote only page P0 (the placeholder) and signalled transfer-complete; decode generated while reading pages P1–P7, which still held earlier requests' KV data. Result: HTTP 200 (not 400), prompt_tokens matching the long prompt, a coherent reply arriving faster than a real 8-page prefill should take, containing other users' content. Every request over the threshold behaved this way — deterministically, not by timing accident.

  4. Two independent fixes, belt-and-suspenders. (a) Config: matched GLM 5.3's config to GLM 5.2, freeing enough memory that the prefill KV pool now holds ~2.04M tokens (≈2× the 1,048,576 context) — a valid prompt always fits, so the rejection path is never reached. (b) Engine: patched SGLang so a request carrying a pending rejection is retired at the disaggregation admission queue, with its original error, before any bootstrap, allocation, or KV transfer (Cloudflare draft PR #38704; the decode node also verifies that tokens-transferred matches tokens-reserved). The fix was already on main as #36988 (merged 09-06 for a related path, unreleased); maintainers merged #38935 to reject these requests even earlier, following Cloudflare's approach.

  5. This belongs to a broader class of KV-cache memory-safety bugs. Cloudflare frames it as "a page is freed, reused, or read while something else still has a claim on it" — a distributed, cross-node analogue of a use-after-free. Two common causes: page-lifetime races within one node (token generation, host-memory offload, KV-transfer receiver, and abort/error paths all holding the same page — prior fixes #24226, #30899, #30937) and cross-node races (a request aborted mid-RDMA-write: the already-sent writes can't be recalled, and if decode frees the pages immediately the late writes land in another request's cache — #31357, #38961, repro #32564). This incident was different — a rejected request went through the handoff every time — but had the same symptom: decode reading pages this request never wrote.

  6. Hardening deployed fleet-wide. Extended deferred KV release to hold prefill's source pages too and cover every failure type (not just aborts); bounded Mooncake transfer batches so one timed-out call can't leave an unbounded tail of in-flight writes; changed the load balancer so a client disconnect no longer aborts a request mid-transfer (holds prefill until it finishes, decode until first token); and added a pre-deploy check to every disaggregated deployment (KV pool vs context length, deferred release on both roles, bounded transfer batches, all required patches). Also shipped --disaggregation-enable-kv-checksum (decode verifies a checksum of the transferred KV before generating, #39500) and fixes to SGLang's KV canary corruption detector.

  7. Pages are not zeroed between requests — by design, for performance. The KV cache is a shared pool of fixed-size pages; a finished request releases pages and the next request simply overwrites them. "This approach is common among inference engines and is necessary to achieve the best performance," which is exactly why the engine is responsible for tracking page ownership and lifetime — including ownership technically shared with the remote prefill node (which holds a simultaneous mutable reference over RDMA). A draft to zero KV pages on release exists (#46435).

  8. The durable answer is integrity by construction / by mathematical impossibility. Short term: configuration checks + lifecycle fixes reduce the ways a page can be misused but don't make misuse impossible. Cloudflare's longer-term direction: (a) per-page tags checked at read time — every physical KV page carries a tag that changes on reallocation, each request records the tags it expects, and the mapping is validated before decode reads the cache (prototype cost <1% throughput/tail-latency; tested on GLM 5.2 mid-August, disabled for reliability; the per-transfer checksum is a first step, page-tags check every read, not just the handoff). (b) Formal verification of the RDMA-based KV-transfer protocol — model production SGLang + Mooncake, prove the safety properties, track and fix divergence; the same formal-methods muscle Cloudflare applied to Topaz (DNS) and Meerkat/QuePaxa. "This bug is an example of a consistency failure in a distributed system, in which the two machines had inconsistent views of the state of that system."

Systems / concepts / patterns extracted

  • Primary subject — SGLang (the open-source inference engine Cloudflare runs): disaggregated prefill/decode admission-queue bug; finished()-vs-rejected check; placeholder-prompt handling; KV-page lifetime and ownership; deferred KV release; per-transfer KV checksum; KV canary. This is the first wiki source that describes SGLang's disaggregated-mode internals in substance (heavy enrich of the stub).
  • Workers AI — the platform serving GLM 5.3 in a PD-disaggregated SGLang configuration (prefill GPU pool → RDMA/Mooncake → decode GPU pool). New incident/remediation section.
  • Mooncake Transfer Engine — the RDMA KV-transfer substrate between prefill and decode; the hardening bounded its transfer batches so one timed-out call can't leave unbounded in-flight writes.
  • Tenant isolation — the property that failed: one user's KV pages leaked into another user's response. A GPU-memory / inference-serving instance of cross-tenant data exposure.
  • KV cache — the shared, fixed-size-page, not-zeroed- between-requests pool whose ownership/lifetime mismanagement caused the leak.
  • Memory safety — the bug class, generalized to a distributed setting: a page read while another request's data still occupies it (a cross-node use-after-write / stale-page-read).
  • Fail-open vs fail-closed — the rejection path failed open (returned 200 with contaminated data instead of 400); the fix makes it fail closed at the admission queue; the corrected config makes the pool always large enough so the path is never reached.
  • Coordinated disclosure — HackerOne report → same-day fix → ~6-week embargo while upstream SGLang patches landed → joint publication with the researcher and RadixArk (SGLang maintainers).
  • Formal verification — the longer-term "integrity by mathematical impossibility" direction: model and prove the RDMA KV-transfer protocol (cited alongside Topaz + Meerkat/QuePaxa).
  • Upstream the fix — Cloudflare authored/co- authored the SGLang admission, checksum, deferred-release, and abort-ACK patches upstream (v0.5.20–v0.5.22) so every operator benefits; recommends all disaggregated-SGLang operators upgrade to v0.5.22+.

Operational numbers

  • Context window configured: 1,048,576 tokens; prefill KV pool at launch: 681,280 tokens (the mismatch). Corrected pool: ~2.04M tokens (≈2× context).
  • 155 responses affected, all to prompts > 681,280 tokens, between 2026-08-27 (launch) and 2026-09-09 (restore).
  • Timeline (UTC): report 09-08 23:11 → verify 09-09 08:00 → incident declared 09-09 13:22 (reproduced with two accounts in 25 min) → model set private 09-09 14:00 (within ~40 min of declaring) → corrected config on all prefill nodes 09-09 14:52 → full rebuild + engine patch 09-09 19:04 → restored 19:40. SGLang v0.5.20 (first fix) → v0.5.21 (10-02) → v0.5.22 (10-19).
  • Per-page integrity-tag prototype: <1% throughput + tail-latency cost.
  • GLM 5.3 was the only affected model; every other disaggregated Workers AI deployment has KV pools larger than its context length on both roles, so no request could reach the rejection path.

Caveats

  • GLM-5.3 is named as the affected model (@cf/zai-org/glm-5.3, a Z.ai text model for agentic coding) but its own architecture is not described — it is the subject of the incident, not of an architecture write-up. Listed in systems: frontmatter; mentioned as plain text, no dedicated page (Systems gate: not substantively architected, single source).
  • RadixArk (SGLang maintainer org), Knostic / Gadi Evron (reporter), Mooncake / Mori / NIXL (transfer backends named in the PR table) are named-in-passing; no pages.
  • No throughput/latency numbers for the Mooncake transfer path itself; the per-transfer-checksum overhead beyond "a first step" is not quantified.

Source

Last updated · 854 distilled / 2,448 read