Skip to content

CLOUDFLARE 2026-10-14 Tier 1

Read original ↗

Inside Hyperdrive: How we solved a rare Hyperdrive MySQL caching bug

Summary

Cloudflare brought Hyperdrive support for MySQL to GA — framed as "20 bug fixes in a trenchcoat" over the beta — and uses the post to walk through one deep, rare bug. Hyperdrive is a distributed database connection pooler + query cache for Workers: it holds warm SQL connections open on Cloudflare's network and can cache common query results in the data-center CDN cache. The headline bug: the highest-volume MySQL customer saw a ~0.001% rate of empty query responses where a non-empty response was always expected, occurring only on Hyperdrive instances with caching enabled. The root cause tied together two facts: (1) Hyperdrive's split Client/Endpoint topology shares a data-center cache when both nodes sit in the same data center, and (2) the MySQL client/server protocol negotiates per-connection capabilities, so the exact result-set bytes depend on which capabilities were negotiated. Hyperdrive translates origin responses encoded with its own fixed capability set into the capabilities the Worker's driver expects. A faulty assumption — carried over from the Postgres implementation — that "we can cache exactly the bytes we wrote back to the Worker" meant a Client could cache Worker-capability bytes that an Endpoint later read back (or vice-versa), serving a response encoded under the wrong capabilities → empty results or hangs. The fix: always write a canonical cache entry encoded with Hyperdrive's own capabilities, regardless of which node stores it. After the change the empty-row problem disappeared and Client-connection success reached the 99.99% availability target.

Key takeaways

  1. Hyperdrive is a two-component (Client + Endpoint) pooler, and the two can share a data-center cache. The Client node runs on the same machine as the Worker, handles local caching, and forwards queries to the Endpoint when needed; the Endpoint node holds the connection pool and sits in the data center closest to the origin DB. Each node uses the Cloudflare CDN cache local to its data center — so if both run in the same data center, they share one cache. Cache keys are a hash of the query's wire messages (normalized) → result bytes. (Source: this article)

  2. The scheme only works if the same bytes are cached for the same response regardless of which node stored it — the exact invariant the bug broke. (Source: this article)

  3. MySQL negotiates per-connection capabilities, and result-set framing depends on them. The protocol's Connection Phase agrees auth method + capability flags; the Command Phase then uses only mutually-supported features, or a protocol desync occurs. This is a textbook protocol capability negotiation. (Source: this article)

  4. CLIENT_DEPRECATE_EOF changes result-set termination — and a popular Node.js driver (mysql2) doesn't support it. Without the capability, the server sends an EOF packet between column definitions and rows, then a second EOF at the end of the result set. With it negotiated, the metadata EOF is omitted and the final EOF is replaced by an OK packet (MySQL 5.7.5+). Hyperdrive negotiates CLIENT_DEPRECATE_EOF toward the origin but must rewrite result sets — reinserting EOFs and translating OK→EOF — when the Worker's driver lacks it. (Source: this article)

  5. Hyperdrive is simultaneously a MySQL server (to the Worker) and a MySQL client (to the origin), with a fixed capability set toward the origin. Because the Worker's capabilities can differ from Hyperdrive's, Hyperdrive translates every origin response from its own capability encoding into the Worker's expected encoding before writing it back. (Source: this article)

  6. The bug: caching the translated (Worker-capability) bytes violated the shared-cache invariant. Carried over from Postgres — whose wire protocol made "cache exactly what we wrote back" safe — the assumption no longer held for MySQL. Two symmetric poisoning scenarios:

  7. Endpoint caches, Client reads: Endpoint caches R_hyperdrive (Hyperdrive-capability bytes); a Client serving the same query finds it in the shared cache and returns it to the Worker without translating → wrong-capability response.

  8. Client caches, Endpoint reads: a Client writes translated R_worker bytes to cache; an Endpoint servicing the same query finds R_worker, sends it to a different Client which tries to translate the already-translated bytes → early EOF or hang. The trigger requires both nodes in the same data center sharing a cache, and one reading a value the other cached. (Source: this article)

  9. The fix is to normalize the cache representation. Always write a canonical result-byte form encoded with Hyperdrive's capabilities, no matter which node stored it; translation to the Worker's capabilities then always happens on the serving hop. Empty-row problem gone; Client-connection success rate reached 99.99%. (Source: this article)

  10. Two earlier, smaller encoding bugs set up the lesson. (a) Hyperdrive initially negotiated utf8 — a deprecated alias for utf8mb3 (1–3 bytes per code point) — which mangled 4-byte emoji; the fix was to negotiate utf8mb4 (full UTF-8). (b) Adding connection cleanup via COM_RESET_CONNECTION (to clear session state between pool checkouts) reset the character set to the server default (e.g. latin1), not the negotiated utf8mb4; the fix is to re-issue the charset-setting command after every connection reset. (Source: this article)

Systems / concepts / patterns extracted

Operational numbers

  • Establishing a SQL connection: minimum five network round trips.
  • Bug incidence: ~0.001% of responses were wrongly empty — only on caching-enabled instances.
  • MySQL CLIENT_DEPRECATE_EOF support: server 5.7.5+.
  • utf8 = utf8mb3 = 1–3 bytes per code point; full UTF-8 (utf8mb4) = up to 4 bytes.
  • Post-fix Client-connection success rate: 99.99% (the stated production availability target).

Caveats

  • This is a GA-announcement + single-bug deep-dive, not a full Hyperdrive internals paper. Pool sizing, per-tenant connection budgets, cache TTL/eviction for the MySQL path, and failure semantics when the origin is unreachable are not covered here. (The broader Hyperdrive cache-key/TTL semantics are disclosed in the 2026-10-13 D1 control-plane post — see sources/2026-10-13-cloudflare-d1-control-plane.)
  • The two symmetric poisoning scenarios reference diagrams in the original post that are not reproduced in the raw markdown.

Source

Last updated · 854 distilled / 2,448 read