CLOUDFLARE 2026-10-13

Under the hood, off the hot path: how D1 scales to millions of databases¶

Summary¶

A behind-the-scenes deep dive into the control plane of Cloudflare D1, Cloudflare's serverless SQLite database, and how it was re-architected to support platforms with millions of created databases, hundreds of thousands active at any moment. The driving workload shift is AI-assisted / "vibe coding" platforms that turn the one-big-cluster model inside out and create one small database per user, app, preview environment, or tenant. The central design question — "how do you scale the control plane for millions of databases without making every query slower by sending it through centralized bottlenecks?" — is answered by a strict control-plane / data-plane separation: keep all query traffic (the data plane) at the edge and off any central single-region component, while letting control-plane operations (create/delete/list, import/export, time travel) depend on a central Postgres cluster reached via Hyperdrive. The post is explicitly pitched as a reusable blueprint for anyone building control + data planes on Cloudflare's Developer Platform.

Growth numbers (as of 2026-10-08)¶

15× growth in live, non-deleted databases over the past year; 65× over 2 years.
18× growth in daily-active databases over the past year (active = executed ≥1 query that day).
Load-tested throughout against a target well above expected growth to find bottlenecks before they hit production.
Named production user: ChatGPT Sites @ OpenAI uses D1 to give its users custom metrics, leaderboards, and comments.

Seven guiding engineering principles (verbatim intent)¶

Design all processes with failure and idempotency in mind; the system self-corrects and continues making progress.
Avoid centralized single-region components in any hot path for data-plane operations (all query requests).
Use Workers KV (globally distributed KV) for critical routing metadata so cache misses only rarely hit central databases.
Use Hyperdrive for central Postgres access — connection pooling, optimized routing + caching, read replicas wherever possible.
Periodically precompute aggregations in the database to avoid scan queries during user request processing.
Use eventual consistency and queuing between components that don't need strong consistency, to raise reliability and availability.
Load-test with realistic workloads to find bottlenecks and optimize those first.

High-level architecture¶

Each D1 database is backed by one SQLite Durable Object (DO) — this is what makes millions of them cheap to create.
The D1 Edge API Worker serves all API surfaces (dashboard, Wrangler, CLI) behind Cloudflare's edge gateway.
Query endpoints (execute SQL) → data plane.
CRUD endpoints (create/delete/list, import/export, time travel) → control plane, which uses Hyperdrive against Postgres.
Two Hyperdrive configurations: one cache-enabled targeting read replicas (few-second TTL), one cache-disabled targeting the primary.
The D1 Eyeball Worker is the proxy for all requests hitting the underlying DOs; it owns query handling, observability, automatic retries, error handling, and its own rate limiting. A user Worker's D1 Binding is a Service Binding to the Eyeball Worker and carries encoded authorization for exactly one database.
Workers KV is the edge caching layer for read-heavy routing metadata (D1 UUID → DO ID + metadata).
Operation taxonomy: (1) data plane — any query, must be fast and always at the edge; (2) control plane — HTTP API ops / internal flows, may depend on central single-region components; (3) background asynchronous plane — analytical / billing workflows, eventually consistent, runs anywhere.

Key takeaways¶

Database creation is forwarded near Postgres, DO placed near the user. Creating a DB does ≥2 Postgres round-trips, so the Edge API Worker forwards the request to a data center near Postgres (a far round-trip is hundreds of ms vs single-digit ms near Postgres). But to keep the SQLite DO close to the user, the DO-init step is itself forwarded back to a data center near the original request location. Endpoints doing ≤1 Postgres query skip this forwarding since Hyperdrive already routes efficiently. (Source: sources/2026-10-13-cloudflare-d1-control-plane)
The "creation checkpoint" is the write to Postgres, not the user response. Steps: check account limits (read replica via Hyperdrive, cached, few-second TTL) → generate DB UUID + a newUniqueId() DO ID → init(options) RPC on the DO stub (DO persists metadata to durable storage) → persist one metadata row to the Postgres primary. Once that row is written the database is considered created even if the final response never reaches the user. The DO-init + Postgres-write are wrapped in a retry loop that uses fresh DB UUID + DO IDs on each retry. A final confirmation RPC to the DO runs in waitUntil() and is allowed to fail; it tells the DO it's active and the DO writes its metadata to Workers KV. (Source: sources/2026-10-13-cloudflare-d1-control-plane)
Why not one big Postgres transaction for creation? It can't scale — connections would be held open while the DO initializes, possibly on the other side of the world. (Source: sources/2026-10-13-cloudflare-d1-control-plane)
Heartbeats make databases self-correcting (anti-entropy). Every D1 DO heartbeats the control plane every 30 s (+ per-database jitter) — tunable per-DB/account, or disablable — reading its latest state (active / deleted) and account limits. Heartbeats tolerate slightly stale data, so they're served from Hyperdrive caches or a Postgres read replica. Because Postgres is the source of truth, an orphaned DO (e.g. created then its Postgres write failed and a retry created a different valid DO) learns within 30 s that "its database does not exist" and marks itself deleted. Fresh IDs per retry are essential: reusing a DB UUID with a new DO ID would make two DOs believe they own one database. The DO is also the serialization point for Workers KV writes (KV has no consistent read), so the best-effort KV update at creation/deletion is eventually repaired by the next heartbeat. The DO stores active/deleted state locally so it can accept/reject queries immediately even if KV is stale. (Source: sources/2026-10-13-cloudflare-d1-control-plane)
The query hot path never touches Postgres. HTTP query flow: Edge API Worker loads D1-UUID → metadata from Workers KV; on the rare KV miss it falls back to Hyperdrive → Postgres read replicas (Hyperdrive checks caller-location cache, then Postgres-location cache, then the replica); then calls the Eyeball Worker over Service Bindings, which validates and forwards to the DO; the DO runs SQL and returns up the chain. Query requests from a user Worker with a D1 Binding skip the KV lookup entirely — the binding already encodes the mapping. (Source: sources/2026-10-13-cloudflare-d1-control-plane)
Workers KV is used as a durable store, not merely a cache, to prevent a metastable failure. "If an item falls out of KV's edge cache, the read is served from KV's own storage rather than Postgres." In a catastrophic cache-purge scenario, "a million databases trying to fetch their routing information would lead to metastable failures" — i.e. a thundering herd onto Postgres. KV's own durable storage keeps Postgres off the data plane even under cache loss. This is why KV is used despite Hyperdrive already having a query cache. (Source: sources/2026-10-13-cloudflare-d1-control-plane)
Why newUniqueId() instead of addressing DOs by name. Name-addressed DOs require a lookup from name → DO ID on every query (a central mapping round-trip); and D1 needs more than the DO ID (metadata to pick the right DO Namespace in the Eyeball Worker), so name-addressing would replace one lookup with two. General advice given: start with name-addressed DOs; only move to unique DO IDs if you must. (Source: sources/2026-10-13-cloudflare-d1-control-plane)
Storage-limit enforcement is async + eventually consistent, with a precomputed per-account total. Each DO publishes storage-usage updates to a Kafka-like queue (aggregated locally per edge server first); a bounded set of queue consumers in Core writes them to Postgres, capping write load so analytics can't overload the DB. A naive SELECT sum(storage) FROM database_stats WHERE account_id = ? scales linearly with per-account database count and doesn't work for accounts with millions of DBs. Instead they maintain a precomputed account_database_stats lookup table via a hand-rolled incremental-view-maintenance scheme (see numbers below). (Source: sources/2026-10-13-cloudflare-d1-control-plane)
The deltas-table trick avoids single-row write contention. Rather than have the consumer update account_database_stats per update (hot single-row lock for big accounts), a third table account_database_stats_changes acts as a log/queue of per-delta rows; a scheduled job every 1 s aggregates all pending deltas per account and applies one summed update. The job uses a DELETE ... RETURNING data-modifying CTE (materialized) to atomically consume pending changes, then GROUP BY account_id and INSERT ... ON CONFLICT DO UPDATE SET storage = storage + EXCLUDED.storage. Totals are eventually consistent (acceptable — a queue already made the pipeline eventually consistent). "This schema design is a pattern used by many other Cloudflare teams." (Source: sources/2026-10-13-cloudflare-d1-control-plane)
Account-keyed cache + dynamic-TTL trick keeps enforcement reads off Postgres. Each DO reads SELECT storage FROM account_database_stats WHERE account_id = ? via Hyperdrive's cache on heartbeat; keying only on account_id (not database_id) means all DBs on an account share one cached value → very high cache-hit rate, most heartbeats never reach Postgres. Deliberate trade-off: including database_id would give exact per-DB headroom and earlier enforcement, but would shatter the cache key and tank hit rate for million-DB accounts. A dynamic-TTL trick shortens Hyperdrive's default 60 s cache: adding $2 = Math.floor(Date.now() / (X*1000)) as a dynamic_ttl parameter changes the cache key every X seconds, forcing fresh reads sooner. (Source: sources/2026-10-13-cloudflare-d1-control-plane)
Row-usage (daily) enforcement uses a different pattern — ClickHouse + Quicksilver, not Postgres. Per-query rows-read/written are published as usage events into a ClickHouse cluster (also the source of truth for row-based billing). A central "Limiter" in Core periodically runs an analytical GROUP BY account_id HAVING reads > ? or writes > ? over today's events (ClickHouse handles this far better than Postgres, and keeps analytics off Postgres). For free-tier accounts over the limit it emails them (deduped) and sets an end-of-day-expiry key in Quicksilver (Cloudflare's globally-distributed KV with sub-second replication; underlies Workers KV Instant). Each DO periodically checks Quicksilver at the edge and rejects queries while limited; an upgrade to paid clears the key via a queue message. (Source: sources/2026-10-13-cloudflare-d1-control-plane)
Right tool per job (summary). Postgres for transactional metadata, ClickHouse for analytics, Workers KV for globally-replicated routing data, Quicksilver for globally-replicated enforcement flags — reducing central regional bottlenecks via caching + read replicas + eventual consistency. (Source: sources/2026-10-13-cloudflare-d1-control-plane)

Operational numbers¶

D1 DB creation example: wrangler d1 create "demo-01" → ~1.6 s total wall clock (0.56 s user / 0.13 s system).
Heartbeat interval: 30 s + per-database jitter (tunable / disablable).
Orphaned-DO self-correction window: within 30 s (next heartbeat).
account_database_stats_changes aggregation job cadence: every 1 s.
Hyperdrive default cache TTL: 60 s; account-limit read uses a few-second TTL via the dynamic_ttl cache-key trick; account-count check at creation uses a few-second cached read replica.
Free-plan storage limit 5 GB/account; Workers Paid 1 TB (raisable on request). Free-tier daily row-execution limit enforced via Quicksilver (expiry at end of day).
Growth: 15×/yr, 65×/2yr live DBs; 18×/yr daily-active DBs (2026-10-08).

Caveats / gaps¶

The post links the companion D1 read-replication deep dive for the actual query-execution + DO internals; those are not re-derived here.
The architecture diagrams referenced in the post ("boxes and arrows") are not reproduced in the scraped markdown; flows are described in prose.
Exact queue technology ("Kafka-like"), consumer count, and Quicksilver internals are not disclosed beyond the behavior described.
The deltas-table / account-aggregation design is described as common across Cloudflare teams but the post does not name the other teams.

Systems¶

Cloudflare D1 — the primary subject: serverless SQLite, one SQLite DO per database, millions of DBs.
Durable Objects — the per-database backing object; newUniqueId(), init() RPC, durable storage, heartbeat origin, Workers-KV write serialization point.
Workers KV — durable edge store for routing metadata (D1 UUID → DO ID), used as a store (not just a cache) to keep Postgres off the data plane under cache purge.
Hyperdrive — connection pooling + query caching + read-replica routing for central Postgres; the dynamic-TTL cache-key trick rides on its query cache.
Postgres — central transactional metadata store (one row per database); source of truth for create/delete; primary + read replicas.
ClickHouse — analytical store for per-query usage events; source of truth for row-based billing; the Limiter's GROUP BY ... HAVING scan runs here, not on Postgres.
Quicksilver — globally-distributed KV with sub-second replication; holds end-of-day row-limit flags checked by DOs at the edge; underlies Workers KV Instant.
SQLite — the per-DO embedded database engine.
Workers — Edge API Worker + Eyeball Worker
user Workers with D1 Bindings (Service Bindings).

Concepts¶

concepts/control-plane-data-plane-separation — the organizing principle; three-way op taxonomy (data / control / background-async).
concepts/eventual-consistency — KV routing, the usage-total pipeline, Quicksilver flags; used wherever strong consistency isn't required.
concepts/idempotent-operations — retry-with-fresh-IDs creation loop; best-effort waitUntil() KV writes repaired by heartbeat.
concepts/hot-path — the data plane (every query) must stay at the edge and off central single-region components.
concepts/incremental-view-maintenance — the precomputed account_database_stats total maintained via a per-delta changes table + 1 s aggregation job (hand-rolled IVM).
concepts/network-round-trip-cost — forward-near-Postgres for multi-round-trip endpoints; account-keyed cache to avoid replica reads.
concepts/cache-hit-rate — account-only cache key chosen deliberately to maximize hit rate across a million databases.
concepts/thundering-herd — the metastable failure averted by making KV a durable store rather than a Postgres-backed cache.
Self-correcting heartbeat / anti-entropy reconciliation (recorded as a mention, not a dedicated page): the DO periodically re-reads the Postgres source of truth and converges local + KV state to it.

Patterns¶

patterns/batch-over-network-to-broker — DOs publish storage-usage deltas to a Kafka-like queue; bounded consumers smooth/batch the write load onto Postgres so analytics can't overload it.

Source¶

Original: https://blog.cloudflare.com/d1-control-plane/
Raw markdown: raw/cloudflare/2026-10-13-under-the-hood-off-the-hot-path-how-d1-scales-to-millions-of-6ec21beb.md