Skip to content

SYSTEM Cited by 1 source

Cloudflare DNS Zone Builder

The DNS Zone Builder is the stage of Cloudflare's DNS propagation pipeline that turns a DNS record or zone-setting change into the keys its authoritative DNS servers read from Quicksilver. A build reads current database state, combines records with zone settings, applies DNS serving rules, encodes the result into edge keys, diffs it against Quicksilver, and writes only changed keys. In 2026 it handled ~700 record changes/s plus ~100 other events/s, on zones up to >20M records (Source: sources/2026-11-02-cloudflare-how-we-halved-dns-build-delay-and-cut-memory-use-by-85).

Pipeline position

DB change → Kafka event → DNS Kafka Aggregator → Zone Builder → Quicksilver → global edge. Each Zone Builder deployment writes to one network — an isolated Quicksilver copy per environment (e.g. global network vs China Network); a zone can be enabled on some networks and not others.

DNS Kafka Aggregator

Sub-component that combines events from several Kafka topics and forwards them to the Zone Builder. It also delays each event long enough to cover normal database-replica lag, because Kafka delivery and replication progress independently (concepts/asynchronous-replication). Once the databases began rejecting reads from lagging replicas, this buffer could shrink — the first step of the 2026 median-delay halving.

Output model in Quicksilver

  • Record keys — DNS answers for one zone + name.
  • Shared metadata key — one per domain name, shared by every internal zone object with that name. Acts as a directory: which zone IDs may answer, and where special records (nameservers, wildcards) live. Several zone IDs can briefly exist for one domain during onboarding/offboarding; a priority policy picks a bounded set to publish.

Build types (targeted build planner, 2026)

  1. Resources only — rebuild specific names; ordinary A/AAAA/TXT/CNAME.
  2. Resources + metadata — names + shared key; nameserver/wildcard changes.
  3. Apex + metadata — e.g. Universal SSL toggles, which change apex CAA records.
  4. Purge — zone should no longer be published on this network.
  5. Full zone — everything, when smaller work cannot be proven safe.

Invariant: every key a targeted build writes is byte-for-byte identical to full-build output; otherwise escalate to full. Uncertainty (timestamp-only updates, missing details, unknown events) always escalates — a fail-safe default (concepts/fail-open-vs-fail-closed). Stale-key cleanup scans Quicksilver under each rebuilt name and deletes keys the new build did not write — incremental recomputation of a derived view (concepts/incremental-view-maintenance).

Large-zone exception: above a record-count threshold, the builder never escalates to full (a full build holds every DB record and every Quicksilver key in memory and cannot finish, stalling its partition). It rebuilds apex, metadata, and every name the queued events identify; zone holds and manual per-name rebuilds cover the residual risk.

Plan, then verify

The event produces a conservative work list. Just before execution the builder reads a fresh publication snapshot from a replica (current settings, whether to publish on this network, which same-name zone IDs belong in the metadata key). This can upgrade to full, convert to purge, skip, or rebuild the shared key from the current candidate set. Shared metadata is always rebuilt whole from the fresh view (never patched), and zone records are written before the metadata that references them.

Unified scheduler

Replaced separate record and full-zone schedulers. The planner decides what; the scheduler decides when. Queued work per zone is merged — merging only adds work (any full → full; purge overrides all). DNS work and pending Kafka messages are tracked separately so retries keep both. A plan-dependent per-zone rebuild interval absorbs bursts and stops one busy zone from taking every worker (concepts/noisy-neighbor), while old queued work may bypass it before it blocks a partition; zone locks and retry delays prevent tight retry loops.

Kafka offset handling (soft commit)

Builds complete out of order in memory, but the committed offset advances only past contiguous completed messages. A slow build blocks commits for its partition (concepts/head-of-line-blocking); a restart replays everything after it (concepts/at-least-once-delivery). Builds converge on DB state, so replay is repeated work, not wrong output (concepts/idempotent-operations). Shorter targeted builds shrink how long the oldest message stays open: before, consumption swung 300–1,550 msg/s in catch-up bursts; after, it tracks the ~600–800 msg/s production rate.

Release safety

  • Equivalence tests: full vs targeted from the same DB state, time pinned, byte-for-byte comparison of every key the targeted plan owns.
  • Temporary production mode: a full build beside every targeted build.
  • Mirror Zone Builder: consumes the production Kafka stream and production DB state, writes to an isolated Quicksilver; a comparer samples 20% of zones for missing/extra/different keys, reported to a DNS Mirror Dashboard built on Workers + Durable Objects behind Access. Watch for new kinds of differences, not transient timing diffs (0.31% diff rate over 234,313 zones) (patterns/shadow-migration).
  • Rollout by hash of zone name (so same-name zones share one scheduler and never race on the metadata key), free plans first, then paid (patterns/staged-rollout).

Results (2026)

  • 92.95% of builds avoid a full-zone rebuild; full builds −~40%; remaining full builds average 0.35 s (mostly first publication of small new zones).
  • Working-set memory avg 49 → 7 GiB, peak 79 → 12 GiB (concepts/working-set-memory).
  • Worst p99 build-completion delay 14.6 → 1.21 min (concepts/tail-latency-at-scale).
  • Median delay ~7.7 s → ~3.3 s after spending headroom on shorter rebuild intervals and a smaller aggregator delay; native Prometheus histograms put p90 ≈ 5.5 s, p99 ≈ 12 s.

History

  • 2022 — per-record builds introduced (150× faster than full on average, 4,250× on a 1M-record zone); goal stated to eliminate full builds.
  • 2026 — targeted planner + unified scheduler for zone-setting events; goal of eliminating full builds dropped as neither realistic nor useful.

Seen in

Last updated · 854 distilled / 2,448 read