Skip to content

CLOUDFLARE 2026-11-02 Tier 1

Read original ↗

Cloudflare: How we halved DNS build delay and cut memory use by 85%

Summary

Cloudflare's DNS Zone Builder turns every DNS record or zone-setting change into the key format its authoritative DNS servers read from Quicksilver. It now receives ~700 record changes/s (vs ~250/s in 2022) plus ~100 other events/s, and some zones hold >20 million records. Record events already had cheap per-record builds (2022), but zone setting changes still triggered full builds that read, encode, compare, and hold the whole zone in memory — the largest zones do not fit at all. The post describes a targeted build planner that chooses the smallest build it can prove is safe (five build types, with the invariant that every written key is byte-for-byte identical to what a full build would write), a plan-then-verify step that re-reads a fresh publication snapshot from a replica (with a staleness check) just before execution, and a unified scheduler that merges queued work per zone while tracking Kafka messages separately so soft-committed offsets never skip unfinished work. Validated with byte-for-byte equivalence tests, a temporary side-by-side full build, and a mirror Zone Builder consuming the production Kafka stream into an isolated Quicksilver; rolled out by hash of zone name. Results: 92.95% of builds no longer rebuild the full zone, full builds fell ~40%, peak memory 79 → 12 GiB (−85%), worst p99 build delay 14.6 → 1.21 min, and — by spending the new headroom on shorter resilience waits — median delay 7.7 s → 3.3 s.

Key takeaways

  1. The problem is work-per-event, not event rate. A full build's cost grows with zone size (tens of millions of records), so a single setting toggle on a huge zone dominates. Record events say what changed; zone events can come from many products, may not say why, and can alter data shared across same-name zones — so targeting them is a correctness problem first. (Source: this page)
  2. Smallest provably safe build, never less. Five build types — resources-only, resources+metadata, apex+metadata, purge, full zone. The rule: every key a targeted build writes must be byte-for-byte identical to the full-build output; if that cannot be proved, do a full build. Missing details, unknown events, and timestamp-only "something changed" updates all escalate: "The planner never turns uncertainty into less work." — a fail-safe default in the sense of concepts/fail-open-vs-fail-closed.
  3. Targeted builds are incremental recomputation of a derived view. The Zone Builder derives an edge representation from DB state; per-name rebuilds plus "scan Quicksilver under each rebuilt name and delete keys the new build didn't write" is concepts/incremental-view-maintenance with an explicit stale-key cleanup step.
  4. Shared metadata needs a fresh, whole rebuild. One Quicksilver metadata key per domain name acts as a directory of which internal zone IDs (several can exist during onboarding/offboarding) may answer. When the builder writes it, it rebuilds the whole key from a fresh read instead of patching an older copy, and writes the zone's records before the metadata that points to them.
  5. Plan from the event, verify before writing. The event yields a conservative work list; immediately before execution a fresh publication snapshot can upgrade it (→ full), turn it into a purge, skip it, or rebuild the shared key. Reads come from DB replicas; because Kafka delivery and replication move independently, the DNS Kafka Aggregator delays events to cover normal lag and a final DB check rejects reads from a replica that is still behind.
  6. Merge only adds work. The unified scheduler (replacing separate record and full-zone schedulers) unions queued work per zone; any full-zone need makes the merged build full; a purge overrides every rebuild. DNS work and pending Kafka messages are tracked separately so retries preserve both and messages stay attached until the result is safely written.
  7. Soft commit trades repeated work for no lost changes. Builds finish out of order and are marked done in memory, but the Kafka committed offset only advances past contiguous completed messages. One slow build blocks commits for its partition (concepts/head-of-line-blocking); a restart replays everything after it (concepts/at-least-once-delivery) — acceptable because builds converge on DB state (concepts/idempotent-operations).
  8. Fairness via a per-zone rebuild interval. After a zone builds, its next build waits a plan-dependent interval and absorbs changes arriving meanwhile — so one busy zone cannot take every worker (concepts/noisy-neighbor) — but old queued work is let through before it blocks a partition too long.
  9. A deliberate large-zone exception. Above a fixed record-count threshold the builder never escalates to full: a full build cannot finish (it holds every DB record and every existing Quicksilver key in memory) and, while retrying, stalls its whole Kafka partition. Instead it rebuilds apex, metadata, and every name the queued events identify; zone holds reduce the same-name risk, and rare remaining cases get manual per-name rebuilds.
  10. Verify a cheaper path by running it beside the expensive one. Equivalence tests (full vs targeted from the same DB state, time pinned, byte compare), a temporary full-build-beside-every-targeted-build in production, then a mirror Zone Builder on the production Kafka stream writing to an isolated Quicksilver with a comparer sampling 20% of zones (patterns/shadow-migration). Rollout by hash of zone name — not zone ID — so all same-name zones share a scheduler and two schedulers never write the same metadata key; free zones first, then paid (patterns/staged-rollout).

Pipeline (as described)

  1. Customer changes a record (DNS Records API / dashboard) or another product changes a zone setting.
  2. Change stored in a database.
  3. DB change (or another system) produces a Kafka event (concepts/change-data-capture-shaped).
  4. DNS Kafka Aggregator merges events from several topics and forwards to the Zone Builder (and delays them to cover replica lag).
  5. Zone Builder reads zone data, combines records + settings, applies DNS serving rules, encodes keys for authoritative DNS.
  6. Diffs against Quicksilver, writes only changed keys; Quicksilver propagates globally. Each Zone Builder deployment writes to one network (an isolated Quicksilver copy per environment, e.g. global network vs China Network).

Operational numbers

Metric Before After
Record changes received ~250/s (2022) ~700/s + ~100 other events/s
Largest zones — >20M records, heading to 100s of millions
Builds avoiding full-zone rebuild (24 h) — 92.95% (resources-only 86.81%, res+meta 2.54%, apex+meta 2.30%, purge 1.30%)
Full builds share 11.88% 7.04% (~40% fewer); remaining avg 0.35 s
Kafka consumption bursts 300–1,550 msg/s vs ~600–800 produced tracks production ~600–800 msg/s
Worst p99 build-completion delay (24 h) 14.6 min 1.21 min (12×)
Avg working-set memory 49 GiB 7 GiB (−86%)
Peak working-set memory 79 GiB 12 GiB (−85%)
Median build-completion delay ~7.7 s ~3.3 s (after shorter rebuild interval + smaller aggregator delay)
p90 / p99 (native histogram) — ~5.5 s / ~12 s
Mirror diff rate — 0.31% across 234,313 matched zones (mostly timing)
2022 per-record vs full build 150× faster avg; 4,250× on a 1M-record zone —

Systems / concepts extracted

  • systems/cloudflare-dns-zone-builder — primary subject (new page).
  • DNS Kafka Aggregator — sub-component; documented as a section on the Zone Builder page.
  • systems/quicksilver — target KV store; record keys per zone+name, one shared metadata key per domain name; per-environment "networks".
  • systems/kafka — ordered partitions, committed offset, soft commit.
  • systems/cloudflare-universal-ssl — example zone setting whose toggle changes apex CAA records + metadata (apex+metadata build).
  • Cloudflare China Network — a separate network a zone can also be published to; named in passing (listed in systems: only, no page).
  • systems/cloudflare-workers, systems/cloudflare-durable-objects, systems/cloudflare-access — host the internal DNS Mirror Dashboard.
  • systems/prometheus — fixed-bucket histograms overstated p90/p99; switched to native histograms.
  • Tag-only (single-source named ideas, not minted): targeted build planner, plan-then-verify publication snapshot, merge-only-adds-work scheduler, per-zone rebuild interval, large-zone no-escalation threshold, hash-of-name rollout for shared-key ownership, histogram-bucket interpolation bias.

Caveats

  • Delay figures combine all build types and plans and exclude Quicksilver's own global distribution; not a per-zone guarantee. A large share of the delay is the deliberate rebuild interval, not build work.
  • The 14.6 → 1.21 min comparison is production (old schedulers) vs mirror (unified scheduler) over the same 24 h of events — controlled, but the mirror serves no traffic.
  • Early p90/p99 values were biased high by coarse 1/5/10/30 s buckets with linear interpolation; the exact average confirmed the median halving.
  • The large-zone no-escalation rule is an acknowledged correctness trade-off (mitigated by zone holds + manual rebuilds), not a proof.
  • Full builds are not going away: first publication and zone-wide changes still need them (the 2022 goal of eliminating them was dropped).

Source

Last updated · 854 distilled / 2,448 read