Skip to content

CLOUDFLARE 2026-10-13

Read original ↗

Rewinding the exponential: How we keep hardware usage down

Summary

Cloudflare's Data team owns the pipeline that plumbs telemetry from the global edge (tens of thousands of servers, 20%+ of the web) to central data centers, feeding Cloudflare Logs, the SQL API, observability, internal analytics, and billing. By May 2026 it carried ~2 billion events/second. The post is a five-year retrospective on fighting the structural mismatch of exponential traffic growth against linear hardware growth — the cumulative optimizations are "equivalent to deferring two and a half years of hardware expansion." It walks five interventions: (2022) breaking up a monolithic message schema into product-specific schemas; (2024) switching compression from LZ4 to zstd; (spring 2026) rearchitecting cross-region redundancy from double-write to a shared ingest pool + late replication; (summer 2026) decentralizing sampling into the edge-side forwarder; and (fall 2026) introducing incremental aggregation at every pipeline stage. The unifying thesis: push data-reduction work as far upstream in the pipeline as possible so each downstream stage carries less, and treat headroom reclaimed — not just bytes saved — as the metric, because organic growth re-consumes savings within months.

Pipeline architecture (background)

The data path, edge → center:

  1. Logfwdr — internal edge-side service that collects events from highly distributed producers (Workers, FL/FL2 proxy) and forwards them to centralized receivers.
  2. Centralized receivers — ingest the forwarded stream, apply sampling, and push onto the buffering layer.
  3. Buftee — Cloudflare's log-based buffering system; downstream consumers fetch from it.
  4. Consumers / inserters — distribute data to final destinations like ClickHouse (used heavily; see Ready-Analytics).

Key takeaways

  • Breaking the monolith (2022) cut backbone bandwidth 35-40%. The single firehose schema packed every product's fields into one stream, forcing internal consumers to ingest everything then demux/discard via filters. Hypothesis: "none of these teams require 100% of fields in the monolithic schema." Replacing it with product-specific schemas — even though some fields are now duplicated across messages — reduced compressed backbone bandwidth 35-40%, "knocking out roughly a year's worth of compressed traffic growth." (Source: sources/2026-10-13-cloudflare-rewinding-the-exponential) An instance of separating concerns paying off as a bandwidth win: send only the fields a consumer needs.

  • LZ4 → zstd (2024) gave ~40% compressed-throughput reduction at negligible CPU cost. Facing daily network-congestion firefighting, the team swapped the compression codec. A prior zstd experiment had shown higher CPU, so they rolled out cautiously (a few servers → whole fleet), tracking CPU the whole way; the feared CPU regression "didn't" materialize. Net: ~40% reduction in compressed throughput; uncompressed volume unchanged, so "we could support more than 1.5x the amount of traffic with the same size pipe" — "about 10 months prior" on the growth curve. Canonical LZ4-vs-zstd production datapoint at data-pipeline-backbone altitude. (Version 2024.6.2 pre-change, 2024.7.1 post-change.)

  • Shared ingest pool + late replication (spring 2026) saved ~36 GB/s net across WANs. For disaster resilience, non-regional analytics must live in multiple central regions. The old model double-wrote: Logfwdr sent two full copies of every non-regional event — one to each region — and each region independently processed a complete copy through to ClickHouse. As resilience expanded to nearly all analytics streams, "Logfwdr traffic doubled across the board" — unsustainable. The fix decoupled pipeline transit from database redundancy: (1) shared ingest pool — adopt Logpush's model so Logfwdr sends each event exactly once to whichever central receiver is available; (2) late replication — replicate as late as possible, after data lands in Buftee, via remote inserters that stream cross-data-center copies downstream; (3) resilient failover — if a region dies, Logfwdr routes all non-regional traffic to the survivor; if the inter-DC link blips, each cluster keeps ingesting its local share without backpressure and remote inserters catch up from Buftee's retention buffer once the backbone recovers. For migrated datasets: −43 GB/s compressed transit while adding only ~7 GB/s cross-DC replication → net −36 GB/s across WANs. (Source: sources/2026-10-13-cloudflare-rewinding-the-exponential)

  • The asymmetry (43 saved vs 7 added) has two structural causes. (1) Replication happens downstream of sampling — the old model shipped two copies of raw, unsampled events over the WAN; the new one ships a single copy to receivers that immediately sample before buffering, so the replicated stream is already compacted. (2) Better compression through uniform partitioning — at source, data is fragmented across thousands of servers; after centralization, streams are aggregated and partitioned into uniform batches over wider time windows, and this structural uniformity lets compression achieve "dramatically higher compression ratios than Logfwdr transit batches ever could." First-cutover numbers: HTTP analytics ~−20 GB/s (50+%); Ready-Analytics −43% (~28 → ~16 GB/s).

  • Decentralized ("from the get-go") sampling (summer 2026) halved on-the-wire compressed data for top datasets — 25 GB/s, 2+ PB/day. Sampling already ran in centralized receivers; the team pulled it earlier, into Logfwdr itself, while keeping receiver-side sampling too. The algorithm: each dataset has a producer-chosen key; records are shuffled per key to eliminate bias, then for each group a sample of size √(total count) is selected; estimates scale by the sample rate (sample 10 of 100 → rate 10/100 → unbiased count 10×10=100). Sampling twice (once in Logfwdr, once in receivers) still yields a representative sample. Applied to the top 3 datasets, each at least halved compressed data to receivers — 25 GB/s total, >2 PB/day. See the square-root-sampling detail below.

  • Double sampling flattens the "most seen" keys without hurting "least seen" keys. Power-law analysis: double-sampled stored rows grow as ~x^0.319 vs single-sampled control at ~x^0.485. A key with 100× more data than another would store ~10× the rows if single-sampled, but only ~4× if double-sampled. Concretely, 11.6M events → 18k rows in ClickHouse under double sampling vs 79k rows single-sampled — same statistical estimate, far less storage. (x^0.25 was not observed because the two sampling passes happen on separate machines/batches, not on one combined pool.) Validated by comparing ClickHouse row counts against estimated sums across keys spanning least-seen → most-seen.

  • Pipeline-wide incremental aggregation (fall 2026) cut HTTP rollup traffic ~80% (9+ → <2 GB/s). Rollup datasets answer a fixed set of questions exactly (chosen dimensions + metrics up front) — the opposite tradeoff to sampling. Historically aggregation ran only at the tail (inserters reading from Buftee). The team built aggregation that runs at every stage: aggregate on each Logfwdr instance → aggregate again at receivers (pushed onto dedicated partitioned buffers) → aggregate again after inserters batch-read from Buftee → final aggregation in ClickHouse via aggregating table engines. Records group by a key that determines which can combine, so matching partial aggregates merge repeatedly as data converges. On the (highest-cardinality) HTTP dataset alone: 9+ GB/s → <2 GB/s compressed (~80% reduction). Productized as Ready-Aggregates, a generic in-house one-click aggregating analytics product (sibling to Ready-Analytics). An incremental-view-maintenance / streaming-aggregation instance applied along the whole pipeline rather than at a single tier.

  • Headroom, not bytes, is the real metric — organic growth re-eats savings in months. Ready-Analytics (load balancing + sampling): mid-May load balancing dropped it ~26 → ~13 GB/s (50%); two months later organic growth had clawed it back to ~20 GB/s (half the reduction erased); sampling in July then took it to ~8 GB/s. Without both interventions it "would have breached 60 GB/s by September." Processing less in central DCs also saved ~750 CPUs and 3.5 TiB memory (~50% reduction) on the Ready-Analytics receivers alone.

  • Aggregate pipeline-wide result: ~70 GB/s of compressed backbone traffic eliminated spring→fall 2026. Load balancing ~38 GB/s + sampling ~25 GB/s (top datasets) + aggregating ~7+ GB/s so far = ~70 GB/s removed from the backbone; throughput now "roughly eight months behind its previously projected growth trajectory", and the 2026 work halves the projected 2031 data volume.

Operational numbers

Metric Value
Pipeline throughput (May 2026) ~2 billion events/sec
Monolith → product schemas (2022) −35-40% backbone bandwidth (~1 yr growth)
LZ4 → zstd (2024) ~−40% compressed throughput; 1.5× same-pipe capacity (~10 mo)
Shared pool + late replication −43 GB/s transit, +7 GB/s cross-DC = −36 GB/s net
HTTP analytics first cutover ~−20 GB/s (50+%)
Ready-Analytics load-balancing cutover −43% (~28 → ~16 GB/s)
Decentralized sampling (top 3 datasets) ≥50% each; 25 GB/s total; >2 PB/day
Double vs single sampling (11.6M events) 18k rows vs 79k rows in ClickHouse
Power-law growth exponents double ~x^0.319 vs single ~x^0.485
HTTP rollup aggregation 9+ GB/s → <2 GB/s (~80%)
Ready-Analytics receiver savings ~750 CPUs + 3.5 TiB memory (~50%)
Total backbone reduction (spring→fall 2026) ~70 GB/s compressed (~8 months of growth)
Long-term 2026 work halves projected 2031 data volume

Square-root sampling (detail)

The sampling algorithm is a key-grouped, bias-eliminated, square-root-sized sampler:

  1. Each dataset has a producer-chosen key used to index and sample.
  2. Records are shuffled within each key to eliminate bias (Source: Cloudflare's earlier "how we make sense of too much data" post).
  3. For each group of records, select a sample of size √(total count).
  4. To produce estimates, scale by the sample rate — sample 10 of 100 → rate 10/100 → unbiased original count = 10×10 = 100.

Because the sample size is sub-linear in the group count, high-volume ("most seen") keys are aggressively down-weighted in storage while low-volume ("least seen") keys are preserved comparably — the detail per rare event is unharmed. Running the sampler twice (Logfwdr + receivers, on separate machines/batches) compounds the flattening (x^0.319) without the x^0.25 a single combined-pool double sample would give. Recorded here as a named technique (single source); not minted as its own concept page.

Progressive data reduction (the unifying thesis)

Every intervention is the same move at a different pipeline stage: do the data-reduction work as far upstream as possible so each downstream hop carries less.

  • Schema split → send only the fields a consumer needs (producer-side).
  • zstd → fewer bytes per unit of information, everywhere downstream.
  • Sampling moved from receivers → into Logfwdr (earliest possible).
  • Replication moved from source double-write → downstream of sampling (latest possible), inverting where the duplication cost is paid.
  • Aggregation moved from tail-only (inserters) → every stage, with repeated partial-aggregate merges as data converges.

The companion insight is economic: because growth is exponential and hardware linear, a one-time reduction only rewinds the curve by a fixed offset (months). The reductions compound against the long-run projection (2031 volume halved), which is why the team frames the payoff as headroom / growth-equivalent deferral rather than absolute savings.

Caveats

  • The post is a retrospective narrative, not a design doc — algorithms (sampling, aggregation) are described conceptually; exact buffer formats, key-selection APIs, and the aggregating-table-engine config are not disclosed.
  • Ready-Aggregates is named once as the productized aggregation offering; no architecture beyond "generic one-click aggregating analytics product." Recorded as prose, not a system page.
  • Numbers are per-dataset and per-cutover; the ~70 GB/s total is "so far" (aggregation still rolling out).
  • Customer data under Customer Metadata Boundaries is explicitly not cross-region replicated — it stays strictly within its region; the load-balancing/late-replication work applies only to non-regional analytics.

Source

Last updated · 854 distilled / 2,448 read