Skip to content

SYSTEM Cited by 1 source

Logfwdr

Logfwdr is the edge-side ingestion / forwarding service at the head of Cloudflare's internal Data team pipeline. It collects events from highly distributed producers on the global network — Workers, the FL/FL2 proxy, and other internal sources — and forwards them to a set of centralized receivers in Cloudflare's core data centers, which then push onto the Buftee buffering layer for downstream consumers (ultimately ClickHouse). At May-2026 scale the pipeline carried ~2 billion events/second. (Source: sources/2026-10-13-cloudflare-rewinding-the-exponential)

Logfwdr sits at the widest, most fragmented point of the pipeline: data originates across tens of thousands of servers. That position makes it the highest-leverage place to do data reduction — anything dropped or compacted here never crosses the wide-area backbone. Over 2026 Cloudflare pushed three successive reductions into or around Logfwdr.

Cross-region load balancing — from double-write to a shared ingest pool

For disaster resilience, non-regional analytics must be stored in multiple central regions (data under Customer Metadata Boundaries stays strictly within its region). The original design had Logfwdr double-write: it transmitted two full copies of every non-regional event — one to each central region — and each region processed a complete copy through to ClickHouse. As resilience expanded to nearly all analytics streams, "Logfwdr traffic doubled across the board" — unsustainable against exponential growth.

The redesign (spring 2026) adopts the stateless shared-receiver-pool model already used for Logpush:

  • Shared ingest pool. Logfwdr dispatches each event exactly once across wide-area transit to whichever centralized receiver is available, rather than one copy per region.
  • Resilient failover. If a central region goes down, Logfwdr immediately routes all non-regional traffic to the surviving region. If the inter-DC link blips, each cluster keeps ingesting its local share without backpressure, and remote inserters catch up from Buftee's retention buffer once the backbone recovers.

Database redundancy is decoupled from transit: the second ClickHouse copy is made late, by remote inserters downstream of Buftee (see late replication), not by Logfwdr at source. For migrated datasets the cutover eliminated ~43 GB/s of compressed transit while adding only ~7 GB/s of cross-DC replication — a net −36 GB/s across WANs. HTTP analytics saw ~−20 GB/s (50+%); Ready-Analytics −43% (~28 → ~16 GB/s).

Decentralized sampling — "from the get-go"

Sampling already ran in the centralized receivers; summer 2026 pushed it earlier, into Logfwdr itself (while keeping receiver-side sampling), so less data crosses the wire in the first place. The sampler is key-grouped and square-root-sized:

  1. Each dataset has a producer-chosen key.
  2. Records are shuffled per key to eliminate bias.
  3. For each group, select a sample of size √(total count).
  4. Estimates scale by the sample rate (10 of 100 → 10×10 = 100).

Because the sampler now runs twice (Logfwdr + receivers, on separate machines/batches), the "most seen" keys are doubly flattened in storage (power-law exponent ~x^0.319 vs single-sampled ~x^0.485) while "least seen" keys are preserved comparably — representativeness is kept, storage is cut. Applied to the top 3 datasets, each at least halved compressed data to receivers: 25 GB/s total, >2 PB/day. Downstream, double sampling turned 11.6M events into 18k ClickHouse rows vs 79k single-sampled. (Running on separate batches is also why the exponent is x^0.319 and not the x^0.25 a single combined-pool double sample would give.)

Per-instance aggregation

Fall 2026's pipeline-wide incremental aggregation starts on each Logfwdr instance: records group by a key, and matching partial aggregates are combined on the instance before forwarding. Receivers re-aggregate onto dedicated partitioned buffers, inserters aggregate again after batch-reading Buftee, and ClickHouse does a final pass via aggregating table engines (see concepts/streaming-aggregation / concepts/incremental-view-maintenance). For the HTTP rollup dataset the whole chain cut 9+ GB/s → <2 GB/s compressed (~80%).

Why Logfwdr is the leverage point

All three reductions exploit the same structural fact: data is cheapest to shrink before it fans into the backbone. Sampling and aggregating at Logfwdr means fewer bytes centralized; load-balancing to a shared pool means each byte crosses the WAN once. Processing less centrally also saved ~750 CPUs and 3.5 TiB memory (~50%) on the Ready-Analytics receivers alone. The counterpoint — compression — actually improves downstream of Logfwdr, because post-centralization streams are partitioned into uniform batches over wider time windows that compress far better than Logfwdr's source-fragmented transit batches.

Seen in

  • sources/2026-10-13-cloudflare-rewinding-the-exponential — the canonical wiki source. Positions Logfwdr as the pipeline head and the site of the 2026 load-balancing (double-write → shared ingest pool), decentralized square-root sampling, and per-instance aggregation changes. Operational numbers: ~2 B events/s pipeline; −36 GB/s net WAN from load balancing; 25 GB/s / >2 PB/day from sampling top 3 datasets; HTTP rollup 9+ → <2 GB/s from aggregation; ~750 CPUs + 3.5 TiB freed on Ready-Analytics receivers.
Last updated · 854 distilled / 2,448 read