Skip to content

SYSTEM Cited by 2 sources

Cloudflare Logpush

Logpush is Cloudflare's log-export service — it "delivers logs in batches as quickly as possible," exporting Cloudflare logs to SIEMs, analytics platforms, and object storage. It was made available on all self-serve plans during Birthday Week 2026, alongside the GA of its Transformers feature. (Source: sources/2026-10-12-cloudflare-filter-reshape-redact-logs-transformers)

The delivery substrate is Bufstream, which "delivers more than 110 million Logpush records per second to customer destinations," claiming time slices of data from Buftee (Cloudflare's distributed log-based in-house message broker) across hundreds of Kubernetes pods.

The stateless shared-receiver-pool model (reused for analytics transit)

Logpush's delivery is stateless: "customer destinations do not care which centralized data center pushes the logs," so Logfwdr can treat the central data centers as a unified, shared pool of receivers — each message is sent to only one region, failing over to another destination if a machine or facility is unreachable. This automatic failover "had already proven its worth during large incidents, shifting traffic seamlessly from one location to another without human intervention." (Source: sources/2026-10-13-cloudflare-rewinding-the-exponential)

In spring 2026 Cloudflare's Data team adopted this exact model for analytics streams, which had previously double-written every non-regional event to two regions. Analytics was trickier than logs because — unlike stateless logs — ClickHouse clusters in both regions need a full copy for independent query failover; the resolution was to keep Logpush's send-once shared-pool transit but make the second database copy downstream via remote inserters out of Buftee (see late replication). Logpush is thus the origin of the shared-ingest-pool pattern that the analytics pipeline generalized — the stateless-delivery property is what made it directly reusable for logs and only adaptable (not copyable) for stateful analytics.

Transformers — SQL filter/reshape/redact before export

A Transformer is a SQL query written for a specific Logpush dataset. Logpush applies the query to each batch before export:

  • Rows that fail the WHERE clause are removed.
  • For the rest, the SELECT output becomes the exported record.
  • Input and output are both NDJSON — a Transformer reshapes records, it does not convert serialization formats.

Transformers can add, rename, remove, or nest fields and redact values (e.g. masking the low-order octets of a client IP with regexp_replace). Customers create/preview/manage them in Transformer Studio in the dashboard or via the API, then attach a saved Transformer to a compatible NDJSON Logpush job that uses the same dataset. Each update creates a new version so changes are reviewable over time.

Why it exists

Before export, teams often need to "remove health checks, omit sensitive fields, or reshape records for dashboards and investigations" — which traditionally means deploying and maintaining a separate processing service. Transformers move that processing into the Logpush export workflow so each destination receives data in the shape it needs. A second payoff is cost: "teams also pay to export, ingest, and store records they discard as soon as they arrive, so filtering them before delivery can reduce downstream data volume and cost" (concepts/egress-cost). This is ELT's transform step pushed to the producer side — filter/redact at the source rather than after load.

A representative use case: a team collecting logs from multiple CDNs adds a static 'cloudflare' AS provider field to normalize records to a common schema before they reach an analytics platform, and drops internal + health-check traffic (/healthz%, /metrics%) so it is never exported.

How the SQL executes (the architecture)

The engineering substance lives in Bufstream and is covered in depth on that page and on Apache DataFusion's. In brief:

  • Cloudflare embedded Apache DataFusion (Rust, Arrow-native, vectorized columnar SQL engine) in-process inside Bufstream (Go), rejecting an initial prototype that ran customer WebAssembly Transformers in Workers because a remote call "would insert a network call into the Logpush export path" with transient timeouts/retries. DataFusion was chosen partly because it already powers Cloudflare Basin.
  • Typed batches cross the Go↔Rust boundary as Arrow IPC over a C ABI via cgo, with explicit ownership so neither runtime leaks the other's memory.
  • Schema comes from the dataset specification (not inferred per batch); the FROM table selects which dataset schema the query validates against.
  • The SQL (stored in R2, fetched through a multi-layer cache) compiles to a DataFusion logical plan prepared once per time slice and reused for every Arrow batch in that slice.

Reliability — fail closed

Transformers are designed to fail closed: if a transform cannot run, Bufstream "neither advances the job's delivery offset nor sends the untransformed batch," because raw data could violate downstream schema/filter/redaction assumptions. It re-attempts with progressively delayed retries, reports errors on the job, and auto-disables jobs that stay failing for an extended period — so one bad job doesn't starve healthy ones. See concepts/fail-open-vs-fail-closed. Customers monitor successes, failures, pre/post-transform data volume, and transform duration in the Transformer Health view of Logpush's Health tab.

Caveats

  • No end-to-end latency figure for a Transformer-attached job is published (only an isolated ~1.6M rows/s DataFusion micro-benchmark — see systems/apache-datafusion).
  • Transformers require NDJSON jobs on a matching dataset.
  • The auto-disable threshold is not quantified.

Seen in

  • sources/2026-10-13-cloudflare-rewinding-the-exponential — Logpush's stateless shared-receiver-pool + automatic-failover delivery model as the design Cloudflare's analytics pipeline adopted for cross-DC transit (replacing double-write with send-once + late replication). Names the key difference — logs are stateless (any DC can push) whereas analytics needs a full ClickHouse copy per region — and how the pipeline kept the shared-pool transit but moved DB redundancy downstream of Buftee.
  • sources/2026-10-12-cloudflare-filter-reshape-redact-logs-transformers — canonical wiki source; Logpush export pipeline + Transformers GA + the in-process DataFusion/Arrow architecture.
Last updated · 854 distilled / 2,448 read