Skip to content

SYSTEM Cited by 1 source

Bufstream (Cloudflare Logpush delivery service)

Bufstream is Cloudflare's Logpush delivery service — the Go-based system that delivers "more than 110 million Logpush records per second to customer destinations." It runs across "hundreds of Kubernetes pods, each claiming a time slice of data from Buftee," Cloudflare's distributed log-based in-house message broker. This model distributes work horizontally across Logpush jobs, whose volume follows a Pareto-like distribution — most jobs are low-volume, a few high-volume jobs process millions of logs/second (concepts/partition-skew-data-skew). (Source: sources/2026-10-12-cloudflare-filter-reshape-redact-logs-transformers)

Bufstream is the service into which Cloudflare embedded Apache DataFusion in-process to run Logpush Transformers — the architecture this page documents.

Design requirement: don't perturb the export hot path

Transformers "needed to meet the same operational requirements as the rest of Logpush: reliable delivery, high throughput, and predictable resource use without introducing additional failure scenarios into the export hot path." An initial prototype ran customer WebAssembly Transformers in Workers, but "invoking a Worker would insert a network call into the Logpush export path. That remote dependency could introduce transient timeouts and retries, reducing delivery throughput and increasing delivery latency." The decision was to keep Transformer execution in-process within the delivery service — embedding the SQL engine next to the data instead of calling out to it (patterns/co-located-inference-in-serving-layer). Embedding DataFusion in Bufstream lets transform batches "use the service's existing delivery infrastructure, including its scheduling, batching, and destination controls, without introducing a separate transformation tier or network hop."

The Go↔Rust boundary (the hard part)

Choosing DataFusion meant connecting Bufstream (Go) to an engine written in Rust. Bufstream moves typed batches across that boundary as Arrow IPC exposed through a C ABI and called from Go using cgo.

From records to columns

  1. Bufstream receives compressed Logpush batches of Cap'n Proto records from Buftee.
  2. For a job with a Transformer attached, it decompresses, parses the eligible logs, and builds a typed Arrow RecordBatch, using a generated schema from the Logpush dataset specification (schema is supplied, not inferred per batch — a stable view for query validation).
  3. The Rust DataFusion wrapper reads typed RecordBatch values via an Arrow StreamReader, "rather than parse individual rows from an intermediate format such as JSON and rebuild typed Arrow columns for execution." Field types, nullability, and nested values are consistent on both sides.

Explicit ownership across the boundary

"The Go and Rust runtimes do not share ownership of transform data."

  • Go passes a pointer to its input Arrow IPC buffer to Rust only for the duration of the cgo call; Rust does not retain that pointer after returning.
  • Rust owns the transformed output buffer; Go copies it into Go-managed memory and then explicitly releases the Rust allocation.

This makes "ownership and lifetimes explicit across the language boundary, avoiding cross-runtime memory-management bugs." Crossing the boundary is "not free" — batches still need to be serialized, decoded, and managed in memory — but Arrow IPC keeps the work batch-oriented, avoiding per-record parsing/conversion, which fits a high-throughput pipeline while retaining schema validation.

Executing the transform

  • A customer saves a Transformer; its SQL is stored in R2.
  • Bufstream retrieves the SQL through a multi-layer cache and translates it into a reusable DataFusion logical plan (scan, filters, projections, expressions), prepared once per time slice of a job's data and reused for every Arrow batch in that slice.
  • For each new batch, the Rust wrapper binds the batch as the plan's table input, executes, and returns Arrow batches with the selected records/fields.

Fail-closed delivery + fairness

Transformers are designed to fail closed: if a transform cannot execute, Bufstream "neither advances the job's delivery offset nor sends the untransformed batch" — delivering raw data could violate the downstream schema/filtering/redaction assumptions. It re-attempts the failed transform instead (concepts/fail-open-vs-fail-closed). To stop a persistently failing job from consuming capacity needed by healthy jobs, Bufstream progressively delays retries, reports errors on the job, and automatically disables jobs that remain failing for an extended period.

Per-job parallelism + adaptive upload

  • Attaching a Transformer adds CPU to a job's path, so Bufstream uses per-job transform parallelism — high-volume jobs execute transform batches concurrently.
  • On output it converts Arrow → NDJSON, then batches, gzip-compresses, and uploads to the destination. Because destinations accept data at different rates and can temporarily slow down, Bufstream adjusts the number of upload goroutines while respecting destination concurrency limits (concepts/backpressure).

Operational numbers

  • 110M Logpush records/second delivered.

  • Hundreds of Kubernetes pods; each claims a time slice from Buftee.
  • Transform micro-benchmark: ~1.6M rows/s single goroutine (see systems/apache-datafusion).

Caveats

  • "Buftee" is named as the upstream broker but not architecturally described here (log-based, distributed, in-house) — no dedicated page.
  • cgo call overhead and the Arrow-IPC copy cost at 110M records/s are not quantified.
  • The multi-layer SQL cache's layers/invalidation are not detailed.

Seen in

Last updated · 854 distilled / 2,448 read