Skip to content

SYSTEM Cited by 1 source

Apache DataFusion

Apache DataFusion (datafusion.apache.org) is a Rust query engine built on Apache Arrow, with a SQL planner and a vectorized, columnar execution engine. It is designed to be embedded in-process inside a host application rather than run as a standalone database — the role Cloudflare uses it in for Logpush Transformers. (Source: sources/2026-10-12-cloudflare-filter-reshape-redact-logs-transformers)

Why Cloudflare chose it

For Logpush Transformers, Cloudflare needed to run customer-defined SQL inside the export hot path without introducing a network hop or a separate tier. After evaluating several SQL execution engines, it picked DataFusion because:

  • Rust + Arrow-native. DataFusion reads typed Arrow RecordBatch values directly in its native columnar model — no intermediate row-by-row JSON parse-and-rebuild step.
  • Vectorized, columnar execution. Fixed-width Arrow columns make SIMD-friendly, branch-predictable processing natural (concepts/columnar-storage-format).
  • Already in production at Cloudflare. DataFusion "already powers Cloudflare Basin," so it was a known quantity.
  • Embeddable in-process. It runs inside Bufstream (Go) via a Rust wrapper, letting transform batches reuse Bufstream's existing scheduling, batching, and destination controls "without introducing a separate transformation tier or network hop."

Logical plan: prepare once, reuse per batch

DataFusion separates planning from execution:

  • A Transformer's SQL compiles into a reusable logical plan that "defines the scan, filters, projections, and expressions."
  • Bufstream prepares this plan once per time slice of a job's data and reuses it for every Arrow batch in that slice.
  • Per batch, the Rust wrapper binds the batch as the plan's table input, executes the query, and returns Arrow batches with the selected records and fields.

The table named in the SQL's FROM clause selects which dataset schema the query validates against — Bufstream supplies a generated schema from the Logpush dataset specification rather than inferring it per batch, giving DataFusion a stable view for query validation.

Performance

In a Cloudflare benchmark, "executing a prepared Transformer against a synthetic 5,000-row http_requests batch from a single Go goroutine took about 3 ms, equivalent to roughly 1.6 million rows per second." This is a single-goroutine figure; Bufstream adds per-job transform parallelism so high-volume jobs run transform batches concurrently.

Relationship to DuckDB / other engines

DataFusion occupies the same embeddable analytical SQL engine niche as DuckDB, but its Rust + Arrow-native design makes it a natural fit for embedding inside a Rust (or, via FFI, Go) service that already speaks Arrow — which is exactly why it, rather than an alternative, sits inside Bufstream and Basin.

Caveats

  • The post does not name which other SQL engines were evaluated or the comparative benchmarks that led to DataFusion.
  • The 1.6M rows/s figure is a single-goroutine micro-benchmark on synthetic data, not an end-to-end production throughput number.

Seen in

Last updated · 854 distilled / 2,448 read