Skip to content

SYSTEM Cited by 1 source

Cloudflare K2

Cloudflare K2 is Cloudflare's durable event-streaming primitive on the Developer Platform (public beta, 2026-10-01). You send events to a K2 stream, which stores them as an ordered log; consumers read them back by splitting reads across a set of consumers (read parallelism) or by delivering all messages to all consumers (pub/sub fan-out). It is "fully serverless, scales to vast quantities of data, and supports long-term retention, so even long periods of consumer downtime do not lose data." Documented at developers.cloudflare.com/k2.

K2 solves the classic producer/consumer coupling problem — in RPC architectures "producers and consumers must align in scale and in time" — by decoupling them with a durable buffer in the middle that absorbs writes while letting independent readers consume at their own pace (the backpressure-absorbing buffer shape).

The core bet: a durable log on R2, not Kafka

"Under the hood, K2 implements a partitioned, durable log on top of R2 object storage, which allows it to scale to huge volumes of storage."

K2 was first built as the durable buffer behind Basin Pipelines — Pipelines' pull-based stream-processing engine needs some system to durably store events before they're read, transformed, and written to R2, and Cloudflare commits to never dropping events once accepted into the Pipelines Stream.

"This is where most companies would deploy Apache Kafka. However, Pipelines runs on the Cloudflare edge, which spans a huge number of servers across over 335 cities. Our unique architecture means we often cannot run traditional distributed systems software like Kafka, and need to rethink how these systems are built and operated."

The edge is hostile to stateful services as Kafka assumes them:

  • "relatively small slices of machines" (not dedicated brokers),
  • "those machines are relatively ephemeral" (not long-lived),
  • "networking is often over the public Internet" (not a datacenter LAN).

…but has two superpowers: proximity to users worldwide and an "incredible capacity to scale horizontally." So K2 relies on the state primitive Cloudflare already has — R2:

"Object storage systems like R2 combine extremely durable storage (11 9s!) with strongly consistent APIs. Offloading replication and consensus to the storage layer allows us to make the application layer (K2 in this case) radically simpler, cheaper, and higher performance. A secondary benefit is that it separates compute and storage, meaning each can be scaled independently."

This is the object-storage-as-disk-root move applied to a streaming broker, and the compute-storage separation it unlocks (store vast historical data cheaply, scale compute separately).

Building a log on an object store that has no append

The central engineering problem: R2 — "like other object stores — does not support appends, the standard operation on a log." K2's answer:

  1. Accumulate writes in-memory on an edge service.
  2. Wait a short period for data to arrive.
  3. Write all events as a segment file — large enough to "overcome the cost of writing and reading each one" (small-file avoidance / micro-batching / batch-before-write).
  4. Order + strictly-incrementing offsets come from R2's atomic operations "without needing a separate coordination service" — a conditional/atomic write on the storage layer substitutes for leader election / consensus (offset-preserving).

The downside is produce latency: object-store writes are slower than local disk, and K2 must wait for the local batch to accumulate before starting the write — "about 1 second of produce latency at the 99th percentile" in the initial release (the explicit batching-latency tradeoff).

Consumption: subscriptions, leases, ack/nack/extend

Produce via HTTP API or Worker binding (env.EVENTS.send([...]) returning { success, error: { message, retryable } }); K2 represents data as bytes, so any encoding works.

Subscriptions divide work between consumers for read parallelism — "scaling out to multiple readers to handle more load than a single server can manage." Each consumer polls /subscriptions/{id}/consume with a worker_id + max_records.

On consume, the client gets a lease for that batch of events for 5 minutes and can do one of three things:

  • ack — mark the batch processed; it will not be redelivered.
  • nack (negative ack) — processing failed; redeliver.
  • extend — needs more time; extend the lease.

Lease-expiry-redelivers, which makes K2 at-least-once — a crash after processing but before ack produces a duplicate, so consumers must be idempotent.

Two read topologies (mix-and-matchable):

  • Read-parallel — one subscription, work split across a consumer pool (each consumer sees a portion of the data).
  • Pub/sub fan-out — a separate subscription per consumer, so each consumer sees all messages. "Or you can mix-and-match … having multiple independent consumer pools."

Streams, Queues, or Pipelines?

K2 is positioned alongside, not replacing, Cloudflare's other async primitives:

Primitive Shape Use when
Queues Individual expensive work items; per-item retries / delays / dead-letter queues Tracking discrete units of async work with message-level logic
K2 Streams High-scale data movement, long-term retention, batch-grained produce/consume, fan-out Moving lots of data, replaying history, multiple independent consumers
Basin Pipelines Serverless ingestion that transforms + writes to R2 / Basin Catalog (Iceberg) The end result is writing events to object storage or Iceberg tables

K2 trades Queues' message-level retries for batch efficiency ("produced and consumed as batches — enabling efficient processing at the expense of message-level retries"), which also drives K2's higher producer latency vs Queues. Reach for Pipelines when writing to object storage / Iceberg; reach for K2 "when doing custom processing or writing to other destinations."

Operational facts (beta)

  • Edge footprint: 335+ cities.
  • R2 durability inherited: 11 nines.
  • Produce latency: ~1 s at p99 (initial release).
  • Consume lease: 5 minutes per batch (extendable).
  • Beta limits: max 10 GB storage; 30 MB/s produce per stream.
  • Create example defaults: retention_seconds: 604800 (7 days), HTTP endpoint + Worker binding optional.
  • Creation surfaces: cf (cf k2 streams create --name app_events --http-enabled), Wrangler, dashboard, or API.
  • Anticipated pricing (post-beta): $0.04/GB produced, $0.04/GB consumed, $0.02/GB/month retained; unbilled during beta.
  • Access: Workers Paid accounts.

Roadmap (stated)

  • Higher write parallelism — up to multi-GB/s streams.
  • Message keys + key-based ordering guarantees.
  • Push-based worker consumers.
  • An Express tier with lower produce + end-to-end latencies.
  • Drop-in Apache Kafka client support.

Seen in

Last updated · 766 distilled / 2,225 read