Skip to content

PATTERN Cited by 1 source

Diskless / disk-lite hybrid streaming

Intent

Capture the cross-AZ bandwidth savings of a fully diskless streaming shape (object-storage-as-data-plane) without sacrificing the Kafka-protocol guarantees — transactions, idempotency, fast leader elections, single-digit-ms metadata consistency — that depend on fast local coordination storage.

The pattern's proposition: move the bulk bytes to object storage; keep the consensus-critical state on local NVMe. The broker remains a "disk-lite" participant — still has a local disk, but uses it only for metadata and Raft log state, not for message payloads.

Canonical framing

From the sources/2026-03-31-redpanda-261-delivers-the-industrys-first-adaptable-streaming-engine|26.1 launch post:

"The industry's answer has been diskless architectures. But diskless isn't riskless. By moving everything off of local disks in favor of object storage and external databases, these systems sacrifice data integrity, availability, and core Kafka features (like transactions) on the altar of cost."

"Enter the Redpanda 'disk-lite' approach… Cloud Topics use a pass-through write model that saves the bulk of your messages directly to object storage. We stream the heavy message payloads directly to S3 or GCS, but we keep the brains of the operation—the metadata and Raft consensus—on high-performance local NVMe."

"You get over 90% lower networking costs and the economics of a diskless system, while keeping the battle-hardened reliability of a local-disk broker that can safely run low-latency workloads alongside these topics. No broken transactions. No metadata lag. No external control plane dependencies. Just efficient, Raft-consistent streaming."

"Disk-lite" is Redpanda's coined name for this pattern, positioned against "diskless" (WarpStream-class) and against traditional full-disk-resident streaming brokers.

The storage-path design space

Shape Payload Metadata Coordination Transactions Canonical example
Traditional Local NVMe (RF copies) Local NVMe (RF copies) Local NVMe Yes Classic Kafka, pre-tiering Redpanda
Diskless Object storage External metadata store External coordination service Often limited WarpStream
Disk-lite Object storage Local NVMe (RF copies via Raft) Local NVMe (Raft) Yes Redpanda Cloud Topics
Tiered storage Local NVMe (hot) + object storage (cold) Local NVMe Local NVMe Yes Kafka tiered storage, Redpanda tiered topics

Disk-lite sits at the corner of the design space that preserves Kafka-protocol semantics (transaction support, idempotency markers, fast leader elections) while paying only metadata- volume cross-AZ cost instead of payload-volume cross-AZ cost.

Mechanism

Two layers with different substrates:

  1. Data plane (bulk bytes) — payload batches stream via pass-through write to object storage. Durability is inherited from the object store's native multi-AZ replication (S3's 11 9s, GCS equivalent). No per-byte cross-AZ network billing.

  2. Metadata plane (coordination state) — offsets, consumer- group state, transactional control markers, Raft log entries, and placeholder batches pointing to the object-store location of each payload batch — all flow through standard Raft replication on local NVMe. This is the "disk-lite" part — every broker still has a disk, but it's used for small-but-latency- sensitive state, not for storing the producer-sent bytes that dominate storage volume.

The anti-pattern: whole-cluster diskless

The post frames the anti-pattern sharply: "diskless isn't riskless." Moving all broker state off local disks — including metadata and consensus state — produces three structural failures:

  • No broken transactions — Kafka transactions depend on ordering-preserving metadata commits on replication quorums. An external metadata store incurs object-store-grade latency on every commit, breaking transaction performance.
  • No metadata lag — when metadata lives in a remote service, any request to resolve an offset, identify a leader, or look up a consumer group's position adds a network hop. Disk-lite metadata lives on the same node as the producer- ack thread.
  • No external control plane dependencies — a fully diskless broker has a hard dependency on a coordination service. If that service is unavailable, the broker is unavailable. The disk-lite broker is self-contained.

When to use

  • Latency-tolerant, high-volume, replication-cost-dominated workloads — observability telemetry, compliance archives, model-training feature streams, batch analytics inputs. These workloads tolerate object-storage write latency (multi-hundred-ms range) in exchange for the 90%+ cross-AZ cost reduction.

  • Single-cluster multi-tier environments — when you want per-topic storage-tier selection in one cluster rather than spinning up a separate diskless cluster for cost-sensitive topics and managing three security models.

  • Mission-critical topics co-hosted with cost-optimised topics — "a local-disk broker that can safely run low-latency workloads alongside these topics."

When to avoid

  • Sub-10-ms produce p99 requirements — the object-store upload step on the write path adds latency bounded only by S3 / GCS PUT latency. Write-caching or traditional topics are required for sub-10-ms commits.

  • Small-batch steady-state workloads — if bytes-per-batch is low, the per-PUT object-store request cost dominates cross-AZ transfer savings.

Trade-offs

  • Cross-AZ cost reduction: 90%+ vendor-claimed (no production measurements disclosed).
  • Per-produce latency floor: bounded by object-store upload latency; e.g. 250 ms on the 26.1 default batch window.
  • Object-store PUT cost: small-batch workloads pay per- request overhead. The 26.1 architecture mitigates via multi-partition batch aggregation at the broker (placeholder batch design).
  • Operational simplicity: single cluster, single security model, single API — vs running a separate diskless cluster for the cost-sensitive topics.

Canonical instance

Redpanda Cloud Topics — Generally Available in Redpanda Streaming 26.1 (2026-03-31 GA; 25.3 beta preview). Each batch of producer messages lands directly on object storage (S3 / GCS / ADLS); a placeholder batch pointing to the object-store location is replicated via Raft to the metadata quorum on local NVMe; producer ack fires only after the Raft replication succeeds. Transaction, idempotency, and consumer-group offset primitives inherit standard Kafka semantics because the metadata plane is untouched.

Anti-pattern instance

WarpStream — the post's named competitive target for the "diskless" pattern. WarpStream moves both data and metadata out of broker-local storage; transactions and the full Kafka semantics are consequently constrained.

Seen in

Source

Last updated · 550 distilled / 1,221 read