Skip to content

SYSTEM Cited by 2 sources

Redpanda Cloud Topics

Cloud Topics is a Redpanda 25.3 feature (2025-11-06 preview, beta at launch) that adds a new topic class whose data is written directly to object storage (S3 / ADLS / GCS) while topic metadata is managed in-broker and replicated via Raft. Cloud Topics coexist with traditional Redpanda topics (NVMe-backed, Raft-replicated data), Iceberg topics (Parquet-projected lakehouse sink), and write-caching topics in the same cluster — delivering "one system for all streams" rather than separate clusters per workload class.

Canonical verbatim

From the sources/2025-11-06-redpanda-253-delivers-near-instant-disaster-recovery-and-more|25.3 launch post:

"With Cloud Topics, each batch of messages is passed straight through and written to cost-effective object storage (S3/ADLS/GCS) while topic metadata is managed in-broker — replicated via Raft for high availability — so the cluster can do its job (partition leadership, replica placement, quotas, governance)."

"This approach virtually eliminates the cross-AZ network traffic associated with data replication. You keep millisecond performance where it matters, and pay object-store prices for replication where it doesn't, without sacrificing durability."

Architectural shape

Cloud Topics sits at a specific point in the streaming-broker storage-tiering design space:

Shape Hot data Cold data Metadata Cross-AZ replication
Traditional Redpanda topic NVMe NVMe (no tiering) Raft (in-broker) Yes — RF-1 copies
Redpanda tiered topic NVMe Object store Raft (in-broker) Yes — hot tier only
Cloud Topic (25.3) Object store Object store Raft (in-broker) No — durability inherited from object store
WarpStream Object store Object store External / metadata store No
Confluent Kora (standard/dedicated) Replica disks Tiered storage In-broker Yes

Cloud Topics is within-cluster tiering — topics pick their storage substrate independently. Customers can host mission- critical payments topics on NVMe and latency-tolerant observability streams on object storage in the same cluster, with one set of IAM policies, one GitOps workflow, one Kafka API endpoint.

Why per-topic, not per-cluster

The post canonicalises the latency-critical-vs-latency-tolerant workload distinction as the motivation:

"Some data sets are latency-critical (e.g., payments, trading, cybersecurity), and others are latency-tolerant (e.g., observability, model training, compliance reporting). Treating those workloads the same is inefficient."

Per-topic tiering means cost structure aligns with business value at the topic granularity rather than forcing a cluster-level commitment.

Cost framing

The load-bearing cost claim:

"This approach virtually eliminates the cross-AZ network traffic associated with data replication."

See concepts/cross-az-replication-bandwidth-cost for the cost axis Cloud Topics attacks. Multi-AZ Raft replication — the default HA shape on Redpanda traditional topics — writes each produced byte to RF-1 brokers in other AZs, each of which is a billed cross- AZ transfer. Cloud Topics route writes directly to object storage (which inherits its own multi-AZ durability from the cloud provider), so the per-write cross-AZ cost is replaced by object- store PUT cost.

Business-impact framing from the post:

  • "Dramatically lower TCO: Sidestep steep cloud provider networking charges for compliance, security, training data, or batch analytics."
  • "Architectural simplification: Stop using a separate platform or cluster just to handle latency-tolerant streaming workloads."
  • "A single multimodal streaming engine: … all in one platform. That's less infrastructure to manage and a cleaner mental model for every team."

Position against Confluent

Explicit comparison from the post:

"Contrast this with Confluent, where you may need a mix of Kora-powered Confluent Cloud clusters (standard/dedicated or Freight) and the separate Confluent WarpStream engine (BYOC) to satisfy different requirements."

Redpanda's differentiator: one cluster, multi-tier storage vs. Confluent's multi-cluster, per-cluster-tier split.

Multimodal-streaming composition

"Run traditional Redpanda topics for low latency and data safety, use write caching for ultra-low latency, Iceberg Topics for push-button lakehouse ingestion, and Cloud Topics for cost- efficient, high-throughput streaming — all in one platform."

Cloud Topics is the fourth topic class in Redpanda's composition vocabulary alongside:

  • Traditional topics — NVMe-backed, Raft-replicated, lowest latency.
  • Write-caching topics — ack-on-memory with background flush, ultra-low-latency.
  • Iceberg topics — simultaneously a Kafka topic and an Iceberg table.
  • Cloud topics — object-storage-backed, cost-optimised.

Status (2026-03-30)

Generally Available in Redpanda Streaming 26.1 (per the 2026-03-30 architecture deep-dive: "with the release of Redpanda Streaming 26.1, Cloud Topics has officially entered General Availability."). Beta in the prior 25.3 preview (2025-11-06).

Architecture

The 2026-03-30 deep-dive is the first detailed public description of Cloud Topics' internals. The load-bearing architectural split:

"The Cloud Topics architecture separates where metadata is stored (each partition's Raft log) and where data is stored (object storage). Traditionally, the data and metadata for the records that are produced are written and replicated using the Raft consensus protocol. Since Cloud Topics writes data directly to object storage, we can bypass the Cross-AZ networking tax incurred when replicating via Raft."

Write path

  1. Kafka API entry — producer records enter the standard Kafka API layer.
  2. Cloud Topics Subsystem staging — instead of appending to the local Raft log's on-disk payload, records are routed to an in-memory multi-partition staging buffer.
  3. Batch trigger — the buffer flushes on time or size: "we batch incoming data in memory for a short window defined by time (e.g., 0.25 seconds) or size (e.g., 4MB). We collect this data across all partitions and topics simultaneously. We do this specifically to minimize the cost of object storage; by aggregating smaller writes into larger batches, we significantly reduce the number of PUT requests sent to S3."
  4. L0 file upload — buffer flushes to cloud object storage as a single file. "We flush this batch directly to cloud object storage. We call this an L0 (Level 0) File." See concepts/l0-l1-file-compaction-for-object-store-streaming.
  5. Placeholder batch replication — "once the L0 file is safely durable in the cloud, we replicate a placeholder batch containing the location of the data to the corresponding Raft log for each batch involved in the upload." See concepts/placeholder-batch-metadata-in-raft.
  6. Producer ack — "then we send an acknowledgement to the producer that the batch is safely persisted."

The batch window (0.25 s example) places a floor on produce p99 latency for Cloud Topics — the feature is positioned for latency-tolerant workloads where this is acceptable.

Semantics preservation

"Because we still use the Raft log for this metadata, Cloud Topics inherit the same transaction and idempotency logic as our standard topics. The data payload lives in the cloud, but the guarantees live in Redpanda."

Kafka's transactional and idempotency protocols operate on record metadata (offsets, sequence numbers, transactional control records) — which still flow through Raft in order. The placeholder batch carrying an object-storage pointer is indistinguishable from a standard payload-carrying batch from the perspective of those protocols.

The Reconciler

L0 files are ingest-optimal but read-unfriendly: they contain data from many different partitions batched together, so reading a single partition's history would require "'scattered reads' across many different files."

A background process called the Reconciler continuously rewrites L0 into L1 files:

"The Reconciler continuously optimizes the storage layout. It reads the L0 files and reorganizes the data, grouping messages that belong to the same partition and writing them into L1 (Level 1) Files."

"L1 Files are: Much larger: Optimized for high-throughput object storage reading. Co-located: All data for a specific partition range is physically together. Sorted: Organized by offset."

"Once L0 data is successfully moved into L1, it's eligible for garbage collection and will eventually be removed."

L1 metadata lives in a separate shared metadata tier:

"Metadata for L1 files are stored in a shared metadata tier that's backed by an internal topic and a key-value store. This ensures that the system maintains a robust, consistent view of where your optimized data resides. This includes updating metadata as the underlying data is rewritten by compaction, and removed as the retention policy kicks in."

See patterns/background-reconciler-for-read-path-optimization.

Read path

The read router uses a per-partition watermark called the Last Reconciled Offset:

*"When a consumer requests data, Redpanda routes the request based on where the data currently lives in its lifecycle. Each partition tracks a Last Reconciled Offset.

  • Reads > Last Reconciled Offset: The system reads from L0. The system follows the pointers in the local Raft logs to find the specific batches in object storage if not found in the local cache.
  • Reads < Last Reconciled Offset: The system reads from L1. This is the highly optimized path for historical reads, allowing us to open large, sorted files and stream data efficiently without scattering."*

Tailing consumers "which represent the vast majority of streaming workloads" typically hit the memory cache first — the L0-scattered-read cost only appears for consumers that fall behind both the memory cache's and the Reconciler's trailing edges. Reconciler cadence effectively bounds that window.

Architecture-at-a-glance

Producer
  │ produce()
  ▼
┌──────────────────────────────────────┐
│ Kafka API layer                       │
│   │                                   │
│   ▼                                   │
│ Cloud Topics Subsystem (in-memory)    │
│   batch across ALL partitions/topics  │
│   trigger: 0.25s or 4MB (example)    │
└──────────────────────────────────────┘
    │ PUT (single file)
    ▼
┌──────────────┐        ┌──────────────────────┐
│ Object store │───┐    │ Per-partition Raft   │
│   L0 files   │   │    │ log: placeholder     │
│ (mixed       │   │    │ batch (object-store  │
│  partitions) │   │    │ pointer)             │
└──────────────┘   │    └──────────────────────┘
    │              │            │
    │              │ Reconciler │ Producer ack
    │              │ reads L0   │ after both
    │              ▼            ▼
┌──────────────┐   Background rewrite
│ Object store │◀──per-partition, offset-sorted
│   L1 files   │
└──────────────┘
    │
    │ Read router: requested_offset vs Last Reconciled Offset
    │  > LRO → memory cache → L0 (via Raft placeholder)
    │  < LRO → L1 (via shared L1 metadata tier)
    ▼
Consumer

What's not disclosed

The 2026-03-30 post stops short of several axes that will matter in production:

  • Absolute latency — no P50/P99 for produce ack, no per-read latency for L0-miss vs L1-hit.
  • Net cost — eliminating cross-AZ cost replaces it with PUT cost + background-compaction egress/storage cost; the net delta is not quantified.
  • Reconciler placement — no disclosure of whether the Reconciler runs on partition leaders, a separate pool, or a detached fleet.
  • Metadata-tier scale — the L1 metadata tier's internal topic name, KV-store engine, and scale ceiling are not disclosed.
  • Failure-mode behaviour — partial L0 upload failure, placeholder-without-data gap, Reconciler crash mid-rewrite, and L1-metadata-tier unavailability aren't covered.
  • Cache design — the "memory cache" that serves tailing consumers isn't architected (per-broker? LRU? tail-truncating?).

Seen in

Last updated · 470 distilled / 1,213 read