Skip to content

PATTERN Cited by 1 source

Per-topic storage tier within one cluster

Pattern

Let each topic in a streaming cluster pick its own storage substrate (NVMe / tiered / object-storage-native / lakehouse- native), independently of other topics in the same cluster. Topic-granularity storage tiering aligns operational cost with business value per data stream without forcing a cluster-level commitment.

The pattern composes:

  • Cluster-shared metadata plane — partition leadership, replica placement, quotas, ACLs, schema registry, governance — managed in the broker's Raft-replicated metadata layer, common to all topics.
  • Topic-selected data plane — per-topic configuration chooses where record bytes live: local NVMe for latency-critical, NVMe with write-caching for ultra-low-latency, object storage (S3 / GCS / ADLS) for latency-tolerant / cost-optimised, Iceberg-projected for lakehouse-native.

Customers pick storage substrate per topic, aligned with the workload's latency criticality.

Canonical instance

Redpanda Cloud Topics (25.3, 2025-11-06 beta) is the canonical wiki instance. Cloud Topics adds object-storage-direct as a new topic-substrate option alongside the existing traditional / write-caching / Iceberg substrates, all co-existing in one Redpanda cluster.

Canonical verbatim from the sources/2025-11-06-redpanda-253-delivers-near-instant-disaster-recovery-and-more|25.3 launch post:

"With Cloud Topics, each batch of messages is passed straight through and written to cost-effective object storage (S3/ADLS/GCS) while topic metadata is managed in-broker — replicated via Raft for high availability — so the cluster can do its job (partition leadership, replica placement, quotas, governance)."

"Run traditional Redpanda topics for low latency and data safety, use write caching for ultra-low latency, Iceberg Topics for push-button lakehouse ingestion, and Cloud Topics for cost-efficient, high-throughput streaming — all in one platform."

Contrast: per-cluster tiering

The prior Kafka-ecosystem shape is per-cluster tier choice: customers pick one posture per cluster (standard replicated Kafka / Kafka + Tiered Storage / WarpStream S3- native / Confluent Freight) and run separate clusters for separate workload classes. Migration between classes requires spinning up a new cluster and moving data.

From the launch post's explicit contrast:

"Contrast this with Confluent, where you may need a mix of Kora-powered Confluent Cloud clusters (standard/dedicated or Freight) and the separate Confluent WarpStream engine (BYOC) to satisfy different requirements."

Per-topic tiering within one cluster gives:

  • Single control plane. One IAM surface, one GitOps pipeline, one set of Kafka bootstrap endpoints, one version upgrade, one observability surface.
  • Cluster-shared primitives. Schema registry, ACLs, RBAC, and governance apply across all topic tiers — no cross-cluster federation.
  • In-place topic-class migration. Changing a topic's substrate is a configuration change, not a data migration.

The per-topic design commitment

For the pattern to work, the broker must be architecturally able to do each substrate — not just to tier old data off to object storage, but to serve entire topics off object storage while keeping other topics on NVMe, without one workload's substrate choice degrading another's.

This implies:

  • Shared metadata plane for topic placement / leadership / ACLs across substrates.
  • Substrate-specific read/write paths inside the broker — NVMe for hot-path topics, S3 SDK for object-storage topics.
  • Per-topic observability — latency distributions, throughput, and cost attribution all reported at topic granularity, because they diverge wildly by substrate.

Trade-offs

Pick per-topic tiering when:

  • The workload mix spans both latency-critical and latency- tolerant classes in the same domain (e.g., a fintech that has payments topics and observability topics).
  • Operational simplicity (one cluster) outweighs the marginal efficiency of per-class-specialised clusters.
  • Cross-cluster governance (schemas, ACLs, IAM) is painful enough that one-cluster-many-tiers wins on that axis alone.

Pick per-cluster tiering when:

  • The workload is pure latency-critical or pure latency- tolerant and you don't need both.
  • Vendor support constrains topic-substrate choice per cluster (most non-Redpanda Kafka-API implementations fall here at 2025).
  • Cross-substrate shared-metadata plane isn't available or trusted for multi-tenancy.

Seen in

Last updated · 470 distilled / 1,213 read