Skip to content

PATTERN Cited by 1 source

Hibernate unused topics on tiered storage

Pattern

For workloads a cluster is no longer actively serving but whose history the operator still wants preserved, unmount the topic — detach it from the cluster's active set without destroying its data — and re-mount it later when the data is needed again, potentially on a different cluster or under a different name. The topic's segments stay in object storage the whole time via tiered storage.

Canonical Redpanda instantiation is Mountable Topics, introduced in the 2024-12-03 24.3 release:

"'Hibernate' unused topics to conserve cluster resources and maximize cost savings and flexibility. ... This allows you to use 100% of your cluster capacity to support your primary workloads while preserving the full history of other workloads."

(Source: sources/2024-12-03-redpanda-redpanda-243-extends-lakehouses-with-streaming-data-cdc)

When to apply

  • Retired workloads with compliance-retention requirements. The business stopped producing to the topic six months ago, but regulation requires seven years of retention. Keeping the topic mounted pays metadata + replica cost for seven years; unmounting pays object-storage cost only.
  • Seasonal workloads. Quarterly batch-report feeds, end-of-month reconciliation topics — active for days per quarter, dormant otherwise. Unmount between active windows.
  • Migration-in-progress staging topics. Created for a specific data-migration project, drained at cutover. Unmount once the migration completes; re-mount only if a re-run is required.
  • Multi-cluster archive pattern. Unmount retired topics from expensive high-spec production clusters and re-mount them on a cheaper archive cluster for ad-hoc access.

When NOT to apply

  • Active low-traffic topics. An occasionally-consumed topic is still active — unmounting forces a mount round-trip before any read or write. Just let it sit at low cost with tiered storage.
  • Topics without tiered storage configured. The pattern depends on the data already being in object storage. A local-log-only topic can't be unmounted without data loss.
  • Topics with active consumer groups. Consumer-group state across unmount/remount cycles is not fully specified in the 24.3 announcement; verify the behaviour before applying to topics with committed consumer offsets.

Mechanics

rpk + the Redpanda Cloud API expose mount / unmount verbs:

rpk topic unmount <topic>
rpk topic mount <topic> [--new-name <name>]

(See Redpanda Mountable Topics docs.)

Relationship to cluster-level decommission

Tiered-storage fast decommission is the broker-lifecycle analogue: a broker leaves the cluster quickly because its historical data lives in object storage. Hibernate-unused-topics-on-tiered-storage is the topic-lifecycle analogue: a topic leaves the cluster's active set quickly for the same reason.

Both patterns capitalise on the same structural property of tiered storage — the bulk of the data doesn't move with the compute — applied at different granularities.

Seen in

Last updated · 550 distilled / 1,221 read