PATTERN Cited by 1 source

Metastore bootstrap from object storage

Pattern

Design the metadata store so its complete state (manifest + SSTables) exists in object storage at all times. This enables two key use cases without connecting to the source cluster:

1. Disaster recovery (full cluster rebuild): Reconstruct the metadata store entirely from object storage, then resume operations from the last-flushed state.
2. Read replicas: A remote cluster bootstraps by downloading the manifest, then lazily pulls SSTables on demand to serve consumer queries.

When to use

• The system must support recovery from complete cluster loss.
• Read-only replicas in different regions should not add load to the source cluster.
• The RPO guarantee (bounded by flush interval) is acceptable.

Mechanism

1. Every flush writes SSTables + manifest to object storage (the source of truth for external consumers).
2. DR path: New cluster downloads manifest → knows all SSTables → rebuilds internal metadata topic → determines last-written offset per partition → resumes writes.
3. Read-replica path: Remote cluster downloads manifest → pulls SSTables on demand → serves Kafka consumers against them.

Trade-offs

• Pro: No cross-cluster network links needed for DR or read replicas.
• Pro: Object storage provides the durability guarantee.
• Con: Staleness bounded by flush interval (default 10 min for Redpanda's metastore).
• Con: Bootstrap time depends on manifest/SSTable download speed.

(Source: sources/2026-06-09-redpanda-cloud-topics-the-metastore)

Seen in

• systems/redpanda-cloud-topics-metastore — Whole Cluster Restore and read-replica topics bootstrap from object-storage metastore state