PATTERN Cited by 1 source
Single-copy compaction on object store¶
Context¶
In traditional disk-based Kafka (or Redpanda NVMe topics), log data is replicated across replication.factor brokers. Each broker independently compacts its local copy, leading to:
- RF × redundant CPU work — the same logical data is scanned and rewritten RF times
- Broker contention — compaction competes with produce/consume for CPU and I/O
- Tombstone coordination problem — safely removing delete markers requires cross-broker agreement (see concepts/compaction-replication-race)
Solution¶
Cloud Topics stores committed data as immutable L1 objects in object storage — a single canonical copy. Compaction operates on this one copy via the metastore, which tracks which objects comprise each partition's log.
Because idempotency and transactions are already resolved at the L0 layer, compaction of L1 data need not concern itself with transaction control batch removal.
Consequences¶
- Compaction cost is O(1) per partition, not O(RF)
- Broker CPU is fully available for produce/consume
- Tombstone coordination is unnecessary — there's nothing to diverge
- Compaction can be offloaded to any node in the cluster
Trade-offs¶
- Object-store latency for reads/writes during compaction (mitigated by multi-part upload)
- Requires optimistic concurrency control (
compaction_epoch) to handle concurrent compactors