PATTERN Cited by 1 source

Coordinated compaction protocol¶

Pattern¶

A leader-driven two-phase coordination protocol layered on top of per-replica log compaction to prevent premature deletion of metadata records (tombstones, transaction markers) that other replicas may still need.

Structure¶

Per-replica progress tracking — each replica maintains a monotonically-advancing watermark of its local compaction progress (MCCO for key-space compaction, MXFO for transaction resolution).
Leader-driven collection — the partition leader periodically queries all followers for their current watermark values.
Min-aggregation — the leader computes the safe-deletion threshold as min(all replica watermarks), including last-known values for unavailable replicas.
Distribution — the leader pushes the computed threshold (MTRO/MXRO) back to all replicas. Each replica uses this as the upper bound for physically removing tombstones or transaction markers.

Key properties¶

Correctness over liveness — an offline replica freezes the safe-deletion threshold for the entire replica set. Storage grows but data is never corrupted.
Monotonicity — thresholds only advance forward. Late RPCs from previous leaders are rejected.
Unavailability-tolerant — uses last-known values for unreachable replicas, which is always conservative-safe.
Leader-change safe — new leader bootstraps from existing MTRO/MXRO, re-collects, and re-broadcasts.

When to use¶

Any replicated log where background cleanup (compaction, GC, tombstone eviction) runs independently per replica.
Any system where metadata records serve as signals to other replicas and their premature deletion creates unrecoverable divergence.

Trade-offs¶

Benefit	Cost
Eliminates time-based heuristic failure modes	Additional RPCs per compaction cycle
Correctness under unbounded replica unavailability	Storage growth when replicas are slow/offline
No operator-tuned retention windows	Added protocol complexity

Seen in¶

sources/2026-06-25-redpanda-kafkas-log-compaction-corrupts-data — canonical disclosure of the protocol as Redpanda's fix for Kafka's compaction–replication race.

concepts/coordinated-compaction — the concept this pattern instantiates
patterns/leader-driven-safe-deletion-watermark — the specific watermark aggregation shape
patterns/epoch-stamp-on-object-id-for-gc — analogous pattern in Cloud Topics L0 GC
patterns/per-partition-rsm-for-gc-tracking — analogous per-partition state machine for GC
concepts/epoch-based-distributed-gc — the GC family this pattern belongs to
systems/redpanda