PATTERN Cited by 1 source
Compaction epoch optimistic concurrency¶
Context¶
When compaction is decoupled from partition leadership and any node can compact any partition (as in Cloud Topics), two nodes might attempt to compact the same partition concurrently. Without coordination, a stale compactor could re-register already-superseded data in the log.
Solution¶
The metastore maintains a per-partition integer compaction_epoch, incremented on every successful compaction metadata update. A compactor reads the current epoch before starting, performs the rewrite and multi-part upload, then submits an update to the metastore. The metastore rejects the update if the epoch has advanced since the compactor's read — a simple compare-and-swap.
Consequences¶
- Correctness guaranteed — stale data is never re-added to the log
- Lock-free — no distributed locks or leases required; conflicts resolved at commit time
- Wasted work on conflict — the losing compactor discards its upload, but this is rare in practice and not a correctness issue
- Simple implementation — a single integer comparison replaces complex distributed locking