PATTERN Cited by 1 source

Raft log as LSM WAL

Pattern

Replace the traditional file-based write-ahead log of an LSM database with a Raft consensus log. Writes to the LSM are submitted as Raft proposals; once a majority of the Raft group acknowledges, the write is durable and ordered.

When to use

• The LSM store must be replicated for high availability anyway.
• A separate WAL + separate replication layer would duplicate durability work.
• Fast failover is required — the new leader already has the WAL (Raft log).
• The system already operates Raft groups (e.g., a streaming broker like Redpanda).

Mechanism

1. Client write → Raft propose → majority ack → applied to in-memory memtable.
2. Memtable flush → manifest update also committed via Raft.
3. Raft log entries prior to manifest commit can be truncated (they're subsumed by the flushed SSTables).

Trade-offs

• Gains: unified ordering, single durability path, replicated state for failover.
• Costs: Raft majority latency per write (mitigated by batching); log size management couples with Raft compaction.

(Source: sources/2026-06-09-redpanda-cloud-topics-the-metastore)

Seen in

• systems/redpanda-cloud-topics-metastore — each metastore partition's WAL is a Raft group within an internal Redpanda topic