Skip to content

CONCEPT Cited by 1 source

Pluggable persistence layer

Definition

A pluggable persistence layer is a storage architecture where each logical tier of a data structure (WAL, data files, manifest/metadata) can be independently backed by a different physical storage substrate — local disk, object storage, a consensus log, etc. — without changing the core data structure logic.

In the LSM-tree context, this means the memtable flush target, SSTable storage, manifest storage, and WAL substrate are all independently configurable rather than uniformly bound to local disk.

Motivation

Traditional LSM databases (LevelDB, RocksDB) assume all persistence layers live on local disk. When metadata must survive node loss or scale beyond single-node capacity, each layer needs a different backing store tuned to its access pattern:

  • SSTables → object storage (scale) + local cache (latency)
  • Manifest → object storage (durability) + Raft (availability for failover)
  • WAL → Raft log (replicated durability)

(Source: sources/2026-06-09-redpanda-cloud-topics-the-metastore)

Trade-offs

  • Pro: Each layer gets the optimal durability/latency/cost profile for its access pattern.
  • Pro: Metadata scales beyond single-node memory/disk.
  • Con: More moving parts; failure modes per layer differ.
  • Con: Consistency across layers requires careful coordination (e.g., manifest must be consistent with SSTable set).

Seen in

Last updated · 542 distilled / 1,571 read