PATTERN Cited by 1 source
Storage-layer unification over engine unification¶
Definition¶
Instead of building a single engine that handles both transactional (OLTP) and analytical (OLAP) workloads (the HTAP approach), unify the two worlds at the storage layer — keep the best engine for each job (e.g., Postgres for ACID transactions, Spark/Lakehouse for analytics) but have them share a single durable copy of data in an open columnar format.
Motivation¶
HTAP systems historically fail because: 1. A single engine cannot match the feature depth of mature, specialized engines. 2. A new engine lacks the ecosystem (drivers, extensions, operational knowledge) of established systems. 3. Transactions and analytics contend for the same hardware without performance isolation.
By unifying at storage rather than compute, each engine retains its full feature set, ecosystem, and independent scaling/isolation, while eliminating the data duplication and replication lag of CDC pipelines.
Mechanism (as implemented in Lakebase LTAP)¶
- The PageServer transcodes Postgres row data into Parquet columnar layout as it materializes pages to object storage.
- Open table formats (Delta/Iceberg) provide snapshot-consistent views for analytical engines.
- Analytical queries ask Postgres for the current LSN, read bulk from object store, and merge only the recent unmaterialized delta from PageServer.
- Every table is automatically available for analytics — no opt-in, no replication pipeline.
Trade-offs¶
| Advantage | Limitation |
|---|---|
| No CDC pipeline to build/monitor/break | Requires tight coupling between PageServer and table format evolution |
| Zero replication lag (LSN-consistent) | Columnar transcoding adds CPU load to storage tier |
| Full performance isolation (separate engines) | Novel architecture; production maturity still being proven |
| Full ecosystem support on both sides | Very small tables skip columnar conversion (minor gap) |
| One governed copy = no permission drift | Analytical freshness depends on materialization throughput |
Seen in¶
- systems/lakebase — Databricks' LTAP architecture (Source: sources/2026-06-30-databricks-from-monolith-to-lakebase-to-ltap)