CONCEPT Cited by 7 sources

Compute–storage separation¶

Compute–storage separation is the architectural property where a system's persistence layer and its query/compute layer are decoupled and scale independently. Storage is a durable, shared substrate (object store, shared log, distributed KV); compute is a pool of workers that can be added/removed without touching the data.

Why it matters¶

Scale the expensive resource independently. Aggregation workloads are compute-heavy but bounded by data size; you can burst compute up for the job without over-provisioning storage, and vice-versa.
Multiple compute tiers against one source of truth. Serving, analytics, backfill, and disaster-recovery workers can all read the same storage without replica sprawl.
Elastic pricing. Compute goes to zero when idle; storage bill stays flat. Aligns with a pay-per-use model (see elasticity).
Horizontal aggregation. An OLAP engine with separated compute can parallelise an aggregation across many workers that all read the same storage; most of the calculation ends up in memory.

Contrast with shared-nothing OLTP¶

Classic MySQL / Postgres setups bundle compute and storage on the same box. To scale, you grow the box (Canva doubled RDS instance size every 8–10 months until the model broke) or shard the data, which complicates the application. OLAP warehouses like Snowflake split these layers: storage lives in the cloud object layer, compute is a "virtual warehouse" cluster you size independently. Canva explicitly calls this out as the reason Snowflake can aggregate billions of records in a few minutes, "several orders of magnitude faster" than the MySQL round-trip approach. (Source: sources/2024-04-29-canva-scaling-to-count-billions)

Seen elsewhere¶

S3 + analytics engines — S3 is the durable store; engines like Spark, Trino, Athena, and Snowflake run compute over the same object data. (systems/aws-s3, systems/apache-iceberg)
Aurora DSQL — separates the journal (durability) from the adjudicator/crossbar/storage/execution tiers, each scaling on its own axis. (systems/aurora-dsql)
Lakebase — Neon-descended serverless Postgres; externalises page and WAL storage into systems/pageserver-safekeeper, leaving the Postgres compute VMs ephemeral and scale-to-zero. A cleaner example of the OLTP-shape separation than DSQL (which keeps Postgres's page/WAL layer and rewrites replication/concurrency). (systems/lakebase)
Lambda — stateless compute with state pushed to managed stores is the same separation at the request-level. (concepts/stateless-compute)

Caveats¶

Query latency ≠ OLTP latency. Compute is elastic but not free to spin up; cold queries can be seconds. OLAP stores aren't a serving tier — see patterns/warehouse-unload-bridge.
Network between layers matters. Separation implies moving data or metadata between compute and storage; engine designs work hard on caching, local-disk staging, and predicate push-down to keep that in check.
Consistency story gets more complex. The storage layer must give the compute layer enough consistency guarantees for correctness; S3 moving to strong read-after-write consistency in 2020 is one enabling step for this (see concepts/strong-consistency).

Seen in¶

sources/2026-05-07-databricks-how-lakebase-architecture-delivers-5x-faster-postgres-writes — First canonical wiki instance of compute-storage separation enabling structural elimination (not just relocation) of a durability primitive. Classical Postgres's Full Page Write exists to tolerate torn pages on the local-disk page heap. On Lakebase's stateless-compute-streaming-WAL-to- safekeeper-quorum architecture, "there is no local-disk page to tear, the failure mode FPW was designed to prevent simply does not exist" — FPW is structurally redundant on compute. Databricks disabled compute-side FPW across the global fleet; the incidental read-path reset-point role FPW had on the pageserver's delta-chain replay is preserved by patterns/image-generation-pushdown-to-storage — image generation moved from the compute's WAL stream to the storage tier's background processing, with per-page-threshold cadence replacing checkpoint-scoped FPW cadence. First wiki disclosure of the Paxos-based safekeeper quorum as the durability substrate that makes stateless compute structurally safe. Quantified outcomes: 94% compute WAL-volume reduction; 5× write throughput at 32 vCPU; p99 read latency −30% to −50%. This ingest extends the four prior Lakebase sources into a five-axis canonicalisation of compute-storage separation at serverless-Postgres altitude: (a) forcing function for two-tier encryption + CMK revocation (2026-04-20); (b) operational payoff for bursty workloads — no data movement on compute spin-up (2026-04-27); (c) enabler for agent-driven per-request lifecycle (2026-04-29); (d) enabler for PITR as a routine sub-10-second operation (2026-04-30); (e) enabler for elimination of durability primitives specific to local-disk failure modes (2026-05-07).
sources/2026-04-30-databricks-backstage-with-lakebase — Fourth Lakebase-altitude confirmation; new axis: PITR-as-storage-fork-cheap-enough-to-be-routine. Thoughtworks Backstage POC demonstrates that concepts/point-in-time-recovery collapses into a copy-on-write storage fork at a past timestamp only because the storage tier is independent of compute + keeps historical page versions as durable shared state. Measured: 3.78 seconds end-to-end PITR for a 32-row deletion recovery; same primitive as forward-branching (1.09 s for 63 MB Backstage catalog), differing only in the source_branch_time parameter. Canonicalises the architectural unification [[patterns/branching-is-pitr- with-time-now]]: on a compute-storage-separated substrate with copy-on-write storage, branching and PITR are the same operation with different time inputs. Extends the four Lakebase sources into a four-axis canonicalisation of compute-storage separation at serverless-Postgres altitude: (a) forcing function for two-tier encryption
CMK revocation (2026-04-20); (b) operational payoff for bursty workloads — no data movement on compute spin-up (2026-04-27); (c) enabler for agent-driven per-request lifecycle (2026-04-29); (d) enabler for PITR as a routine sub-10-second operation rather than last-resort disaster recovery (2026-04-30). The common thread: decoupling storage from compute doesn't just scale-independently — it makes every storage-addressing operation (clone, fork, recover, historical-query) structurally cheap because compute is stateless + disposable.
sources/2026-04-29-databricks-and-stripe-projects-infrastructure-built-for-agents — Compute-storage separation as agent-lifecycle enabler. Third canonical cross-source confirmation on the wiki that compute-storage separation is the load-bearing property behind Lakebase's serverless- Postgres contract. Verbatim: "By decoupling compute from storage, agents can create, build, and tear down OLTP databases in seconds." New axis the prior confirmations didn't cover: the separation as the forcing function for per-request compute lifecycle at agent-initiated cadence — not just bursty workload tolerance (LangGuard 2026-04-27) or two-tier encryption (CMK 2026-04-20). Creating an OLTP database becomes a sub-350 ms operation because storage already exists and only compute needs standing up; tearing one down is cheap because only compute is de-allocated. This makes database instances structurally disposable — the pre-condition for concepts/agent-provisioned-database as a resource primitive. The three Lakebase sources now form a coherent canonicalisation of compute-storage separation at serverless-Postgres altitude: (a) forcing function for two-tier encryption + CMK revocation semantics (2026-04-20); (b) operational payoff for bursty workloads — no data movement on compute spin-up (2026-04-27); (c) enabler for agent-driven per-request lifecycle (2026-04-29).
sources/2026-04-27-databricks-inside-one-of-the-first-production-deployments-of-lakebase-langguard — LangGuard on Lakebase: the second production-deployment datapoint for Lakebase's compute/storage-separated Postgres. Canonicalises the specific operational payoff for bursty agentic workloads: "Because durable state lives in the storage layer, not in the compute node, spinning up a new compute instance requires no data movement. It simply attaches to the existing database history and begins serving queries immediately." No cold-start penalty on the data side — only on the compute-VM side — which is what makes scale-to-zero between bursts operationally viable for a latency-sensitive-enforcement workload. Former IBM QRadar team frames this as the "answer we had always needed but didn't have access to" when building SIEM at petabyte scale.
sources/2024-04-29-canva-scaling-to-count-billions — Snowflake's separated compute is the reason Canva's end-to-end recompute is feasible at billions-per-month scale.
sources/2026-04-20-databricks-take-control-customer-managed-keys-for-lakebase-postgres — Lakebase's Pageserver (durable pages) + Safekeeper (durable WAL) storage tier is independent of the ephemeral Postgres compute VMs that scale up/down/to-zero; the separation is explicitly called out as the forcing function motivating a two-tier encryption design (persistent envelope + per-boot ephemeral keys).
sources/2024-07-29-aws-amazons-exabyte-scale-migration-from-apache-spark-to-ray-on-ec2 — Amazon Retail's post-Oracle BI stack named as the canonical application of compute-storage separation: S3 for storage, swappable compute engines on top (systems/amazon-redshift, systems/aws-rds, systems/apache-hive, systems/apache-spark on systems/amazon-emr, later systems/ray on systems/aws-ec2). The architectural payoff is named explicitly: swapping compute engines is architecturally cheap because the source of truth — S3 — doesn't move. BDT's migration of the compactor workload from Spark to Ray for the largest ~1% of tables is the case study — storage untouched, compute engine replaced, 82% better cost per GiB.
— PlanetScale's structural pushback: compute-storage separation via EBS imposes a variance floor that translates directly into customer-visible partial failure on OLTP workloads ("this is the cost of separating storage and compute and the sheer complexity of the software and networking components between the client and the backing disks"). The post argues OLTP databases should go the other way — re-collocate compute and storage on direct-attached NVMe + pay for durability via cluster replication. See systems/planetscale-metal + patterns/shared-nothing-storage-topology + patterns/direct-attached-nvme-with-replication. Canonical wiki datapoint against compute-storage separation as the default for low-latency OLTP.
— Brian Morrison II (PlanetScale, 2024-01-24). Canonical OLTP illustration of compute-storage separation via Amazon Aurora: the writer compute node and read-only compute nodes share the same distributed storage substrate (10 GiB segments × 6 copies × 3 AZs, concepts/aurora-storage-quorum|4-of-6 write-ack quorum). "Since data is replicated on the storage level, read-only compute nodes can be started at any time in an availability zone containing a copy of the data for that node to read." Unlike the analytics/warehouse instances (Snowflake, Databricks), Aurora runs the separation at the OLTP tier with writer-driven page-update notifications to keep reader buffer caches coherent. See patterns/storage-forwarded-redo-log-replication for the full substrate pattern. This is the structural counter- model to PlanetScale's shared-nothing direct-attached-NVMe substrate — both architectures ship under the Aurora-style "primary + read replicas" surface but use opposite substrate primitives underneath.