Skip to content

DATABRICKS 2026-10-08

Read original ↗

Load terabytes of data in minutes into Lakebase Postgres

Summary

Databricks describes how the LTAP architecture lets Lakebase offload bulk loads of operational Postgres data onto Spark instead of funnelling every row through the Postgres primary. Traditional Postgres bulk ingest is bottlenecked by a single writer: even a distributed Spark job that reads source partitions in parallel must stream every row through one Postgres primary via COPY, which turns rows into heap/index pages and writes WAL on the exact instance serving live traffic — a noisy neighbor that forces teams to overprovision, run loads off-hours, and manage checkpointing by hand. Because Lakebase's transactional compute is stateless and durable state lives in the Pageserver/Safekeeper tier on object storage (compute–storage separation), "Postgres is a client of storage" — it no longer has to be the process that materializes every new page. Spark executors build valid Postgres heap and index pages in parallel inside sandboxed Postgres instances, upload them to object storage, and the primary publishes the result with a single compact WAL record. This is the write-path accelerator behind Synced Tables: a customer loading ~1 billion rows/day saw an 8+ hour, CPU/memory- saturating load replaced by one that leaves live traffic untouched, and internal benchmarks load 1 TB in under 5 minutes.

Key takeaways

  1. The single-writer bottleneck is the whole problem. "Traditional Postgres architectures make the primary the sole gatekeeper of durable state. Bulk loads are forced through this single bottleneck, meaning every imported row generates heap pages, updates indexes, and writes WAL records on the exact same instance serving your live application traffic." Spark can scale the source (parallel partition reads) but not the destination — every row still passes through one Postgres COPY writer, then WAL, then an fsync to durable storage before commit. Adding executors speeds the scan, not the write. (Source: this article)

  2. LTAP removes the gatekeeper. Transactional compute is stateless; durable state lives in the distributed storage layer, not on the primary's local disk. So "Spark can build Postgres state and write it into storage, while the primary only publishes the result." Operational consequences: bulk loads don't compete with OLTP for CPU/I/O/connections/WAL bandwidth; multiple large loads into the same destination run concurrently (each ends in a single WAL record, so they don't queue behind each other's COPY streams); and a 1 CU primary keeps serving traffic while Spark loads billions of rows on separate, disposable compute. (Source: this article)

  3. Executors build the heap in parallel using pg_upgrade's binary-upgrade mode. Each Spark executor launches a sandboxed Postgres instance in binary-upgrade mode — the same mechanism pg_upgrade uses to preserve catalog OIDs across major-version upgrades. The driver transplants the destination's catalog OIDs into each sandbox (so OIDs embedded in generated pages match the destination) and assigns non-overlapping OID ranges so concurrently-created objects can't collide. Inside each sandbox a binary COPY with FREEZE builds that executor's slice of the heap; freezing marks tuples as already-committed so Postgres treats them as visible without consulting the sandbox's transaction history. (Source: this article)

  4. Each generated page is checksummed; the pageserver validates on ingest. Workers compute checksums for every generated page; Lakebase's pageservers validate those checksums when they ingest the files, detecting corruption before imported pages become authoritative. Together — binary-upgrade mode, frozen tuples, coordinated OIDs, checksum validation — Spark produces pages the destination reads as ordinary Postgres pages, implemented via Postgres extensions and a custom table access method, "without modifying Postgres core." Disposable sandboxes also enable UNLOGGED helper tables (an optimization, not a safety mechanism — no app traffic shares the sandbox, so no WAL/lock contention). (Source: this article)

  5. The index is built without downloading the heap — the clever bit. A B-tree builder only needs the stream of (key, ctid) pairs a heap scan would produce, not the heap contents. Each heap worker exports its key columns + tuple IDs to a separate object-storage file while building its slice. Those records feed a helper table holding just the key columns + tuple ID; during CREATE INDEX, a custom table access method scans the helper table like heapam but writes the real ctid into each index entry. Crucially, each slice-local ctid is shifted by the cumulative size of the preceding heap slices, so it points to the tuple's final location in the concatenated heap. Postgres's standard B-tree builder runs unchanged and produces an index over a heap it never downloaded — "this replaces movement of the entire heap with movement of the much smaller key-and-tuple-ID representation." (Source: this article)

  6. Handoff is one SQL function + a manifest; the WAL cost is a single record. Once heap and index slices are in object storage, the Spark driver writes a manifest and invokes one SQL function on the destination. The primary records the import as "a compact WAL record" — "A conventional COPY would send the full data volume through the safekeeper quorum. We only send the import description." Pageservers claim the uploaded files as authoritative pages, validate checksums, and prewarm them onto local SSD so initial reads don't pay a cold object-storage fetch. The primary then atomically swaps the staged data into the user-visible synced table; until that transaction commits the imported table stays isolated, and afterward the primary sees "ordinary heap and B-tree pages." (Source: this article)

  7. Measured impact. Beta customer: ~1 billion rows/day via Synced Tables; previously 8+ hours, saturating CPU and memory and forcing heavy OLTP overprovisioning "just to survive the sync." After offloading to Spark via LTAP: load throughput scales "almost linearly as your data grows," the primary is unaffected, and internal benchmarks load 1 TB in under 5 minutes. Caveat stated in-post: "This benchmark measures data load time (building heap pages). We are actively working on parallelizing index builds for this loaded data as well." (Source: this article)

Systems / concepts / patterns extracted

Operational numbers

  • Beta customer bulk load: 8+ hours → unaffected primary after LTAP offload; previously saturated CPU + memory and forced OLTP overprovisioning.
  • Internal benchmark: 1 TB loaded in < 5 minutes (heap-build time only; index builds not yet parallelized for the loaded data).
  • Throughput scales "almost linearly" with data size (distributed-engine property), on compute that terminates as soon as the load finishes.
  • A 1 CU primary keeps serving traffic while Spark loads billions of rows on separate compute.
  • Each concurrent load ends in one WAL record on the primary rather than a full COPY stream, so loads don't serialize behind one another.

Caveats / limitations

  • Index builds for the loaded data are not yet parallelized — the sub-5-minute 1 TB figure is heap-build time; the post says this is actively being worked on.
  • Vendor-specific and partly aspirational: this is a Lakebase/LTAP capability; benchmark numbers are Databricks-internal; the post closes by saying they want to "generalize this protocol to handle large operations and maintenance, like building indexes or even migrations" — i.e. the generalization is future work.
  • The mechanism is tied to Postgres internals (catalog OIDs, pg_upgrade binary-upgrade mode, COPY FREEZE, table access methods, page checksums). It's "just Postgres" in the sense that the output is ordinary Postgres pages, but it depends on a specific extension + table-access-method implementation around that path.
  • Positioned as the acceleration layer for Synced Tables and a replacement for manual ReverseETL jobs from the Lakehouse; broader applicability (arbitrary user bulk loads vs. governed Delta→Postgres syncs) is implied more than detailed.

Source

Last updated · 854 distilled / 2,448 read