Skip to content

DATABRICKS 2026-09-30

Read original ↗

A practical guide to cost optimization with Lakebase Postgres

Summary

Databricks lays out where Lakebase Postgres cost efficiencies come from and how to configure for them. The thesis is that separated storage and compute plus a serverless compute layer make several otherwise-expensive database operations cheap by design — branching, read replicas, and high availability all add compute without duplicating storage; autoscaling with scale-to-zero bills compute to actual demand. On top of those built-ins the post gives concrete operational levers: sync only the working set of Lakehouse data into Lakebase via Synced Tables (not whole Delta tables), match the sync mode (Snapshot / Triggered / Continuous) to how fresh the data really needs to be, bin-pack multiple tables into one sync pipeline, right-size compute to the working set (not the on-disk database size), put a connection pooler in front to avoid the max_connections ceiling, and tune the PITR window + snapshot schedule to actual recovery requirements. Costs fall into three metered buckets: database compute, database storage (branch / PITR / snapshot), and Synced-Table pipeline compute — all queryable from system.billing.usage.

Key takeaways

  1. Separation of storage and compute is the root cost lever. Branches, read replicas, and HA all add compute that reads the same underlying storage layer as the primary, so none of them duplicates the storage footprint. "Read replicas are independent compute instances that read from the same underlying storage layer as the primary, so scaling read capacity does not require creating and paying for another copy of the database." (Source: this article; architecture background: sources/2026-05-07-databricks-how-lakebase-architecture-delivers-5x-faster-postgres-writes).

  2. Branching shares storage via copy-on-write. "Unlike approaches that require creating a separate physical copy of the database for each environment, Lakebase branches share the same underlying storage and track changes as the branch diverges from its parent." This makes short-lived dev/test environments with production-like data cheap — you pay only for the divergence, not a full copy. See concepts/database-branching + concepts/copy-on-write-storage-fork.

  3. Autoscaling + scale-to-zero bills compute to demand. You set a min/max CU range; compute scales within it and, with scale-to-zero on, suspends entirely after inactivity, dropping compute cost to zero. "Compute resumes from scale to zero in a few hundred milliseconds" — attractive for dev workflows, non-prod app variants, and prod apps that don't need double-digit -ms latency. See concepts/scale-to-zero + concepts/elasticity.

  4. "Always on" pricing is a 25% discount for compute that can't scale to zero. When scale-to-zero is turned off, Lakebase applies always-on pricing — a 25% discount on baseline capacity. This also covers compute that cannot scale to zero, such as HA configurations. The cost model deliberately doesn't punish you for the workloads that must stay warm.

  5. Sync only the working set, not the whole Delta table. A common misstep: pushing a large Delta table into Lakebase when the app only queries a small active subset. This inflates Lakebase storage, raises sync cost, and can degrade performance. The fix: define exactly what the app needs with a Materialized View (e.g. a rolling 60-day window) and sync just that. The MV's automatic change data feed lets the sync pipeline compute row-level changes (including deletions as rows age out of the rolling window) and propagate them incrementally — a cost-efficient reverse-ETL pattern where the fresh, low-latency active subset lives in Lakebase while full history stays in Delta. See concepts/change-data-capture.

  6. Match the sync mode to required freshness — the single biggest Synced-Table cost knob. Synced Tables are managed Serverless Spark Declarative Pipelines under the hood and run for the duration of the sync, so cost is driven by data volume moved and Lakebase instance size (CUs). Three modes:

  7. Snapshot — full copy each cycle. Best when >10% of rows change per cycle; up to 10× more efficient than Triggered for highly-volatile or infrequently-updated tables.
  8. Triggered — incremental (insert/update/delete) on demand or on an interval. The cost/latency sweet spot for incremental workloads; pair with a table-update trigger to run only when source data actually changes (near-continuous freshness on a budget).
  9. Continuous — real-time streaming, seconds of lag, highest cost because the pipeline runs continuously and consumes compute even when there are no changes.

Rule of thumb: avoid long intervals between runs — a massive backlog makes the next sync slow and costly.

  1. Bin-pack multiple tables into one sync pipeline. The same pipeline can sync changes from multiple Delta tables into Lakebase, letting those tables share the same underlying compute. Especially valuable for Continuous syncs (always-running), where it avoids paying for a separate always-on pipeline per table. See concepts/bin-packing.

  2. Right-size compute to the working set, not the disk size. "The single most important input to sizing is your working set: the data and indexes your application accesses frequently, as opposed to the full size of your database on disk. A 2500 GB database with a 20 GB hot working set does not need 2500 GB of RAM." RAM scales linearly with compute size, and up to 75% of a compute's RAM is available as compute cache; when the working set fits the cache, most reads are memory hits (fast + consistent latency), otherwise Postgres fetches missing pages from storage (slower, with latency variability). Sizing is largely "choose a compute whose cache exceeds your working set, plus headroom" — but also weigh query complexity, concurrency, and latency targets. See concepts/working-set-memory + concepts/cache-hit-rate.

  3. max_connections is set by compute size — pool rather than oversize. The hard ceiling on concurrent Postgres connections is determined by compute size, and for an autoscaling compute follows a specific rule: the limit is the smaller of (maximum CU) and (8 × minimum CU). So raising the maximum only adds connections up to 8× the minimum; a small minimum caps how far a larger maximum can take you. Apps that open many connections hit this ceiling and start rejecting new connections. The fix: factor connection volume into your minimum CU, and put a connection pooler in front (supports up to 10,000 concurrent client connections) — cheaper than sizing compute up purely to raise the connection ceiling. See concepts/connection-pool-exhaustion + systems/pgbouncer.

  4. Undersizing shows up as symptoms, not failures. The post enumerates the tells: slow/inconsistent latency (p95/p99 climb while median looks fine as reads miss cache and hit storage); a falling cache hit ratio (the leading indicator, slips before latency visibly degrades); CPU saturation + query queuing; "too many clients" connection errors; and a cold cache penalty after a scale-to-zero wake or an autoscale-up (cache starts empty and must re-warm) — which is why minimum compute size matters as much as the maximum. The Lakebase Metrics dashboard reports working-set size over 5-min / 15-min / 1-hour windows alongside available compute cache; for stable access patterns, compare the 1-hour working set against available cache.

  5. Tune PITR window + snapshots to actual recovery needs. PITR continuously retains history to restore to any moment within a configurable 2–30 day window; snapshots are discrete captures (manual or scheduled daily/weekly/monthly). PITR storage grows with write activity × window length, so a long window on a write-heavy app is expensive. Cost-conscious approach: pick the shortest PITR window that meets incident-recovery needs and supplement with scheduled snapshots for longer-term recovery points. Both are cheaper than regular branch storage — snapshot storage $0.090/GB-month (~74% cheaper), PITR storage $0.200/GB-month (~42% cheaper) — and scheduled snapshots are incremental after the first full one. Use PITR for unexpected incidents (accidental deletes, bad writes); use snapshots for planned recovery points (before a risky migration/bulk update). See concepts/point-in-time-recovery + concepts/rpo-rto.

  6. Three metered cost areas, all in billing system tables. (1) Compute — metered on CU usage over time (follows autoscaling). (2) Storage — branch storage, PITR history, snapshot storage, each metered separately. (3) Synced-Table pipeline compute — the managed serverless pipeline is billed separately from the Lakebase database compute. All queryable from system.billing.usage, with storage broken down by product_features.lakebase.storage_type: BRANCH_DATA_STORAGE (non-expiring branches), BRANCH_CHANGE_STORAGE (changed data for expiring branches), BRANCH_HISTORY_STORAGE (PITR history). Join to system.billing.list_prices to estimate daily cost at list price (contractual discounts not reflected). Synced-Table pipeline cost is found by filtering on the pipeline ID.

Operational numbers

  • Default new-project compute: autoscale 8–16 CU, scale-to-zero after 24 hours of inactivity (resize at provisioning time via SDK or Declarative Automation Bundles rather than after-the-fact).
  • Scale-to-zero resume: a few hundred milliseconds.
  • Always-on pricing discount (scale-to-zero off): 25% off baseline capacity.
  • Compute cache: up to 75% of a compute's RAM.
  • max_connections (autoscaling): min(max CU, 8 × min CU).
  • Connection pooler ceiling: up to 10,000 concurrent client connections.
  • Snapshot-mode guidance: use when >10% of rows change per cycle; up to 10× more efficient than Triggered in those scenarios.
  • PITR window: configurable 2–30 days.
  • Storage pricing: snapshot $0.090/GB-month (~74% cheaper than branch storage), PITR $0.200/GB-month (~42% cheaper).
  • Working-set dashboard windows: 5 min / 15 min / 1 hour.

Caveats / undisclosed

  • The CU→(vCPU, RAM) mapping and exact autoscale step granularity are not specified here (see sources/2026-08-31-databricks-autoscaling-lakebase-postgres for the autoscaling mechanism).
  • The regular branch-storage $/GB-month baseline is given only relatively (snapshot ~74% cheaper, PITR ~42% cheaper) — the absolute branch rate is not stated.
  • The SDK / DABs code snippets for setting initial compute range are referenced but not reproduced in the raw article body.
  • Pricing is list price only; customer-specific contractual discounts are not reflected in the billing-table cost estimates.
  • This is a Tier-3 (Databricks) vendor post with a cost-optimization framing, but it clears scope on real system internals: storage/compute separation economics, COW branching, CDC-based incremental reverse-ETL, working-set-driven sizing, and the max_connections autoscaling rule.

Source

Last updated · 766 distilled / 2,225 read