Skip to content

Lakebase branching for CI/CD: lessons from 100 PR databases

Summary

A Databricks (Tier-3) customer case study of Snorkel AI using Lakebase database branching to give each CI environment its own production-like database. Snorkel's CI runs on Kubernetes: every open PR (plus ad-hoc commits) gets an ephemeral "PR namespace". Production is an Amazon Aurora PostgreSQL cluster, and restoring a copy per namespace took ~20 minutes — too slow for a merge queue that needs a fresh environment on every run. The new design: AWS DMS replicates Aurora into the Lakebase project's main branch (full load, then ongoing CDC, ~2 min behind production), which becomes the "golden copy". CI calls the Lakebase API to create a copy-on-write branch per PR with a TTL, injects the connection details into the namespace, runs tests (migrations, writes, schema changes) against real data, and deletes the branch on PR close — with TTL as the backstop. Developers authenticate via Okta SSO; pipelines run as per-team / per-repo service principals that fetch short-lived OAuth tokens; all traffic stays on private networking. The post's most useful non-obvious lesson is about cost drift of long-lived branches: because a branch pins the pages that existed at fork time while CDC keeps rewriting main, a branch that lives too long gradually stops sharing storage — so TTLs are framed as required, not optional.

Key takeaways

  1. Restore-per-environment does not scale to a merge queue; copy-on-write branching does. A ~20 min Aurora restore per PR namespace (the results section says "over 30 minutes" end-to-end) was replaced by a branch that is "ready in under a minute" because creation is a metadata operation independent of database size — "a ~1 TB production copy takes about as long as … a 1 GB database." Target: p95 < 60 s. (Source: sources/2026-10-31-databricks-lakebase-branching-for-cicd-lessons-from-100-pr-databases)

  2. A continuously replicated main branch is the branching root. Because Lakebase is standard Postgres, AWS DMS targets it with no custom tooling: full load then CDC — a textbook patterns/snapshot-plus-catchup-replication — keeps main ~2 min behind Aurora. Full LOB mode carries 10–20 MB JSONB values. Snorkel already ran DMS, so there was no new pipeline to own. Every PR branch forks from this fresh golden copy rather than from a stale snapshot.

  3. Per-branch storage is proportional to writes, not database size. Each PR branch writes ~5–10 GB on top of a ~1 TB shared base. Combined with per-branch autoscaling compute that scales to zero, ~100 provisioned branches need compute for only ~20–30 concurrently active ones at ~5 QPS each at peak.

  4. Long-lived branches silently lose the copy-on-write discount. A branch keeps the page versions from its fork point; as CDC keeps updating main, the pages the branch still references diverge from main's current pages, so the branch's effective unique storage grows over time even with zero writes of its own. TTL on every CI branch keeps total storage ≈ production base + active-PR deltas. Corollary best practice: create a new branch instead of refreshing an old one (seconds to create), and monitor storage per branch.

  5. Explicit delete + TTL backstop. The pipeline deletes the branch on merge, close, or test pass; if cleanup never runs (crashed job, abandoned PR) the branch expires on its own — the patterns/ttl-based-resource-lifecycle shape applied to databases.

  6. Split identity by caller type. Interactive developers: Okta federated to the Databricks account, users provisioned via SCIM (concepts/sso-authentication). Non-interactive CI: each PR namespace runs as a Databricks service principal (scoped per team or repo, also SCIM-provisioned) and obtains short-lived Lakebase OAuth tokens via the Databricks SDK, refreshing them during long test runs — no long-lived DB passwords in CI (concepts/workload-identity). Postgres audit logs record the service-principal identity per query, so every query traces to a pipeline (concepts/audit-trail).

  7. Private networking was a hard gate. Production data flows Aurora → Lakebase → CI cluster over private connectivity, never the public internet; this plus identity requirements "ruled out other options". The developer quote: prior physical-copy jobs "took hours and frequently ran into network and other bottlenecks" across CI platforms.

  8. Same primitive for AI agents. One branch per agent task; dozens of parallel agents cost ≈ their deltas; review the branch's changes before applying to production, or delete the branch — "nothing to roll back"; TTLs clean up abandoned agent branches; agents run as service principals with short-lived tokens so their queries are auditable. Reinforces branch-per-agent.

Architecture

Aurora PostgreSQL (prod)
   │  AWS DMS: full load → ongoing CDC (Full LOB mode), private network
   ▼
Lakebase project: main branch  ("golden copy", ~2 min behind prod, ~1 TB)
   │  CI → Lakebase API: create branch from main + set TTL      (<60 s)
   ├──► pr-101 branch ── own autoscaling compute (scale-to-zero) ──► K8s PR namespace
   ├──► pr-102 branch ── …                                        ──► K8s PR namespace
   └──► … (~100 branches; ~20–30 active at once, ~5 QPS each)
        on PR merge/close/test-pass → delete branch; TTL expiry as backstop
Identity: devs = Okta SSO (SCIM); CI = service principal → short-lived OAuth (SDK)

Systems / concepts / patterns extracted

Operational numbers

Metric Value
Old per-environment Aurora restore ~20 min (post also says "over 30 minutes")
New branch provisioning target < 60 s p95
Concurrent PR branches at peak ~100 (designed for)
Branches actively serving at once ~20–30, ~5 QPS each at peak
Base size ~1 TB (architecture section); ~2.5 TB "production dataset" (results)
Per-branch write delta ~5–10 GB
Main-branch replication lag vs Aurora ~2 min (DMS CDC)
Largest JSONB/LOB values 10–20 MB
Rollout 60–70 developers across teams

Caveats

  • Vendor customer story with self-reported numbers. The published results still contain an editorial placeholder — "under 60 seconds with Lakebase [VERIFY p95 measured]" — so the p95 claim is a success criterion, not a confirmed measurement.
  • Internal inconsistencies: baseline restore "~20 minutes" vs "over 30 minutes"; base "~1 TB" vs "~2.5 TB production dataset". "Designed for" 100 concurrent branches — not stated as observed.
  • Compatibility is asserted as "no major compatibility issues" and "performance comparable to Aurora" with no benchmarks.
  • No detail on how PR schema migrations are reconciled with DMS CDC on main (CDC only writes main, so branches are unaffected after fork — but the post does not discuss DMS behaviour during production schema changes).
  • Mechanism depth (pageserver layers, branch-at-LSN) lives in earlier Lakebase sources, e.g. sources/2026-10-01-databricks-lakebase-postgres-branch-based-restores-for-fast-recovery-at-scale.

Source

Last updated · 854 distilled / 2,448 read