Skip to content

PATTERN Cited by 1 source

Foreign-catalog federation for operational-DB governance

Pattern

Expose the operational database as a foreign catalog inside the analytical platform's unified catalog, rather than governing it independently with database-tier RBAC. Once the operational DB is a foreign catalog, all the analytical platform's governance primitives — GRANT model, audit substrate, cost-attribution substrate, ABAC policies, masking policies — apply to operational reads and writes the same way they apply to analytical assets. The compliance side- channel that the operational DB historically created collapses into a single substrate.

Canonical instance

Backstage with Lakebase, Part 2: Governance (Thoughtworks, 2026- 05-15).

The Thoughtworks Backstage POC exposes the Backstage Postgres database (running on Lakebase) as a foreign catalog lakebase_bs in Unity Catalog via Lakehouse Federation. Verbatim:

"Because Lakebase is natively embedded inside Databricks, Unity Catalog extends directly over the operational Postgres database. In this POC, we used Lakehouse Federation to expose the Backstage catalog as a foreign catalog (lakebase_bs) in Unity Catalog. Once it's there, standard UC grants control who can see what, no Postgres-level role management required."

Architectural shape

                           ┌─────────────────────────────────┐
                           │   Unity Catalog (one substrate) │
                           │                                 │
                           │   GRANT model                   │
                           │   system.access.audit           │
                           │   system billing tables         │
                           │   ABAC + governed tags + masks  │
                           └────────────┬────────────────────┘
                                        │
                  ┌─────────────────────┴────────────────┐
                  │                                      │
                  ▼                                      ▼
        ┌───────────────────┐                  ┌─────────────────────┐
        │ Native UC catalogs│                  │ lakebase_bs         │
        │ (Delta tables)    │                  │ (foreign catalog    │
        │                   │                  │  via Federation)    │
        │ ANALYTICAL TIER   │                  │ OPERATIONAL TIER    │
        └───────────────────┘                  │   ↓                 │
                                               │ Lakebase Postgres   │
                                               │ (Backstage data)    │
                                               └─────────────────────┘

Both tiers governed by the same UC primitives. Audit trails for operational events land in the same system.access.audit table that captures analytical events. Cost attribution joins the same (project_id, branch_id, endpoint_id) identity space.

What it replaces

Before (independent governance) After (foreign-catalog federation)
Postgres native GRANTs UC GRANT
CloudTrail (IAM principal) system.access.audit
pgaudit / pg_stat_activity (SQL statement audit) system.access.audit
CloudWatch (timestamp + cost tags) UC system billing tables
Three services to cross-reference per audit query One SQL statement
Custom CloudWatch tagging on every ephemeral RDS instance Native branch-identity attribution
Manual data-masking on dev/test provisioning UC masking policies, branch-propagated

When it fits

  • Operational DB and analytical platform share the same vendor platform (Lakebase + Databricks UC; AWS-side equivalents in development; not yet generalisable across vendor boundaries).
  • The compliance burden of cross-system audit cross-referencing is material — regulated industries, multi-team access patterns, or high-cardinality ephemeral environments where the CloudTrail+pgaudit+CloudWatch path is the operational pain point.
  • The operational DB supports a query-federation interface that preserves enough wire-protocol semantics for the catalog to reach in (Lakebase's native embedding inside Databricks is the Part 2 enabler; less-tightly-integrated stores may not support this).

When it doesn't fit

  • The operational DB lives in a different vendor ecosystem than the analytical platform. Federation across vendor boundaries (e.g., RDS Postgres into UC) typically degrades to read-only query federation without the full audit / cost / policy substrate symmetry that the pattern depends on.
  • The application's hot-path latency budget can't absorb the federation-engine overhead. The Part 2 source does not disclose the latency envelope of UC-mediated reads vs direct Postgres reads; the pattern's value is governance, not performance.
  • The team's compliance posture doesn't require operational-DB audit unification. If the operational DB is auditable in isolation and there's no cross-tier audit query that needs answering, the federation overhead is unnecessary.

What it composes with

Anti-patterns

  • Federate without identity-dimension propagation. If (project_id, branch_id, endpoint_id) aren't stamped on operational events at emission time, the federation surface shows the data but the audit + billing substrate can't attribute it. Cost-tracking and audit query become "federate, then enrich with tags" rather than "federate, then query."
  • Federate read-path without write-path. If only reads go through UC and writes bypass the federation engine, the audit trail diverges from reality. The Part 2 source is implicit on this — the worked examples are control-plane (branch lifecycle) rather than DML — but the pattern depends on all operational traffic reaching the audit substrate.
  • Treat the foreign catalog as read-only analytical mirror. The pattern's value is OLTP-tier governance unification, not zero-copy analytics. Federating an operational DB read-only into the warehouse for analyst use is a different (sibling) pattern with different constraints.

Caveats

  • Mechanism not fully disclosed in canonical instance. The Part 2 source establishes the capability and the UC GRANT model but not the federation engine's policy-pushdown behaviour, the enforcement-point topology, or the latency envelope. Future Databricks Lakehouse Federation disclosures or Part 3 of the series may fill this in.
  • Vendor-platform-specific in this canonical instance. The pattern is currently demonstrated only inside Databricks (Lakebase + UC + Lakehouse Federation). The architectural shape is portable in principle, but the "natively embedded inside" precondition is what makes the audit + cost + policy symmetry hold in practice.

Seen in

  • sources/2026-05-15-databricks-backstage-with-lakebase-part-2 — First canonical wiki disclosure of the foreign-catalog- federation-for-operational-DB-governance pattern. Backstage Postgres on Lakebase exposed as lakebase_bs in Unity Catalog via Lakehouse Federation; UC GRANT replaces Postgres native grants; system.access.audit + system billing tables replace CloudTrail + pgaudit + CloudWatch; "one SQL query instead of three services." The Part 1 disaster-recovery experiment is the worked-example anchor — branch-lifecycle audit queryable as a single UC SQL statement.
Last updated · 542 distilled / 1,571 read