Skip to content

DATABRICKS 2026-10-05

Read original ↗

Unlocking Data Portability: Preventing Catalog Lock-in with REGISTER and UNREGISTER APIs

Summary

Databricks describes how a table in an open table format can be moved between Iceberg catalogs without copying, exporting, or rewriting a single data file — the mechanism for "catalog portability" and the anti-lock-in argument for the lakehouse. Because an Iceberg catalog is only the authority for a table's latest metadata pointer (data and metadata both live in object storage, decoupled from compute), handing a table off is fundamentally a pointer-ownership transfer, not a data migration. The post explains why REGISTER alone is unsafe (it creates a split-brain where two catalogs both believe they own the table, and the first write silently forks it), and introduces UNREGISTER — which Databricks contributed to the Apache Iceberg REST catalog specification — as the missing half: it tells the old catalog to relinquish the table and returns the exact metadata-location pointer the next catalog needs to assume commit coordination. The safe handoff is the ordered three-step sequence unregister → receive pointer → register (Source: sources/2026-10-05-databricks-unlocking-data-portability-preventing-catalog-lock-in-with-register-and-unregister).

Key takeaways

  1. The catalog is the single authority for a table's latest state. When an engine queries a table it first asks the catalog to load the table; the catalog returns the current metadata (schema, history, statistics) and the data's location in object storage; the engine then reads Parquet directly from the bucket. The catalog "coordinates commits and ensures that two writers can never silently overwrite each other" — it is the commit-coordination and single-source-of-truth point for an otherwise storage-decoupled table (concepts/open-table-format).

  2. REGISTER attaches an existing table to a catalog by pointer, with zero data movement. A client POSTs the desired table name + the URI of the existing metadata.json location; after basic validation the catalog "writes a single record linking the table name to the existing metadata.json. No data files are copied, moved, or altered." REGISTER is already defined in the Iceberg REST spec.

  3. REGISTER alone creates a split-brain. Iceberg catalogs are independent and do not communicate, so registering a table in catalog B while it remains active in catalog A leaves two catalogs each believing they are the sole owner. "Neither will throw an error, but the first write operation will fork the table, leading to inconsistent query results and silent data loss." A write via Catalog A becomes invisible to Catalog B — "permanently destroying your single source of truth" (concepts/split-brain).

  4. DROP TABLE cannot be used to clean up the old side — it deletes the table's underlying data and metadata. Historically the ecosystem "lacked a standard way to cleanly terminate a catalog's management of a table" without destroying the data, so the REGISTER handoff was missing its second half.

  5. UNREGISTER is the contributed missing piece. Databricks contributed UNREGISTER to the Apache Iceberg REST catalog specification: it removes the table's entry from the managing catalog "without touching a single underlying data file," and — crucially — returns the table's latest metadata location, the exact pointer the next catalog needs to assume commit coordination. By making the original catalog explicitly relinquish control, it eliminates the split-brain risk.

  6. The safe handoff is an ordered three-step sequence. unregister (old catalog forgets the table; files stay put) → receive the returned metadata location → register (new catalog picks up the pointer and becomes the sole managing catalog). "The handoff ensures the table has exactly one managing catalog at all times" — a revoke-before-establish ordering (patterns/separate-revoke-from-establish).

  7. This is the structural basis of the "not lock-in" claim. Because data never moves and ownership is a transferable pointer, organizations retain "the freedom to choose and move between catalogs as their architecture evolves." Databricks positions Unity Catalog as the open catalog delivering this portability (REGISTER/UNREGISTER available on UC in private preview at publication).

Wire detail

  • UNREGISTER request (empty POST to the table resource): POST <uc-iceberg-rest-base>/v1/{prefix}/namespaces/{namespace}/tables/{table}/unregister
  • UNREGISTER response: the table's latest metadata-location pointer — the handoff token the next catalog registers with.
  • REGISTER request: a POST with the desired table name + the URI of the existing metadata.json; catalog writes a single name→metadata link record after basic validation.

Operational / architectural numbers

  • Zero data files copied, moved, or altered during either REGISTER or UNREGISTER — the entire handoff is metadata-pointer manipulation.
  • Exactly one managing catalog is guaranteed at all times by the ordered sequence (the invariant the design protects).
  • One record written on REGISTER (the name → metadata.json link).

Caveats

  • Production migration involves more than the three API calls. The post explicitly flags that a real migration "still involves operational steps, like safely stopping writers and repointing jobs," deferred to a follow-up post. The three-step API sequence is the correctness core, not the full operational runbook.
  • No numbers on latency or scale. The post is a mechanism explainer; it provides no throughput/latency figures for the endpoints.
  • UC REGISTER/UNREGISTER is private preview at publication ("reach out to your account team"). UNREGISTER is contributed to the Iceberg REST spec, but availability in other catalog implementations (Apache Polaris, Snowflake Open Catalog, AWS Glue) is not stated.
  • Split-brain window during a naïve REGISTER-only move is silent — no error is thrown by either catalog; detection is only after divergent reads/writes manifest. This is why the explicit relinquish step matters.

Source

Last updated · 854 distilled / 2,448 read