Skip to content

DATABRICKS 2026-10-19

Read original ↗

What makes a catalog open? Unity Catalog Advances the Open Lakehouse

Summary

A Databricks positioning post that defines an open catalog by three requirements beyond merely supporting open file/table formats, and argues Unity Catalog meets all three. The thesis: supporting Apache Iceberg in name is not enough — if your tables sit in a bucket the provider owns, the format is "open in name only." A truly open catalog must keep data in storage you own, deliver its best features on those owned tables (not reserve them for provider-controlled storage), and allow data portability: moving a table to another catalog with the data staying in place. The post's headline claim is that UC is the first catalog to support both REGISTER and UNREGISTER from the Iceberg REST catalog spec (Databricks contributed UNREGISTER to the spec), making no-copy catalog handoff an open standard. It is the "what makes a catalog open" framing companion to the earlier, mechanism-level REGISTER/UNREGISTER post (2026-10-05).

The three requirements of an open catalog

The post's central model — beyond open formats, an open catalog must satisfy three properties, each with a named failure mode when absent:

# Requirement What it means Failure mode without it
1 Storage you own Tables live in customer-owned storage, not provider-controlled ("managed") storage Bucket sits in the provider's cloud account; you pay storage "rent" + egress to move data out
2 Best features included Automatic optimizations + advanced table features work on customer-owned-storage tables, not only provider-controlled ones Teams move tables into provider storage to get features → provider's roadmap decides what you can do with your data
3 Free to move without copy Tables enter/leave via REGISTER / UNREGISTER, data stays in place Changing catalogs means exporting and rewriting every table out of provider storage

Key takeaways

  • "Open format" ≠ "open catalog." Many platforms support Iceberg but store the primary table type in a proprietary format inside provider-controlled storage by default. The format being open does not make the catalog open; the lock-in moves up a layer to where the data physically lives and who controls the feature set (Source: this page).
  • "Managed" describes who maintains, not who owns. UC's distinction: a managed table is one the catalog optimizes/maintains/upgrades — its files still live in a cloud storage account you own, not in "managed storage" the provider controls. This is the semantic pivot the whole argument turns on, and it is the UC managed tables model (Source: this page).
  • 99% of UC managed tables are in customer-owned storage — stated inverse of "platforms built around provider-controlled storage," where "nearly all" tables sit in provider storage, proprietary-by-default. "The difference shows up the day you decide to leave a catalog" (Source: this page).
  • Best features on owned tables (req. 2), with named competitor gaps:
  • Fail-safe / recovery — Snowflake "and other platforms reserve fail-safe recovery for tables in their own storage"; UC managed tables in customer-owned storage support data recovery — query earlier versions or restore dropped tables up to 30 days.
  • Automatic optimizations — Predictive Optimization runs on UC managed tables in both Delta and Iceberg in customer-owned storage; the post claims Snowflake data compaction is "costlier in provider-controlled storage."
  • Storage cost — the post claims Snowflake "doesn't delete orphaned files from Iceberg tables in your own bucket" (files left by failed writes), so you keep paying for unreferenced files; UC's Predictive Optimization removes unreferenced files from your bucket automatically (Source: this page).
  • Portability is the most important test (req. 3). The ecosystem had no standard way to release a table from a catalog's management (DROP destroys data); Databricks contributed UNREGISTER to the Iceberg REST spec. REGISTER + UNREGISTER work as a pair: UNREGISTER removes a table and returns its latest metadata location (the pointer the next catalog needs to take over commits); REGISTER attaches that metadata location to a new catalog as a single record (Source: this page).
  • Portability is only possible because the data is in storage you own. "Unregistering isn't possible with provider-controlled storage since the exit API can't hand you a pointer to files that you can't take with you." This is the structural link between requirement 1 (owned storage) and requirement 3 (no-copy exit) — compute-storage separation plus owned storage is what makes catalog ownership a transferable pointer (Source: this page).
  • Iceberg alone does not prevent lock-in (FAQ). "An Iceberg table in provider-controlled storage can be as hard to move as a proprietary table, and an Iceberg table in your own storage may get fewer features." Avoiding lock-in requires owned storage + full features on those tables + a clean no-copy handoff — i.e. all three requirements together (Source: this page).
  • Foreign Iceberg tables are read-only in Databricks because another catalog owns their commits (two catalogs cannot write the same table — the split-brain hazard). To have UC manage such a table in customer-owned storage, REGISTER it; the roadmap item SET MANAGED [IN PLACE] will convert foreign/external tables to UC managed tables keeping the same customer-defined storage path with no data movement (Source: this page).

Systems / concepts / patterns extracted

  • Systems: Unity Catalog (the catalog making the open-catalog claim); Iceberg REGISTER/UNREGISTER APIs (the no-copy handoff mechanism); Apache Iceberg
  • Iceberg REST catalog spec (the open standard REGISTER/UNREGISTER extend); Delta Lake (the other managed-table format); UC managed tables (the maintained-by-catalog / owned-by-you table type); Predictive Optimization (the automatic-optimization engine that runs on owned-storage tables); Snowflake (the named contrast platform — provider-controlled-storage feature gates). Apache Polaris / Snowflake Open Catalog / AWS Glue named only as the ecosystem UNREGISTER adoption is not yet confirmed for.
  • Concepts: open table format (the catalog owns a pointer, not the data); egress cost (the "storage rent + exit toll" of provider-controlled storage); compute-storage separation (why a table's data lives independent of the catalog); open lakehouse (the umbrella architecture); object storage (the customer-owned bucket where data files live); point-in-time recovery (the 30-day time-travel / dropped-table restore the post calls fail-safe recovery).
  • Patterns: no new pattern. The REGISTER/UNREGISTER handoff is a concrete instance of the already-canonical separate-revoke-from-establish ordering (revoke on A before establish on B), documented on the REGISTER/UNREGISTER system page.

Operational numbers / specifics

  • 30 days — window for UC managed-table data recovery (query earlier versions / restore dropped tables) in customer-owned storage.
  • 99% — share of UC managed tables Databricks says are in customer-owned storage (vs "nearly all" on provider-controlled-storage platforms).
  • $1M+/year — storage-cost saving a named adopter (Zepto) attributes to UC managed tables' automatic optimizations ("while eliminating the need for tedious manual effort on a daily basis"). Vendor-supplied customer quote, not an independent measurement.
  • Private Preview — REGISTER and UNREGISTER availability in UC at publication (reach out to account team for access).

Caveats

  • Tier-3 vendor positioning post. The three-requirement model is a framing device Databricks uses to score itself favorably against Snowflake; the competitor "gaps" (fail-safe only on provider storage, orphaned-file billing, costlier compaction) are Databricks' characterization with doc links, not neutral benchmarking. Treat as architecture argument, not measured comparison.
  • Mechanism depth lives in the 2026-10-05 post. This page is the "why it matters / what open means" framing; the actual REGISTER/UNREGISTER wire semantics, the split-brain hazard of REGISTER-alone, and the ordered safe-handoff are on the earlier source and its system page. Cite those for internals.
  • Only one quantitative hard number is Databricks' (30-day recovery); the 99% and $1M figures are a vendor stat and a customer quote.
  • REGISTER/UNREGISTER is Private Preview; broad cross-catalog adoption of UNREGISTER (Polaris, Snowflake Open Catalog, Glue) is not asserted.
  • SET MANAGED [IN PLACE] (in-place foreign→managed conversion with no data movement) is stated roadmap, not shipped.

Source

Last updated · 854 distilled / 2,448 read