Skip to content

Databricks — Advancing Apache Iceberg on Databricks: Iceberg v3 GA, Open Sharing, and Unified Governance

Summary

A Databricks Blog post (2026-05-28, Tier 3) announcing a coordinated set of Apache Iceberg capability releases — the bulk reaching General Availability — that reposition Unity Catalog as a fully Iceberg-native catalog. The architecturally load-bearing items are Iceberg v3 GA (deletion vectors, row tracking, VARIANT type), Foreign Iceberg GA (governing tables managed in external catalogs through UC), External Sharing to Iceberg clients GA (Delta Sharing emits Iceberg REST endpoints to non-Databricks recipients), Cross-engine ABAC (Beta) (governance evaluated server-side during Iceberg REST scan planning), and a forward-looking disclosure that Iceberg v4 and Delta 5.0 will share an adaptive metadata tree structure. The post is structurally a feature-roundup / marketing piece — five "things that make Unity Catalog the most interoperable Iceberg catalog" with customer testimonials and a comparison-chart frame — but each named primitive is a real architectural element with consequences for the open-table-format ecosystem.

Context for the wiki

Tier-3 source. Treated as scope-passing because the post names multiple genuine architectural primitives — file-level deletion vectors, row tracking, the VARIANT type, server-side scan-planning policy evaluation, and shared-metadata format co-evolution — even though the post does not provide deep internals on any single primitive (most go to docs links). The architectural content is approximately 25-30% of the body; customer testimonials, comparison framing, and conference CTAs make up the rest. The wiki ingests it as a catalog-level GA-roundup canonical that consolidates several Iceberg-on-UC primitives into named entities; deep-dive sources (Iceberg spec / Databricks docs) would be needed for mechanism-level coverage of any individual primitive.

Key takeaways

  1. Iceberg v3 reaches GA on Databricks across managed, foreign, and UniForm-enabled tables. Three format-level features ship together: deletion vectors (file-level row-delete representation that accelerates updates / merges / deletes without rewriting whole data files), row tracking (stable per-row identity that supports more efficient incremental processing), and the VARIANT type (a standard representation for semi-structured data). "Native support for deletion vectors, row tracking, and the new VARIANT type across managed, foreign, and UniForm-enabled tables." Verbatim summary of v3's positioning: "deletion vectors accelerate updates, merges, and deletes; row tracking supports more efficient incremental processing; and VARIANT provides a standard representation for semi-structured data. These features also work seamlessly across both Delta and Iceberg tables, enabling interoperability without rewriting data." (Source: this article.)

  2. Managed Iceberg is GA — Iceberg tables can be created, read, written, optimised, governed, and shared directly in Unity Catalog, with Predictive Optimization and Liquid Clustering applying "eliminating the manual work required to keep tables performant". Unity Catalog vending short-lived credentials over the Iceberg REST Catalog API lets any conforming engine — "Spark, Trino, Flink, Snowflake, DuckDB, pandas, or another Iceberg-compatible client" — perform reads and writes against the managed tables without copy or broad storage permissions. "customers can create, read, and write to Iceberg tables in Unity Catalog from any engine using UC's Iceberg REST Catalog APIs." This is the external-engine-write-to-managed-table shape (already canonicalised on the wiki from the 2026-05-14 source) extended to Iceberg as the format.

  3. Foreign Iceberg is GA — Unity Catalog can govern Iceberg tables managed in external catalogs. Customers register Iceberg tables held in AWS Glue, Snowflake Horizon, Hive Metastore, Salesforce Data Cloud, Apache Polaris, Google Cloud Lakehouse Runtime Catalog, Palantir, and Workday into UC; UC then "discovers, secures, queries, and shares external Iceberg tables through Databricks while leaving the data and source catalog in place." Credential Vending for Foreign Iceberg also reaches GA: UC mints short-lived scoped credentials for federated tables. The architectural property is catalog-of-catalogs — UC becomes a single governance plane over Iceberg tables produced by other catalogs. New wiki concept: concepts/foreign-iceberg-table.

  4. External Sharing to Iceberg clients is GA via Delta Sharing. "Databricks customers can share live data externally with any recipient that supports the Iceberg REST Catalog API. Recipients can query shared data from Iceberg-compatible clients such as Snowflake, Trino, Flink, and Spark, without manual ingestion or copies." Delta Sharing — historically Delta-native — now emits Iceberg REST endpoints, making the protocol bi-format on the recipient side. Public Preview separately announces External Sharing of Foreign Iceberg tables: customers can include Iceberg tables managed outside Databricks but registered in UC into a Delta Sharing share. UC becomes the sharing layer for both managed and foreign Iceberg tables, "keeping data in place and governance centralized."

  5. Cross-engine ABAC reaches Beta — Unity Catalog ABAC evaluates policies during Iceberg REST scan planning, returning a filtered scan plan to external engines. This is the architecturally most novel primitive in the post. "With cross-engine attribute-based access controls (ABAC) now in Beta, Unity Catalog extends attribute-based access control to Iceberg clients using the Iceberg REST Catalog Scan APIs." Mechanism: "Administrators define policies once in UC, including column masks, row filters, and tag-based policies. When an external Iceberg engine requests access, UC evaluates the applicable policies during server-side scan planning. UC then returns a filtered scan plan so the engine only reads authorized data when processing the query." Compatibility hook: "Any engine, such as Apache Spark or DuckDB, which implements the Iceberg REST catalog scan planning client (added in the Iceberg 1.11 release) can access data with ABAC enforced." This is the wiki's first canonical instance of scan-planning-as-policy-enforcement-point — governance evaluated by the catalog server during scan planning, returning an already-filtered plan rather than relying on each engine to apply policies post-read. New wiki concept: concepts/cross-engine-abac.

  6. Iceberg-compatible materialized views (Gated Public Preview). Customers create materialized views in Databricks and "expose them as Iceberg tables to downstream consumers." Forward-looking syntax disclosed: CREATE MATERIALIZED VIEW my_mv USING ICEBERG. Architectural shape: a managed MV with refresh ownership inside the catalog, exposed to the open ecosystem as an ordinary Iceberg table — adds a managed-aggregation primitive to the Iceberg-native surface area.

  7. New catalog federation connectors in Preview extend UC's federation beyond the existing AWS Glue / Snowflake Horizon / Hive Metastore / Salesforce Data Cloud set to Google Cloud Lakehouse and Palantir. "making Unity Catalog your single pane of glass." Generalises the catalog-of-catalogs role to a longer list of upstream metadata systems.

  8. Iceberg v4 and Delta 5.0 will share an adaptive metadata tree structure. Forward-looking disclosure: "With Iceberg v4, we are rethinking the core metadata structure from the ground up for better performance, scalability, and interoperability. […] we are also proposing that the next version of Delta, Delta 5.0, adopts the adaptive metadata tree structure." This is format co-evolution — the explicit alignment of two open table formats on a shared metadata data structure rather than two parallel evolutions. New wiki concept: concepts/format-co-evolution-iceberg-delta. The post does not describe the adaptive metadata tree internals; the conference session "Format Co-Evolution: How Iceberg v4 and Delta 5.0 Share a Unified Metadata" is the deferred-internals reference.

Architectural numbers / disclosed properties

The post is qualitative and contains no quantitative numbers. Customer testimonials are present (AT&T's Andy Markus: "a future-proof data foundation: interoperable, governed, and primed for AI applications that move the needle"; Rippling's Tae Lee: "native performance for our AI and ML pipelines, and open interoperability for every downstream consumer. One write path, zero duplication, and a governance layer every engine respects, including the AI-driven products we're building for Rippling's Data Cloud"; Magnite's Kayvon Raphael: "native performance for our AI and ML pipelines, and open interoperability") but no latency / throughput / hit-rate / cost numbers are disclosed. Disclosed structural attributes:

  • Iceberg v3 features scope: managed Iceberg, foreign Iceberg, and UniForm-enabled managed tables — three table types covered.
  • Cross-engine ABAC scan-planning client is part of the Iceberg 1.11 release — the version-floor for ABAC compatibility.
  • Materialized-view-as-Iceberg-table SQL syntax: CREATE MATERIALIZED VIEW <name> USING ICEBERG.
  • Federation set at the time of the post: AWS Glue, Snowflake Horizon, Hive Metastore, Salesforce Data Cloud (existing GA); Google Cloud Lakehouse, Palantir (Preview); Workday is named in the body but not explicitly placed in the GA/Preview taxonomy.
  • Iceberg-client recipients of Delta Sharing: Snowflake, Trino, Flink, Spark named explicitly.

Caveats

  • Tier-3 marketing roundup framing. The post is a "5 things that make Unity Catalog the most interoperable Iceberg catalog" feature-roundup with comparison chart and customer testimonials. Architectural depth is shallow; each primitive links to docs for mechanism-level details. Wiki ingest is a shape-canonicalisation (entity names + architectural roles), not a deep dive on any one primitive.
  • No quantitative numbers. Zero latency / throughput / cost / hit-rate / scale numbers anywhere in the post. The promotion of "the most interoperable Iceberg catalog on the market" and "the most comprehensive set of Iceberg capabilities available on any lakehouse catalog" is unbenchmarked vendor positioning.
  • Iceberg v3 internals deferred to spec. The post names deletion vectors / row tracking / VARIANT but provides no on-disk format detail, no compaction policy, no compatibility-matrix detail. Reader has to follow Iceberg v3 spec and Databricks Iceberg-v3 docs for mechanism depth.
  • Cross-engine ABAC mechanism light on details. The post describes the shape (UC evaluates policies during server-side scan planning, returns filtered scan plan) but not the wire-level protocol additions, the policy-evaluation latency cost on the catalog server, the failure mode when an external engine doesn't support the scan-planning client, or the client-trust assumptions (does UC trust the engine to honour the filtered plan, or are masks applied client-side via UDF interception?).
  • Iceberg v4 / Delta 5.0 adaptive metadata tree is unbuilt. Forward-looking disclosure with no spec, no benchmarks, no commitment date. Captured here as a directional concept, not an in-production primitive.
  • Materialized-view-as-Iceberg surface in Gated Public Preview — CREATE MATERIALIZED VIEW ... USING ICEBERG is announced, broader availability "in the coming weeks".
  • External Sharing of Foreign Iceberg tables is in Public Preview, not GA — distinct from the External Sharing to Iceberg clients GA.
  • No cell-architecture / blast-radius / SLO content — the post is a feature catalog, not an operational reliability post. For UC reliability properties, see the 2026-05-27 How the lakebase architecture stays resilient to cloud failures source.
  • Comparison chart referenced but not reproduced — image URLs (/sites/default/files/inline-images/...) referenced in the post body cannot be verified from the markdown. Comparison-chart claims ("Unity Catalog is the only catalog that delivers on all five requirements") are vendor positioning.
  • No discussion of how Iceberg v3 features interact with UC ABAC. E.g., does a row filter work correctly against a deletion-vector-based table where some rows are logically deleted? Does row tracking interact with column masking? Not addressed.
  • No latency disclosure for cross-engine ABAC evaluation during scan planning. Server-side policy evaluation per scan-plan request adds catalog-side compute on the query path; cost/latency profile is undisclosed.

Source

Last updated · 542 distilled / 1,571 read