Skip to content

SYSTEM Cited by 1 source

UC Credential Vending

Unity Catalog Credential Vending is the API by which Unity Catalog mints short-lived, scoped credentials on demand for external engines to read or write the underlying object-store data backing UC-governed assets. Generally Available for tables on 2026-05-14; Public Preview for UC Volumes (unstructured assets — images, PDFs, videos) on the same date.

The architectural primitive: instead of granting external engines broad, static cloud-storage permissions (or per-engine service-account keys in the customer's cloud), UC issues per-request, per-resource, minutes-scale credentials that can only access the specific resource the request targeted, only for the lifetime of the credential, and only as governed by the requester's UC privileges.

Two dimensions of GA-vs-PuPr

Surface Status (2026-05-14)
Tabular data — UC managed + external Delta tables Generally Available
Unstructured data — UC Volumes (images / PDFs / videos) Public Preview

The unification claim: "The same access control model, audit trail, and scoped credentials apply whether you're querying a table or processing a raw video file externally." (Source: sources/2026-05-14-databricks-expanded-interoperability-with-unity-catalog-open-apis)

Two production-readiness additions disclosed at GA

The 2026-05-14 post calls out two enterprise-grade additions that flipped credential vending from preview-quality to GA-quality:

  1. M2M OAuth replaces PATs. "External engines can now authenticate to UC using machine-to-machine (M2M) OAuth, meeting enterprise security requirements without relying on personalized access tokens (PATs), which are per-user, long-lived, and hard to rotate."

The structural failure modes of PATs the post calls out: - Per-user: the credential binds to a human identity, breaking when the human leaves the org. - Long-lived: the compromise window is indefinite by default. - Hard to rotate: rotation requires per-engine reconfiguration.

M2M OAuth dissolves all three: service-to-service identity (no human binding), short-lived tokens (compromise bounded), automatic re-mint (rotation invisible to operators). Canonical instance of concepts/m2m-oauth-vs-pat.

  1. Engine-side automatic refresh. "Credentials are refreshed automatically by engines via the UC credential vending APIs, so pipelines that run for hours complete reliably without tokens expiring mid-job."

This is the property that makes minutes-scale credential lifetimes operationally tractable for hour-scale Spark / Flink pipelines. The engine SDK (e.g., Delta-Spark 4.2 + Unity Catalog 0.4.1 — the Beta version pinning) detects approaching expiry and re-invokes the vending API transparently. Without this, every pipeline that ran longer than the credential TTL would have to handle expiry mid-run, which the prior generation of static-credential designs simply avoided by setting credentials never to expire.

Architectural shape

┌─────────────────┐            ┌─────────────────┐
│ External Engine │            │ Unity Catalog   │
│ (Spark/Flink/   │  M2M OAuth │                 │
│  DuckDB)        │ ─────────> │ ─ AuthN         │
│                 │            │ ─ AuthZ via UC  │
│                 │  scoped    │   privileges    │
│                 │  +         │ ─ Mint scoped   │
│                 │  short-    │   short-lived   │
│                 │  lived     │   creds         │
│                 │  cred      │                 │
│                 │ <───────── │                 │
└────┬────────────┘            └─────────────────┘
     │
     │ data path (cloud storage, scoped to requested resource)
     ▼
┌─────────────────┐
│ Object store    │
│ (S3/ADLS/GCS)   │
└─────────────────┘

Three principal architectural properties:

  • Catalog mediation — credentials are minted by the catalog, not shared as static cloud-IAM principals. The catalog is the single point where credential issuance can be observed and revoked.
  • Resource-scoped — each credential carries authorisation only for the specific table / volume / path that the requester asked for, governed by UC privileges.
  • Time-scoped — minutes-scale TTL bounds the compromise window; auto-refresh bridges the gap to long-running pipelines.

Composition with surrounding UC primitives

Composes with Role
systems/unity-catalog The minting authority + privilege evaluation point.
systems/uc-managed-tables The most common credential-vending consumer in the 2026-05-14 disclosure (managed Delta tables external engines write to).
systems/unity-catalog-volumes The Public-Preview second class of asset under credential vending.
systems/delta-kernel Library shape that integrates the auto-refresh on engine side.
systems/unity-catalog-abac (forward) Per the 2026-05-14 roadmap: row + column-level ABAC will compose onto credential-vending so external reads inherit the same fine-grained policy enforcement as first-party reads.

Why this is a credential-vending rather than a credential-sharing primitive

The 2026-05-14 post implicitly distinguishes credential vending from two adjacent shapes:

  • Static IAM principal sharing — granting an external engine a cloud-IAM principal (e.g., an AWS IAM role) with broad permissions. Compromise is unbounded; revocation requires IAM-side action; scoping is permission-policy-only, not request-by-request.
  • Catalog-as-authoriser-but-storage-direct-grant — the catalog decides who can access what but the cloud storage itself enforces via static credentials. Same blast-radius problem as static IAM.

Credential vending substitutes both with catalog-mints-per-request, storage-trusts-the-token: every access has a fresh, scoped, short- lived token, with the catalog as the only authoritative issuer. Canonical instance of concepts/credential-vending applied at the data-platform / object-storage altitude.

Seen in

  • sources/2026-05-14-databricks-expanded-interoperability-with-unity-catalog-open-apis — First wiki canonicalisation as a distinct named API. Discloses two productionisation additions (M2M OAuth + auto- refresh), GA-for-tables and Public-Preview-for-volumes scope, and the composition with UC Managed Tables as the primary consumer. Reserved for future ingests: credential vending throughput / latency, scaling envelope per metastore, the M2M OAuth client-credential lifecycle (issuance / rotation / revocation), the per-request-scope enforcement mechanism inside the cloud storage backend, ABAC-for-external-reads composition.
Last updated · 542 distilled / 1,571 read