Skip to content

PATTERN Cited by 1 source

Credential vending for external engine access

Credential vending for external engine access is the deployment pattern in which a central catalog or governance plane mints short-lived, resource-scoped credentials on demand for external compute engines, instead of granting those engines static cloud-IAM principals or sharing personal access tokens. The pattern composes M2M OAuth (engine ↔ catalog auth) with scoped, short-lived credentials (catalog → cloud-storage data path) and engine-side automatic refresh (long-running pipelines remain credentialed without operator intervention).

The pattern is the auth-side half of the external-engine- write-to-managed-table shape; the commit-side half is patterns/catalog-managed-commits-for-external-write-safety.

Canonical instance: Unity Catalog Credential Vending GA (2026-05-14)

"Unity Catalog handles this through credential vending… UC issues short-lived, scoped credentials to external engines on demand, with access policies enforced centrally." (sources/2026-05-14-databricks-expanded-interoperability-with-unity-catalog-open-apis)

The 2026-05-14 disclosure put two productionisation properties on the wiki:

  1. M2M OAuth replaces personal access tokens (PATs) — "per-user, long-lived, and hard to rotate" — as the engine-to-catalog auth primitive.
  2. Engine-side automatic refresh — "credentials are refreshed automatically by engines via the UC credential vending APIs, so pipelines that run for hours complete reliably without tokens expiring mid-job."

Both are GA for tables; Volume Credential Vending (the same primitive applied to unstructured assets in UC Volumes) is in Public Preview on the same date.

Implementation shape

External engine                 Catalog (UC)              Object store
     │                              │                          │
     │ 1. M2M OAuth client_creds    │                          │
     │    grant ──────────────────> │                          │
     │ <── access_token (mins TTL)  │                          │
     │                              │                          │
     │ 2. Vending request           │                          │
     │    (resource path,           │                          │
     │     read|write,              │                          │
     │     access_token) ─────────> │                          │
     │                              │ AuthN: validate token    │
     │                              │ AuthZ: UC privileges     │
     │                              │        + ABAC policies   │
     │                              │ Mint: scoped + minutes-  │
     │                              │       TTL credential     │
     │ <── scoped storage cred      │                          │
     │                              │                          │
     │ 3. Read/write (scoped cred) ──────────────────────────> │
     │                              │                          │
     │ 4. Engine SDK: detect        │                          │
     │    approaching expiry        │                          │
     │    → goto step 2 (refresh    │                          │
     │      vending) and/or         │                          │
     │    → goto step 1 (refresh    │                          │
     │      access token)           │                          │

Three load-bearing pattern properties:

  • Two layers of short-lived credentials: the M2M OAuth access token (engine ↔ catalog) and the vended storage credential (catalog → cloud-storage). Both auto-refresh; both are minutes-scale.
  • Catalog as the only authoritative issuer: the cloud-storage layer trusts catalog-issued tokens; there is no path by which the engine acquires direct cloud-storage credentials outside the catalog.
  • Scope enforced at issuance: the catalog evaluates UC privileges + (forward roadmap) ABAC policies at vending time and bakes the scope into the issued credential, so the credential itself cannot be used outside its intended scope even if leaked.

Three property guarantees

Property What it eliminates
Catalog-mediated minting Static-IAM-principal sharing — no broad cloud credentials cross the customer's trust boundary.
Resource-scoped credentials Lateral access — a credential issued for table A cannot read table B even though both live on the same cloud-storage account.
Minutes-scale TTL + auto-refresh Long-lived-credential blast radius (compromise window) — combined with auto-refresh, neither credential class outlives the period of legitimate use.

Composition with surrounding patterns

Composes with Role
patterns/catalog-managed-commits-for-external-write-safety Auth-side; catalog-managed-commits is the write-coordination side.
patterns/connector-library-as-protocol-abstraction Auto-refresh logic typically lives in the shared library (e.g., Delta Kernel) so engines don't re-implement it.
patterns/external-credential-store-with-principal-rewrite Adjacent shape — vault-as-credential-broker; credential vending is the catalog-as-credential-broker variant where authZ + minting are both catalog-side.

When this pattern applies

  • Multi-tenant managed-table substrate with external engine consumers — the canonical instance.
  • Per-resource scoping requirements — when "this engine should only see this specific table / volume" is structurally required.
  • Heterogeneous engine access to the same governed assets — Spark + Flink + DuckDB / Trino / partner products all needing governed read/write to the same table set.
  • Long-running pipelines — auto-refresh is the property that makes minutes-scale TTL operationally tractable.

When this pattern doesn't fit

  • Single-engine, static-permission deployments — operational overhead of the vending flow exceeds the blast-radius savings.
  • Latency-sensitive inner-loop access where adding a vending round-trip per request is unaffordable. (Vending is fine for query-class operations; not for sub-millisecond hot paths.)
  • Resources whose downstream layer can't verify catalog-issued tokens — would force the catalog to proxy the data path, an anti-pattern relative to direct-engine-to-storage access.

Trade-offs vs adjacent shapes

Shape Pro Con
Static IAM principal sharing Simple, no runtime issuance cost. Long-lived credentials; broad permissions; revocation requires IAM-side action; no central audit point.
Catalog-as-vault (long-lived stored creds) Central audit point. Compromise window unbounded; per-vended-credential rotation problem.
Credential vending (this pattern) Short-lived + scoped + auto-refresh + central audit + central revocation. Per-request issuance latency; engine-side refresh complexity (typically library-handled).

Operational disclosure (2026-05-14)

The 2026-05-14 post does not disclose quantitative figures on vending throughput, latency, or per-metastore capacity. Inferred from the architectural shape: vending is in the connection-setup path, not the per-row path, so latency overhead is amortised across many storage-layer operations. Reserved for future ingest with operational data.

Seen in

Last updated · 542 distilled / 1,571 read