Skip to content

DATABRICKS Tier 3

Read original ↗

Why R&D Data Belongs in the Lakehouse — and Why Agents Need It There

Summary

A case study from cellcentric (Daimler Truck × Volvo Group joint venture building hydrogen fuel cell systems) describing how a four-year data-platform investment on Azure + Databricks became the governed context layer for AI agents. The core thesis: agents are only useful if they reason over the same governed context an engineer trusts — where data came from, what it means, how complete it is, which caveats matter, and whether the user is allowed to see it. The architecture exposes one Data Hub with two interfaces: a marketplace UI for human employees and an MCP server for AI agents, both backed by the same Unity Catalog governance layer. Identity flows end-to-end from Azure AD through OAuth 2.0 token exchange (on-behalf-of) so agents never hold bulk credentials — if the user lacks access, neither does the agent.

Key takeaways

  1. Data products are the unit of AI context, not raw tables. A data product bundles ownership, lifecycle state, domain assignment, linked governed assets, and rich markdown context entries explaining what the product is for and how to use it. "Agents need more than schema metadata, and they need more than a document." (Source: raw article)

  2. Context coverage is a first-class data quality metric. The platform carries a context-coverage badge per data product measuring table-description presence and column-level documentation completeness. "Coverage went up because engineers could see the gap directly in the product surface where the data would be consumed." (Source: raw article)

  3. Documentation written during development captures reasoning that disappears after the fact. AI-assisted column comments (reviewed by engineers before merge) + long-form markdown catalog entries are authored while project context is fresh; they become context for the next human or agent.

  4. One UI for humans, one MCP for agents — same governance. The Data Hub exposes Unity Catalog metadata and data product context to human employees via a web marketplace and to AI agents via MCP. "Any MCP-aware coding agent or assistant can access the same Unity Catalog metadata and data product context that the Data Hub uses." (Source: raw article)

  5. Agent access is governed by identity delegation, not shared credentials. Identity starts in Azure AD → flows through Data Hub → into Databricks via OAuth 2.0 token exchange (on-behalf-of pattern). "If the user lacks access to a table, masked column, or governed view, the agent receives the same boundary." (Source: raw article)

  6. Agents cannot become production writers. The operating model separates production ownership/deployment (code-reviewed, governed deployment paths) from consumption. Consumers including agents "do not receive production write identities." (Source: raw article)

  7. Hybrid MCP architecture. Databricks-managed MCP delivers governed access to Databricks capabilities (Genie tool calls); cellcentric's own MCP delivers the richer context layer — Unity Catalog metadata, data product objects, catalog markdown, and business-specific tools. "Agents need both." (Source: raw article)

  8. Unity AI Gateway as single observability choke-point. Model calls route through AI Gateway → inference tables land in Unity Catalog Delta tables → analysable with standard SQL patterns already used for business data. (Source: raw article)

  9. MLflow tracing + continuous evaluation framework. Standardized traces across agent adapters; continuous evaluation with deterministic correctness scorers, SQL-grounding scorers, cache-aware cost scorers, and LLM-judge scorers with domain-expert alignment. "Traces and evals let us test the agent, the tools, and the context it relies on together." (Source: raw article)

  10. Skills Marketplace as the next pattern. Packaging instructions, approved tools, templates, guardrails, and evaluation checks into versioned, reviewed "skills" that agents invoke by name — extending the data-product lifecycle pattern to procedural knowledge. (Source: raw article)

Architecture & numbers

Datum Value
Data products published 27
Column-comment coverage (published products) ~90% avg
Source systems integrated (Fuel Cell Passport) 5 (SAP S/4HANA, 2× MES, laboratory DB, IoT telemetry)
Hierarchy levels modelled 7 (system → raw-material batch)
Refresh cadence Daily
Identity substrate Azure AD → OAuth 2.0 token exchange
LLM substrate Databricks Model Serving + Claude via Foundation Model APIs
Observability Unity AI Gateway → inference tables (Delta) + MLflow tracing
MCP architecture Hybrid: Databricks-managed + cellcentric custom
Temporal model State-based (point-in-time + rework-history)

Extracted entities

Systems

Concepts

Patterns

Caveats

  1. Tier-3 customer case study / product showcase. Architecture content is ~40-50% of body; the rest is product-marketing positioning for Unity Catalog, Genie, MLflow, Asset Bundles.
  2. No quantitative scale numbers. No QPS, no latency p-values, no data volumes beyond "27 products" and "5 source systems".
  3. Temporal model described but not sized. "State-based temporal model" for point-in-time + rework-history — no row counts, no storage footprint.
  4. MCP server implementation undisclosed. Described as hybrid (Databricks-managed + custom) but no protocol details, tool schema, or latency numbers.
  5. OAuth 2.0 token exchange mechanism referenced but not detailed. No OIDC flow diagrams, no token lifetime, no refresh policy.
  6. Evaluation framework scorers named but not specified. "Deterministic correctness", "SQL-grounding", "cache-aware cost", "LLM-judge" — no thresholds, no scoring rubric, no agreement rates.
  7. Skills Marketplace is roadmap/aspirational. "The next wave" — no shipped implementation.
  8. No comparison with alternative architectures. No mention of competing patterns (e.g. RAG over raw tables, vector-store-first, graph-based context).
  9. Single-company single-industry case study. Hydrogen fuel cell R&D at cellcentric; generalisability to other domains asserted but not demonstrated.
  10. AI-assisted column comments quality undisclosed. "AI-assisted during data engineering work and reviewed by the engineer before merge" — no acceptance rate, no correction rate.

Source

Last updated · 590 distilled / 1,788 read