Skip to content

DATABRICKS 2026-09-25

Read original ↗

From Data to Dialogue: How S&P Global Energy Made Its Structured Data Estate Conversational with Databricks Genie Agents and MCP

Summary

S&P Global Energy needed to make its entire structured-data estate — spanning Chemicals, Crude Oil, Refined Products, Gas & Power, LNG and more, each a rich family of datasets living across Databricks and several non-Databricks sources — available for external consumption by AI agents through the Model Context Protocol (MCP), so that customers' agents and its own could ask natural-language questions and get trusted, governed answers. After evaluating hand-built text-to-SQL pipelines, custom per-use-case APIs, and exporting data into external AI tools (each brittle, slow, or governance- breaking), the winning architecture was a three-layer design: (1) subject- matter experts (SMEs) curate one focused Genie Agent per dataset group — no agent code — enriching each with table/column descriptions, example queries, trusted assets, and business definitions; (2) every Genie Agent is automatically a Databricks-managed MCP server at /api/2.0/mcp/genie/{genie_space_id} with a two-tool ask-then-poll surface, governed end-to-end by Unity Catalog; (3) a FastMCP-based proxy composes the group-level Genie MCP servers into per-commodity composite endpoints with name-spaced tools, so an agent connects to one endpoint per commodity and its LLM routes (or fans out) questions across group Genies. The organizational shift is the headline: curation became a domain activity, not an engineering activity — SMEs became publishers, not requesters — and time-to-market for a new conversational data experience collapsed from months to days.

Key takeaways

  1. The bridge problem is context, not the LLM. "Agents are only as good as the context they can reach, and enterprise data rarely lives in one neat, well-documented place." The three historical bridges each fail: hand-built text-to-SQL is brittle (every schema change / ambiguous column / metric definition becomes an engineering task); custom APIs per use case need a new endpoint + sprint + release each; exporting data into external AI tools duplicates data, breaks freshness, and leaves the governance perimeter. (Source: sources/2026-09-25-databricks-from-data-to-dialogue-how-sp-global-energy-made-its-structured-data-estate-conversational)

  2. One Genie Agent per dataset group, not per commodity — deliberately narrow. Within LNG alone there are separate Genie Agents for Assets & Contracts, Cargo, Tenders, Outages, Supply & Demand, Netbacks, and Prices; Chemicals is split into capacity, production, utilization, trade, demand-by- end-use/derivative, inventory change, and country/region supply–demand balances. "The result is a fleet of small, sharply scoped Genie Agents rather than a handful of sprawling disconnected AI tools." This is the canonical specialized-agent- decomposition shape applied to a structured-data estate — one giant agent "degrades answer quality."

  3. The semantic layer is the differentiator, and SMEs own it. Inside each agent, SMEs add table/column descriptions, example queries, trusted assets for high-stakes metrics, and business definitions — e.g. "floating storage is defined as cargoes idling for 3 days or more in vessels travelling below a threshold speed." "This is the step that generic text-to-SQL solutions skip — and it is the step that determines whether users trust the answers." The person who knows what "floating storage" means is the person teaching the Genie what it means.

  4. Every Genie Agent is automatically a managed MCP server — zero deploy. Each is exposed at https://<workspace-hostname>/api/2.0/mcp/genie/{genie_space_id} with nothing to deploy and nothing to host, exposing essentially two tools: genie_query_space (submit a natural-language question) and genie_poll_response (poll by conversation + message ID for the full response, including generated SQL and result set). This ask-then-poll pattern fits agentic workloads: questions run asynchronously against a SQL warehouse and the agent polls until ready.

  5. Governance is inherited, not built. The managed servers are governed by Unity Catalog: a Genie Agent — or the user behind it — "can only reach the agents and underlying tables they have permission to see," authentication is handled by the platform, and every question runs through UC permissions on governed tables (native or federated) with full auditability. "We did not have to build a security layer around our AI access; we inherited the one we already had." (See concepts/governed-agent-data-access.)

  6. Non-Databricks sources join via Lakehouse Federation — no ETL. Where the data lives outside Databricks, SMEs bring it in through Lakehouse Federation connectors: "no data movement, no duplicate pipelines. The federated tables appear alongside native tables and inherit the same governance." Best-practice lesson: "Use Lakehouse Federation before you build pipelines… You can always materialize hot paths later."

  7. FastMCP composes group Genies into commodity bundles for cross-domain questions. Real questions cross groups ("How did the recent outages at Sabine Pass affect cargo premiums into Asia?" touches Outages + Cargo) and commodities ("How are naphtha prices affecting chemical production margins?" crosses Refined Products + Chemicals). Rather than one giant agent or forcing clients to configure a dozen servers, FastMCP's proxy + composition mounts each group-level Genie MCP server behind a single commodity server with name-spaced tools (cargo_genie_query_agent, outages_genie_query_agent, …); the agent's LLM decides which group Genie to route to, or fans a cross-group question across several and synthesizes. This is a content-aware instantiation of MCP as centralized integration proxy at the analytics altitude — "narrow, high-accuracy group-level Genie Agents underneath, and broad, commodity- and estate-wide conversational access on top."

  8. One MCP-standard bridge serves internal and external consumers. Because MCP is an open standard, the same composite endpoints serve internal agents, customer-facing AI experiences, and — critically — external customers connecting their own MCP-compatible agents directly to governed S&P Global Energy data. "We built the bridge once; every MCP client, internal or external, can cross it." A distinctive external-consumer framing for concepts/governed-agent-data-access.

  9. Answer quality is measured with Genie Agent Benchmarks, continuously. SMEs define test questions mirroring how users actually ask (including multiple phrasings of the same question) and score accuracy automatically against verified answers; benchmarks are rerun after any change to instructions, data, or business logic. "An efficient continuous quality loop: curate, benchmark, improve, and re-benchmark." The best leading indicator of adoption they tracked was how often SMEs agreed with Genie's generated SQL during curation — "Measure trust, not just latency."

Architecture (three layers, each owned by the right people)

Layer Owner What it is
1. Semantic layer SMEs (no code) One Genie Agent per dataset group; native tables via Unity Catalog, non-Databricks tables via Lakehouse Federation; enriched with descriptions, example queries, trusted assets, business definitions
2. Integration contract Databricks platform Each Genie Agent auto-exposed as a managed MCP server at /api/2.0/mcp/genie/{genie_space_id}; two-tool ask-then-poll surface; UC-governed, platform-authenticated; async execution on a SQL warehouse
3. Composition layer Engineering (one thin proxy) FastMCP proxy mounts group Genie MCP servers into per-commodity composite endpoints with name-spaced tools; higher-level composites bundle several commodities

Illustrative FastMCP composition (from the post)

from fastmcp import FastMCP

# Each group-level Genie Agent is a managed MCP server on Databricks
cargo    = FastMCP.as_proxy(genie_mcp_config("lng_cargo_agent_id"),    name="cargo")
outages  = FastMCP.as_proxy(genie_mcp_config("lng_outages_agent_id"),  name="outages")
netbacks = FastMCP.as_proxy(genie_mcp_config("lng_netbacks_agent_id"), name="netbacks")

# Compose the group Genies into one commodity bundle
lng = FastMCP(name="lng-composite")
lng.mount(cargo,    prefix="cargo")
lng.mount(outages,  prefix="outages")
lng.mount(netbacks, prefix="netbacks")

# The same pattern repeats for Chemicals, Crude Oil, Refined Products, Coal …

(Illustrative snippet from the post — "adapt to your FastMCP version and auth setup.")

Business outcomes

  • Time to market collapsed — new conversational data domains go live in days, not development cycles. Launching a new dataset group or entire commodity means an SME curates the Genie Agents; "the MCP endpoint exists the moment the agent does."
  • SMEs became publishers, not requesters — domain experts no longer file tickets to expose their data; engineering effort shifted from building bespoke access layers to maintaining "one thin, reusable proxy layer."
  • Governance came built-in — every question runs through Unity Catalog on governed native/federated tables with full auditability.
  • One integration pattern, many consumers — internal agents, customer-facing AI, and external customers' own MCP agents all cross the same bridge.
  • Measurable, maintained accuracy — Genie Agent Benchmarks provide a continuous curate → benchmark → improve → re-benchmark loop.

Best practices (verbatim intent)

  1. Keep Genie Agents narrow and well-curated — highest quality when one agent covers one specific domain; "resist the temptation to build one agent per commodity — or worse, one agent to rule them all. Bring together multiple data domains at the MCP layer instead."
  2. Use Lakehouse Federation before you build pipelines — federation reaches "conversational" without a single new ETL job; materialize hot paths later.
  3. Invest in the semantic layer — column descriptions, business definitions, trusted example queries "are what separate a demo from a product."
  4. Namespace your composite tools clearly — prefixes like cargo_ and outages_ help the LLM route correctly across many group Genies.
  5. Measure trust, not just latency — track how often SMEs agree with Genie's generated SQL during curation; it's the best leading indicator of business-user adoption.

Caveats / what's not disclosed

  • Customer-authored case study. This is a Databricks-blog customer post (S&P Global Energy VP Priyanka John quoted throughout); it is architecture- and-outcomes narrative, not an internals paper.
  • No hard operational numbers. No latency envelopes, per-query cost, QPS, concurrency limits, dataset/table counts, or benchmark accuracy figures are given.
  • FastMCP proxy internals not specified. The composition snippet is explicitly "illustrative"; auth passthrough details from the composite proxy down to each managed Genie MCP server (how the end-user identity/OBO token flows through the FastMCP mount to Unity Catalog) are not spelled out.
  • Federation enforcement mechanism unstated (consistent with the existing systems/lakehouse-federation page's open questions): how UC GRANT semantics push down to non-Databricks sources, and the latency of a federated Genie query vs a native one, are not disclosed.
  • Genie Agent Benchmarks are described at a capability level only — scoring method, verified-answer authoring workflow, and thresholds are not detailed.

Source

Last updated · 766 distilled / 2,225 read