Skip to content

DATABRICKS 2026-08-10

Read original ↗

How to ground Genie Agents in both structured data and documents without losing governance

Summary

Databricks explains how Genie Agents — the natural-language analytics agents built on Genie — can be grounded in both structured tables and unstructured documents while keeping Unity Catalog (UC), not the LLM, as the security perimeter. The central architectural claim is that Genie Agents run with the end user's credentials: every query is filtered at the data layer against the calling user's existing identity and permissions before any row leaves the Lakehouse, so the agent is incapable of returning a record the user is not authorized to see. This directly repudiates the common homegrown pattern of granting an agent broad access and relying on prompt engineering to filter results at the model layer — "making the LLM your security perimeter, a dangerous bet." The post walks a fictional global brick retailer, Brickstore, through four steps: (0) get identities right via Automatic Identity Management + Just-in-Time provisioning; (1) ground structured data and layer four access-control mechanisms — object privileges, ABAC, row filters, column masks; (2) extend the same governance to documents by landing them in UC Volumes; (3) validate in production by asking the same question as two differently-entitled users and comparing the answers.

Key takeaways

  1. Genie Agents run with the end user's credentials — UC is the security perimeter, not the model. "While Genie determines how to query the data, it is incapable of returning a record the end-user is not authorized to see, as every answer is filtered at the data layer before it ever leaves the Lakehouse." This is the run-with-end-user-credentials principle. (Source cited inline throughout.)

  2. Prompt-layer filtering is not a defensible control. "Telling an auditor that 'I added instructions that said to not show restricted data' is not a defensible governance control." Homegrown systems that grant agents broad access and filter in the model layer make the LLM the security perimeter — a bet that fails because models "can be manipulated or bypassed."

  3. Access controls are only as reliable as the identities they evaluate. Step 0 is identity sync: Automatic Identity Management (AIM) for Microsoft Entra ID and Okta syncs users, groups, group memberships, and service principals into Databricks with no SCIM app required, and Just-in-Time (JIT) provisioning is always on so a first-time user arrives already carrying their IdP group memberships. Governance becomes continuous, not a point-in-time setup: a transfer between regions moves the user between IdP groups, the sync propagates, and "the very next question they ask Genie returns the AMER view — without anyone filing a ticket or making changes to the Genie Agent."

  4. Four layers of structured-data access control, routinely conflated. The post tabulates them:

  5. Object Privileges — who can access what resource (GRANT SELECT on catalog/schema/table). The first layer: without SELECT, Genie can't query the table on the user's behalf.
  6. ABAC — which policy applies, via governed-tag-driven rules that attach once and propagate (e.g. any column tagged pii:email is masked).
  7. Row filters — which rows a user sees, via a SQL UDF evaluated per row at query time (rows where the function returns FALSE are excluded).

  8. Column Masks — which columns are masked and how, via a SQL UDF taking the column value and returning the original or a masked version. Row filters and column masks key off the same groups the grants use (is_account_group_member('brickstore_apac')).

  9. ABAC inverts per-table security: tag once, policy propagates. The old way was per-table (write a filter, attach to orders; write a mask, attach elsewhere; repeat) — "gap-prone across hundreds of tables." ABAC (GA in UC with governed tags + automated data classification) lets you tag sensitive data with governed tags (account-level, access-controlled key/value pairs) and write one policy: "wherever this tag appears, apply this protection." New tables inherit protection the moment they're tagged — no per-table work. See tag-driven-attribute-based-access-control.

  10. Metric Views are the governed semantic layer over the facts. Delta tables (brickstore.sales.orders, brickstore.sales.products) are the facts/dimensions; Metric Views encode business-metric definitions once in YAML (what "net revenue" means, how "bricks sold" is calculated) so every consumer computes them the same way. Genie can also read views, materialized views, streaming tables, and foreign tables federated from external systems.

  11. Documents are governed by putting them inside the same plane: UC Volumes. Historically unstructured data lived in isolated storage with separate ACLs. The fix: land files in Unity Catalog Volumes so they become securables like tables. GRANT READ VOLUME to the groups that should see them, and Genie reasons over them under the same identity contract. Supported formats: PDF, images (JPG/JPEG/PNG/TIFF/TIF), Office (DOC/DOCX/ PPT/PPTX), plus plain text and Markdown.

  12. An attached volume is a required source — volume grants gate agent use, not just document visibility. When a volume is attached to a Genie Agent it becomes a required source: the agent validates access to every attached source at load, so "a user who lacks READ VOLUME on an attached volume can't use that agent at all." A volume is also the smallest securable unit — permissions are all-or-nothing per volume, not per-file. Corollary: scope each agent's document sources to the audience that should use it; if two audiences need different documents, give them different Genie Agents. See volume-as-agent-knowledge-source-with-required-access.

  13. Same question, different correct answers — with zero per-user prompt engineering. Two concurrent sessions grounded in identical assets (orders, products, market_report volume) — one requester in brickstore_apac, one in brickstore_amer — ask the same question ("top seller this quarter, what's driving demand, list top customers and their emails"). Each gets a different, correct answer: "The numbers are different, and both are correct… the difference is purely the rows each is entitled to, not a difference in how the metric was computed. The difference required zero per-user prompt engineering." UC filtered rows and masked customer_email at query time.

  14. Test governance by impersonation, not inspection. "Don't validate governance by reading the policy and convincing yourself it's right — ask the same question as a member of each group and compare the responses. Make it a regression test and run it whenever policies or groupings change." See test-governance-by-impersonation.

Patterns to watch (from the post)

  • Tag, then policy. Don't mask table-by-table. Define governed tags + ABAC policies to future-proof governance.
  • One audience per volume. The volume is the smallest grantable unit, so decide document access at the volume boundary; different readers → different volumes → different agents. Plan the layout upfront.
  • Handle identity carefully when surfacing Genie externally via MCP or API. Outside the Databricks UI you are not always granted the end user's identity (e.g. when a Service Principal authenticates). Databricks details the U2M, M2M, and OBO configurations in "Access Genie everywhere."
  • Test by impersonation, not inspection (as above).

Operational specifics

  • Identity providers: Microsoft Entra ID, Okta (AIM); JIT provisioning always on; no SCIM application required.
  • Group-membership predicate: is_account_group_member('brickstore_apac').
  • Governed-tag example: pii:email (account-level key/value pair).
  • Supported document formats in volumes: PDF; JPG, JPEG, PNG, TIFF, TIF; DOC, DOCX, PPT, PPTX; plain text; Markdown.
  • External-access auth modes: U2M (user-to-machine), M2M (machine-to-machine), OBO (on-behalf-of).

Caveats

  • The post is a Databricks product/best-practice piece (Tier-3 source), but the architecture content — credential propagation, query-time enforcement, four-layer access control, volume-as-securable, impersonation testing — is well above the 20% threshold, so it is in scope. Marketing framing ("Brickstore", CTAs to docs) is stripped here.
  • Volume-level all-or-nothing granularity is a real design constraint, not a limitation to be worked around: it forces the "one audience per volume / per agent" layout.
  • The exact ABAC policy SQL is shown as headers in the raw post but the code blocks are not rendered in the fetched markdown; the behavior (mask every email column in one statement; region-scoped row filter by group) is captured above.

Source

Last updated · 766 distilled / 2,225 read