Skip to content

CONCEPT Cited by 1 source

Schema discovery vs data access separation

A data-platform governance affordance in which users can see what tables exist (their names, possibly some lineage and ownership metadata) but cannot see column-level information or data for tables / columns they haven't been authorised for. DESCRIBE, SHOW COLUMNS, and SELECT * honour the access boundary; bare table-name discovery does not.

Canonical wiki instance

Cloudflare Town Lake, from the 2026-05-28 launch post:

"We separate schema discovery from data access. Users can see what tables exist, but unreviewed columns are hidden from DESCRIBE and SHOW COLUMNS and from SELECT *. That subtle distinction matters: it means a new unreviewed column doesn't break existing dashboards built on the rest of an approved table."

Why the separation matters

The structural argument is dashboard / pipeline stability under governance churn. Without this separation, every new column added to an existing table — even if no one queries it yet — would change SELECT * results and break consumers that rely on column ordering or schema introspection.

Two concrete failure modes the separation prevents:

  1. SELECT * schema drift — a dashboard that did SELECT * FROM dim.accounts would break the moment a new column lands, until that column completes review. The separation lets new columns enter as invisible to SELECT * until reviewed, while existing approved columns continue to be returned.
  2. DESCRIBE-driven validation — pipelines that introspect schemas (e.g., dbt-style downstream validation) would see "new column" signals before review is complete, potentially triggering false alerts or pipeline reroutes.

What "schema discovery" still exposes

The Town Lake post doesn't enumerate the line precisely, but the affordances that survive across the access boundary are implicitly:

  • Table existence — "users can see what tables exist."
  • Table-level lineage / ownership / tags — DataHub-resident metadata on the table as a whole.
  • The fact that columns exist (count, possibly names) — enough for self-serve permission requests to be meaningful ("ask for access to this table because it has the column you need").

What's hidden:

  • Column-level details for unreviewed columns — DESCRIBE TABLE and SHOW COLUMNS filter unreviewed entries out.
  • Column data — SELECT * won't return them.

Engine-level mechanism (Trino)

In Town Lake's implementation, the enforcement runs through Lifeguard's JSON policy (served to Trino over HTTP). Trino's column-level authorisation is the engine-side substrate; Lifeguard supplies the per-column allow / deny / pending state, and Trino filters schema-introspection results accordingly.

Why default-closed governance depends on this

Without separation, default-closed becomes operationally hostile. Every schema migration would break dashboards until the review queue caught up. Teams would either:

  • Delay schema migrations (slow data engineering velocity), or
  • Skip review for new columns (defeats the governance posture).

The separation lets new columns enter the platform invisibly — discoverable as "there's something here, ask for access if you want it" but not breaking any existing approved-column consumer.

This makes the separation a load-bearing affordance for default-closed table allowlisting — it's the property that keeps the governance posture sustainable at organisational scale.

Seen in

Last updated · 542 distilled / 1,571 read