PATTERN Cited by 1 source

PromQL-to-SQL over Delta tables¶

Build a translation layer that accepts PromQL from Grafana (or any Prometheus-compatible tool) and executes the query as SQL against a Delta Lake / Iceberg lakehouse, rather than against an in-memory TSDB.

The user-facing interface (Grafana + PromQL) stays identical; the storage substrate changes from a TSDB to a columnar lakehouse. Engineers keep their dashboards, alert rules, and workflows unchanged — the platform team gets lakehouse economics for the raw-data tier.

Why this is the load-bearing enabler for lakehouse observability¶

Without the translation layer, moving observability data to a lakehouse would force engineers to rewrite every dashboard and alert rule in SQL. That migration cost is so large it blocks the substrate change. With translation, the change is invisible to the dashboard layer — see concepts/critical-user-journey for the user-facing-interface- is-the-stable-contract framing.

Architecture¶

  Grafana                                        Translation Layer
   PromQL query  ─▶    (intercepts PromQL)   ─▶   Parser ─▶  Rewriter ─▶  SQL
                                                                           │
                                                                           ▼
                                                                  Lakehouse SQL engine
                                                                  (Databricks SQL / Photon)
                                                                           │
                                                                           ▼
  Grafana   ◀────  PromQL response format   ◀────  Result shape adapter

PromQL parser — builds the abstract query tree (instant vectors, range vectors, aggregation operators, rate / irate / increase, histogram quantile, subqueries, label matchers).
Rewriter — translates the tree into SQL against the Delta schema. Metrics typically land in tables keyed on timestamp, metric_name, plus label columns or a typed map column. Partitioning by region / time.
SQL engine — executes the query using the lakehouse's query engine (Databricks SQL Warehouse / Photon / Spark SQL / Trino / DuckDB over Delta).
Result adapter — reshapes the SQL rowset into the PromQL-response-format Grafana expects.

Query-plan optimizations the lakehouse engine provides¶

Predicate pushdown — label filters translate to column / map predicates that Delta scans use for file-skipping.
Z-order clustering on high-cardinality labels (metric name, tenant, region) co-locates the rows for a given series on disk — critical for bounded query time on a multi-billion- row table.
Partition pruning — time-range and region predicates drop entire partitions from the scan plan.
Columnar scan — the SQL engine reads only the columns actually used by the query (timestamp + value + filtered labels), not the full row.

Fidelity concerns¶

Not all of PromQL round-trips cleanly:

Range-vector semantics — rate(x[5m]) needs per-series windowing; requires SQL window functions + carefully-defined "same series" partitioning.
rate() on partial data — PromQL extrapolates across gaps; faithful translation must do the same.
Histogram quantile arithmetic — PromQL's histogram_quantile(0.99, ...) operates on bucket pairs; the SQL version needs equivalent bucket-bounded interpolation (SQL has no native histogram quantile function, so this is typically a UDF).
Staleness markers — PromQL has a concept of "stale"; the lakehouse representation must track staleness explicitly or approximate it.
Instant-vector vector-match semantics — on() / ignoring() / group-left / group-right join semantics need faithful rendering in SQL.

A production translation layer tracks all of these — Hydra claims Grafana dashboards work without modification, but the post acknowledges ("improving the performance of Hydra so it achieves similar data freshness to Pantheon and the two experiences converge even further") that freshness + full parity are ongoing work.

Seen in¶

sources/2026-05-05-databricks-10-trillion-samples-a-day-scaling-beyond-traditional-monitoring — canonical instance. "Hydra integrates directly with Grafana by enabling PromQL queries to run against data stored in Databricks. We built a PromQL-to-SQL conversion layer that translates PromQL expressions into SQL queries executed on Delta tables in the Lakehouse. This approach allows engineers to continue using familiar PromQL syntax and dashboards without modification. At the same time, the underlying queries are executed against large-scale Delta tables rather than an in-memory TSDB."