Skip to content

How to get the most out of running dbt on Databricks

Summary

A Databricks practices roundup for the 3000+ organizations running dbt on Databricks, organized around one thesis: hand operational work to the platform and keep only version-controlled, tested transformation logic in your dbt project. Two halves — "cheaper and faster" (incremental strategies, materialized views, automatic data layout, query-tag cost attribution) and "simpler to operate" (streaming-table ingestion, Unity Catalog metric views, native Lakeflow orchestration, deploy-as-code, in-project governance). The post is a lighter, feature-oriented companion to the much deeper IFCO case study (sources/2026-10-06-databricks-scaling-and-operating-a-large-dbt-project-on-databricks-ifco): it names the same dbt-databricks levers (incremental strategies, liquid clustering, Predictive Optimization) but as a checklist rather than a measured rebuild, and adds product features that article didn't cover (query tags, streaming-table materialization, metric views, DABs, in-project ABAC). Tier-3, borderline-INCLUDE: product-advocacy framing, but the incremental-processing and data-layout trade-offs are genuine data-warehouse design content.

Key takeaways

  1. Full table rebuilds are the largest avoidable cost; pick the incremental strategy that matches how data arrives. dbt-databricks supports the full matrix — append (insert-only), merge (keyed upsert / SCD / dedupe / late-arriving updates), replace_where (re-derive a narrow time window, dbt-databricks-specific), insert_overwrite (whole partitions), delete+insert (replace-by-key without column-level update logic, dbt-databricks-specific), and microbatch (time-series with an event_time column + structured backfill). (Source: this article)
  2. Most projects default to merge when a narrower strategy would do. If the changed data is describable by a predicate, replace_where does less work than a merge that must scan the target to find matches; if not, incremental_predicates narrows the merge scan. This is the same "touch as few rows as possible" principle the IFCO rebuild measured in depth.
  3. Materialized views automate incrementalization so you don't maintain a strategy at all. A materialized view is a one-line native dbt-databricks materialization: define plain SQL, and Databricks tracks which underlying data changed and refreshes only affected results — the IVM engine does the work. Best fit is the serving layer (aggregations over frequently changing sources that back dashboards). TPC-DI benchmark: switching full recomputes to materialized views at the serving layer delivered 17× speedup. (Source: this article)
  4. Automate data layout instead of hand-tuning partitions and maintenance. Predictive Optimization runs OPTIMIZE / VACUUM / statistics collection on managed tables automatically (enable once at catalog/schema level → retire the OPTIMIZE/VACUUM/ANALYZE post-hooks dbt projects accumulate). Liquid Clustering replaces partition design with a self-tuning layout that avoids the small-file explosion of over-partitioned tables and adapts as query patterns drift; CLUSTER BY AUTO lets Predictive Optimization (via AutoLiquid) choose and adjust the keys from query history. Both are one-time decisions; Automatic Upgrades keep managed tables picking up new optimizations. (Source: this article)
  5. Query tags attribute every dollar back to the dbt model that spent it. Every query dbt runs on Databricks is automatically stamped with model name, dbt version, and materialization type; custom DIY tags add team/domain/etc. Tags surface in system.query.history; joining against system.billing.usage attributes serverless warehouse spend to any grouping — cost by model, team, domain, or materialization. Closes the "finance asks why the bill went up and nobody can answer" gap because a raw query carried no link to its dbt model. (Source: this article)
  6. Bring ingestion into the dbt DAG with the streaming_table materialization. Ingestion historically lived outside dbt (split architecture, no single lineage). A dbt model materialized as a streaming table incrementally picks up new files/events as they arrive, so ingestion becomes just another model in the DAG — silver/gold models ref() it, and lineage, tests, and docs cover the whole path from raw files to final mart ( bronze→silver→gold). (Source: this article)
  7. Standardize metrics once with Unity Catalog metric views. A metric view moves a metric definition into one governed place (declare measures + dimensions once; every consumer queries the same logic via MEASURE()), exposed as a native dbt-databricks materialization so the definition is versioned and reviewed like any model. One definition serves the SQL editor, BI dashboards, notebooks, and Genie.
  8. Orchestrate dbt natively with Lakeflow Jobs instead of a third-party orchestrator. Lakeflow Jobs runs dbt as a native task inside one DAG — an ingestion task feeds the dbt task, downstream tasks trigger on completion, dependencies enforced by the platform — with scheduling, automatic retries, run history, and alerts built in. Supports dbt Core and dbt Platform; dbt Core v2.0 support is on the way. (Source: this article)
  9. Deploy the whole pipeline as code with Declarative Automation Bundles (DABs). A UI-assembled job is hard to version/review/reproduce. DABs declare the Lakeflow Job, schedule, compute settings, and dbt project in YAML, deployed as one package: --target dev creates an isolated copy of the whole pipeline, --target prod promotes it. Tag models by cadence in dbt YAML and give each cadence its own job → moving a model from daily to hourly is a one-line change. (Source: this article)
  10. Define governance in the dbt project, not after it runs. Tags, grants, and row/column protection all live in project-wide dbt_project.yml config (databricks_tags drives classification/policy, grants controls read access, row_filters/column_masks control per-user visibility) — shipped with the code, reviewed like the code, enforced by Unity Catalog on every query. At scale, ABAC removes the repetition: one policy on a governed tag applies the right filters/masks to every table carrying that tag, so a new model tagged pii is covered the moment it builds. Reach for per-model filters/masks only for genuinely model-specific logic (e.g. multi-tenant isolation). (Source: this article)

Systems / concepts / patterns extracted

Operational numbers

  • 3000+ organizations run dbt on Databricks.
  • 17× speedup — switching full recomputes to materialized views at the serving layer, on the TPC-DI benchmark.
  • --target dev / --target prod as the isolated-copy vs promote deploy toggle; cadence-tagged jobs make a daily→hourly move a one-line YAML change.

Caveats

  • Tier-3 Databricks product-advocacy post, not a measured engineering writeup. The only quantitative claim is the 17× TPC-DI materialized-view figure (benchmark methodology not detailed in-post) and the 3000+-orgs count.
  • Substantially duplicative of the deeper IFCO source (sources/2026-10-06-databricks-scaling-and-operating-a-large-dbt-project-on-databricks-ifco), which measures the incremental-strategy / liquid-clustering / Predictive- Optimization levers in a real rebuild (≈66% runtime cut, 58% compute cut). Treat this page as the feature-checklist companion; cite IFCO for the internals.
  • Every feature is described at capability altitude — no internals for the MV refresh engine, the CLUSTER BY AUTO key-selection algorithm, query-tag storage, or the streaming-table ingestion path. The product links point to dbt-databricks and Databricks docs, not architecture writeups.
  • dbt Core v2.0 support in Lakeflow Jobs is stated as forthcoming, not GA.

Source

Last updated · 854 distilled / 2,448 read