Skip to content

SYSTEM Cited by 1 source

databricks-dbt-factory

databricks-dbt-factory is a standalone, MIT-licensed open-source library (GitHub, PyPI) that reads a dbt manifest and a job template and produces a Databricks Asset Bundle job with one task per dbt node — model, test, seed, and snapshot. It is the mechanism that turns a dbt project from a single opaque Lakeflow Jobs task into a per-model task graph. (Source: sources/2026-10-06-databricks-scaling-and-operating-a-large-dbt-project-on-databricks-ifco)

Why it exists — the black-box problem

The simplest way to run a dbt project on Databricks Workflows is as a single dbt task. It works, but it's a black box: one model fails and the entire job fails, with no way to see, rerun, or watch individual models. At IFCO's scale (a large semantic layer over billions of tracking events) that is an operational risk. databricks-dbt-factory inverts this: it fans the dbt DAG out into individual Databricks tasks so operations gets task-level visibility, targeted reruns of only the failed model and its dependents, per-model logging, alerting, and testing.

How it works

It consumes two inputs — the dbt manifest (the compiled DAG + node metadata) and a job template — and emits a Databricks Asset Bundle job whose task graph mirrors the dbt node graph. Per-task granularity only pays off if each task is cheap to start, which rests on three mechanisms IFCO pairs it with:

  • Notebook task type. A small shared runner notebook triggers dbt through its Python API; dbt computes the SQL and sends it to the SQL warehouse, so the notebook compute crunches no data.
  • Base environments. A pre-built serverless Python environment snapshot already containing dbt-databricks, so each task skips the pip install a fresh task would pay.
  • Manifest injection. A pre-built manifest is handed straight to dbt so parsing is skipped, and each task writes artifacts to a private local directory — on a large project run by many tasks at once, the difference between minutes of re-reading the project and useful work.

Operational properties

  • Output is a Databricks Asset Bundle, so the generated job deploys through the same path as any other DAB — IFCO ships it via a path-aware GitHub Actions matrix.
  • The runner notebook is extensible: load secrets, tag a run with a git SHA, or post to Slack in a few lines.
  • Composes with per-model observability: query tags + system tables feed a dashboard tracking per-model runtime and cost, so a regression surfaces in a day rather than in a monthly bill. (See patterns/measurement-driven-micro-optimization.)

Seen in

Last updated · 854 distilled / 2,448 read