PATTERN Cited by 1 source
Job of jobs orchestration¶
The "Job of Jobs" pattern is the orchestration shape used by the Octopus Energy MHHS rebuild to manage the three grain-aligned streams of its margin data pipeline: a parent job composing child jobs, with the parent responsible for dependency management and parallel execution across the children while each child preserves its own independent tuning profile. The verbatim disclosure: "A 'Job of Jobs' orchestration pattern manages dependencies and parallel execution across all three streams." (Source: sources/2026-05-23-databricks-scaling-for-mhhs-octopus-energy-50x-cost-reduction)
Shape¶
┌──────────────────────────────────┐
│ Parent: Job of Jobs │
│ - dependency management │
│ - parallel execution │
│ - cross-stream coordination │
└────────────────┬─────────────────┘
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Child A │ │ Child B │ │ Child C │
│ (Stream 1) │ │ (Stream 2) │ │ (Stream 3) │
│ own tuning │ │ own tuning │ │ own tuning │
│ profile │ │ profile │ │ profile │
└─────────────┘ └─────────────┘ └─────────────┘
The pattern's defining property is the split of responsibilities between parent and children:
| Concern | Parent (Job of Jobs) | Child (per-stream job) |
|---|---|---|
| Inter-stream dependencies | Owns (e.g., monthly stream depends on settlement having completed for the period) | Doesn't see |
| Parallel execution | Owns (decides which children run concurrently) | Doesn't see |
| Stream-internal tuning | Doesn't see | Owns (Spark config, optimiser hints, partition strategy) |
| Per-stream alerting | Aggregates from children | Owns the per-stream signal |
When to apply¶
Apply when all are true:
- Multiple pipelines (or streams) share inputs / a substrate but have independent tuning profiles.
- There are cross-stream dependencies that must be enforced (stream B depends on stream A having advanced its watermark past version N).
- Parallel execution within and across streams is a first-order concern.
- You don't want a single Spark / DLT config to apply to all streams.
The Octopus rebuild satisfies all four — "Each stream is independently tunable — what works as a Spark optimisation for Settlement is not necessarily right for NHH".
Steps¶
- Define each stream as an independently-tunable job. Each carries its own cluster config, Spark config, optimiser hints, partition strategy. Per-stream alerting and SLAs.
- Define a parent job that invokes the children. Common implementations: Databricks Workflows / Lakeflow Jobs task graphs, Apache Airflow DAGs (parent DAG triggering child DAGs), Dagster job factories.
- Encode inter-stream dependencies as parent-job edges, not as inside-stream wait logic. The parent decides "settlement must complete before monthly's reconciliation step"; the children stay simple.
- Use the parent for parallelism control. The parent decides "settlement and HH run in parallel; monthly waits for both". This keeps each child agnostic about who else is running.
- Compose with patterns/grain-aligned-stream-split — the parent-child split is the orchestration realisation of a grain-aligned multi-stream pipeline.
Why this beats a single monolithic job¶
The naive alternative — a single monolithic Spark job with all three streams' logic in it — has well-known failure modes:
| Failure mode | What goes wrong |
|---|---|
| Shared tuning profile | Choices that help one stream hurt another |
| Cascading failures | One stream's failure aborts all of them |
| Coarse-grained alerting | Hard to attribute a slowdown to a specific stream |
| Parallel-execution opacity | Unclear which streams could run concurrently |
The Job-of-Jobs split addresses each — children are isolated for tuning, alerting, and failure containment; the parent owns the cross-stream coordination explicitly.
Why this beats N independent jobs with no parent¶
The opposite extreme — N independent jobs with no orchestrator — loses cross-stream coordination:
- Inter-stream dependencies become wait-loop hacks baked into each child.
- Parallelism becomes an external concern — manual scheduling, or a separate scheduler nobody owns.
- Reconciliation breaks — no single place to enforce "all three streams have advanced past version N" before downstream consumers read.
The parent job is the explicit owner of the cross-stream coordination that would otherwise leak into the children.
Trade-offs¶
- Two-level operational surface. Failures can be at parent (orchestration / dependency) or child (stream-internal) level. Alerting and runbooks have to distinguish.
- Parent becomes a critical path. Parent-job bugs affect all children. The parent's own change cadence has to be slower / more carefully tested than the children's.
- Implementation discipline required. Cross-stream coupling that should live in the parent often leaks into children when teams optimise locally. Code review needs to push it back up.
Implementation choices¶
The Octopus source does not name the orchestration runtime. Likely candidates given the Databricks platform:
- Databricks Workflows / Lakeflow Jobs task graphs — first-class support for parent jobs invoking child jobs as tasks; native UI for dependency visualisation; per-task cluster choice.
- Apache Airflow — parent DAG triggering child DAGs via
TriggerDagRunOperator; widely adopted but adds out-of-platform state. - Dagster — job-of-jobs is a first-class concept; tighter Python-side typing of inter-stream contracts.
Composition with related patterns¶
- patterns/grain-aligned-stream-split — the architectural shape; Job-of-Jobs is its orchestration realisation.
- patterns/cdf-incremental-replacing-full-rescan — each child stream consumes the shared substrate's CDF feed at its own watermark; the parent doesn't own watermark state.
- concepts/remove-before-add-optimization — applies inside each child independently; per-stream measurement makes per-stream audit feasible.
Seen in¶
- sources/2026-05-23-databricks-scaling-for-mhhs-octopus-energy-50x-cost-reduction — canonical disclosure. "A 'Job of Jobs' orchestration pattern manages dependencies and parallel execution across all three streams. Each stream is independently tunable — what works as a Spark optimisation for Settlement is not necessarily right for NHH." Applied to the three-stream margin data pipeline at Octopus Energy.