Skip to content

SYSTEM Cited by 2 sources

Netflix Maestro

Maestro is Netflix's horizontally scalable workflow orchestrator, designed to manage large-scale data and ML workflows. It oversees the entire lifecycle of a workflow — retries, queuing, task distribution to compute engines — and provides Workflow-as-a-Service to thousands of Netflix end users, applications, and services. Production scale as of 2024-07: ~500,000 jobs/day on average, ~2,000,000 jobs on peak days, thousands of workflow instances launched daily, 87.5% YoY growth in executed jobs, hundreds of thousands of workflows migrated internally (Source: sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator). Open-sourced at github.com/Netflix/maestro.

Architectural stance

  • Single cluster, horizontally scalable. "Since Netflix's data tables are housed in a single data warehouse, we believe a single orchestrator should handle all workflows accessing it." Splitting workflows across clusters "adds unnecessary complexity… [and] requires additional mechanisms to coordinate these fragmented workflows."
  • Both acyclic AND cyclic workflows — not just DAGs, unlike most mainstream orchestrators. See concepts/dag-vs-cyclic-workflow.
  • JSON workflow definition — two sections: properties (author, owner, execution settings preserved across versions) + versioned workflow (unique ID, name, description, tags, timeouts, low / medium / high criticality for prioritisation, steps with step type / input-output parameters / dependencies / retry policies / failure mode / outputs). Each workflow change creates a new version; active-or-latest version used by default.

Primitives / building blocks

Run strategies — concepts/workflow-run-strategy

Maestro ships five predefined run strategies:

Strategy Semantics Canonical use
Sequential (default) FIFO; runs on terminal state transitions regardless of success/failure Normal pipelines with independent runs
Strict Sequential FIFO, but blocks future runs when a prior instance is in a blocking-error state until manual mark-unblocked or restart "Time-insensitive but business-critical" pipelines
First-only If one running, drop newly queued; "effectively turning off queuing" Non-idempotent flows — "helps to avoid idempotency issues by not queuing new workflow instances"
Last-only Stop running instance when a newer one arrives; keep only latest "Process the latest snapshot of an entire table each time"
Parallel with Concurrency Limit Many runs in parallel, capped Backfilling old data within a deadline

Composite step types — patterns/composite-workflow-pattern

Three named patterns implemented directly in the engine (rather than in user code) so that they can be optimised and so their semantics are consistent across the organisation:

  1. Foreach — "Each iteration of the foreach loop is internally treated as a separate workflow instance, which scales similarly as any other Maestro workflow." The foreach step monitors + collects statuses of the per-iteration workflow instances. Canonical use: data backfills, ML model tuning.
  2. Conditional branch — subsequent steps run only if SEL conditions in the upstream step are met. Composes with other primitives to build e.g. audit-check remediation flows.
  3. Subworkflow — "workflow as a function"; a step runs another workflow, enabling a graph of workflows with shared common functions. Netflix runs "complex workflows consisting of hundreds of subworkflows to process data across hundreds tables, where subworkflows are provided by multiple teams."

Composite patterns (loops over subworkflows, nested foreach, conditional-subworkflow-with-recovery) are explicitly supported — see the auto-recovery example in the post (subworkflow job1 → status-check → conditional branch → recovery subworkflow → retry via subworkflow job2).

Parameters + SEL — concepts/parameterized-workflow · systems/netflix-sel · patterns/sel-sandboxed-expression-language

  • Dynamic parameters with code injection for sharing state between workflows + steps + upstream/downstream data flow.
  • Threat model is explicit: "users might unintentionally write an infinite loop that creates an array and appends items to it, eventually crashing the server with out-of-memory (OOM) issues."
  • Solution: SEL — a homemade subset of the Java Language Specification with runtime limits (loop- iteration, array-size, object-memory) + Java Security Manager sandbox. "SEL supports code injection while incorporating validations during syntax tree parsing to protect the system."
  • Callable step execution — step runs return output parameters back to Maestro via REST API; step runtime has no direct Maestro- DB access.
  • Parameterized workflows strike balance between static (simple but duplication-prone) and fully dynamic (flexible but hard to debug / reuse).

Step parameter merging — concepts/step-parameter-merging

Seven-layer deterministic merge at step runtime:

  1. Default general parameters (reserved — user cannot override): workflow_instance_id, step_instance_uuid, step_attempt_id, step_id.
  2. Injected parameters from step runtime, dynamically generated from step schema — each step type has its own schema evolving independent of Maestro.
  3. Default typed parameters — step-type-specific defaults (e.g. loop_params, loop_index for foreach).
  4. Workflow + step info parameters — identity metadata, e.g. workflow_id.
  5. New undefined parameters user supplies at start/restart.
  6. Step definition parameters from the workflow JSON.
  7. Run + restart override parameters — "merged at the end so that step runtime can see the most recent and accurate parameter space."

Signals + step dependencies — concepts/signal-based-step-dependency · patterns/signal-publish-subscribe-step-trigger

  • Signals are messages carrying parameter values, produced either by step outputs or external systems (systems/kafka / SNS).
  • Step dependency specifies data-related conditions required before execution starts.
  • Signal matching — a subset of mapped parameter fields compared with operators (<, >, =, etc.) — see SignalOperator.java.
  • Dual semantics: signals serve both publish-subscribe patterns (one producer unblocks many consumers) AND trigger patterns (external event starts a new flow).
  • Exactly-once execution guarantee — "Signal triggering guarantees exactly-once execution for the workflow subscribing a signal or a set of joined signals." See concepts/exactly-once-signal-trigger.
  • Signal lineage — "navigate all historical instances of signals and the workflow steps that match (i.e. publishing or consuming) those signals." Query-able history of producer / consumer / match events.
  • Canonical use: ETL workflow updates a table and publishes a signal with the partition key; downstream workflows subscribed to that table/partition run exactly once per commit.

Breakpoints — concepts/workflow-breakpoint · patterns/workflow-step-breakpoint

  • IDE-style pause-at-step for workflow debugging.
  • Per-instance resume: pausing one instance doesn't affect others; deleting the breakpoint resumes all paused instances.
  • Foreach-aware — "Setting a single breakpoint on a step will cause all iterations of the foreach loop to pause at that step for debugging purposes."
  • Production uses: initial workflow development (inspect step executions / outputs), debug foreach iterations with many parameter combinations, mutate step state while workflow is running.

Timeline + aggregated view + rollup

  • Timeline — audit trail of significant events per step (state-machine transitions, reasoning). Examples: "Created," "Evaluating params." See sample-step-instance-failed.json#L137-L150.
  • concepts/workflow-aggregated-view — merge base aggregated view (statuses from prior runs) with current-run statuses on multi-run workflow instances. Useful across restarts: run 1 succeeds step1+step2, fails step3, skips step4-5; restart begins at step3 in run 2 with step1+step2 treated as skipped-but-succeeded; aggregated view shows terminal status across all steps once complete.
  • concepts/workflow-rollup — recursive leaf-step status counts across nested subworkflows + foreach iterations. "Only leaf steps are counted in the rollup, as other steps serve merely as pointers to concrete workflows." Retains references to non-successful steps for navigation. "The rollup model is eventually consistent" — derived from (prior-run base rollup
  • current run's leaf statuses + recursive subworkflow / foreach rollup merges).

Retry policies

  • Platform retries — for infrastructure errors unrelated to user logic.
  • User retries — for user-defined conditions.
  • Each has its own policy: count, delay strategy (fixed interval / exponential backoff), per-error-type rules.
  • Zero retries for non-idempotent steps — "Maestro provides the flexibility to set retries to zero for non-idempotent steps to avoid retry."

Event publishing — patterns/internal-external-event-pipeline

Two-tier queue pipeline:

  1. Internal events — workflow / instance / step-instance state-machine changes published to an internal Maestro queue. See com.netflix.maestro.engine.jobevents.
  2. Event processor — consumes internal events, dispatches based on type, converts to external events where applicable. See com.netflix.maestro.models.events.
  3. External events — emitted to external queues (Kafka / SNS) for downstream event-driven services.

Two classes of external events:

  • Workflow change events — workflow-level actions (definition / properties change).
  • Instance status change events — workflow-instance or step- instance state transitions.

Position in the Metaflow stack

For open-source Metaflow users the supported orchestrators are AWS Step Functions, Argo Workflows (Kubernetes-native), and Airflow. Inside Netflix, Maestro replaces all three and adds the deeper primitives above (signal triggering, per-step breakpoints, run strategies, composite step types, safe parameterized workflows via SEL). "a hugely important detail that often goes overlooked: [event-triggering] allows a team to integrate their Metaflow flows to surrounding systems upstream (e.g. ETL workflows), as well as downstream (e.g. flows managed by other teams), using a protocol shared by the whole organization." The named canonical production workload on Maestro is Netflix's Content Decision Making system — a flow graph of hundreds of models + intricate business logic supporting content decisions for 260M+ subscribers across 190+ countries, "managed by a relatively small team of engineers and data scientists autonomously." (Source: sources/2024-07-22-netflix-supporting-diverse-ml-systems-at-netflix).

Comparison — Maestro vs named peer orchestrators

Axis Maestro Temporal / Cadence Airflow / Step Functions / Argo
Primary workload Data + ML pipelines General durable workflows Data pipelines
Cyclic support ✅ ✅ (via loops in code) ❌ (DAG only)
Composite primitives Engine-native (foreach, subworkflow, cond) Code-level (loops, child workflows) Limited (subDAG, branch operators)
Parameterised workflows SEL safe-DSL Native code Templated via Jinja / expr
Code-injection safety Java Security Manager + SEL limits N/A (full code) N/A (template-scoped)
Run strategies 5 named (Seq / Strict / First / Last / Parallel) User-implemented Schedule / sensor based
Signal-based deps ✅ Exactly-once + signal lineage Signal channels (Temporal) Sensors
Breakpoints ✅ Per-step pause Debug via replay ❌
Open source ✅ (2024-07) ✅ ✅

Seen in

Last updated · 319 distilled / 1,201 read