Skip to content

CONCEPT Cited by 1 source

Step parameter merging

Step parameter merging is the deterministic pipeline that constructs the effective parameter space a workflow step sees at execution time, by combining parameter sources with well-defined precedence. Getting this merge order right is essential for debuggability: "step runtime can see the most recent and accurate parameter space" (Source: sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator).

Canonical wiki instance is the seven-layer merge pipeline in Netflix Maestro's step-runtime interface, applied in this order (each later layer shadows or augments the earlier layers):

The seven layers, in order

1. Default general parameters (reserved)

Every Maestro step receives a handful of system-supplied, reserved identity parameters that users cannot override:

  • workflow_instance_id
  • step_instance_uuid
  • step_attempt_id
  • step_id

These anchor the step to its concrete execution context and are used by Maestro's own lineage + retry + rollup subsystems.

2. Injected parameters

Injected from step runtime at evaluation time, dynamically generated from the step schema. Each step type has its own schema that evolves independently of the Maestro core — so the injected-parameter set is step-type-dependent.

3. Default typed parameters

Step-type-specific defaults — e.g. foreach steps receive loop_params + loop_index automatically; subworkflow steps receive subworkflow-specific defaults. Merged after injected parameters.

4. Workflow + step info parameters

Identity metadata about the enclosing workflow: workflow_id, workflow properties, step info. Merged if present.

5. New undefined parameters (user-supplied at start/restart)

At start or restart time users can supply parameters not present in the original step definition. "When starting or restarting a maestro workflow instance, users can specify new step parameters that are not present in initial step definition. ParamsManager merges these parameters to ensure they are available at execution time."

6. Step definition parameters

Parameters the workflow author declared in the step's JSON definition — the "statically-declared" layer. Merged if non-empty.

7. Run + restart override parameters

At start or restart time users can also override previously defined parameters — these merge last, on top of everything.

"These two types of parameters are merged at the end so that step runtime can see the most recent and accurate parameter space." (Source: sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator)

Why the order matters

The pipeline encodes three architectural principles:

  1. Reserved system identity parameters are inviolable — they come first and cannot be shadowed by any later layer because user parameters don't reach them (or they're explicitly protected). This preserves auditability + lineage.
  2. Static before dynamic; defaults before overrides — step- type defaults → workflow info → user-defined static → run-time override lets the operator break-glass at execution time without modifying the workflow definition. Essential for backfills, ad-hoc re-runs, and incident remediation.
  3. Transparency — by fixing the order, the effective parameter space is derivable from the inputs, not a function of evaluation order. Users can answer "why does my step see value X?" by tracing down the seven layers.

Why this isn't just "config merge"

The merge pipeline is specifically per-step at each invocation — every foreach iteration, every restart attempt, every subworkflow entry reconstructs its own parameter space. This differs from traditional configuration merging (which runs once at service start) in three ways:

  • Runtime values from prior steps flow in via injected parameters — the merge is not pure-static.
  • Attempt-level identity (step_attempt_id) appears in layer 1, so retries see a distinct parameter space from the original attempt.
  • Override parameters at start / restart time are persisted in the workflow instance + flow through to every step instance they apply to.

Seen in

Last updated · 319 distilled / 1,201 read