Skip to content

CONCEPT Cited by 1 source

Parameterized workflow

A parameterized workflow is a workflow whose structure (number of steps, which steps run, values passed between steps) is initialised step-by-step at runtime based on user-defined parameters — a middle ground between two unsatisfying extremes:

  1. Static workflows — simple to reason about, but users must duplicate the workflow definition every time they want minor variations. No state-sharing between workflows / jobs without out-of-band mechanisms.
  2. Fully dynamic workflows — flexible but "can be challenging to manage and support. They are difficult to debug or troubleshoot and hard to be reused by others."

Canonical wiki statement + design stance from Netflix Maestro (Source: sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator):

"Parameterized workflows strike a balance by being initialized step by step at runtime based on user defined parameters. This approach provides great flexibility for users to control the execution at runtime while remaining easy to manage and understand."

How parameterization differs from simple parameter-passing

At first glance, most orchestrators support "parameters" — Airflow templated tasks, Step Functions Choice states, Temporal workflow arguments. What distinguishes a genuinely parameterized workflow:

  • Parameters drive control flow at runtime, not just data values. The set of steps that execute depends on parameter values computed mid-workflow.
  • State-sharing between steps + between workflows via parameters (step outputs become downstream parameters; subworkflow outputs flow back to callers).
  • Code injection at the expression level — conditional branches, foreach ranges, subworkflow inputs are computed from expressions over parameters rather than being hard-coded.

The code-injection threat

Once you allow users to inject expressions into workflow definitions, you've imported a classic security / safety problem: tenant-supplied code runs in a shared process. The specific threat Netflix enumerates:

"Users might unintentionally write an infinite loop that creates an array and appends items to it, eventually crashing the server with out-of-memory (OOM) issues." (Source: sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator)

Two structural responses exist:

  1. Move the code out of the workflow definition — require users to embed parameter-expression logic in their own business- logic code (which then runs in their own containers, not the orchestrator process). Netflix explicitly rejects this: "this would impose additional work on users and tightly couple their business logic with the workflow. In certain cases, this approach blocks users to design some complex parameterized workflows."
  2. Sandbox the code inside the orchestrator — provide a safe expression language with runtime limits + platform sandbox. This is Maestro's choice — see systems/netflix-sel.

Canonical uses of parameterized workflows

From the Maestro post:

  • Backfill pipelines — one workflow definition parameterised by date range; runs N times across historical partitions (paired with Parallel-with- Concurrency-Limit run strategy).
  • ML model tuning — hyperparameter sweeps with foreach over parameter grids; each iteration treated as a separate workflow instance but scheduled/orchestrated uniformly.
  • Cross-table ETL — one workflow parameterised by source table + destination table, reused across hundreds of tables.
  • Conditional remediation — audit-check step's output drives whether a remediation subworkflow runs.

Seen in

Last updated · 319 distilled / 1,201 read