Skip to content

CONCEPT Cited by 1 source

Workflow run strategy

A workflow run strategy is the policy that determines whether and when a newly-triggered workflow instance actually runs, given the state of prior/concurrent instances of the same workflow. Different real-world pipelines have fundamentally different concurrency + ordering + failure-propagation requirements; the run strategy names these shapes explicitly at the orchestrator level.

Canonical wiki taxonomy is the five named strategies in Netflix Maestro (Source: sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator).

The five Maestro strategies

1. Sequential (default)

  • FIFO queue; runs workflows in trigger order.
  • Does not couple state across runs — execution does not depend on prior run statuses.
  • Once a run reaches any terminal state (succeeded or failed), Maestro starts the next queued instance.

Canonical use: ordinary pipelines where time-ordering matters but failure in one run shouldn't block the next.

2. Strict Sequential

  • FIFO, but blocks on failure — when an instance fails with a blocking error, all subsequent instances queue indefinitely until an operator manually marks failed runs unblocked or restarts them.

"This run strategy is useful for time insensitive but business critical workflows. This gives the workflow owners the option to review the failures at a later time and unblock the executions after verifying the correctness." (Source: sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator)

Canonical use: compliance / finance / audit pipelines where skipping past a failure would silently corrupt downstream state.

3. First-only

  • If a run is in progress, drop newly queued instances — "effectively turning off queuing with this run strategy."
  • Maestro only starts a new instance when there's no running one.

"This approach helps to avoid idempotency issues by not queuing new workflow instances."

Canonical use: non-idempotent workflows — pipelines whose repeat execution would produce wrong results or duplicate writes.

4. Last-only

  • Kill running, run newest — when a new instance arrives, Maestro stops the running one and starts the new one. Only the most recently triggered instance runs.

"This is useful if a workflow is designed to always process the latest data, such as processing the latest snapshot of an entire table each time."

Canonical use: snapshot-processing pipelines — any outcome of an older run is strictly worse than starting fresh on current data.

5. Parallel with Concurrency Limit

  • Multiple triggered instances run concurrently, capped at a predefined limit.
  • Distributes execution to fan out data processing under a deadline.

"A common use case for this strategy is for backfilling the old data."

Canonical use: backfills. Parallelism under a concurrency cap prevents backfills from starving steady-state traffic.

How this concept maps onto peer orchestrators

Most orchestrators expose the same shapes but spread across separate config surfaces rather than as named first-class strategies:

  • Apache Airflow — max_active_runs + depends_on_past approximates Strict-Sequential; max_active_runs=1 + catchup=False approximates First-only; there's no direct Last-only built-in (users implement via sensors).
  • AWS Step Functions — state machine concurrency is implicit in triggering logic (EventBridge / manual invoke); no named strategy taxonomy.
  • Temporal — WorkflowIDReusePolicy + cron overlap policy cover similar ground (AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate + WorkflowIDConflictPolicy).

Maestro's explicit five-strategy naming is a wiki-canonical taxonomy useful for comparing orchestrators.

Why this matters

A workflow's run strategy is part of its correctness contract — the same step logic + same trigger cadence can produce wildly different outcomes depending on the strategy:

  • Sequential on a backfill would serialise everything, missing the deadline.
  • Parallel on a non-idempotent pipeline would corrupt data.
  • First-only on a "process latest snapshot" job wastes CPU on stale data.
  • Strict-Sequential without the manual-unblock discipline silently stops the pipeline.

Naming the strategies explicitly forces the choice at design time.

Seen in

Last updated · 319 distilled / 1,201 read