CONCEPT Cited by 1 source
DAG vs cyclic workflow¶
Most mainstream workflow orchestrators constrain their workflows to Directed Acyclic Graphs (DAGs) — a dependency graph with no loops. The DAG constraint simplifies scheduling, termination analysis, and visualisation, but excludes patterns that need legitimate iteration at the workflow level — foreach loops over data partitions, retry-until-success loops, interactive refinement loops.
Netflix Maestro explicitly takes the opposite stance:
"Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc." (Source: sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator)
Why most orchestrators are DAG-only¶
- Termination is obvious — a DAG cannot have infinite execution; every node has finite depth.
- Dependency resolution is linear — topological-sort gives deterministic execution order.
- Visualisation is natural — tree / Sugiyama-layout renderers all assume acyclicity.
- Failure-propagation analysis is bounded — a node's blast radius is its descendants.
Airflow's DAG is in the name; Step Functions' state machines
are strict state transitions (conditionals, not loops); Argo
Workflows treats cycles as errors.
Why cyclic is sometimes needed¶
- Foreach over partitions / model hyperparameters / dates — a fixed-count loop.
- Retry-until-success / audit-then-remediate — a conditional loop bounded by a stopping criterion.
- Iterative refinement — ML workflows that loop until a metric threshold is met.
- Workflow-as-function composition — calling the same subworkflow multiple times with different inputs can be naturally expressed cyclically.
How orchestrators get around the constraint¶
Three typical workarounds + their limitations:
| Workaround | Limitation |
|---|---|
| External driver — user script loops + re-triggers the workflow | Loses orchestrator visibility; no unified lineage |
| Subworkflow with dynamic definition — template a loop by generating a DAG at runtime | Definition is runtime-computed, hard to statically validate |
Operator-level iteration (e.g. Airflow's dynamic task mapping since 2.3, Argo's withSequence) |
Usually hides the iteration from the DAG visualisation; still DAG-under-the-hood |
Maestro's stance — native cyclic workflow support plus three engine-level composite primitives (foreach, subworkflow, conditional branch) — avoids the workaround tax by making iteration a first-class workflow-level construct.
Termination semantics¶
The open question for any cyclic-workflow orchestrator: how do you prevent infinite execution? Maestro's post doesn't spell out termination rules in detail, but the article's framing implies:
- Foreach steps are bounded by the iterable's size at evaluation time.
- Conditional branches with back-edges require user-supplied stopping conditions — with SEL's runtime loop-iteration limits bounding any runaway behaviour at the expression level, and presumably engine-level instance-count caps.
- Subworkflow recursion can in principle loop infinitely but is bounded by engine-level depth + throughput limits.
The canonical instance in the post is a conditional-retry / audit-remediate pattern — bounded in practice by a retry-count ceiling.
Industry positioning¶
| Orchestrator | Cyclic workflows | How |
|---|---|---|
| Netflix Maestro | ✅ native | engine-level composite primitives |
| Temporal | ✅ | workflow code is regular Go / Java / Python — can contain loops |
| Cadence | ✅ | same as Temporal |
| Airflow | ❌ (DAG-level), ✅ dynamic task mapping since 2.3 | per-task iteration; no DAG cycles |
| Step Functions | ❌ | state machine; Map state for parallel iteration only |
| Argo | ❌ | withSequence / withItems parallel iteration only |
Temporal + Cadence achieve cyclic semantics by expressing workflows as code (loops are language-native) rather than as a declarative graph. Maestro's stance is distinct: declarative JSON workflow definition + engine-level cyclic primitives.
Seen in¶
- sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator — the explicit DAG-vs-cyclic framing