Skip to content

CONCEPT Cited by 1 source

Signal-based step dependency

A signal-based step dependency is a data-condition-driven way to unblock a workflow step: instead of waiting on upstream step completion (the usual DAG dependency shape), the step waits until a signal — a message carrying parameter values — is published that matches its subscribed criteria.

Canonical wiki instance is Netflix Maestro's signal mechanism (Source: sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator).

Anatomy of a signal

"Signals, which are pieces of messages carrying information such as parameter values and can be published through step outputs or external systems like SNS or Kafka messages."

Three surface properties:

  1. Producers — either a workflow step that writes to its output.signal definition, or an external system (Kafka / SNS / webhook).
  2. Consumers — steps whose signal definition declares a set of mapped parameters + match conditions.
  3. Matching — Maestro performs "'signal matching' on a subset of fields" using operators <, >, =, and variants (SignalOperator.java).

Dual semantics — pub-sub AND trigger

The same signal primitive serves two named patterns:

Publish-subscribe

One step publishes, many downstream steps are unblocked. Classic use-case: an ETL step writes a table + publishes a signal containing the partition key; all downstream workflows depending on that table/partition unblock in parallel.

Trigger

A signal arrival starts a new workflow instance (rather than unblocking an existing step). External events (SNS / Kafka message) can act as triggers when they match a workflow's trigger-signal definition.

See patterns/signal-publish-subscribe-step-trigger for the composed pattern.

Why this differs from classical DAG dependency

  • Decouples producer + consumer workflows — producer doesn't know who subscribes; consumer doesn't know which specific workflow produced.
  • Survives workflow boundaries — signals are a cross-team protocol. The Metaflow-at-Netflix post:

"It allows a team to integrate their Metaflow flows to surrounding systems upstream (e.g. ETL workflows), as well as downstream (e.g. flows managed by other teams), using a protocol shared by the whole organization." (Source: sources/2024-07-22-netflix-supporting-diverse-ml-systems-at-netflix)

  • Carries parameter values — the signal doesn't just announce "event happened" but carries data (partition key, timestamp, size, …) that downstream consumers use to parameterise their own execution.
  • External origin — signals can come from outside the orchestrator entirely (SNS, Kafka), closing the loop with non- orchestrator event sources.

Signal lineage

"Maestro supports 'signal lineage,' which allows users to navigate all historical instances of signals and the workflow steps that match (i.e. publishing or consuming) those signals."

A queryable history of producer / consumer / match events — useful for debugging ("why didn't my downstream trigger?"), auditing ("what flows consumed this table update?"), and capacity planning.

Efficiency + exactly-once

"This approach is efficient, as it conserves resources by only executing the workflow or step when the specified conditions in the signals are met."

"Signal triggering guarantees exactly-once execution for the workflow subscribing a signal or a set of joined signals."

See concepts/exactly-once-signal-trigger — the exactly-once guarantee closes the duplication threat from external event sources + retry mechanisms.

Composed architectural shape

Signals become the organisational protocol for inter-team workflow integration:

        ┌──────────────┐    signal(table=X, partition=Y)
Team A  │  ETL flow    │ ──────────────────────────┐
        │  (produces)  │                           │
        └──────────────┘                           │
                                                   ▼
                                          ┌─────────────────┐
                                          │  Signal service │
                                          │  (match + route)│
                                          └────────┬────────┘
                                   subscribes      │
                                   to table=X      │
                       ┌──────────────┬────────────┤
                       ▼              ▼            ▼
              ┌──────────────┐ ┌──────────┐ ┌──────────────┐
Teams         │ ML training  │ │ Audit    │ │ Reporting    │
B/C/D/E/…     │ flow         │ │ flow     │ │ flow         │
              └──────────────┘ └──────────┘ └──────────────┘

No team needs to know the identity of other teams consuming their signals — the signal protocol is the integration contract.

Seen in

Last updated · 319 distilled / 1,201 read