Skip to content

CONCEPT Cited by 1 source

Exactly-once signal trigger

An exactly-once signal trigger is a signal-driven workflow invocation where the orchestrator guarantees the subscribing workflow runs exactly once per matching signal — even if the signal is delivered multiple times (as is typical of at-least-once messaging substrates like Kafka / SNS).

"Signal triggering guarantees exactly-once execution for the workflow subscribing a signal or a set of joined signals." (Source: sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator)

Why this matters

Most event buses are at-least-once — retries, redrives, and broker failovers all legitimately reproduce messages. Naively wiring an orchestrator to subscribe to such a bus yields duplicate workflow runs, which in turn produce:

  • Duplicate writes to downstream tables.
  • Duplicate billing events.
  • Idempotency-key collisions.
  • Double-counting in aggregates.
  • Wasted compute + duplicate alerts.

If the workflow's run strategy is [[concepts/workflow-run-strategy# first-only|First-only]] or the steps are idempotent by construction, duplicates are harmless but wasteful. For non-idempotent workflows (billing, order placement, content-decision-making) they are a correctness bug.

The joined-signal case

Maestro's guarantee extends to joined signals: if a workflow subscribes to signals A and B, it runs exactly once per matching (A, B) pair. This is stronger than per-signal dedup — requires the orchestrator to track which joined-signal combinations have already fired a workflow.

Mechanism — implicit

The post doesn't describe the dedup mechanism in detail. Plausible implementations:

  1. Signal-ID deduplication — each signal carries a unique ID; the orchestrator persists a (workflow_id, signal_id) seen-set and rejects repeat matches.
  2. Consumed-signal watermark — the orchestrator advances a watermark as it consumes signals; signals below the watermark that match a subscriber don't fire.
  3. Mapped-field hash — the mapped parameter subset used for matching forms an idempotency key; workflows are keyed by (workflow_id, mapped-fields-hash).

What the post does disclose: "signal lineage" — a queryable history of publisher/consumer match events — implies persistent per-signal metadata, consistent with (1) or (3).

Comparison: delivery-guarantee spectrum

Guarantee Typical substrate Workflow-trigger consequence
At-most-once Fire-and-forget UDP May miss legitimate triggers; dangerous
At-least-once Kafka default, SNS, SQS Duplicate workflow runs unless dedup
Exactly-once (Maestro signal trigger) Orchestrator-level dedup over at-least-once substrate Safe for non-idempotent downstream work

Kafka's "exactly-once semantics" (EOS) applies to a single producer → consumer pipe under specific transactional guarantees; Maestro's exactly-once is orchestrator-level and spans any event source.

Why this is canonical

Exactly-once is one of the well-known hard problems in distributed systems — usually considered infeasible across network boundaries without additional assumptions. The Maestro instance shows the feasible engineering shape: delegate delivery to an at-least-once substrate, build exactly-once at the consumer side via persistent dedup state keyed on idempotency-relevant fields. This is the same shape as:

  • AWS Lambda's event-source idempotency controls.
  • Stripe's idempotency keys on API requests.
  • Kafka's idempotent producers (for the producer → broker pipe).
  • HTTP Idempotency-Key header.

Seen in

Last updated · 319 distilled / 1,201 read