Skip to content

PATTERN Cited by 1 source

Ignore self-generated shutdown signal

Problem

An orchestrator generates shutdown signals (unavailability events) that propagate to all managed services — including the orchestrator's own control-plane services. When the orchestrator shuts itself down, orphaned services that never received a shutdown signal remain running but unreapable (because the reaper is dead). The signal generator is impacted by its own signal — a "boomerang" effect.

Solution

Allow control-plane services to simply ignore shutdown signals associated with the triggering event class (e.g., power-related unavailability events). The orchestrator stays alive to continue coordinating the shutdown and eventual recovery of all other services.

Why This Beats the "Complex" Alternatives

Meta explicitly chose this simple approach over more intricate solutions such as maintaining an exclusion list of services exempt from UE dispatch:

  • Simpler — no exclusion-list maintenance burden; no risk of the list becoming stale
  • More sustainable — no coupling between signal dispatch logic and the evolving set of control-plane services
  • Self-contained — each control-plane service owns its own ignore-rule rather than depending on external configuration

Canonical Instance

Meta's Twine orchestrator: unavailability events (UEs) used to orchestrate shutdown and recovery ended up shutting down the Scheduler, Allocator, Broker, and Zelos services that generate and dispatch those UEs. Orphaned services could never be reaped. The fix: control-plane services ignore power-related UE shutdown signals — a one-line policy per service, not a system-wide coordination change.

Generalisation

Any system where a control-plane component both generates and is subject to a broadcast signal faces this risk. Common in:

  • Container orchestrators generating drain signals
  • Service meshes propagating circuit-breaker state
  • Config systems pushing config that includes their own config

The resolution is always the same shape: exempt the signal source from its own signal, either by ignoring it at the receiver or by routing around it at the dispatcher.

Seen in

Last updated · 542 distilled / 1,571 read