Skip to content

PATTERN Cited by 1 source

Global pause for lifecycle enforcement

Pattern

Provide a fleet-wide kill switch ("big red button") that immediately halts all automated instance replacement/rotation across the entire fleet. The mechanism should be simple, instant, and accessible during incidents — not gated behind deployment pipelines or config changes.

Mechanism

  1. An operator places a control object in a well-known location (e.g., an S3 object, a feature flag, a DynamoDB row).
  2. The lifecycle enforcement system checks for this sentinel before scheduling any replacement.
  3. If present, all replacement activity stops immediately — fleet-wide.
  4. When the incident resolves, the operator removes the control object (or it TTLs out), and normal lifecycle enforcement resumes.
  5. CLI tooling lets operators activate/release the pause without console access.

Why it matters

During incidents or periods of elevated risk, automated instance rotation can compound problems (replacing healthy instances during a partial outage, for example). A global pause gives operators confidence that the lifecycle system won't make a bad situation worse.

Slack's implementation

The Reaper in Shipyard provides "a global pause mechanism—the 'big red button.' By placing a control object in S3, teams can temporarily halt all Reaper activity across the fleet. This provides a safe and immediate way to stop instance cycling during incidents or periods of elevated risk" (Source: sources/2026-07-14-slack-shipyard-next-generation-ec2-platform).

Seen in

Last updated · 585 distilled / 1,765 read