Skip to content

PATTERN Cited by 1 source

Tainted instance replacement

Pattern

When an external system detects that a running instance has deviated from its desired state (via security signals, manual access, AWS events, or drift detection), it emits a taint signal that schedules the instance for graceful replacement. The instance is not immediately terminated — it is replaced according to the service's replacement policy and rate limits.

Mechanism

  1. An event occurs that marks an instance as "tainted":
  2. Security tooling detects a vulnerability or configuration deviation.
  3. A human SSHs into a production instance (manual access signal).
  4. An AWS EC2 event indicates hardware degradation or scheduling changes.
  5. The taint signal is consumed by a lifecycle enforcement system.
  6. The instance is scheduled for replacement according to service policies.
  7. Replacement respects rate limits (per-service/region/AZ) to prevent capacity impact.
  8. The instance continues serving until its replacement is healthy.
  9. On replacement completion, the tainted instance is terminated.

Benefits

  • Reinforces immutability: Any deviation from desired state triggers eventual replacement, not remediation.
  • Reduces vulnerability window: Tainted instances have bounded remaining lifetime.
  • Discourages ad-hoc changes: Operators know manual changes will be reverted via replacement.
  • Graceful: Services remain healthy throughout the replacement process.

Slack's implementation

The Reaper in Shipyard implements this pattern. Notably, "while manual remote access remains available for emergency scenarios, manually accessing a production-class node will generate a signal that marks the instance for eventual replacement, supporting our immutable infrastructure goals" (Source: sources/2026-07-14-slack-shipyard-next-generation-ec2-platform).

Seen in

Last updated · 585 distilled / 1,765 read