Skip to content

SYSTEM Cited by 1 source

Slack Reaper

What it is

The Reaper is Shipyard's instance lifecycle enforcement system. It ensures that EC2 instances are regularly replaced rather than mutated in place, reinforcing Slack's immutable infrastructure commitment.

Mechanism

The Reaper evaluates two primary inputs:

1. Taint signals

External systems mark instances as no longer in their desired state:

  • Security tooling detecting drift or vulnerabilities.
  • AWS EC2 events indicating hardware or scheduling changes.
  • Manual SSH access to production instances — deliberately generates a taint signal supporting immutable infrastructure goals.

2. Age-based rotation

Periodic checks (integrated with Peekaboo) identify instances exceeding their maximum allowed lifespan. Once a node reaches its limit, it follows the same graceful replacement workflow as tainted instances.

Safety controls

Rate limiting

Service owners define how many instances can be replaced at a time, scoped by: - Service - Region - Availability zone

This prevents sudden capacity impacts from bulk rotation.

Global pause ("big red button")

A control object placed in S3 temporarily halts all Reaper activity across the fleet. Provides immediate, safe cessation of instance cycling during incidents or elevated-risk periods.

CLI

Service owners can: - Manage rate limits - Inspect configuration - Activate or release the global pause

Short-lived SSH access

Controlled access via short-lived SSH certificate workflows for break-glass scenarios, while still maintaining overall lifecycle safety commitments (accessed instances are still tainted for eventual replacement).

Design philosophy

The Reaper is designed to be both predictable and controllable: enforcing instance immutability by default while giving teams visibility and safeguards to operate confidently. Key principle: replace instances rather than modify them in place — this reduces configuration drift, limits vulnerability exposure windows, and makes the fleet auditable.

Future direction

Plans to make the system context-aware: only meaningful changes (software updates, configuration drift) would trigger replacement, while read-only or low-risk actions would not create unnecessary churn.

Caveats

  • Maximum lifespan thresholds not quantified.
  • Rate-limiting specifics (instances/hour, etc.) not disclosed.
  • Graceful replacement workflow mechanics not detailed (drain → terminate? ASG-managed?).
  • Taint signal schema and propagation latency not described.

Seen in

Last updated · 585 distilled / 1,765 read