Skip to content

PATTERN Cited by 1 source

Rate-limited fleet rotation

Pattern

When enforcing instance lifecycle policies (rotation, taint-based replacement), apply rate limits scoped by service, region, and availability zone to prevent bulk rotation from causing sudden capacity loss. The lifecycle system replaces instances gradually rather than all at once.

Mechanism

  1. Lifecycle enforcement system identifies instances eligible for replacement (age-expired or tainted).
  2. Before scheduling replacement, check the current rate against configured limits per service × region × AZ.
  3. If under the limit, schedule replacement. If at the limit, defer to next evaluation cycle.
  4. Service owners configure their own rate limits via CLI/API.

Why it matters

Without rate limiting, a batch of simultaneously-aged instances or a fleet-wide taint signal could trigger mass replacement, causing capacity drops and service degradation. Rate limiting bounds the blast radius of lifecycle enforcement itself.

Slack's implementation

The Reaper in Shipyard has "built-in rate limiting [that] allows service owners to define how many instances can be replaced at a time, scoped by service, region, or availability zone, preventing sudden capacity impacts" (Source: sources/2026-07-14-slack-shipyard-next-generation-ec2-platform).

Seen in

Last updated · 585 distilled / 1,765 read