Skip to content

PATTERN Cited by 1 source

Fleet replacement over in-place mutation

Pattern

When deploying changes to a fleet of instances, replace instances with new ones built from the updated image rather than mutating existing instances in place (via SSH, configuration management runs, or hot-patching). The deployment unit is the image, not the patch.

Mechanism

  1. Build a new AMI incorporating the desired changes.
  2. Update the fleet's launch template / node class to reference the new AMI.
  3. Perform a controlled replacement of instances (Instance Refresh for ASGs, Karpenter disruption for K8s nodes, custom executors otherwise).
  4. Old instances are terminated after replacement completes.
  5. Emergency in-place changes are allowed but instances are marked for replacement afterward.

Benefits

  • Zero drift: Every running instance matches its declared image.
  • Simple rollback: Deploy the previous image version.
  • Auditable: The fleet state is a function of which image version is deployed — not an accumulation of patches.
  • Eliminates "works on my instance" failures: All instances are identical by construction.

Trade-offs

  • Longer change-to-deploy cycle for trivial changes (must rebuild an image even for small tweaks).
  • Long-lived stateful instances (databases, message brokers) can't use this pattern directly.
  • Image storage costs grow with version frequency.

Slack's implementation

Shipyard uses this pattern as its core deployment model. "Instead of patching existing instances, fleets are updated through controlled replacements, keeping everything consistent and predictable." Even emergency fixes applied via AWS SSM to running instances are "meant for emergencies only. Affected instances are expected to be replaced afterward via regular deployment pipelines" (Source: sources/2026-07-14-slack-shipyard-next-generation-ec2-platform).

Seen in

Last updated · 585 distilled / 1,765 read