Skip to content

SLACK 2026-07-14

Read original ↗

Shipyard: How We Built Slack's Next-Generation EC2 Platform

Summary

Slack's Compute Platform Team describes Shipyard, their next-generation EC2 platform that replaces the legacy Chef-based continuously-mutated-instance model with an immutable infrastructure approach. Shipyard treats infrastructure as deployable artifacts (AMIs) rather than endlessly mutable instances, bringing modern deployment primitives — progressive rollouts, metric-driven safety, automated rollback — directly to EC2 workloads that cannot migrate to containers. This is the third post in Slack's Chef infrastructure series, following the AZ-bucketed environment split and signal-driven Chef runs from the 2025-10-23 post.

Key takeaways

  1. Immutable-first with a secrets exception: Shipyard instances are built as immutable AMIs — packages and configuration are baked at image-build time. The one exception is secrets: Consul Template dynamically refreshes credentials from Vault without cycling the fleet, making the architecture "semi-immutable" (Source: article, "A Caveat on Immutability" section).

  2. Layered image model (slack-zero): A shared "golden" base image called slack-zero — maintained by Compute, Security, and Monitoring teams — contains OS hardening, networking, service discovery, monitoring agents, and common tooling. Service teams build their own AMIs on top, analogous to Docker base images. This gives platform stability with team-level flexibility (Source: article, "Golden Base Images" section).

  3. AWS Image Builder over Packer: Slack chose AWS Image Builder for slack-zero construction because of built-in AMI lifecycle management (auto-cleanup), SSM Parameter publishing (downstream pipelines always read latest AMI), EventBridge-triggered downstream rebuilds, and built-in image validation testing (Source: article, "slack-zero" section).

  4. Bake-heavy, provision-light: Heavy operations (package installation, configuration) happen at image bake time. Provisioning at boot is intentionally lightweight — dropping environment-specific secrets, regional config, and starting services — so instances become operational in seconds rather than minutes (Source: article, "Baking and Provisioning" section).

  5. Gondola as deployment orchestrator: Gondola is the global deployment orchestrator. It produces deployable artifacts (AMI + versioned Chef artifact as a single unit), supports pluggable executors (ASG Instance Refresh, Karpenter for K8s workers, custom executors), progressive metric-monitored rollouts, and automatic rollback on health regression (Source: article, "Deployments and Fleet Updates" section).

  6. The Reaper enforces instance lifecycle: A dedicated system called The Reaper enforces short instance lifespans by consuming taint signals (from security tooling, AWS events, or manual SSH access) and performing age-based rotation checks. Instances are scheduled for graceful replacement, not abrupt termination. Rate limiting is per-service/region/AZ, and a "big red button" (S3 control object) provides a global pause mechanism (Source: article, "The Reaper" section).

  7. Peekaboo for real-time fleet inventory: Peekaboo replaces Chef Server as the fleet source-of-truth by consuming AWS EventBridge cloud events and instance metadata, providing near-real-time visibility via UI, API, and CLI. It tracks both Shipyard and non-Shipyard instances (Source: article, "Real-Time Inventory and Fleet Visibility" section).

  8. Ship Quick for pre-merge testing: A developer workflow called Ship Quick lets teams test cookbook changes on real infrastructure before merging. A CLI packages the cookbook to S3, a message hits a queue, and a fleet of "Longshoremen" worker instances (split into vanilla Ubuntu fleet for slack-zero testing and slack-zero fleet for service cookbooks) picks up the job, runs the bake/provision workflow, and streams logs back (Source: article, "How Do We Test Changes?" section).

  9. Multi-architecture and multi-OS from day one: Shipyard supports AMD64 + ARM (Graviton) and multiple OS families (Ubuntu, RHEL, Amazon Linux), enabling cost/performance optimisation without separate platform implementations (Source: article, "Multi-Architecture and Multi-OS Support" section).

  10. Staged progressive deployment: Customer pipelines in Gondola can have multiple stages (e.g., canary ASG + production ASGs per AZ), with sequential flow and per-stage metric monitoring + automatic rollback — preventing issues from spreading across the fleet (Source: article, "What Do Customer Pipelines Look Like?" section).

Architectural components

Component Role
slack-zero Golden base AMI (OS + hardening + monitoring + networking)
Service AMIs Team-built images layered on slack-zero
Gondola Global deployment orchestrator (artifact build + staged rollout + auto-rollback)
Peekaboo Real-time fleet inventory (EventBridge + OpenSearch + Lambda)
The Reaper Instance lifecycle enforcer (taint signals + age rotation + rate limiting)
Ship Quick / Longshoremen Pre-merge testing on real infrastructure
Consul Template Dynamic secret refresh from Vault (the "semi-immutable" exception)

Operational numbers

  • Tens of thousands of EC2 instances in scope (from earlier posts in series)
  • Instance startup: seconds (vs minutes under legacy model) due to bake-heavy approach
  • Multiple CPU architectures (AMD64 + Graviton)
  • Multiple OS families (Ubuntu, RHEL, Amazon Linux)
  • Worker fleets auto-scale with demand for Ship Quick testing

Caveats

  • No specific fleet size, instance-count, or cost-savings numbers disclosed in this post.
  • No Gondola latency/throughput benchmarks.
  • No specific SLOs for Peekaboo's "near real-time" inventory freshness.
  • The Reaper's maximum-lifespan threshold is not quantified.
  • Rate-limiting configuration details (how many replacements per unit time) not disclosed.
  • Emergency Chef workflow details (SSM document) are mentioned but not detailed.
  • No specific metric types used for deployment health gates (error rate? latency? custom?).
  • Long-lived instance migration (data nodes, JIRA, GitHub Enterprise) acknowledged as unsolved next challenge.
  • No mention of cost comparison with the legacy Chef model.

Source

Last updated · 585 distilled / 1,765 read