Skip to content

SYSTEM Cited by 2 sources

Slack Shipyard

What it is

Shipyard is Slack's next-generation EC2 platform — a complete reimagining of how EC2-based services are run, replacing the legacy Chef-based continuously-mutated-instance model with an immutable infrastructure approach. Shipyard treats infrastructure as deployable artifacts (AMIs) rather than endlessly mutable instances, and gives teams service-level deployment primitives with the same safety and predictability expected from modern container-based platforms.

Built by Slack's Compute Platform Team, Shipyard is designed for workloads that cannot migrate to containers: infrastructure components, Kubernetes worker nodes, egress network stacks, and other EC2-bound services.

Core architecture

Layered image model

Shipyard uses a layered image approach inspired by container image layering:

┌─────────────────────────────────┐
│ Service AMI (team-owned)        │  ← team-specific software + config
├─────────────────────────────────┤
│ slack-zero (platform-owned)     │  ← OS + hardening + networking +
│                                 │     monitoring agents + common tooling
├─────────────────────────────────┤
│ Vendor OS image (Ubuntu/RHEL/AL)│
└─────────────────────────────────┘
  • slack-zero: The golden base image maintained collaboratively by Compute, Security, and Monitoring teams. Contains OS baseline + hardening, networking + service discovery, monitoring + security agents, and common tooling.
  • Service AMIs: Team-built images layered on slack-zero. Each team controls their runtime environment while inheriting standardised platform components.

slack-zero is built using AWS Image Builder (chosen over Packer) with built-in lifecycle management, SSM Parameter publishing, EventBridge-triggered downstream rebuilds, and pre-publish image validation.

Bake vs provision

Shipyard separates instance preparation into two distinct phases (bake-heavy, provision-light):

  • Bake phase: Install packages and apply configuration that is consistent across environments. Most work happens here — instances start from a fully prepared, known-good state.
  • Provision phase: Apply environment-specific settings (secrets, regional config, deployment metadata) at boot. Intentionally lightweight — typically just dropping configuration, retrieving secrets, and starting services.

This means instances become operational in seconds rather than minutes.

Deployment orchestration (Gondola)

Gondola is the global deployment orchestrator. When building an artifact, it produces a deployable package with two parts:

  1. The AMI to roll out
  2. The Chef artifact (versioned recipes at a Git commit)

These are treated as a single deployable unit. Gondola supports pluggable executors:

  • ASG-based: Updates launch template with new AMI + config; uses AWS Instance Refresh.
  • Kubernetes workers: Tells Karpenter which AMI to use; same bootstrapping as ASG nodes.
  • Custom executors: Extensible for services with special deployment needs.

Customer pipelines can have multiple stages (e.g., canary ASG + production ASGs per AZ) with sequential flow and per-stage metric monitoring + automatic rollback.

Fleet inventory (Peekaboo)

Peekaboo provides near-real-time visibility into the EC2 fleet by tapping EventBridge cloud events and instance metadata (rather than relying on Chef Server). Built on EventBridge + OpenSearch + Lambda with UI, API, and CLI interfaces. Tracks both Shipyard and non-Shipyard instances.

Instance lifecycle enforcement (The Reaper)

The Reaper enforces short instance lifespans through two inputs:

  1. Taint signals: External systems (security tooling, AWS EC2 events, manual SSH access) mark instances as no longer in desired state.
  2. Age-based rotation: Periodic checks via Peekaboo identify instances exceeding their maximum lifespan.

Safety controls: - Rate limiting per service/region/AZ prevents sudden capacity impacts. - Global pause ("big red button") via S3 control object halts all Reaper activity fleet-wide. - CLI for service owners to manage rate limits and inspect configuration.

Semi-immutable secrets exception

While Shipyard instances are mostly immutable, Consul Template allows dynamic secret refresh from Vault without cycling the fleet. This is explicitly scoped to secrets only — the core system and service layers remain fixed at bake time.

Multi-architecture and multi-OS

Designed from day one for: - CPU architectures: AMD64 + ARM (Graviton) - Operating systems: Ubuntu, RHEL, Amazon Linux

This lets teams optimise for cost, performance, or compatibility without separate platform implementations.

Testing infrastructure (Ship Quick)

Ship Quick is a developer workflow for testing cookbook changes on real infrastructure before merging:

  1. Developer runs CLI from cookbook repo (test cases defined in YAML).
  2. CLI packages cookbook to S3, sends workflow message to queue.
  3. Longshoremen worker instances pick up jobs, detach from ASG, run the Chef workflow, stream logs back to CLI, then terminate.

Two worker fleets: - Vanilla Ubuntu fleet: For baking/testing slack-zero itself (can't build on top of itself). - slack-zero fleet: For service team cookbooks (validates against the same foundation used in production).

Both fleets auto-scale with demand; slack-zero workers continuously updated to latest images.

Emergency pathways

For urgent situations, targeted configuration changes on running instances are allowed via AWS Systems Manager with a predefined SSM document to run selected Chef recipes. Affected instances are expected to be replaced afterward via regular deployment pipelines — returning to the immutable state.

Shared responsibility model

Layer Owner Scope
slack-zero base Compute + Security + Monitoring teams OS, hardening, agents, networking
Service AMI Service team Application software, service configuration
Deployment pipeline Service team (using Gondola) Rollout strategy, metric gates
Instance lifecycle Platform (Reaper) + Service team (rate config) Rotation cadence, taint policy

When the platform team updates slack-zero (security patch, agent update, networking improvement), service teams rebuild their images on the updated base to inherit the latest fixes.

Relationship to existing Slack compute stack

                    ┌─────────────────────────────┐
                    │ Slack Deploy Safety Program │
                    │ (10m auto / 20m manual SLOs)│
                    └──────────────┬──────────────┘
                                   │
                ┌──────────────────┼──────────────────┐
                │                  │                  │
                ▼                  ▼                  ▼
          ┌───────────┐      ┌──────────┐       ┌──────────┐
          │ Slack     │      │ Shipyard │       │ legacy   │
          │ Bedrock   │      │ (EC2     │       │ Chef EC2 │
          │ (K8s PaaS)│      │ platform)│       │ (maint-  │
          │ preferred │      │ prod     │       │ enance)  │
          └───────────┘      └──────────┘       └──────────┘
  • Bedrock remains the preferred target for containerisable workloads.
  • Shipyard is for EC2-only workloads (infrastructure components, K8s worker nodes, egress network stacks).
  • Legacy Chef-based EC2 is feature-complete + maintenance-mode; teams are actively migrating to Shipyard.

What's next (disclosed roadmap)

  • Long-lived instances: Data nodes, singleton services (GitHub Enterprise), third-party instances (Atlassian JIRA) that can't be cycled quickly. New deploy executors in Gondola under development.
  • Context-aware Reaper: Only meaningful changes (software updates, configuration drift) trigger replacement; read-only or low-risk actions won't create unnecessary churn.
  • Future posts promised on: Shipyard API, image pipelines, developer workflows, and Peekaboo inventory system.

Caveats

  • No fleet-size or cost-savings numbers disclosed.
  • No specific Reaper maximum-lifespan thresholds.
  • No Gondola latency/throughput benchmarks.
  • No specific metric types for deployment health gates.
  • Long-lived instance support explicitly acknowledged as unsolved.
  • Rate-limiting configuration details undisclosed.
  • Emergency Chef SSM document details not specified.

Seen in

Last updated · 608 distilled / 1,858 read