SYSTEM Cited by 1 source

EC2 control plane

What it is

The EC2 control plane is the service layer that takes what exists physically in AWS data centers and presents it to customers via the EC2 API. It is responsible for launching and terminating VMs, health-checking the fleet, reconciling desired state with actual state, and maintaining the record of which resources exist for each customer.

At its core: "there is a MySQL database, and what's in that database is supposed to match reality." (Source: sources/2026-08-04-allthingsdistributed-on-building-scalable-control-planes)

Architecture evolution

The EC2 control plane went through a canonical database scaling ladder over ~14 years:

1. Single MySQL primary. All reads and writes through one server. A single RunInstances call triggers "hundreds or thousands of internal API calls between micro and macro-services."
2. Hot standby. Continuous replication (milliseconds of lag); manual failover limiting outage to seconds — but requiring human on-call judgment at 3am.
3. Read replicas. Describe APIs routed to replicas → massive reduction of primary load. Trade-off: eventual consistency exposed to customers (EC2 describe APIs are eventually consistent because of this design decision).
4. AZ sharding. Each availability zone got its own independent control plane and MySQL database. Dual benefit: independent failure domains + smaller dataset per shard.
5. Cell sharding. Internal subdivision of each zone into cells. Required changes across every service that talks to the database — took years.

Design principles

• Static stability: "No matter what happens to the control plane, VMs that are already running need to keep working." (Source: sources/2026-08-04-allthingsdistributed-on-building-scalable-control-planes)
• Desired-state reconciliation: The thermostat model — continuously watching, comparing, correcting.
• Human removal from loop: Early fleet patching was manual (divide hosts among team, assign shifts). Mature control plane automates this entirely.

Key decisions and trade-offs

• Sharding dimension: "Do you shard by account or by resource? Different services choose differently depending on their access patterns, and there's no universally right answer."
• Eventual consistency cost: Read replicas made EC2 APIs eventually consistent — acknowledged as "unfortunate cognitive load on customers."
• Years-long projects: Both AZ sharding and cell sharding took years because routing logic must be threaded through every database-talking service.

Seen in

• sources/2026-08-04-allthingsdistributed-on-building-scalable-control-planes