Skip to content

On building scalable control planes

Summary

A first-person retrospective by Zak van der Merwe (14 years at AWS: EC2 → DSQL) on the evolution of control plane architecture at AWS. The post traces the entire scaling journey of EC2's control plane — from a single MySQL primary, through hot standby failover, read replicas (introducing eventual consistency as a trade-off), AZ-sharding, and cell-based architecture — and then shows how Aurora DSQL was designed specifically to eliminate those same pain points for the next generation of AWS control plane builders. DSQL self-hosts its own control plane on DSQL, closing the loop. Werner Vogels's intro frames this as a companion to the Marc Brooker/Marc Bowes DSQL writing.

Key takeaways

  1. Control planes are reconciliation loops. The thermostat analogy: continuously compare desired state (what should exist) with actual state (what does exist), and correct the difference. This is the fundamental abstraction. (Source: body, "thermostat" section)

  2. Static stability is the cardinal rule. "No matter what happens to the control plane, VMs that are already running need to keep working." Every new feature is a chance to accidentally violate this guarantee — and violating it turns a launch-outage into a total outage. (Source: body, "static stability" section)

  3. The EC2 scaling ladder was: single primary → hot standby → read replicas → AZ sharding → cell sharding. Each rung took years of engineering, and each introduced new operational complexity (replication lag, routing logic, split-brain risk, cross-shard joins). (Source: body, "Living inside the control plane" section)

  4. Read replicas bought time but caused eventual consistency. EC2's describe APIs were eventually consistent because they hit read replicas. The author explicitly calls out this was an unfortunate cognitive load on customers — and DSQL was designed to remove it by providing strongly consistent reads that auto-scale. (Source: body; links Marc Brooker's consistency post)

  5. AZ sharding served dual purposes. Splitting the control plane per-AZ gave independent failure domains (blast-radius reduction) AND scaling headroom (smaller dataset per shard). This became the foundational building block customers use to build resilient architectures. (Source: body, "split each AWS region into multiple availability zones")

  6. Cell-based sharding was the internal second phase. After AZ sharding, EC2 internally sharded each zone into "cells" — taking years because every service that talks to the database must know which shard to route to. (Source: body)

  7. Sharding decisions have no universally right answer. "Do you shard by account or by resource? Different services choose differently depending on their access patterns." (Source: body)

  8. Human cost of immature control planes. Before automation, patching the EC2 fleet meant literally dividing all hosts among the team and assigning shifts. "You can patch a fleet of a few hundred hosts that way. You cannot patch a fleet of millions that way." (Source: body, "human cost" section)

  9. DSQL removes the scaling ladder for control plane builders. Micro-VM per connection eliminates standby failover; automatic read replicas with strong consistency eliminate the replica-lag trade-off; automatic partitioning eliminates the multi-year sharding project. (Source: body, "DSQL" comparison section)

  10. Self-hosting creates a virtuous cycle. DSQL's control plane runs on DSQL itself. Benefits: (a) bookkeeping DB scales automatically with adoption, (b) AZ outages don't kill the control plane because DSQL survives single-zone failures. Trade-off acknowledged: circular dependency (control plane can't depend on the thing it controls) must be carefully broken. (Source: body, "Self-hosting" section)

Operational numbers

  • EC2 control plane: single MySQL primary at its core (originally); "hundreds or thousands of internal API calls" per single RunInstances request.
  • Hot standby replication lag: "ideally only milliseconds behind."
  • Failover target: "seconds" (when humans executed correctly).
  • DSQL: Firecracker micro-VM per connection; strongly consistent auto-scaling reads; automatic partitioning.

Caveats

  • The post is intentionally retrospective and does not provide current EC2 architecture details (cell boundaries, shard counts, etc.).
  • DSQL gaps acknowledged: no foreign key constraints yet; single-node Postgres cached reads are faster than DSQL's latency (though DSQL is more predictable at scale).
  • Migrating EC2's control plane to DSQL "would take years even if we started today."
  • The post is partly DSQL advocacy — the comparison is made by a DSQL team member, not a neutral third party.

Source

Last updated · 609 distilled / 1,902 read