PLANETSCALE 2023-04-06

PlanetScale — What is database sharding and how does it work?¶

Summary¶

Justin Gage (writer of the Technically newsletter, guest-posting on PlanetScale's blog, 2023-04-06) publishes a broad pedagogical primer on database sharding: what it is, how it differs from replication, the three-to-four canonical partition strategies, how to pick between them as a function of your data model, what the operational-maintenance work looks like, how the routing layer is built, how to plan the migration, and which open-source + managed tools exist in the ecosystem. The post is written for an "overwhelmed developer" facing a scaling-up-vs-out decision, not for a database engineer — the tone is decision-framework over internals. Load-bearing framings verbatim:

"Sharding is a strategy for scaling out your database by storing partitions of your data across multiple servers instead of putting everything on a single giant one. Each partition of data is called a shard."

"Sharding and replication are complementary but distinct strategies. Sharding splits data across multiple servers so each server holds only a subset of the total data, reducing write load and storage pressure. Replication copies the same data to multiple servers, improving read performance and redundancy. Many production systems use both: sharded databases are often replicated within each shard for high availability."

"How you decide to split up your data into shards – also referred to as your partition strategy – should be a direct function of how your business runs, and where your query load is concentrated."

Gage canonicalises the sharding-vs-replication orthogonality (a common conflation), names partition strategy as the umbrella noun for the hash / range / directory-based choice, canonicalises vertical sharding as schema / table-level split (a different axis from row-level horizontal sharding), pulls Notion's double-write → backfill → verification → switch-over framework into a single-paragraph summary that's become a widely-cited canonical migration sequence, names the divisible-by-many-smaller-numbers shard-count heuristic, names Citus (Postgres) and Vitess (MySQL) as the canonical open-source sharding substrates, names ProxySQL as a "rough shard-routing service" distinct from a full substrate, and closes with a provocative question — "why are you not using a database that does sharding for you?" — pointing at Cloud Spanner / CockroachDB / shard-native managed offerings as the architectural alternative.

Key takeaways¶

Sharding and replication are orthogonal, not alternatives. Gage's framing — "Sharding splits data across multiple servers so each server holds only a subset of the total data, reducing write load and storage pressure. Replication copies the same data to multiple servers, improving read performance and redundancy." — corrects the common conflation. Production systems usually do both (each shard is replicated for HA). Canonical wiki framing of the two-axis design space.
Vertical sharding is schema / table-level, not row-level. "Vertical sharding, for example, is when you split things at the schema or table level." — Gage's canonicalisation of vertical sharding distinct from horizontal (row-level) sharding. Most wiki sources treat vertical sharding as the move tables to another cluster lever; Gage's framing includes schema-column splits as the same axis. The PlanetScale / Vitess vocabulary (Berquist, Dicken) uses the table-move framing.
Partition strategy is a "direct function of how your business runs." Gage's canonical framing: the hash / range / directory-based choice isn't arbitrary — it's derived from the dominant query-load shape and the data model. B2B SaaS with per-org queries → split by org_id. Consumer with uniform activity → hash on order_id. Notion → team-ID. Amazon example walked verbatim: customer orders have no meaningful clustering → hash-based sharding on order ID. Canonical partition-strategy concept.
Three algorithms named: hash / range / directory-based. Gage names "directory based sharding" for what the Dicken (sources/2026-04-21-planetscale-database-sharding) and Guevara (sources/2026-04-21-planetscale-sharding-strategies-directory-based-range-based-and-hash-based) posts also call lookup sharding — synonyms on this wiki. Gage's definition verbatim: "Pick a column, allocate shards manually, and maintain a lookup table so you know where each row is stored." Notion's team-ID split is an instance: one column, operator-authored placement, lookup-maintained.
Sharding maintenance is ongoing work — hotspots redistribute. "Sharding maintenance is an oft underappreciated piece of scaling out your relational database. Depending on what your partition strategy is, you'll likely end up with hotspots, where a particular server in your cluster is either storing too much data or handling too much throughput." Amazon example: "In our Amazon example, it could be because a large business started ordering a metric-ton of stuff, and all of their data is on one server. Managing those hotspots, redistributing data and load, and reorganizing your partition strategy to prevent future issues is part of what you're signing up for when you shard." Canonical cross-ref to concepts/hot-shard-write-frontier; extends the hotspot discussion beyond range-sharding into hash-sharding-on-low-cardinality or concentrated-tenant cases.
Joins across shards are expensive → table colocation is core to the design. "Joins across databases in a distributed system are difficult and costly, so ideally all of the data you need to answer a particular query exists on the same physical machine. For Amazon, that means the orders table and the products table containing the products in the orders table need to be physically colocated." Canonical framing that colocation is a requirement of the design, not an optimisation — because cross-shard joins undermine the horizontal-sharding payoff. "This also requires incremental maintenance: if a customer makes a new order, the product data for that order needs to be included in the new shard so it can be read quickly later on."
Shard count should be divisible by many smaller numbers for flexible rebalance. "This is why companies like to choose a number of shards that's divisible by a lot of smaller numbers; it allows you to scale out the number of servers incrementally while maintaining that smooth, even distribution." Canonical highly-composite-shard-count concept — a heuristic that lets the operator add hosts at many intermediate points without rebalancing the logical partition layer.
Routing layer in application code is the most common starting point, and its operational maintenance is the hidden cost. "For those building sharding from scratch, the most common answer is in the application layer. You need to build logic into your application code that decides which database (and schema) to connect to for a particular query." Gage walks a Python-like pseudocode snippet (if data.sharding_key in database_1.sharding_keys: …connect to database_1), then flags the real cost: "Building this for the first time is actually not that difficult; it's the operational maintenance that becomes the real problem over time. If you move shards from database to database, rebalance, add new machines, remove machines, change any database properties…you'll need to update that application logic to account for it." Canonical reinforcement of the application-level sharding post-Vitess-era framing. ProxySQL named as "a rough 'shard routing' service" — distinct from a full substrate.
Notion's four-step migration framework is the canonical zero-downtime sharding cutover. Gage's verbatim paraphrase of Notion's Sharding Postgres at Notion post: "1. Double-write: Incoming writes get applied to both the old and new databases. 2. Backfill: Once double-writing has begun, migrate the old data to the new database. 3. Verification: Ensure the integrity of data in the new database. 4. Switch-over: Actually switch to the new database. This can be done incrementally, e.g. double-reads, then migrate all reads." Canonical four-step pattern; sibling to the generic dual-write-migration pattern but with explicit verification-before-switch and incremental-switch-over steps. Also the honest caveat: "Each of these steps still introduces the possibility of downtime; it's just a risk you're going to have to take for changes at this scale."
Open-source sharding ecosystem: Vitess (MySQL) + Citus (Postgres). Vitess framed as the origin-at-YouTube + served-all-YouTube-DB-traffic + production-at-Slack/GitHub/NewRelic/Pinterest/Square reference; Citus as the Postgres-extension equivalent, "minus some more flashy features", in production at Algolia / Heap / Cisco. First wiki disclosure of Citus as a named system.
The provocative closing question: why aren't you using a shard-native database? "I suppose the more fundamental question is: why are you not using a database that does sharding for you? … Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB." Names Cloud Spanner + CockroachDB + PlanetScale as shard-native alternatives that shift the decision from "how do I shard?" to "why is sharding not the database's job?" Gage coins "shard-native" for the category and positions it as the direction the industry is moving.

Operational / architectural numbers¶

The post has no empirical numbers — it is pedagogy rather than a production report. The quantitative claims are all qualitative framings ("many" production Vitess users, "several" NoSQL failures cited, "3rd time this quarter" RDS upsize opener). For empirical numbers on related sharding posts on the wiki:

Linear QPS scaling: see Berquist 2022 (16 → 32 shards = 2× QPS on sysbench-tpcc).
Shard-parallel backup: see Dicken 2025-01-09 (4 TB @ 100 MB/s → 11 h unsharded vs 2.7 h on 4 shards).
IOPS-cost reduction: see Dicken 2024-08-19 (8× workload: $20k-$24k/mo unsharded io1 vs $14k/mo on 8-shard cheap-volume).

Caveats and scope¶

Tier-3 pedagogy; Justin Gage is a guest author, not a PlanetScale engineer. Gage runs the Technically newsletter for non-engineer audiences learning the tech industry; this post is his sharding primer reposted to PlanetScale's blog. That makes it slightly less authoritative than the Dicken / Burns / Guevara PlanetScale-internal voices on internals, but the framing is broad-audience-friendly and the canonical terminology (partition strategy, shard-native, the four-step migration framework as a packaged summary) makes the post referenced heavily in external sharding discussions.
The distinction between "hash", "key-based", and "consistent hashing" is flattened. Gage uses "Hash based sharding (also known as key based)" as a single label; this wiki separates hash sharding (the strategy) from consistent hashing (the specific family of hash algorithms that makes resharding 1/N cost). Gage doesn't make the distinction.
Vertical sharding framing is broader than the Vitess vocabulary. Gage includes column-level splits under vertical sharding; the PlanetScale / Vitess wiki vocabulary uses vertical sharding / vertical partitioning for table-level moves across clusters (not column-level splits within a table). Readers should note the vocabulary difference when cross-referencing.
No substrate-vs-substrate feature comparison. The Citus-vs-Vitess framing is one paragraph ("minus some more flashy features") without specifics on what those features are. Dicken's posts on Vitess's online-resharding / VReplication / MoveTables primitives fill that gap.
Notion-4-step framework is one-paragraph synthesis, not a deep dive. The canonical source is Notion's 2021 Sharding Postgres at Notion post, which walks each phase with Notion-specific numbers and caveats. Gage's summary is the packaged-for-reuse form.
No discussion of cross-shard queries as a scale cap. Gage mentions joins-are-expensive but doesn't surface the scale-cap framing that cross-shard queries turn the cluster into an unsharded-sized workload — Dicken (sources/2026-04-21-planetscale-database-sharding) and Figma (sources/2026-04-21-figma-how-figmas-databases-team-lived-to-tell-the-scale) both make this explicit.

Contributions to the wiki¶

New canonical pages (3): concepts/partition-strategy (umbrella noun for the hash/range/directory choice); concepts/highly-composite-shard-count (the divisible-by-many-smaller-numbers heuristic); patterns/notion-double-write-backfill-verify-switchover (Notion's four-step migration framework as a first-class pattern).

New system pages (2): systems/citus (Postgres sharding extension; Algolia / Heap / Cisco); systems/proxysql (MySQL routing proxy framed as "rough shard-routing service").

Extensions to existing pages: Seen-in entries on concepts/horizontal-sharding, concepts/vertical-sharding, concepts/hash-sharding, concepts/lookup-sharding (directory-based-naming confirmation), patterns/application-level-sharding (Gage's "operational maintenance is the real problem" framing), and patterns/dual-write-migration (Notion-4-step as named specialisation).