Skip to content

PATTERN Cited by 3 sources

Application-level sharding

Context

A team outgrew single-server vertical scaling and decided to horizontally partition a table across multiple database instances, but no sharding substrate was available or trusted — either because the substrate didn't yet exist (pre-2010 era), because the available NoSQL systems didn't meet durability / feature-set requirements, or because the operational expertise to adopt a substrate layer wasn't in place. The team chose to embed the shard-routing logic directly in the application code.

Pattern

All routing logic lives in the application layer. Given a row / query, the application:

  1. Extracts the shard key from the request.
  2. Applies a hash or range function shard = f(shard_key) to select a shard.
  3. Opens a connection to that specific shard's database and issues the SQL.
  4. For multi-shard queries: fans out N connections, aggregates client-side (scatter-gather implemented in app code).
  5. For cross-shard transactions: either skips them (make the product resilient to partial failures) or hand-rolls compensation logic.

The database is treated as a dumb storage layer that does not know it is part of a shard set. Each shard is a plain MySQL/Postgres instance with an identical schema and a disjoint key range.

Canonical precedents (2010–2015 era)

  • Pinterest (2012) — "Sharding Pinterest: How we scaled our MySQL fleet". Marty Weiner on the context: "We had several NoSQL technologies, all of which eventually broke catastrophically" — NoSQL immaturity drove the return to sharded MySQL. Pinterest mapped data by primary key and used it to route rows to shards. Trade-offs they accepted: no cross-shard joins, no foreign keys (Source: Burns/Barnett).
  • Etsy — "Scaling the Database: Shared Nothing Architecture". Added a two-way lookup primary-key → shard_id mapping and packed multiple shards onto a single host, automating some of the shard-management work. Same fundamental architecture: routing in the app, plain MySQL instances beneath.
  • Figma (2022–2024) — How Figma's databases team lived to tell the scale. A modern re-instantiation of the pattern: Figma built an in-house DBProxy query engine that routes queries across hash-sharded RDS Postgres. Not strictly application-level (routing is in a dedicated proxy), but the same substrate-not-available motivation: runway pressure + deep operational expertise + a relational model that existing NewSQL options couldn't express.

Failure modes

Burns/Barnett name three compounding failure modes (Source: sources/2026-04-21-planetscale-horizontal-sharding-for-mysql-made-easy):

  1. Routing logic must be updated for every new feature. Every new table, every new query pattern, every new access path forces a routing-code change. The application's sharding complexity grows monotonically with product surface area — it is a developer-capacity tax that cannot be amortised.
  2. Cross-shard features must be reimplemented in the app. Joins, aggregations, distributed transactions — all of these need bespoke application-side implementations. Each one is a small engineering project that re-derives a primitive the database would normally provide.
  3. Resharding becomes a daunting operational challenge. When the initial shard count runs out of capacity, the operator must split shards while serving live traffic — a coordinated dance of dual-writes, verification, cutover, and rollback that the application layer has to orchestrate itself. "Ongoing management of shards, including splitting shards after the initial resharding, presented significant challenges."

The compounding effect: (1) makes adding features slow, (2) makes the application code base drift toward database-internals territory, and (3) makes capacity planning adversarial to product work. Engineering capacity that should be going into product features is consumed by sharding plumbing.

Why the pattern lost favour

Burns/Barnett: "From experiences like these, there is an increasing need to separate sharding logic from the application as it introduces a plethora of complexity, making the application and your database harder to manage, which, in turn, drains developer capacity and pulls your team away from building and improving on great products for your customer base."

The rise of Vitess (2010, open-sourced by YouTube) and later PlanetScale as a managed Vitess offering provided the substrate-layer alternative: routing + failover + backup + pooling handled below the application, the app sees what looks like a single MySQL database. Both Pinterest and Etsy later migrated some workloads to Vitess — the self-correction from within the reference-class precedents.

When this pattern is still reasonable

Despite the operational tax, the pattern has two defensible use cases:

  • Substrate-layer is a poor fit for the data model. Figma's 2022 in-house build is the modern exemplar: their relational-model complexity + aggressive runway pressure made an out-of-the-box substrate (Vitess, Spanner, TiDB, CockroachDB) an unacceptable compromise. When the substrate's feature-set shrinkage is worse than the application-layer tax, in-house routing wins.
  • Single-axis workload at moderate scale. A workload that genuinely only ever needs one shard key and doesn't need cross-shard transactions can remain manageable at the application layer for a long time — the feature-growth-curve's compounding hurts less when there are fewer axes and fewer query shapes.

Figma's in-house build also produced architect-sharding-from-first-principles-per-engine — designing the sharding layer per-engine rather than adopting a substrate that abstracts the engine away.

Seen in

  • sources/2026-04-21-planetscale-horizontal-sharding-for-mysql-made-easy — Burns + Barnett canonicalise the pattern's pre-Vitess-era historical position, naming Pinterest + Etsy as the two reference-class precedents and the three compounding failure modes. The piece frames Vitess as the explicit alternative that "separates sharding logic from the application" — positioning application-level sharding as a legacy pattern.
  • sources/2026-04-21-figma-how-figmas-databases-team-lived-to-tell-the-scale — Figma's 2022-era in-house build revives the pattern (in a proxy-layer form, via DBProxy) under modern conditions: aggressive runway + relational-model mismatch with mature NewSQL options. The post explicitly notes the choice is worth re-evaluating once initial runway is bought.
  • sources/2026-04-21-planetscale-what-is-database-sharding-and-how-does-it-work — Justin Gage (guest post, 2023-04-06) canonicalises the "building this for the first time isn't hard; operational maintenance is the real problem" diagnosis of the pattern. Verbatim: "Building this for the first time is actually not that difficult; it's the operational maintenance that becomes the real problem over time. If you move shards from database to database, rebalance, add new machines, remove machines, change any database properties…you'll need to update that application logic to account for it." Sibling framing to Burns + Barnett's "routing logic must be updated for every new feature" failure mode, but with a different axis: Burns + Barnett focus on the feature-addition tax; Gage focuses on the shard-topology-change tax. Both are compounding failures that motivate the shift to a substrate layer. Also names ProxySQL as a "rough shard-routing service" — the closest-thing-to-a-helper for teams still committed to app-layer routing on top of MySQL, short of adopting a full substrate. Gage's framing completes the failure-mode picture: three axes of compounding cost (feature-addition tax / shard-topology-change tax / cross-shard-feature re-implementation tax), all of which the Vitess-era substrate solutions amortize.
Last updated · 550 distilled / 1,221 read