Skip to content

PATTERN Cited by 1 source

Architect sharding from first principles per engine

Problem

A managed-database vendor has built a mature, battle-tested sharding layer for one engine (e.g. Vitess on MySQL). Demand arrives for an equivalent sharding capability on a second engine (e.g. Postgres). The tempting shortcut is port the existing sharding layer — reuse the proxy topology, reshard algorithms, query-rewrite logic, tooling, operational runbooks, and engineering muscle.

The shortcut has a structural problem: a sharding layer's correctness and performance depend on engine-specific properties of the underlying database. Replication protocol, binlog / WAL format, transaction isolation semantics, DDL characteristics, MVCC shape, lock-granularity, server-side pooling primitives — these are all different across engines. A MySQL-assumption-bearing sharding layer ported onto Postgres inherits none of the engine leverage that made it work for MySQL and gains all the engine-specific failure modes it wasn't designed for.

Solution

Architect the sharding layer for the new engine from first principles. Keep the surface-level abstractions (shard map, query router, online resharding, schema-change workflow) that your customers and operators already understand, but derive the implementation from the new engine's own properties.

Transferable across engines:

  • Operator knowledge (how to think about shards, pivots, resharding, topology awareness).
  • Customer-facing API shape (sharded DDL, reshard kickoff, schema-deployment semantics).
  • Tooling surface (dashboards, observability, failure-mode runbooks).

Not transferable — must be re-derived from the new engine:

  • Replication protocol (how the router tails changes for online resharding).
  • Transaction isolation assumptions (how the router handles cross-shard transactions).
  • Schema-change protocol (how the router orchestrates non-blocking DDL).
  • Query-rewrite semantics (how the router parses and routes a given SQL dialect's queries).
  • Failure-handling semantics (how to fence a failed primary, when to promote, how to catch up).

Canonical wiki instance

Neki — PlanetScale's from-first-principles sharding system for Postgres, announced alongside PlanetScale for Postgres on 2025-07-01.

PlanetScale's explicit framing:

Vitess is one of PlanetScale's greatest strengths and has become synonymous with database scaling. Contemporary Vitess is the product of PlanetScale's experience running at extreme scale. We have made explicit sharding accessible to hundreds of thousands of users and it is time to bring this power to Postgres. We will not however be using Vitess to do this.

Vitess' achievements are enabled by leveraging MySQL's strengths and engineering around its weaknesses. To achieve Vitess' power for Postgres we are architecting from first principles. We are well under way with building this new system and will be releasing more information and early access as we progress. As with all PlanetScale products we work with customers at scale to build and validate maturity.

(Source: sources/2025-07-01-planetscale-planetscale-for-postgres.)

The pattern's load-bearing claim is that "Vitess' achievements are enabled by leveraging MySQL's strengths and engineering around its weaknesses" — i.e. Vitess's design is specifically MySQL-shaped, and porting it to Postgres would lose exactly the architectural leverage that made it valuable.

Caveats

  • Customer-facing stability. Customers don't want two fundamentally different sharding APIs between engines. The pattern requires re-using the external shape even while rebuilding the internals.
  • Engineering cost. A from-first-principles rebuild is years of work; early-access / waitlist posture is architecturally necessary.
  • Organisational-learning transfer. The expertise PlanetScale accumulated running Vitess at scale transfers as "what can go wrong at scale" knowledge even when the specific implementation doesn't transfer — a non-trivial asset.
  • Counter-pattern: port the layer instead. [[systems/ vitess|Vitess itself]] has had experimental Postgres support attempts; the stance here is that the official maintainers don't believe the port is the right architecture.

Seen in

Last updated · 319 distilled / 1,201 read