Skip to content

PATTERN Cited by 1 source

Dual-write branch cutover via new cluster

Problem. A sharded cluster needs to grow shard count (4 → 8, 8 → 16, etc). A live Vitess / logical-replication reshard is available but:

  • Shares the source cluster's production traffic throughout.
  • Failure rollback requires tearing down an in-flight replication stream.
  • Provides no independent validation window before the new shard count is in the critical path.

Solution. Provision the new-shard-count cluster empty. Point a second consumer fleet (reading the same upstream durable stream) at the new cluster. Dual-write for at least the retention window; the new cluster accumulates a full retention's worth of data naturally. Validate the new cluster with load tests + production metrics during the dual-write window. Cut reads over once the new cluster is trusted.

   upstream stream (Kafka, CDC, event bus)
         │
         ├────────► old-consumers ──► old-cluster  (4 shards) ← reads
         │
         └────────► new-consumers ──► new-cluster  (8 shards)
                                             │
                                          validate
                                       load-tests + prod
                                       metrics
                                             │
                            day N+retention: cut reads over

(Source: sources/2026-04-21-planetscale-storing-time-series-data-in-sharded-mysql-to-power-query-insights.)

Canonical PlanetScale Insights application

Rafer Hazen, 2023-08-10: "Insights originally shipped with four shards, but we increased this to eight earlier this year to keep up with increased write volume and to build operation experience resharding. Vitess can re-shard an actively used database, but we opted to provision a new, larger, PlanetScale database when we needed to increase the number of shards. Since Insights currently stores eight days of data, we provisioned a new set of consumers, let the new branch receive duplicate writes for eight days, and then cut the application over to read from the new database. This method allowed us to test and gain confidence in the new cluster before placing it in the critical path. Based on load tests and resource utilization metrics in production, we've found that our maximum write throughput has so far scaled linearly with the number of shards."

Structural preconditions

  1. Durable upstream stream. The cluster's data must be re-derivable from an upstream source (Kafka topic, CDC stream, event bus). If the cluster is the primary store, this pattern doesn't apply — use live reshard.
  2. Short retention window. The dual-write duration has a floor equal to the retention window: the new cluster isn't complete until it has stored a full retention's worth of data. For 8-day Insights retention, the dual-write is 8 days. For a 90-day retention this is 3 months — borderline, but still reasonable. For year-plus retention, the cost of holding two clusters online becomes prohibitive and live reshard is the right call.
  3. Consumer-side fanout is cheap. Running a second consumer fleet doubles the consumer-side cost; this is acceptable only when consumer cost is much smaller than DB storage cost.

Why it's chosen over Vitess live reshard

  • Independent validation. The new cluster is exercised on real production write load during the dual-write window without any read traffic. Load tests and resource-utilisation metrics are measured on a cluster that's functionally but not critically in the path — if metrics look bad, rollback is "don't cut over."
  • Clean state. The new cluster ends the dual-write window holding exactly retention-window data with no replication-generated historical backfill. Anything weird in the backfill process that would appear on a live-reshard target doesn't exist here.
  • Operator experience. Hazen's framing: "to build operation experience resharding." Running two distinct reshard mechanisms on different parts of the fleet builds muscle on both. PlanetScale operates Vitess live reshard for customer databases; for Insights' retention-bounded own-fleet data, the new-cluster path is viable and cheaper to validate.

Cost profile

  • Consumer cost: 2× during the dual-write window. For a consumer fleet that's small relative to the DB, this is negligible fleet-wide.
  • Storage cost: 2× during the dual-write window. For Insights' 8-day retention, this is 8 cluster-days of duplicate storage — a small one-time cost.
  • Write cost: 2× during the dual-write window to the downstream MySQL cluster fleet.
  • Read cost: 1× throughout (reads always go to a single cluster).

When this pattern is the wrong answer

  • Primary data store. No re-derivable upstream stream exists; use live reshard / logical replication.
  • Long retention. Dual-write duration is prohibitive.
  • Cardinality blow-up during dual-write. If the source data is growing fast, the new cluster's steady-state size after retention is larger than the old cluster's current size — operator capacity planning has to account for the target-steady-state, not the source-current-size.
  • Cross-database transactions / foreign keys. The dual consumer fleets don't coordinate; strongly-consistent cross-shard invariants during the dual-write window aren't enforceable.

Variant — Kafka offset-pin for bootstrap

A variant of this pattern keeps the new consumer fleet pinned to the oldest Kafka offset the retention window reaches, so the new cluster is deterministically reproducible from the source stream. Insights' post doesn't explicitly describe this but it's the natural implementation.

Seen in

Last updated · 470 distilled / 1,213 read