PATTERN Cited by 4 sources

Shadow migration (dual-run reconciliation)

Shadow migration (a.k.a. dual-run with reconciliation) is the pattern of running the new engine in parallel with the old, feeding both the same inputs, producing both outputs, and reconciling results before any consumer sees the new engine's output. It is the canonical risk-reduction pattern for migrating business-critical data pipelines between engines.

Shape

1. Manual shadow — pick a small representative subset; run the new engine against the same inputs as the old. Compare outputs.
2. Automated shadow — wire the new engine into production alongside the old; every job runs in both engines simultaneously. Compare outputs continuously. Keep the old engine's output authoritative throughout.
3. Reconciliation — two layers:
4. Dataset-level statistical equivalence — record counts, cardinalities, min/max/avg/distribution per column. Good for "did we lose any rows? did any column's distribution shift?"
5. Real-query reconciliation — run actual consumer queries (across multiple consumer engines, not just one) against both outputs, compare results. Good for "does a real reader see a difference?"
6. Subscriber switchover — once shadow agrees and production confidence is earned, move individual consumers off the old engine with the freedom to reverse course per-consumer (subscriber-switchover).

Cost side

• You temporarily double the compute cost of the pipeline during the shadow phase. This is a real operational expense that must be budgeted, not wished away.
• Amazon BDT: "If everything crashed and burned at this point, it would become a capital loss for the business and filed away as a hard lesson learned."

Why byte-for-byte is the wrong bar

At TB/PB scale, queries run on different compute frameworks against the same data almost never produce byte-equal results:

• Decimal rounding differences
• Non-deterministic execution plans (unstable sort)
• Adding/removing metadata in Parquet files
• Different value-overflow/underflow handling
• Type-comparison opinions: -0 == 0? timezone-aware vs -naive timestamp equality?
• Pre-Gregorian calendar date interpretation drift
• Leap-second handling

Trying to enforce byte-for-byte equivalence is a false bar that will burn the migration team chasing non-bugs. Statistical + real-query equivalence is the pragmatic bar that survives real cross-framework comparison.

Amazon BDT's instantiation

On the Spark → Ray migration:

• DQ Service (Ray-based) for dataset-level statistical equivalence — counts, cardinalities, min/max/avg, Parquet feature parity (e.g. Bloom filters present iff both sides produced them).
• Data Reconciliation Service for real-query equivalence — ran queries through systems/amazon-redshift, systems/apache-spark, and systems/amazon-athena against both outputs.
• Progression: manual shadow (2022) → automated 1:1 shadow (2023) → subscriber switchover (2024).

(Source: sources/2024-07-29-aws-amazons-exabyte-scale-migration-from-apache-spark-to-ray-on-ec2)

Related

• subscriber-switchover — the cutover pattern shadow migration earns you the right to do.
• dual-write-migration — the writes-side dual.
• achievable-target-first-migration — the complement pattern of picking the first migration target that you can actually finish.
• concepts/copy-on-write-storage-fork — the workload in Amazon BDT's case.

Seen in

• sources/2026-10-01-aws-accelerating-airline-retailing-innovation-datalex-modernization — non-regression parallel validation during a Strangler Fig migration. Datalex runs new Spring Boot microservices in parallel with the existing n-tier system; the business service proxy can route to the old REST API or the modernized API "through a simple parameter change," enabling rapid non-regression testing of each migrated slice before cutover — the validation half of a proxy-routed incremental migration.

• sources/2024-07-29-aws-amazons-exabyte-scale-migration-from-apache-spark-to-ray-on-ec2 — canonical instantiation across a multi-year migration on a business-critical pipeline, with explicit progression from manual shadow to automated 1:1 shadow to switchover, and explicit statistics-over-bytes reconciliation bar.

• sources/2026-06-19-netflix-the-evolution-of-cassandra-data-movement-at-netflix — Netflix's Casspactor→Move Data migration used shadow mode with a formal C = M row-level set-equality proof as the trust metric. Any non-zero difference triggered immediate investigation. See patterns/shadow-migration for the formalised variant.
• sources/2026-09-18-cloudflare-saving-another-100tb-of-ram-with-math-and-rust — dual-run variant applied to a consistent-hash ring swap. PBR carried both the old ketama v1 ring and the new smaller v2 ring in memory, choosing per request via the migration framework (stable per request hash → clean rollback). The reconciliation bar here is not byte-for-byte output equality but cache/origin behavior — a bad ring swap re-homes cache keys and floods origins, so they watched backend-selection traces, ring-version counters, connection errors, memory, startup time, cache behavior, and origin traffic. Pair with patterns/progressive-configuration-rollout — the where it's allowed to move (data-center-scoped) dimension is that pattern.

Merged aliases

• shadow-pipeline-migration
• shadow-then-reverse-shadow-migration- api-parity-migration
• backfill-replay-cutover
• canary-shard-substrate-migration
• dark-ship-for-behavior-parity
• event-type-by-event-type-shadow-cutover
• nondisruptive-migration
• notion-double-write-backfill-verify-switchover
• parallel-run-pattern
• shadow-application-readiness
• shadow-mode-bytes-comparison
• shadow-mode-parallel-validation
• shadow-traffic-plus-reindex-blue-green-upgrade
• transparent-migration
• vdiff-verify-before-cutover- audit-then-refactor-migration
• dual-route-stateless-legacy-migration
• dual-system-sync-during-migration
• scoped-migration-with-fast-follows
• shadow-table-online-schema-change
• shadow-validation-dependency-graph