PATTERN Cited by 1 source
Backfill-replay-cutover¶
A general pattern for zero-downtime data migration: create a new target structure, bulk-copy existing data (backfill), continuously replay changes that arrived during the copy (replay/converge), then atomically redirect traffic (cutover).
Structure¶
1. Create target (new schema / shard layout / index)
2. Backfill: bulk copy from source → target
3. Replay: stream ongoing changes (CDC / changelog) to target
4. Converge: repeat replay until lag < threshold
5. Cutover: brief pause/block → final catchup → atomic swap → resume
Key Decisions¶
| Decision | Options |
|---|---|
| Change stream source | WAL/binlog (MySQL gh-ost), Lucene changes snapshot (AOSC), DynamoDB Streams, Kafka offset |
| Backfill mechanism | Point-in-time snapshot, Engine.Searcher pin, SELECT scan |
| Cutover strategy | Write-block + swap, held-writes + alias flip, double-write + compare |
| Convergence metric | Operations gap, replication lag, doc-count delta |
Trade-offs¶
- Pro: Source stays writable and readable during migration (no long downtime).
- Pro: Migration logic lives in infrastructure, not application code.
- Con: Requires a change stream / CDC primitive from the storage layer.
- Con: Brief cutover window still exists (sub-second to seconds) where writes may be rejected or delayed.
- Con: Correctness validation (source == target) is non-trivial at scale.
Seen In¶
- systems/opensearch-aosc — Atlassian's AOSC uses retention leases + Lucene changes snapshot for the replay phase, with a write-block cutover
- systems/gh-ost — GitHub's MySQL online schema change uses binlog streaming for the replay phase
- systems/pt-online-schema-change — Percona's tool uses triggers for change capture
- patterns/shadow-table-online-schema-change — the MySQL-specific instantiation of this pattern
(Source: sources/2026-07-24-atlassian-online-index-migration-and-shard-scaling-in-opensearch-with-aosc)