Skip to content

PATTERN Cited by 1 source

Shadow pipeline migration

Pattern

Run a complete shadow copy of a new stream-processing pipeline alongside the production system, consuming the same input streams but writing to dedicated shadow output streams. Compare shadow output against production output for correctness, completeness, and latency before cutting over.

Problem

Migrating stateful stream-processing systems is high-risk: the new system must produce identical (or provably equivalent) output at production scale, and stream processing bugs often only manifest under real traffic patterns, late-arriving events, and failure scenarios that synthetic tests cannot reproduce.

Solution structure

Input Streams ──┬── Production Pipeline ──→ Production Output
                └── Shadow Pipeline ──→ Shadow Output
                                              │
                                   Comparison Service
                                   (compares against production)

Validation protocol: 1. Run shadow pipeline consuming real production inputs 2. Compare output metrics (match rate, enrichment success, latency) 3. Multi-week observation period (Zalando: 4 weeks) 4. A/B test on downstream consumer to validate business outcomes (Zalando: 1 week) 5. Config-flip cutover on downstream service

Seen in

  • Zalando Ad Platform (2026-07): Full shadow pipeline with dedicated Nakadi input/output streams. Downstream service consumed both production and shadow streams in parallel, writing to a shadow table for comparison. Validation criteria: 99.9% enrichment success rate match, per-campaign-per-day billing equivalence, processing latency within 15-minute window, ~0.2% duplicate rate acceptable. After 4 weeks shadow + 1 week A/B, cutover was a config update (Source: sources/2026-07-23-zalando-from-homegrown-to-flink-migrating-a-stateful-ad-event-join).

When to use

  • Business-critical data pipelines where incorrect output has financial impact
  • Stateful systems where bugs depend on production traffic patterns
  • When you can afford the infrastructure cost of running two pipelines temporarily
Last updated · 595 distilled / 1,807 read