PATTERN Cited by 1 source

Dual-pipeline deprecation

Intent

Safely decommission legacy data tables by running old and new pipelines in parallel for validation before cutting over, minimizing risk to downstream consumers.

How it works

1. Identify legacy tables that no longer meet new standards
2. Build the replacement pipeline/tables following new conventions
3. Run both pipelines simultaneously — old and new produce outputs
4. Validate that the new pipeline produces equivalent or correct results
5. Communicate the migration to downstream consumers (potentially hundreds)
6. Gradually cut consumers over to the new tables
7. Only decommission the old pipeline after all consumers have migrated

The process is intentionally slow and careful because legacy tables can have hundreds of downstream dependencies (Source: sources/2026-06-09-airbnb-scaling-beyond-one-data-architecture).

When to use

• Legacy data tables with many downstream consumers
• New modeling standards that make old tables non-compliant
• Any data pipeline migration where correctness is critical

Trade-offs

• Pro: Minimizes risk — old pipeline continues to work during transition
• Pro: Enables validation before commitment
• Con: Resource-intensive — running both pipelines doubles compute cost
• Con: Slow — the deprecation cycle can take months
• Con: Organizational overhead — requires extensive communication with consumers

Related patterns

• patterns/dual-write-migration — similar dual-run concept for online systems

Seen in

• sources/2026-06-09-airbnb-scaling-beyond-one-data-architecture — Airbnb's approach to migrating legacy Experiences tables to the new multi-product framework