Skip to content

PATTERN Cited by 1 source

Staging validation before production deploy

Pattern

Before deploying updated streaming-layer code to production, first deploy to a staging pipeline (same Kafka source → staging Flink → staging CDC table), validate that the new code processes records correctly, then apply the production table change and deploy to production.

Rationale

Streaming jobs (Flink) are more sensitive than batch jobs (Spark) because: - A failed Flink job combined with Kafka retention expiration causes irreversible data loss - Spark is watermark-based and can resume from the last successful checkpoint - Streaming cannot be replayed once retention expires

The staging validation step catches parsing failures, type-conversion errors, and code-generation bugs before they can cause production data loss.

Seen in

Last updated · 559 distilled / 1,651 read