CONCEPT Cited by 2 sources

Flink incremental checkpoint

Definition

A Flink checkpoint mode where only the delta (new/changed SST files) since the last checkpoint is uploaded to durable storage, rather than the full state snapshot. Built on RocksDB's immutable SST file design — once written, SST files never change, so previously uploaded files don't need re-upload.

Why it matters

For jobs with large state (tens to hundreds of GB), full checkpoints dominate cluster resources: - Upload bandwidth scales with total state size - Checkpoint duration grows linearly with state - CPU spikes during checkpoint can cause backpressure

Incremental checkpoints break this proportionality — upload cost scales with the write rate (new SST files since last checkpoint), not total state size.

Seen in

• Zalando Ad Platform (2026-07): 3-minute incremental checkpoints to RocksDB state backend enabled recovery from last checkpoint instead of losing up to 15 minutes of buffered events. Made aggressive autoscaling safe (1–8 pods vs constant 20) without state loss. The previous homegrown solution had no checkpointing at all (Source: sources/2026-07-23-zalando-from-homegrown-to-flink-migrating-a-stateful-ad-event-join).

• Zalando Search & Browse (2026-03): Hourly savepoints on 235 GB state consumed 12 minutes of 100% CPU and caused backpressure. After state reduction to 56 GB, snapshot duration dropped to 2.5 minutes (Source: sources/2026-03-03-zalando-why-we-ditched-flink-table-api-joins-cutting-state-by-75-with-datastream-unions).

Relationship to autoscaling

Incremental checkpoints are what make Flink autoscaling safe: on rescale, the job restarts from the last checkpoint with state redistributed across the new parallelism. Without checkpointing, any rescale loses all buffered state — forcing operators to overprovision pods to avoid rescaling entirely.