CONCEPT Cited by 1 source

Flink Interval Join seek overhead

Definition

A pathological performance characteristic of Flink's built-in Interval Join operator when used with unique-key partitions at high throughput. The Interval Join implementation opens a RocksDB iterator and performs a seek() before reading any records on the matching path — regardless of how many entries the partition contains. When each partition holds exactly one entry (unique keys), the seek fires on every element processed, paying the cost of locating a starting position across sorted files on disk with no amortisation.

Two sources of overhead

1. Per-element seek on the matching path: Every incoming element triggers a seek into the other stream's state, even for single-entry partitions where a direct point-get would suffice.

2. Expiration timer queue: Interval Join uses explicit timers stored in a RocksDB-backed sorted set with an in-memory cache. At high throughput with a wide window (e.g., 15 minutes), this queue grows to tens of millions of entries (~70 million at Zalando's scale). When the cache drains, Flink seeks into RocksDB to reload the next batch — generating a second source of constant scans.

Seen in

• Zalando Ad Platform (2026-07): Flame graph showed 15–30% of CPU spent on seek operations. The ~70 million timer entries with a 15-minute window at 200 MB/s throughput were the second overhead source. Even increasing cache block sizes couldn't solve the algorithmic unsuitability (Source: sources/2026-07-23-zalando-from-homegrown-to-flink-migrating-a-stateful-ad-event-join).

Remediation

Replace Interval Join with KeyedCoProcessFunction for unique-key workloads — direct point lookups instead of seeks, RocksDB TTL compaction instead of explicit timers.