Skip to content

PATTERN Cited by 1 source

KeyedCoProcessFunction over Interval Join

Pattern

For time-bounded event joins where keys are unique (or very low cardinality per partition), replace Flink's built-in Interval Join with a custom KeyedCoProcessFunction that uses direct ValueState point lookups and RocksDB TTL compaction for state expiry.

Problem

Flink's Interval Join is algorithmically suited for partitions with multiple entries — it maintains sorted state with iterators and explicit timer queues for expiration. When each key has exactly one entry (unique IDs), the generic algorithm becomes pathological: - One seek per element (expensive disk positioning for a single-entry read) - Millions of timer registrations that are never amortised - 15–30% CPU consumed on seek overhead alone at high throughput

Solution structure

Stream A (bids) ──┐
                  ├─ keyBy(bidId) → KeyedCoProcessFunction
Stream B (clicks) ┘
                      ├── ValueState<BidEvent> bidState
                      ├── ValueState<ClickEvent> clickState
                      └── No timers — TTL compaction handles expiry

On each element: 1. Point-get the other stream's state for this key 2. If found → emit match, clear both states 3. If not found → store this element in its own ValueState

State expiry: RocksDB TTL configured to the join window duration; background compaction removes expired entries.

Seen in

When to use

  • Keys are unique or very low-cardinality per partition
  • High throughput where per-element overhead compounds
  • You have Java/Kotlin expertise to implement and maintain custom join logic

When NOT to use

  • Many-to-many joins where iterator-based matching is genuinely needed
  • Low throughput where Interval Join's overhead is negligible
  • Teams without Flink internals knowledge for debugging custom state logic
Last updated · 595 distilled / 1,807 read