CONCEPT Cited by 1 source

Flink KeyedCoProcessFunction

Definition

KeyedCoProcessFunction is a low-level Flink DataStream API primitive that processes elements from two connected keyed streams with access to per-key state and timers. Unlike high-level join operators (Interval Join, Window Join), it gives full control over matching logic, state management, and expiration strategy.

Why it matters

High-level join operators like Interval Join impose their own state management and expiry mechanisms that may be pathological for specific access patterns. KeyedCoProcessFunction allows:

• Direct point lookups in RocksDB (no iterator/seek) when keys are unique
• Custom expiry via RocksDB TTL compaction instead of per-element timer registration
• Asymmetric logic — different handling for each stream's elements

Seen in

• Zalando Ad Platform (2026-07): Two ValueState objects (bid events + interaction events) partitioned by bid ID. Since bid IDs are unique, each lookup is a direct point-get — no iterator, no seek. Replaced Interval Join which used 15–30% CPU on seeks alone. Result: 3× fewer subtasks, 50% less CPU, 67% less memory (Source: sources/2026-07-23-zalando-from-homegrown-to-flink-migrating-a-stateful-ad-event-join).

• Zalando Search & Browse (2026-03): MultiStreamJoinProcessor via union(...) → keyBy(SKU) → process() with single ValueState[EnrichmentState] per SKU, replacing a 4-way Table API JOIN chain. State reduced 235 GB → 56 GB (Source: sources/2026-03-03-zalando-why-we-ditched-flink-table-api-joins-cutting-state-by-75-with-datastream-unions).

Trade-offs

• Pro: Full algorithmic control, optimal for unique-key workloads, no hidden seek/timer overhead
• Con: More code to write and maintain; correctness burden shifts from framework to developer; no built-in watermark-based window semantics
• When to use: Unique-key or low-cardinality-per-key partitions where high-level operators' generic algorithms produce pathological overhead