Skip to content

CONCEPT Cited by 3 sources

Sticky partitioner

Definition

The sticky partitioner is a Kafka producer-side partitioning strategy that concentrates all unkeyed records from a single client onto one partition at a time, rotating to the next partition only when the current batch fills up to batch.size. It replaces the classic round-robin partitioner (which would distribute each unkeyed record across all partitions of a topic) for the default unkeyed-record case. Introduced in Apache Kafka via KIP-480 (2.4, 2019) and the default unkeyed partitioner since.

Why it exists

Under the classic round-robin partitioner, increasing partition count reduces effective batch size per client because each partition gets only a 1/N share of the client's record rate. A client writing 100 records/sec to a 1-partition topic fills a batch of 100 records/sec on that partition; the same client writing 100 records/sec to a 10-partition topic fills only 10 records/sec per partition, so each batch takes 10× longer to form.

The sticky partitioner breaks this pinning:

"Increased partition count doesn't affect batch size when the sticky partitioner is in use since that partitioner writes all messages to a single partition (read: batch) until a size threshold is reached based on batch.size." (Source: sources/2024-11-19-redpanda-batch-tuning-in-redpanda-for-optimized-performance-part-1)

The effect: at steady state, the client forms a full-batch.size batch on partition A, dispatches it, then rotates to partition B for the next batch, then C, and so on. Each partition still sees its fair share over time, but batches are big instead of diluted.

Composing with concepts/effective-batch-size

The sticky partitioner is the canonical mitigation for factor 4 (partitioning) in the seven-factor effective-batch-size framework. With sticky active, factor 4 is effectively neutralised — adding more partitions doesn't shrink per-client batch size. This makes partition count a pure parallelism knob rather than a batching regression.

Scope — only unkeyed records

The sticky partitioner applies only when records have no key. Keyed records still hash to partitions deterministically (same key ⇒ same partition), because key-to-partition determinism is a correctness requirement for ordering guarantees in Kafka partitions. Sticky changes only how the client chooses a partition for records that don't pin one via key.

Trade-offs

  • Local ordering of unkeyed records within a batch is still preserved by the partition's append-only log.
  • Per-partition temporal distribution of records within a short window is more bursty under sticky (all records for a given batch-size window land on one partition, then rotate) than under round-robin. For consumers reading from many partitions simultaneously, this is typically invisible; for consumers reading from one partition under load-balance-style semantics, sticky can concentrate bursts.
  • Partition balance over long windows is approximately uniform. Under steady-state load, each partition receives ~1/N of records.

Seen in

Last updated · 470 distilled / 1,213 read