Skip to content

PATTERN Cited by 1 source

Consumer-group partition exclusivity

Pattern

Within a consumer group, each partition is assigned to exactly one consumer instance at a time. No two consumers in the same group can read the same partition simultaneously.

Canonical production instance: Apache Kafka consumer groups.

Why the invariant matters

Kafka guarantees record ordering per partition, not across partitions. Record-ordering is therefore preserved end-to-end only if the consumer side preserves it per-partition too. Kozlovski's Kafka-101 framing:

"The records in any single partition are ordered within it. Consumers are guaranteed to read it in the right order. To ensure that the order is preserved, the consumer group protocol ensures that no two consumers in the same consumer group can read from the same partition." (Source: sources/2024-05-09-highscalability-kafka-101)

Partition-exclusivity is the mechanism Kafka uses to lift its producer-side ordering guarantee (per partition) into a consumer-side ordering guarantee (per partition, per group) without requiring consumers to coordinate directly.

Scale implication

Since a partition is the unit of exclusive consumption, a consumer group can have at most partition_count actively-consuming members on a given topic. Consumers beyond that number sit idle — they're failover capacity.

This is the dial operators tune: topic partition count sets the ceiling on consumer-group parallelism. Increase partitions to increase parallelism; more partitions mean more replicas, more metadata, more IO, more rebalance work.

Rebalance protocol

Partition-to-consumer assignment is mutable. Membership changes (a consumer joins or leaves the group, or fails heartbeats) trigger a rebalance: partitions are re-distributed across the current members. The Group Coordinator (concepts/consumer-group) runs this protocol.

  • Multi-group fan-out — different consumer groups are independent; the same topic can have many groups at different positions. Producer/consumer decoupling.
  • Kafka Streams builds on this. A Streams application is a consumer group under the hood; tasks map 1:1 onto partitions within the application's sub-topologies. Sub-topology-scoped cross-topic colocation (sources/2025-11-11-expedia-kafka-streams-sub-topology-partition-colocation) is partition-exclusivity generalised across multiple topics consumed by the same logical task.
  • Kafka Connect tasks also distribute via the consumer-group protocol, inheriting partition-exclusivity.

Seen in

Last updated · 319 distilled / 1,201 read