CONCEPT Cited by 1 source

Consumer group¶

Definition¶

A consumer group in Apache Kafka is a set of consumers that cooperatively divide the partitions of a topic among themselves. Group membership + partition-to-consumer assignment are coordinated through a broker — consumers "synchronize each other through talking to the broker — they are not connected to one another" (Kozlovski, sources/2024-05-09-highscalability-kafka-101).

Invariants¶

Partition exclusivity within a group — no two consumers in the same group read from the same partition (see patterns/consumer-group-partition-exclusivity). This is what preserves in-partition record ordering end-to-end: since a partition is consumed by exactly one group member, and records within a partition are ordered, the group member observes records in order.
Progress persisted — a consumer group persists how far it has consumed each partition (the committed offset) in a special internal topic __consumer_offsets.
Horizontal scale limit — a group can have at most as many actively-consuming members as the topic has partitions. Extra members sit idle.
Multi-group fan-out — the same topic can have many consumer groups simultaneously, each at its own position. This is the producer/consumer decoupling that let Kafka beat classical message buses.

Group Coordinator¶

Each consumer group has a Group Coordinator, which is the broker that leads the partition of __consumer_offsets the group's offsets are stored in:

"The broker that is the leader of the partition acts as the so-called Group Coordinator for that consumer group, and it is this Coordinator that is responsible for maintaining the consumer group membership and liveliness." (Source: sources/2024-05-09-highscalability-kafka-101)

The Group Coordinator:

Accepts join / leave messages from group members.
Runs the rebalance protocol when membership changes, reassigning partitions across current members.
Receives offset commits and persists them to __consumer_offsets.

Where consumer groups show up in the ecosystem¶

Application consumers — the ordinary case.
Kafka Streams — layered on top of consumer groups; sub-topology tasks become group members. See sources/2025-11-11-expedia-kafka-streams-sub-topology-partition-colocation.
Kafka Connect — workers use the consumer-group protocol to distribute connector tasks and to rebalance on worker failure.

Seen in¶

sources/2024-05-09-highscalability-kafka-101 — canonical definition of consumer groups, Group Coordinator, __consumer_offsets, partition-exclusivity, and multi-group fan-out.

systems/kafka
systems/kafka-connect — task-distribution built on the consumer-group protocol.
systems/kafka-streams — sub-topology tasks run as consumer group members.
concepts/kafka-partition — consumer groups divide partitions.
patterns/consumer-group-partition-exclusivity — the architectural pattern the partition-exclusivity invariant names.