CONCEPT Cited by 1 source

Leader-follower replication (Kafka partition)

Definition

In Apache Kafka, every partition has exactly one leader at a time; the other replicas are followers. Writes always go to the leader; followers pull from the leader and apply the same log; if the leader dies, one of the in-sync followers is elected the new leader.

Kozlovski's Kafka-101 framing:

"The replication is leader-based, which is to say that only a single broker leads a certain partition at a time. […] Writes can only go to that leader, which then asynchronously replicates the data to the N-1 followers." (Source: sources/2024-05-09-highscalability-kafka-101)

See concepts/asynchronous-replication for the general asynchronous-replication primitive; this page is the Kafka partition-scoped specialisation.

Invariants

• Single writer per partition — the leader is the only replica accepting writes; followers are read-only copies maintained via replication. Keeps the log linear and the offset monotone.
• Followers can serve reads — since Kafka 2.4+, consumers can "read from the closest replica in the network topology" (not just the leader); leader-based writes + any-replica reads.
• Leader election is Controller-owned — when a leader dies or shuts down, the active Controller (pre-3.3: ZooKeeper + /controller zNode; 3.3+: KRaft quorum member that leads the __cluster_metadata partition) picks a new leader from the partition's ISR: "This is most notable in failover cases where a leader broker dies, or even when it is shutting down. In both of these cases, the Controller reacts by gracefully switching the partition leadership to another broker in the replica set."

Composes with

• Producer acks — the producer's acks=0/1/all dial is evaluated against this leader/follower topology. acks=all means leader waits until the whole ISR has persisted.
• ISR — leader election only chooses from the ISR (by default); replicas outside the ISR are ineligible for leadership.

Seen in

• sources/2024-05-09-highscalability-kafka-101 — canonical statement of Kafka's leader-based partition replication + Controller-mediated leader election + async-replicate-to-followers shape.

Related

• systems/kafka
• concepts/kafka-partition — the unit over which leader-follower replication runs.
• concepts/in-sync-replica-set — leader-election eligibility.
• concepts/acks-producer-durability — producer durability dial evaluated against this shape.
• concepts/asynchronous-replication — the general class.
• patterns/leader-based-partition-replication — the architectural pattern this concept names.