Skip to content

SYSTEM Cited by 1 source

KRaft (Kafka Raft)

KRaft ("Kafka Raft") is Apache Kafka's in-tree distributed consensus mechanism — a Raft dialect heavily influenced by Kafka's existing replication protocol — that replaces ZooKeeper as the coordinator for cluster metadata. First production-ready release shipped in Kafka 3.3 (October 2022); Kafka 4.0 (expected Q3 2024) removes ZooKeeper entirely.

Stub page — expand on future KRaft-internals sources (quorum membership changes, snapshotting, metadata-log semantics at scale, failure-mode post-mortems). The canonical wiki entry point today is the 2024-05-09 Kafka 101 explainer.

Core idea — cluster metadata as an ordinary Kafka log

Kozlovski frames KRaft's foundational realisation:

"A key realization is that the cluster's metadata can be easily expressed in a regular log through the ordered record of events that happened in the cluster. Brokers could then replay these events to build up to the latest state of the system." (Source: sources/2024-05-09-highscalability-kafka-101)

Concretely:

  • Metadata topic: a special Kafka topic named __cluster_metadata with a single partition.
  • Raft-managed leader election: unlike every other topic in Kafka (where the Controller picks the partition leader), the metadata topic's leader election runs via Raft across the KRaft quorum.
  • Leader is the active Controller: whichever KRaft quorum member wins the metadata-partition leader election is the active Controller.
  • Hot standbys: other KRaft quorum members keep the metadata log in memory, ready for failover.
  • Brokers tail the log: regular (non-controller) brokers replicate the __cluster_metadata topic like any other topic; they no longer subscribe to ZooKeeper watches or directly RPC the Controller for metadata, they simply keep up with the log.

See concepts/kraft-metadata-log.

Deployment modes

Kozlovski: "KRaft supports two modes of deployment — combined and isolated mode."

  • Combined — a broker serves both the regular-broker role and the controller role (similar to the pre-KRaft model where a broker could be elected Controller on top of its broker duties).
  • Isolated — dedicated controller nodes serving only the KRaft quorum and hosting the metadata log; no ordinary topic-partition data. Typical deployments use 3 controller nodes.

Why replace ZooKeeper

  • One consensus system, not two — operators no longer run ZooKeeper alongside Kafka.
  • Uniform vocabulary — metadata lives in a Kafka log, consumable via the same tailing abstraction used everywhere else.
  • Scale — watch-driven pull from ZooKeeper strained at high partition counts; log-tailing scales with the existing replication substrate.

Seen in

Last updated · 319 distilled / 1,201 read