Skip to content

CONCEPT Cited by 1 source

KRaft metadata log (cluster metadata as a Kafka log)

Definition

The KRaft metadata log is Apache Kafka's post-ZooKeeper mechanism for storing cluster metadata. A special topic __cluster_metadata (single partition) whose leader election runs via Raft across a small KRaft quorum (typically 3 controllers). The leader of that partition is the active Controller; other quorum members are hot standbys. Regular brokers replicate this topic asynchronously — they learn cluster metadata by tailing the log, not by watching a coordinator.

Core insight — metadata is a log

Kozlovski's Kafka-101 framing:

"A key realization is that the cluster's metadata can be easily expressed in a regular log through the ordered record of events that happened in the cluster. Brokers could then replay these events to build up to the latest state of the system." (Source: sources/2024-05-09-highscalability-kafka-101)

"In this new model, Kafka has a quorum of N controllers (usually 3). These brokers host a special topic called the metadata topic ('__cluster_metadata'). This topic has a single partition whose leader election is managed by Raft (as opposed to the Controller for every other topic). The leader of the partition becomes the currently active Controller. The other controllers act as hot standbys, storing the latest metadata in memory." (Source: sources/2024-05-09-highscalability-kafka-101)

So the Controller-election protocol for ordinary Kafka topics requires a Controller to already exist — but the __cluster_metadata topic's leader election is the Controller-election itself, run by Raft. This is the bootstrapping trick.

Consequences

  • One consensus system, not two. Operators no longer run ZooKeeper alongside Kafka; KRaft is in-tree.
  • Uniform vocabulary. Cluster metadata lives in a Kafka log, consumable by the same tail-this-log abstraction used for every other topic.
  • Push is replaced with pull. Pre-KRaft, Kafka used ZooKeeper's watch mechanism to push metadata change notifications to subscribers. Post-KRaft, regular brokers tail the metadata log asynchronously — a large step in the direction of patterns/pull-on-demand-replacing-push at the control plane.
  • Scale. The pull-by-tailing substrate scales with Kafka's existing log-replication substrate rather than ZooKeeper's watch-subscriber count.
  • Deployment-mode flexibility. KRaft supports combined mode (a broker doubles as a controller — ZooKeeper-era shape) and isolated mode (dedicated controller nodes).

Timeline

  • First production-ready KRaft: Kafka 3.3 (October 2022).
  • ZooKeeper fully removed: Kafka 4.0 (expected Q3 2024 at time of post).

Seen in

Last updated · 319 distilled / 1,201 read