Definition
- KRaft (Kafka Raft) – Kafka's built-in metadata management mode where cluster metadata is stored in an internal replicated log managed by controller nodes using the Raft consensus protocol.
- ZooKeeper – the external coordination service Kafka historically depended on for cluster metadata, broker registration, topic configuration, and leader election.
- Controller quorum – a set of dedicated controller nodes (typically 3 or 5) that form a Raft group, elect an active controller, and replicate metadata changes.
__cluster_metadata– the internal Kafka topic that stores all cluster metadata (topics, partitions, replica assignments, configurations, ACLs) in KRaft mode.process.roles– the KRaft configuration setting that defines whether a node acts as a broker, a controller, or both (combined mode for small clusters).
Imagine National Bank of India used to rely on a separate outside agency (ZooKeeper) to keep the master list of which vault holds which ledger, who is in charge, and the rules. Every time something changed, the bank had to phone that outside agency, which was slow and one more thing that could fail. KRaft is like the bank bringing that master list in-house and letting its own senior managers (controller nodes) agree on it among themselves using a clear voting rule (Raft). Now there is one organization to run, decisions are faster, and there is no outside dependency to break.
Understanding the Concepts
What was ZooKeeper's role in Kafka?
For most of its history, Kafka used ZooKeeper as an external coordination service to store cluster metadata: the list of brokers, topic and partition definitions, which broker leads each partition, configurations, and ACLs. Brokers watched ZooKeeper for changes, and a single broker acted as the controller that pushed metadata updates.
package com.bank.kafka;
public class ZooKeeperRole {
public static void main(String[] args) {
System.out.println("ZooKeeper stored (before KRaft):");
System.out.println(" - Broker registration and health");
System.out.println(" - Topic definitions and partition count");
System.out.println(" - Replica assignments per partition");
System.out.println(" - Leader election state");
System.out.println(" - Topic configurations");
System.out.println(" - ACLs (access control lists)");
System.out.println();
System.out.println("Problems:");
System.out.println(" - Separate system to deploy, secure, tune, monitor");
System.out.println(" - Scalability bottleneck for large partition counts");
System.out.println(" - Slower leader election and metadata propagation");
}
}
Expected output:
ZooKeeper: separate system for metadata, ACLs, leader election
Problems: operational overhead, scalability bottleneck
What is KRaft and how does it work?
KRaft removes ZooKeeper entirely. Metadata now lives in an internal, replicated Kafka log topic named __cluster_metadata, and a dedicated set of controller nodes forms a Raft quorum to manage it. Raft is a consensus algorithm where the quorum elects a leader (the active controller) and replicates an ordered log of metadata changes to the follower controllers.
package com.bank.kafka;
public class KRaftArchitecture {
public static void main(String[] args) {
System.out.println("KRaft Architecture:");
System.out.println();
System.out.println("Controller Quorum (Raft):");
System.out.println(" Controller 1: ACTIVE (elected leader)");
System.out.println(" Controller 2: FOLLOWER");
System.out.println(" Controller 3: FOLLOWER");
System.out.println();
System.out.println("Internal topic: __cluster_metadata");
System.out.println(" Stores: topics, partitions, replicas, configs, ACLs");
System.out.println(" Replicated across controller quorum");
System.out.println();
System.out.println("Brokers follow the metadata log:");
System.out.println(" Apply changes incrementally (like a consumer)");
System.out.println(" No ZooKeeper watching, no full snapshots");
}
}
Expected output:
KRaft: 3-node controller quorum managing __cluster_metadata
Brokers consume metadata changes incrementally
No external ZooKeeper dependency
How is KRaft different from ZooKeeper? (comparison table)
| Feature | ZooKeeper Mode | KRaft Mode |
|---|---|---|
| Metadata storage | External ZooKeeper ensemble | Internal __cluster_metadata topic |
| Dependencies | Kafka + ZooKeeper (2 systems) | Kafka only (1 system) |
| Controller election | ZooKeeper ephemeral nodes | Raft consensus |
| Metadata propagation | Watch notifications + snapshots | Incremental log consumption |
| Partition scale | ~200K partition limit | Millions of partitions |
| Controller failover | Slower (ZooKeeper session timeout) | Faster (Raft leader election) |
| Security | Two systems to secure | One system to secure |
package com.bank.kafka;
public class ZooKeeperVsKRaft {
public static void main(String[] args) {
System.out.println("ZooKeeper vs KRaft:");
System.out.println();
System.out.println("ZooKeeper: 2 systems, external dependency,");
System.out.println(" ~200K partition limit, slower failover");
System.out.println();
System.out.println("KRaft: 1 system, self-contained,");
System.out.println(" millions of partitions, faster failover");
}
}
Expected output:
ZooKeeper: 2 systems, 200K partition limit, slower failover
KRaft: 1 system, millions of partitions, faster failover
How do you configure KRaft?
In KRaft mode, you configure each node with process.roles (broker, controller, or both) and a controller.quorum.voters list. Startup uses a formatted cluster id rather than ZooKeeper connection strings.
package com.bank.config;
public class KRaftConfiguration {
public static void main(String[] args) {
System.out.println("KRaft Configuration (server.properties):");
System.out.println();
System.out.println("# Controller-only node:");
System.out.println("process.roles=controller");
System.out.println("node.id=1");
System.out.println("controller.quorum.voters=1@ctrl1:9093,"
+ "2@ctrl2:9093,3@ctrl3:9093");
System.out.println("controller.listener.names=CONTROLLER");
System.out.println();
System.out.println("# Broker-only node:");
System.out.println("process.roles=broker");
System.out.println("node.id=101");
System.out.println("controller.quorum.voters=1@ctrl1:9093,"
+ "2@ctrl2:9093,3@ctrl3:9093");
System.out.println();
System.out.println("# Combined mode (small cluster):");
System.out.println("process.roles=broker,controller");
System.out.println("node.id=1");
}
}
Expected output:
Controller-only: process.roles=controller
Broker-only: process.roles=broker
Combined: process.roles=broker,controller (small clusters)
Why does KRaft matter for performance?
Because metadata is itself an event log, brokers consume metadata changes the same way consumers read a topic: they follow the metadata log and apply updates incrementally. This is far more efficient than the old model of watching ZooKeeper and reloading large snapshots. Leader elections and metadata propagation are much faster.
package com.bank.performance;
public class KRaftPerformanceGains {
public static void main(String[] args) {
System.out.println("KRaft Performance Improvements:");
System.out.println();
System.out.println("Controller failover:");
System.out.println(" ZooKeeper: 10-30 seconds (session timeout)");
System.out.println(" KRaft: 1-3 seconds (Raft election)");
System.out.println();
System.out.println("Topic creation:");
System.out.println(" ZooKeeper: write to ZK, notify controller, propagate");
System.out.println(" KRaft: controller writes to metadata log, done");
System.out.println();
System.out.println("Partition scale:");
System.out.println(" ZooKeeper: practical limit ~200K partitions");
System.out.println(" KRaft: tested to millions of partitions");
System.out.println();
System.out.println("For banking: faster failover = less disruption");
}
}
Expected output:
Controller failover: ZK 10-30s -> KRaft 1-3s
Partition scale: ZK ~200K -> KRaft millions
Faster failover = less disruption for banking
What is the migration path from ZooKeeper to KRaft?
KRaft became production-ready in Kafka 3.3, ZooKeeper was deprecated through the 3.x line, and Kafka 4.0 removed ZooKeeper support completely. Teams upgrading from older ZooKeeper-based clusters must perform a migration to KRaft before moving to 4.0.
package com.bank.migration;
public class ZooKeeperToKRaftMigration {
public static void main(String[] args) {
System.out.println("ZooKeeper to KRaft Migration Path:");
System.out.println();
System.out.println("Kafka 3.3: KRaft production-ready (opt-in)");
System.out.println("Kafka 3.4-3.9: ZooKeeper deprecated");
System.out.println("Kafka 4.0: ZooKeeper REMOVED (KRaft only)");
System.out.println();
System.out.println("Migration Steps:");
System.out.println("1. Upgrade to Kafka 3.6+ (bridge version)");
System.out.println("2. Provision controller quorum nodes");
System.out.println("3. Run migration tool to copy metadata");
System.out.println("4. Switch brokers to KRaft mode");
System.out.println("5. Validate in staging");
System.out.println("6. Decommission ZooKeeper ensemble");
System.out.println("7. Upgrade to Kafka 4.0");
}
}
Expected output:
Kafka 3.3: KRaft ready
Kafka 4.0: ZooKeeper removed
Migrate: provision controllers -> copy metadata -> switch -> validate
What are common mistakes with KRaft?
The most common mistake is trying to upgrade directly to Kafka 4.0 from a ZooKeeper-based cluster without performing the KRaft migration first. Another mistake is sizing the controller quorum with an even number of nodes.
package com.bank.mistakes;
// WRONG: Even number of controllers.
// 4 controllers need 3 for majority = tolerates only 1 failure.
// 3 controllers need 2 for majority = also tolerates 1 failure.
// 4 controllers adds cost without improving fault tolerance.
public class WrongControllerCount {
public static void main(String[] args) {
System.out.println("WRONG: 4 controllers");
System.out.println(" Majority = 3, tolerates 1 failure");
System.out.println(" Same as 3 controllers but costs more");
System.out.println();
System.out.println("CORRECT: odd number of controllers");
System.out.println(" 3 controllers: majority=2, tolerates 1 failure");
System.out.println(" 5 controllers: majority=3, tolerates 2 failures");
}
}
WRONG: even controllers (4) = same fault tolerance as 3 but higher cost
CORRECT: odd controllers (3 or 5) = optimal cost/fault tolerance ratio
What should a senior engineer know about KRaft?
The durability and fault tolerance principles of the controller quorum mirror those of data replication: a quorum of three controllers tolerates one failure, a quorum of five tolerates two. Controller nodes should be on reliable storage and in different availability zones.
package com.bank.senior;
public class KRaftSeniorInsights {
public static void main(String[] args) {
System.out.println("Senior KRaft Insights:");
System.out.println("1. Controller quorum: 3 (prod) or 5 (large/critical)");
System.out.println("2. Always odd number (Raft majority rule)");
System.out.println("3. Controllers in different AZs (rack-awareness)");
System.out.println("4. Reliable storage for metadata log");
System.out.println("5. Combined mode for dev, separate for production");
System.out.println("6. Monitor: active controller health");
System.out.println("7. Monitor: metadata log lag on followers");
System.out.println("8. Test controller failover regularly");
}
}
Expected output:
Senior insights: odd quorum, cross-AZ placement, monitor controller health
Visual Overview
KRaft Architecture =================== ZooKeeper Mode (before 4.0): ┌──────────┐ ┌──────────────────┐ │ Broker 1 │─────▶│ ZooKeeper │ │ Broker 2 │─────▶│ Ensemble │ │ Broker 3 │─────▶│ (external system)│ └──────────┘ └──────────────────┘ Two systems to operate KRaft Mode (Kafka 4.0+): ┌───────────────────────────────────────┐ │ Controller Quorum (Raft) │ │ ┌────────┐ ┌────────┐ ┌────────┐ │ │ │Ctrl 1 │ │Ctrl 2 │ │Ctrl 3 │ │ │ │(ACTIVE)│ │(FOLLOW)│ │(FOLLOW)│ │ │ └────┬───┘ └────────┘ └────────┘ │ │ │ │ │ │ __cluster_metadata topic │ │ │ (replicated across quorum) │ │ │ │ │ ┌────▼───┐ ┌────────┐ ┌────────┐ │ │ │Broker 1│ │Broker 2│ │Broker 3│ │ │ │ follow │ │ follow │ │ follow │ │ │ │metadata│ │metadata│ │metadata│ │ │ └────────┘ └────────┘ └────────┘ │ └───────────────────────────────────────┘ One system. Self-contained. Scalable.
Key Takeaway
Kafka 4.0 fully removed ZooKeeper and replaced it with KRaft, which stores all cluster metadata in an internal replicated Kafka log managed by a controller quorum using the Raft consensus protocol. This eliminates the external ZooKeeper dependency that added operational complexity, created a scalability bottleneck for large partition counts, and slowed down controller failover.
KRaft makes Kafka a single, self-contained system. Controller failover drops from 10-30 seconds with ZooKeeper to 1-3 seconds with Raft election. The cluster can support millions of partitions instead of the practical ~200K limit under ZooKeeper. There is one system to install, secure, upgrade, and monitor instead of two, which shrinks the attack surface and the number of failure modes.
The migration path is important for teams running older ZooKeeper-based clusters: KRaft became production-ready in Kafka 3.3, ZooKeeper was deprecated through 3.x, and 4.0 is KRaft-only. The controller quorum should always be an odd number (3 or 5) for optimal Raft majority-based fault tolerance, deployed across availability zones.
Remember one sentence: KRaft replaces ZooKeeper by storing Kafka's cluster metadata in an internal replicated log managed by a Raft controller quorum, making Kafka simpler, faster, and able to scale to millions of partitions.
---