Definition
- Load balancing – distributing a topic's partitions roughly evenly across the consumers in a group so each consumer handles a similar share of the work.
- Rebalance – the process by which the group coordinator redistributes partition ownership when group membership changes, triggered by a consumer joining, leaving, or failing.
- Heartbeat – a periodic signal sent by each consumer on a background thread every
heartbeat.interval.msto prove it is alive to the coordinator. - Session timeout – the
session.timeout.mssetting that defines how long the coordinator waits without a heartbeat before declaring a consumer dead. max.poll.interval.ms– the maximum time allowed between calls topoll(); exceeding it causes the coordinator to remove the consumer as unhealthy.- Static group membership – assigning a
group.instance.idto a consumer so it can rejoin after a brief restart and reclaim its partitions without triggering a rebalance.
Picture a team of clerks at National Bank of India sharing stacks of transaction forms. The supervisor splits the stacks evenly so everyone has about the same workload. If a new clerk joins, the supervisor takes a few stacks from the busy clerks and hands them over. If a clerk goes home sick without telling anyone and stops responding, the supervisor notices the silence, takes that clerk's stacks, and shares them among the remaining clerks so no form is left unprocessed. This reshuffling is the rebalance.
Understanding the Concepts
How does Kafka balance load across consumers?
Load balancing in Kafka means fair distribution of partitions. The group coordinator and the elected group leader compute an assignment using the configured partition.assignment.strategy. The goal is that each consumer owns about the same number of partitions so the processing load is even.
package com.bank.config;
import java.util.Properties;
public class BalancedConsumerConfig {
public static Properties createConfig() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "settlement-group");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
// Cooperative sticky for minimal rebalance disruption
props.put("partition.assignment.strategy",
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
// Heartbeat and session timeout tuning
props.put("heartbeat.interval.ms", "3000");
props.put("session.timeout.ms", "10000");
props.put("max.poll.interval.ms", "300000");
System.out.println("Consumer configured for balanced load distribution");
return props;
}
public static void main(String[] args) {
createConfig();
}
}
Expected output:
Consumer configured for balanced load distribution
With 12 partitions and 4 consumers:
Consumer A: P0, P1, P2 (3 partitions)
Consumer B: P3, P4, P5 (3 partitions)
Consumer C: P6, P7, P8 (3 partitions)
Consumer D: P9, P10, P11 (3 partitions)
What are the partition assignment strategies?
Kafka offers four built-in strategies. Each strategy divides partitions differently, and the choice affects how stable assignments are during rebalances.
| Strategy | How It Works | Stability | Recommended |
|---|---|---|---|
| RangeAssignor | Contiguous ranges per topic | Low | No |
| RoundRobinAssignor | Spread one by one evenly | Medium | Development |
| StickyAssignor | Minimize movement on rebalance | High | Yes |
| CooperativeStickyAssignor | Sticky + incremental rebalance | Highest | Production |
package com.bank.assignment;
public class AssignmentStrategyComparison {
public static void main(String[] args) {
System.out.println("RangeAssignor (3 consumers, 6 partitions):");
System.out.println(" A: P0, P1 | B: P2, P3 | C: P4, P5");
System.out.println();
System.out.println("RoundRobinAssignor:");
System.out.println(" A: P0, P3 | B: P1, P4 | C: P2, P5");
System.out.println();
System.out.println("CooperativeStickyAssignor (D joins):");
System.out.println(" Before: A:P0,P1 B:P2,P3 C:P4,P5");
System.out.println(" After: A:P0 B:P2,P3 C:P4,P5 D:P1");
System.out.println(" Only P1 moved (minimal disruption)");
}
}
Expected output:
RangeAssignor (3 consumers, 6 partitions):
A: P0, P1 | B: P2, P3 | C: P4, P5
RoundRobinAssignor:
A: P0, P3 | B: P1, P4 | C: P2, P5
CooperativeStickyAssignor (D joins):
Before: A:P0,P1 B:P2,P3 C:P4,P5
After: A:P0 B:P2,P3 C:P4,P5 D:P1
Only P1 moved (minimal disruption)
What triggers a rebalance?
Three events trigger a rebalance. A consumer joining the group (scaling up), a consumer leaving gracefully (clean shutdown), and a consumer failing (crash or stuck processing that causes missed heartbeats).
package com.bank.rebalance;
public class RebalanceTriggers {
public static void main(String[] args) {
System.out.println("Rebalance Triggers:");
System.out.println();
System.out.println("1. Consumer JOINS (scaling up):");
System.out.println(" New instance starts -> coordinator redistributes");
System.out.println(" Some partitions move from existing to new consumer");
System.out.println();
System.out.println("2. Consumer LEAVES (graceful shutdown):");
System.out.println(" Instance sends LeaveGroup -> immediate reassignment");
System.out.println(" Fast because coordinator knows immediately");
System.out.println();
System.out.println("3. Consumer FAILS (crash or stuck):");
System.out.println(" Heartbeats stop -> coordinator waits session.timeout.ms");
System.out.println(" After timeout -> declares dead -> reassigns partitions");
System.out.println(" Slower because of timeout wait");
}
}
Expected output:
Rebalance Triggers:
1. Consumer JOINS: new instance -> coordinator redistributes
2. Consumer LEAVES: sends LeaveGroup -> immediate reassignment
3. Consumer FAILS: heartbeats stop -> wait session.timeout.ms -> reassign
How does heartbeat and session timeout work?
Each consumer sends a heartbeat on a background thread every heartbeat.interval.ms. If the coordinator does not hear from a consumer within session.timeout.ms, it declares that consumer dead and starts a rebalance. There is also max.poll.interval.ms: if a consumer takes too long between calls to poll(), the coordinator removes it.
package com.bank.config;
import java.util.Properties;
public class HeartbeatConfig {
public static Properties configureTimeouts() {
Properties props = new Properties();
// Heartbeat sent every 3 seconds
props.put("heartbeat.interval.ms", "3000");
// If no heartbeat for 10 seconds, consumer is dead
props.put("session.timeout.ms", "10000");
// If no poll() call for 5 minutes, consumer is stuck
props.put("max.poll.interval.ms", "300000");
System.out.println("Heartbeat: every 3s");
System.out.println("Session timeout: 10s (dead if no heartbeat)");
System.out.println("Max poll interval: 300s (stuck if no poll)");
System.out.println();
System.out.println("Rule: heartbeat.interval.ms < session.timeout.ms / 3");
return props;
}
public static void main(String[] args) {
configureTimeouts();
}
}
Expected output:
Heartbeat: every 3s
Session timeout: 10s (dead if no heartbeat)
Max poll interval: 300s (stuck if no poll)
Rule: heartbeat.interval.ms < session.timeout.ms / 3
How does eager vs cooperative rebalancing differ?
In the classic eager protocol, every consumer revokes all its partitions, the leader computes a fresh assignment, and consumers receive new partitions. This causes a stop-the-world pause. The cooperative incremental protocol only revokes the specific partitions that need to move, keeping the rest flowing.
package com.bank.rebalance;
public class EagerVsCooperativeRebalance {
public static void main(String[] args) {
System.out.println("Eager Rebalance (stop-the-world):");
System.out.println(" Step 1: ALL consumers stop processing");
System.out.println(" Step 2: ALL consumers revoke ALL partitions");
System.out.println(" Step 3: Leader computes new assignment");
System.out.println(" Step 4: ALL consumers receive new assignment");
System.out.println(" Step 5: Processing resumes");
System.out.println(" Impact: FULL PAUSE for entire group");
System.out.println();
System.out.println("Cooperative Rebalance (incremental):");
System.out.println(" Step 1: Only affected consumers revoke partitions");
System.out.println(" Step 2: Other consumers KEEP processing");
System.out.println(" Step 3: Revoked partitions assigned to new consumer");
System.out.println(" Impact: MINIMAL PAUSE, most partitions unaffected");
}
}
Expected output:
Eager Rebalance: FULL PAUSE for entire group
Cooperative Rebalance: MINIMAL PAUSE, most partitions unaffected
What is static group membership?
Static group membership via group.instance.id avoids rebalances during brief restarts. When a consumer with a static instance id restarts within session.timeout.ms, it reclaims its previous partitions without triggering a rebalance. This is essential for rolling deployments.
package com.bank.config;
import java.util.Properties;
public class StaticMembershipConfig {
public static Properties createConfig(String instanceId) {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "settlement-group");
// Static membership: no rebalance on brief restart
props.put("group.instance.id", instanceId);
props.put("session.timeout.ms", "30000");
System.out.println("Static member: " + instanceId);
System.out.println("If restarted within 30s, reclaims partitions");
return props;
}
public static void main(String[] args) {
createConfig("settlement-instance-1");
}
}
Expected output:
Static member: settlement-instance-1
If restarted within 30s, reclaims partitions
Without static membership:
Restart -> LeaveGroup -> rebalance -> JoinGroup -> rebalance again
Two rebalances per restart!
With static membership:
Restart -> rejoin with same instance id -> reclaim partitions
Zero rebalances if restart < session.timeout.ms
What are common mistakes with rebalancing?
The most common mistake is not tuning max.poll.interval.ms for slow consumers. If processing takes longer than this interval, the coordinator removes the consumer, triggering an unnecessary rebalance and causing duplicate processing.
package com.bank.mistakes;
// WRONG: Default max.poll.interval.ms (5 minutes) with slow processing.
// If a batch takes 6 minutes, the consumer is evicted and rebalanced.
public class WrongPollInterval {
public static void main(String[] args) {
System.out.println("WRONG: max.poll.interval.ms = 300000 (5 min)");
System.out.println("Processing a batch of transactions takes 6 min");
System.out.println("Result: consumer evicted, rebalance triggered");
System.out.println("Partitions reassigned, messages reprocessed");
}
}
package com.bank.mistakes;
// CORRECT: Set max.poll.interval.ms > max expected processing time.
// Or reduce max.poll.records to process smaller batches faster.
public class CorrectPollInterval {
public static void main(String[] args) {
System.out.println("CORRECT: max.poll.interval.ms = 600000 (10 min)");
System.out.println("Or: max.poll.records = 50 (smaller batches)");
System.out.println("Result: consumer stays alive, no false eviction");
}
}
WRONG: slow processing exceeds max.poll.interval.ms -> false eviction
CORRECT: increase interval or reduce batch size -> stable group
What should a senior engineer know about rebalancing?
Rebalances are necessary but costly, so you minimize unnecessary ones. Use the cooperative sticky assignor, tune poll intervals, use static membership for rolling restarts, and monitor rebalance frequency. Frequent rebalances directly cause consumer lag and duplicate processing.
package com.bank.senior;
public class RebalanceBestPractices {
public static void main(String[] args) {
System.out.println("Senior Rebalancing Best Practices:");
System.out.println("1. Use CooperativeStickyAssignor (no stop-the-world)");
System.out.println("2. Set max.poll.interval.ms > max processing time");
System.out.println("3. Use static group membership for rolling deploys");
System.out.println("4. Set max.poll.records to control batch size");
System.out.println("5. Monitor rebalance frequency (high = problem)");
System.out.println("6. Commit offsets in onPartitionsRevoked callback");
System.out.println("7. Investigate root cause of frequent rebalances");
}
}
Expected output:
Senior Rebalancing Best Practices:
1. Use CooperativeStickyAssignor
2. Set max.poll.interval.ms > max processing time
3. Use static group membership for rolling deploys
4. Set max.poll.records to control batch size
5. Monitor rebalance frequency
6. Commit offsets in onPartitionsRevoked callback
7. Investigate root cause of frequent rebalances
Visual Overview
Rebalance Flow =============== Before (3 consumers, 6 partitions): ┌──────────┐ ┌──────────┐ ┌──────────┐ │Consumer A│ │Consumer B│ │Consumer C│ │ P0, P1 │ │ P2, P3 │ │ P4, P5 │ └──────────┘ └──────────┘ └──────────┘ Consumer D joins: ┌──────────────────────────────────────────┐ │ Coordinator detects new member │ │ Triggers rebalance │ │ │ │ Cooperative: only P1 moves │ │ A keeps P0, B keeps P2,P3, C keeps P4,P5│ │ D receives P1 │ └──────────────────────────────────────────┘ After (4 consumers, 6 partitions): ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │Consumer A│ │Consumer B│ │Consumer C│ │Consumer D│ │ P0 │ │ P2, P3 │ │ P4, P5 │ │ P1 │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ Consumer B fails (heartbeat stops): ┌──────────────────────────────────────────┐ │ Coordinator waits session.timeout.ms │ │ No heartbeat -> declares B dead │ │ Redistributes P2, P3 to survivors │ └──────────────────────────────────────────┘ After failure (3 consumers, 6 partitions): ┌──────────┐ ┌──────────┐ ┌──────────┐ │Consumer A│ │Consumer C│ │Consumer D│ │ P0, P2 │ │ P4, P5 │ │ P1, P3 │ └──────────┘ └──────────┘ └──────────┘
Key Takeaway
Kafka balances load across a consumer group by distributing partitions evenly among consumers using a configurable strategy. The CooperativeStickyAssignor is the recommended default because it minimizes partition movement during rebalances and avoids the stop-the-world pause of the eager protocol. At National Bank of India, the settlement-group uses this strategy so that scaling up or down moves only the partitions that need to change owner, keeping most processing uninterrupted.
Three events trigger a rebalance: a consumer joining (scaling up), a consumer leaving gracefully (clean shutdown with LeaveGroup), and a consumer failing (heartbeats stop within session.timeout.ms). During any rebalance, the consumer giving up a partition should commit its offset first so the new owner resumes at exactly the right record. Static group membership avoids unnecessary rebalances during rolling deployments by letting a restarting consumer reclaim its previous partitions.
The senior insight is that rebalances are necessary but costly. Frequent rebalances cause consumer lag and duplicate processing. You minimize them by tuning max.poll.interval.ms to avoid false evictions of slow consumers, using static membership for brief restarts, and monitoring rebalance frequency as a production health metric.
Remember one sentence: Kafka rebalances partitions when consumers join, leave, or fail, and the cooperative sticky assignor minimizes disruption by moving only the partitions that need to change owner.
---