Definition
- Eager rebalance -- the default protocol where on any membership change every consumer revokes all partitions and re-joins, causing a stop-the-world pause.
- 2N rebalances -- a rolling deploy of N pods triggers roughly 2N stop-the-world rebalances (one per stop, one per start), each pausing the entire group.
- Cooperative sticky assignor -- consumers only revoke the specific partitions that actually need to move, keeping all other partitions consuming throughout.
- Static group membership -- giving each pod a stable
group.instance.idlets the coordinator skip the rebalance when a pod restarts within the session timeout. - Stacking fixes -- cooperative sticky removes most of the cost from unavoidable rebalances; static membership eliminates rebalances for quick restarts.
Imagine National Bank of India has ten tellers at ten counters, and customers are split by account number. When the manager sends each teller for a coffee break one at a time, with the old rule every single time one teller leaves or comes back, all ten tellers close their counters, huddle in the back room, and re-divide the entire customer list from scratch. The fix is a smarter rule where only the affected counter is touched and the other nine keep serving.
Understanding the Concepts
Why does a rolling deployment hurt more than expected?
The core reason is the default eager rebalance protocol. On any membership change, every consumer revokes all of its partitions, the coordinator recomputes the assignment, and every consumer re-joins. During a rolling deployment, each pod stop and start triggers a separate membership change, so a group of N pods pays roughly 2N stop-the-world rebalances.
package com.bank.kafka;
public class RebalanceCostCalculator {
public static void main(String[] args) {
int pods = 6;
int rebalancesPerPod = 2; // one for stop, one for start
int total = pods * rebalancesPerPod;
int pausePerRebalanceMs = 5000; // typical eager rebalance
int totalPauseMs = total * pausePerRebalanceMs;
System.out.println("Rolling deploy of " + pods + " pods:");
System.out.println(" Eager rebalances: " + total);
System.out.println(" Pause per rebalance: " + pausePerRebalanceMs + "ms");
System.out.println(" Total group pause: " + totalPauseMs + "ms (" + totalPauseMs/1000 + "s)");
System.out.println(" During each pause: NO partition in the group is consumed");
}
}
Expected output:
Rolling deploy of 6 pods:
Eager rebalances: 12
Pause per rebalance: 5000ms
Total group pause: 60000ms (60s)
During each pause: NO partition in the group is consumed
How does the cooperative sticky assignor fix this?
With cooperative rebalancing, consumers only revoke the specific partitions that actually need to move, and they keep consuming everything else throughout.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;
public class CooperativeStickyConfig {
public static Properties create() {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "settlement-service");
// Switch from eager to cooperative sticky
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
System.out.println("Cooperative sticky assignor enabled");
System.out.println(" Only partitions that need to move are paused");
System.out.println(" Other partitions keep consuming throughout");
return props;
}
public static void main(String[] args) {
create();
}
}
Expected output:
Cooperative sticky assignor enabled
Only partitions that need to move are paused
Other partitions keep consuming throughout
How does static group membership eliminate rebalances?
With a static member, you give each pod a stable identity via group.instance.id. When that pod restarts within session.timeout.ms, the coordinator recognizes it as the same member returning and skips the rebalance entirely.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;
public class StaticMembershipConfig {
public static Properties create(String podOrdinal) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "settlement-service");
// Static membership: stable identity per pod
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG,
"settlement-consumer-" + podOrdinal);
// Session timeout must exceed pod restart time
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "60000"); // 60s
// Cooperative sticky for remaining rebalances
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
System.out.println("Static member: settlement-consumer-" + podOrdinal);
System.out.println(" Quick restart -> zero rebalances");
System.out.println(" Session timeout: 60s (exceeds restart time)");
return props;
}
public static void main(String[] args) {
// In Kubernetes, derive ordinal from StatefulSet pod name
create("3"); // settlement-consumer-3
}
}
Expected output:
Static member: settlement-consumer-3
Quick restart -> zero rebalances
Session timeout: 60s (exceeds restart time)
How do eager and cooperative rebalancing compare?
| Aspect | Eager (Default) | Cooperative Sticky |
|---|---|---|
| On membership change | All consumers revoke all partitions | Only affected partitions revoke |
| Group pause | Full stop-the-world | Only moved partitions pause |
| Rolling deploy cost | 2N full pauses | 2N minimal pauses |
| Partition affinity | Partitions reshuffled | Returning pod gets old partitions |
| Cache warming | Lost on every rebalance | Preserved (sticky) |
package com.bank.kafka;
public class RebalanceComparison {
public static void main(String[] args) {
System.out.println("6-pod rolling deploy comparison:");
System.out.println();
System.out.println("Eager rebalance:");
System.out.println(" 12 stop-the-world pauses");
System.out.println(" All 12 partitions paused each time");
System.out.println(" Caches cold after every rebalance");
System.out.println();
System.out.println("Cooperative sticky + static membership:");
System.out.println(" 0 rebalances (pod restarts within session timeout)");
System.out.println(" Partitions never paused");
System.out.println(" Caches stay warm (sticky assignment)");
}
}
Expected output:
6-pod rolling deploy comparison:
Eager rebalance:
12 stop-the-world pauses
All 12 partitions paused each time
Caches cold after every rebalance
Cooperative sticky + static membership:
0 rebalances (pod restarts within session timeout)
Partitions never paused
Caches stay warm (sticky assignment)
How do you ensure graceful shutdown?
Make sure the pod's preStop hook calls consumer.close() so the consumer leaves the group gracefully rather than timing out.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.KafkaConsumer;
public class GracefulShutdown {
public static void registerShutdownHook(KafkaConsumer<String, String> consumer) {
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
System.out.println("Graceful shutdown: closing consumer");
consumer.close(); // Leaves group cleanly
System.out.println("Consumer closed -- group notified immediately");
}));
}
}
What about Kubernetes StatefulSet configuration?
In Kubernetes, derive the pod ordinal from the StatefulSet pod name for the group.instance.id.
package com.bank.kafka;
public class KubernetesOrdinal {
public static String getPodOrdinal() {
String hostname = System.getenv("HOSTNAME"); // e.g., settlement-2
if (hostname != null && hostname.contains("-")) {
String ordinal = hostname.substring(hostname.lastIndexOf('-') + 1);
System.out.println("Pod hostname: " + hostname);
System.out.println("Ordinal: " + ordinal);
return ordinal;
}
return "0";
}
public static void main(String[] args) {
String ordinal = getPodOrdinal();
String instanceId = "settlement-consumer-" + ordinal;
System.out.println("group.instance.id: " + instanceId);
}
}
Expected output:
Pod hostname: settlement-2
Ordinal: 2
group.instance.id: settlement-consumer-2
What are common mistakes when fixing rebalance storms?
The most common mistake is enabling cooperative sticky but forgetting to set a stable group.instance.id, so every restart still triggers a rebalance. Another error is setting session.timeout.ms shorter than the pod restart time, which causes the coordinator to evict the static member before it returns. A third mistake is mixing eager and cooperative assignors in the same group during migration, which causes a protocol mismatch.
package com.bank.kafka;
public class RebalanceMistakes {
public static void main(String[] args) {
System.out.println("WRONG: cooperative sticky without static membership");
System.out.println(" partition.assignment.strategy=CooperativeStickyAssignor");
System.out.println(" // No group.instance.id -> still triggers rebalance");
System.out.println();
System.out.println("WRONG: session timeout too short");
System.out.println(" group.instance.id=settlement-1");
System.out.println(" session.timeout.ms=10000 // Pod takes 30s to restart");
System.out.println(" // Coordinator evicts before pod returns!");
System.out.println();
System.out.println("CORRECT: cooperative + static + generous timeout");
System.out.println(" CooperativeStickyAssignor + group.instance.id");
System.out.println(" session.timeout.ms=60000 (exceeds restart time)");
}
}
Expected output:
WRONG: cooperative sticky without static membership
partition.assignment.strategy=CooperativeStickyAssignor
// No group.instance.id -> still triggers rebalance
WRONG: session timeout too short
group.instance.id=settlement-1
session.timeout.ms=10000 // Pod takes 30s to restart
// Coordinator evicts before pod returns!
CORRECT: cooperative + static + generous timeout
CooperativeStickyAssignor + group.instance.id
session.timeout.ms=60000 (exceeds restart time)
What should a senior engineer know?
These two fixes stack. Static membership removes rebalances for quick restarts, and the cooperative assignor minimizes the cost of rebalances that remain. The sticky assignor returns a pod to its old partitions, so warmed caches are preserved. Tune session.timeout.ms to comfortably exceed your pod's restart time. Measure the rebalance count during a deploy before and after the fix.
Visual Overview
Eager vs Cooperative Rolling Deploy
====================================
Eager Rebalance (default):
─────────────────────────
Pod 1 stops → ALL consumers stop, revoke ALL partitions
↓ reassign ↓ resume
Pod 1 starts → ALL consumers stop, revoke ALL partitions
↓ reassign ↓ resume
Pod 2 stops → ALL consumers stop, revoke ALL partitions
↓ reassign ↓ resume
...repeat for every pod...
Total: 2N stop-the-world pauses
Cooperative Sticky + Static Membership:
───────────────────────────────────────
Pod 1 stops → static member, session timeout running
Pod 1 starts → same instance.id, coordinator skips rebalance
↓ zero pause
Pod 2 stops → static member, session timeout running
Pod 2 starts → same instance.id, coordinator skips rebalance
↓ zero pause
...repeat for every pod...
Total: 0 rebalances (if restart < session timeout)
Key Takeaway
A rolling deployment hurts far more than expected because the default eager rebalance protocol is stop-the-world. Each pod stop and start triggers a separate rebalance where every consumer revokes all partitions, so a deploy of N pods costs roughly 2N full group pauses during which no partition is consumed.
The fix has two parts that stack. First, the cooperative sticky assignor ensures that only partitions needing to move are paused while everything else keeps consuming. Second, static group membership with a stable instance id per pod lets the coordinator skip the rebalance entirely when a pod restarts quickly.
In my National Bank of India settlement service, six consumer pods on twelve partitions caused visible lag spikes on every deploy. After switching to cooperative sticky plus static membership, deploys caused zero rebalances and latency stayed flat throughout.
Remember one sentence: switch to cooperative sticky assignor and static group membership so rolling deploys cause zero stop-the-world rebalances and partitions are never paused.
---