Definition
- Eager protocol -- a stop-the-world rebalance where every consumer revokes all its partitions, the group recomputes assignments from scratch, and all consumers pause during the shuffle.
- Cooperative-sticky protocol -- an incremental rebalance where consumers keep partitions they already own and only the specific partitions that must move are revoked, usually over two short rebalances.
- RangeAssignor / RoundRobinAssignor -- built-in assignors that use the eager protocol.
- CooperativeStickyAssignor -- the built-in assignor that uses the cooperative-sticky protocol.
- Static group membership -- setting
group.instance.idso a briefly restarting consumer rejoins without triggering any rebalance within the session timeout. - onPartitionsRevoked -- a callback where you commit offsets before partitions are taken away to prevent double-processing.
Imagine National Bank of India reassigning counters among tellers when a new teller joins. Eager rebalancing is like telling every teller to leave their counter and stand in the hallway, then handing out counter assignments again from scratch -- the whole bank stops serving customers during the reshuffle. Cooperative-sticky rebalancing is like saying most of you keep your current counters; only two counters need to move to the new teller. Almost everyone keeps serving customers and only the few affected counters pause briefly.
Understanding the Concepts
What triggers a rebalance?
A rebalance happens whenever group membership or topic metadata changes: a consumer joins, a consumer leaves or crashes, or partitions are added. The protocol decides how disruptive that reassignment is.
package com.bank.consumer;
public class RebalanceTriggers {
public static void main(String[] args) {
System.out.println("Rebalance triggers:");
System.out.println(" 1. Consumer joins the group (new instance or restart)");
System.out.println(" 2. Consumer leaves (shutdown or crash)");
System.out.println(" 3. Consumer misses heartbeat (session timeout)");
System.out.println(" 4. Consumer exceeds max.poll.interval.ms");
System.out.println(" 5. Topic partition count changes");
System.out.println(" 6. Topic is created/deleted matching subscription");
}
}
Rebalance triggers:
1. Consumer joins the group (new instance or restart)
2. Consumer leaves (shutdown or crash)
3. Consumer misses heartbeat (session timeout)
4. Consumer exceeds max.poll.interval.ms
5. Topic partition count changes
6. Topic is created/deleted matching subscription
How does the eager protocol work?
The eager protocol works in one phase. Every consumer first revokes all of its currently owned partitions, which means it stops consuming and commits offsets. Then the group leader computes a brand-new assignment and every consumer picks up its newly assigned partitions. Even a consumer whose partitions do not change still gives them up and reacquires them.
package com.bank.consumer;
import java.util.Properties;
public class EagerConfig {
public static Properties eagerConsumer() {
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092");
props.put("group.id", "settlement-service");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
// Eager assignors -- stop-the-world rebalance
props.put("partition.assignment.strategy",
"org.apache.kafka.clients.consumer.RangeAssignor");
return props;
}
public static void main(String[] args) {
System.out.println("Eager rebalance steps:");
System.out.println(" 1. ALL consumers revoke ALL partitions");
System.out.println(" 2. ALL consumers STOP processing");
System.out.println(" 3. Leader computes new assignment");
System.out.println(" 4. ALL consumers get new assignment");
System.out.println(" 5. ALL consumers resume");
System.out.println(" Impact: entire group pauses");
}
}
Eager rebalance steps:
1. ALL consumers revoke ALL partitions
2. ALL consumers STOP processing
3. Leader computes new assignment
4. ALL consumers get new assignment
5. ALL consumers resume
Impact: entire group pauses
How does the cooperative-sticky protocol work?
The cooperative-sticky protocol works incrementally over two short rebalances. On the first rebalance it computes the desired assignment and only revokes the partitions that need to move. Consumers that keep their partitions never stop. The second short rebalance assigns the freed partitions to their new owners.
package com.bank.consumer;
import java.util.Properties;
public class CooperativeConfig {
public static Properties cooperativeConsumer() {
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092");
props.put("group.id", "settlement-service");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
// Cooperative-sticky -- incremental rebalance
props.put("partition.assignment.strategy",
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
return props;
}
public static void main(String[] args) {
System.out.println("Cooperative rebalance steps:");
System.out.println(" 1. Compute desired assignment");
System.out.println(" 2. Only MOVING partitions are revoked");
System.out.println(" 3. Non-moving consumers KEEP processing");
System.out.println(" 4. Second short rebalance assigns freed partitions");
System.out.println(" Impact: only migrating partitions pause");
}
}
Cooperative rebalance steps:
1. Compute desired assignment
2. Only MOVING partitions are revoked
3. Non-moving consumers KEEP processing
4. Second short rebalance assigns freed partitions
Impact: only migrating partitions pause
How do the two protocols compare? (comparison table)
| Feature | Eager | Cooperative-Sticky |
|---|---|---|
| Revocation scope | All partitions on all consumers | Only partitions that move |
| Processing pause | Entire group stops | Only migrating partitions pause |
| Number of rebalance rounds | 1 | 2 (short) |
| Assignment stability | New assignment from scratch | Sticky -- tries to keep same mapping |
| Suitable for rolling deploys | Poor (each restart stalls group) | Excellent (localized pause) |
| Assignor class | RangeAssignor, RoundRobinAssignor | CooperativeStickyAssignor |
How do you migrate from eager to cooperative?
You cannot mix eager and cooperative assignors in the same group because they disagree on revocation. The migration is a two-phase rolling upgrade.
package com.bank.migration;
public class RebalanceMigration {
public static void main(String[] args) {
System.out.println("Phase 1: Rolling deploy with BOTH assignors");
System.out.println(" partition.assignment.strategy = "
+ "RangeAssignor, CooperativeStickyAssignor");
System.out.println(" All consumers now support both protocols");
System.out.println();
System.out.println("Phase 2: Rolling deploy with ONLY cooperative");
System.out.println(" partition.assignment.strategy = "
+ "CooperativeStickyAssignor");
System.out.println(" Migration complete");
}
}
Phase 1: Rolling deploy with BOTH assignors
partition.assignment.strategy = RangeAssignor, CooperativeStickyAssignor
All consumers now support both protocols
Phase 2: Rolling deploy with ONLY cooperative
partition.assignment.strategy = CooperativeStickyAssignor
Migration complete
How do you handle offsets during rebalance?
Commit offsets in onPartitionsRevoked so no work is double-processed when partitions move to another consumer.
package com.bank.consumer;
import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;
import java.util.Collection;
public class OffsetCommitOnRebalance implements ConsumerRebalanceListener {
private final KafkaConsumer<String, String> consumer;
public OffsetCommitOnRebalance(KafkaConsumer<String, String> consumer) {
this.consumer = consumer;
}
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
System.out.println("Partitions revoked: " + partitions);
consumer.commitSync();
System.out.println("Offsets committed before handoff");
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
System.out.println("Partitions assigned: " + partitions);
}
}
How does static group membership complement cooperative rebalancing?
Setting group.instance.id makes the consumer a static member. If it restarts and rejoins within the session timeout, no rebalance is triggered at all.
package com.bank.consumer;
import java.util.Properties;
public class StaticMembershipConfig {
public static Properties staticConsumer(String instanceId) {
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092");
props.put("group.id", "settlement-service");
props.put("group.instance.id", instanceId);
props.put("session.timeout.ms", "45000");
props.put("partition.assignment.strategy",
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
return props;
}
public static void main(String[] args) {
System.out.println("Static membership + cooperative:");
System.out.println(" Quick restart within 45s -> no rebalance at all");
System.out.println(" Long outage beyond 45s -> cooperative rebalance");
}
}
Static membership + cooperative:
Quick restart within 45s -> no rebalance at all
Long outage beyond 45s -> cooperative rebalance
What should a senior engineer know?
Cooperative-sticky dramatically reduces the impact of routine events like rolling deployments. In a bank running continuous deployments, the eager protocol makes the whole settlement group stall on every consumer restart, while cooperative-sticky confines the pause to just the partitions owned by the restarting consumer. The migration caution is that you cannot flip the whole group at once -- you must do a two-phase rolling upgrade. Static membership with a group instance id eliminates rebalance entirely for quick restarts.
Visual Overview
Eager Rebalance (stop-the-world)
┌─────────┐ ┌─────────┐ ┌─────────┐
│ C1 │ │ C2 │ │ C3 (new)│
│ P0,P1 │ │ P2,P3 │ │ │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
▼ ▼ ▼
REVOKE ALL REVOKE ALL (joins)
ALL STOP ALL STOP
│ │ │
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ C1 │ │ C2 │ │ C3 │
│ P0 │ │ P1,P2 │ │ P3 │
└─────────┘ └─────────┘ └─────────┘
Resume Resume Resume
Cooperative-Sticky Rebalance (incremental)
┌─────────┐ ┌─────────┐ ┌─────────┐
│ C1 │ │ C2 │ │ C3 (new)│
│ P0,P1 │ │ P2,P3 │ │ │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
KEEP P0 KEEP P2 (joins)
REVOKE P1 REVOKE P3
│ │ │
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ C1 │ │ C2 │ │ C3 │
│ P0 │ │ P2 │ │ P1,P3 │
│ RUNNING │ │ RUNNING │ │ assigned│
└─────────┘ └─────────┘ └─────────┘
Key Takeaway
The eager and cooperative-sticky protocols are two ways a consumer group redistributes partitions when membership changes. The eager protocol is stop-the-world: every consumer revokes all partitions, the group recomputes a fresh assignment, and then everyone picks up their new partitions. Even consumers whose partitions do not change still give them up and take them back, so the entire group pauses for the duration.
The cooperative-sticky protocol is incremental. Consumers keep the partitions they already own, and only the specific partitions that need to move are revoked over two short rebalances. At National Bank of India, Pankaj moved the settlement consumers from eager to cooperative-sticky because every rolling deploy produced visible latency spikes from the whole group stopping. Now a deploy only pauses the partitions owned by the restarting consumer.
Static group membership with group.instance.id complements cooperative rebalancing by letting a quickly restarting consumer rejoin without triggering any rebalance at all. The migration from eager to cooperative requires a two-phase rolling upgrade because the assignors disagree on revocation semantics.
Remember one sentence: cooperative-sticky turns a routine consumer restart from a group-wide outage into a localized, barely noticeable event.
---