Definition
- Consumer group – a set of consumer instances sharing the same
group.idthat cooperate to consume a topic's partitions, with each partition assigned to exactly one consumer in the group. - Group coordinator – a broker role that manages consumer group membership, receives heartbeats, and triggers rebalances when the group changes.
- Partition assignment strategy – the algorithm that divides partitions among consumers in a group, including
RangeAssignor,RoundRobinAssignor,StickyAssignor, andCooperativeStickyAssignor. - Rebalance – the process of redistributing partition ownership when a consumer joins, leaves, or fails, temporarily pausing consumption.
__consumer_offsets– the internal Kafka topic that stores committed offsets per consumer group per partition, enabling resume-from-where-you-left-off behavior.
Think of a consumer group as a team of clerks at National Bank of India who share one job, such as processing transactions. The team lead (group coordinator) hands each clerk a set of ledger drawers (partitions) so no two clerks on the same team touch the same drawer, and together they get through all the work once. A different team, say the fraud team, has its own lead and its own copy of all the drawers to read independently. Same records, two teams, each doing their own job without stepping on each other.
Understanding the Concepts
What is a consumer group?
A consumer group is identified by its group id, set with group.id in consumer configuration or spring.kafka.consumer.group-id in Spring Boot. All consumers with the same group id belong to one group and cooperate. The purpose is load sharing: if a topic has twelve partitions and the group has four consumers, Kafka gives each consumer three partitions, so the group processes four times faster than a single consumer while still processing each record exactly once within that group.
package com.bank.consumer;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class SettlementConsumer {
// All instances of this service share group-id = settlement-group
// Kafka assigns partitions across instances automatically
@KafkaListener(topics = "transactions", groupId = "settlement-group")
public void processTransaction(String event) {
System.out.println("[settlement-group] Processing: " + event);
}
}
Expected output (3 instances, 6 partitions):
Instance A [settlement-group]: P0, P1 -> processes events from these partitions
Instance B [settlement-group]: P2, P3 -> processes events from these partitions
Instance C [settlement-group]: P4, P5 -> processes events from these partitions
Each event processed ONCE by the group.
How does partition assignment work within a group?
Assignment within a group is performed by the group coordinator (a broker role) together with a group leader (one of the consumers chosen to compute the assignment). The actual division follows a partition assignment strategy configured via partition.assignment.strategy.
| Strategy | Behavior | Best For |
|---|---|---|
| RangeAssignor | Contiguous ranges per topic | Simple, but can cause imbalance |
| RoundRobinAssignor | Spreads partitions one by one | Even distribution |
| StickyAssignor | Keeps assignments stable across rebalances | Minimize disruption |
| CooperativeStickyAssignor | Incremental rebalance, no stop-the-world | Production default |
package com.bank.config;
import java.util.Properties;
public class ConsumerGroupConfig {
public static Properties createConfig() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "settlement-group");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
// Cooperative sticky: best for production
props.put("partition.assignment.strategy",
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
System.out.println("Consumer configured with CooperativeStickyAssignor");
return props;
}
public static void main(String[] args) {
createConfig();
}
}
Expected output:
Consumer configured with CooperativeStickyAssignor
How does partition assignment work across groups?
Across groups, assignment is completely independent. Each group has its own group id, its own coordinator state, and its own committed offsets stored in __consumer_offsets keyed by group id. So the audit group and the fraud group each receive all partitions and track their own progress. One group falling behind has zero effect on another group.
package com.bank.consumer;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class IndependentGroupConsumers {
@KafkaListener(topics = "transactions", groupId = "settlement-group")
public void settlementProcessor(String event) {
System.out.println("[settlement] " + event);
}
@KafkaListener(topics = "transactions", groupId = "fraud-group")
public void fraudDetector(String event) {
System.out.println("[fraud] " + event);
}
@KafkaListener(topics = "transactions", groupId = "audit-group")
public void auditLogger(String event) {
System.out.println("[audit] " + event);
}
}
Expected output:
[settlement] {"accountId":"SAV2001","type":"DEBIT",...}
[fraud] {"accountId":"SAV2001","type":"DEBIT",...}
[audit] {"accountId":"SAV2001","type":"DEBIT",...}
-> Same event processed by ALL three groups independently
-> Each group has its own offset tracking
What is the one-partition-to-one-consumer rule?
The core invariant within a group is that one partition is assigned to exactly one consumer. A partition is never shared by two consumers in the same group at the same instant. If there are more consumers than partitions, the surplus consumers get nothing and stay idle. If there are fewer consumers than partitions, some consumers own multiple partitions.
package com.bank.consumer;
public class PartitionConsumerMapping {
public static void main(String[] args) {
int partitions = 6;
for (int consumers = 1; consumers <= 8; consumers++) {
int active = Math.min(consumers, partitions);
int idle = consumers - active;
int partitionsPerConsumer = partitions / active;
int extra = partitions % active;
System.out.println(consumers + " consumers, " + partitions
+ " partitions: " + active + " active, " + idle + " idle"
+ " (" + partitionsPerConsumer + "-"
+ (partitionsPerConsumer + (extra > 0 ? 1 : 0))
+ " partitions each)");
}
}
}
Expected output:
1 consumers, 6 partitions: 1 active, 0 idle (6-6 partitions each)
2 consumers, 6 partitions: 2 active, 0 idle (3-3 partitions each)
3 consumers, 6 partitions: 3 active, 0 idle (2-2 partitions each)
6 consumers, 6 partitions: 6 active, 0 idle (1-1 partitions each)
7 consumers, 6 partitions: 6 active, 1 idle (1-1 partitions each)
8 consumers, 6 partitions: 6 active, 2 idle (1-1 partitions each)
What happens during a rebalance?
When the group membership changes: a consumer starts, crashes, or is added for scaling, the coordinator triggers a rebalance. During a classic eager rebalance, all consumers briefly stop, give up their partitions, and receive new assignments. The cooperative incremental protocol only revokes the partitions that actually need to move, keeping the rest flowing.
package com.bank.rebalance;
import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;
import org.apache.kafka.common.TopicPartition;
import java.util.Collection;
public class BankRebalanceListener implements ConsumerRebalanceListener {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
// Commit offsets before giving up partitions
System.out.println("Partitions revoked: " + partitions);
System.out.println("Committing offsets before handover...");
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
System.out.println("Partitions assigned: " + partitions);
System.out.println("Resuming processing from committed offsets");
}
}
Expected output (consumer C joins):
Partitions revoked: [transactions-4, transactions-5]
Committing offsets before handover...
Partitions assigned: [transactions-4]
Resuming processing from committed offsets
-> Consumer C received transactions-5
What are common mistakes with consumer groups?
The most common mistake is accidentally giving each consumer instance a unique group id, which turns each instance into its own group and causes every message to be processed by every instance instead of once.
package com.bank.mistakes;
import java.util.UUID;
// WRONG: Each instance gets a unique group id.
// Every instance processes every message -> triple debit!
public class WrongGroupIdConfig {
public static void main(String[] args) {
// Bad: unique group id per instance
String groupId = "settlement-" + UUID.randomUUID();
System.out.println("WRONG group id: " + groupId);
System.out.println("Each instance is its own group -> all read all partitions");
System.out.println("Result: Rajesh debited 3 times!");
}
}
package com.bank.mistakes;
// CORRECT: All instances share the same group id.
// Each message processed by exactly one instance.
public class CorrectGroupIdConfig {
public static void main(String[] args) {
String groupId = "settlement-group";
System.out.println("CORRECT group id: " + groupId);
System.out.println("All instances share the group -> partitions divided");
System.out.println("Result: Rajesh debited exactly once.");
}
}
WRONG: unique group id = every instance reads everything = triple debit
CORRECT: shared group id = partitions divided = processed once
How do you monitor consumer group health?
The key metric is consumer lag: the difference between the latest offset in a partition and the committed offset for a group on that partition. High lag means the group is falling behind. You monitor lag using kafka-consumer-groups.sh --describe or tools like Burrow and Grafana.
package com.bank.monitoring;
public class ConsumerLagMonitor {
public static void main(String[] args) {
System.out.println("Consumer Group Lag Report:");
System.out.println("Group: settlement-group");
System.out.println();
System.out.println("Partition | Latest Offset | Committed | Lag");
System.out.println("P0 | 15000 | 14998 | 2");
System.out.println("P1 | 12000 | 12000 | 0");
System.out.println("P2 | 18000 | 17500 | 500");
System.out.println("P3 | 14000 | 14000 | 0");
System.out.println();
System.out.println("ALERT: P2 lag = 500 (threshold: 100)");
System.out.println("Action: Check consumer processing speed on P2");
}
}
Expected output:
Consumer Group Lag Report:
Group: settlement-group
Partition | Latest Offset | Committed | Lag
P0 | 15000 | 14998 | 2
P1 | 12000 | 12000 | 0
P2 | 18000 | 17500 | 500
P3 | 14000 | 14000 | 0
ALERT: P2 lag = 500 (threshold: 100)
What should a senior engineer know about consumer groups?
A senior engineer focuses on capacity planning and offset management. Because a group cannot have more active consumers than partitions, you size partitions for your target parallelism with headroom. You decide the offset commit strategy per group: auto-commit is simplest but risks reprocessing or loss on crashes, while manual commit after processing gives at-least-once semantics.
package com.bank.senior;
public class ConsumerGroupBestPractices {
public static void main(String[] args) {
System.out.println("Senior Consumer Group Best Practices:");
System.out.println("1. Size partitions >= max expected consumers");
System.out.println("2. Use CooperativeStickyAssignor (minimize rebalance)");
System.out.println("3. Disable auto-commit for financial topics");
System.out.println("4. Commit manually AFTER processing");
System.out.println("5. Monitor consumer lag per group per partition");
System.out.println("6. Use static group membership for rolling deploys");
System.out.println("7. Set max.poll.interval.ms > max processing time");
}
}
Expected output:
Senior Consumer Group Best Practices:
1. Size partitions >= max expected consumers
2. Use CooperativeStickyAssignor (minimize rebalance)
3. Disable auto-commit for financial topics
4. Commit manually AFTER processing
5. Monitor consumer lag per group per partition
6. Use static group membership for rolling deploys
7. Set max.poll.interval.ms > max processing time
Visual Overview
Consumer Group: Within vs Across
=================================
Within a Group (settlement-group, 6 partitions):
┌─────────────────────────────────────────────────┐
│ Group: settlement-group │
│ │
│ Consumer A ──▶ P0, P1 (2 partitions) │
│ Consumer B ──▶ P2, P3 (2 partitions) │
│ Consumer C ──▶ P4, P5 (2 partitions) │
│ │
│ Each partition owned by exactly ONE consumer. │
│ Each record processed ONCE by the group. │
└─────────────────────────────────────────────────┘
Across Groups (same topic, different groups):
┌─────────────────────┐ ┌─────────────────────┐
│ settlement-group │ │ fraud-group │
│ A->P0,P1 B->P2,P3 │ │ X->P0,P1 Y->P2,P3 │
│ C->P4,P5 │ │ Z->P4,P5 │
│ Own offsets: 500 │ │ Own offsets: 300 │
└─────────────────────┘ └─────────────────────┘
│ │
│ Same topic data │
│ Independent offsets │
│ No interference │
▼ ▼
Process once Process once
(settlement) (fraud detection)
Key Takeaway
A consumer group is a set of consumer instances that share the same group id and cooperate to consume a topic. The key rule is that within a single group, each partition is assigned to exactly one consumer, so the group processes each record once while spreading the work across instances. At National Bank of India, the settlement-group has three instances that divide six partitions, each handling two, so the group processes transactions three times faster than a single consumer.
Across groups, assignment is completely independent. Each group has its own coordinator state and committed offsets, so the fraud-group and the audit-group both read every partition and track their own progress without interfering with the settlement-group. This is exactly how Kafka supports the publish-subscribe pattern: add a new group and it independently consumes the whole topic.
The assignment strategy and rebalancing mechanics are important for production stability. The CooperativeStickyAssignor minimizes disruption during rebalances by only moving the partitions that actually need to change owner. Monitoring consumer lag per group per partition tells you whether each team is keeping up with the event stream.
Remember one sentence: within a consumer group, each partition is owned by exactly one consumer for load-sharing, and across groups, every group independently reads all partitions for publish-subscribe.
---