Definition
- Already committed -- Kafka does not redeliver committed messages to the same group; the fix is to reset offsets or use a new group.id.
- Latest reset -- a brand-new group with
auto.offset.reset=latestsilently skips all pre-existing messages; setearliestfor the backlog. - Idle consumers -- more consumers than partitions means some sit idle because a partition is owned by exactly one consumer per group.
- Stuck rebalance -- members keep joining and leaving, so the group never settles into stable assignment and processes little.
- Diagnostic tool --
kafka-consumer-groups --describeshows per-partition current offset, log-end offset, lag, member assignment, and group state.
Imagine transfer slips piling up in a tray at National Bank of India while the clerk insists they are working fine. Maybe the clerk already stamped those slips as done earlier. Maybe a new clerk was told to only handle slips arriving after they sat down, so the old pile is invisible. Maybe there are five clerks but only three trays, so two clerks sit idle. Or maybe the team is reshuffling who owns which tray and nobody is actually working yet.
Understanding the Concepts
What are the four classic causes of unconsumed messages with no errors?
When a consumer runs cleanly but messages appear unconsumed, the cause is almost always about offsets, group membership, or partition assignment rather than a crash.
Cause 1: Group already committed those offsets
Kafka does not redeliver committed messages to the same group. If someone ran the consumer earlier and it committed through the latest offset, a fresh run sees nothing new.
# Diagnose: check if current offset equals log-end offset
kafka-consumer-groups.sh --describe --group settlement-service \
--bootstrap-server kafka-1:9092
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
settlement-service transfers 0 45100 45100 0
settlement-service transfers 1 44950 44950 0
# LAG = 0 everywhere: group already consumed everything
# Fix: reset offsets to earliest to reprocess
kafka-consumer-groups.sh --reset-offsets --to-earliest --execute \
--group settlement-service --topic transfers \
--bootstrap-server kafka-1:9092
# Alternative: use a completely new group.id
Cause 2: auto.offset.reset=latest on a new group
For a group with no committed offset, latest starts at the end, silently skipping all existing messages. The app waits for new records while the backlog sits untouched.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;
public class LatestResetTrap {
// WRONG: new group with latest -- skips entire backlog
public static Properties wrongConfig() {
Properties props = new Properties();
props.put(ConsumerConfig.GROUP_ID_CONFIG, "brand-new-group");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
System.out.println("WRONG: new group with 'latest' skips all existing messages");
return props;
}
// CORRECT: use earliest to read the backlog
public static Properties correctConfig() {
Properties props = new Properties();
props.put(ConsumerConfig.GROUP_ID_CONFIG, "brand-new-group");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
System.out.println("CORRECT: new group with 'earliest' reads entire backlog");
return props;
}
public static void main(String[] args) {
wrongConfig();
correctConfig();
}
}
Expected output:
WRONG: new group with 'latest' skips all existing messages
CORRECT: new group with 'earliest' reads entire backlog
Cause 3: More consumers than partitions
Within a consumer group, each partition is assigned to exactly one consumer. If a topic has three partitions and you run five consumers, two get no partitions and sit idle.
package com.bank.kafka;
public class IdleConsumerDemo {
public static void main(String[] args) {
int partitions = 3;
int consumers = 5;
int active = Math.min(partitions, consumers);
int idle = consumers - active;
System.out.println("Topic partitions: " + partitions);
System.out.println("Consumers in group: " + consumers);
System.out.println("Active consumers: " + active);
System.out.println("Idle consumers: " + idle);
System.out.println("Fix: reduce consumers to " + partitions
+ " or add partitions to " + consumers);
}
}
Expected output:
Topic partitions: 3
Consumers in group: 5
Active consumers: 3
Idle consumers: 2
Fix: reduce consumers to 3 or add partitions to 5
Cause 4: Group stuck rebalancing
If members keep joining and leaving, perhaps because of a blown poll interval or flapping instances, the group spends its time rebalancing and never settles into stable assignment.
# Check group state
kafka-consumer-groups.sh --describe --group settlement-service \
--bootstrap-server kafka-1:9092
Consumer group 'settlement-service' is rebalancing.
# No stable member assignment -- group is stuck
# Fix: stabilize membership, fix slow processing, stop flapping
How do you use kafka-consumer-groups --describe to diagnose?
The describe command shows per-partition current offset, log-end offset, lag, and member assignment. Reading this tells you the cause directly.
| Observation | Likely Cause |
|---|---|
| LAG=0 everywhere | Already committed (Cause 1) |
| LAG=high, CURRENT-OFFSET=0 | Latest reset skipped backlog (Cause 2) |
| Some partitions have no CONSUMER-ID | Idle consumers or rebalance (Cause 3/4) |
| Group state: PreparingRebalance | Stuck rebalance (Cause 4) |
# Full diagnostic command
kafka-consumer-groups.sh --describe --group settlement-service \
--bootstrap-server kafka-1:9092
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
settlement-service transfers 0 45100 45100 0 consumer-1-abc123 /10.0.1.5
settlement-service transfers 1 0 44950 44950 consumer-2-def456 /10.0.1.6
settlement-service transfers 2 0 43200 43200 - -
# Partition 0: LAG=0, already consumed
# Partition 1: LAG=high, CURRENT-OFFSET=0 -- reset to latest on new group?
# Partition 2: no consumer assigned -- idle or rebalancing
What about topic name typos?
A subtle fifth cause is that the consumer is subscribed to the wrong topic name. A typo produces the same symptom of no errors but no messages, because the consumer happily subscribes to a non-existent or empty topic.
package com.bank.kafka;
public class TopicTypoDemo {
public static void main(String[] args) {
String intended = "transfers";
String actual = "transfer"; // Typo: missing 's'
System.out.println("Intended topic: " + intended);
System.out.println("Actual subscription: " + actual);
System.out.println("Result: consumer subscribes to empty/nonexistent topic");
System.out.println("Fix: verify topic name matches exactly");
}
}
Expected output:
Intended topic: transfers
Actual subscription: transfer
Result: consumer subscribes to empty/nonexistent topic
Fix: verify topic name matches exactly
What should a senior engineer know?
Always check the group state field: Stable, PreparingRebalance, or CompletingRebalance. Confirm the consumer is subscribed to the topic you think it is. For Cause 2, the setting does nothing when a committed offset exists, so changing it will not help an existing group that already committed.
Visual Overview
Diagnostic Decision Tree
=========================
Consumer running, no errors, messages unconsumed
│
▼
kafka-consumer-groups --describe
│
┌──────────┴──────────┐
│ │
LAG = 0 LAG > 0
everywhere on some partitions
│ │
▼ ▼
Cause 1: Already ┌─────────────────┐
committed │ CURRENT-OFFSET=0 │
Fix: reset offsets │ on high-lag │
or new group.id │ partitions? │
└────┬────────────┘
yes │ no
│ │
┌─────────┘ ┌───┴──────────┐
▼ │ CONSUMER-ID │
Cause 2: latest │ missing? │
reset skipped └──┬────────────┘
backlog yes│ no
Fix: set earliest │ │
▼ ▼
Cause 3/4: Check:
Idle or - Topic name typo?
rebalancing - Processing too slow?
Check group
state field
Key Takeaway
When a consumer runs cleanly but messages sit unconsumed, I look at offsets, membership, and assignment, not at a crash. The four classic causes are: the group already committed those offsets, a brand-new group with latest reset skipped the backlog, more consumers than partitions leaving some idle, or a stuck rebalance.
The diagnostic tool is kafka-consumer-groups --describe, which shows per-partition current offset, log-end offset, lag, and member assignment. LAG of zero means the group already consumed everything. High lag with a current offset of zero points to a latest reset. Missing consumer IDs point to idle consumers or a rebalance.
In my National Bank of India work, a new reconciliation consumer appeared to process nothing. The describe output showed large lag with offset zero and latest reset on a fresh group. We changed to earliest, reset offsets, and the backlog drained immediately.
Remember one sentence: use kafka-consumer-groups --describe to check lag, current offset, and member assignment, because unconsumed messages with no errors is always an offset, membership, or assignment problem.
---