Definition
Consumer lag and pipeline health involve several related mechanisms.
- Consumer lag – The gap between the log-end offset (newest message) and the committed offset (last processed), meaning unprocessed backlog per partition.
- Partition rebalancing – When a consumer joins, leaves, or misses heartbeats, the group coordinator reassigns partitions among surviving consumers.
- Back-pressure – A slow downstream stage signals upstream to slow down so buffers do not overflow and the pipeline degrades gracefully.
- Dead-letter queue – A separate topic where poison records are routed after bounded retries so one bad message never blocks its partition.
Keeping the pipeline healthy requires scaling consumers up to the partition count, monitoring lag, making processing idempotent, and handling poison records.
Picture the National Bank of India's transaction counter where slips arrive on a conveyor belt from Kafka. Consumer lag is the growing pile of unprocessed slips when the clerk is too slow. If a clerk goes on break, the remaining clerks reshuffle who handles which tray: that reshuffle is a rebalance, and during it processing briefly pauses. Back-pressure is the slow clerk telling the conveyor "stop feeding me, I am swamped" so slips do not spill on the floor. When one slip is so garbled it keeps jamming the machine, you set it aside in a reject tray (dead-letter queue) so the rest keep flowing.
Understanding the Concepts
What is consumer lag?
Consumer lag is the most important health signal for a streaming pipeline. Each Kafka partition has a log-end offset (the newest message) and each consumer group commits the offset it has processed. Lag is the difference. At the National Bank of India, steadily growing lag means fraud alerts arrive minutes late.
# Monitor consumer lag using kafka-consumer-groups CLI
# Run from the bank's monitoring server
# Check lag for the transaction-processor consumer group
kafka-consumer-groups.sh \
--bootstrap-server kafka-broker:9092 \
--describe \
--group transaction-processor
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
transaction-processor bank-transactions 0 4500 5000 500
transaction-processor bank-transactions 1 4800 5000 200
transaction-processor bank-transactions 2 3200 5000 1800
Partition 2 has 1800 messages of lag, which needs investigation.
Why does lag grow?
Lag grows when the consumer cannot keep up with the production rate. The common causes at a bank are slow per-message processing (complex fraud checks), too few consumers relative to partitions, or a slow downstream dependency (database write bottleneck).
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import java.time.Duration;
import java.math.BigDecimal;
// Slow consumer: fraud check takes too long per record
KafkaConsumer<String, Transaction> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("bank-transactions"));
while (true) {
ConsumerRecords<String, Transaction> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, Transaction> record : records) {
Transaction txn = record.value();
// This fraud check is slow and causes lag to grow
FraudResult result = fraudService.deepCheck(txn); // 500ms per record
processResult(result);
}
consumer.commitSync();
}
What is partition rebalancing?
Rebalancing happens whenever group membership changes: a consumer joins, leaves, or is considered dead after missing heartbeats. The group coordinator reassigns partitions so each live consumer owns a share.
import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;
import org.apache.kafka.common.TopicPartition;
import java.util.Collection;
public class BankRebalanceListener implements ConsumerRebalanceListener {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
// Commit offsets for partitions being taken away
System.out.println("Revoking: " + partitions);
consumer.commitSync();
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
System.out.println("Assigned: " + partitions);
// Resume processing from committed offsets
}
}
consumer.subscribe(List.of("bank-transactions"), new BankRebalanceListener());
During a naive (eager) rebalance, a stop-the-world pause occurs where no consumer processes messages. Cooperative incremental rebalancing reduces this by only moving the partitions that must move.
// Enable cooperative rebalancing to minimize pause
props.put("partition.assignment.strategy",
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
How does back-pressure protect the pipeline?
Back-pressure is how a pipeline protects itself from overload. If a downstream stage like a database write is slower than ingestion, an unbounded buffer would grow until memory is exhausted. Back-pressure propagates the slowness upstream so the whole pipeline runs at the speed of its slowest stage rather than crashing.
// In a pull-based consumer, back-pressure is natural:
// you simply poll less frequently when processing is slow
while (true) {
ConsumerRecords<String, Transaction> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, Transaction> record : records) {
// If this is slow, the next poll is delayed
// which naturally reduces consumption rate
bankDatabase.writeTransaction(record.value());
}
consumer.commitSync();
}
// Tune fetch sizes to control batch pressure
props.put("max.poll.records", "100"); // Smaller batches when processing is slow
props.put("fetch.max.bytes", "1048576"); // 1 MB max per fetch
How do you scale consumers?
Within a consumer group, a partition is owned by exactly one consumer, so you cannot have more active consumers than partitions. Extra consumers sit idle. Partition count sets your maximum parallelism.
Partitions: 6
Consumers: 3 --> Each consumer gets 2 partitions (good)
Consumers: 6 --> Each consumer gets 1 partition (max parallelism)
Consumers: 8 --> 6 active, 2 idle (wasted)
# Increase partitions to allow more consumers
kafka-topics.sh --alter \
--topic bank-transactions \
--partitions 12 \
--bootstrap-server kafka-broker:9092
How do you handle poison records with a dead-letter queue?
A poison record is a malformed message that keeps failing. You retry a bounded number of times and then route it to a dead-letter queue so one bad message never blocks its partition forever.
import java.math.BigDecimal;
int MAX_RETRIES = 3;
for (ConsumerRecord<String, Transaction> record : records) {
int attempts = 0;
boolean success = false;
while (attempts < MAX_RETRIES && !success) {
try {
processTransaction(record.value());
success = true;
} catch (Exception e) {
attempts++;
if (attempts >= MAX_RETRIES) {
// Route to dead-letter queue
dlqProducer.send(new ProducerRecord<>(
"bank-transactions-dlq",
record.key(),
record.value()
));
System.err.println(
"Poison record sent to DLQ: " + record.key());
}
}
}
}
What monitoring should you set up?
Monitor lag, throughput, processing latency, and per-stage error rate. Alert when lag grows or errors spike.
# Prometheus alerting rule for consumer lag
# bank_kafka_alerts.yml
alert_rules = """
groups:
- name: kafka_consumer_health
rules:
- alert: HighConsumerLag
expr: kafka_consumer_group_lag > 10000
for: 5m
labels:
severity: critical
team: bank-data-platform
annotations:
summary: "Consumer lag exceeds 10k for {{ $labels.group }}"
- alert: DLQRateSpike
expr: rate(kafka_dlq_messages_total[5m]) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "Dead-letter queue receiving >10 msgs/sec"
"""
What should a senior engineer know?
- Monitor the dead-letter queue itself: a rising DLQ rate signals an upstream data-quality problem, not just a bad record.
- Cooperative incremental rebalancing (CooperativeStickyAssignor) is the modern default: it reduces pause time from seconds to milliseconds.
- Idempotent processing is essential because a rebalance can cause reprocessing of in-flight messages.
- Partition count is a permanent decision: you can increase it but not decrease it without recreating the topic.
- Consider static group membership for long-lived consumers to avoid unnecessary rebalances on restart.
// Static group membership to avoid rebalance on restart
props.put("group.instance.id", "bank-processor-instance-1");
// Consumer can restart within session.timeout without triggering rebalance
Visual Overview
Producer (bank app)
│
▼
┌──────────────────────────────────┐
│ Kafka Topic: bank-transactions │
│ ┌────┐ ┌────┐ ┌────┐ ┌────┐ │
│ │ P0 │ │ P1 │ │ P2 │ │ P3 │ │
│ └──┬─┘ └──┬─┘ └──┬─┘ └──┬─┘ │
└─────┼──────┼──────┼──────┼───────┘
│ │ │ │
▼ ▼ ▼ ▼
┌─────────┐ ┌─────────┐
│Consumer1│ │Consumer2│ (2 consumers, 4 partitions)
│ P0 + P1 │ │ P2 + P3 │
└────┬────┘ └────┬────┘
│ │
▼ ▼
┌──────────────────────┐
│ Bank Database │
│ (downstream sink) │
└──────────────────────┘
If Consumer2 dies:
Rebalance → Consumer1 gets P0+P1+P2+P3
Lag on P2, P3 grows until scaling
Poison record flow:
Record fails 3x → DLQ topic → Alert → Investigation
Key Takeaway
Consumer lag is the single most important health signal for a streaming pipeline. Each partition has a newest offset, each consumer group commits the offset it has processed, and lag is the difference: the unprocessed backlog. Small stable lag is fine, but steadily growing lag means the consumer cannot keep up, which at the National Bank of India could make fraud alerts arrive minutes late.
Partition rebalancing happens whenever group membership changes, and the coordinator reassigns partitions so each live consumer owns a share. A naive rebalance causes a brief stop-the-world pause that spikes lag and can reprocess in-flight work, which is why idempotent processing matters. Cooperative incremental rebalancing reduces that pause to milliseconds. The hard constraint is that a partition is owned by exactly one consumer, so partition count sets maximum parallelism. Back-pressure propagates slowness upstream so the pipeline degrades gracefully instead of crashing.
To keep the pipeline healthy, scale consumers up to the partition count, monitor lag with alerts, make processing idempotent, and handle poison records with a dead-letter queue and bounded retries. Monitor the DLQ itself, because a rising rate points to upstream data quality problems rather than isolated bad records.
At the National Bank of India, when a malformed transaction with a null amount kept failing, the consumer retried it three times and diverted it to the dead-letter queue, committing past it so the partition kept flowing. We watched lag closely, used cooperative rebalancing, and attached clear ownership to the DLQ because in a bank a poison transaction must be investigated rather than silently dropped.
Remember one sentence: monitor consumer lag as the first health signal, scale consumers up to the partition count, make processing idempotent, and dead-letter poison records so they never block the pipeline.
---