Definition
- Consumer lag -- the difference between the log end offset (latest produced record) and the consumer group's committed offset (last acknowledged position) for a partition.
- Per-partition lag -- the lag computed individually for each partition, which reveals imbalances and stuck partitions.
- Aggregate lag -- the sum of lag across all partitions, which hides per-partition problems because 600 total could be 50 on each or 600 on one.
- Lag in records vs time -- lag counts records, not seconds; the actual delay depends on how expensive each record is to process.
- Rate of change -- the derivative of lag over time; a constant lag might be fine, but rising lag means the consumer is falling behind.
- Symptom, not diagnosis -- lag tells you something is wrong but never why; the cause could be too few consumers, a slow dependency, a hot key, or a poison pill.
Imagine each teller window at National Bank of India has a queue of customers. Consumer lag is the number of customers still waiting at each window. It tells you a window is busy or falling behind. But it does not tell you how long each customer will wait, because one might be a quick balance check and another a complex loan. It does not tell you whether one window is completely jammed while you only look at the total across all windows. And it does not tell you why a window is slow.
Understanding the Concepts
What is consumer lag?
Lag is computed per partition as the log end offset minus the committed offset. If a partition's latest produced record is at offset 1000 and the consumer has committed offset 940, the lag is 60 records.
package com.bank.monitoring;
public class LagCalculation {
public static void main(String[] args) {
long logEndOffset = 1000;
long committedOffset = 940;
long lag = logEndOffset - committedOffset;
System.out.println("Partition 3:");
System.out.println(" Log end offset : " + logEndOffset);
System.out.println(" Committed : " + committedOffset);
System.out.println(" Lag : " + lag + " records");
}
}
Partition 3:
Log end offset : 1000
Committed : 940
Lag : 60 records
What does lag tell you?
Lag tells you three useful things: whether the consumer is keeping up, whether it is falling behind, and whether partitions are balanced.
package com.bank.monitoring;
public class WhatLagTellsYou {
public static void main(String[] args) {
System.out.println("What lag TELLS you:");
System.out.println(" Lag ~0 : consumer is keeping up");
System.out.println(" Lag rising : consumer cannot keep pace");
System.out.println(" Lag spike+dip : transient slowdown, recovered");
System.out.println(" Lag imbalanced: some partitions hotter than others");
}
}
What lag TELLS you:
Lag ~0 : consumer is keeping up
Lag rising : consumer cannot keep pace
Lag spike+dip : transient slowdown, recovered
Lag imbalanced: some partitions hotter than others
What does lag NOT tell you?
Lag has three critical limitations that interviewers probe.
package com.bank.monitoring;
public class WhatLagDoesNotTell {
public static void main(String[] args) {
// Limitation 1: records, not time
System.out.println("1. Lag is in RECORDS, not TIME");
System.out.println(" 60 records of lag:");
System.out.println(" If each record is trivial: ~100 ms behind");
System.out.println(" If each hits credit bureau: ~30 minutes behind");
System.out.println();
// Limitation 2: level, not rate
System.out.println("2. Lag is a LEVEL, not a RATE");
System.out.println(" Constant lag of 1000:");
System.out.println(" Could be fine (production = consumption)");
System.out.println(" Need derivative to know catching up or behind");
System.out.println();
// Limitation 3: aggregate hides per-partition problems
System.out.println("3. Aggregate lag HIDES per-partition problems");
System.out.println(" Total lag 600 across 12 partitions:");
System.out.println(" Healthy: 50 on each partition");
System.out.println(" Incident: 600 on ONE partition, 0 on rest");
System.out.println(" SAME aggregate number, VERY different reality!");
}
}
1. Lag is in RECORDS, not TIME
60 records of lag:
If each record is trivial: ~100 ms behind
If each hits credit bureau: ~30 minutes behind
2. Lag is a LEVEL, not a RATE
Constant lag of 1000:
Could be fine (production = consumption)
Need derivative to know catching up or behind
3. Aggregate lag HIDES per-partition problems
Total lag 600 across 12 partitions:
Healthy: 50 on each partition
Incident: 600 on ONE partition, 0 on rest
SAME aggregate number, VERY different reality!
Why is aggregate lag dangerous?
package com.bank.monitoring;
public class AggregateVsPerPartition {
public static void main(String[] args) {
// Scenario A: healthy, even distribution
int[] healthy = {50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50};
// Scenario B: one stuck partition
int[] stuck = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 600};
int totalA = 0, totalB = 0;
for (int lag : healthy) totalA += lag;
for (int lag : stuck) totalB += lag;
System.out.println("Scenario A (healthy): aggregate = " + totalA);
System.out.println("Scenario B (incident): aggregate = " + totalB);
System.out.println("SAME aggregate! Partition 11 is stuck at 600.");
System.out.println("Aggregate gauge sees no difference.");
}
}
Scenario A (healthy): aggregate = 600
Scenario B (incident): aggregate = 600
SAME aggregate! Partition 11 is stuck at 600.
Aggregate gauge sees no difference.
How do you read lag programmatically?
package com.bank.monitoring;
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.ListOffsetsResult;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.common.TopicPartition;
import java.util.Map;
import java.util.Properties;
public class LagReader {
public static void readLag(AdminClient admin, String groupId, String topic,
int partitions) throws Exception {
// Get committed offsets
Map<TopicPartition, OffsetAndMetadata> committed =
admin.listConsumerGroupOffsets(groupId).partitionsToOffsetAndMetadata().get();
for (int p = 0; p < partitions; p++) {
TopicPartition tp = new TopicPartition(topic, p);
long end = 0; // Would call admin.listOffsets for latest
long comm = committed.getOrDefault(tp,
new OffsetAndMetadata(0)).offset();
System.out.printf(" %s : lag = %d%n", tp, end - comm);
}
}
public static void main(String[] args) {
System.out.println("Read lag per partition with AdminClient:");
System.out.println(" 1. listConsumerGroupOffsets(groupId)");
System.out.println(" 2. listOffsets(LATEST) for each partition");
System.out.println(" 3. lag = latest - committed");
}
}
Read lag per partition with AdminClient:
1. listConsumerGroupOffsets(groupId)
2. listOffsets(LATEST) for each partition
3. lag = latest - committed
How do you add time-based lag?
For time-based staleness, compare the timestamp of the latest produced record with the timestamp of the last committed record.
package com.bank.monitoring;
public class TimeLag {
public static void main(String[] args) {
long latestTimestamp = System.currentTimeMillis();
long committedTimestamp = latestTimestamp - 45000; // 45 seconds behind
long timeLagMs = latestTimestamp - committedTimestamp;
System.out.println("Record lag : 60 records");
System.out.println("Time lag : " + timeLagMs / 1000 + " seconds");
System.out.println("Time lag is more meaningful for SLA monitoring");
}
}
Record lag : 60 records
Time lag : 45 seconds
Time lag is more meaningful for SLA monitoring
What causes can lag indicate?
Lag is the symptom, not the diagnosis. It could be caused by many different problems.
package com.bank.diagnosis;
public class LagCauses {
public static void main(String[] args) {
System.out.println("Possible causes of high lag:");
System.out.println(" 1. Too few consumers for partition count");
System.out.println(" 2. Slow downstream dependency (credit bureau)");
System.out.println(" 3. Hot partition from bad key choice");
System.out.println(" 4. Long GC pauses");
System.out.println(" 5. Rebalance storm");
System.out.println(" 6. Poison pill blocking one partition");
System.out.println(" 7. max.poll.records too high (batch too large)");
}
}
Possible causes of high lag:
1. Too few consumers for partition count
2. Slow downstream dependency (credit bureau)
3. Hot partition from bad key choice
4. Long GC pauses
5. Rebalance storm
6. Poison pill blocking one partition
7. max.poll.records too high (batch too large)
What should a senior engineer know?
Always monitor lag per partition and by rate of change, not just a single aggregate gauge. Pair record lag with time-based lag for SLA monitoring. Treat lag as the alarm that starts an investigation, never as the explanation.
Visual Overview
Consumer Lag Visualization Partition 3: ┌─────────────────────────────────────────────┐ │ offset 0 committed (940) latest │ │ ├────────────────────┤ (1000) │ │ ├──── lag=60 ──┤ │ │ records records │ └─────────────────────────────────────────────┘ Aggregate vs Per-Partition: ┌────────────────────────────────────────┐ │ Aggregate = 600 │ │ │ │ Healthy: P0=50 P1=50 ... P11=50 │ │ Incident: P0=0 P1=0 ... P11=600 │ │ │ │ SAME number, DIFFERENT reality │ └────────────────────────────────────────┘
Key Takeaway
Consumer lag is the gap between the latest produced offset and the committed offset per partition, and it is the single most important consumer health signal. Near-zero lag means the consumer is keeping up, steadily rising lag means it cannot process as fast as records arrive, and comparing lag across partitions reveals imbalance.
But lag has critical limitations. It is measured in records, not time, so 60 records could be milliseconds or minutes. It is a level, not a rate, so you need the derivative to know if you are catching up or falling behind. Most dangerously, aggregate lag hides per-partition problems: a total of 600 could be 50 on each partition or 600 on one stuck partition, and the aggregate is identical.
At National Bank of India, the settlement team once watched a single aggregate gauge that read 600 and assumed all was well, while one partition was stuck at 600 and the other eleven were at zero. Pankaj switched to per-partition lag alerting and added time-based lag because some partitions processed cheap records and others hit a slow credit bureau.
Remember one sentence: lag is essential but blunt -- watch it per partition and by rate of change, pair it with time-based lag, and treat it as the alarm that starts an investigation, never as the explanation.
---