Definition
- No backpressure -- Kafka decouples producers and consumers through a durable log; a producer is never slowed by a slow consumer.
- Unbounded lag -- if consumer rate is permanently below producer rate, the gap accumulates as ever-growing consumer lag.
- Latency growth -- end-to-end latency grows proportionally to lag divided by consumer throughput.
- Silent data loss -- when lag exceeds the retention horizon, the broker deletes unread segments and the consumer skips forward.
- Fix the imbalance -- scale consumer parallelism, speed up per-record processing, and build application-level backpressure with pause/resume.
Imagine National Bank of India has one clerk reading transfer letters while ten branches keep dropping letters in faster than the clerk can read. The mailbox is huge so the branches never wait. But the unread pile grows every day. Rajesh's letter sinks deeper, and by the time the clerk reaches it, hours have passed. Worse, letters older than seven days are shredded. If the clerk never catches up, Rajesh's letter is shredded before being read.
Understanding the Concepts
Why does Kafka have no end-to-end backpressure?
Kafka deliberately decouples producers from consumers. A producer writes to the log and gets acknowledged based only on broker replication; it does not know or care how far behind consumers are. This decoupling is a feature for independent scaling, but it means the system will never automatically slow the producer to match a slow consumer.
package com.bank.kafka;
public class NoBackpressureExplained {
public static void main(String[] args) {
System.out.println("Kafka's design: no end-to-end backpressure");
System.out.println(" Producer ack depends on: broker replication only");
System.out.println(" Producer knows about consumer lag: NO");
System.out.println(" System will slow producer for slow consumer: NO");
System.out.println();
System.out.println("If consumer rate < producer rate permanently:");
System.out.println(" Lag grows without bound");
System.out.println(" Latency climbs proportionally");
System.out.println(" Eventually: retention deletes unread messages");
}
}
Expected output:
Kafka's design: no end-to-end backpressure
Producer ack depends on: broker replication only
Producer knows about consumer lag: NO
System will slow producer for slow consumer: NO
If consumer rate < producer rate permanently:
Lag grows without bound
Latency climbs proportionally
Eventually: retention deletes unread messages
How do you measure consumer lag?
Lag is the difference between the log-end offset (newest record) and the consumer's committed offset. You read it with the consumer group describe command.
kafka-consumer-groups.sh --describe --group settlement-service \
--bootstrap-server kafka-1:9092
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
settlement-service transfers 0 30000 45000 15000
settlement-service transfers 1 28000 44000 16000
settlement-service transfers 2 31000 46000 15000
# Lag growing every check -- structural imbalance
What happens when lag exceeds retention?
When lag grows past the retention horizon, the broker deletes the oldest segments. The consumer's committed offset points at data that no longer exists, and it jumps forward based on auto.offset.reset, silently losing those messages.
package com.bank.kafka;
public class RetentionDataLoss {
public static void main(String[] args) {
long producerRate = 2000; // records/sec
long consumerRate = 1200; // records/sec
long deficit = producerRate - consumerRate; // 800/sec
long retentionHours = 6;
long retentionRecords = producerRate * retentionHours * 3600;
long hoursToLoss = retentionRecords / deficit / 3600;
System.out.println("Producer: " + producerRate + " rec/s");
System.out.println("Consumer: " + consumerRate + " rec/s");
System.out.println("Deficit: " + deficit + " rec/s");
System.out.println("Retention: " + retentionHours + " hours (" + retentionRecords + " records)");
System.out.println("Time until data loss: ~" + hoursToLoss + " hours");
System.out.println(" After this: oldest unread messages are deleted by retention");
}
}
Expected output:
Producer: 2000 rec/s
Consumer: 1200 rec/s
Deficit: 800 rec/s
Retention: 6 hours (43200000 records)
Time until data loss: ~15 hours
After this: oldest unread messages are deleted by retention
How do you fix the throughput imbalance?
The most direct fix is consumer parallelism: add consumers up to the partition count. Beyond that, add partitions to support more consumers, speed up each consumer, and batch expensive operations.
package com.bank.kafka;
public class ThroughputFixes {
public static void main(String[] args) {
System.out.println("Fix 1: Scale consumer parallelism");
System.out.println(" Add consumers to the group up to partition count");
System.out.println(" If at limit, add partitions (keyed topic caveat)");
System.out.println();
System.out.println("Fix 2: Speed up each consumer");
System.out.println(" Batch database writes (e.g., 100 inserts in one tx)");
System.out.println(" Remove per-record synchronous calls to slow APIs");
System.out.println(" Parallelize processing with a worker pool");
System.out.println();
System.out.println("Fix 3: Tune consumer config");
System.out.println(" Increase max.poll.records for bulk work");
System.out.println(" Increase fetch.max.bytes for network efficiency");
}
}
Expected output:
Fix 1: Scale consumer parallelism
Add consumers to the group up to partition count
If at limit, add partitions (keyed topic caveat)
Fix 2: Speed up each consumer
Batch database writes (e.g., 100 inserts in one tx)
Remove per-record synchronous calls to slow APIs
Parallelize processing with a worker pool
Fix 3: Tune consumer config
Increase max.poll.records for bulk work
Increase fetch.max.bytes for network efficiency
How do you build application-level backpressure?
Since Kafka provides no backpressure, you build it yourself. Use consumer.pause() and consumer.resume() to stop fetching when the internal work queue is full.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;
import java.util.Set;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class AppBackpressure {
private static final int QUEUE_CAPACITY = 1000;
private static final BlockingQueue<String> workQueue = new LinkedBlockingQueue<>(QUEUE_CAPACITY);
public static void managePressure(KafkaConsumer<String, String> consumer) {
Set<TopicPartition> assigned = consumer.assignment();
if (workQueue.remainingCapacity() < 100) {
consumer.pause(assigned);
System.out.println("PAUSE: work queue near full (" + workQueue.size() + "/" + QUEUE_CAPACITY + ")");
} else if (workQueue.size() < QUEUE_CAPACITY / 2) {
consumer.resume(assigned);
System.out.println("RESUME: work queue drained (" + workQueue.size() + "/" + QUEUE_CAPACITY + ")");
}
}
public static void main(String[] args) {
System.out.println("Application-level backpressure with pause/resume:");
System.out.println(" Queue full -> pause(partitions) -> stop fetching");
System.out.println(" Queue drained -> resume(partitions) -> fetch again");
System.out.println(" poll() still called for heartbeats when paused");
}
}
Expected output:
Application-level backpressure with pause/resume:
Queue full -> pause(partitions) -> stop fetching
Queue drained -> resume(partitions) -> fetch again
poll() still called for heartbeats when paused
What about bursty vs structural imbalance?
If the imbalance is only bursty, lag is healthy and self-correcting. The log is the buffer designed to absorb bursts. Do not over-engineer for a steady-state problem you do not have.
| Scenario | Lag Behavior | Action |
|---|---|---|
| Bursty traffic | Spikes then drains | None -- log absorbs it |
| Structural imbalance | Grows indefinitely | Scale consumers, speed up processing |
| Single hot partition | One partition lagging | Key bucketing or salting |
What metrics should you alert on?
Alert on lag trend, not just absolute lag. A steadily rising lag is the early warning of a structural imbalance long before retention deletes anything.
# Monitor lag trend over time
kafka-consumer-groups.sh --describe --group settlement-service \
--bootstrap-server kafka-1:9092
# Run every 5 minutes; if LAG column increases each time -> structural problem
What should a senior engineer know?
Size retention so it exceeds your worst-case recovery time. Watch for a single hot partition masquerading as a global throughput problem. Reactive frameworks like Reactor Kafka expose real backpressure to the application layer by requesting only as many records as downstream demand allows.
Visual Overview
No Backpressure: Lag Growth Over Time ====================================== Producer rate: 2000 rec/s Consumer rate: 1200 rec/s Deficit: 800 rec/s Lag (records) │ │ ╱ retention deletes │ ╱ unread messages │ ╱ │ ╱ ← silent data loss │ ╱ │ ╱ │ ╱ │ ╱ │ ╱ │ ╱ │ ╱ │ ╱ │╱ └──────────────────────────────────► Time 0h 2h 4h 6h 8h 10h Fix: Scale Consumers ===================== Before: 3 consumers, 6 partitions ┌─────┐ ┌─────┐ ┌─────┐ │ C1 │ │ C2 │ │ C3 │ │P0,P1│ │P2,P3│ │P4,P5│ 1200 rec/s total └─────┘ └─────┘ └─────┘ After: 6 consumers, 6 partitions ┌───┐ ┌───┐ ┌───┐ ┌───┐ ┌───┐ ┌───┐ │C1 │ │C2 │ │C3 │ │C4 │ │C5 │ │C6 │ │P0 │ │P1 │ │P2 │ │P3 │ │P4 │ │P5 │ 2400 rec/s total └───┘ └───┘ └───┘ └───┘ └───┘ └───┘
Key Takeaway
Kafka deliberately has no end-to-end backpressure. A producer is acknowledged based only on broker replication and never learns how far behind consumers are. If a consumer group's rate is permanently below the producer's rate, lag grows without bound, latency climbs, and eventually retention deletes unread messages.
The fix is scaling consumer parallelism, speeding up each consumer, and building application-level backpressure with pause/resume since Kafka provides none. For bursty traffic, the log itself is the buffer and lag self-corrects.
In my National Bank of India card-authorization pipeline, fraud scoring could not keep up with 2000 events per second. We scaled consumers to the partition count, batched the database writes, and used pause/resume for a bounded internal queue.
Remember one sentence: Kafka will not throttle the producer for you, so a structural throughput imbalance must be fixed by scaling and speeding up the consumer side and sizing retention above your worst-case recovery time.
---