Definition
- Pull model -- consumers call
poll()to request records at their own pace; the broker never pushes records onto a consumer. - Durable lag -- unprocessed records remain safely in the partition log, bounded only by retention, so backpressure means a growing backlog, not lost data.
- max.poll.records -- caps how many records one poll returns, bounding the batch the consumer must process before the next poll.
- max.poll.interval.ms -- the deadline by which the consumer must call poll again; exceeding it triggers a rebalance.
- Manual offset commit -- committing only after processing keeps the committed position truthful, so lag honestly reflects outstanding work.
- Scaling consumers -- adding consumers up to the partition count raises throughput; beyond that, add partitions.
Imagine National Bank of India's documents arriving in a big filing cabinet (the Kafka log). A push system would be a conveyor belt dumping documents onto a clerk's desk faster than the clerk can file them, burying the clerk. Kafka's pull model is different: the clerk walks to the cabinet and takes only as many documents as they can handle. If the clerk is slow, they take fewer and the cabinet holds the rest safely. The growing pile in the cabinet is the lag. To go faster, you hire more clerks, up to the number of drawers.
Understanding the Concepts
What is backpressure?
Backpressure is the mechanism by which a slow consumer signals that it cannot keep up, so the system avoids overwhelming it. In push-based systems, this requires explicit flow control. In Kafka, the pull model provides it automatically.
package com.bank.backpressure;
public class PushVsPull {
public static void main(String[] args) {
System.out.println("Push-based (e.g., RabbitMQ default):");
System.out.println(" Broker pushes messages to consumer");
System.out.println(" Slow consumer -> memory overflow / drop");
System.out.println();
System.out.println("Pull-based (Kafka):");
System.out.println(" Consumer calls poll() at own pace");
System.out.println(" Slow consumer -> polls less -> lag grows");
System.out.println(" Nothing lost, nothing overflows");
}
}
Push-based (e.g., RabbitMQ default):
Broker pushes messages to consumer
Slow consumer -> memory overflow / drop
Pull-based (Kafka):
Consumer calls poll() at own pace
Slow consumer -> polls less -> lag grows
Nothing lost, nothing overflows
How does the pull model handle backpressure?
Consumers call poll() to request records. The broker never forces records onto a consumer. A busy consumer simply does not poll as often or fetches fewer records per poll. The records it has not fetched stay in the partition log, which is durable and bounded only by retention.
package com.bank.consumer;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class PullModelDemo {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092");
props.put("group.id", "settlement-service");
props.put("max.poll.records", "100");
props.put("enable.auto.commit", "false");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(Collections.singletonList("transfers"));
while (true) {
// Consumer controls intake rate via poll()
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofMillis(100));
// Process at own pace
records.forEach(r -> processTransfer(r.value()));
consumer.commitSync();
}
}
}
private static void processTransfer(String value) {
// Slow processing is OK -- just means fewer polls per second
}
}
How does backpressure affect lag?
When a consumer is slower than production, its pull rate falls below the produce rate and lag grows. Lag is the visible, durable record of backpressure.
package com.bank.backpressure;
public class LagAsBackpressure {
public static void main(String[] args) {
int produceRate = 1000; // records/sec
int consumeRate = 800; // records/sec
int deficit = produceRate - consumeRate;
System.out.println("Produce rate : " + produceRate + " records/sec");
System.out.println("Consume rate : " + consumeRate + " records/sec");
System.out.println("Lag growth : " + deficit + " records/sec");
System.out.println("After 1 min : " + (deficit * 60) + " records behind");
System.out.println("After 1 hour : " + (deficit * 3600) + " records behind");
System.out.println();
System.out.println("All records safe in the log -- nothing lost");
}
}
Produce rate : 1000 records/sec
Consume rate : 800 records/sec
Lag growth : 200 records/sec
After 1 min : 12000 records behind
After 1 hour : 720000 records behind
All records safe in the log -- nothing lost
How do you tune poll size safely?
max.poll.records bounds the batch, and max.poll.interval.ms is the deadline. If processing a batch overruns the interval, the broker thinks the consumer is dead and triggers a rebalance.
package com.bank.config;
public class PollSizeTuning {
public static void main(String[] args) {
// WRONG: huge batch, slow processing
System.out.println("WRONG: max.poll.records=10000, processing=500ms each");
System.out.println(" Batch time: 10000 * 500ms = 5000 seconds");
System.out.println(" max.poll.interval.ms=300000 (5 min)");
System.out.println(" Result: REBALANCE triggered!");
System.out.println();
// CORRECT: smaller batch that fits within poll interval
System.out.println("CORRECT: max.poll.records=100, processing=500ms each");
System.out.println(" Batch time: 100 * 500ms = 50 seconds");
System.out.println(" Well within 5 min poll interval");
}
}
WRONG: max.poll.records=10000, processing=500ms each
Batch time: 10000 * 500ms = 5000 seconds
max.poll.interval.ms=300000 (5 min)
Result: REBALANCE triggered!
CORRECT: max.poll.records=100, processing=500ms each
Batch time: 100 * 500ms = 50 seconds
Well within 5 min poll interval
Why use manual offset commit?
Manual commit keeps lag honest. With auto-commit, offsets advance before processing finishes, which risks data loss on a crash and makes lag understate the true backlog.
package com.bank.config;
public class ManualVsAutoCommit {
public static void main(String[] args) {
System.out.println("Auto-commit (enable.auto.commit=true):");
System.out.println(" Offset committed on a timer, before processing done");
System.out.println(" Crash -> records lost (committed but not processed)");
System.out.println(" Lag understates true backlog");
System.out.println();
System.out.println("Manual commit (enable.auto.commit=false):");
System.out.println(" Offset committed only after processing succeeds");
System.out.println(" Crash -> resume from last truly completed record");
System.out.println(" Lag honestly reflects outstanding work");
}
}
Auto-commit (enable.auto.commit=true):
Offset committed on a timer, before processing done
Crash -> records lost (committed but not processed)
Lag understates true backlog
Manual commit (enable.auto.commit=false):
Offset committed only after processing succeeds
Crash -> resume from last truly completed record
Lag honestly reflects outstanding work
How do you scale throughput?
Throughput for a consumer group is limited by partition count. Each partition is consumed by at most one consumer in a group.
package com.bank.scaling;
public class ConsumerScaling {
public static void main(String[] args) {
int partitions = 6;
for (int consumers = 1; consumers <= 8; consumers++) {
int active = Math.min(consumers, partitions);
int idle = consumers - active;
System.out.printf(" %d consumers: %d active, %d idle%n",
consumers, active, idle);
}
System.out.println(" Beyond " + partitions + " consumers, add partitions!");
}
}
1 consumers: 1 active, 0 idle
2 consumers: 2 active, 0 idle
3 consumers: 3 active, 0 idle
4 consumers: 4 active, 0 idle
5 consumers: 5 active, 0 idle
6 consumers: 6 active, 0 idle
7 consumers: 6 active, 1 idle
8 consumers: 6 active, 2 idle
Beyond 6 consumers, add partitions!
What are common mistakes?
package com.bank.mistake;
public class BackpressureMistakes {
public static void main(String[] args) {
System.out.println("WRONG: huge poll batch that overruns poll interval");
System.out.println(" Consumer kicked out, rebalance makes things worse");
System.out.println();
System.out.println("CORRECT: small batches that finish within poll interval");
System.out.println();
System.out.println("WRONG: 8 consumers for 6 partitions");
System.out.println(" 2 consumers sit idle -- wasted resources");
System.out.println();
System.out.println("CORRECT: consumers <= partitions, add partitions first");
}
}
WRONG: huge poll batch that overruns poll interval
Consumer kicked out, rebalance makes things worse
CORRECT: small batches that finish within poll interval
WRONG: 8 consumers for 6 partitions
2 consumers sit idle -- wasted resources
CORRECT: consumers <= partitions, add partitions first
What should a senior engineer know?
Kafka converts overload into durable lag instead of failure, which buys time to react. Partition count is a capacity-planning decision made early because scaling consumers only helps up to that number. The pull model is why Kafka stays stable under bursty load compared to push-based queues.
Visual Overview
Kafka Backpressure (Pull Model)
Producer ──1000 rec/s──▶ Partition Log
┌────────────────────────────┐
│ offset 0 ... 999 1000 ... │
│ ▲ ▲ │
│ │ │ │
│ committed latest │
│ │ │ │
│ ├─ lag ──┤ │
└────────────────────────────┘
▲
Consumer ──poll()──▶ fetch 100 ──┘
(800 rec/s) at own pace
Scaling:
┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐
│ C1 │ │ C2 │ │ C3 │ │ C4 │ │ C5 │ │ C6 │
│ P0 │ │ P1 │ │ P2 │ │ P3 │ │ P4 │ │ P5 │
└─────┘ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘
Max throughput = 6 consumers for 6 partitions
Key Takeaway
Kafka handles backpressure almost for free because it is pull-based. Consumers call poll() to fetch records at their own pace, so a busy consumer simply polls less and fetches smaller batches. Unprocessed records remain safely in the durable partition log, and the growing backlog shows up as consumer lag rather than lost data or memory pressure.
Managing backpressure means tuning poll size to stay under the poll interval deadline, committing offsets manually so lag is honest, and scaling consumers up to the partition count. Beyond that, add partitions. The pull model is why Kafka stays stable under bursty load while push-based queues can overwhelm a slow consumer.
At National Bank of India, the settlement consumer calls a slow risk engine, so during market hours it falls behind and lag grows. Rajesh reduced max.poll.records to keep each batch within the poll interval, switched to manual commit, and scaled the consumer group to match the partition count. The lag drains after market hours.
Remember one sentence: Kafka turns overload into durable lag instead of failure, which buys you time to react, and throughput scales by adding consumers up to the partition count.
---