Definition
- heartbeat.interval.ms -- how often the background heartbeat thread sends a heartbeat to the group coordinator (default 3 seconds); keeps it well below the session timeout.
- session.timeout.ms -- how long the coordinator waits without a heartbeat before declaring the consumer dead and triggering a rebalance (default 45 seconds); detects crashed or unreachable consumers.
- max.poll.interval.ms -- the maximum time allowed between successive
poll()calls on the application thread (default 5 minutes); detects a live but stalled consumer whose processing has frozen. - Two threads -- the background heartbeat thread proves the process is up; the application thread's poll proves processing is progressing; these are independent liveness signals.
- Why slow handlers rebalance -- heartbeats keep flowing from the background thread, but the application thread is stuck in the handler and has not called
poll()within the interval, so the coordinator evicts it.
Imagine a settlement clerk at National Bank of India. The heartbeat is the clerk waving at the supervisor every few seconds to say they are alive. The session timeout is the rule that if no wave comes, assume the clerk collapsed and reassign their tray. But waving does not prove the clerk is clearing slips; the clerk could be frozen staring at one impossible slip while still waving. So there is a second rule: if the clerk does not grab the next handful of slips within a set time, the supervisor assumes they are stuck, even though they keep waving, and reassigns the tray.
Understanding the Concepts
What are the two liveness mechanisms in a Kafka consumer?
Modern Kafka consumers use two separate threads for liveness. A background heartbeat thread handles group membership, and the application thread handles polling and processing. This separation is the key to understanding these three settings.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;
public class LivenessConfigDemo {
public static Properties create() {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "settlement-service");
// Background heartbeat thread settings
props.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, "3000"); // Wave every 3s
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "45000"); // Dead if no wave for 45s
// Application thread setting
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "300000"); // Stalled if no poll for 5min
System.out.println("Heartbeat interval: 3s (background thread)");
System.out.println("Session timeout: 45s (dead consumer detection)");
System.out.println("Max poll interval: 5min (stalled consumer detection)");
return props;
}
public static void main(String[] args) {
create();
}
}
Expected output:
Heartbeat interval: 3s (background thread)
Session timeout: 45s (dead consumer detection)
Max poll interval: 5min (stalled consumer detection)
How does heartbeat.interval.ms work?
The heartbeat.interval.ms, default three seconds, is how often the background thread sends a heartbeat to the group coordinator. Heartbeats are cheap and frequent, and they say the consumer process is alive and connected. You keep this well below the session timeout, commonly a third of it.
| Setting | Default | Thread | Purpose |
|---|---|---|---|
| heartbeat.interval.ms | 3000 | Background | Send liveness signal |
| session.timeout.ms | 45000 | Background | Detect dead consumer |
| max.poll.interval.ms | 300000 | Application | Detect stalled consumer |
How does session.timeout.ms detect a crashed consumer?
The session.timeout.ms, default forty-five seconds, is how long the coordinator waits without a heartbeat before declaring the consumer dead. This detects a crashed consumer, a killed process, or a network partition, because in those cases the heartbeat thread stops sending.
package com.bank.kafka;
public class SessionTimeoutScenario {
public static void main(String[] args) {
System.out.println("Scenario: settlement-consumer process killed");
System.out.println(" t=0s: Last heartbeat sent");
System.out.println(" t=3s: Heartbeat missed (process dead)");
System.out.println(" t=6s: Heartbeat missed again");
System.out.println(" t=45s: Session timeout expires");
System.out.println(" t=45s: Coordinator declares consumer dead");
System.out.println(" t=45s: Rebalance triggered -- partitions reassigned");
System.out.println();
System.out.println("Tuning: shorter timeout = faster detection, but risks false eviction");
System.out.println("Tuning: longer timeout = more tolerant, but slower reaction");
}
}
Expected output:
Scenario: settlement-consumer process killed
t=0s: Last heartbeat sent
t=3s: Heartbeat missed (process dead)
t=6s: Heartbeat missed again
t=45s: Session timeout expires
t=45s: Coordinator declares consumer dead
t=45s: Rebalance triggered -- partitions reassigned
Tuning: shorter timeout = faster detection, but risks false eviction
Tuning: longer timeout = more tolerant, but slower reaction
How does max.poll.interval.ms detect a stalled consumer?
The max.poll.interval.ms, default five minutes, bounds the time between successive poll() calls on the application thread. If your handler takes longer than this, the consumer proactively leaves the group and its partitions are reassigned.
package com.bank.kafka;
public class PollIntervalScenario {
public static void main(String[] args) {
System.out.println("Scenario: slow compliance check blocks handler");
System.out.println(" t=0s: poll() returns 100 records");
System.out.println(" t=10s: Processing record 50, compliance API slow");
System.out.println(" t=60s: Still processing, heartbeats flowing normally");
System.out.println(" t=180s: Still stuck on compliance call");
System.out.println(" t=300s: max.poll.interval.ms (5 min) exceeded");
System.out.println(" t=300s: Consumer leaves group -- rebalance triggered");
System.out.println(" NOTE: Heartbeats were flowing the entire time!");
System.out.println();
System.out.println("Fix: lower max.poll.records so batch finishes in time");
System.out.println("Fix: or offload slow work to a worker thread");
}
}
Expected output:
Scenario: slow compliance check blocks handler
t=0s: poll() returns 100 records
t=10s: Processing record 50, compliance API slow
t=60s: Still processing, heartbeats flowing normally
t=180s: Still stuck on compliance call
t=300s: max.poll.interval.ms (5 min) exceeded
t=300s: Consumer leaves group -- rebalance triggered
NOTE: Heartbeats were flowing the entire time!
Fix: lower max.poll.records so batch finishes in time
Fix: or offload slow work to a worker thread
Why does a slow handler trigger rebalance even with heartbeats flowing?
The heartbeat thread runs independently on a background thread, so from the coordinator's membership view the consumer looks alive. But the application thread is stuck inside the handler and has not called poll() again within the poll interval. The two mechanisms answer different questions: is the process up (heartbeat), and is the processing progressing (poll).
package com.bank.kafka;
public class TwoLivenessSignals {
public static void main(String[] args) {
System.out.println("Two independent liveness signals:");
System.out.println();
System.out.println(" Background thread (heartbeat):");
System.out.println(" Question: Is the process UP?");
System.out.println(" Signal: heartbeat.interval.ms");
System.out.println(" Timeout: session.timeout.ms");
System.out.println(" Detects: CRASHED consumer (killed, network down)");
System.out.println();
System.out.println(" Application thread (poll):");
System.out.println(" Question: Is processing PROGRESSING?");
System.out.println(" Signal: time between poll() calls");
System.out.println(" Timeout: max.poll.interval.ms");
System.out.println(" Detects: STALLED consumer (slow handler, deadlock)");
}
}
Expected output:
Two independent liveness signals:
Background thread (heartbeat):
Question: Is the process UP?
Signal: heartbeat.interval.ms
Timeout: session.timeout.ms
Detects: CRASHED consumer (killed, network down)
Application thread (poll):
Question: Is processing PROGRESSING?
Signal: time between poll() calls
Timeout: max.poll.interval.ms
Detects: STALLED consumer (slow handler, deadlock)
How do you tune these settings together?
Keep the heartbeat interval at about a third of the session timeout. Size max.poll.records and max.poll.interval.ms together so each batch finishes within the interval.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;
public class TuningExample {
public static Properties tunedConfig() {
Properties props = new Properties();
// Heartbeat: 1/3 of session timeout
props.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, "10000"); // 10s
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000"); // 30s
// Processing: batch size * processing time < poll interval
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "50");
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "600000"); // 10 min
// 50 records * 10 sec each = 500 sec < 600 sec poll interval
System.out.println("Tuned: heartbeat=10s, session=30s, poll-records=50, poll-interval=10min");
return props;
}
public static void main(String[] args) {
tunedConfig();
}
}
Expected output:
Tuned: heartbeat=10s, session=30s, poll-records=50, poll-interval=10min
What about pause and resume for long-running work?
For genuinely long work, offload it to a worker thread and use consumer.pause() and consumer.resume() to keep polling (and heartbeating) while the work proceeds, decoupling processing time from the poll interval.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;
import java.util.Set;
public class PauseResumePattern {
public static void handleSlowWork(KafkaConsumer<String, String> consumer) {
Set<TopicPartition> assigned = consumer.assignment();
// Pause fetching while slow work proceeds
consumer.pause(assigned);
System.out.println("Paused partitions -- poll() still called for heartbeats");
// Submit slow work to a thread pool
// ... worker processes the batch ...
// Resume fetching when work completes
consumer.resume(assigned);
System.out.println("Resumed partitions -- fetching new records again");
}
}
What should a senior engineer know?
After a poll-interval eviction, partitions are reassigned and likely reprocessed, so idempotency matters. The reprocessing plus the rebalance itself can cascade if many consumers are slow. Doing heavy work on a separate thread with pause/resume is the pattern to decouple processing time from the poll interval.
Visual Overview
Two-Thread Liveness Model
==========================
┌────────────────────────────────────────────────────┐
│ Consumer Process │
│ │
│ ┌──────────────────────┐ ┌──────────────────────┐ │
│ │ Background Thread │ │ Application Thread │ │
│ │ │ │ │ │
│ │ heartbeat ─────────► │ │ poll() ──► process │ │
│ │ every 3 seconds │ │ ──► commitSync │ │
│ │ │ │ ──► poll() │ │
│ │ Proves: process UP │ │ Proves: processing │ │
│ │ │ │ PROGRESSING │ │
│ └───────────┬───────────┘ └──────────┬────────────┘ │
│ │ │ │
└──────────────┼──────────────────────────┼──────────────┘
│ │
▼ ▼
┌──────────────────────┐ ┌──────────────────────────┐
│ Group Coordinator │ │ Group Coordinator │
│ │ │ │
│ No heartbeat for 45s? │ │ No poll() for 5 min? │
│ → Consumer DEAD │ │ → Consumer STALLED │
│ → Rebalance │ │ → Rebalance │
└───────────────────────┘ └───────────────────────────┘
Slow handler scenario:
Background thread: ✓ heartbeats flowing
Application thread: ✗ stuck in handler, no poll()
Result: max.poll.interval.ms expires → rebalance
Key Takeaway
Kafka detects a failed consumer in two independent ways, on two different threads. A background heartbeat thread sends heartbeats and the coordinator evicts the consumer if none arrive within the session timeout, detecting a crashed or unreachable consumer. The max-poll-interval bounds the time between poll calls on the application thread, detecting a consumer that is alive but stalled.
That is exactly why a slow handler triggers a rebalance even with heartbeats flowing. The heartbeat thread runs independently, so from the coordinator's view the process looks alive, but the application thread is stuck and has not called poll within the interval.
In my National Bank of India settlement service, we saw rebalances with no crashes. The cause was a slow compliance call that pushed a batch past the poll interval while heartbeats stayed healthy. We fixed it by lowering max-poll-records and using pause/resume for long work.
Remember one sentence: heartbeats prove the process is up, the poll interval proves processing is progressing, and a slow handler trips the poll interval even while heartbeats flow.
---