Back to Consumer: Commits, Polling, Deserialization
Q2Question 2 of 7

What do the key consumer configs do?

Key Concepts
  • enable.auto.commit=false gives clean at-least-once by committing only after processing succeeds.
  • auto.offset.reset=earliest reads the full backlog for a new group; latest silently skips existing messages.
  • max.poll.records caps per-poll batch size (default 500) and is not a total consumption limit.
  • max.poll.interval.ms evicts a consumer that takes too long between polls, triggering a rebalance.
  • auto.offset.reset is irrelevant when a committed offset already exists; the consumer resumes there.
  • Size max.poll.records so each batch finishes processing within the max.poll.interval.ms window.

Definition

enable.auto.commit controls safe manual commit for at-least-once, auto.offset.reset picks backlog versus skip for a fresh group, max.poll.records sizes each poll batch, and max.poll.interval.ms evicts a consumer that processes a batch too slowly.
  • enable.auto.commit -- when true (default), offsets commit on a timer; when false, you commit manually after processing, giving clean at-least-once delivery.
  • auto.offset.reset -- decides where a consumer starts when there is no committed offset: earliest reads the whole backlog, latest skips to the end, none throws an error.
  • max.poll.records -- caps how many records a single poll() returns (default 500); it is a per-batch limit, not a total consumption cap.
  • max.poll.interval.ms -- bounds the time allowed between successive poll() calls (default 5 minutes); exceeding it triggers a rebalance because the coordinator assumes the consumer is stalled.

Think of a settlement clerk at National Bank of India. The auto-commit setting is whether the clerk marks slips cleared automatically on a timer or only after actually doing the work. The offset-reset setting is, for a brand-new clerk, whether they begin from the oldest slip in the tray or only from slips arriving after they sit down. The max-poll-records setting is how many slips the clerk grabs in one handful. And the max-poll-interval setting is the rule that if the clerk takes too long before grabbing the next handful, the supervisor assumes they are stuck and reassigns their tray.

Understanding the Concepts

What does enable.auto.commit do?

The enable.auto.commit setting, default true, makes the consumer commit the last polled offsets automatically every auto.commit.interval.ms, by default five seconds. The problem is that this timer is independent of whether you have finished processing, so auto-commit can advance the offset past records you have not yet handled.

JAVA
package com.bank.kafka;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;

public class AutoCommitConfig {
    // WRONG: auto-commit risks skipping unprocessed records
    public static Properties dangerousConfig() {
        Properties props = new Properties();
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
        props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "5000");
        System.out.println("WRONG: auto-commit can lose records on crash");
        return props;
    }

    // CORRECT: manual commit for at-least-once
    public static Properties safeConfig() {
        Properties props = new Properties();
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        System.out.println("CORRECT: manual commit after processing");
        return props;
    }

    public static void main(String[] args) {
        dangerousConfig();
        safeConfig();
    }
}
TEXT
Expected output:
WRONG: auto-commit can lose records on crash
CORRECT: manual commit after processing

What does auto.offset.reset do?

The auto.offset.reset setting decides where to start only when there is no valid committed offset for the group and partition, which happens for a brand-new group or when the committed offset has been deleted by retention.

| Value | Behavior | Use Case |
|---|---|---|
| earliest | Start at oldest available record | Process full history |
| latest | Start at end, skip existing | Only new messages |
| none | Throw exception | Fail loudly rather than guess |

JAVA
package com.bank.kafka;

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class OffsetResetDemo {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "new-fraud-scorer");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

        // earliest: new group reads ALL existing transactions
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        System.out.println("New group 'new-fraud-scorer' will read entire backlog");

        // latest would skip everything, which is a classic cause of
        // "my new consumer appears to receive nothing"

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(Collections.singletonList("transfers"));
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
            System.out.println("Received " + records.count() + " backlog records");
        }
    }
}
TEXT
Expected output:
New group 'new-fraud-scorer' will read entire backlog
Received 45000 backlog records

What does max.poll.records do?

The max.poll.records setting, default 500, caps how many records a single call to poll() returns. It is not a total limit on consumption and not a byte limit; it just shapes each batch. You tune it down when each record is expensive to process.

JAVA
package com.bank.kafka;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;

public class MaxPollRecordsDemo {
    public static void main(String[] args) {
        // For lightweight records: larger batches for throughput
        Properties highThroughput = new Properties();
        highThroughput.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "1000");
        System.out.println("High throughput: 1000 records per poll");

        // For expensive fraud scoring: smaller batches to finish within poll interval
        Properties expensiveProcessing = new Properties();
        expensiveProcessing.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "50");
        System.out.println("Expensive processing: 50 records per poll");
        System.out.println("  Ensures batch completes within max.poll.interval.ms");
    }
}
TEXT
Expected output:
High throughput: 1000 records per poll
Expensive processing: 50 records per poll
  Ensures batch completes within max.poll.interval.ms

What does max.poll.interval.ms do?

The max.poll.interval.ms setting, default five minutes, bounds the time allowed between two successive poll() calls. If your handler takes longer than this to process a batch, the group coordinator assumes the consumer is stuck, removes it from the group, and triggers a rebalance.

This is why a slow message handler causes a rebalance even when heartbeats are still flowing: the heartbeat thread proves the process is up, while the poll interval proves processing is progressing.

JAVA
package com.bank.kafka;

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class PollIntervalDemo {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "fraud-scorer");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "50");
        props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "300000"); // 5 minutes

        System.out.println("Config: 50 records per poll, 5 min poll interval");
        System.out.println("Each record must be scored in < 6 seconds to stay safe");
        // 50 records * 6 sec each = 300 sec = 5 min poll interval
    }
}
TEXT
Expected output:
Config: 50 records per poll, 5 min poll interval
Each record must be scored in < 6 seconds to stay safe

How do these four settings interact?

Manual commit plus a right-sized max.poll.records plus a sane max.poll.interval.ms together give a consumer that is both safe and stable: it processes bounded batches, commits only after success, and does not get evicted mid-batch.

JAVA
package com.bank.kafka;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;

public class RecommendedConsumerConfig {
    public static Properties create() {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "settlement-service");

        // Safe commit: manual after processing
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

        // Start from beginning for new group
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        // Bounded batches for expensive processing
        props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "100");

        // Generous but bounded poll interval
        props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "300000");

        System.out.println("Recommended config: safe, complete, and stable");
        return props;
    }

    public static void main(String[] args) {
        create();
    }
}
TEXT
Expected output:
Recommended config: safe, complete, and stable

What is the auto.offset.reset=none option?

Setting auto.offset.reset=none throws a NoOffsetForPartitionException if there is no committed offset, which you use when you want to fail loudly rather than silently guess where to start.

JAVA
package com.bank.kafka;

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class OffsetResetNoneDemo {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "strict-audit-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(Collections.singletonList("transfers"));
            consumer.poll(Duration.ofSeconds(5));
        } catch (org.apache.kafka.clients.consumer.NoOffsetForPartitionException e) {
            System.out.println("ERROR: No committed offset found for group");
            System.out.println("Use 'none' to fail loudly rather than guess");
        }
    }
}
TEXT
Expected output:
ERROR: No committed offset found for group
Use 'none' to fail loudly rather than guess

What are common mistakes with consumer configs?

The most common mistake is leaving auto-commit enabled and assuming at-least-once delivery. Another frequent error is setting auto.offset.reset=latest for a new group and then wondering why the consumer appears to receive nothing, because it skipped the entire backlog. A third mistake is setting max.poll.records very high without raising max.poll.interval.ms, causing the consumer to be evicted mid-batch.

JAVA
package com.bank.kafka;

public class ConsumerConfigMistakes {
    public static void main(String[] args) {
        System.out.println("WRONG: auto-commit + at-least-once claim");
        System.out.println("  enable.auto.commit=true (default)");
        System.out.println("  Crash between poll and commit -> records lost");
        System.out.println();
        System.out.println("WRONG: latest for a new group that needs history");
        System.out.println("  auto.offset.reset=latest");
        System.out.println("  New group skips entire backlog silently");
        System.out.println();
        System.out.println("WRONG: large batch + short poll interval");
        System.out.println("  max.poll.records=5000, max.poll.interval.ms=30000");
        System.out.println("  Expensive processing -> eviction -> rebalance storm");
        System.out.println();
        System.out.println("CORRECT: manual commit, earliest, sized batches");
    }
}
TEXT
Expected output:
WRONG: auto-commit + at-least-once claim
  enable.auto.commit=true (default)
  Crash between poll and commit -> records lost

WRONG: latest for a new group that needs history
  auto.offset.reset=latest
  New group skips entire backlog silently

WRONG: large batch + short poll interval
  max.poll.records=5000, max.poll.interval.ms=30000
  Expensive processing -> eviction -> rebalance storm

CORRECT: manual commit, earliest, sized batches

What should a senior engineer know?

After a rebalance caused by a blown poll interval, the reassigned partitions may be reprocessed, so idempotency still matters. When the setting does nothing: auto.offset.reset is irrelevant when a committed offset already exists, because the consumer simply resumes there. A common misconception is that max.poll.records limits total consumption; it only limits per-poll batch size.

Visual Overview

DIAGRAM
 Consumer Config Decision Flow
 ==============================

 ┌─────────────────────────────────────┐
 │        Consumer Starts Up            │
 └──────────────┬──────────────────────┘
                │
                ▼
 ┌─────────────────────────────────────┐
 │ Does a committed offset exist?       │
 └───────┬─────────────────┬───────────┘
    yes  │                 │  no
         ▼                 ▼
 ┌───────────────┐  ┌─────────────────────────┐
 │ Resume from   │  │ auto.offset.reset?       │
 │ committed     │  ├─────────┬───────┬───────┤
 │ offset        │  │earliest │latest │ none  │
 └───────────────┘  │         │       │       │
                    ▼         ▼       ▼
              ┌─────────┐ ┌──────┐ ┌───────┐
              │ Read all│ │ Skip │ │ Throw │
              │ backlog │ │ old  │ │ error │
              └─────────┘ └──────┘ └───────┘


 Poll Loop Timing
 ================

  poll() ──► process batch ──► commitSync() ──► poll()
    │                                             │
    │◄──────── max.poll.interval.ms ─────────────►│
    │                                             │
    │   batch size = max.poll.records (default 500)
    │
    │   If processing takes > max.poll.interval.ms:
    │     Coordinator evicts consumer → rebalance

Key Takeaway

These four consumer settings control correctness, startup behavior, and liveness. The auto-commit setting decides whether offsets commit on a timer or manually. For clean at-least-once I disable auto-commit and commit only after processing succeeds, so a crash before commit causes a safe reprocess rather than a loss.

The offset-reset setting only matters for a brand-new group: earliest reads the whole backlog, latest starts at the end. The max-poll-records setting caps each poll batch, and the max-poll-interval bounds the time between polls, evicting a consumer that processes too slowly.

In my National Bank of India fraud-scoring consumer, I set reset to earliest for the full backlog, disabled auto-commit, lowered max-poll-records so each batch finished inside the poll interval, and the consumer was safe, complete, and stable.

Remember one sentence: disable auto-commit for safety, set earliest for completeness, size max-poll-records for your processing speed, and keep each batch inside the poll interval to avoid eviction.

---

Share
Use ← → arrow keys to navigate