Definition
- bootstrap.servers -- comma-separated list of broker host:port pairs for initial cluster discovery; list at least two for fault tolerance.
- key.serializer / value.serializer -- convert Java objects to bytes on the producer side; must match the consumer's deserializers.
- acks -- durability level: 0 (fire-and-forget), 1 (leader only), all (all in-sync replicas).
- partition count + replication factor -- set at topic creation; partitions control parallelism, replication factor controls durability.
- group.id -- ties consumers into a group that shares partitions for load balancing.
- auto.offset.reset -- where a new consumer group starts:
earliest(whole backlog) orlatest(only new messages).
Think of National Bank of India sending a transfer instruction. The producer is the branch clerk who must know the central office address (bootstrap.servers), how to write the form in a language the office understands (serializers), and how many confirmations to wait for (acks). The broker is the central mailroom with pigeonholes (partitions) and backup copies (replication). The consumer is the settlement team that must know the address, how to read the form (deserializers), and which team they belong to (group.id).
Understanding the Concepts
What does the producer need?
package com.bank.producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;
public class MinimalProducerConfig {
public static void main(String[] args) {
Properties props = new Properties();
// Required: where to find the cluster
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"broker1:9092,broker2:9092");
// Required: how to serialize the key
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
// Required: how to serialize the value
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
// Recommended: durability
props.put(ProducerConfig.ACKS_CONFIG, "all");
// Recommended: prevent retry duplicates
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
producer.send(new ProducerRecord<>("transfers", "SAV2001",
"{\"amount\":50000,\"type\":\"DEBIT\"}"));
System.out.println("Transfer sent for SAV2001");
}
}
}
Transfer sent for SAV2001
What does the broker/topic need?
# Create topic with partition count and replication
kafka-topics --create --topic transfers \
--partitions 6 --replication-factor 3 \
--bootstrap-server broker1:9092
# Set min.insync.replicas for durability
kafka-configs --alter --topic transfers \
--add-config min.insync.replicas=2 \
--bootstrap-server broker1:9092
package com.bank.topic;
public class TopicParameters {
public static void main(String[] args) {
System.out.println("Topic parameters:");
System.out.println(" partitions = 6 (parallelism)");
System.out.println(" replication.factor = 3 (durability)");
System.out.println(" min.insync.replicas = 2 (acks=all safety)");
System.out.println(" retention.ms = 604800000 (7 days)");
}
}
Topic parameters:
partitions = 6 (parallelism)
replication.factor = 3 (durability)
min.insync.replicas = 2 (acks=all safety)
retention.ms = 604800000 (7 days)
What does the consumer need?
package com.bank.consumer;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class MinimalConsumerConfig {
public static void main(String[] args) {
Properties props = new Properties();
// Required: where to find the cluster
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
// Required: which group this consumer belongs to
props.put(ConsumerConfig.GROUP_ID_CONFIG, "settlement-service");
// Required: how to deserialize key and value
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
// Recommended: where to start if no committed offset
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
// Recommended: manual commit for at-least-once
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(Collections.singletonList("transfers"));
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
records.forEach(r ->
System.out.println("Received: key=" + r.key() + " value=" + r.value()));
consumer.commitSync();
}
}
}
Received: key=SAV2001 value={"amount":50000,"type":"DEBIT"}
How do the parameters relate? (summary table)
| Layer | Parameter | Purpose | Banking value |
|---|---|---|---|
| Producer | bootstrap.servers | Cluster discovery | List 2+ brokers |
| Producer | key/value.serializer | Object to bytes | StringSerializer for JSON |
| Producer | acks | Durability | all |
| Producer | enable.idempotence | No retry duplicates | true |
| Broker | partitions | Parallelism | 6+ |
| Broker | replication.factor | Durability | 3 |
| Broker | min.insync.replicas | acks=all safety | 2 |
| Consumer | bootstrap.servers | Cluster discovery | Same as producer |
| Consumer | group.id | Load sharing | settlement-service |
| Consumer | key/value.deserializer | Bytes to object | Must match producer |
| Consumer | auto.offset.reset | New group start | earliest |
| Consumer | enable.auto.commit | Commit control | false |
What are common mistakes?
package com.bank.mistake;
public class ConfigMistakes {
public static void main(String[] args) {
// WRONG: serializer/deserializer mismatch
System.out.println("WRONG: producer=StringSerializer, consumer=AvroDeserializer");
System.out.println(" Result: garbage or deserialization crash");
System.out.println();
// WRONG: single bootstrap server
System.out.println("WRONG: bootstrap.servers=broker1:9092 (only one)");
System.out.println(" If broker1 is down, client cannot bootstrap");
System.out.println();
// CORRECT
System.out.println("CORRECT: bootstrap.servers=broker1:9092,broker2:9092");
System.out.println(" Survives one listed broker being down");
}
}
WRONG: producer=StringSerializer, consumer=AvroDeserializer
Result: garbage or deserialization crash
WRONG: bootstrap.servers=broker1:9092 (only one)
If broker1 is down, client cannot bootstrap
CORRECT: bootstrap.servers=broker1:9092,broker2:9092
Survives one listed broker being down
How does the message key affect flow?
The key determines which partition a record goes to. In banking, key by account id for per-account ordering.
package com.bank.producer;
import org.apache.kafka.clients.producer.ProducerRecord;
public class KeyUsage {
public static void main(String[] args) {
// Key = account id -> all events for SAV2001 on same partition
ProducerRecord<String, String> record = new ProducerRecord<>(
"transfers", "SAV2001", "{\"amount\":50000}");
System.out.println("Key: " + record.key());
System.out.println("All SAV2001 events go to the same partition");
System.out.println("Ordering guaranteed for this account");
}
}
Key: SAV2001
All SAV2001 events go to the same partition
Ordering guaranteed for this account
What are the default values for key parameters?
Knowing the defaults helps you understand what happens when a parameter is not explicitly set. Many production issues stem from relying on defaults that are optimized for safety or throughput but not for the specific use case.
| Parameter | Default | Notes | |---|---|---| | acks | all (Kafka 3.0+) | Was 1 before 3.0 | | retries | MAX_INT (Kafka 2.1+) | Was 0 before 2.1 | | enable.idempotence | true (Kafka 3.0+) | Was false before 3.0 | | max.in.flight.requests | 5 | Must be <=5 for idempotence | | enable.auto.commit | true | Dangerous for at-least-once | | auto.offset.reset | latest | Skips backlog for new groups | | max.poll.records | 500 | Per-poll batch size | | replication.factor | 1 | Must be >=3 for production |
package com.bank.kafka;
public class DefaultValues {
public static void main(String[] args) {
System.out.println("Key defaults that catch people:");
System.out.println(" acks=all (safe since Kafka 3.0, was 1 before)");
System.out.println(" enable.auto.commit=true (dangerous for at-least-once)");
System.out.println(" auto.offset.reset=latest (skips backlog)");
System.out.println(" replication.factor=1 (no redundancy)");
}
}
Expected output:
Key defaults that catch people:
acks=all (safe since Kafka 3.0, was 1 before)
enable.auto.commit=true (dangerous for at-least-once)
auto.offset.reset=latest (skips backlog)
replication.factor=1 (no redundancy)
What should a senior engineer know?
The serializer on the producer and the deserializer on the consumer must agree. bootstrap.servers should list at least two brokers. The partition count is a capacity decision made early. Manual commit with auto-commit off gives at-least-once semantics. acks=all plus min.insync.replicas gives durability.
Visual Overview
Message Flow: Producer -> Broker -> Consumer Producer Broker Consumer ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ bootstrap.servers│ │ Topic: transfers │ │ bootstrap.servers│ │ key.serializer │ │ partitions: 6 │ │ group.id │ │ value.serializer │ │ RF: 3 │ │ key.deserializer │ │ acks=all │───send────▶│ min.isr: 2 │──poll───▶│ value.deserializer│ │ idempotence=true │ │ │ │ auto.commit=false│ └──────────────────┘ └──────────────────┘ └──────────────────┘ key=SAV2001 ──hash──▶ partition 3 ──▶ consumer C2 (assigned P3)
Key Takeaway
For a message to flow from producer to broker to consumer, each side needs a small set of parameters. The producer needs bootstrap.servers for cluster discovery, serializers to convert objects to bytes, and the acks setting for durability. The broker needs the topic with a partition count for parallelism, a replication factor for durability, and min.insync.replicas for acks=all safety.
The consumer needs bootstrap.servers, deserializers that match the producer's serializers, a group id for load sharing, and decisions about offset reset and auto-commit. In banking, Rajesh uses acks=all, idempotence, and manual commit for at-least-once delivery.
At National Bank of India, the account number is the message key so all of Priya's events stay ordered in the same partition. acks=all with min.isr=2 on RF=3 gives durability. Manual commit after writing to the ledger ensures the committed position reflects real completed work.
Remember one sentence: the producer needs servers, serializers, and acks; the broker needs partitions and replication; the consumer needs servers, deserializers, a group id, and manual commit.
---