Definition
- acks=all -- producer waits for all in-sync replicas to persist the record; paired with
min.insync.replicasfor true durability. - Idempotent producer --
enable.idempotence=trueso retries never create duplicates; the default in modern Kafka. - Effectively infinite retries -- set
retriesto max and bound withdelivery.timeout.msso transient failures are retried automatically. - Manual offset commit --
enable.auto.commit=falsewith commit after processing, so a crash resumes from the last truly completed record. - read_committed -- consumer isolation level that skips aborted and in-flight transaction records.
- Dead letter queue -- isolates records that cannot be processed after bounded retries, so they are preserved, not lost or blocking.
Think of guaranteeing that every transfer instruction at National Bank of India is never lost and always acted upon. The producer side is like registered post with delivery confirmation: the sender keeps resending until the recipient confirms (acks and retries), and the post office files a copy in multiple vaults before confirming (min in-sync replicas). The consumer side is like the clerk only stamping a form as done after actually completing the work. The dead letter queue is the problem tray for forms that genuinely cannot be processed.
Understanding the Concepts
What does guaranteed delivery mean?
Guaranteed delivery means a produced record is reliably persisted and eventually processed, with no silent loss and no unhandled failure. No single setting provides it; it is a coordinated stack across the whole path.
package com.bank.delivery;
public class GuaranteedDeliveryStack {
public static void main(String[] args) {
System.out.println("Guaranteed delivery = three layers:");
System.out.println(" 1. PRODUCER: acks=all + min.isr + idempotence + retries");
System.out.println(" 2. CONSUMER: manual commit + read_committed + idempotent processing");
System.out.println(" 3. OPERATIONAL: dead letter queue + monitoring");
System.out.println();
System.out.println("Any weak link breaks the guarantee.");
}
}
Guaranteed delivery = three layers:
1. PRODUCER: acks=all + min.isr + idempotence + retries
2. CONSUMER: manual commit + read_committed + idempotent processing
3. OPERATIONAL: dead letter queue + monitoring
Any weak link breaks the guarantee.
How do you configure the producer?
package com.bank.producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import java.util.Properties;
public class GuaranteedProducerConfig {
public static Properties config() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"broker1:9092,broker2:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
// Durability
props.put(ProducerConfig.ACKS_CONFIG, "all");
// Idempotence (prevents retry duplicates)
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
// Retries bounded by delivery timeout
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120000);
// Max in-flight for idempotence
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
return props;
}
public static void main(String[] args) {
Properties p = config();
System.out.println("Producer config for guaranteed delivery:");
System.out.println(" acks = " + p.get(ProducerConfig.ACKS_CONFIG));
System.out.println(" idempotence = " + p.get(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG));
System.out.println(" retries = MAX");
System.out.println(" delivery.timeout = 120s");
}
}
Producer config for guaranteed delivery:
acks = all
idempotence = true
retries = MAX
delivery.timeout = 120s
How do you configure the consumer?
package com.bank.consumer;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;
public class GuaranteedConsumerConfig {
public static Properties config() {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "settlement-service");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
// Manual commit -- only after processing succeeds
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
// Read only committed transaction records
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
// Start from earliest if no committed offset
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
return props;
}
public static void main(String[] args) {
System.out.println("Consumer config for guaranteed delivery:");
System.out.println(" auto.commit = false");
System.out.println(" isolation = read_committed");
System.out.println(" offset.reset = earliest");
}
}
Consumer config for guaranteed delivery:
auto.commit = false
isolation = read_committed
offset.reset = earliest
How do you configure the topic?
# Create topic with replication and min ISR
kafka-topics --create --topic transfers \
--partitions 6 --replication-factor 3
kafka-configs --alter --topic transfers \
--add-config min.insync.replicas=2
package com.bank.topic;
public class TopicConfig {
public static void main(String[] args) {
System.out.println("Topic config for guaranteed delivery:");
System.out.println(" partitions = 6");
System.out.println(" replication.factor = 3");
System.out.println(" min.insync.replicas = 2");
}
}
Topic config for guaranteed delivery:
partitions = 6
replication.factor = 3
min.insync.replicas = 2
Why must consumers be idempotent?
At-least-once delivery means a record can be redelivered. Without idempotent processing, a redelivery causes double side effects.
package com.bank.consumer;
import java.math.BigDecimal;
public class IdempotentProcessing {
public static void main(String[] args) {
String tradeId = "TRD-001";
BigDecimal amount = new BigDecimal("50000.00");
// WRONG: non-idempotent -- double-settles on redelivery
System.out.println("WRONG: settle(tradeId, amount) -- no dedup check");
System.out.println(" First delivery : settle 50000 INR");
System.out.println(" Redelivery : settle 50000 INR AGAIN!");
System.out.println();
// CORRECT: idempotent -- keyed on trade id
System.out.println("CORRECT: if (!settled(tradeId)) settle(tradeId, amount)");
System.out.println(" First delivery : settle 50000 INR");
System.out.println(" Redelivery : already settled, SKIP");
}
}
WRONG: settle(tradeId, amount) -- no dedup check
First delivery : settle 50000 INR
Redelivery : settle 50000 INR AGAIN!
CORRECT: if (!settled(tradeId)) settle(tradeId, amount)
First delivery : settle 50000 INR
Redelivery : already settled, SKIP
How does the dead letter queue fit?
Guaranteed delivery does not mean guaranteed success for a broken record. It means the record is never silently lost. A DLQ isolates unprocessable records so the partition keeps flowing.
What are common mistakes?
package com.bank.mistake;
public class GuaranteedDeliveryMistakes {
public static void main(String[] args) {
System.out.println("WRONG: acks=1 with min.insync.replicas=2");
System.out.println(" acks=1 never checks ISR -- min.isr is useless");
System.out.println();
System.out.println("WRONG: auto.commit=true for guaranteed delivery");
System.out.println(" Offset advances before processing finishes");
System.out.println(" Crash = data loss");
System.out.println();
System.out.println("WRONG: no DLQ -- poison pill blocks entire partition");
}
}
WRONG: acks=1 with min.insync.replicas=2
acks=1 never checks ISR -- min.isr is useless
WRONG: auto.commit=true for guaranteed delivery
Offset advances before processing finishes
Crash = data loss
WRONG: no DLQ -- poison pill blocks entire partition
What should a senior engineer know?
Guaranteed delivery is end to end. A weak link anywhere breaks it. At-least-once delivery with idempotent processing is the practical shape of exactly-once for most banking systems. Full transactions are reserved for places that truly need atomic multi-write. Everything must be monitored: ISR, producer errors, consumer lag, DLT depth.
Visual Overview
Guaranteed Delivery Stack
PRODUCER BROKER CONSUMER
┌─────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ acks=all │ │ RF=3 │ │ auto.commit=false│
│ idempotence=true│──send─▶│ min.isr=2 │─poll▶│ isolation= │
│ retries=MAX │ │ 3 replicas │ │ read_committed │
│ delivery.timeout│ │ │ │ commit after │
│ =120s │ │ │ │ processing │
└─────────────────┘ └──────────────────┘ └────────┬─────────┘
│
┌────────▼─────────┐
│ Idempotent │
│ processing │
│ (keyed on biz id)│
└────────┬─────────┘
│ if fails
┌────────▼─────────┐
│ Dead Letter Topic│
│ (isolate, don't │
│ lose) │
└──────────────────┘
Key Takeaway
Guaranteed delivery in Kafka is a coordinated stack, and any weak link breaks it. On the producer, set acks=all with min.insync.replicas of at least 2, enable idempotence, and set effectively infinite retries bounded by the delivery timeout. On the consumer, turn off auto-commit and commit only after processing succeeds, set isolation to read_committed if producers use transactions, and make processing idempotent on a business id.
Operationally, a dead letter queue isolates records that can never be processed so they are preserved, not lost. The practical shape of guaranteed delivery for most banking systems is at-least-once delivery with idempotent processing, reserving full transactions for atomic multi-write scenarios.
At National Bank of India, the transfers path uses this stack: acks=all with min.isr=2 on RF=3, idempotence on, manual commit after settlement, and idempotent processing keyed on the trade id. Permanently bad records go to a monitored DLT.
Remember one sentence: guaranteed delivery is end to end, and the guarantee is only as strong as its weakest configured link.
---