Definition
- Idempotent producer -- the foundation of exactly-once on the write side; guarantees that a record sent once is written exactly once even if the producer retries after a lost acknowledgement.
- Producer id -- a unique identifier assigned by the broker to the producer session; used with the sequence number for deduplication.
- Sequence number -- a monotonically increasing per-partition counter the producer attaches to each record; if the broker sees a duplicate sequence, it acknowledges without writing.
- Exactly-once semantics -- each record is produced once, grouped writes are atomic, and consumers never see duplicates or partial results.
- Transaction layer -- builds on idempotence to add atomicity across partitions and tie offset commits to output records.
- read_committed -- consumer isolation level that only delivers committed transaction records, completing the exactly-once chain.
Imagine Rajesh instructs National Bank of India to transfer fifty thousand rupees. The phone line drops before he hears done, so his app resends. An idempotent producer is like stamping each instruction with a unique serial number: the bank sees the same serial twice and files it only once, so Rajesh is debited exactly once, not twice. That single guarantee is the first brick of exactly-once; stacking transactions and read-committed on top makes the whole money movement happen once and only once.
Understanding the Concepts
Why do duplicates happen in Kafka?
Duplicates happen because of the network gap between the broker writing a record and the producer receiving the acknowledgement. The producer sends a record. The broker appends it to the log. But the acknowledgement is lost on the way back. The producer cannot tell whether the write succeeded, so it retries. Without protection, the same record is now in the log twice.
package com.bank.duplicate;
public class DuplicateScenario {
public static void main(String[] args) {
System.out.println("Step 1: Producer sends transfer for SAV2001");
System.out.println("Step 2: Broker writes to partition 3 at offset 100");
System.out.println("Step 3: ACK lost on the network");
System.out.println("Step 4: Producer retries the same record");
System.out.println("Step 5: Broker writes AGAIN at offset 101");
System.out.println("Result: Priya debited TWICE for the same transfer!");
}
}
Step 1: Producer sends transfer for SAV2001
Step 2: Broker writes to partition 3 at offset 100
Step 3: ACK lost on the network
Step 4: Producer retries the same record
Step 5: Broker writes AGAIN at offset 101
Result: Priya debited TWICE for the same transfer!
How does the idempotent producer fix this?
When idempotence is enabled, the broker gives the producer a unique producer id (PID). The producer attaches a sequence number to each record per partition, increasing by one each time. The broker remembers the last sequence it accepted for that PID and partition. If a retried record carries a sequence that equals or is less than the last written, the broker recognizes it as a duplicate, acknowledges it, but does not append it again.
package com.bank.producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;
public class IdempotentFixDemo {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
// Enable idempotence (default in recent Kafka)
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
producer.send(new ProducerRecord<>("transfers", "SAV2001",
"{\"amount\":50000,\"type\":\"DEBIT\"}"));
// If ACK is lost and producer retries, broker drops the duplicate
System.out.println("Idempotent: retry-safe, no double debit for Priya");
}
}
}
Idempotent: retry-safe, no double debit for Priya
What are the configuration requirements?
Idempotence requires three settings to work:
| Setting | Required value | Why | |---|---|---| | enable.idempotence | true | Turns on PID + sequence tracking | | acks | all | Record must be on all ISR before ack | | max.in.flight.requests.per.connection | <= 5 | Broker can detect and reorder within 5 | | retries | > 0 | Retrying is the whole point |
package com.bank.config;
public class IdempotenceRequirements {
public static void main(String[] args) {
System.out.println("Idempotence requirements:");
System.out.println(" enable.idempotence = true (default in Kafka 3.0+)");
System.out.println(" acks = all");
System.out.println(" max.in.flight.requests.per.connection <= 5");
System.out.println(" retries > 0");
}
}
Idempotence requirements:
enable.idempotence = true (default in Kafka 3.0+)
acks = all
max.in.flight.requests.per.connection <= 5
retries > 0
How does idempotence connect to exactly-once?
Exactly-once has three parts. First, the write must not duplicate on retry -- the idempotent producer. Second, grouped writes must be atomic -- the transactional producer. Third, in a consume-process-produce loop, offsets and output must be committed together with consumers reading only committed data.
package com.bank.eos;
public class ExactlyOnceLayers {
public static void main(String[] args) {
System.out.println("Exactly-once = three layers:");
System.out.println(" Layer 1: Idempotent producer (no retry duplicates)");
System.out.println(" Layer 2: Transactional producer (atomic multi-write)");
System.out.println(" Layer 3: read_committed consumer (see only committed)");
System.out.println();
System.out.println("Idempotence is the BASE. Without it, exactly-once fails.");
}
}
Exactly-once = three layers:
Layer 1: Idempotent producer (no retry duplicates)
Layer 2: Transactional producer (atomic multi-write)
Layer 3: read_committed consumer (see only committed)
Idempotence is the BASE. Without it, exactly-once fails.
What are the limitations of idempotence?
Idempotence is scoped to a single producer session and per partition. Across a producer restart the producer id changes, so idempotence alone does not dedup across sessions. It also does not protect against the application choosing to resend different data, nor against duplicate side effects in external systems.
package com.bank.limitation;
public class IdempotenceLimitations {
public static void main(String[] args) {
System.out.println("Idempotence does NOT protect against:");
System.out.println(" 1. Application-level resends (different PID after restart)");
System.out.println(" 2. External system side effects (database insert)");
System.out.println(" 3. Cross-partition duplicates (scope is per-partition)");
System.out.println();
System.out.println("For these, you need:");
System.out.println(" - Transactions (stable transactional.id survives restarts)");
System.out.println(" - Idempotent consumers keyed on business id");
}
}
Idempotence does NOT protect against:
1. Application-level resends (different PID after restart)
2. External system side effects (database insert)
3. Cross-partition duplicates (scope is per-partition)
For these, you need:
- Transactions (stable transactional.id survives restarts)
- Idempotent consumers keyed on business id
Why does this matter in banking?
A duplicated fee, settlement, or transfer is money charged twice or created out of nothing. This is a correctness and compliance failure, not a performance blip. Removing retry duplicates is a hard requirement.
package com.bank.impact;
import java.math.BigDecimal;
public class BankingImpact {
public static void main(String[] args) {
BigDecimal transferAmount = new BigDecimal("50000.00");
System.out.println("Without idempotence:");
System.out.println(" Priya's balance debited: " +
transferAmount.multiply(new BigDecimal("2")) + " INR (DOUBLE!)");
System.out.println();
System.out.println("With idempotence:");
System.out.println(" Priya's balance debited: " +
transferAmount + " INR (correct)");
System.out.println(" Retry duplicate silently dropped by broker");
}
}
Without idempotence:
Priya's balance debited: 100000.00 INR (DOUBLE!)
With idempotence:
Priya's balance debited: 50000.00 INR (correct)
Retry duplicate silently dropped by broker
What are common mistakes with idempotent producers?
The most common mistake is assuming idempotence protects against application-level duplicate sends, such as a retry in your business logic that creates a new producer record with a new sequence number. Another error is setting max.in.flight.requests.per.connection above 5, which silently disables idempotence in older Kafka versions. A third mistake is relying on idempotence across producer restarts, since the producer id changes on restart and the broker treats it as a new producer.
package com.bank.kafka;
public class IdempotenceMistakes {
public static void main(String[] args) {
System.out.println("WRONG: assume app-level resend is deduped");
System.out.println(" producer.send(record); // seq=5, PID=7");
System.out.println(" // App retries with new record");
System.out.println(" producer.send(newRecord); // seq=6, PID=7 -- NOT a dup!");
System.out.println();
System.out.println("WRONG: max.in.flight > 5");
System.out.println(" max.in.flight.requests.per.connection=10");
System.out.println(" // Idempotence silently disabled in Kafka < 3.0");
System.out.println();
System.out.println("CORRECT: enable.idempotence=true, acks=all, in-flight<=5");
System.out.println(" And use transactions for cross-restart guarantees");
}
}
Expected output:
WRONG: assume app-level resend is deduped
producer.send(record); // seq=5, PID=7
// App retries with new record
producer.send(newRecord); // seq=6, PID=7 -- NOT a dup!
WRONG: max.in.flight > 5
max.in.flight.requests.per.connection=10
// Idempotence silently disabled in Kafka < 3.0
CORRECT: enable.idempotence=true, acks=all, in-flight<=5
And use transactions for cross-restart guarantees
What should a senior engineer know?
Idempotence is the non-negotiable base layer. Transactions use a stable transactional id to fence zombies and preserve guarantees across restarts. True end-to-end exactly-once also requires making downstream consumers idempotent on a business id, because Kafka cannot roll back a row already inserted in PostgreSQL. The cost of idempotence is negligible; the cost of transactions is real latency and coordinator overhead.
Visual Overview
Without Idempotence:
Producer ──send──▶ Broker writes (offset 100)
◀──ACK LOST──
Producer ──retry─▶ Broker writes AGAIN (offset 101)
DUPLICATE!
With Idempotence:
Producer ──send──▶ Broker writes (offset 100, PID=7, seq=5)
(PID=7) ◀──ACK LOST──
Producer ──retry─▶ Broker checks: PID=7, seq=5 already written
(seq=5) ACK but NO WRITE
No duplicate!
Exactly-Once Stack:
┌─────────────────────────────────────────┐
│ Layer 3: read_committed consumer │
├─────────────────────────────────────────┤
│ Layer 2: Transactional producer │
│ (atomic multi-partition write) │
├─────────────────────────────────────────┤
│ Layer 1: Idempotent producer (BASE) │
│ (PID + sequence dedup) │
└─────────────────────────────────────────┘
Key Takeaway
Idempotent producers are the foundation of exactly-once semantics because they eliminate the one unavoidable source of duplicates on the write path: retries after a lost acknowledgement. The broker gives the producer a unique producer id, and every record carries a per-partition sequence number. A retried record with a sequence the broker has already seen is acknowledged but never written again.
By itself, idempotence gives exactly-once on a single partition within one producer session. Full end-to-end exactly-once stacks transactions on top for atomic multi-partition writes, ties offset commits to output inside the same transaction, and uses read_committed consumers. The idempotent producer is the base brick; transactions and read-committed complete the structure.
At National Bank of India, every producer runs idempotent so Priya is never double-charged. For the settle-and-charge flow, Rajesh uses a transactional producer that writes the settlement event and the fee event and commits the trade offset in one transaction. The billing consumer reads with read_committed.
Remember one sentence: idempotent producers make the word exactly-once honest on the write side by using a producer id and sequence number to drop retry duplicates, and they are the non-negotiable first layer on which transactions build the complete guarantee.
---