Definition
- Idempotent producer -- enabled with
enable.idempotence=true, uses producer id, epoch, and per-partition sequence numbers to deduplicate the client's own retries on the broker. - What it solves -- if the broker writes a record but the ack is lost, the client retries and the broker discards the duplicate based on the sequence number.
- What it does NOT solve -- two separate
send()calls from your application code get different sequence numbers and are both stored; the broker cannot tell they mean the same thing. - Application-level fix -- use an idempotency key (e.g., transfer id) with a dedup store (database unique constraint or processed-ids table) on the consumer side.
- Layered model -- idempotent producer handles transport retries, Kafka transactions handle within-Kafka atomicity, application-level dedup handles business duplicates.
Imagine Rajesh mails a transfer instruction to National Bank of India. The post office guarantees that if it resends the same sealed envelope because of a delivery hiccup, the bank only processes it once, because each envelope has a tracking number. But if Rajesh himself writes a brand new letter asking for the same transfer, those are two different envelopes with two different tracking numbers, and the bank processes both. Stopping that requires Rajesh's own reference number and the bank rejecting a reference it already honored.
Understanding the Concepts
What does the idempotent producer solve?
Without idempotence, a common duplicate scenario is: the producer sends a record, the broker writes it, but the ack is lost. The producer retries, and the broker now has the record twice. With enable.idempotence=true, the broker deduplicates using sequence numbers.
package com.bank.kafka;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class IdempotentProducerConfig {
public static Properties create() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
// Deduplicates client retries only -- not application duplicates
System.out.println("Idempotent producer enabled");
System.out.println(" Deduplicates: client retransmissions (same sequence number)");
System.out.println(" Does NOT deduplicate: two separate send() calls");
return props;
}
public static void main(String[] args) {
create();
}
}
Expected output:
Idempotent producer enabled
Deduplicates: client retransmissions (same sequence number)
Does NOT deduplicate: two separate send() calls
What does it NOT protect against?
The sequence number is assigned per send() call. Two separate sends for the same business event get different sequence numbers. The broker stores both.
package com.bank.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class ApplicationDuplicateDemo {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
// Upstream HTTP request retried -- settle() runs twice
producer.send(new ProducerRecord<>("settlements", "SAV2001",
"SETTLE TXF-123 50000")).get();
System.out.println("First send: sequence N -- stored");
// Same business event, NEW send() call -- different sequence
producer.send(new ProducerRecord<>("settlements", "SAV2001",
"SETTLE TXF-123 50000")).get();
System.out.println("Second send: sequence N+1 -- ALSO stored");
System.out.println(" Two records on broker! Idempotent producer did NOT help.");
System.out.println(" Priya gets double credit unless consumer deduplicates.");
}
}
}
Expected output:
First send: sequence N -- stored
Second send: sequence N+1 -- ALSO stored
Two records on broker! Idempotent producer did NOT help.
Priya gets double credit unless consumer deduplicates.
How do you fix application-level duplicates?
Use an idempotency key with consumer-side deduplication. The consumer checks a processed-ids store before applying the effect.
package com.bank.kafka;
import java.math.BigDecimal;
import java.util.HashSet;
import java.util.Set;
public class ConsumerSideDedup {
// Simulates a database table with unique constraint on transfer_id
private static final Set<String> processedTransfers = new HashSet<>();
public static void applySettlement(String transferId, String accountId, BigDecimal amount) {
if (processedTransfers.contains(transferId)) {
System.out.println("SKIP: " + transferId + " already processed -- no double credit");
return;
}
// Apply in same DB transaction: credit + insert processed id
processedTransfers.add(transferId);
System.out.println("APPLY: " + transferId + " credit " + amount + " to " + accountId);
}
public static void main(String[] args) {
applySettlement("TXF-123", "SAV3001", new BigDecimal("50000"));
applySettlement("TXF-123", "SAV3001", new BigDecimal("50000")); // Duplicate
applySettlement("TXF-124", "SAV2001", new BigDecimal("25000")); // Different transfer
}
}
Expected output:
APPLY: TXF-123 credit 50000 to SAV3001
SKIP: TXF-123 already processed -- no double credit
APPLY: TXF-124 credit 25000 to SAV2001
How does the layered dedup model work?
| Layer | Mechanism | Protects Against |
|---|---|---|
| Idempotent producer | Sequence numbers per send | Client retry duplicates |
| Kafka transactions | Atomic commit of offsets + output | Within-Kafka exactly-once |
| Application idempotency key | Business id + dedup store | Application-level duplicates |
| Transactional outbox | Single DB transaction | Cross-system exactly-once |
What about the transactional outbox pattern?
When a publish must be exactly-once together with a database write, use the transactional outbox so there is a single source of truth for whether the event exists.
package com.bank.kafka;
import java.math.BigDecimal;
public class OutboxPattern {
// @Transactional -- one database transaction
public static void transferWithOutbox(String from, String to, BigDecimal amount) {
// debit(from, amount);
// credit(to, amount);
// outboxRepo.save(new OutboxEvent("TXF-125", "TransferCompleted", payload));
System.out.println("DB transaction: debit + credit + outbox row");
System.out.println(" No Kafka call here -- relay publishes from outbox");
System.out.println(" Single source of truth: the database");
}
public static void main(String[] args) {
transferWithOutbox("SAV2001", "SAV3001", new BigDecimal("50000"));
}
}
Expected output:
DB transaction: debit + credit + outbox row
No Kafka call here -- relay publishes from outbox
Single source of truth: the database
What about producer restart and session scope?
Idempotence is scoped to a single producer session. If the producer restarts and gets a new producer id, sequence numbers reset and the broker cannot correlate. This is another reason it cannot be your business dedup mechanism.
What are common mistakes when relying on idempotent producers?
The most common mistake is equating enable.idempotence=true with end-to-end exactly-once. Idempotence only deduplicates broker-level retries within one session. If your application logic resends a message with a new sequence number, the broker treats it as a new record. Another mistake is ignoring external side effects: even if the broker deduplicates the Kafka write, an HTTP call or database insert triggered during processing still executes twice.
package com.bank.kafka;
public class IdempotenceScopeMistakes {
public static void main(String[] args) {
System.out.println("WRONG: assume idempotence = end-to-end exactly-once");
System.out.println(" enable.idempotence=true does NOT cover:");
System.out.println(" - App-level resends (new sequence number)");
System.out.println(" - External calls (HTTP, DB insert)");
System.out.println(" - Cross-session duplicates (PID changes on restart)");
System.out.println();
System.out.println("CORRECT: layer defenses");
System.out.println(" Broker side: idempotent producer (retry dedup)");
System.out.println(" App side: transactional outbox (app-level dedup)");
System.out.println(" Consumer side: idempotent handler (business-key dedup)");
}
}
Expected output:
WRONG: assume idempotence = end-to-end exactly-once
enable.idempotence=true does NOT cover:
- App-level resends (new sequence number)
- External calls (HTTP, DB insert)
- Cross-session duplicates (PID changes on restart)
CORRECT: layer defenses
Broker side: idempotent producer (retry dedup)
App side: transactional outbox (app-level dedup)
Consumer side: idempotent handler (business-key dedup)
What should a senior engineer know?
Kafka transactions build on idempotence for within-Kafka exactly-once, but even transactions do not know that two sends represent the same business event. The most reliable dedup is end-to-end, on the consumer side, keyed by the business id. The idempotent producer is necessary but never sufficient for real-world duplicate prevention.
Visual Overview
Idempotent Producer Scope ========================== Client retry (SAME batch, SAME sequence number): ┌──────────┐ send(seq=5) ┌──────────┐ │ Producer │ ──────────────► │ Broker │ writes, ack lost │ │ ──────────────► │ │ retry seq=5 │ │ │ Already │ seq=5 seen │ │ ◄────────────── │ DEDUP! │ ack without write └──────────┘ └──────────┘ ✓ Idempotent producer handles this Application duplicate (DIFFERENT sends, DIFFERENT sequence numbers): ┌──────────┐ send(seq=5) ┌──────────┐ │ Producer │ ──────────────► │ Broker │ writes at seq=5 │ │ send(seq=6) │ │ writes at seq=6 │ (HTTP │ ──────────────► │ TWO │ both stored! │ retry) │ │ RECORDS │ └──────────┘ └──────────┘ ✗ Idempotent producer CANNOT help ✓ Consumer-side dedup with transfer id
Key Takeaway
No, the idempotent producer does not protect against application-level duplicate publishes. It only deduplicates the client's own retries within a single session using sequence numbers. Two separate send calls for the same business event get different sequence numbers, and the broker stores both.
The correct fix is an application-level idempotency key with consumer-side dedup. The consumer checks a processed-transfers table before applying the credit, so a duplicate is detected and skipped.
In my National Bank of India transfer API, Rajesh's app timed out and retried the HTTP call. Both requests published the same settlement, and the idempotent producer was powerless because they were separate sends. Consumer-side dedup on the transfer id caught it.
Remember one sentence: the idempotent producer deduplicates transport-layer retries, not application-level duplicates -- for those you need a business idempotency key and a dedup store on the consumer side.
---