Definition
- Idempotent producer -- uses a broker-assigned producer id and per-partition sequence numbers to deduplicate retries; requires
enable.idempotence=true,acks=all, and at most five in-flight requests. - Transactional producer -- builds on idempotence and adds a stable
transactional.idto atomically commit writes across multiple partitions and topics; fences zombie producers via epoch bumping. - Producer id + sequence number -- the mechanism the broker uses to detect duplicate writes from retries within one producer session.
- transactional.id -- a stable identifier that survives producer restarts and lets the transaction coordinator fence out old zombie instances.
- sendOffsetsToTransaction -- commits consumer offsets inside the same transaction as output records, making the consume-process-produce loop exactly-once.
- isolation.level=read_committed -- a consumer setting that skips aborted and in-flight transaction records, only delivering committed results.
Think of Rajesh sending a transfer instruction to the bank. An idempotent producer is like each instruction having a unique serial number, so if Rajesh's phone retries on a timeout, the bank sees the same serial and records it only once -- no accidental double transfer. A transactional producer is like a notary who wraps several instructions into one sealed envelope: either all of them are filed together or none are, and clerks can be told to ignore envelopes that were never sealed. Idempotence stops one instruction being counted twice; transactions make a whole batch of instructions atomic.
Understanding the Concepts
What is an idempotent producer?
The idempotent producer solves a narrow but critical problem. When a producer sends a record and the broker writes it but the acknowledgement is lost on the network, the producer retries. Without idempotence, that retry becomes a duplicate. With idempotence enabled, the broker assigns the producer a unique producer id and tracks a sequence number per partition. If it sees a record whose sequence it has already written, it acknowledges it again but does not write it twice.
package com.bank.producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;
public class IdempotentProducerDemo {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
producer.send(new ProducerRecord<>("transfers", "SAV2001",
"{\"amount\":50000,\"type\":\"DEBIT\"}"));
System.out.println("Idempotent producer: retry duplicates are dropped");
}
}
}
Idempotent producer: retry duplicates are dropped
What is a transactional producer?
A transactional producer is a superset of the idempotent producer. It requires a transactional.id, which lets Kafka fence out a previous instance of the same producer so a zombie cannot keep writing. You call initTransactions() once, then wrap work in beginTransaction(), sends, and commitTransaction() or abortTransaction().
package com.bank.producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;
public class TransactionalProducerDemo {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "settlement-tx-1");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.ACKS_CONFIG, "all");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();
try {
producer.beginTransaction();
// Debit from SAV2001
producer.send(new ProducerRecord<>("debits", "SAV2001",
"{\"amount\":50000}"));
// Credit to SAV3045
producer.send(new ProducerRecord<>("credits", "SAV3045",
"{\"amount\":50000}"));
producer.commitTransaction();
System.out.println("Both debit and credit committed atomically");
} catch (Exception e) {
producer.abortTransaction();
System.out.println("Transaction aborted: " + e.getMessage());
} finally {
producer.close();
}
}
}
Both debit and credit committed atomically
How do the two compare? (comparison table)
| Feature | Idempotent Producer | Transactional Producer |
|---|---|---|
| Deduplication | Yes, per partition per session | Yes (inherits idempotence) |
| Scope | One partition, one session | Multiple partitions and topics |
| Atomicity | No | Yes, all-or-nothing |
| Zombie fencing | No (producer id changes on restart) | Yes (epoch bumping via transactional.id) |
| Offset commit | Separate | Can be inside the transaction |
| Performance cost | Minimal | Higher (two-phase commit) |
| Config | enable.idempotence=true | transactional.id + enable.idempotence |
How does sendOffsetsToTransaction work?
The crucial capability of a transactional producer is sendOffsetsToTransaction, which commits the consumer offsets inside the same transaction as the output records. This makes the consume-process-produce loop exactly-once.
package com.bank.eos;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.TopicPartition;
import java.time.Duration;
import java.util.HashMap;
import java.util.Map;
public class ExactlyOnceLoop {
public static void processLoop(KafkaConsumer<String, String> consumer,
KafkaProducer<String, String> producer) {
producer.initTransactions();
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
if (records.isEmpty()) continue;
producer.beginTransaction();
records.forEach(record -> {
// Process and produce output
producer.send(new ProducerRecord<>("settlements", record.key(),
"{\"settled\":true}"));
});
// Commit offsets INSIDE the transaction
Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
records.partitions().forEach(tp -> {
long lastOffset = records.records(tp)
.get(records.records(tp).size() - 1).offset();
offsets.put(tp, new OffsetAndMetadata(lastOffset + 1));
});
producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
producer.commitTransaction();
}
}
}
How does the consumer participate?
Consumers participate through isolation.level. With read_committed, a consumer only sees records from committed transactions. With the default read_uncommitted, it sees everything immediately.
package com.bank.consumer;
import java.util.Properties;
public class ReadCommittedConsumer {
public static Properties config() {
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092");
props.put("group.id", "ledger-service");
props.put("isolation.level", "read_committed");
props.put("enable.auto.commit", "false");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
return props;
}
public static void main(String[] args) {
System.out.println("read_committed: only sees committed transaction records");
System.out.println("read_uncommitted: sees all records immediately");
}
}
read_committed: only sees committed transaction records
read_uncommitted: sees all records immediately
What are common mistakes?
package com.bank.mistake;
public class IdempotentVsTransactionalMistakes {
public static void main(String[] args) {
// WRONG: thinking idempotence prevents all duplicates
System.out.println("WRONG: idempotence prevents application-level resends");
System.out.println(" Reality: it only prevents RETRY duplicates within one session");
System.out.println();
// WRONG: using transactions everywhere
System.out.println("WRONG: wrapping every produce in a transaction");
System.out.println(" Reality: transactions add latency and coordinator overhead");
System.out.println(" Use only where atomicity is a hard business requirement");
System.out.println();
// WRONG: forgetting read_committed on consumers
System.out.println("WRONG: transactional producer + read_uncommitted consumer");
System.out.println(" Reality: consumer sees aborted records too!");
}
}
WRONG: idempotence prevents application-level resends
Reality: it only prevents RETRY duplicates within one session
WRONG: wrapping every produce in a transaction
Reality: transactions add latency and coordinator overhead
Use only where atomicity is a hard business requirement
WRONG: transactional producer + read_uncommitted consumer
Reality: consumer sees aborted records too!
When should you use each?
- Idempotent producer: always on (it is the default). It removes retry duplicates for free.
- Transactional producer: when you need atomic writes across multiple partitions or topics, or exactly-once read-process-write.
- In banking: idempotence for every producer. Transactions for money movement spanning a debit topic and a credit topic, so consumers never see a lonely debit.
What should a senior engineer know?
Idempotence is cheap and should essentially always be enabled. Transactions add a two-phase commit through a transaction coordinator, extra latency, and transaction markers in the log, so reserve them for hard atomicity requirements. In Spring Kafka, setting a transactionIdPrefix on the producer factory turns the KafkaTemplate transactional. External side effects (database writes) are not covered by Kafka transactions, so make those idempotent or use the outbox pattern.
Visual Overview
Idempotent Producer
┌──────────┐ ┌──────────┐
│ Producer │──send──▶│ Broker │
│ PID=100 │ │ seq=5 OK │
│ seq=5 │ └──────────┘
│ │ │
│ (ack lost on network) │
│ │ │
│ retry │ ┌──────────┐
│ seq=5 │──send──▶│ Broker │
│ │ │ seq=5 │
│ │ │ DUPLICATE│
│ │ │ ack, no │
│ │ │ write │
└──────────┘ └──────────┘
Transactional Producer
┌──────────┐ ┌──────────┐ ┌──────────┐
│ begin() │ │ debits │ │ credits │
│ │ │ P0: debit│ │ P2: cred │
│ send() │───▶│ │ │ │
│ send() │───▶│ │───▶│ │
│ commit() │ │ COMMIT │ │ COMMIT │
└──────────┘ │ marker │ │ marker │
└──────────┘ └──────────┘
Both visible atomically
Key Takeaway
An idempotent producer and a transactional producer solve two different reliability problems, and the transactional one is a superset. The idempotent producer fixes the duplicate-from-retry problem using a producer id and per-partition sequence numbers, so a retried record is acknowledged but not stored twice. It is cheap, it is the default, and it should be on everywhere.
The transactional producer builds on idempotence and adds atomicity across multiple partitions and topics. It uses a stable transactional id that fences out zombie producers, and it can commit consumer offsets inside the same transaction as output records, which is how exactly-once consume-process-produce works. Consumers must set isolation level to read_committed to see only committed results.
At National Bank of India, every producer runs idempotent so Priya's transfer is never double-counted. For money movement that spans a debit topic and a credit topic, Rajesh uses a transactional producer so consumers never see a lonely debit without the matching credit.
Remember one sentence: idempotence is table stakes for correctness under retries, while transactions are a deliberate, more expensive tool you apply where all-or-nothing semantics are a hard business requirement.
---