Definition
- Zombie producer -- a transactional producer instance that was presumed dead and replaced, but is actually still alive and may try to write after a network partition heals.
- Epoch fencing -- when a new producer initializes transactions with the same
transactional.id, the coordinator bumps the epoch; any write from the old, lower epoch is rejected. - ProducerFencedException -- the fatal, non-retriable error the zombie receives; the correct response is to close the producer immediately.
- Three identifiers --
transactional.id(stable logical name), producer id (assigned by coordinator), and epoch (incremented on each init). - Read-committed isolation -- consumers with
isolation.level=read_committednever see the zombie's uncommitted records because there is no valid commit marker.
Imagine National Bank of India issues a teller a numbered authorization stamp to approve transfers. The teller walks into the vault, the door jams (network partition), and the branch promotes a replacement with a higher stamp number, declaring the old number void. Later the original teller walks out and tries to approve a transfer with the old stamp. The clerk checks the stamp number, sees it is lower than the current valid one, and refuses it. That stamp-number check is epoch fencing.
Understanding the Concepts
What is a zombie producer?
A zombie producer is a transactional producer instance that everyone assumed was dead and replaced, but which is actually still alive and might try to write again once a network partition heals. Without fencing, it could commit stale writes and cause duplicates or data corruption.
package com.bank.kafka;
public class ZombieScenario {
public static void main(String[] args) {
System.out.println("Zombie producer timeline:");
System.out.println(" t=0s: Producer A starts, gets epoch 5");
System.out.println(" t=10s: Producer A begins transaction for Priya's credit");
System.out.println(" t=15s: Network partition isolates Producer A");
System.out.println(" t=30s: Orchestrator starts Producer B, same transactional.id");
System.out.println(" t=30s: Coordinator bumps epoch to 6, aborts A's transaction");
System.out.println(" t=60s: Network heals, Producer A (zombie) tries to commit");
System.out.println(" t=60s: Broker rejects: epoch 5 < current epoch 6");
System.out.println(" t=60s: Producer A gets ProducerFencedException -- must close");
}
}
Expected output:
Zombie producer timeline:
t=0s: Producer A starts, gets epoch 5
t=10s: Producer A begins transaction for Priya's credit
t=15s: Network partition isolates Producer A
t=30s: Orchestrator starts Producer B, same transactional.id
t=30s: Coordinator bumps epoch to 6, aborts A's transaction
t=60s: Network heals, Producer A (zombie) tries to commit
t=60s: Broker rejects: epoch 5 < current epoch 6
t=60s: Producer A gets ProducerFencedException -- must close
How does epoch fencing work?
Three identifiers make fencing possible. The transactional.id is a stable logical name you choose. The producer id is assigned by the coordinator. The epoch is incremented every time a producer calls initTransactions() with that transactional id. Every write carries the epoch, and the broker checks it.
package com.bank.kafka;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class TransactionalProducerConfig {
public static Properties create() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// Transactional id: stable name for this producer role
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "settlement-tx-producer-1");
// Idempotence is implied by transactions
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
System.out.println("Transactional producer configured");
System.out.println(" transactional.id: settlement-tx-producer-1");
System.out.println(" On initTransactions(): coordinator assigns producer id + epoch");
return props;
}
public static void main(String[] args) {
create();
}
}
Expected output:
Transactional producer configured
transactional.id: settlement-tx-producer-1
On initTransactions(): coordinator assigns producer id + epoch
How do you handle ProducerFencedException in code?
The exception is fatal and non-retriable by design. The correct response is to close the producer immediately.
package com.bank.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.KafkaException;
import org.apache.kafka.common.errors.ProducerFencedException;
import org.apache.kafka.common.errors.OutOfOrderSequenceException;
import org.apache.kafka.common.errors.AuthorizationException;
import org.apache.kafka.common.serialization.StringSerializer;
import java.math.BigDecimal;
import java.util.Properties;
public class FencedProducerHandling {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "settlement-tx-producer-1");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();
try {
producer.beginTransaction();
producer.send(new ProducerRecord<>("settlements", "SAV3001",
"CREDIT " + new BigDecimal("50000")));
producer.commitTransaction();
System.out.println("Transaction committed for Priya's credit");
} catch (ProducerFencedException | OutOfOrderSequenceException
| AuthorizationException fatal) {
// Zombie detected or unrecoverable -- MUST close
System.out.println("FATAL: " + fatal.getClass().getSimpleName());
System.out.println(" This producer is a zombie -- closing immediately");
producer.close();
} catch (KafkaException e) {
// Transient error -- safe to abort and retry
System.out.println("Transient error: " + e.getMessage());
producer.abortTransaction();
}
}
}
Expected output (normal case):
Transaction committed for Priya's credit
Expected output (zombie case):
FATAL: ProducerFencedException
This producer is a zombie -- closing immediately
How does read-committed isolation protect consumers?
Consumers with isolation.level=read_committed only see records up to the last stable offset. Aborted or never-committed transactions are skipped, so the zombie's uncommitted records are never visible.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;
public class ReadCommittedConsumer {
public static Properties create() {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "settlement-verifier");
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
System.out.println("Consumer with read_committed isolation");
System.out.println(" Sees only committed transactions");
System.out.println(" Zombie's uncommitted records: invisible");
return props;
}
public static void main(String[] args) {
create();
}
}
Expected output:
Consumer with read_committed isolation
Sees only committed transactions
Zombie's uncommitted records: invisible
How is epoch fencing different from idempotent dedup?
| Mechanism | Scope | Protects Against |
|---|---|---|
| Idempotent producer (sequence numbers) | Single session | Client retry duplicates |
| Epoch fencing (transactional.id + epoch) | Across instances | Zombie producer writes |
| Application-level idempotency key | End-to-end | Application-level duplicates |
What are the rules for transactional.id?
Each logical producer role needs its own stable, unique transactional.id. If two genuinely different producers share one, they will constantly fence each other.
package com.bank.kafka;
public class TransactionalIdRules {
public static void main(String[] args) {
// CORRECT: unique id per producer role
System.out.println("CORRECT: settlement-tx-producer-1 (pod 1)");
System.out.println("CORRECT: settlement-tx-producer-2 (pod 2)");
System.out.println(" Each pod has its own transactional.id");
// WRONG: two different roles sharing one id
System.out.println("WRONG: both settlement and analytics use 'shared-tx-id'");
System.out.println(" They fence each other constantly -- neither makes progress");
}
}
Expected output:
CORRECT: settlement-tx-producer-1 (pod 1)
CORRECT: settlement-tx-producer-2 (pod 2)
Each pod has its own transactional.id
WRONG: both settlement and analytics use 'shared-tx-id'
They fence each other constantly -- neither makes progress
What about cross-system exactly-once?
Kafka's fencing only protects writes inside Kafka. If the producer also writes to a database, epoch fencing does not roll back the database write. For cross-system atomicity, use the transactional outbox pattern.
What should a senior engineer know?
The transaction.timeout.ms setting bounds how long a producer can hold an open transaction. If exceeded, the coordinator aborts it, which prevents a stuck producer from blocking the last stable offset indefinitely. Idempotence is required for transactions and provides per-session dedup via sequence numbers, while epoch fencing protects across sessions.
Visual Overview
Epoch Fencing Timeline
=======================
Producer A Coordinator Producer B
────────── ─────────── ──────────
initTransactions()
────────────────────► assigns PID=1, epoch=5
beginTransaction()
send("CREDIT SAV3001")
│
═══ NETWORK PARTITION ═══
│
│ Orchestrator detects A is gone
│ initTransactions()
│ ────────────────────────────► same tx.id
│ bumps epoch to 6
│ aborts A's open transaction
│ assigns PID=1, epoch=6 to B
│
│ beginTransaction()
│ send("CREDIT SAV3001")
│ commitTransaction()
│
═══ NETWORK HEALS ═══
│
commitTransaction()
────────────────────► epoch 5 < current 6
REJECT!
◄────────────────────
ProducerFencedException
producer.close()
Result: Priya credited exactly once by Producer B
Zombie A's commit rejected
Key Takeaway
A zombie producer is a transactional instance that everyone assumed was dead, but which comes back after a network partition heals and tries to write. Kafka handles this with epoch fencing: the transactional id combined with a monotonically increasing epoch. When a replacement producer calls initTransactions with the same id, the coordinator bumps the epoch and aborts the old transaction.
When the zombie tries to commit at its old epoch, the broker rejects it with ProducerFencedException. This is fatal by design -- the zombie must close. Consumers with read-committed isolation never see the zombie's uncommitted records.
In my National Bank of India settlement service, epoch fencing stops a double credit. If the pod crediting Priya loses the network and a replacement takes over, the replacement bumps the epoch, so the old pod's commit is fenced and Priya is credited exactly once.
Remember one sentence: epoch fencing rejects a zombie producer's stale writes by checking the epoch on every commit, so a replaced instance can never land stale data after a network partition heals.
---