Definition
- Exactly-once processing guarantees that each input record produces its effect on the output and state store exactly one time, even if the application crashes and restarts.
- processing.guarantee is the StreamsConfig property that controls the guarantee level; set it to
exactly_once_v2for exactly-once or leave the defaultat_least_once. - Changelog topic is a compacted Kafka topic that stores every state change for a state store; it serves as the durable backup for recovery.
- State store recovery is the process of replaying the changelog topic to rebuild the local RocksDB state store after a crash or rebalance.
- Standby replicas are warm copies of state stores maintained on other Kafka Streams instances so that recovery after rebalancing is near-instant instead of replaying the full changelog.
Think of National Bank of India processing a fund transfer from Priya's account to Rajesh's account. Exactly-once is like an atomic ledger entry: debiting Priya, crediting Rajesh, and recording the transaction in the audit log either all happen or none of them happen. If the clerk's pen runs out of ink mid-entry, the entire entry is void and retried from scratch. The changelog topic is the audit log. If the ledger book is lost, the bank rebuilds it by replaying every entry in the audit log from the beginning.
Understanding the Concepts
What does exactly-once processing mean?
Exactly-once processing means that for every input record, its effects appear in the output topic and the state store exactly one time. Without exactly-once, a crash after writing the output but before committing the input offset causes the record to be reprocessed, producing a duplicate output.
- The three guarantees in Kafka Streams are:
- At-most-once: process the record, commit the offset, then write output. If the write fails, the record is lost.
- At-least-once (default): write the output, then commit the offset. If the commit fails, the record is reprocessed, producing a duplicate.
- Exactly-once: write output, update state, and commit offset in a single atomic transaction.
package com.bank.streams;
public class ExactlyOnceExplained {
public static void main(String[] args) {
System.out.println("Processing guarantees:");
System.out.println();
System.out.println("at_least_once (default):");
System.out.println(" 1. Read record from input");
System.out.println(" 2. Process and write to output");
System.out.println(" 3. Commit input offset");
System.out.println(" If crash between 2 and 3: record reprocessed, duplicate output");
System.out.println();
System.out.println("exactly_once_v2:");
System.out.println(" 1. Begin transaction");
System.out.println(" 2. Read record from input");
System.out.println(" 3. Process");
System.out.println(" 4. Write to output (in transaction)");
System.out.println(" 5. Write to changelog (in transaction)");
System.out.println(" 6. Commit offset (in transaction)");
System.out.println(" 7. Commit transaction (all or nothing)");
System.out.println(" If crash: transaction rolls back, no duplicates");
}
}
Expected output:
Processing guarantees:
at_least_once (default):
1. Read record from input
2. Process and write to output
3. Commit input offset
If crash between 2 and 3: record reprocessed, duplicate output
exactly_once_v2:
1. Begin transaction
2. Read record from input
3. Process
4. Write to output (in transaction)
5. Write to changelog (in transaction)
6. Commit offset (in transaction)
7. Commit transaction (all or nothing)
If crash: transaction rolls back, no duplicates
How do you enable exactly-once in Kafka Streams?
Set the processing.guarantee config to exactly_once_v2. This tells Kafka Streams to use Kafka transactions for the read-process-write cycle.
package com.bank.streams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
public class ExactlyOnceConfig {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "balance-updater");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
// Enable exactly-once processing
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
StreamsConfig.EXACTLY_ONCE_V2);
System.out.println("Exactly-once configuration:");
System.out.println(" processing.guarantee = exactly_once_v2");
System.out.println();
System.out.println("Requirements:");
System.out.println(" Kafka broker version >= 2.5 (for exactly_once_v2)");
System.out.println(" min.insync.replicas >= 2 on output topics");
System.out.println(" replication.factor >= 3 on output topics");
System.out.println();
System.out.println("Previous versions:");
System.out.println(" exactly_once (deprecated, one producer per task)");
System.out.println(" exactly_once_v2 (recommended, one producer per thread)");
}
}
Expected output:
Exactly-once configuration:
processing.guarantee = exactly_once_v2
Requirements:
Kafka broker version >= 2.5 (for exactly_once_v2)
min.insync.replicas >= 2 on output topics
replication.factor >= 3 on output topics
Previous versions:
exactly_once (deprecated, one producer per task)
exactly_once_v2 (recommended, one producer per thread)
How does state store recovery work?
Every state store in Kafka Streams is backed by a changelog topic. When you call aggregate(), count(), or reduce() with a Materialized store, Kafka Streams creates a changelog topic named {application-id}-{store-name}-changelog. Every state change is written to this topic.
On recovery (after crash or rebalance), Kafka Streams reads the changelog topic from the beginning and replays every record to rebuild the local RocksDB store. This process takes time proportional to the state size.
package com.bank.streams;
public class StateStoreRecovery {
public static void main(String[] args) {
System.out.println("State store recovery process:");
System.out.println();
System.out.println("Normal operation:");
System.out.println(" 1. Process record");
System.out.println(" 2. Update local RocksDB store");
System.out.println(" 3. Write change to changelog topic");
System.out.println();
System.out.println("After crash:");
System.out.println(" 1. Application restarts");
System.out.println(" 2. Kafka Streams reads changelog from beginning");
System.out.println(" 3. Replays every record into fresh RocksDB store");
System.out.println(" 4. Store is now identical to pre-crash state");
System.out.println(" 5. Resume processing from last committed offset");
System.out.println();
System.out.println("Recovery time depends on state size:");
System.out.println(" 1 GB state = ~30 seconds recovery");
System.out.println(" 10 GB state = ~5 minutes recovery");
}
}
Expected output:
State store recovery process:
Normal operation:
1. Process record
2. Update local RocksDB store
3. Write change to changelog topic
After crash:
1. Application restarts
2. Kafka Streams reads changelog from beginning
3. Replays every record into fresh RocksDB store
4. Store is now identical to pre-crash state
5. Resume processing from last committed offset
Recovery time depends on state size:
1 GB state = ~30 seconds recovery
10 GB state = ~5 minutes recovery
How do at_least_once and exactly_once_v2 compare?
| Feature | at_least_once (default) | exactly_once_v2 |
|---|---|---|
| Duplicate output possible | Yes (on crash/rebalance) | No |
| State store consistency | May have stale data after crash | Always consistent |
| Kafka transactions used | No | Yes |
| Throughput overhead | None | 10-30% reduction |
| Broker version required | Any | >= 2.5 |
| Producer per | Thread | Thread |
| Commit interval | commit.interval.ms (30s default) | commit.interval.ms (100ms default) |
| Use case | High throughput, duplicates OK | Financial, exactly-once required |
package com.bank.streams;
public class GuaranteeComparison {
public static void main(String[] args) {
System.out.println("at_least_once:");
System.out.println(" Throughput: 100,000 records/sec");
System.out.println(" Duplicates: possible on crash");
System.out.println(" Use case: metrics, analytics, logging");
System.out.println();
System.out.println("exactly_once_v2:");
System.out.println(" Throughput: 70,000-90,000 records/sec (10-30% less)");
System.out.println(" Duplicates: never");
System.out.println(" Use case: banking, payments, financial ledgers");
System.out.println();
System.out.println("For National Bank of India: always use exactly_once_v2");
System.out.println("A duplicate debit of 50,000 rupees is unacceptable.");
}
}
Expected output:
at_least_once:
Throughput: 100,000 records/sec
Duplicates: possible on crash
Use case: metrics, analytics, logging
exactly_once_v2:
Throughput: 70,000-90,000 records/sec (10-30% less)
Duplicates: never
Use case: banking, payments, financial ledgers
For National Bank of India: always use exactly_once_v2
A duplicate debit of 50,000 rupees is unacceptable.
What are standby replicas?
Standby replicas are warm copies of state stores maintained on other Kafka Streams instances. They continuously read from the changelog topic and keep a local copy of the state. When a rebalance moves a partition to a new instance, the standby replica is already up to date, so recovery is near-instant.
Configure standby replicas with num.standby.replicas. The default is 0 (no standbys).
package com.bank.streams;
import org.apache.kafka.streams.StreamsConfig;
import java.util.Properties;
public class StandbyReplicasConfig {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
System.out.println("Standby replicas configuration:");
System.out.println(" num.standby.replicas = 1");
System.out.println();
System.out.println("Without standbys:");
System.out.println(" Rebalance -> replay changelog from scratch -> 5 min recovery");
System.out.println();
System.out.println("With 1 standby:");
System.out.println(" Rebalance -> standby already has state -> seconds recovery");
System.out.println();
System.out.println("Trade-off:");
System.out.println(" Standbys consume extra disk and network bandwidth");
System.out.println(" 1 standby = 2x disk usage per state store");
System.out.println(" Worth it for large state stores (> 1 GB)");
}
}
Expected output:
Standby replicas configuration:
num.standby.replicas = 1
Without standbys:
Rebalance -> replay changelog from scratch -> 5 min recovery
With 1 standby:
Rebalance -> standby already has state -> seconds recovery
Trade-off:
Standbys consume extra disk and network bandwidth
1 standby = 2x disk usage per state store
Worth it for large state stores (> 1 GB)
What is a complete banking project for an idempotent balance updater?
This project reads deposit and withdrawal events and maintains a running balance per account in a state store. It uses exactly_once_v2 to ensure no duplicate balance updates.
Project structure
balance-updater/
├── pom.xml
└── src/
├── main/
│ └── java/
│ └── com/
│ └── bank/
│ └── streams/
│ ├── BalanceUpdaterApp.java
│ ├── BalanceUpdaterTopology.java
│ └── BalanceEvent.java
└── test/
└── java/
└── com/
└── bank/
└── streams/
└── BalanceUpdaterTopologyTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>balance-updatersyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
BalanceEvent.java
package com.bank.streams;
import java.math.BigDecimal;
public class BalanceEvent {
private String accountId;
private String type;
private BigDecimal amount;
public BalanceEvent() {}
public BalanceEvent(String accountId, String type, BigDecimal amount) {
this.accountId = accountId;
this.type = type;
this.amount = amount;
}
public String getAccountId() { return accountId; }
public String getType() { return type; }
public BigDecimal getAmount() { return amount; }
@Override
public String toString() {
return accountId + "|" + type + "|" + amount;
}
public static BalanceEvent fromString(String value) {
String[] parts = value.split("\\|");
return new BalanceEvent(parts[0], parts[1], new BigDecimal(parts[2]));
}
}
BalanceUpdaterTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;
public class BalanceUpdaterTopology {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> events = builder.stream("balance-events");
// Aggregate: maintain running balance per account
KTable<String, String> balances = events
.groupByKey()
.aggregate(
() -> "0",
(accountId, eventStr, currentBalance) -> {
BalanceEvent event = BalanceEvent.fromString(eventStr);
java.math.BigDecimal balance = new java.math.BigDecimal(currentBalance);
if ("DEPOSIT".equals(event.getType())) {
balance = balance.add(event.getAmount());
} else if ("WITHDRAWAL".equals(event.getType())) {
balance = balance.subtract(event.getAmount());
}
return balance.toPlainString();
},
Materialized.as("account-balances")
);
balances.toStream()
.mapValues((accountId, balance) -> "BALANCE|" + accountId + "|" + balance)
.to("balance-updates");
return builder.build();
}
}
BalanceUpdaterApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class BalanceUpdaterApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "balance-updater");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);
props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
KafkaStreams streams = new KafkaStreams(BalanceUpdaterTopology.build(), props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
}));
try {
streams.start();
System.out.println("Balance updater started (exactly-once)");
System.out.println(" Input: balance-events");
System.out.println(" Output: balance-updates");
System.out.println(" Store: account-balances");
System.out.println(" Standbys: 1");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
BalanceUpdaterTopologyTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.state.KeyValueStore;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class BalanceUpdaterTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
try (TopologyTestDriver driver = new TopologyTestDriver(BalanceUpdaterTopology.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"balance-events", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> output = driver.createOutputTopic(
"balance-updates", new StringDeserializer(), new StringDeserializer());
// Priya deposits 50,000
input.pipeInput("SAV2001", "SAV2001|DEPOSIT|50000");
// Priya deposits another 30,000
input.pipeInput("SAV2001", "SAV2001|DEPOSIT|30000");
// Priya withdraws 10,000
input.pipeInput("SAV2001", "SAV2001|WITHDRAWAL|10000");
// Rajesh deposits 100,000
input.pipeInput("CUR3001", "CUR3001|DEPOSIT|100000");
System.out.println("Balance updates:");
while (!output.isEmpty()) {
System.out.println(" " + output.readValue());
}
// Assert state store
KeyValueStore<String, String> store = driver.getKeyValueStore("account-balances");
System.out.println();
System.out.println("State store:");
System.out.println(" SAV2001 balance: " + store.get("SAV2001"));
System.out.println(" CUR3001 balance: " + store.get("CUR3001"));
}
}
}
Expected output:
Balance updates:
BALANCE|SAV2001|50000
BALANCE|SAV2001|80000
BALANCE|SAV2001|70000
BALANCE|CUR3001|100000
State store:
SAV2001 balance: 70000
CUR3001 balance: 100000
How to run the project
# Step 1: Start Kafka (min 3 brokers for exactly-once in production)
docker-compose up -d kafka zookeeper
# Step 2: Create topics
kafka-topics.sh --create --topic balance-events --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic balance-updates --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Step 3: Build
cd balance-updater
mvn clean package
# Step 4: Run
java -cp target/balance-updater-1.0.0.jar com.bank.streams.BalanceUpdaterApp
# Step 5: Produce balance events
kafka-console-producer.sh --topic balance-events --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# SAV2001:SAV2001|DEPOSIT|50000
# SAV2001:SAV2001|WITHDRAWAL|10000
# Step 6: Consume balance updates
kafka-console-consumer.sh --topic balance-updates --from-beginning --bootstrap-server localhost:9092
What are common mistakes with exactly-once processing?
The most common mistake is assuming exactly-once works end-to-end without understanding its boundaries.
Wrong: assuming exactly-once covers external systems
package com.bank.streams;
public class WrongExactlyOnceScope {
public static void main(String[] args) {
System.out.println("WRONG: Assuming exactly-once covers external systems");
System.out.println();
System.out.println("Scenario:");
System.out.println(" 1. Read from Kafka (exactly-once)");
System.out.println(" 2. Call REST API to update balance in database");
System.out.println(" 3. Write to output topic (exactly-once)");
System.out.println();
System.out.println("Problem: Step 2 is OUTSIDE the Kafka transaction.");
System.out.println("If the app crashes after the REST call but before");
System.out.println("committing, the REST call happens again on retry.");
System.out.println("The database gets a duplicate update.");
}
}
Expected output:
WRONG: Assuming exactly-once covers external systems
Scenario:
1. Read from Kafka (exactly-once)
2. Call REST API to update balance in database
3. Write to output topic (exactly-once)
Problem: Step 2 is OUTSIDE the Kafka transaction.
If the app crashes after the REST call but before
committing, the REST call happens again on retry.
The database gets a duplicate update.
Correct: exactly-once is Kafka-to-Kafka only, use idempotent external writes
package com.bank.streams;
public class CorrectExactlyOnceScope {
public static void main(String[] args) {
System.out.println("CORRECT: Understand exactly-once boundaries");
System.out.println();
System.out.println("Exactly-once covers:");
System.out.println(" Input offset commit + state store + output write = atomic");
System.out.println();
System.out.println("Exactly-once does NOT cover:");
System.out.println(" REST API calls, database writes, email sends");
System.out.println();
System.out.println("For external systems, make the external call idempotent:");
System.out.println(" 1. Use a unique transaction ID in the external call");
System.out.println(" 2. The external system deduplicates by transaction ID");
System.out.println(" 3. Retries are safe because duplicates are ignored");
}
}
Expected output:
CORRECT: Understand exactly-once boundaries
Exactly-once covers:
Input offset commit + state store + output write = atomic
Exactly-once does NOT cover:
REST API calls, database writes, email sends
For external systems, make the external call idempotent:
1. Use a unique transaction ID in the external call
2. The external system deduplicates by transaction ID
3. Retries are safe because duplicates are ignored
What should a senior or principal engineer know?
At the senior level, exactly-once and state recovery are production architecture decisions.
- Transaction overhead: exactly_once_v2 uses one producer per stream thread (not per task like the deprecated exactly_once). This reduces the number of transactional producers and improves throughput.
- Commit interval: with exactly_once_v2, the default commit.interval.ms is 100ms (vs 30s for at_least_once). This increases write frequency to the changelog and broker. Tune it based on your latency vs throughput needs.
- Changelog compaction: changelog topics are log-compacted by default. For large state stores, compaction reduces the data replayed during recovery. Monitor compaction lag.
- State store directory: configure state.dir to a fast SSD. RocksDB performance depends heavily on disk I/O. Default is /tmp/kafka-streams.
- Standby replicas cost: each standby reads from the changelog topic continuously. 1 standby doubles the read traffic from changelog topics. Use standbys only for state stores with recovery time > your SLA.
- Fencing: exactly_once_v2 uses producer fencing. If a zombie instance (thought-dead but still running) tries to produce, its transaction is fenced (rejected), preventing split-brain duplicates.
package com.bank.streams;
public class SeniorExactlyOnceKnowledge {
public static void main(String[] args) {
System.out.println("Senior/Principal exactly-once checklist:");
System.out.println(" 1. exactly_once_v2 = 1 producer per thread (efficient)");
System.out.println(" 2. commit.interval.ms default = 100ms (tune for throughput)");
System.out.println(" 3. Changelog topics are log-compacted (monitor lag)");
System.out.println(" 4. state.dir on fast SSD (not /tmp for production)");
System.out.println(" 5. Standby replicas: 1 standby = 2x changelog read traffic");
System.out.println(" 6. Producer fencing prevents zombie duplicates");
System.out.println(" 7. Exactly-once is Kafka-to-Kafka only");
System.out.println(" 8. External calls need idempotent design separately");
}
}
Expected output:
Senior/Principal exactly-once checklist:
1. exactly_once_v2 = 1 producer per thread (efficient)
2. commit.interval.ms default = 100ms (tune for throughput)
3. Changelog topics are log-compacted (monitor lag)
4. state.dir on fast SSD (not /tmp for production)
5. Standby replicas: 1 standby = 2x changelog read traffic
6. Producer fencing prevents zombie duplicates
7. Exactly-once is Kafka-to-Kafka only
8. External calls need idempotent design separately
What are common interview follow-up questions?
- What is the difference between exactly_once and exactly_once_v2? (exactly_once creates one producer per task, which is expensive with many partitions. exactly_once_v2 creates one producer per stream thread. Use v2 for all new applications.)
- Can you have exactly-once without a state store? (Yes. Even stateless topologies benefit: the input offset and output write are atomic. Without exactly-once, a crash can cause duplicate output records.)
- What happens to in-flight transactions during a crash? (The transaction is never committed. On restart, the consumer reads from the last committed offset, and the uncommitted output records are invisible to downstream consumers with read_committed isolation.)
- How does read_committed isolation work? (Downstream consumers with isolation.level=read_committed only see records from committed transactions. Uncommitted records are buffered by the consumer until the transaction is committed or aborted.)
- How do you measure the throughput cost of exactly-once? (Run the same topology with at_least_once and exactly_once_v2 on the same input, and compare records-processed-per-second metrics.)
Visual Overview
Exactly-Once Transaction Flow
==============================
┌──────────────┐
│ Input Topic │
│ (balance- │
│ events) │
└──────┬───────┘
│ read
▼
┌──────────────────────────────┐
│ BEGIN TRANSACTION │
│ │
│ 1. Process record │
│ 2. Update state store │──> State Store (RocksDB)
│ 3. Write to changelog │──> Changelog Topic
│ 4. Write to output topic │──> Output Topic
│ 5. Commit input offset │──> Consumer Offset
│ │
│ COMMIT TRANSACTION │
│ (all or nothing) │
└──────────────────────────────┘
State Store Recovery
=====================
┌──────────────┐
│ Changelog │
│ Topic │
│ (compacted) │
└──────┬───────┘
│ replay from beginning
▼
┌──────────────────────────────┐
│ Fresh RocksDB Store │
│ │
│ key=SAV2001 val=70000 │
│ key=CUR3001 val=100000 │
│ │
│ State rebuilt, resume │
│ processing from last offset │
└──────────────────────────────┘
Key Takeaway
Exactly-once processing in Kafka Streams wraps the entire read-process-write cycle in a Kafka transaction. Input offset commit, state store changelog write, and output record write either all succeed or all roll back. This guarantees that each input record produces its effect exactly one time, even after crashes.
State stores are backed by changelog topics. When an application crashes, Kafka Streams replays the changelog from the beginning to rebuild the local RocksDB store. Standby replicas maintain warm copies of state stores on other instances, reducing recovery time from minutes to seconds after a rebalance.
In our balance updater project at National Bank of India, we maintained running account balances using exactly_once_v2. Priya's deposits and withdrawals updated the state store atomically with the output. The TopologyTestDriver confirmed that three events on SAV2001 (deposit 50,000, deposit 30,000, withdrawal 10,000) produced a final balance of 70,000.
The most critical thing to understand is that exactly-once is Kafka-to-Kafka only. It covers input offsets, state stores, and output topics. If your processing calls a REST API or writes to a database, that call is outside the transaction. For external systems, you must design idempotent writes separately.
Remember one sentence: exactly_once_v2 uses Kafka transactions to make input offset commit, state store update, and output write atomic, and changelog topics make state stores recoverable after any failure.
---