Definition
Delivery semantics describe how many times a message may be processed in the presence of failures.
- At-most-once – The consumer commits its offset before processing, so a crash mid-processing means the message is skipped on restart. Nothing is duplicated, but messages can be lost.
- At-least-once – The consumer commits its offset after processing succeeds, so a crash after processing but before committing means the message is reprocessed on restart. Nothing is lost, but duplicates are possible.
- Exactly-once – Each message affects the result exactly one time despite failures, achieved through idempotent producers, transactional or idempotent writes, and checkpointed consumer offsets committed atomically with the output.
The choice is deliberate: for money movement you insist on exactly-once; for low-value telemetry you may accept at-least-once.
Imagine the National Bank of India posting transfer instructions. At-most-once is like mailing a letter and not keeping a copy: if it gets lost, the transfer never happens, but it is never done twice. At-least-once is like resending the letter until you get a confirmation: the transfer definitely happens, but a slow confirmation might make you send it twice, so Priya could be credited twice. Exactly-once is the bank's gold standard: every transfer happens once and only once, because the bank uses unique instruction numbers and commits the money movement and the "done" mark together in one atomic step.
Understanding the Concepts
What are the three delivery semantics?
The three semantics are a trade-off between the risk of loss and the risk of duplication, forced by the reality that networks and processes fail at the worst moments. At the National Bank of India, this trade-off directly affects account balances.
Semantic Offset Commit Risk of Loss Risk of Duplication ───────────── ────────────── ──────────── ─────────────────── At-most-once Before process Yes No At-least-once After process No Yes Exactly-once Atomic with out No No
How does at-most-once work?
The consumer commits its offset before processing. If it crashes mid-processing, the message is already marked done and is skipped on restart. Nothing is duplicated, but messages can be lost.
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import java.time.Duration;
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("bank-metrics"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
consumer.commitSync(); // Commit BEFORE processing
for (ConsumerRecord<String, String> record : records) {
// If we crash here, the message is already committed
// and will be skipped on restart -- LOST
metricsService.record(record.value());
}
}
// Acceptable only for data you can afford to drop, like branch metrics
// NEVER acceptable for money movement
How does at-least-once work?
The consumer commits the offset only after processing succeeds. If it crashes after processing but before committing, it reprocesses that message on restart.
KafkaConsumer<String, Transaction> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("bank-transactions"));
while (true) {
ConsumerRecords<String, Transaction> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, Transaction> record : records) {
Transaction txn = record.value();
bankDatabase.applyDebit(txn); // Process first
// If we crash HERE, offset is not committed
// On restart, the debit is applied AGAIN -- DUPLICATED
}
consumer.commitSync(); // Commit AFTER processing
}
// Default for most systems because losing data is worse than duplicating
// But duplicates are deadly in finance: applying a debit twice is wrong
How do idempotent producers prevent duplicate writes?
Idempotent producers ensure that a producer retry does not write the same record twice into the log. The broker deduplicates using a producer ID and sequence number.
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;
Properties props = new Properties();
props.put("bootstrap.servers", "kafka-broker:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
// Enable idempotent producer
props.put("enable.idempotence", "true");
// Implies acks=all, retries=Integer.MAX_VALUE, max.in.flight.requests.per.connection<=5
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
// Even if the producer retries due to a network glitch,
// the broker uses (producerId, sequenceNumber) to deduplicate
producer.send(new ProducerRecord<>("bank-transfers", "TXN-1001", transferJson));
How do Kafka transactions achieve exactly-once?
Kafka transactions commit produced messages and consumed offsets atomically in one transaction. This is the read-process-write exactly-once pattern.
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.common.TopicPartition;
import java.util.Map;
import java.util.HashMap;
// Producer with transactional ID
Properties prodProps = new Properties();
prodProps.put("transactional.id", "bank-transfer-processor-1");
prodProps.put("enable.idempotence", "true");
KafkaProducer<String, String> producer = new KafkaProducer<>(prodProps);
producer.initTransactions();
// Consumer with read-committed isolation
Properties consProps = new Properties();
consProps.put("isolation.level", "read_committed");
consProps.put("enable.auto.commit", "false");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consProps);
consumer.subscribe(List.of("bank-transfers"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
producer.beginTransaction();
try {
for (var record : records) {
// Process and produce output
String enriched = enrichTransfer(record.value());
producer.send(new ProducerRecord<>("enriched-transfers", record.key(), enriched));
}
// Commit offsets AND produced messages atomically
Map<TopicPartition, OffsetAndMetadata> offsets = getOffsetsToCommit(records);
producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
producer.commitTransaction();
} catch (Exception e) {
producer.abortTransaction();
}
}
How does Flink achieve exactly-once?
Flink uses checkpoint barriers and a two-phase commit sink. The barriers flow through the dataflow graph, and when all operators have checkpointed, the sink commits. On failure, Flink restores state from the last checkpoint and replays from that position.
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.CheckpointingMode;
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// Enable exactly-once checkpointing
env.enableCheckpointing(60000); // Every 60 seconds
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(30000);
// Kafka source reads from committed checkpoint offset
// Kafka sink commits in two-phase commit aligned with checkpoints
// Result: exactly-once from source to sink
What is effectively-once with idempotent writes?
When crossing into an external system that cannot participate in Kafka transactions, you often drop to at-least-once plus idempotent writes. This gives effectively-once results.
import java.math.BigDecimal;
// At-least-once delivery + idempotent upsert = effectively-once
// The database uses the transaction_id as a unique key
// A duplicate message simply overwrites the same row
String upsertSql = """
INSERT INTO bank_transfers (transaction_id, account_id, amount, status)
VALUES (?, ?, ?, ?)
ON CONFLICT (transaction_id)
DO UPDATE SET amount = EXCLUDED.amount, status = EXCLUDED.status
""";
try (PreparedStatement stmt = conn.prepareStatement(upsertSql)) {
stmt.setString(1, txn.getTransactionId()); // Stable unique ID
stmt.setString(2, txn.getAccountId());
stmt.setBigDecimal(3, txn.getAmount());
stmt.setString(4, txn.getStatus());
stmt.executeUpdate();
}
// A replay after crash simply re-upserts: no duplicate effect
When do you choose each semantic?
The choice is deliberate and depends on the data's value.
| Semantic | Use Case | Banking Example |
|---|---|---|
| At-most-once | Low-value, droppable data | Branch temperature metrics |
| At-least-once | Most data where loss is worse than duplication | Audit log events (deduplicate downstream) |
| Exactly-once | Money movement, balances | Debit/credit transactions |
| Effectively-once | External systems without transaction support | Writing transfers to PostgreSQL via upsert |
// Decision tree in code
if (isMoneyMovement(txn)) {
// Use Kafka transactions for exactly-once
processWithTransaction(txn);
} else if (isAuditEvent(txn)) {
// At-least-once with idempotent downstream
processWithRetry(txn);
} else {
// At-most-once for low-value metrics
processFireAndForget(txn);
}
What should a senior engineer know?
- Exactly-once is not free: Kafka transactions add latency and complexity. Use them only where the data demands it.
- End-to-end exactly-once requires every component in the chain to participate. A non-transactional sink breaks the guarantee.
- The
read_committedisolation level on the consumer ensures it never reads uncommitted (in-flight) transaction data. - Flink's exactly-once and Kafka's exactly-once use different mechanisms but achieve the same goal.
- For most banking pipelines, the pragmatic pattern is at-least-once plus idempotent upserts, which gives effectively-once with simpler operations.
Visual Overview
At-Most-Once:
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Consume │───►│ Commit │───►│ Process │
│ message │ │ offset │ │ (may │
└──────────┘ └──────────┘ │ crash) │
└──────────┘
Crash after commit → message LOST
At-Least-Once:
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Consume │───►│ Process │───►│ Commit │
│ message │ │ │ │ offset │
└──────────┘ └──────────┘ └──────────┘
Crash after process → message REPROCESSED
Exactly-Once (Kafka Transactions):
┌──────────┐ ┌──────────────────────────┐
│ Consume │───►│ BEGIN TRANSACTION │
│ message │ │ Process + Produce │
└──────────┘ │ Commit offsets + output │
│ COMMIT TRANSACTION │
└──────────────────────────┘
Crash → abort → replay from last committed offset
Key Takeaway
Delivery semantics describe how many times a message may be processed when failures happen, and they are a trade-off between losing data and duplicating it. At-most-once commits the offset before processing, so a crash means the message is skipped on restart and lost. At-least-once commits after processing, so a crash causes reprocessing and possible duplication. Exactly-once means each message affects the result exactly one time despite failures.
Exactly-once is not magic: it is engineered from three cooperating pieces. Idempotent producers ensure a retry does not write duplicate records into the log. Transactional writes commit output and consumer offsets atomically. And checkpointed offsets tie recovery to a consistent point. Flink achieves this with checkpoint barriers, and Kafka achieves it with read-process-write transactions.
At the National Bank of India this distinction was the difference between correct and catastrophic. For a debit message, at-most-once could lose the debit, and at-least-once could apply it twice and wrongly reduce Rajesh's balance. So for money movement we insisted on exactly-once: idempotent writes keyed by a stable transaction ID meant a replay after crash simply skipped the already-applied debit. For low-value telemetry we used at-least-once to save complexity.
The pragmatic rule for most banking pipelines is at-least-once plus idempotent upserts when crossing into external systems, giving effectively-once results without the full weight of Kafka transactions.
Remember one sentence: exactly-once is achieved by combining idempotent producers, transactional writes, and atomically committed offsets, so each message affects the result once despite any failure.
---