Definition
- Durable log – Kafka stores records in an append-only log on disk, and the record's lifetime is governed by the retention policy, not by consumption.
- Retention policy – the topic configuration (
retention.ms,retention.bytes, orcleanup.policy=compact) that determines when records are removed, independent of whether they were consumed. auto.offset.reset– the consumer setting that controls where a new consumer group starts reading:earliestreads the entire retained backlog,lateststarts from the current end.- Decoupled producers and consumers – Kafka's design where the producer writes independently of whether any consumer exists, and consumers read independently of the producer.
- Traditional queue – a message broker like JMS or SQS where a message is typically removed once a consumer acknowledges it, and may be lost if no consumer is present.
Imagine National Bank of India writes every transaction into a permanent ledger book, whether or not anyone is reading it right now. If the fraud analyst is on holiday when Rajesh makes a transfer, the entry is still written into the ledger and stays there. When the analyst comes back next week, they open the ledger and read everything that happened while they were away. Nothing was lost just because nobody was watching at that moment. A traditional queue would be like a to-do basket where a slip is thrown away if nobody is around to grab it.
Understanding the Concepts
What happens when no consumer is subscribed?
When a producer calls kafkaTemplate.send("transfers", key, value), the broker appends the record to the appropriate partition and assigns it an offset. This write succeeds and is acknowledged based solely on the producer's acks setting and the replication state. It has nothing to do with whether any consumer group exists or is online. A producer can publish to a brand-new topic that no consumer has ever subscribed to, and the record is safely stored.
package com.bank.producer;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class TransferEventProducer {
private final KafkaTemplate<String, String> kafkaTemplate;
public TransferEventProducer(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void publishTransfer(String accountId, BigDecimal amount) {
String event = "{\"accountId\":\"" + accountId
+ "\",\"amount\":" + amount + "}";
// This succeeds even if no consumer exists
kafkaTemplate.send("transfers", accountId, event);
System.out.println("Published transfer for " + accountId
+ " (no consumer needed)");
}
}
Expected output:
Published transfer for SAV2001 (no consumer needed)
-> Record stored at partition 2, offset 0
-> No consumer exists, record waits in log
How is this different from a traditional queue?
In a traditional point-to-point queue like JMS or SQS, a message is removed once a consumer acknowledges it. If no consumer is present, the message waits but will be consumed by exactly one worker and then deleted. In Kafka, consumption never deletes records. Any number of consumer groups can read the same record, and it stays in the log until the retention policy removes it.
| Feature | Kafka (Log) | Traditional Queue (JMS/SQS) |
|---|---|---|
| No consumer present | Record stored in log | Message waits in queue |
| After consumption | Record stays in log | Message deleted |
| Multiple consumers | Yes, via consumer groups | No (one consumer per message) |
| Replay | Yes, seek to old offset | No, message gone after ack |
| Retention | Time/size/compaction based | Until consumed or max retention |
package com.bank.comparison;
public class KafkaVsQueueBehavior {
public static void main(String[] args) {
System.out.println("Kafka (log-based):");
System.out.println(" 1. Producer sends record");
System.out.println(" 2. Record stored in partition log");
System.out.println(" 3. No consumer? Record stays in log");
System.out.println(" 4. Consumer arrives later? Reads from beginning");
System.out.println(" 5. Consumer reads record? Record STAYS in log");
System.out.println(" 6. Second consumer group? Reads same record");
System.out.println();
System.out.println("Traditional Queue:");
System.out.println(" 1. Producer sends message");
System.out.println(" 2. Message waits in queue");
System.out.println(" 3. Consumer reads and ACKs");
System.out.println(" 4. Message DELETED from queue");
System.out.println(" 5. No replay, no second consumer");
}
}
Kafka: record stays after consumption, multiple readers, replay
Queue: message deleted after ACK, one reader, no replay
What governs the record's lifetime?
Retention, not consumption, governs the record's lifetime. With the default delete policy, the record lives until it exceeds retention.ms or the partition exceeds retention.bytes. With a compact policy, the latest record per key is kept indefinitely. In neither case does reading a record remove it.
package com.bank.retention;
public class RecordLifetime {
public static void main(String[] args) {
System.out.println("Record Lifetime in Kafka:");
System.out.println();
System.out.println("cleanup.policy=delete:");
System.out.println(" retention.ms=604800000 (7 days)");
System.out.println(" Record produced Monday -> deleted next Monday");
System.out.println(" Whether consumed or not: same lifetime");
System.out.println();
System.out.println("cleanup.policy=compact:");
System.out.println(" Latest record per key kept indefinitely");
System.out.println(" Only older versions of same key removed");
System.out.println();
System.out.println("Key point: reading does NOT delete records");
System.out.println("Retention policy does NOT consider consumption");
}
}
Expected output:
Retention removes records by age/size, not by consumption.
Reading never deletes. Retention policy ignores consumers.
How does a late-arriving consumer read the backlog?
A consumer that subscribes later controls where it starts with auto.offset.reset. If set to earliest, a new consumer group with no committed offset starts from the very beginning of the retained log and reads every record still present, including those produced while no consumer existed. If set to latest, the new group starts only from records produced after it subscribes.
package com.bank.consumer;
import java.util.Properties;
public class LateConsumerConfig {
public static Properties createEarliestConfig() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "fraud-analytics-group");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
// Read ALL retained records, including those produced before
// this consumer existed
props.put("auto.offset.reset", "earliest");
System.out.println("Fraud analytics consumer configured:");
System.out.println(" auto.offset.reset=earliest");
System.out.println(" Will read entire backlog from offset 0");
return props;
}
public static Properties createLatestConfig() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "live-dashboard-group");
// Read only NEW records produced after subscribing
props.put("auto.offset.reset", "latest");
System.out.println("Live dashboard consumer configured:");
System.out.println(" auto.offset.reset=latest");
System.out.println(" Will skip backlog, read only new events");
return props;
}
public static void main(String[] args) {
createEarliestConfig();
System.out.println();
createLatestConfig();
}
}
Expected output:
Fraud analytics: earliest -> reads 7 days of backlog
Live dashboard: latest -> reads only new events
What are the practical caveats?
The record is only kept as long as retention allows. If no consumer reads it for longer than the retention window, it will eventually be deleted by the cleanup policy, not because it was unconsumed, but because it aged out. Also, a consumer only receives records produced to partitions it is assigned.
package com.bank.caveats;
public class RetentionCaveats {
public static void main(String[] args) {
System.out.println("Practical Caveats:");
System.out.println();
System.out.println("1. Retention expiry:");
System.out.println(" Record produced on Day 1, retention=7 days");
System.out.println(" Consumer subscribes on Day 10");
System.out.println(" Day 1-3 records ALREADY DELETED (aged out)");
System.out.println(" Consumer only sees Day 4-10 records");
System.out.println();
System.out.println("2. auto.offset.reset only applies when:");
System.out.println(" No committed offset exists for this group");
System.out.println(" Existing groups resume from committed offset");
System.out.println();
System.out.println("3. Partition assignment:");
System.out.println(" Consumer only gets records from assigned partitions");
}
}
Expected output:
Caveat 1: records age out per retention, even if never consumed
Caveat 2: auto.offset.reset only for new groups (no committed offset)
Caveat 3: consumer only reads assigned partitions
What are common mistakes with this behavior?
The most common mistake is assuming Kafka drops messages like a queue when no consumer is present. Another mistake is setting auto.offset.reset=latest when you actually need the backlog.
package com.bank.mistakes;
// WRONG: Assuming messages are lost without a consumer.
public class WrongAssumption {
public static void main(String[] args) {
System.out.println("WRONG: 'No consumer online, transfers will be lost'");
System.out.println(" Reality: transfers stored in log for 7 days");
System.out.println(" Consumer can read them anytime within retention");
System.out.println();
System.out.println("WRONG: auto.offset.reset=latest for fraud analytics");
System.out.println(" Fraud team deploys new consumer on Wednesday");
System.out.println(" Wants to analyze Monday-Wednesday transfers");
System.out.println(" But latest skips the Monday-Tuesday backlog!");
System.out.println();
System.out.println("CORRECT: auto.offset.reset=earliest for fraud analytics");
System.out.println(" Reads from beginning of retained log");
System.out.println(" Gets Monday through Wednesday transfers");
}
}
WRONG: messages lost without consumer (Kafka is not a queue)
WRONG: latest when you need the backlog (use earliest)
CORRECT: earliest to read all retained records
How does this enable multiple independent consumers?
Because records persist regardless of consumers, any number of groups can read at their own pace. A record produced with no consumer is not a special case; it is the normal behavior of a log. The producer does not block, does not fail, and does not need a consumer to be present.
package com.bank.patterns;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class IndependentConsumerGroups {
// Consumer 1: deployed on day 1
@KafkaListener(topics = "transfers", groupId = "settlement-group")
public void settlement(String event) {
System.out.println("[settlement] Processing: " + event);
}
// Consumer 2: deployed on day 5 (reads backlog from day 1)
@KafkaListener(topics = "transfers", groupId = "analytics-group")
public void analytics(String event) {
System.out.println("[analytics] Processing: " + event);
}
// Consumer 3: deployed on day 10 (reads backlog from day 4)
@KafkaListener(topics = "transfers", groupId = "ml-training-group")
public void mlTraining(String event) {
System.out.println("[ml-training] Processing: " + event);
}
}
Expected output:
settlement: reads from day 1 (deployed day 1)
analytics: reads backlog from day 1 (deployed day 5, earliest)
ml-training: reads backlog from day 4 (deployed day 10, 7-day retention)
What should a senior engineer know?
The log-centric model is precisely why Kafka suits event streaming, event sourcing, and audit: the data is a durable record of what happened, available to current and future consumers alike, rather than a transient task handed to one worker and discarded.
package com.bank.senior;
public class LogCentricModelSummary {
public static void main(String[] args) {
System.out.println("Why Kafka's log model matters:");
System.out.println("1. Events are facts, not tasks");
System.out.println(" -> 'Rajesh transferred 50000' is a permanent record");
System.out.println("2. Consumers are readers, not removers");
System.out.println(" -> Reading never deletes");
System.out.println("3. New consumers can appear anytime");
System.out.println(" -> ML team deploys next month, reads all history");
System.out.println("4. Replay is always possible");
System.out.println(" -> Seek to old offset, reprocess");
System.out.println("5. Retention is the only cleanup mechanism");
System.out.println(" -> Not consumption, not acknowledgment");
}
}
Expected output:
Kafka log model: events are facts, consumers are readers,
new consumers can appear anytime, replay always possible.
Visual Overview
Producer Sends With No Consumer
================================
Producer Kafka Cluster
│ │
│ send("transfers", │
│ key="SAV2001", │
│ event) │
├────────────────────────▶│
│ │
│ ACK (based on acks, │
│ NOT on consumer │
│ presence) │
│◀────────────────────────│
│ │
│ ┌────┴────────────┐
│ │ Partition 2 │
│ │ [offset 0] │
│ │ Record stored │
│ │ Retention: 7d │
│ └─────────────────┘
│
No consumer online │
Record WAITS in log │
│
Day 5: Consumer arrives │
auto.offset.reset=earliest
│
┌──────────────────────┘
│
▼
Consumer (fraud-analytics-group)
Reads from offset 0 -> gets ALL retained records
Nothing was lost.
Key Takeaway
If a producer sends a message but no consumer is subscribed, the message is not lost. This is one of the most important differences between Kafka and a traditional queue. Kafka stores records in a durable log independently of whether any consumer exists. The broker appends the record, assigns an offset, and acknowledges the write based only on acks and replication, not on the presence of a consumer. The record then simply waits in the log.
How long it waits is governed by the retention policy, not by consumption. With the delete policy, the record lives until it exceeds retention.ms; with compact, the latest record per key is kept indefinitely. Reading a record never deletes it, which is the opposite of a traditional queue where a message is removed after acknowledgment. This is what enables multiple independent consumer groups and replay.
At National Bank of India, the payments service produces transfer events around the clock. When the fraud analytics team deploys a new consumer with auto.offset.reset=earliest, it reads the entire retained backlog, including everything produced while it was offline. A live dashboard, by contrast, uses latest because it only cares about new events. Both patterns work because the log retains everything within the retention window.
Remember one sentence: Kafka retains records per the retention policy regardless of consumers, so a message produced with no consumer is not lost but simply waits in the log for any future reader.
---