Definition
Log compaction promises to retain at least the latest value per key, but that promise has documented exceptions.
- Tombstone – a record with a null value that marks a key for deletion; removed after
delete.retention.ms(default 24 hours). - compact,delete policy – applies both compaction and time/size retention, so even the latest value for a key can expire on age.
- Null-keyed record – a record with no key that compaction cannot deduplicate, governed only by retention.
- Dirty ratio –
min.cleanable.dirty.ratiocontrols how aggressively the cleaner runs; a low value purges superseded records faster. - delete.retention.ms – the window a tombstone is kept so slow consumers can observe the delete before it vanishes.
Imagine National Bank of India keeps one current notice per account on a board, and the rule is to always keep the newest notice for each account. People expect nothing ever vanishes, but four documented rules say otherwise. First, if a clerk pins a blank cancelled slip for Rajesh's account SAV2001 and a day passes, even that cancelled slip is removed, so SAV2001 vanishes from the board. Second, if the board also has a nothing-older-than-seven-days rule, even Rajesh's newest notice disappears on day eight. Third, notices pinned without an account number cannot be matched to any account, so they are thrown out on the age rule. Fourth, if the tidy-up crew runs constantly, old superseded notices are pulled almost the instant a newer one is pinned.
Understanding the Concepts
What is log compaction in Kafka?
Log compaction is a cleanup strategy where Kafka retains at least the latest record for each key in a topic. Unlike delete retention, which removes all data older than a time or size threshold, compaction keeps the most recent value per key indefinitely, making the topic act like a key-value snapshot store. You enable it with cleanup.policy=compact.
At National Bank of India, the account.state topic is compacted so that the latest balance for every account like SAV2001 is always available, even if the account was created months ago.
# Create a compacted topic for account state
kafka-topics.sh --create --topic account.state \
--bootstrap-server kafka-1:9092 \
--partitions 12 --replication-factor 3 \
--config cleanup.policy=compact
How does a tombstone make data disappear?
You delete a key from a compacted topic by producing a record with that key and a null value. This is the tombstone. Compaction keeps the tombstone for at least delete.retention.ms (default 24 hours) so consumers can observe the deletion. After that window and a subsequent cleaning pass, the tombstone itself is removed, and the key is completely gone from the log.
package com.bank.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class TombstoneDemo {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
// Rajesh closes account SAV2001 -- send tombstone
producer.send(new ProducerRecord<>("account.state", "SAV2001", null));
System.out.println("Tombstone sent for SAV2001");
// After delete.retention.ms + cleaning pass, SAV2001 is fully gone
}
}
}
Tombstone sent for SAV2001
Why does cleanup.policy=compact,delete cause the latest value to expire?
A topic with cleanup.policy=compact only compacts and never expires the latest value per key. But cleanup.policy=compact,delete applies both compaction and ordinary time or size retention. A record can be removed because it is older than retention.ms, even if it is the latest value for its key. A rarely updated key's latest record can expire purely on age.
# Topic with both policies: compacted AND time-bounded
kafka-configs.sh --alter --topic account.changelog \
--bootstrap-server kafka-1:9092 \
--add-config cleanup.policy=compact\,delete,retention.ms=604800000
# Priya's account SAV3001 was last updated 10 days ago.
# Even though it is the latest value for SAV3001, it exceeds 7-day retention.
# Result: SAV3001 is deleted. This is expected, not a bug.
Why do null-keyed records bypass compaction?
Compaction is keyed: it reasons about the latest record per key. A record with a null key has no key to deduplicate on, so compaction skips it entirely. Such records are governed only by ordinary retention if the topic allows deletion. If a producer accidentally sends records with a null key on a compacted topic, those records are not protected by the latest-per-key guarantee.
package com.bank.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class NullKeyMistakeDemo {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
// WRONG: null key -- compaction cannot track this record
producer.send(new ProducerRecord<>("account.state", null, "{\"balance\":50000}"));
System.out.println("Sent with null key -- not compactable!");
// CORRECT: always set the account ID as the key
producer.send(new ProducerRecord<>("account.state", "SAV2001", "{\"balance\":50000}"));
System.out.println("Sent with key SAV2001 -- compaction will retain latest");
}
}
}
Sent with null key -- not compactable!
Sent with key SAV2001 -- compaction will retain latest
How do aggressive compaction settings purge old records quickly?
How quickly superseded records are removed depends on tunables. A small segment.bytes rolls segments sooner so recent updates become eligible for cleaning quickly. A low min.cleanable.dirty.ratio makes the cleaner run even when only a small fraction of the log is stale. A short log.cleaner.backoff.ms makes it run often.
# Aggressive compaction: old values purged almost immediately
kafka-configs.sh --alter --topic account.state \
--bootstrap-server kafka-1:9092 \
--add-config segment.bytes=16777216,min.cleanable.dirty.ratio=0.1,log.cleaner.backoff.ms=5000
# Result: a superseded balance for SAV2001 is cleaned within seconds
# If you expected to find the previous balance for debugging, it is already gone
# Relaxed compaction: keep older versions longer
kafka-configs.sh --alter --topic account.state \
--bootstrap-server kafka-1:9092 \
--add-config segment.bytes=268435456,min.cleanable.dirty.ratio=0.5,log.cleaner.backoff.ms=30000
What are common mistakes with compacted topics?
The most common mistake is assuming compaction keeps the latest value forever regardless of configuration, and then being surprised when data disappears.
# WRONG: using compact,delete but expecting latest value to live forever
kafka-configs.sh --alter --topic account.state \
--bootstrap-server kafka-1:9092 \
--add-config cleanup.policy=compact\,delete,retention.ms=86400000
# Rajesh's account last updated 2 days ago -> GONE. Not a bug.
# CORRECT: use compact only if you want an unbounded key-value store
kafka-configs.sh --alter --topic account.state \
--bootstrap-server kafka-1:9092 \
--add-config cleanup.policy=compact
# Rajesh's latest value is retained indefinitely
How should consumers handle tombstones?
Consumers must treat a null value as an explicit delete. If a consumer ignores tombstones, it will hold stale state for a key that was deleted. After delete.retention.ms the tombstone itself is gone, so a consumer that starts fresh will never know the key existed.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.ConsumerRecord;
public class TombstoneConsumerDemo {
public void processRecord(ConsumerRecord<String, String> record) {
if (record.value() == null) {
// Tombstone: key is being deleted
System.out.println("Account " + record.key() + " closed -- removing from cache");
accountCache.remove(record.key());
} else {
// Normal update: store latest state
System.out.println("Account " + record.key() + " updated");
accountCache.put(record.key(), record.value());
}
}
}
Account SAV2001 closed -- removing from cache
Account SAV3001 updated
What about compaction and offset gaps?
Compaction preserves offsets: surviving records keep their original offsets even though intermediate offsets are gone. After compaction, the log has gaps in offset numbering, which consumers handle transparently because the broker returns the next available offset.
Before compaction (account.state, partition 0):
Offset 0: key=SAV2001 value={balance:100000}
Offset 1: key=SAV3001 value={balance:200000}
Offset 2: key=SAV2001 value={balance:95000} <-- supersedes offset 0
Offset 3: key=SAV2001 value={balance:90000} <-- supersedes offset 2
After compaction:
Offset 1: key=SAV3001 value={balance:200000} <-- latest for SAV3001
Offset 3: key=SAV2001 value={balance:90000} <-- latest for SAV2001
(Offsets 0 and 2 are gone -- gaps are normal)
What should a senior engineer know?
Compaction's guarantee is narrow and precise: it retains at least the latest value per key, subject to tombstone expiry, subject to any additional delete policy, only for properly keyed records, and on a schedule you control. The compaction cleaner runs per partition and lazily, so the same key may be visible on one partition's uncleaned segment while gone from another. Enforce non-null, correct keys at the producer, ideally with a schema or validation. Lengthen delete.retention.ms if slow consumers must observe deletes. The unifying rule is that when data disappears from a compacted topic, the first thing to check is the topic's exact configuration.
# Inspect compaction state for a topic
kafka-log-dirs.sh --describe --bootstrap-server kafka-1:9092 \
--topic-list account.state
# Key metrics to watch
# kafka.log:type=LogCleaner,name=max-clean-time-secs
# kafka.log:type=LogCleaner,name=cleaner-recopy-percent
# kafka.log:type=Log,name=LogEndOffset (per partition)
Visual Overview
Compacted Topic: account.state
┌──────────────────────────────────────────────────┐
│ Closed Segment (eligible for compaction) │
│ │
│ Offset 0: SAV2001 = {balance: 100000} ──┐ │
│ Offset 1: SAV3001 = {balance: 200000} │ │
│ Offset 2: SAV2001 = {balance: 95000} ──┤ same │
│ Offset 3: SAV2001 = {balance: 90000} ──┘ key │
└──────────────────────────────────────────────────┘
│
▼ Log Cleaner
┌──────────────────────────────────────────────────┐
│ After Compaction (only latest per key kept) │
│ │
│ Offset 1: SAV3001 = {balance: 200000} │
│ Offset 3: SAV2001 = {balance: 90000} │
│ (offsets 0, 2 removed -- gaps are normal) │
└──────────────────────────────────────────────────┘
Four reasons data disappears:
┌──────────────────┐ ┌───────────────────────┐
│ 1. Tombstone │ │ 2. compact,delete │
│ null value │ │ time retention │
│ expires after │ │ removes even │
│ delete.ret.ms │ │ latest value │
└──────────────────┘ └───────────────────────┘
┌──────────────────┐ ┌───────────────────────┐
│ 3. Null key │ │ 4. Aggressive settings│
│ not compactable│ │ small segments + │
│ plain retention│ │ low dirty ratio │
└──────────────────┘ └───────────────────────┘
Key Takeaway
Log compaction guarantees that a compacted topic retains at least the latest value per key, but that guarantee has four documented exceptions. A tombstone, a record with a null value, marks a key for deletion and is removed after delete.retention.ms. A topic with cleanup.policy=compact,delete applies time or size retention on top of compaction, so even the latest value can expire. Records with null keys are not compactable and are governed only by retention. Aggressive settings like small segments and a low dirty ratio purge superseded records almost instantly.
On the National Bank of India account-state topic, Pankaj's team chased all four of these as suspected bugs before realizing each was documented behaviour. A closed account's key SAV2001 vanished a day after its tombstone, a dormant account's snapshot expired on a compact-plus-delete topic, some null-keyed events disappeared under retention, and on an aggressively tuned topic the previous balance was purged the instant a new one arrived.
The practical design rules are concrete: make every consumer treat a null value as a delete so tombstone expiry never leaves stale state, choose compact versus compact-plus-delete deliberately based on whether you need an unbounded store or a bounded changelog, enforce non-null correct keys at the producer with schema validation, and tune compaction aggressiveness to match how long you need superseded versions to survive.
Remember one sentence: compaction keeps the latest value per key only within the bounds of tombstone expiry, any additional delete policy, correct keying, and the schedule you configured.
---