Definition
- Tombstone -- a record with the key you want to delete and a null value; it is how you delete a key in a compacted topic.
- Log compaction -- a background process that rewrites closed segments to keep only the latest value per key; it is asynchronous and not instant.
- Active segment -- the newest segment being appended to; the log cleaner never touches it, only closed segments.
- delete.retention.ms -- how long the tombstone itself is kept (default 24 hours) so consumers have time to read the delete before it vanishes.
- min.cleanable.dirty.ratio -- the fraction of the log that must be stale before the cleaner runs on that partition.
Imagine National Bank of India keeps one noticeboard per account where only the latest notice should remain. To cancel Rajesh's standing instruction, a clerk pins a blank cancelled slip. But the tidy-up crew works on a schedule and never touches the last few pinned slips. The rule also says to leave the cancelled slip up for a full day so every branch reads it. If you glance at the board right after pinning, the old instruction is still visible. It is not a bug; the cleanup has not reached that part of the board yet.
Understanding the Concepts
What does log compaction guarantee?
A compacted topic with cleanup.policy=compact promises to retain at least the latest value for every key. It does not promise to remove old values immediately. The removal is done by a background log cleaner that periodically scans and rewrites segments.
# Create a compacted topic for account profiles
kafka-topics.sh --create --topic account.profiles \
--partitions 6 --replication-factor 3 \
--config cleanup.policy=compact \
--config min.cleanable.dirty.ratio=0.5 \
--config delete.retention.ms=86400000 \
--bootstrap-server kafka-1:9092
How do you delete a key in a compacted topic?
You send a tombstone: a record with the key and a null value. Compaction eventually removes all earlier records for that key.
package com.bank.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class TombstoneProducer {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
// Write a profile
producer.send(new ProducerRecord<>("account.profiles", "SAV2001",
"{\"name\":\"Rajesh\",\"tier\":\"gold\"}")).get();
System.out.println("Wrote profile for SAV2001");
// Delete with tombstone: value = null
producer.send(new ProducerRecord<>("account.profiles", "SAV2001", null)).get();
System.out.println("Sent tombstone for SAV2001 (value=null)");
System.out.println(" Old records will be removed after compaction runs");
System.out.println(" Tombstone stays for delete.retention.ms (24h default)");
}
}
}
Expected output:
Wrote profile for SAV2001
Sent tombstone for SAV2001 (value=null)
Old records will be removed after compaction runs
Tombstone stays for delete.retention.ms (24h default)
Why are old records still visible after the tombstone?
Three timing rules explain why. First, compaction runs in the background, so there is always a lag. Second, the cleaner only works on closed segments, never the active one. Third, the tombstone is kept for delete.retention.ms so slow consumers can read the delete.
package com.bank.kafka;
public class WhyOldRecordsVisible {
public static void main(String[] args) {
System.out.println("Why old records are still visible after tombstone:");
System.out.println();
System.out.println("1. Compaction is asynchronous (background log cleaner)");
System.out.println(" Gap between write and physical removal");
System.out.println();
System.out.println("2. Active segment is never compacted");
System.out.println(" Recent writes stay until segment rolls");
System.out.println(" Controlled by segment.ms or segment.bytes");
System.out.println();
System.out.println("3. Tombstone retention (delete.retention.ms = 24h)");
System.out.println(" Tombstone kept so slow consumers see the delete");
System.out.println(" Old values removed only after cleaner processes segments");
}
}
Expected output:
Why old records are still visible after tombstone:
1. Compaction is asynchronous (background log cleaner)
Gap between write and physical removal
2. Active segment is never compacted
Recent writes stay until segment rolls
Controlled by segment.ms or segment.bytes
3. Tombstone retention (delete.retention.ms = 24h)
Tombstone kept so slow consumers see the delete
Old values removed only after cleaner processes segments
How do consumers handle tombstones?
A consumer reading a compacted topic must treat a null value as delete this key from my local view. If it ignores nulls, it keeps a stale entry forever.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.ConsumerRecord;
public class TombstoneConsumerHandler {
// WRONG: ignoring null values
public static void wrongHandler(ConsumerRecord<String, String> record) {
if (record.value() != null) {
System.out.println("Cache put: " + record.key() + " = " + record.value());
}
// Null values ignored -- stale entry stays in cache forever!
}
// CORRECT: treating null as delete
public static void correctHandler(ConsumerRecord<String, String> record) {
if (record.value() == null) {
System.out.println("Cache EVICT: " + record.key() + " (tombstone)");
// cache.remove(record.key());
} else {
System.out.println("Cache put: " + record.key() + " = " + record.value());
// cache.put(record.key(), record.value());
}
}
public static void main(String[] args) {
System.out.println("WRONG handler (ignores nulls):");
wrongHandler(new ConsumerRecord<>("profiles", 0, 1, "SAV2001", null));
System.out.println(" Stale entry stays in cache!");
System.out.println();
System.out.println("CORRECT handler (evicts on null):");
correctHandler(new ConsumerRecord<>("profiles", 0, 1, "SAV2001", null));
}
}
Expected output:
WRONG handler (ignores nulls):
Stale entry stays in cache!
CORRECT handler (evicts on null):
Cache EVICT: SAV2001 (tombstone)
How do you tune compaction for faster deletes?
Lower segment.ms or segment.bytes to roll segments sooner. Lower min.cleanable.dirty.ratio to make the cleaner run more aggressively. Shorten delete.retention.ms carefully.
| Config | Default | Tune For Faster Deletes |
|---|---|---|
| segment.ms | 7 days | Lower (e.g., 1 hour) |
| segment.bytes | 1 GB | Lower (e.g., 100 MB) |
| min.cleanable.dirty.ratio | 0.5 | Lower (e.g., 0.1) |
| delete.retention.ms | 24 hours | Lower (but risk slow consumer missing delete) |
# Tune compaction for faster cleanup
kafka-topics.sh --alter --topic account.profiles \
--config segment.ms=3600000 \
--config min.cleanable.dirty.ratio=0.1 \
--config delete.retention.ms=3600000 \
--bootstrap-server kafka-1:9092
What about records with null keys?
A record with a null key cannot be compacted at all, because compaction is keyed. A mistakenly null-keyed record will persist until ordinary retention removes it.
package com.bank.kafka;
public class NullKeyWarning {
public static void main(String[] args) {
System.out.println("WARNING: null-key records in compacted topics");
System.out.println(" Compaction only works on keyed records");
System.out.println(" A null-key record lingers until time/size retention deletes it");
System.out.println(" If cleanup.policy=compact only, it may stay indefinitely");
}
}
Expected output:
WARNING: null-key records in compacted topics
Compaction only works on keyed records
A null-key record lingers until time/size retention deletes it
If cleanup.policy=compact only, it may stay indefinitely
What about cleanup.policy=compact,delete?
When both policies are set, normal time/size retention applies on top of compaction. Old segments are compacted AND eventually deleted, which interacts with tombstones.
What should a senior engineer know?
Compaction is per-partition, so the cleaner's progress differs across partitions. Setting delete.retention.ms too low risks a slow consumer missing the tombstone and keeping a stale key forever. Always test that your consumers handle null values as evictions.
Visual Overview
Compaction and Tombstone Timeline
===================================
t=0: Producer writes SAV2001 = {name: Rajesh, tier: gold}
t=1: Producer writes SAV2001 = {name: Rajesh, tier: platinum}
t=2: Producer sends tombstone: SAV2001 = null
Log state after t=2:
┌──────────────────────────────────────┐
│ Closed Segment 1 (eligible) │
│ offset 0: SAV2001 = {tier: gold} │
│ offset 1: SAV2001 = {tier: platinum}│
├──────────────────────────────────────┤
│ Active Segment 2 (NOT eligible) │
│ offset 2: SAV2001 = null (tombstone)│
└──────────────────────────────────────┘
Cleaner has NOT run yet:
All three records visible to consumer
After compaction runs on Segment 1:
┌──────────────────────────────────────┐
│ Compacted Segment 1 │
│ (offset 0 and 1 removed) │
├──────────────────────────────────────┤
│ Active Segment 2 │
│ offset 2: SAV2001 = null (tombstone)│
└──────────────────────────────────────┘
After delete.retention.ms expires:
┌──────────────────────────────────────┐
│ Tombstone also removed │
│ SAV2001 completely gone │
└──────────────────────────────────────┘
Key Takeaway
In a compacted topic, deleting a key means publishing a tombstone with a null value. Compaction then promises to keep only the latest value per key and eventually remove old values and the tombstone. The reason you still see old records after sending a tombstone is that compaction is asynchronous, never touches the active segment, and retains the tombstone for delete.retention.ms so slow consumers can read the delete.
Consumers must treat null values as evictions. A consumer that ignores nulls keeps a stale entry forever, even after compaction removes the old records from the log.
In my National Bank of India compliance work, we sent a tombstone for Rajesh's profile and immediately replayed the topic -- three old versions were still visible. The cleaner had not yet run. After compaction the old versions collapsed, and the tombstone stayed for the retention window.
Remember one sentence: compaction is a lazy background job that keeps tombstones for delete.retention.ms, so old records after a delete is expected timing, not a failure -- and consumers must handle null values as evictions.
---