Definition
- Log retention – the default cleanup policy (
cleanup.policy=delete) that deletes whole log segments once they exceedretention.ms(time) orretention.bytes(size), regardless of record keys. - Log compaction – the cleanup policy (
cleanup.policy=compact) where the log cleaner keeps the latest record per distinct key and removes older records for the same key, creating a durable snapshot. - Tombstone – a record with a key and a null value in a compacted topic that signals the key should be deleted; kept for
delete.retention.msbefore purging. cleanup.policy– the topic configuration that controls which cleanup mechanism applies:delete,compact, orcompact,delete(both).- Log cleaner – the background thread that scans compacted partitions and removes older records for keys that have a newer entry.
Imagine National Bank of India keeps a log of balance updates. With retention, the bank keeps every update slip for the last seven days and then shreds the older slips in batches: you see recent history but old slips are gone. With compaction, the bank keeps only the most recent balance slip for each account and shreds the older slips for that same account: for account SAV2001 you always have the latest balance, forever, but not the full history of every change.
Understanding the Concepts
What is log retention?
Time-and-size retention is the default cleanup policy, configured with retention.ms (how long to keep records) and retention.bytes (maximum partition size). Kafka stores a partition as a series of segment files. When the oldest segment's records exceed the retention time, or the partition exceeds the retention size, Kafka deletes that whole segment. Retention does not look at keys at all; it simply removes old data in bulk.
package com.bank.retention;
public class RetentionPolicyExample {
public static void main(String[] args) {
System.out.println("Retention Policy (cleanup.policy=delete):");
System.out.println();
System.out.println("retention.ms = 604800000 (7 days)");
System.out.println("retention.bytes = -1 (unlimited size)");
System.out.println();
System.out.println("Day 1: [transfer-1] [transfer-2] [transfer-3]");
System.out.println("Day 7: [transfer-1..1000] (7 days of transfers)");
System.out.println("Day 8: [transfer-100..1000] (day 1 segment deleted)");
System.out.println();
System.out.println("All keys affected equally. No key awareness.");
}
}
Expected output:
Retention: keeps records for 7 days, then deletes old segments.
No key awareness. All keys age out equally.
What is log compaction?
Log compaction keeps, for each distinct key, at least the latest record and removes older records with the same key. The result is that the partition always holds the latest value for every key that has ever been written, so it acts like a durable key-value snapshot.
package com.bank.compaction;
public class CompactionExample {
public static void main(String[] args) {
System.out.println("Compaction (cleanup.policy=compact):");
System.out.println();
System.out.println("Before compaction:");
System.out.println(" offset 0: key=SAV2001 balance=10000");
System.out.println(" offset 1: key=SAV3002 balance=25000");
System.out.println(" offset 2: key=SAV2001 balance=15000");
System.out.println(" offset 3: key=SAV2001 balance=20000");
System.out.println(" offset 4: key=SAV3002 balance=30000");
System.out.println();
System.out.println("After compaction:");
System.out.println(" offset 3: key=SAV2001 balance=20000 (latest)");
System.out.println(" offset 4: key=SAV3002 balance=30000 (latest)");
System.out.println();
System.out.println("Offsets 0, 1, 2 removed (older versions of same keys)");
}
}
Expected output:
Before: 5 records (3 for SAV2001, 2 for SAV3002)
After: 2 records (latest per key)
Older versions removed by log cleaner.
How is retention different from compaction? (comparison table)
| Feature | Retention (delete) | Compaction (compact) |
|---|---|---|
| Trigger | Age or size threshold | Duplicate keys |
| What is removed | Entire old segments | Older records per key |
| Key awareness | No | Yes |
| Preserves history | Rolling window (recent) | No (only latest per key) |
| Preserves latest per key | Not guaranteed | Guaranteed |
| Storage growth | Bounded by time/size | Bounded by unique key count |
| Use case | Event streams (transfers, logs) | State snapshots (balances, configs) |
package com.bank.comparison;
public class RetentionVsCompaction {
public static void main(String[] args) {
System.out.println("Retention: rolling window of full event history");
System.out.println(" transfers topic: 7 days of all transfers");
System.out.println(" After 7 days, old transfers deleted");
System.out.println();
System.out.println("Compaction: latest value per key forever");
System.out.println(" account-balances topic: current balance per account");
System.out.println(" Old balance updates removed, latest kept");
}
}
Retention: keeps recent history, removes by age
Compaction: keeps latest per key, removes older duplicates
What is a tombstone in compaction?
A record with a key and a null value is called a tombstone. It signals that the key should be deleted. Compaction keeps the tombstone for delete.retention.ms so downstream consumers learn the key was removed, then eventually purges it.
package com.bank.compaction;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
@Service
public class AccountBalancePublisher {
private final KafkaTemplate<String, String> kafkaTemplate;
public AccountBalancePublisher(
KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void updateBalance(String accountId, String balanceJson) {
kafkaTemplate.send("account-balances", accountId, balanceJson);
System.out.println("Balance updated: " + accountId);
}
public void closeAccount(String accountId) {
// Tombstone: null value signals deletion
kafkaTemplate.send("account-balances", accountId, null);
System.out.println("Tombstone sent for closed account: " + accountId);
// Downstream caches will drop this key after reading the tombstone
}
}
Expected output:
Balance updated: SAV2001
Tombstone sent for closed account: SAV2001
Compacted topic after tombstone:
SAV2001: null (tombstone, kept for delete.retention.ms)
SAV3002: {"balance":30000} (latest balance)
After delete.retention.ms expires:
SAV2001: removed entirely
SAV3002: {"balance":30000}
When do you use retention vs compaction?
Event topics are historical facts ("a transfer happened"), so they use retention with a window sized for replay and compliance. State topics are current values ("this account's balance"), so they use compaction. You can combine both with compact,delete.
package com.bank.config;
public class CleanupPolicyDecision {
public static void main(String[] args) {
System.out.println("Cleanup Policy Decision:");
System.out.println();
System.out.println("Event topics (cleanup.policy=delete):");
System.out.println(" transfer-events: retention 7 days");
System.out.println(" audit-events: retention 90 days (compliance)");
System.out.println(" app-logs: retention 3 days");
System.out.println();
System.out.println("State topics (cleanup.policy=compact):");
System.out.println(" account-balances: latest balance per account");
System.out.println(" customer-profiles: latest profile per customer");
System.out.println(" __consumer_offsets: latest offset per group");
System.out.println();
System.out.println("Both (cleanup.policy=compact,delete):");
System.out.println(" session-state: compact recent + delete old");
}
}
Expected output:
Events: retention (delete old by time)
State: compaction (keep latest per key)
Both: compact,delete (compact recent, delete very old)
What are common mistakes with cleanup policies?
The most dangerous mistake is compacting an event topic, which silently drops intermediate transactions. Another mistake is using plain retention for a state topic, which eventually deletes the only copy of a current balance.
package com.bank.mistakes;
// WRONG: Compacting an event feed.
public class WrongCompactionOnEvents {
public static void main(String[] args) {
System.out.println("WRONG: cleanup.policy=compact on transfer-events");
System.out.println();
System.out.println("Before compaction:");
System.out.println(" SAV2001: DEBIT 50000 (offset 0)");
System.out.println(" SAV2001: CREDIT 20000 (offset 1)");
System.out.println(" SAV2001: DEBIT 30000 (offset 2)");
System.out.println();
System.out.println("After compaction:");
System.out.println(" SAV2001: DEBIT 30000 (only offset 2 kept)");
System.out.println(" Offsets 0 and 1 SILENTLY DROPPED!");
System.out.println(" Audit trail broken, transfers lost from history");
}
}
package com.bank.mistakes;
// CORRECT: Retention for events, compaction for state.
public class CorrectCleanupPolicy {
public static void main(String[] args) {
System.out.println("CORRECT:");
System.out.println(" transfer-events: cleanup.policy=delete, retention=7d");
System.out.println(" -> All transfers kept for 7 days (full history)");
System.out.println();
System.out.println(" account-balances: cleanup.policy=compact");
System.out.println(" -> Latest balance per account kept forever");
}
}
WRONG: compacting events = silently drops intermediate transactions
CORRECT: retention for events, compaction for state
How does compaction work operationally?
The log cleaner background thread scans the partition and builds a map of keys to their latest offset. It then rewrites the older segments, keeping only the latest record per key. There is always a head segment that has not yet been compacted.
package com.bank.operations;
public class CompactionOperations {
public static void main(String[] args) {
System.out.println("Compaction Operations:");
System.out.println();
System.out.println("min.cleanable.dirty.ratio = 0.5");
System.out.println(" Cleaner runs when 50% of log is dirty (uncompacted)");
System.out.println();
System.out.println("delete.retention.ms = 86400000 (24 hours)");
System.out.println(" Tombstones kept for 24 hours before removal");
System.out.println();
System.out.println("Active segment: never compacted (head of log)");
System.out.println(" Recent writes may have duplicate keys temporarily");
System.out.println(" Consumer reading from start sees latest after compaction");
}
}
Expected output:
Cleaner runs when 50% dirty. Tombstones kept 24h.
Active segment not compacted (temporary duplicates possible).
What should a senior engineer know?
The classic pattern is event sourcing plus a compacted state topic: raw events live in a retention topic for audit and replay, while a derived compacted topic holds the current materialized state for fast lookups.
package com.bank.senior;
public class EventSourcingPlusCompaction {
public static void main(String[] args) {
System.out.println("Event Sourcing + Compacted State:");
System.out.println();
System.out.println("transfer-events (cleanup.policy=delete, 90 days):");
System.out.println(" Full history of every transfer");
System.out.println(" Used for: audit, replay, compliance");
System.out.println();
System.out.println("account-balances (cleanup.policy=compact):");
System.out.println(" Derived from transfer-events via Kafka Streams");
System.out.println(" Latest balance per account");
System.out.println(" Used for: fast current-value lookups");
System.out.println();
System.out.println("Both topics serve different needs from same data.");
}
}
Expected output:
transfer-events (retention): full history for audit
account-balances (compaction): latest value for lookups
Visual Overview
Retention vs Compaction
========================
Retention (cleanup.policy=delete):
Day 1 Day 2 Day 7 Day 8
[e1][e2] -> [e1..e50] -> [e1..e1000] -> [e100..e1000]
▲
Day 1 segment
DELETED (aged out)
Entire segments removed by time/size.
Compaction (cleanup.policy=compact):
Before cleaner runs:
[SAV2001=10K][SAV3002=25K][SAV2001=15K][SAV2001=20K][SAV3002=30K]
▲ ▲ ▲ ▲
older older latest latest
After cleaner runs:
[SAV2001=20K][SAV3002=30K]
latest latest
Only the latest value per key survives.
Tombstone (key deletion):
[SAV2001=null] <- tombstone
Consumers see: SAV2001 is deleted
After delete.retention.ms: tombstone removed entirely
Key Takeaway
Log retention and log compaction are two fundamentally different cleanup policies that determine what a Kafka partition stores over time. Retention deletes entire old segments by age or size, keeping a rolling window of the complete event history. Compaction keeps the latest record per distinct key and removes older records, creating a durable latest-value-per-key snapshot. The choice is set per topic with cleanup.policy.
The key difference is history versus state. Retention preserves the full ordered sequence of events within the retained window, so replaying gives every recent event. Compaction does not preserve intermediate updates; replaying a compacted topic gives only the current value per key. Tombstones (null-value records) signal key deletion in compacted topics.
At National Bank of India, the transfer-events topic uses retention of seven days for full audit history and replay, while the derived account-balances topic uses compaction so there is always a current balance per account without storing every historical change. Closing an account writes a tombstone so downstream caches drop it. The classic pattern is event sourcing: raw events in a retention topic for compliance, plus a compacted state topic for fast current-value lookups.
Remember one sentence: retention keeps a time window of all events, compaction keeps the latest value per key, and choosing the wrong policy is dangerous because compacting events drops history and retaining state eventually deletes current values.
---