Compaction & Retention

Deletion in a compacted topic is a tombstone, and compaction is a lazy background job that skips the active segment and keeps tombstones for `delete.retention.ms`, so seeing old records for a deleted key right after the delete is expected timing, not a failure.

Compaction keeps the latest value per key, but tombstones expire after delete.retention.ms, compact-plus-delete adds time expiry, null keys bypass compaction entirely, and aggressive settings purge old records instantly -- all expected, all documented.

Retention cleanup deletes big segment files in periodic bursts, and those deletes are metadata-heavy random I/O that also evict page cache, so you see spiky disk I/O and latency; on EBS the virtualized storage amplifies each delete, making it worse.

A partition is a directory of immutable closed segments plus one active segment, each paired with an offset index and a time index; retention deletes whole aged segments while compaction rewrites them to latest-per-key.

Retention deletes old records by age or size; compaction keeps the latest record per key and removes older ones.