Why does Kafka retention cleanup cause spikes in disk I/O and temporarily degrade broker performance, and why is it worse on cloud disks like EBS?

Kafka retention cleanup causes disk I/O spikes because of how segments are stored and deleted.

• Segment file – the on-disk unit Kafka stores records in, typically up to 1 GB, with matching index files.
• Retention cleanup – the background thread that scans and deletes whole closed segments once they age out by time or size.
• Random I/O – scattered metadata operations (inode updates, block deallocation) that are far slower than Kafka's normal sequential appends.
• Page cache eviction – when a segment is deleted, the OS frees its cached pages, so consumers that relied on memory reads now hit disk.
• EBS amplification – a cloud block device turns a single logical delete into multiple backend operations, draining provisioned IOPS faster than local disks.

What is retention cleanup in Kafka?

Retention cleanup is the background process that removes old data from a Kafka partition. A partition is stored as a series of segment files on disk. When a closed segment's records are all older than retention.ms (default seven days) or the partition exceeds retention.bytes, the cleanup thread deletes the entire segment file plus its index files. This is the cleanup.policy=delete behaviour. The cleanup thread runs on an interval controlled by log.retention.check.interval.ms (default five minutes), scanning eligible segments and deleting them in a batch.

At National Bank of India, the settlement topic keeps seven days of transaction history. Every five minutes the cleaner scans partition directories and removes segments whose newest record is older than seven days.

# Check current retention settings for the settlements topic
kafka-configs.sh --describe --topic settlements \
  --bootstrap-server kafka-1:9092

# Output:
# retention.ms=604800000   (7 days)
# segment.bytes=1073741824 (1 GB)
# cleanup.policy=delete

Why does deleting segment files cause random I/O?

Kafka's normal workload is sequential: producers append to the end of the active segment, and consumers read forward through recent segments. Sequential I/O is fast because the disk head barely moves on spinning disks and SSDs write contiguous blocks efficiently.

Deleting a file is different. The filesystem must update the inode to mark it deleted, free the blocks the file occupied in the block allocation bitmap, and update directory entries. These are small, scattered, metadata-heavy operations spread across the disk. When retention deletes many one-gigabyte segments at once, it fires a burst of random metadata I/O on top of the steady sequential traffic, and random I/O is orders of magnitude slower per operation.

package com.bank.ops;

import java.io.IOException;
import java.nio.file.*;
import java.util.stream.Stream;

/**
 * Demonstrates the kind of file-delete operation Kafka's cleanup does.
 * Each delete triggers filesystem metadata updates (inode, blocks).
 */
public class SegmentCleanupDemo {
    public static void main(String[] args) throws IOException {
        Path partitionDir = Paths.get("/data/kafka/settlements-0");
        long retentionMs = 7 * 24 * 60 * 60 * 1000L; // 7 days
        long cutoff = System.currentTimeMillis() - retentionMs;

        try (Stream<Path> files = Files.list(partitionDir)) {
            files.filter(p -> p.toString().endsWith(".log"))
                 .filter(p -> {
                     try { return Files.getLastModifiedTime(p).toMillis() < cutoff; }
                     catch (IOException e) { return false; }
                 })
                 .forEach(p -> {
                     try {
                         Files.delete(p); // random I/O: inode + block dealloc
                         System.out.println("Deleted segment: " + p.getFileName());
                     } catch (IOException e) {
                         System.err.println("Failed: " + e.getMessage());
                     }
                 });
        }
    }
}

Deleted segment: 00000000000000000000.log
Deleted segment: 00000000000001048576.log
Deleted segment: 00000000000002097152.log

How does retention cleanup disrupt the OS page cache?

Kafka deliberately relies on the OS page cache so that consumers reading recent data are served from memory, not disk. When a segment file is deleted, the kernel invalidates and frees the page cache entries backing that file. During a burst of deletes, large amounts of cache are churned, and the freed pages may be reallocated to other uses. Consumers that were served from memory now miss the cache and read from disk, which both slows them and adds more disk load precisely when the disk is already busy with deletes.

package com.bank.ops;

/**
 * Conceptual illustration: after cleanup, consumer reads miss the page cache.
 * In reality Kafka relies on OS-level caching, not application-level.
 */
public class PageCacheImpactDemo {
    public static void main(String[] args) {
        // Before cleanup: consumer reads recent segments from page cache
        System.out.println("Consumer read latency (cached):   ~0.1 ms");

        // During cleanup: deleted segment pages are evicted
        System.out.println("Cleanup deletes 5 x 1 GB segments...");
        System.out.println("Page cache entries for those 5 GB are freed.");

        // After cleanup: nearby pages may also be evicted under memory pressure
        System.out.println("Consumer read latency (disk):     ~5.0 ms");
        System.out.println("Consumer lag spikes until cache warms again.");
    }
}

Consumer read latency (cached):   ~0.1 ms
Cleanup deletes 5 x 1 GB segments...
Page cache entries for those 5 GB are freed.
Consumer read latency (disk):     ~5.0 ms
Consumer lag spikes until cache warms again.

Why are the I/O spikes periodic and bursty?

The log cleanup thread does not trickle deletes continuously. It runs on log.retention.check.interval.ms, scanning for eligible segments, deleting a batch, then sleeping until the next cycle. This scan-delete-sleep rhythm means the I/O cost arrives in periodic spikes rather than a smooth background load. Monitoring shows regular saw-tooth patterns in disk utilization and latency aligned to the check interval.

# Broker server.properties: how often the cleaner checks
log.retention.check.interval.ms=300000  # every 5 minutes

# Observation: every 5 minutes, disk util jumps from 20% to 80%
# and settle back after the batch is done

Why is this worse on cloud disks like EBS?

A local SSD handles a delete close to the hardware. An EBS volume is a network-attached, virtualized block device. Operations traverse a storage backend that must track and reclaim blocks across a distributed system. A delete that is one operation locally can expand into multiple backend operations. EBS also enforces provisioned IOPS and throughput limits, so a burst of delete-driven random I/O can exhaust your IOPS budget and get throttled, stretching a brief blip into a sustained latency event.

# AWS CloudWatch metrics to watch during retention cleanup
# VolumeReadOps, VolumeWriteOps    -- spike during deletes
# VolumeQueueLength                -- grows when IOPS exhausted
# BurstBalance (gp2/gp3)           -- drains under sustained random I/O

# Example: a gp3 volume provisioned at 3000 IOPS
# Retention burst uses 5000 IOPS for 30 seconds
# Result: throttled, latency spikes to 50ms+

How do you configure retention to reduce disk I/O spikes?

The mitigations follow from the causes. Smaller segments mean each delete removes a smaller file, so the random I/O burst per delete is smaller and more frequent rather than huge and rare, smoothing the curve. Staggering retention across topics avoids all deletes piling into one cycle. Provisioning cloud storage with enough IOPS headroom absorbs the bursts.

# Smaller segments: 256 MB instead of 1 GB, roll hourly
kafka-configs.sh --alter --topic settlements \
  --bootstrap-server kafka-1:9092 \
  --add-config segment.bytes=268435456,segment.ms=3600000,retention.ms=259200000

# Stagger retention check intervals across broker groups
# Broker-1: log.retention.check.interval.ms=300000
# Broker-2: log.retention.check.interval.ms=330000
# Broker-3: log.retention.check.interval.ms=360000

What is the difference between delete retention and compaction cleanup?

Delete retention (cleanup.policy=delete) removes entire aged segments. Compaction (cleanup.policy=compact) rewrites segments to keep only the latest value per key. Both cause I/O, but the pattern differs.

| Aspect | Delete Retention | Compaction |
|---|---|---|
| Operation | Deletes whole segment files | Reads and rewrites segments |
| I/O type | Random metadata I/O (inode, block free) | Sequential read + write (rewrite) |
| Page cache impact | Evicts cached pages of deleted files | Replaces old pages with rewritten data |
| Frequency | Periodic bursts on check interval | Continuous based on dirty ratio |
| Tuning knob | segment.bytes, retention.check.interval | min.cleanable.dirty.ratio |

# Delete retention topic
kafka-configs.sh --alter --topic settlements \
  --bootstrap-server kafka-1:9092 \
  --add-config cleanup.policy=delete

# Compaction topic
kafka-configs.sh --alter --topic account-state \
  --bootstrap-server kafka-1:9092 \
  --add-config cleanup.policy=compact

What are common mistakes when tuning retention cleanup?

The most common mistake is making segments too small to smooth cleanup and then suffering excessive file-handle and index overhead during steady-state operations.

# WRONG: segments so small that the broker has thousands of open files
kafka-configs.sh --alter --topic settlements \
  --bootstrap-server kafka-1:9092 \
  --add-config segment.bytes=16777216  # 16 MB -- too small!

# Result: 'Too many open files' errors, high index overhead
# Each segment = 3 files (.log, .index, .timeindex)
# 1 TB topic / 16 MB = 65,536 segments = 196,608 files!

# CORRECT: balance cleanup smoothness against file-handle pressure
kafka-configs.sh --alter --topic settlements \
  --bootstrap-server kafka-1:9092 \
  --add-config segment.bytes=268435456  # 256 MB -- good balance

# Result: 1 TB topic / 256 MB = 4,096 segments = 12,288 files

What should a senior engineer know?

This is specifically about delete cleanup for cleanup.policy=delete. Log compaction has its own cleaner with different but related I/O characteristics. Watch filesystem busy percentage, average I/O service time, and for EBS the volume queue length and burst-balance credits. A draining burst balance is the early sign that delete bursts are exhausting your IOPS. Tiered storage moves older segments to object storage so local deletes shrink, reducing both local delete pressure and the amplified operations on the hot block device.

# Monitor key metrics
# JMX: kafka.log:type=LogCleanerManager,name=max-dirty-percent
# OS:  iostat -x 1 | grep sda   (watch await and util columns)
# AWS: aws cloudwatch get-metric-data --metric-name BurstBalance

# Tiered storage (KIP-405, available in newer Kafka)
# Moves cold segments to S3, reducing local deletes
kafka-configs.sh --alter --topic settlements \
  --bootstrap-server kafka-1:9092 \
  --add-config remote.storage.enable=true,local.retention.ms=86400000