Definition
- IO errors -- when a log directory fills up, the broker cannot append records or write index entries, so it logs IO exceptions and marks the directory offline.
- Partition unavailability -- partitions whose leaders are on the full broker become unavailable for writes, and producers get
NotEnoughReplicasor timeout errors. - Replication stall -- follower replicas on the full broker cannot persist fetched data and fall out of the in-sync replica set, reducing durability.
- Prevention -- set
retention.msorretention.bytesper topic, monitor disk usage with alerts at 70-80%, use producer quotas, and consider tiered storage. - Recovery -- reduce retention to let the cleaner free space, add disk or brokers, and let replicas catch up and rejoin the ISR.
Imagine the clerk's desk at National Bank of India is completely covered in ledgers with no room left to write. The clerk cannot record any new transaction, so new entries get rejected and pile up at the counter. The backup clerks who copy the ledgers also cannot keep up, so their copies fall behind. The fix is to archive old ledgers off the desk regularly, watch the desk's fullness and get warned before it overflows, and get a bigger desk or more clerks before the rush.
Understanding the Concepts
What happens when a broker's disk fills up?
Kafka stores each partition replica as log segment files plus index files in a configured log directory on local disk. When that directory's filesystem runs out of space, any attempt to append a record or extend an index fails with an IO exception. Kafka marks that log directory offline, takes the partitions hosted there out of service, and may shut the broker down.
package com.bank.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.math.BigDecimal;
import java.util.Properties;
import java.util.concurrent.ExecutionException;
public class DiskFullProducerDemo {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.ACKS_CONFIG, "all");
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
try {
producer.send(new ProducerRecord<>("transfers", "SAV2001",
"DEBIT " + new BigDecimal("50000"))).get();
System.out.println("Transfer recorded successfully");
} catch (ExecutionException e) {
// When broker disk is full, producer gets this error
System.out.println("ERROR: " + e.getCause().getClass().getSimpleName());
System.out.println("Broker disk may be full -- transfer not recorded");
// NotEnoughReplicasException or TimeoutException
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
Expected output (when broker disk is full):
ERROR: NotEnoughReplicasException
Broker disk may be full -- transfer not recorded
How does disk full affect replication and durability?
Replication is affected because a follower on the full broker cannot write the data it fetches, so it stops making progress and is removed from the in-sync replica set. This reduces the redundancy of every partition that had a replica there, so a second failure becomes more dangerous.
| Scenario | Impact |
|---|---|
| Leader on full broker | Writes fail, producers get errors |
| Follower on full broker | Falls out of ISR, reduces durability |
| All replicas full | Partition completely unavailable |
| Other brokers healthy | Kafka elects new leader from ISR |
# Check which partitions are under-replicated after disk full
kafka-topics.sh --describe --under-replicated-partitions \
--bootstrap-server kafka-2:9092
Topic: transfers Partition: 3 Leader: 2 Replicas: 1,2,3 Isr: 2,3
(Broker 1 dropped from ISR -- disk full)
Topic: transfers Partition: 7 Leader: 2 Replicas: 1,2,3 Isr: 2,3
How do you prevent disk full with retention settings?
You set retention.ms or retention.bytes per topic so old data is deleted on a schedule. The log cleaner runs periodically based on log.retention.check.interval.ms.
# Set retention to 7 days and 100 GB max for the transfers topic
kafka-topics.sh --alter --topic transfers \
--config retention.ms=604800000 \
--config retention.bytes=107374182400 \
--bootstrap-server kafka-1:9092
# Verify retention settings
kafka-configs.sh --describe --topic transfers \
--bootstrap-server kafka-1:9092
Dynamic configs for topic transfers:
retention.ms=604800000
retention.bytes=107374182400
How do you monitor disk usage and set alerts?
Monitor disk usage per log directory and alert well before full, commonly at seventy or eighty percent, so operators can act before the broker goes offline.
package com.bank.kafka;
import java.io.File;
public class DiskMonitor {
public static void checkDiskUsage(String logDir) {
File dir = new File(logDir);
long total = dir.getTotalSpace();
long free = dir.getFreeSpace();
long used = total - free;
double usagePercent = (double) used / total * 100;
System.out.printf("Kafka log dir: %s%n", logDir);
System.out.printf(" Total: %.1f GB%n", total / 1e9);
System.out.printf(" Used: %.1f GB (%.1f%%)%n", used / 1e9, usagePercent);
System.out.printf(" Free: %.1f GB%n", free / 1e9);
if (usagePercent > 80) {
System.out.println(" ALERT: Disk usage above 80%% -- take action!");
} else if (usagePercent > 70) {
System.out.println(" WARNING: Disk usage above 70%% -- monitor closely");
} else {
System.out.println(" OK: Disk usage within safe limits");
}
}
public static void main(String[] args) {
checkDiskUsage("/var/kafka/data");
}
}
Expected output:
Kafka log dir: /var/kafka/data
Total: 500.0 GB
Used: 410.0 GB (82.0%)
Free: 90.0 GB
ALERT: Disk usage above 80% -- take action!
How do you use quotas to prevent runaway producers?
Producer quotas stop a runaway producer from filling disks faster than cleanup frees space. You set byte-rate limits per client id or per user.
# Set a 10 MB/s produce quota for the batch-import client
kafka-configs.sh --alter --add-config 'producer_byte_rate=10485760' \
--entity-type clients --entity-name batch-import \
--bootstrap-server kafka-1:9092
How do you recover from a disk full situation?
Recovery means freeing space or adding it. Reduce retention temporarily so the cleaner deletes more, add disk capacity, or add brokers and reassign partitions. Never hand-delete segment files, which is risky; let Kafka manage cleanup through retention settings.
# Emergency: reduce retention temporarily to free space
kafka-topics.sh --alter --topic transfers \
--config retention.ms=86400000 \
--bootstrap-server kafka-1:9092
# After space is freed, restore normal retention
kafka-topics.sh --alter --topic transfers \
--config retention.ms=604800000 \
--bootstrap-server kafka-1:9092
# Check that the affected replicas have rejoined the ISR
kafka-topics.sh --describe --topic transfers --bootstrap-server kafka-1:9092
What are common mistakes in disk management?
package com.bank.kafka;
public class DiskManagementMistakes {
public static void main(String[] args) {
// WRONG: manually deleting segment files
System.out.println("WRONG: rm /var/kafka/data/transfers-3/*.log");
System.out.println(" Risk: corrupts partition, broker may crash");
// CORRECT: reduce retention and let Kafka clean up
System.out.println("CORRECT: kafka-topics --alter --config retention.ms=86400000");
System.out.println(" Safe: Kafka's log cleaner handles deletion properly");
// WRONG: Kafka data on same volume as OS
System.out.println("WRONG: /var/kafka/data on root filesystem");
System.out.println(" Risk: full data disk crashes the OS and logging");
// CORRECT: separate data volume
System.out.println("CORRECT: /var/kafka/data on dedicated volume");
System.out.println(" Safe: OS stays up even if data disk fills");
}
}
Expected output:
WRONG: rm /var/kafka/data/transfers-3/*.log
Risk: corrupts partition, broker may crash
CORRECT: kafka-topics --alter --config retention.ms=86400000
Safe: Kafka's log cleaner handles deletion properly
WRONG: /var/kafka/data on root filesystem
Risk: full data disk crashes the OS and logging
CORRECT: /var/kafka/data on dedicated volume
Safe: OS stays up even if data disk fills
What should a senior engineer know?
Tiered storage largely prevents this class of incident for high-retention topics by keeping only recent segments locally. Spread partitions and data evenly because an unbalanced cluster can fill one broker while others are near empty. Separate the Kafka data volume from the operating-system volume so a full data disk does not also crash the OS. Alert on disk usage trend, not just absolute level, to catch gradual growth before it becomes urgent.
Visual Overview
Disk Full Impact Chain
=======================
┌──────────────────┐
│ Broker Disk Full │
└────────┬─────────┘
│
▼
┌──────────────────────────────────────┐
│ IO Error: cannot append to segments │
└────────┬─────────────────────────────┘
│
┌────┴────────────┐
▼ ▼
┌────────────┐ ┌──────────────────┐
│ Leader on │ │ Follower on │
│ this broker│ │ this broker │
└─────┬──────┘ └───────┬──────────┘
│ │
▼ ▼
┌────────────┐ ┌──────────────────┐
│ Writes fail│ │ Falls out of ISR │
│ Producers │ │ Durability │
│ get errors │ │ reduced │
└─────┬──────┘ └───────┬──────────┘
│ │
▼ ▼
┌────────────────────────────────────┐
│ If healthy ISR exists elsewhere: │
│ New leader elected on other │
│ broker -- partition recovers │
│ If no healthy ISR: │
│ Partition completely unavailable │
└────────────────────────────────────┘
Prevention Stack
================
retention.ms / retention.bytes ──► Old data cleaned up
Disk monitoring (alert at 70%) ──► Early warning
Producer quotas ──► Prevent runaway writes
Tiered storage ──► Local disk stays small
Separate data volume ──► OS survives disk full
Key Takeaway
When a Kafka broker's log directory fills up, the broker can no longer append records or write index files, so it logs IO errors, marks that log directory offline, and takes its hosted partitions out of service. For partitions whose leader was on the full broker, writes fail and producers get rejected. Follower replicas on the full broker fall out of the in-sync replica set, reducing durability across the cluster.
I prevent it with retention settings, disk monitoring with alerts at seventy to eighty percent, producer quotas, and tiered storage for high-retention topics. For recovery I never hand-delete segment files; instead I reduce retention to let the cleaner free space, add disk or brokers, and let the recovered replicas catch up and rejoin the ISR.
In my National Bank of India project, a surge plus a retention misconfiguration once filled one broker. We recovered by temporarily cutting retention and adding capacity, and we hardened the setup by separating the data volume from the OS volume, balancing partitions evenly, and adopting tiered storage.
Remember one sentence: a full broker disk takes partitions offline and erodes durability, so prevent it with retention, monitoring, quotas, and separate data volumes.
---