Definition
Kafka achieves very high throughput on commodity hardware through three reinforcing design choices.
- Sequential append-only writes – producers only append to the end of the active segment, so the disk sees large contiguous writes instead of scattered random I/O.
- Zero-copy transfer – consumer reads use the OS
sendfilesystem call (Java'sFileChannel.transferTo) to move bytes directly from the page cache to the network socket without copying through the JVM heap. - OS page cache – Kafka stores no in-process cache; the OS caches recently written segment data in memory, serving tail consumers from RAM for free.
- Small JVM heap – because the cache is in the kernel, the broker runs with a small heap, keeping garbage-collection pauses short and predictable.
- Warm restarts – the page cache survives a process restart, so a restarted broker serves recent data from memory immediately.
Imagine National Bank of India records every transaction in a long ledger. Instead of flipping around the ledger to find a spot for each entry (random I/O), clerks always write the next entry on the very next line at the end (sequential append), which is fast and smooth. When head office asks for a copy of recent pages, instead of a clerk photocopying each page and walking it over, the bank uses a direct pass-through chute from the shelf to the courier (zero-copy). And because the most recent pages are already lying open on the desk from being written, anyone who wants them just reads them off the desk instead of fetching from the vault (page cache).
Understanding the Concepts
What are sequential append-only writes?
A Kafka partition is an ordered log, and producers only ever append to the end of the active segment. Appending sequentially means the disk head on a spinning disk barely moves, and on an SSD the controller writes large contiguous blocks efficiently with minimal write amplification. Random I/O, by contrast, scatters small writes all over the disk, forcing seeks on HDDs and expensive read-modify-write cycles on SSDs. The performance gap is often one to two orders of magnitude.
package com.bank.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.math.BigDecimal;
import java.util.Properties;
public class SequentialAppendDemo {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 65536); // batch for throughput
props.put(ProducerConfig.LINGER_MS_CONFIG, 5); // wait 5ms to fill batch
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
for (int i = 0; i < 100_000; i++) {
String txn = "{\"account\":\"SAV2001\",\"amount\":\"" +
new BigDecimal("100.00") + "\"}";
// Every send appends to the end of the active segment -- sequential
producer.send(new ProducerRecord<>("card.authorizations", "SAV2001", txn));
}
System.out.println("Sent 100K records -- all sequential appends");
}
}
}
Sent 100K records -- all sequential appends
What is zero-copy transfer?
When a consumer fetches records, those bytes often already sit in the OS page cache. Without zero-copy, serving them requires copying data from the kernel page cache into user-space, then back into the kernel socket buffer, then out to the NIC -- multiple copies and context switches. Kafka uses the sendfile system call, which tells the kernel to transfer bytes directly from the page cache to the network socket without ever copying into the JVM.
package com.bank.kafka;
import java.io.IOException;
import java.nio.channels.*;
import java.nio.file.*;
/**
* Conceptual demo: how Kafka serves a fetch using zero-copy.
* In reality Kafka calls FileChannel.transferTo internally.
*/
public class ZeroCopyDemo {
public static void main(String[] args) throws IOException {
Path segmentFile = Paths.get("/data/kafka/card.authorizations-0/00000000000000000000.log");
try (FileChannel fileChannel = FileChannel.open(segmentFile, StandardOpenOption.READ);
SocketChannel socketChannel = SocketChannel.open()) {
// Zero-copy: kernel moves bytes file -> socket directly
long transferred = fileChannel.transferTo(0, fileChannel.size(), socketChannel);
System.out.println("Transferred " + transferred + " bytes via zero-copy");
// No copy into JVM heap, minimal context switches
}
}
}
Transferred 1073741824 bytes via zero-copy
How does the OS page cache replace an in-process cache?
Many systems build a large in-memory cache inside the application process. Kafka deliberately does not. It lets the OS cache file data in the page cache. Recently written data is already in the page cache, so consumers reading the tail of the log (the common case) are served from RAM without any disk read. Because the cache lives in the kernel and not the JVM heap, Kafka runs with a small heap, keeping GC pauses short.
# On a Kafka broker, check how much RAM is used as page cache
free -h
# total used free shared buff/cache available
# Mem: 64G 4.5G 10G 0.1G 49G 58G
# The 49 GB in buff/cache is the OS page cache serving Kafka reads
# Kafka broker JVM heap: deliberately small
# KAFKA_HEAP_OPTS="-Xms6g -Xmx6g"
# The remaining ~58 GB is available for the OS page cache
How do these three pillars reinforce each other?
Sequential writes keep recent data contiguous and hot in the page cache. The page cache makes most reads hit memory. Zero-copy ships those cached bytes to consumers without touching the heap. The net effect is that the common path -- producers appending and consumers reading the tail -- involves almost no random disk access, no user-space copying, and no JVM memory pressure.
Common path (hot tail read): Producer append → OS page cache (write) Consumer fetch → OS page cache (read) → sendfile → socket → NIC Disk not touched. JVM heap not touched. Cold path (old data read): Consumer fetch → page cache miss → disk read → page cache → sendfile → socket Disk touched once. JVM heap still not touched.
When is zero-copy bypassed?
Zero-copy is bypassed when the broker must transform bytes. With TLS encryption enabled, data must pass through user space for encryption. When the broker must down-convert message formats for an older client, it must read, convert, and rewrite. Both add CPU and copies.
package com.bank.kafka;
/**
* Conceptual: TLS forces data through user space, bypassing zero-copy.
*/
public class TlsImpactDemo {
public static void main(String[] args) {
// Without TLS: sendfile path, ~2 Gbps per broker
System.out.println("Zero-copy path: page cache -> socket (no JVM copy)");
System.out.println("Throughput: ~2 Gbps per broker");
// With TLS: data must be encrypted in user space
System.out.println("TLS path: page cache -> JVM -> encrypt -> socket");
System.out.println("Throughput: ~1.2 Gbps per broker (CPU-bound on encryption)");
System.out.println("Keep client and broker formats aligned to preserve zero-copy");
}
}
Zero-copy path: page cache -> socket (no JVM copy)
Throughput: ~2 Gbps per broker
TLS path: page cache -> JVM -> encrypt -> socket
Throughput: ~1.2 Gbps per broker (CPU-bound on encryption)
Keep client and broker formats aligned to preserve zero-copy
How does Kafka achieve durability without fsync per message?
Kafka relies on the page cache and the OS to flush to disk lazily, trading an fsync-per-message for throughput. Durability comes from replication across brokers: with acks=all and min.insync.replicas=2, a record is committed only when multiple brokers have it in their page caches, so even if one broker loses power before flushing, the others still have the data.
package com.bank.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class DurabilityConfigDemo {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// Durability from replication, not fsync
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
producer.send(new ProducerRecord<>("settlements", "SAV2001",
"{\"amount\":50000}")).get();
System.out.println("Settlement committed to all ISR replicas");
}
}
}
Settlement committed to all ISR replicas
How should you size memory for a Kafka broker?
Because Kafka leans on the page cache, give the OS plenty of free RAM for caching rather than sizing a huge JVM heap. The page cache also survives a broker process restart, so the broker serves hot data from a warm cache immediately.
# Good: small heap, lots of free RAM for page cache
KAFKA_HEAP_OPTS="-Xms6g -Xmx6g" # 6 GB heap
# On a 64 GB server, ~58 GB is available for page cache
# Bad: huge heap, page cache starved
KAFKA_HEAP_OPTS="-Xms48g -Xmx48g" # 48 GB heap
# Only ~16 GB left for page cache -- consumers will hit disk
# Also: long GC pauses destabilize the broker
What should a senior engineer know?
Kafka wins by turning messaging into sequential log appends and cached, zero-copy reads. The working set that fits in the page cache determines performance. Monitor page cache hit rates and disk read throughput. For TLS environments, budget CPU for encryption and accept the loss of zero-copy. For inter-broker replication, zero-copy is not used -- replication is a regular network transfer.
# Monitor page cache effectiveness
# If disk reads are high, the working set exceeds available RAM
iostat -x 1 | grep sda
# Look at: r/s (reads/sec), await (latency), %util
# If r/s is low and throughput is high: page cache is serving reads
# If r/s is high: consumers are reading cold data from disk
Visual Overview
Three Pillars of Kafka Throughput
1. Sequential Append-Only Writes
┌────────────────────────────────────────────┐
│ Active Segment (append-only) │
│ [rec1][rec2][rec3][rec4] → → → [recN] │
│ Disk head moves in one direction only │
└────────────────────────────────────────────┘
2. Zero-Copy Transfer (sendfile)
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Segment │───→│ Page │───→│ Network │
│ File │ │ Cache │ │ Socket │
└──────────┘ └──────────┘ └──────────┘
No copy through JVM heap!
3. OS Page Cache
┌──────────────────────────────────────────┐
│ JVM Heap: 6 GB (small, short GC) │
│ OS Page Cache: 58 GB (hot segments) │
│ Disk: cold data only │
└──────────────────────────────────────────┘
Combined Data Path:
Producer ──append──→ Segment ──→ Page Cache
│
Consumer ──fetch──→ sendfile ────────┘──→ Socket ──→ NIC
Key Takeaway
Kafka sustains very high throughput on commodity disks because of three reinforcing design choices. Sequential append-only writes mean the disk sees large contiguous blocks instead of scattered random operations, which is one to two orders of magnitude faster. Zero-copy transfer using the sendfile system call ships bytes directly from the OS page cache to the network socket without copying through the JVM, letting one broker feed many consumers at near network line rate. And relying on the OS page cache instead of an in-process cache keeps the JVM heap small, GC pauses short, and recent data served from RAM for free.
On the National Bank of India card-authorization stream, this is exactly why cheap servers with ordinary SSDs kept up with millions of events per second. The writes were sequential batched appends, the hot tail was served from the page cache, and zero-copy let one broker fan those events out to the fraud scorer, the ledger updater, and analytics without saturating CPU.
The nuances that matter in production are that zero-copy is bypassed when the broker must transform bytes, such as with TLS encryption or message-format down-conversion for old clients, so keep client and broker formats aligned and budget CPU for encryption. Durability comes from replication and the acks contract rather than fsync per message. And because Kafka leans on the page cache, you size the broker with a small heap and plenty of free RAM rather than a giant heap that starves the cache.
Remember one sentence: Kafka wins by turning messaging into sequential log appends and cached, zero-copy reads, which is the pattern commodity hardware is best at.
---