Definition
- batch.size -- how many bytes accumulate per partition before sending; default 16 KB, increase for throughput.
- linger.ms -- how long to wait to fill a batch; default 0 (send immediately), increase to 5-20 ms for better batching.
- compression.type -- shrinks the batch end to end (none, gzip, snappy, lz4, zstd); lz4 and zstd are fast and effective.
- acks -- durability level; keep
allfor banking and optimize throughput elsewhere. - max.in.flight.requests.per.connection -- how many unacknowledged batches fly at once; up to 5 with idempotence preserves ordering.
- buffer.memory -- total bytes the producer can use for batching; raise if many partitions or large batches.
Imagine National Bank of India sending transaction forms to the central office. Instead of one courier trip per form, the clerk waits a few seconds to collect a stack. A bigger envelope means fewer trips. Compressing the stack makes the envelope lighter. Several couriers on the road at once means more mail per minute. But waiting too long delays each form, so there is a balance.
Understanding the Concepts
What are the five knobs for throughput?
package com.bank.producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import java.util.Properties;
public class HighThroughputConfig {
public static Properties config() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
// Throughput knobs
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 65536); // 64 KB
props.put(ProducerConfig.LINGER_MS_CONFIG, 10); // 10 ms
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4"); // fast codec
props.put(ProducerConfig.ACKS_CONFIG, "all"); // keep safe
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); // ordering safe
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67108864); // 64 MB
return props;
}
public static void main(String[] args) {
Properties p = config();
System.out.println("High-throughput producer config:");
System.out.println(" batch.size = 64 KB");
System.out.println(" linger.ms = 10 ms");
System.out.println(" compression = lz4");
System.out.println(" acks = all");
System.out.println(" max.in.flight = 5");
System.out.println(" buffer.memory = 64 MB");
}
}
High-throughput producer config:
batch.size = 64 KB
linger.ms = 10 ms
compression = lz4
acks = all
max.in.flight = 5
buffer.memory = 64 MB
How does batch.size work?
The batch.size setting (default 16,384 bytes) controls how many bytes accumulate per partition before sending. Larger batches mean fewer network requests and better compression.
package com.bank.throughput;
public class BatchSizeEffect {
public static void main(String[] args) {
int recordSize = 200; // bytes per transfer event
int[] batchSizes = {16384, 32768, 65536};
for (int batchSize : batchSizes) {
int recordsPerBatch = batchSize / recordSize;
System.out.printf(" batch.size=%d KB -> ~%d records/batch%n",
batchSize / 1024, recordsPerBatch);
}
System.out.println(" Fewer, larger batches = fewer network requests");
}
}
batch.size=16 KB -> ~81 records/batch
batch.size=32 KB -> ~163 records/batch
batch.size=64 KB -> ~327 records/batch
Fewer, larger batches = fewer network requests
Why is linger.ms essential?
With linger.ms=0 (default), the producer sends as soon as it can. Under light load, batches ship half-empty. A small linger of 5-20 ms lets many records arrive in that window, filling batches efficiently.
package com.bank.throughput;
public class LingerEffect {
public static void main(String[] args) {
System.out.println("linger.ms=0 (default):");
System.out.println(" Batch sent immediately, often half-empty");
System.out.println(" Many small requests -> low throughput");
System.out.println();
System.out.println("linger.ms=10:");
System.out.println(" Wait up to 10ms to fill the batch");
System.out.println(" Fewer, fuller batches -> high throughput");
System.out.println(" Cost: up to 10ms added latency per record");
}
}
linger.ms=0 (default):
Batch sent immediately, often half-empty
Many small requests -> low throughput
linger.ms=10:
Wait up to 10ms to fill the batch
Fewer, fuller batches -> high throughput
Cost: up to 10ms added latency per record
How does compression reinforce batching?
Kafka compresses the whole batch, not individual records. A larger batch compresses better because more repetitive content is available.
package com.bank.throughput;
public class CompressionBatching {
public static void main(String[] args) {
System.out.println("Batch of 10 JSON transfer events:");
System.out.println(" Uncompressed : ~2000 bytes");
System.out.println(" lz4 compressed: ~600 bytes (70% savings)");
System.out.println();
System.out.println("Batch of 300 JSON transfer events:");
System.out.println(" Uncompressed : ~60000 bytes");
System.out.println(" lz4 compressed: ~8000 bytes (87% savings)");
System.out.println();
System.out.println("Larger batch + compression = multiplicative savings");
}
}
Batch of 10 JSON transfer events:
Uncompressed : ~2000 bytes
lz4 compressed: ~600 bytes (70% savings)
Batch of 300 JSON transfer events:
Uncompressed : ~60000 bytes
lz4 compressed: ~8000 bytes (87% savings)
Larger batch + compression = multiplicative savings
How does max.in.flight.requests.per.connection work?
This setting (default 5) controls how many unacknowledged batches can be in flight. Higher values mean the producer does not wait for each batch to be acknowledged. With idempotence enabled, ordering is preserved up to 5 in-flight requests.
package com.bank.throughput;
public class InFlightEffect {
public static void main(String[] args) {
System.out.println("max.in.flight = 1:");
System.out.println(" Send batch, wait for ack, send next");
System.out.println(" Throughput limited by round-trip time");
System.out.println();
System.out.println("max.in.flight = 5 (with idempotence):");
System.out.println(" 5 batches in flight simultaneously");
System.out.println(" Throughput multiplied");
System.out.println(" Ordering preserved (idempotence reorders on retry)");
}
}
max.in.flight = 1:
Send batch, wait for ack, send next
Throughput limited by round-trip time
max.in.flight = 5 (with idempotence):
5 batches in flight simultaneously
Throughput multiplied
Ordering preserved (idempotence reorders on retry)
How do the knobs interact? (comparison)
| Knob | Default | Throughput value | Trade-off |
|---|---|---|---|
| batch.size | 16 KB | 64 KB+ | More memory per partition |
| linger.ms | 0 | 5-20 ms | Adds latency per record |
| compression.type | none | lz4 or zstd | CPU cost (minimal) |
| acks | all | all (keep it!) | No trade-off -- use batching instead |
| max.in.flight | 5 | 5 | >5 risks reordering without idempotence |
| buffer.memory | 32 MB | 64-128 MB | JVM heap pressure |
What are common mistakes?
package com.bank.mistake;
public class ThroughputMistakes {
public static void main(String[] args) {
// WRONG: big batch.size with linger.ms=0
System.out.println("WRONG: batch.size=64KB but linger.ms=0");
System.out.println(" Batches ship half-empty under light load");
System.out.println();
// WRONG: weakening acks for throughput
System.out.println("WRONG: acks=0 for speed in banking");
System.out.println(" Silent data loss is unacceptable");
System.out.println();
// CORRECT: keep acks=all, optimize batching and compression
System.out.println("CORRECT: acks=all + batch=64KB + linger=10 + lz4");
}
}
WRONG: batch.size=64KB but linger.ms=0
Batches ship half-empty under light load
WRONG: acks=0 for speed in banking
Silent data loss is unacceptable
CORRECT: acks=all + batch=64KB + linger=10 + lz4
What should a senior engineer know?
These settings interact: a big batch.size is useless without linger.ms. Compression amplifies the benefit of larger batches. More partitions allow more parallelism on both sides. You should also raise buffer.memory if the producer accumulates many batches. Topic design (partition count) is part of throughput tuning.
Visual Overview
Throughput Optimization ┌─────────────────────────────────────────────┐ │ Producer │ │ │ │ Records ──▶ Batch buffer (batch.size=64KB) │ │ │ │ │ │ linger.ms=10 (wait to fill) │ │ ▼ │ │ Full batch │ │ │ │ │ │ compression=lz4 (shrink) │ │ ▼ │ │ Compressed batch │ │ │ │ │ │ max.in.flight=5 │ │ │ (5 batches flying at once) │ │ ▼ │ │ ─── Network ───▶ Broker │ │ ▲ │ │ │ acks=all (safe) │ │ │ ack arrives │ └─────────────────────────────────────────────┘
Key Takeaway
To optimize Kafka for high throughput, tune five knobs together. Increase batch.size so more records accumulate per request. Add linger.ms of 5-20 milliseconds so batches actually fill up. Enable compression (lz4 or zstd) to shrink batches, which saves network and disk end to end. Keep acks=all for banking correctness and get throughput from batching instead. Set max.in.flight to 5 with idempotence so multiple batches fly at once without risking reordering.
The knobs interact: a big batch.size does nothing without linger.ms because batches ship half-empty. Compression amplifies larger batches because more repetitive data compresses better. Raise buffer.memory if accumulating many batches.
At National Bank of India, the end-of-day job published millions of transaction events. The naive one-record-at-a-time producer could not keep up, but after setting 64 KB batches, 10 ms linger, lz4 compression, and 5 in-flight requests with idempotence, throughput jumped by an order of magnitude while durability stayed at acks=all.
Remember one sentence: bigger batches plus a little linger plus compression plus multiple in-flight requests maximize throughput, and you never weaken acks in banking.
---