Definition
- message.max.bytes -- broker-level cluster default for the maximum record batch size; rejects batches larger than this.
- max.message.bytes -- per-topic override of the broker limit.
- max.request.size -- producer-side cap; records exceeding it throw
RecordTooLargeExceptionbefore anything is sent. - max.partition.fetch.bytes -- consumer-side cap per partition; must be at least as large as the biggest record.
- fetch.max.bytes -- overall consumer fetch cap across all partitions.
- Claim-check pattern -- store the large blob in object storage (S3) and publish only a small reference on Kafka.
Imagine National Bank of India has a rule that any single envelope sent between departments can be at most one kilogram. There are three checkpoints: the mailroom (broker), the sender's desk (producer), and the recipient's inbox tray (consumer). If you want to send a two-kilogram envelope, all three checkpoints must accept the heavier weight. Change only one and the envelope gets stuck somewhere.
Understanding the Concepts
What is the default message size?
Kafka's default maximum message size is approximately 1,048,576 bytes (1 MB). This limit exists at multiple layers and all of them matter.
package com.bank.config;
public class DefaultMessageSize {
public static void main(String[] args) {
System.out.println("Default message size limits:");
System.out.println(" Broker message.max.bytes : 1048588 bytes (~1 MB)");
System.out.println(" Topic max.message.bytes : inherits broker default");
System.out.println(" Producer max.request.size : 1048576 bytes (1 MB)");
System.out.println(" Consumer max.partition.fetch.bytes : 1048576 bytes (1 MB)");
System.out.println(" Consumer fetch.max.bytes : 52428800 bytes (50 MB)");
}
}
Default message size limits:
Broker message.max.bytes : 1048588 bytes (~1 MB)
Topic max.message.bytes : inherits broker default
Producer max.request.size : 1048576 bytes (1 MB)
Consumer max.partition.fetch.bytes : 1048576 bytes (1 MB)
Consumer fetch.max.bytes : 52428800 bytes (50 MB)
Why does the default exist?
Kafka is designed for high-throughput streams of many small records. Large messages increase memory pressure, inflate replication traffic, cause latency spikes, and can stall a partition while processed.
What happens when the limit is exceeded?
package com.bank.error;
public class RecordTooLargeDemo {
public static void main(String[] args) {
System.out.println("Exceeding the limit:");
System.out.println(" Producer (max.request.size exceeded):");
System.out.println(" -> RecordTooLargeException at producer");
System.out.println(" -> Nothing sent to broker");
System.out.println();
System.out.println(" Broker (message.max.bytes exceeded):");
System.out.println(" -> Broker rejects produce request");
System.out.println(" -> Producer sees error");
System.out.println();
System.out.println(" Consumer (max.partition.fetch.bytes too small):");
System.out.println(" -> Record stored but consumer CANNOT read it");
System.out.println(" -> Consumer stalls on that partition");
}
}
Exceeding the limit:
Producer (max.request.size exceeded):
-> RecordTooLargeException at producer
-> Nothing sent to broker
Broker (message.max.bytes exceeded):
-> Broker rejects produce request
-> Producer sees error
Consumer (max.partition.fetch.bytes too small):
-> Record stored but consumer CANNOT read it
-> Consumer stalls on that partition
How do you raise the limits?
All three layers must be raised together.
package com.bank.config;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;
public class LargeMessageConfig {
public static Properties producerConfig() {
Properties props = new Properties();
props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 5242880); // 5 MB
return props;
}
public static Properties consumerConfig() {
Properties props = new Properties();
props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 5242880); // 5 MB
props.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, 52428800); // 50 MB
return props;
}
public static void main(String[] args) {
System.out.println("To support 5 MB messages, raise ALL layers:");
System.out.println(" 1. kafka-configs --alter --topic statements");
System.out.println(" --add-config max.message.bytes=5242880");
System.out.println(" 2. producer max.request.size = 5242880");
System.out.println(" 3. consumer max.partition.fetch.bytes = 5242880");
System.out.println(" 4. broker replica.fetch.max.bytes = 5242880");
}
}
To support 5 MB messages, raise ALL layers:
1. kafka-configs --alter --topic statements
--add-config max.message.bytes=5242880
2. producer max.request.size = 5242880
3. consumer max.partition.fetch.bytes = 5242880
4. broker replica.fetch.max.bytes = 5242880
How does compression help?
package com.bank.config;
public class CompressionForSize {
public static void main(String[] args) {
System.out.println("Compression reduces message size before hitting limits:");
System.out.println(" Original JSON statement : 3.2 MB");
System.out.println(" With zstd compression : 0.6 MB (under 1 MB default!)");
System.out.println();
System.out.println(" props.put(\"compression.type\", \"zstd\")");
System.out.println(" Broker stores and ships compressed batch");
System.out.println(" Consumer decompresses");
}
}
Compression reduces message size before hitting limits:
Original JSON statement : 3.2 MB
With zstd compression : 0.6 MB (under 1 MB default!)
props.put("compression.type", "zstd")
Broker stores and ships compressed batch
Consumer decompresses
What is the claim-check pattern?
For genuinely large payloads like PDFs or images, store the blob in object storage and publish only a small reference on Kafka.
package com.bank.claimcheck;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;
public class ClaimCheckProducer {
public static void main(String[] args) {
// Step 1: Upload PDF to S3
String s3Key = "statements/2024/SAV2001/june.pdf";
System.out.println("Uploaded PDF to S3: " + s3Key);
// Step 2: Publish small reference on Kafka
String reference = "{\"accountId\":\"SAV2001\"," +
"\"s3Key\":\"" + s3Key + "\"," +
"\"sizeBytes\":3200000," +
"\"checksum\":\"abc123\"}";
System.out.println("Kafka message: " + reference);
System.out.println("Size: " + reference.getBytes().length + " bytes (tiny!)");
}
}
Uploaded PDF to S3: statements/2024/SAV2001/june.pdf
Kafka message: {"accountId":"SAV2001","s3Key":"statements/2024/SAV2001/june.pdf","sizeBytes":3200000,"checksum":"abc123"}
Size: 117 bytes (tiny!)
What are common mistakes?
package com.bank.mistake;
public class MessageSizeMistakes {
public static void main(String[] args) {
System.out.println("WRONG: raise only producer max.request.size");
System.out.println(" Broker still rejects -> RecordTooLargeException");
System.out.println();
System.out.println("WRONG: raise producer + broker, forget consumer");
System.out.println(" Record stored but consumer CANNOT read it");
System.out.println();
System.out.println("CORRECT: raise all three layers together");
System.out.println(" Or better: use claim-check pattern");
}
}
WRONG: raise only producer max.request.size
Broker still rejects -> RecordTooLargeException
WRONG: raise producer + broker, forget consumer
Record stored but consumer CANNOT read it
CORRECT: raise all three layers together
Or better: use claim-check pattern
When should you raise limits vs use claim-check?
| Approach | When to use | Trade-off |
|---|---|---|
| Raise limits | Occasional slightly-over-1MB messages | Changes cluster memory/latency profile |
| Compression | Large but compressible data (JSON) | CPU cost, very effective for text |
| Claim-check | Genuinely large blobs (PDFs, images) | Extra S3 round trip, lifecycle management |
What should a senior engineer know?
Raising limits is a last resort. The 1 MB default is a deliberate guardrail. Compression is the right first lever for compressible data. The claim-check pattern is the professional solution for large blobs. Any limit change must be load-tested across the cluster.
Visual Overview
Three-Layer Size Enforcement
Producer Broker Consumer
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ max.request │ │ message.max │ │ max.partition │
│ .size = 1 MB │─────▶│ .bytes = 1 MB │─────▶│ .fetch.bytes │
│ │ │ (per-topic: │ │ = 1 MB │
│ Too big? │ │ max.message │ │ │
│ -> REJECT │ │ .bytes) │ │ Too small? │
│ RecordTooLarge│ │ Too big? │ │ -> STALL │
└───────────────┘ │ -> REJECT │ └───────────────┘
└───────────────┘
Claim-Check Pattern:
┌──────────┐ ┌─────────┐ ┌───────────────┐
│ Producer │────▶│ S3 │ │ Kafka topic │
│ (PDF) │ │ (blob) │ │ (reference) │
└──────────┘ └─────────┘ └───────┬───────┘
▲ │
│ ▼
┌────┴──────────────────────┐
│ Consumer reads reference, │
│ downloads blob from S3 │
└───────────────────────────┘
Key Takeaway
Kafka's default maximum message size is about 1 MB, and this limit is enforced at three layers: the producer's max.request.size, the broker's message.max.bytes (or per-topic max.message.bytes), and the consumer's max.partition.fetch.bytes. All three must be raised together for larger messages; a mismatch at any layer stalls or rejects the message.
Compression is the first lever for large but compressible data like JSON statements -- zstd can shrink a 3 MB JSON statement below 1 MB without changing any size limits. For genuinely large blobs like PDFs, the claim-check pattern is the professional solution: store the blob in S3 and publish only a small reference on Kafka.
At National Bank of India, Priya's monthly statement PDF reached several megabytes. Rather than raising limits cluster-wide, Rajesh adopted the claim-check pattern: the PDF goes to S3 and Kafka carries only a reference. This keeps the cluster healthy while the email service downloads the full statement.
Remember one sentence: the 1 MB default is a deliberate guardrail, and the better engineering answer for large payloads is compression for compressible data and claim-check for blobs, not raising limits.
---