Definition
- RecordTooLargeException -- thrown by the producer when a record exceeds
max.request.size; the record is never sent to the broker. - max.request.size -- producer-side cap (default 1 MB) on the size of a produce request.
- message.max.bytes / max.message.bytes -- broker-level and per-topic caps on the record batch the broker accepts.
- max.partition.fetch.bytes -- consumer-side cap per partition; must be large enough to fetch the biggest record.
- Compression -- shrinks the batch before it hits size limits; zstd and lz4 are fast and effective for text-heavy banking data.
- Claim-check pattern -- store the large payload in external storage and send only a reference through Kafka.
Imagine National Bank of India's mailroom only accepts envelopes up to a certain thickness. If a clerk tries to post a two-hundred-page loan agreement in one envelope, the mailroom refuses it at the counter. To send something big, you either raise the thickness limit at every checkpoint, compress the pages, or put the big document in the vault and mail only a claim ticket.
Understanding the Concepts
What happens when the size is exceeded?
The failure depends on which layer catches it first.
package com.bank.error;
public class SizeExceeded {
public static void main(String[] args) {
System.out.println("Layer 1 -- Producer (max.request.size):");
System.out.println(" Record > 1 MB -> RecordTooLargeException");
System.out.println(" Fails LOCALLY, nothing sent to broker");
System.out.println();
System.out.println("Layer 2 -- Broker (message.max.bytes):");
System.out.println(" Batch > broker limit -> rejected");
System.out.println(" Producer sees error response");
System.out.println();
System.out.println("Layer 3 -- Consumer (max.partition.fetch.bytes):");
System.out.println(" Record stored but fetch size too small");
System.out.println(" Consumer STALLS on that partition");
}
}
Layer 1 -- Producer (max.request.size):
Record > 1 MB -> RecordTooLargeException
Fails LOCALLY, nothing sent to broker
Layer 2 -- Broker (message.max.bytes):
Batch > broker limit -> rejected
Producer sees error response
Layer 3 -- Consumer (max.partition.fetch.bytes):
Record stored but fetch size too small
Consumer STALLS on that partition
How do you raise the limits?
All layers must be raised together, plus the replica fetcher.
package com.bank.config;
public class RaiseLimits {
public static void main(String[] args) {
int targetSize = 5 * 1024 * 1024; // 5 MB
System.out.println("Target: " + targetSize + " bytes (5 MB)");
System.out.println();
System.out.println("Producer:");
System.out.println(" max.request.size = " + targetSize);
System.out.println();
System.out.println("Broker/Topic:");
System.out.println(" message.max.bytes = " + targetSize);
System.out.println(" max.message.bytes = " + targetSize + " (per-topic)");
System.out.println(" replica.fetch.max.bytes = " + targetSize);
System.out.println();
System.out.println("Consumer:");
System.out.println(" max.partition.fetch.bytes = " + targetSize);
System.out.println(" fetch.max.bytes = " + (50 * 1024 * 1024));
}
}
Target: 5242880 bytes (5 MB)
Producer:
max.request.size = 5242880
Broker/Topic:
message.max.bytes = 5242880
max.message.bytes = 5242880 (per-topic)
replica.fetch.max.bytes = 5242880
Consumer:
max.partition.fetch.bytes = 5242880
fetch.max.bytes = 52428800
How does compression help?
package com.bank.compression;
public class CompressionBenefit {
public static void main(String[] args) {
System.out.println("JSON statement for Priya's account:");
System.out.println(" Uncompressed : 3.2 MB (exceeds 1 MB limit)");
System.out.println(" With zstd : 0.6 MB (under limit!)");
System.out.println();
System.out.println("props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, \"zstd\")");
System.out.println("No limit changes needed for this statement");
}
}
JSON statement for Priya's account:
Uncompressed : 3.2 MB (exceeds 1 MB limit)
With zstd : 0.6 MB (under limit!)
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd")
No limit changes needed for this statement
How does the claim-check pattern work?
package com.bank.claimcheck;
public class ClaimCheckFlow {
public static void main(String[] args) {
System.out.println("Step 1: Upload PDF to S3");
System.out.println(" s3://bank-statements/SAV2001/june.pdf (4.5 MB)");
System.out.println();
System.out.println("Step 2: Publish reference on Kafka");
System.out.println(" {accountId: SAV2001, s3Key: .../june.pdf} (~100 bytes)");
System.out.println();
System.out.println("Step 3: Consumer reads reference, downloads from S3");
System.out.println(" Kafka stays fast, no limit changes needed");
}
}
Step 1: Upload PDF to S3
s3://bank-statements/SAV2001/june.pdf (4.5 MB)
Step 2: Publish reference on Kafka
{accountId: SAV2001, s3Key: .../june.pdf} (~100 bytes)
Step 3: Consumer reads reference, downloads from S3
Kafka stays fast, no limit changes needed
What are the three options compared?
| Option | When | Pro | Con |
|---|---|---|---|
| Raise all limits | Occasional slightly-over-1MB | Simple config change | Memory/latency impact on cluster |
| Compression | Large compressible data | No config change if under limit | CPU cost (minimal for lz4/zstd) |
| Claim-check | Genuinely large blobs | Kafka stays fast | Extra storage + round trip |
What are common mistakes?
package com.bank.mistake;
public class SizeMistakes {
public static void main(String[] args) {
System.out.println("WRONG: raise only producer limit");
System.out.println(" Broker still rejects");
System.out.println();
System.out.println("WRONG: raise producer + broker, forget consumer");
System.out.println(" Record stored but consumer stalls");
System.out.println();
System.out.println("WRONG: raise producer + broker + consumer, forget replica fetcher");
System.out.println(" Followers cannot replicate large batches");
System.out.println();
System.out.println("CORRECT: raise all four: producer, topic, consumer, replica fetcher");
}
}
WRONG: raise only producer limit
Broker still rejects
WRONG: raise producer + broker, forget consumer
Record stored but consumer stalls
WRONG: raise producer + broker + consumer, forget replica fetcher
Followers cannot replicate large batches
CORRECT: raise all four: producer, topic, consumer, replica fetcher
How do you detect and monitor oversized messages?
Detecting oversized messages before they cause exceptions is better than reacting after the fact. Use producer interceptors to measure record sizes, and alert when records approach the limit. On the broker side, monitor the kafka.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec metric.
package com.bank.kafka;
public class OversizedMessageMonitoring {
public static void main(String[] args) {
System.out.println("Monitoring oversized messages:");
System.out.println(" 1. Producer interceptor: log warning if size > 80% of limit");
System.out.println(" 2. Broker metric: FailedProduceRequestsPerSec");
System.out.println(" Spike = producers sending oversized records");
System.out.println(" 3. Application logging: catch RecordTooLargeException");
System.out.println(" Log the key, topic, and actual size for debugging");
System.out.println(" 4. Pre-send validation:");
System.out.println(" if (serializedSize > MAX_RECORD_SIZE * 0.8)");
System.out.println(" use claim-check pattern instead");
}
}
Expected output:
Monitoring oversized messages:
1. Producer interceptor: log warning if size > 80% of limit
2. Broker metric: FailedProduceRequestsPerSec
Spike = producers sending oversized records
3. Application logging: catch RecordTooLargeException
Log the key, topic, and actual size for debugging
4. Pre-send validation:
if (serializedSize > MAX_RECORD_SIZE * 0.8)
use claim-check pattern instead
What should a senior engineer know?
Raising limits is a last resort because Kafka is optimized for many small records. Compression is the right first lever. The claim-check pattern keeps Kafka messages small. Any limit change must be rolled out across all layers and load-tested.
Visual Overview
RecordTooLargeException Flow Producer ┌──────────────────────┐ │ Record = 2 MB │ │ max.request.size=1MB │ │ │ │ 2 MB > 1 MB │ │ -> RecordTooLarge │ │ Exception! │ │ Nothing sent. │ └──────────────────────┘ Solutions: ┌──────────────────────────────────────┐ │ Option 1: Raise all limits (5 MB) │ │ Option 2: Compress (zstd: 2MB->0.5)│ │ Option 3: Claim-check (S3 + ref) │ └──────────────────────────────────────┘
Key Takeaway
When a record exceeds the default 1 MB size, the producer throws RecordTooLargeException before anything is sent. The broker independently rejects oversized batches, and the consumer stalls if its fetch size is too small. These limits exist at three layers and all must be raised together for larger messages.
Compression is the first lever: zstd can shrink a multi-megabyte JSON statement well under 1 MB with no limit changes. For genuinely large blobs like PDFs, the claim-check pattern keeps Kafka fast by storing the blob in S3 and publishing only a small reference.
At National Bank of India, Priya's statement PDF exceeded 1 MB. Rather than raising limits cluster-wide, Rajesh enabled zstd compression for JSON statements and used claim-check for PDFs. The cluster stays healthy with no size limit changes.
Remember one sentence: RecordTooLargeException means the record exceeds a default that exists at three layers, and the engineering answer is compression for text and claim-check for blobs, not raising limits.
---