Definition
- No buffering – Never load the entire file into heap memory; a hundred concurrent 100 MB uploads is 10 GB of heap, which is instant OutOfMemoryError.
- Presigned URLs – Object storage issues a short-lived, signed URL so the client uploads directly to S3; your API handles only tiny JSON.
- Multipart upload – For multi-gigabyte files, S3 multipart splits the file into independently uploadable and retryable parts.
- Async processing – Virus scan, MIME validation, and thumbnailing happen off the upload path via a message queue.
- Reactive stack – Spring WebFlux with
Fluxstreams bytes without holding the whole file in memory, with backpressure.
Customers of National Bank of India need to upload large scanned loan files. Imagine the branch clerk personally carrying every customer's heavy suitcase into the vault. With one clerk and a thousand customers, the clerk collapses. Instead, the bank gives each customer a one-time pass (presigned URL) to drive their own suitcase straight to the warehouse (S3). The clerk only writes down "Rajesh dropped off suitcase 7 at 3pm" on a slip and puts it in an in-tray for later filing.
Understanding the Concepts
Why must you never buffer a large file in memory?
A naive Spring MVC controller that takes a MultipartFile and calls getBytes() loads the entire file into the heap. A hundred concurrent 100 MB uploads is 10 GB of heap, which is an instant OutOfMemoryError.
package com.bank.upload.antipattern;
import org.springframework.web.multipart.MultipartFile;
public class BufferingAntipattern {
// WRONG: loads entire file into heap
public void handleUpload(MultipartFile file) throws Exception {
byte[] data = file.getBytes(); // 100 MB in heap PER REQUEST
// 100 concurrent uploads = 10 GB heap -> OutOfMemoryError
}
}
How do presigned URLs keep bytes off your server?
Your API validates the request and returns a short-lived signed URL. The client uploads directly to S3. Your server never touches the file content.
package com.bank.upload.service;
import java.time.Duration;
import java.util.UUID;
public class PresignedUrlService {
private final S3Presigner presigner;
public PresignedUrlService(S3Presigner presigner) {
this.presigner = presigner;
}
public UploadInitResponse initUpload(String userId, String filename,
long sizeBytes, String contentType) {
if (sizeBytes > 500_000_000) { // 500 MB max
throw new IllegalArgumentException("File too large: " + sizeBytes);
}
String key = "uploads/" + userId + "/" + UUID.randomUUID() + "/" + filename;
// Presign a PUT URL valid for 5 minutes
String presignedUrl = presigner.presignPutObject(key,
Duration.ofMinutes(5), contentType);
return new UploadInitResponse(key, presignedUrl);
}
}
# Client calls: POST /api/v1/uploads {filename: "loan-docs.pdf", size: 150000000}
# Server returns: {key: "uploads/RAJESH/abc123/loan-docs.pdf",
# presignedUrl: "https://s3.amazonaws.com/...?X-Amz-Signature=..."}
# Client uploads 150 MB directly to S3 using the presigned URL.
# Server never touches the file content.
How does S3 multipart upload work for very large files?
The client splits the file into parts (10 MB each), gets a presigned URL per part, uploads parts in parallel, and calls complete. A failed part retries without re-uploading the whole file.
package com.bank.upload.service;
import java.util.List;
public class MultipartUploadService {
public String initiateMultipart(String key) {
// Returns an uploadId from S3
return "upload-id-abc123";
}
public String presignPart(String key, String uploadId, int partNumber) {
// Returns a presigned URL for this specific part
return "https://s3.amazonaws.com/...?partNumber=" + partNumber;
}
public void completeMultipart(String key, String uploadId,
List<PartETag> parts) {
// S3 assembles all parts into the final object
System.out.println("Multipart upload complete: " + key
+ " with " + parts.size() + " parts");
}
}
How do you process metadata asynchronously?
When the upload finishes, drop an event on a queue. Background workers do virus scanning, MIME validation, and thumbnailing.
package com.bank.upload.event;
public class UploadCompletedEvent {
private String userId;
private String storageKey;
private String declaredContentType;
private long sizeBytes;
private long timestamp;
// Published to Kafka topic "upload-events"
// Consumers:
// 1. VirusScanWorker -> scans with ClamAV
// 2. MimeValidator -> checks magic bytes
// 3. ThumbnailGenerator -> creates preview
}
How does reactive WebFlux stream bytes with backpressure?
When you must stream through your service, WebFlux's Flux pipes the body in chunks without loading it all into memory. Backpressure signals "slow down" if storage cannot keep up.
package com.bank.upload.controller;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.http.codec.multipart.FilePart;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
public class ReactiveUploadController {
// WebFlux: body arrives as Flux<DataBuffer>, never fully in memory
public Mono<String> uploadReactive(FilePart filePart) {
Flux<DataBuffer> content = filePart.content();
// Pipe directly to S3 reactive client
// Only one chunk in memory at a time
// Backpressure: if S3 is slow, upstream pauses automatically
return content
.map(buffer -> {
// Process chunk (e.g., write to S3)
int size = buffer.readableByteCount();
return size;
})
.reduce(Integer::sum)
.map(totalBytes -> "Uploaded " + totalBytes + " bytes");
}
}
How do you validate MIME type from magic bytes?
Never trust the file extension or declared content type. Read the leading bytes to identify the real format.
package com.bank.upload.validation;
import java.io.InputStream;
import java.util.Set;
public class MimeValidator {
private static final Set<String> ALLOWED_TYPES = Set.of(
"application/pdf", "image/jpeg", "image/png");
public boolean validateMagicBytes(InputStream input) throws Exception {
byte[] header = new byte[8];
input.read(header);
// PDF starts with %PDF (0x25 0x50 0x44 0x46)
if (header[0] == 0x25 && header[1] == 0x50
&& header[2] == 0x44 && header[3] == 0x46) {
return true; // genuine PDF
}
// JPEG starts with FF D8 FF
if (header[0] == (byte) 0xFF && header[1] == (byte) 0xD8
&& header[2] == (byte) 0xFF) {
return true; // genuine JPEG
}
return false; // unknown or disallowed type
}
}
How do you apply rate limiting?
A per-user token bucket prevents one client from flooding storage bandwidth or cost.
package com.bank.upload.ratelimit;
public class UploadRateLimiter {
// Max 5 uploads per user per minute
public boolean allowUpload(String userId) {
int recentUploads = getRecentUploadCount(userId);
if (recentUploads >= 5) {
System.out.println("Rate limited: " + userId);
return false;
}
return true;
}
private int getRecentUploadCount(String userId) {
return 0; // Redis INCR with 60s TTL
}
}
What are common interview follow-up questions?
How do you handle a presigned URL that was leaked? How do you enforce content-length on a presigned URL? What happens if the client abandons a multipart upload? How do you handle very slow mobile connections? How do you quarantine infected files?
Visual Overview
┌──────────────────────────────────────────────────────────┐
│ LARGE FILE UPLOAD ARCHITECTURE │
├──────────────────────────────────────────────────────────┤
│ │
│ Client │
│ │ POST /api/v1/uploads {filename, size} │
│ ▼ │
│ ┌─────────────────┐ │
│ │ API Server │ (only handles tiny JSON) │
│ │ - validate size │ │
│ │ - rate limit │ │
│ │ - mint presigned │ │
│ │ URL │ │
│ └────────┬────────┘ │
│ │ returns presigned URL │
│ ▼ │
│ Client ──PUT bytes──▶ S3 (direct, server not involved) │
│ │ │
│ ┌──────▼───────┐ │
│ │ S3 Event / │ │
│ │ Kafka Event │ │
│ └──────┬───────┘ │
│ │ │
│ ┌──────────▼──────────┐ │
│ │ Async Workers │ │
│ │ - Virus scan │ │
│ │ - MIME magic bytes │ │
│ │ - Thumbnail │ │
│ └─────────────────────┘ │
└──────────────────────────────────────────────────────────┘
Key Takeaway
Designing a microservice for large file uploads under high concurrency starts with one principle: big bytes should never flow through the application server. At National Bank of India, customers like Rajesh upload 150 MB scanned property files during loan campaigns, thousands at once. A naive controller calling getBytes() caused OutOfMemoryError the moment a few hundred files arrived concurrently.
Pankaj's architecture uses presigned URLs: the client calls the API with just the filename and size, the API validates and mints a short-lived signed URL scoped to one S3 key, and the client uploads directly to S3. The server never touches the content. For multi-gigabyte KYC videos, S3 multipart upload splits the file into independently retryable parts. All heavy processing (virus scan, MIME magic-byte validation, thumbnail generation) happens asynchronously through a Kafka queue, decoupling the fast upload path from slow processing.
When bytes must flow through the service, reactive WebFlux with Flux streams chunks without holding the whole file in memory, and backpressure keeps memory bounded. A per-user token-bucket rate limit prevents one client from flooding storage bandwidth or cost. MIME type is validated from magic bytes, never from the extension, and untrusted content sits in a quarantine bucket until ClamAV clears it.
Remember one sentence: Keep the bytes off your server by minting presigned URLs, use multipart for very large files, handle metadata on a non-blocking reactive stack with backpressure, push processing onto an async queue, and validate magic bytes rather than trusting extensions.
---