Definition
- Why not stream – Each upload consumes heap (if buffered), a thread (for the entire transfer duration), and disk (if spooled), which does not scale.
- Presigned URL – A time-limited, signed URL that grants a client permission for one specific operation on one specific S3 key.
- Multipart upload – Splits large files into independently uploaded parts that S3 reassembles, giving parallelism and resumability.
- Magic bytes – The file extension is attacker-controlled; the real format is identified by the first few bytes (e.g.,
%PDF,FF D8 FF). - Virus scan – Uploaded content lands in quarantine; an async scanner promotes clean files and rejects infected ones.
At National Bank of India, imagine every customer's heavy document box had to pass through the manager's office. The desk overflows, the floor fills up, and the manager is stuck holding each box for the whole transfer. Instead the manager writes a one-time gate pass (presigned URL) that lets the customer drive straight to the warehouse. The guard never trusts the label on the box; he opens it and checks what is actually inside.
Understanding the Concepts
Why should bytes not flow through the application server?
Three resources get consumed per upload: heap memory if you buffer, a request thread for the entire upload duration, and local disk if you spool to temp files. A single slow 2 GB upload on a mobile connection can hold a thread for ten minutes.
package com.bank.upload.antipattern;
// WRONG: three scarce resources consumed per upload
public class StreamThroughServer {
// Resource 1: Heap memory (getBytes loads entire file)
// Resource 2: Request thread (held for entire transfer)
// Resource 3: Local disk (temp file spooling)
// A few hundred concurrent large uploads:
// - Heap: OutOfMemoryError
// - Threads: thread pool exhausted
// - Disk: /tmp fills up
// The server is doing zero useful computation, just copying bytes.
}
How does a presigned URL work?
S3 generates a URL embedding a signature derived from your secret credentials, the specific bucket and key, the allowed method, and a short expiry.
package com.bank.upload.service;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import java.time.Duration;
public class S3PresignService {
private final S3Presigner presigner;
public S3PresignService(S3Presigner presigner) {
this.presigner = presigner;
}
public String mintUploadUrl(String bucket, String key, String contentType) {
PutObjectRequest putRequest = PutObjectRequest.builder()
.bucket(bucket)
.key(key)
.contentType(contentType)
.build();
PresignedPutObjectRequest presigned = presigner.presignPutObject(req ->
req.signatureDuration(Duration.ofMinutes(5))
.putObjectRequest(putRequest));
return presigned.url().toString();
// Client uses this URL to PUT bytes directly to S3
// URL expires in 5 minutes
}
}
# Flow:
# 1. Client -> API: POST /uploads {filename: "kyc-video.mp4", size: 3000000000}
# 2. API validates, mints presigned URL (5 min expiry)
# 3. Client -> S3: PUT bytes using presigned URL
# 4. API never touches the file content
How does S3 multipart upload handle multi-GB files?
A single PUT is impractical for very large files because a network blip restarts the whole transfer. Multipart splits into parts (10 MB each), uploads in parallel, and a failed part retries alone.
package com.bank.upload.service;
import java.util.ArrayList;
import java.util.List;
public class MultipartUploadFlow {
public void uploadLargeFile(String bucket, String key, long totalSize) {
// Step 1: Initiate multipart upload
String uploadId = initiateMultipart(bucket, key);
// Step 2: Upload each part (10 MB chunks)
long partSize = 10_000_000; // 10 MB
int partCount = (int) Math.ceil((double) totalSize / partSize);
List<PartETag> completedParts = new ArrayList<>();
for (int i = 1; i <= partCount; i++) {
// Get presigned URL for this part
String partUrl = presignPart(bucket, key, uploadId, i);
// Client uploads this chunk using the URL
// On success, record ETag
completedParts.add(new PartETag(i, "etag-" + i));
// On failure: retry just this part, not the whole file
}
// Step 3: Complete multipart
completeMultipart(bucket, key, uploadId, completedParts);
}
private String initiateMultipart(String b, String k) { return "upload-id"; }
private String presignPart(String b, String k, String u, int p) { return ""; }
private void completeMultipart(String b, String k, String u, List<PartETag> p) { }
}
Why must you validate magic bytes, not the extension?
The file extension and declared Content-Type are attacker-controlled strings. An attacker renames malware.exe to invoice.pdf.
package com.bank.upload.validation;
public class MagicBytesValidator {
// WRONG: trusting the extension
public boolean validateByExtension(String filename) {
return filename.endsWith(".pdf"); // attacker renames .exe to .pdf
}
// CORRECT: checking the actual file header
public boolean validateByMagicBytes(byte[] header) {
// PDF: starts with %PDF (25 50 44 46)
if (header[0] == 0x25 && header[1] == 0x50
&& header[2] == 0x44 && header[3] == 0x46) {
return true;
}
// JPEG: starts with FF D8 FF
if (header[0] == (byte) 0xFF && header[1] == (byte) 0xD8
&& header[2] == (byte) 0xFF) {
return true;
}
// PNG: starts with 89 50 4E 47
if (header[0] == (byte) 0x89 && header[1] == 0x50
&& header[2] == 0x4E && header[3] == 0x47) {
return true;
}
return false; // reject unknown format
}
}
How do you enforce size limits?
Enforce a hard maximum before issuing the presigned URL. For multipart, cap total size.
package com.bank.upload.validation;
public class SizeLimitEnforcer {
private static final long MAX_FILE_SIZE = 500_000_000L; // 500 MB
public void validateSize(long declaredSize) {
if (declaredSize > MAX_FILE_SIZE) {
throw new IllegalArgumentException(
"File too large: " + declaredSize + " bytes. Max: " + MAX_FILE_SIZE);
}
}
// Also set content-length-range on presigned URL conditions
// so S3 rejects uploads exceeding the declared size
}
How does virus scanning work?
Uploaded content lands in a quarantine bucket. An async scanner checks it. Clean files are promoted; infected files are deleted.
package com.bank.upload.security;
public class VirusScanWorker {
public void scanFile(String quarantineKey) {
boolean clean = scanWithClamAV(quarantineKey);
if (clean) {
moveToUsableBucket(quarantineKey);
System.out.println("File clean: " + quarantineKey);
} else {
deleteInfectedFile(quarantineKey);
notifyUser("Your file was rejected due to a security concern.");
}
}
private boolean scanWithClamAV(String key) { return true; }
private void moveToUsableBucket(String key) { }
private void deleteInfectedFile(String key) { }
private void notifyUser(String msg) { }
}
What are the security rules for presigned URLs?
Short-lived (5 minutes), scoped to one key and one method, with a content-length-range condition.
package com.bank.upload.security;
public class PresignedUrlSecurity {
// Rules:
// 1. Expiry: 5 minutes (a leaked URL expires quickly)
// 2. Scoped to one key: /uploads/RAJESH/abc123/file.pdf
// 3. Scoped to one method: PUT only (cannot GET other files)
// 4. Content-length-range: [0, 500MB] (client cannot upload 50 GB)
// 5. Content-type condition: application/pdf (client cannot upload .exe)
}
What are common interview follow-up questions?
How do you handle the case where a client starts a multipart upload and never completes it? How do you implement download presigned URLs? What is the difference between a presigned URL and a presigned POST? How do you handle files that are too large for a single Lambda function to scan?
Visual Overview
┌───────────────────────────────────────────────────────────┐
│ PRESIGNED URL + MULTIPART UPLOAD FLOW │
├───────────────────────────────────────────────────────────┤
│ │
│ Client API Server │
│ │ │ │
│ │ POST /uploads {name,size} │ │
│ │──────────────────────────────▶ │ │
│ │ │ validate size │
│ │ │ mint presigned URL │
│ │ ◀─── presigned URL ───────────│ │
│ │ │
│ │ PUT bytes ──────────────────────────▶ S3 │
│ │ (direct, server not involved) │ │
│ │ │ │
│ │ For multi-GB: │ │
│ │ ┌─Part 1─▶ S3 (parallel) │ │
│ │ ├─Part 2─▶ S3 (parallel) │ │
│ │ └─Part 3─▶ S3 (parallel) │ │
│ │ └─ Complete ─▶ S3 assembles │ │
│ │ │ │
│ │ ┌───────▼────────┐ │
│ │ │ Quarantine │ │
│ │ │ ClamAV scan │ │
│ │ │ Magic bytes │ │
│ │ └───────┬────────┘ │
│ │ clean │ infected │
│ │ ┌───────▼┐ ┌───────┐ │
│ │ │Usable │ │Delete │ │
│ │ │bucket │ │+ alert│ │
│ │ └────────┘ └───────┘ │
└───────────────────────────────────────────────────────────┘
Key Takeaway
File bytes should never flow through the application server because each upload consumes heap, a thread, and potentially disk, all for zero useful computation. At National Bank of India, Priya's 3 GB KYC video previously failed near the end and restarted from zero because the server was streaming it through and the connection dropped. Pankaj moved to presigned URLs: the API mints a short-lived, scoped URL and steps aside, so the client uploads directly to S3.
For multi-gigabyte files, S3 multipart upload splits into independently retryable parts that upload in parallel. A failed part retries alone, giving resumability. This fixed Priya's video problem immediately. On validation, the team never trusts the file extension or declared content type because both are attacker-controlled. Magic bytes (the first few bytes of the file) identify the real format, and a hard maximum size is enforced before issuing any URL.
Virus scanning happens asynchronously in a quarantine bucket. An uploaded file is not marked usable until ClamAV clears it, and infected files are deleted with the user notified. The presigned URL itself is scoped to one key, one method, a 5-minute expiry, and a content-length-range condition, so a leaked URL expires quickly and cannot be abused.
Remember one sentence: Mint a short-lived presigned URL so bytes go directly to S3, use multipart for large files, validate magic bytes instead of extensions, enforce size limits before issuing the URL, and virus-scan in quarantine before marking usable.
---