Design a File Storage System (like Google Drive/S3)

• Chunked upload : splitting a file into fixed-size pieces for resumable, parallel transfers.
• Content-hash deduplication : identifying chunks by their SHA-256 hash so identical chunks are stored only once.
• Metadata database : stores file structure, ownership, permissions, and version history, separate from the blob store.
• Versioning : stores a new metadata record per change while old chunks remain referenced.
• Pre-signed URL : a time-limited URL that lets clients upload/download directly from the blob store without going through application servers.

How do chunked uploads work?

Large files are split into fixed-size chunks (e.g., 4 MB), each identified by its content hash. Uploads are resumable: if the network drops, only missing chunks are resent. At National Bank of India, uploading a 200 MB property valuation report over a slow branch connection works because each 4 MB chunk is independently uploaded and verified.

package com.bank.filestorage;

import java.security.MessageDigest;
import java.util.*;

public class ChunkedUploader {
    public List<Chunk> chunk(byte[] file, int chunkSize) {
        List<Chunk> chunks = new ArrayList<>();
        for (int off = 0; off < file.length; off += chunkSize) {
            byte[] data = Arrays.copyOfRange(file, off,
                Math.min(off + chunkSize, file.length));
            chunks.add(new Chunk(sha256(data), data));
        }
        return chunks;
    }
}

How does deduplication save space?

Before storing a chunk, check if a chunk with that hash already exists. If so, just reference it and skip the upload. Thousands of loan applications at National Bank of India include the same standard terms-and-conditions document. With deduplication, those identical pages are stored only once.

package com.bank.filestorage;

public class DeduplicatingStore {
    public void storeChunk(Chunk chunk) {
        if (blobStore.exists(chunk.hash())) return;  // already stored, skip
        blobStore.put(chunk.hash(), chunk.data());   // store once
    }
}

What is the metadata model?

Metadata records which chunks in which order form each file, plus owner, permissions, timestamps, and version. The metadata database is relational (PostgreSQL) and separate from the blob store. This separation means the metadata layer can be queried, indexed, and backed up independently.

package com.bank.filestorage;

import java.util.List;
import java.util.Set;

public record FileMetadata(String fileId, String owner, long version,
                           List<String> chunkHashes, Set<String> readers,
                           long createdAt, long updatedAt) {}

How does access control work?

Every read and write checks whether the requesting user is the owner or in the allowed readers set. A loan officer can see their own branch's documents but not another branch's files. Access is enforced at the API layer before generating any pre-signed URL.

package com.bank.filestorage;

public class AccessControl {
    public boolean canRead(String userId, FileMetadata file) {
        return userId.equals(file.owner()) || file.readers().contains(userId);
    }
    public boolean canWrite(String userId, FileMetadata file) {
        return userId.equals(file.owner());
    }
}

How does versioning work?

Each change stores a new metadata record pointing to the updated set of chunk hashes. Old chunks remain referenced by older versions, so any previous version can be restored by pointing back to its chunk list. Deletion of a version does not remove shared chunks still referenced by other versions.

package com.bank.filestorage;

import java.util.List;

public class VersioningService {
    public FileMetadata updateFile(String fileId, List<String> newChunkHashes) {
        FileMetadata current = metadataStore.latest(fileId);
        FileMetadata newVersion = new FileMetadata(
            fileId, current.owner(), current.version() + 1,
            newChunkHashes, current.readers(),
            current.createdAt(), System.currentTimeMillis());
        metadataStore.save(newVersion);
        return newVersion;
    }
    public FileMetadata restore(String fileId, long version) {
        return metadataStore.findVersion(fileId, version);
    }
}

How do downloads use CDN?

Popular files are served via CDN with signed, time-limited URLs. The API generates a signed URL only after access control passes. The CDN caches the file at edge locations, so subsequent requests for the same file are served locally.

package com.bank.filestorage;

public class DownloadService {
    public String generateSignedUrl(String userId, String fileId) {
        FileMetadata meta = metadataStore.latest(fileId);
        if (!accessControl.canRead(userId, meta))
            throw new AccessDeniedException();
        return cdnSigner.sign(fileId, /* expiresInSeconds */ 300);
    }
}

How do uploads bypass application servers?

Uploads go directly to the blob store via pre-signed URLs. The application server generates the URL with an embedded expiry and signature. The client uploads chunks directly to the blob store, and file bytes never pass through application servers, saving bandwidth and CPU.

How do you handle orphaned chunks?

When a file version is permanently deleted, its chunk references are removed. A garbage collection job periodically scans for chunks that have zero references across all file versions and deletes them from the blob store to reclaim storage.

What should a senior engineer know?

A senior engineer considers garbage collection of orphaned chunks, encryption at rest for compliance with banking regulations, multi-region replication for disaster recovery, and quota management to prevent a single department from consuming all storage. They also design the metadata schema for efficient listing and search across millions of documents.