Definition
- Small-files problem – performance degradation from storing a dataset as too many tiny files instead of fewer right-sized ones.
- Causes – streaming writes, high-cardinality partitionBy, over-partitioned batch jobs.
- Consequences – excessive metadata, slow file listing (especially on S3), task scheduling overhead (one task per file).
- Write-time fixes – coalesce/repartition before write, maxRecordsPerFile, repartition by partition column.
- Post-write fixes – periodic compaction, Delta Lake OPTIMIZE, Iceberg/Hudi compaction.
- Ideal file size – roughly 128 MB to 1 GB.
At the National Bank of India, imagine filing each transaction slip in its own separate envelope. Finding a day's slips means opening millions of envelopes – the opening takes far longer than reading the slips. The fix is to batch many slips into sensibly sized folders and periodically consolidate scattered envelopes.
Understanding the Concepts
What is the small-files problem?
When a dataset is stored as a huge number of tiny files, every operation that touches those files pays overhead proportional to the file count, not the data size. This overhead can dominate the actual work.
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.appName("SmallFilesDemo").getOrCreate()
# Check how many files a directory has
import subprocess
result = subprocess.run(["hdfs", "dfs", "-count", "/data/transactions"],
capture_output=True, text=True)
print(result.stdout)
# If you see millions of files, you have the small-files problem
What causes the small-files problem?
Three common patterns produce excessive small files.
# Cause 1: Streaming writes (few records per micro-batch)
stream = spark.readStream.format("kafka") \
.option("subscribe", "transactions").load()
stream.writeStream \
.format("parquet") \
.option("checkpointLocation", "/checkpoints/txns") \
.trigger(processingTime="10 seconds") \
.start("/data/streaming_txns")
# Every 10 seconds: one tiny file per partition
# After a day: 8640 micro-batches * partitions = thousands of tiny files
# Cause 2: High-cardinality partitionBy
txns = spark.read.parquet("/data/transactions")
txns.write.partitionBy("txn_id").parquet("/data/bad_partitioned")
# One directory per unique txn_id = millions of one-row directories!
# Cause 3: Over-partitioned batch job
spark.conf.set("spark.sql.shuffle.partitions", "10000")
txns.groupBy("account_id").sum("amount") \
.write.parquet("/data/over_partitioned")
# 10000 output files, most containing a few KB
What are the consequences?
The overhead manifests in three ways that compound at scale.
1. Metadata overhead:
Each file is a metastore/object store entry.
Millions of entries slow metastore operations.
2. File listing overhead:
Before a read, Spark lists all files to plan the job.
On S3: listing 1 million files takes 5-10 minutes!
On HDFS: faster but still significant.
3. Task scheduling overhead:
Spark creates one task per file for reads.
1 million files = 1 million tasks.
Per-task overhead (scheduling, reporting) dwarfs actual work.
Driver memory fills up with task metadata.
How do you fix it at write time?
Control the number of output files before writing.
# Fix 1: Coalesce before write (reduces partitions without shuffle)
txns.coalesce(10).write.parquet("/data/coalesced")
# Produces ~10 files instead of thousands
# Fix 2: Repartition before write (with shuffle for balance)
txns.repartition(20).write.parquet("/data/repartitioned")
# Produces ~20 evenly-sized files
# Fix 3: Repartition by partition column
txns.repartition("txn_date") \
.write.partitionBy("txn_date") \
.parquet("/data/date_partitioned")
# Each date directory gets one file per writer instead of many
# Fix 4: maxRecordsPerFile (caps records per file)
txns.write \
.option("maxRecordsPerFile", 1000000) \
.parquet("/data/capped")
# Each file has at most 1M records, preventing oversized files too
# Fix 5: Avoid high-cardinality partitionBy
# WRONG: partitionBy("txn_id") -> millions of directories
# CORRECT: partitionBy("txn_date") -> 365 directories per year
How do you fix it after writing?
For streaming and append-heavy tables, periodic compaction consolidates small files into larger ones.
# Delta Lake OPTIMIZE: compacts small files
from delta.tables import DeltaTable
delta_table = DeltaTable.forPath(spark, "/data/lake/transactions")
delta_table.optimize().executeCompaction()
# Schedule as a nightly job
# OPTIMIZE runs as a Delta transaction: readers are not affected
# Manual compaction for non-Delta tables:
df = spark.read.parquet("/data/streaming_txns/txn_date=2024-01-15")
df.coalesce(4) \
.write.mode("overwrite") \
.parquet("/data/streaming_txns/txn_date=2024-01-15")
What is the ideal file size?
The target file size depends on the storage system and access pattern, but a general guideline is 128 MB to 1 GB.
| File Size | Effect | Recommendation |
|---|---|---|
| < 1 MB | Severe overhead | Always compact |
| 1-10 MB | Moderate overhead | Compact regularly |
| 10-128 MB | Acceptable for streaming | OK for hot data |
| 128 MB - 1 GB | Ideal | Target for batch |
| > 1 GB | Single task reads whole file | Split if needed |
# Calculate target partition count for a desired file size
data_size_mb = 50000 # 50 GB total
target_file_size_mb = 256
target_partitions = data_size_mb // target_file_size_mb # ~195 files
txns.repartition(target_partitions).write.parquet("/data/right_sized")
What about streaming latency vs file size?
There is a tension between streaming latency and file size: writing less frequently produces larger files but adds latency.
# Trade-off: latency vs file size
# Option A: frequent triggers, small files, compact later
stream.writeStream.trigger(processingTime="10 seconds") \
.start("/data/stream_output")
# Low latency but many small files; schedule nightly OPTIMIZE
# Option B: less frequent triggers, larger files
stream.writeStream.trigger(processingTime="5 minutes") \
.start("/data/stream_output")
# Higher latency but fewer, larger files; less compaction needed
# Option C: use Delta with auto-compaction
spark.conf.set("spark.databricks.delta.autoCompact.enabled", "true")
stream.writeStream.format("delta") \
.start("/data/delta_stream")
# Auto-compaction handles small files automatically
What should a senior engineer know?
- Design the write path to avoid the small-files problem rather than discovering it when reads are slow.
- At the National Bank of India, the streaming transaction table accumulated thousands of small files daily until reads spent minutes listing files; nightly Delta OPTIMIZE fixed it.
- Avoid high-cardinality partitionBy (by transaction id or account id); partition by date or region instead.
- Coalesce before batch writes and schedule compaction for streaming tables.
- Monitor file counts and sizes as part of data quality checks.
Visual Overview
Small Files Problem ┌──┐┌──┐┌──┐┌──┐┌──┐┌──┐┌──┐┌──┐┌──┐┌──┐ │1K││2K││500││1K││3K││800││1K││2K││500││1K│ Thousands of tiny files └──┘└──┘└──┘└──┘└──┘└──┘└──┘└──┘└──┘└──┘ Listing: 10 minutes Tasks: 10000 Overhead: huge After Compaction (coalesce / OPTIMIZE) ┌─────────────────┐ ┌─────────────────┐ │ 256 MB │ │ 256 MB │ 2 right-sized files └─────────────────┘ └─────────────────┘ Listing: instant Tasks: 2 Overhead: minimal
Common Causes
Streaming: High-cardinality Over-partitioned:
┌──┐ every 10s partitionBy: ┌──┐┌──┐┌──┐...
┌──┐ every 10s txn_id=TXN001/ 10000 shuffle
┌──┐ every 10s └── 1 row partitions
┌──┐ every 10s txn_id=TXN002/ each with
8640 files/day └── 1 row a tiny file
millions of dirs!
Key Takeaway
The small-files problem is when a dataset is stored as too many tiny files, degrading performance through excessive metadata, slow file listing, and task scheduling overhead. Each file is an entry the storage system must track, Spark must list all files before reading, and it creates one task per file, so millions of tiny files means millions of tasks whose overhead dwarfs the actual work.
The common causes are streaming writes that produce a few records per micro-batch, high-cardinality partitionBy that creates a directory per value, and over-partitioned batch jobs. The fixes map to these causes: coalesce or repartition before writing to control file count, avoid high-cardinality partitionBy by partitioning on date or region, cap file size with maxRecordsPerFile, and schedule compaction or Delta OPTIMIZE for streaming tables.
At the National Bank of India, the streaming transaction table accumulated thousands of small Parquet files daily until reads spent minutes just listing them. A nightly Delta OPTIMIZE compacted each partition into right-sized files. A separate batch job had mistakenly partitioned by transaction id, creating millions of one-row directories, and was fixed by partitioning on date and coalescing before writing.
The design rule is to prevent the problem at the write path: avoid high-cardinality partitionBy, coalesce to a target file count for batch, and schedule compaction for streaming. The ideal file size is 128 MB to 1 GB, aligned with row-group sizes.
Remember one sentence: too many tiny files make reads slow because of listing and scheduling overhead, so coalesce at write time and compact periodically to keep files right-sized.
---