Definition
- Spill-to-disk – Spark's safety mechanism: when a shuffle, sort, or aggregation produces more intermediate data than fits in execution memory, Spark writes the overflow to local disk rather than crashing.
- Correctness-preserving – spilled data is serialized, written to disk, read back, and merged; the result is correct but much slower.
- Performance red flag – disk I/O is orders of magnitude slower than memory; heavy spill means a stage is under-provisioned or overloaded.
- Diagnosis – the Stages tab in Spark UI reports spill (memory) and spill (disk) per task.
At the National Bank of India a clerk sorting a pile of slips keeps the working pile on the desk (memory), but if the pile grows too tall to fit, instead of giving up the clerk moves the overflow into a filing cabinet drawer (disk) and fetches it back when needed. The work still completes correctly, but every trip to the cabinet is far slower than reaching across the desk, so a clerk who keeps shuttling slips to and from the cabinet all day is a sign the desk is too small or the pile is too big.
Understanding the Concepts
What is spill-to-disk?
Spill happens inside memory-hungry operators. A sort-based shuffle, an external sort for order-by or sort-merge join, and a hash aggregation all build up intermediate data in execution memory. When that data exceeds the memory the task can claim, the operator writes sorted or partitioned runs to local disk, frees the memory, and continues. Multiple spills may occur, producing several runs that are later merged.
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
spark = SparkSession.builder \
.appName("NBI-Spill-Demo") \
.config("spark.executor.memory", "2g") \
.config("spark.sql.shuffle.partitions", "4") \
.getOrCreate()
# Large aggregation that may spill
txn = spark.read.parquet("hdfs:///bank/transactions/2026/")
result = txn.groupBy("account_id", "branch") \
.agg(F.sum("amount"), F.count("*"), F.avg("amount")) \
.orderBy("branch")
result.write.parquet("hdfs:///bank/reports/account-summary/")
# Check Spark UI → Stages tab → spill metrics
# spill (memory): bytes of data that were spilled (in-memory size)
# spill (disk): bytes actually written to disk (serialized, compressed)
How is spill different from storage eviction?
Spill and eviction both involve disk, but they concern different memory regions. Spill is about execution memory: the operator's own active working set is written to disk because it cannot fit. Eviction is about storage memory: cached blocks are removed when execution needs space.
# Spill: execution operator's data goes to disk
# Example: hash aggregation buffer too large for execution memory
result = txn.groupBy("account_id").agg(F.sum("amount")) # may spill
# Eviction: cached blocks removed from storage memory
txn.cache()
txn.count() # materializes cache
# If execution needs memory, cached blocks may be evicted
# Eviction frees storage memory; spill writes execution data to disk
How do you read spill metrics in the Spark UI?
The Stages tab reports spill per task. Look at the summary metrics (min, median, max) and the task-level detail.
Spark UI → Stages tab → Stage detail → Summary Metrics: Metric Min 25th Median 75th Max Input Size 120 MB 125 MB 128 MB 130 MB 132 MB Shuffle Read 50 MB 55 MB 58 MB 60 MB 1.2 GB ← skew! Spill (Memory) 0 0 0 0 2.8 GB ← one task spilling Spill (Disk) 0 0 0 0 1.1 GB ← serialized/compressed Duration 12s 15s 18s 20s 180s ← straggler
# Interpreting spill metrics:
# - Spill concentrated on ONE task (max >> median): DATA SKEW
# - Spill UNIFORM across all tasks: TOO FEW PARTITIONS or TOO LITTLE MEMORY
# - Spill (disk) << Spill (memory): good compression ratio
What causes spill?
The causes map to specific fixes. Too few shuffle partitions, data skew, over-caching, and undersized executors are the main culprits.
# Cause 1: Too few shuffle partitions → each partition too large
spark.conf.set("spark.sql.shuffle.partitions", "200") # default
# Fix: increase partitions
spark.conf.set("spark.sql.shuffle.partitions", "800")
# Cause 2: Data skew → one partition is huge
# Fix: AQE skew join, salt hot keys
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
# Cause 3: Over-caching → steals execution memory
# Fix: unpersist unneeded caches
txn.unpersist()
# Cause 4: Undersized executor
# Fix: increase executor memory
# spark.conf.set("spark.executor.memory", "8g")
How do you distinguish uniform spill from skewed spill?
The distinction decides whether to add partitions/memory (uniform) or attack a hot key (skewed).
# Read spill from Spark UI:
# If ALL tasks spill roughly equally:
# → Partitions too large or executor too small
# → Fix: increase spark.sql.shuffle.partitions or executor memory
# If ONE task spills heavily while others do not:
# → Data skew on a hot key
# → Fix: AQE skew join, salt the hot key, isolate and broadcast
# Example: checking for skew programmatically
per_partition = txn.withColumn("part", F.spark_partition_id()) \
.groupBy("part").count().orderBy(F.desc("count"))
per_partition.show(5)
+----+-------+
|part| count|
+----+-------+
| 17|5000000|
| 3| 25000|
| 12| 24800|
| 0| 24500|
| 7| 24200|
+----+-------+
How does Kryo serialization reduce spill cost?
Spilled data must be serialized to disk and deserialized on read-back. Kryo produces smaller, faster serialization than the default Java serializer, reducing both the I/O volume and the CPU cost of spill.
# Enable Kryo serialization to reduce spill bytes
spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
spark.conf.set("spark.kryo.registrationRequired", "false")
# Kryo typically produces 2-5x smaller serialized output
# This directly reduces spill (disk) bytes and ser/deser CPU time
What is the goal: zero spill or bounded spill?
The goal is not zero spill. A little spill on a large job is normal and acceptable. Forcing everything into memory can cause OOM instead. The goal is bounded spill that is not the dominant cost.
# Acceptable: small spill relative to data processed
# Input: 10 GB, Spill (disk): 200 MB → 2% → fine
# Concerning: large spill relative to data processed
# Input: 10 GB, Spill (disk): 15 GB → 150% → stage bottleneck
# Rule of thumb:
# spill_ratio = spill_disk_bytes / input_bytes
# < 10%: acceptable
# 10-50%: investigate
# > 50%: definitely fix
What should a senior engineer know?
A senior engineer reads spill in context rather than chasing zero spill. They look at the ratio of spill bytes to input bytes, distinguish uniform from concentrated spill, and apply the matching fix.
# Senior diagnostic pattern:
# 1. Open Stages tab, find the slow stage
# 2. Check spill (memory) and spill (disk) in summary metrics
# 3. Compare max to median:
# - max >> median → skew → fix the hot key
# - max ≈ median → uniform → increase partitions or memory
# 4. Check Storage tab: is over-caching squeezing execution?
# 5. Check if Kryo is enabled
# 6. Apply fix, re-run, verify spill decreased
Visual Overview
Spill-to-Disk Mechanism: Task execution ┌──────────────────────────────────────────┐ │ Execution Memory (hash table / sort buf)│ │ ┌──────────────────────────────┐ │ │ │ Data: [A, B, C, D, E, F...]│ │ │ │ Memory full! │ │ │ └──────────┬───────────────────┘ │ │ │ spill │ │ ▼ │ │ ┌──────────────────────────────┐ │ │ │ Local Disk (sorted run #1) │ │ │ │ [A, B, C, D] │ │ │ └──────────────────────────────┘ │ │ Continue processing... │ │ Memory full again! │ │ │ spill │ │ ▼ │ │ ┌──────────────────────────────┐ │ │ │ Local Disk (sorted run #2) │ │ │ │ [E, F, G, H] │ │ │ └──────────────────────────────┘ │ │ ... │ │ Final: merge all spilled runs │ └──────────────────────────────────────────┘ Diagnosing spill pattern: UNIFORM spill: SKEWED spill: [=][=][=][=][=][=] [=][=][=][=][=][========] All tasks spill equally One task spills heavily → partition too large → hot key / data skew → ↑ partitions or memory → fix the skew
Key Takeaway
Spill-to-disk is Spark's safety mechanism for memory-hungry operators. When a shuffle, sort, or hash aggregation builds up more intermediate data than fits in execution memory, Spark writes the overflow to local disk as sorted or partitioned runs, frees the memory, and continues, later reading those runs back and merging them. This is what lets Spark process datasets far larger than memory without crashing, so a small amount of spill on a big job is completely normal.
Excessive spill is a red flag because local disk is orders of magnitude slower than RAM, and spilled data must be serialized out, read back, deserialized, and merged, adding both I/O and CPU cost. When spill bytes are large relative to the data processed, the stage spends most of its time shuttling data to and from disk rather than computing. In the Spark UI the Stages tab reports spill (memory) and spill (disk) per task, and the key diagnostic is whether spill is uniform across all tasks or concentrated on one.
At the National Bank of India I used that distinction constantly. Uniform spill across every task of a sort-merge join meant partitions were too large, so I raised the shuffle partition count and sometimes the executor memory. Spill concentrated on a single task meant skew on a hot account, which I fixed with AQE skew join or salting rather than partition count. I also enabled the Kryo serializer to shrink spilled bytes.
The nuance I share in interviews is that the goal is not zero spill: forcing everything into memory invites OOM. The goal is bounded spill, and reading whether it is uniform or concentrated tells you whether to add partitions and memory or to attack a hot key.
Remember one sentence: spill-to-disk writes overflow execution data to local disk to prevent OOM, and excessive spill is a red flag where uniform spill means partitions are too large and concentrated spill means data skew.
---