Definition
- Driver OOM – caused by pulling large data to the driver:
collect(),toPandas(), large broadcast assembly, or driver-side accumulation. - Executor OOM – caused by a task's working set exceeding execution memory: data skew, too few partitions, oversized broadcast, excessive shuffle, or over-caching.
- Diagnosis tool – the Spark UI: Stages tab for task distribution, Executors tab for GC time, Storage tab for cache size, SQL tab for the physical plan.
- Fix principle – classify the cause from evidence, then apply the matching fix; adding memory only helps genuine volume problems, not skew or logic errors.
At the National Bank of India an OutOfMemory error is like a clerk's desk overflowing with paperwork until they collapse. Debugging it is detective work: you first ask whether it was a field clerk (executor) or the head manager (driver) who collapsed, then check the activity logs (Spark UI) to find whether one clerk was handed a mountain of slips while others sat idle (skew), everyone was drowning in shuffled paperwork (excessive shuffle), a huge reference binder was copied to every desk (large broadcast), there were too few desks so each pile was enormous (too few partitions), or the manager tried to pile the entire bank's records onto their own desk (driver collect).
Understanding the Concepts
What is the first question when debugging a Spark OOM?
The first question is always: driver or executor? A driver OOM and an executor OOM have different causes, different symptoms, and different fixes. Check the error message and the failing component in the logs.
# Driver OOM: typically from collect() or toPandas()
# Error appears in driver logs:
# java.lang.OutOfMemoryError: Java heap space
# at org.apache.spark.sql.Dataset.collectToPython
# Executor OOM: typically from a task's working set
# Error appears in executor logs:
# java.lang.OutOfMemoryError: Java heap space
# ExecutorLostFailure (executor X exited caused by OOM)
How do you fix a driver OOM?
A driver OOM almost always comes from pulling large data back to the driver. The fix is to never collect large results; write to storage instead.
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
spark = SparkSession.builder.appName("NBI-OOM-Debug").getOrCreate()
txn = spark.read.parquet("hdfs:///bank/transactions/2026/")
# WRONG: collect() pulls millions of rows to driver → OOM
# all_rows = txn.collect() # driver OOM!
# CORRECT: write to storage
txn.write.parquet("hdfs:///bank/output/transactions/")
# CORRECT: if you must bring data back, limit it
sample = txn.limit(100).collect()
print(f"Sampled {len(sample)} rows")
# WRONG: toPandas() on large DataFrame → driver OOM
# pdf = txn.toPandas() # driver OOM!
# CORRECT: toPandas() only on small aggregated results
summary = txn.groupBy("branch").agg(F.sum("amount")).toPandas()
print(summary.head())
Sampled 100 rows
branch sum(amount)
0 Mumbai 45750000.00
1 Delhi 32100000.00
How do you diagnose an executor OOM using the Spark UI?
For an executor OOM, the Spark UI is the primary evidence source. Check four tabs in order: Stages, Executors, Storage, and SQL.
# Diagnostic checklist:
# 1. Stages tab → task duration and shuffle-size distribution
# - If max task >> median: DATA SKEW
# - If all tasks have large shuffle: EXCESSIVE SHUFFLE
#
# 2. Executors tab → GC time and failed tasks
# - If GC time > 30% of task time: MEMORY PRESSURE
#
# 3. Storage tab → cached data size
# - If cached data is large: OVER-CACHING
#
# 4. SQL tab → physical plan
# - BroadcastExchange with large size: OVERSIZED BROADCAST
# - Missing predicate pushdown: READING TOO MUCH DATA
How do you identify data skew as the cause?
Data skew is the most common executor OOM cause. In the Stages tab, if the maximum task duration and shuffle read are many times the median, one partition is far larger than the rest.
# Data skew example: one account has millions of transactions
# The partition containing that account's data is huge
# Checking for skew in the data
skew_check = txn.groupBy("account_id").count().orderBy(F.desc("count"))
skew_check.show(5)
# Fix 1: Enable AQE skew join (Spark 3.0+)
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
# Fix 2: Salt the hot key
from pyspark.sql.functions import lit, concat, floor, rand
hot_key = "POOL_ACCOUNT_001"
salt_range = 10
# Salt the large side
txn_salted = txn.withColumn(
"salted_key",
F.when(F.col("account_id") == hot_key,
F.concat(F.col("account_id"), F.lit("_"), F.floor(F.rand() * salt_range).cast("string")))
.otherwise(F.col("account_id"))
)
print("Salted key sample:")
txn_salted.filter(F.col("account_id") == hot_key).select("salted_key").show(3)
+----------+-------+
|account_id| count|
+----------+-------+
|POOL_AC...|5000000|
|SAV2001 | 150|
|SAV2002 | 120|
+----------+-------+
Salted key sample:
+--------------------+
| salted_key|
+--------------------+
|POOL_ACCOUNT_001_3 |
|POOL_ACCOUNT_001_7 |
|POOL_ACCOUNT_001_1 |
+--------------------+
How do you fix too few partitions?
Too few shuffle partitions make each partition too large to fit in memory. The fix is to increase spark.sql.shuffle.partitions.
# Default: 200 shuffle partitions
# If each partition is 2 GB and executor has 4 GB, tasks may OOM
# Fix: increase shuffle partitions
spark.conf.set("spark.sql.shuffle.partitions", "800")
# Now each partition is ~500 MB instead of 2 GB
# Better: enable AQE to auto-tune
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
How do you fix an oversized broadcast?
If a join broadcasts a table that is too large, every executor must hold the whole table, and the driver must assemble it first. Check the SQL tab for BroadcastExchange size.
# Check broadcast threshold
print("Broadcast threshold:", spark.conf.get("spark.sql.autoBroadcastJoinThreshold"))
# Fix 1: disable broadcast for this join
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
# Forces sort-merge join instead of broadcast
# Fix 2: lower the threshold
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "10485760") # 10 MB
# Fix 3: use join hints to control join strategy
from pyspark.sql.functions import broadcast
result = txn.join(broadcast(small_table), "account_id") # explicit broadcast
# Only broadcast when you KNOW the table is small
How do you fix over-caching?
Over-caching steals execution memory and causes spills. The symptom is slow shuffles, not a cache error.
# Diagnostic: check Storage tab in Spark UI
# If cached data uses most of the unified memory region:
# Fix 1: unpersist what you no longer need
txn.unpersist()
# Fix 2: avoid caching data used only once
# BAD: cache everything
txn.cache(); accounts.cache(); ref.cache()
# GOOD: cache only what is reused multiple times
txn.cache()
result1 = txn.groupBy("branch").agg(F.sum("amount"))
result2 = txn.groupBy("account_id").agg(F.count("*"))
txn.unpersist() # done
# Fix 3: use MEMORY_AND_DISK to allow spill instead of eviction
from pyspark import StorageLevel
txn.persist(StorageLevel.MEMORY_AND_DISK)
What are common mistakes when debugging OOM?
The most common mistake is throwing more memory at every OOM without diagnosing the cause.
# WRONG: blindly increase memory for every OOM
# spark.conf.set("spark.executor.memory", "32g") # masks the real problem
# CORRECT: diagnose first, then fix
# 1. Skew → AQE skew join, salt hot keys
# 2. Too few partitions → increase shuffle.partitions
# 3. Oversized broadcast → disable broadcast for that join
# 4. Over-caching → unpersist, cache selectively
# 5. Driver collect → write to storage instead
# 6. Genuine volume → increase executor memory
What should a senior engineer know?
A senior engineer follows a disciplined order of investigation and distinguishes configuration fixes from logic fixes. They read the UI, classify the cause, and apply the matching fix.
# Senior diagnostic flowchart:
# 1. WHERE did OOM happen? → Driver or Executor
# 2. If Driver:
# - collect() / toPandas() → write to storage
# - large broadcast assembly → disable broadcast
# 3. If Executor:
# a. Stages tab: task max >> median? → SKEW
# b. Stages tab: all tasks large shuffle? → increase partitions
# c. Executors tab: high GC? → memory pressure
# d. Storage tab: large cache? → over-caching
# e. SQL tab: large BroadcastExchange? → oversized broadcast
# 4. Apply matching fix, not just more memory
Visual Overview
Spark OOM Debugging Flowchart:
OutOfMemoryError
│
▼
┌──────────────────┐
│ Driver or │
│ Executor? │
└─────┬──────┬─────┘
│ │
DRIVER EXECUTOR
│ │
▼ ▼
┌────────┐ ┌──────────────────────────────────────┐
│collect │ │ Check Spark UI: │
│toPandas│ │ │
│large │ │ Stages tab: task max >> median? │
│bcast │ │ YES → DATA SKEW │
└───┬────┘ │ Fix: AQE skew join, salt hot key │
│ │ │
▼ │ Executors tab: GC > 30%? │
Fix: write │ YES → MEMORY PRESSURE │
to storage │ │
│ Storage tab: large cached data? │
│ YES → OVER-CACHING │
│ Fix: unpersist, cache selectively │
│ │
│ SQL tab: large BroadcastExchange? │
│ YES → OVERSIZED BROADCAST │
│ Fix: disable broadcast │
│ │
│ All tasks uniformly large? │
│ YES → TOO FEW PARTITIONS │
│ Fix: increase shuffle.partitions │
└──────────────────────────────────────┘
Key Takeaway
Debugging a Spark OutOfMemoryError is systematic detective work, and my first step is always to determine whether the driver or an executor ran out of memory, because the causes are different. A driver OOM is almost always from pulling large data back with collect, toPandas, or assembling a huge broadcast, and the fix is to write results to storage instead. An executor OOM is a task's working set exceeding execution memory, and that is where I use the Spark UI.
In the UI I check four tabs. The Stages tab shows the task duration and shuffle-size distribution: a maximum many times the median means data skew. The Executors tab shows GC time and failed tasks: a high GC fraction means the executor is thrashing near its limit. The Storage tab shows cached data size: over-caching steals execution memory. The SQL tab shows the plan: an unexpectedly large BroadcastExchange reveals a table being broadcast that is too big.
At the National Bank of India this playbook resolved real incidents: a transactions join that OOMed turned out to be skew on a pooling account, visible as one task with fifty times the median shuffle read; a daily report that crashed the driver was a collect on millions of rows; and a join that OOMed every executor was broadcasting a reference table that had grown large.
The lesson I stress in interviews is that the junior reflex of adding memory to every OOM only masks skew and accidental collects, whereas the senior approach is to read the UI, classify the cause, and apply the matching fix.
Remember one sentence: debug a Spark OOM by first checking driver vs executor, then using the Spark UI tabs to classify the cause as skew, excessive shuffle, oversized broadcast, too few partitions, or over-caching, and apply the matching fix.
---