Definition
- MEMORY_ONLY – partitions stay deserialized in heap memory; overflow is dropped and recomputed from lineage.
- MEMORY_AND_DISK – partitions prefer memory but spill to local disk instead of being dropped; the safe general-purpose default.
- DISK_ONLY – all partitions stored on disk; for data too large for memory where recomputation is still costlier than disk reads.
- MEMORY_ONLY_2 / MEMORY_AND_DISK_2 – replicated versions keeping two copies on different nodes for fault tolerance.
- OFF_HEAP – partitions stored outside the JVM heap to avoid garbage collection scanning.
- Serialized variants – append
_SERto pack data into compact byte arrays, trading CPU for memory savings.
Picture how a clerk at the National Bank of India stores finished summaries. MEMORY_ONLY is keeping them on the desk: instant to grab, but extras are thrown away when the desk is full. MEMORY_AND_DISK keeps what fits on the desk and files the rest in a nearby cabinet. DISK_ONLY files everything in the cabinet. Replicated versions keep a backup copy on another floor. OFF_HEAP is a special storage room the office cleaner (garbage collector) never disturbs.
Understanding the Concepts
What are Spark storage levels?
A storage level tells Spark how to store cached partitions. Each level is a trade-off across speed, memory usage, durability, and CPU cost. You choose a level with persist(StorageLevel.LEVEL_NAME) when caching a dataset.
from pyspark.sql import SparkSession, functions as F
from pyspark import StorageLevel
spark = SparkSession.builder.appName("NBI-StorageLevels").getOrCreate()
txns = spark.read.parquet("s3://nbi-data-lake/transactions/")
accounts = spark.read.parquet("s3://nbi-data-lake/accounts/")
cleaned = txns.filter(F.col("status") == "SETTLED") \
.join(accounts, "accountId")
How does MEMORY_ONLY work?
MEMORY_ONLY keeps partitions deserialized in JVM heap memory. This is the fastest level because data is ready to use immediately with no deserialization cost. The weakness is that partitions that do not fit are simply not cached. When a later action needs them, they are recomputed from lineage.
# MEMORY_ONLY: fastest, but drops overflow
cleaned.persist(StorageLevel.MEMORY_ONLY)
cleaned.count() # Compute + cache what fits
# If 200 partitions need 4 GB but executor has 2 GB for storage,
# only ~100 partitions are cached. The rest are recomputed on demand.
# Best for: small datasets that fit comfortably in memory
# Example: branch lookup table (200 rows, a few KB)
branches = spark.read.parquet("s3://nbi-data-lake/branches/")
branches.persist(StorageLevel.MEMORY_ONLY)
branches.count()
How does MEMORY_AND_DISK work?
MEMORY_AND_DISK prefers memory but spills overflow partitions to the executor's local disk instead of dropping them. Reading from disk is slower than memory but far faster than recomputing an expensive lineage. This is the safe general-purpose default for DataFrames.
# MEMORY_AND_DISK: safe default, no data loss
cleaned.persist(StorageLevel.MEMORY_AND_DISK)
cleaned.count()
# Partitions that fit → memory (fast)
# Partitions that don't fit → local disk (slower but no recompute)
# Best for: general-purpose caching of medium to large datasets
# This is exactly what cache() uses for DataFrames
How does DISK_ONLY work?
DISK_ONLY stores all partitions on the executor's local disk. No memory is used for caching. This is for datasets too large to fit in memory at all, where recomputation is so costly that reading from disk is still worthwhile.
# DISK_ONLY: all on disk, no memory used
huge_history = spark.read.parquet("s3://nbi-data-lake/transactions/10-years/")
huge_history.persist(StorageLevel.DISK_ONLY)
huge_history.count()
# Best for: very large datasets where:
# - Memory is too scarce to hold any of it
# - Recomputation is expensive (complex joins, external sources)
# - Disk reads are acceptable
How do replicated levels work?
Levels ending in _2 keep two copies of each partition on different executor nodes. If an executor dies, the second copy serves the data immediately without recomputation.
# MEMORY_AND_DISK_2: two copies for fault tolerance
feature_matrix = compute_risk_features(cleaned) # Takes 2 hours
feature_matrix.persist(StorageLevel.MEMORY_AND_DISK_2)
feature_matrix.count()
# If Executor 3 dies:
# Without _2: recompute lost partitions from lineage (hours!)
# With _2: read backup from Executor 7 (instant)
# Cost: 2x storage (memory + disk on two nodes)
# Best for: extremely expensive datasets where recompute is unaffordable
How does OFF_HEAP storage work?
OFF_HEAP stores partitions in memory outside the JVM heap. The garbage collector does not scan this memory, which eliminates GC pauses caused by large caches.
# Enable off-heap memory first
spark.conf.set("spark.memory.offHeap.enabled", "true")
spark.conf.set("spark.memory.offHeap.size", "10g")
# OFF_HEAP: outside GC, serialized format
cleaned.persist(StorageLevel.OFF_HEAP)
cleaned.count()
# The 2.4 GB cache no longer causes GC pauses
# because the garbage collector does not scan off-heap memory.
# Best for: large caches that cause long GC pauses
How do serialized variants save memory?
Appending _SER to a storage level stores data as compact byte arrays instead of deserialized Java objects. This typically uses 2x to 5x less memory but requires CPU to serialize on write and deserialize on read.
# Deserialized: fast access, more memory
cleaned.persist(StorageLevel.MEMORY_AND_DISK) # ~2.4 GB
# Serialized: compact, less memory, CPU overhead
cleaned.persist(StorageLevel.MEMORY_AND_DISK_SER) # ~0.8 GB
# Best for: large datasets in tight memory where you can trade CPU for space
How do you choose the right storage level?
Ask four questions about the dataset:
- Does it fit in memory? If yes, MEMORY_ONLY. If no, MEMORY_AND_DISK.
- Is memory tight? If yes, use a serialized variant (_SER).
- Is recomputation extremely expensive? If yes, use a replicated variant (_2).
- Is the cache causing GC pauses? If yes, use OFF_HEAP.
# Decision tree in code:
def choose_storage_level(fits_in_memory, memory_tight,
recompute_expensive, gc_problem):
if gc_problem:
return StorageLevel.OFF_HEAP
if recompute_expensive:
return StorageLevel.MEMORY_AND_DISK_2
if memory_tight:
return StorageLevel.MEMORY_AND_DISK_SER
if fits_in_memory:
return StorageLevel.MEMORY_ONLY
return StorageLevel.MEMORY_AND_DISK # safe default
# At the bank:
level = choose_storage_level(
fits_in_memory=False, memory_tight=True,
recompute_expensive=False, gc_problem=False
)
cleaned.persist(level) # MEMORY_AND_DISK_SER
What are common mistakes with storage levels?
The most common mistake is using MEMORY_ONLY for a dataset that does not fit, causing silent recomputation that defeats the purpose of caching.
# WRONG: MEMORY_ONLY for data that does not fit
huge_data.persist(StorageLevel.MEMORY_ONLY)
huge_data.count()
# Only 30% is cached. The other 70% is silently recomputed
# on every access, making the job SLOWER than no caching.
# RIGHT: MEMORY_AND_DISK for data that may not fit
huge_data.persist(StorageLevel.MEMORY_AND_DISK)
huge_data.count()
# 100% cached: some in memory, rest on disk. No recomputation.
# WRONG: Replicated level for cheap datasets (wastes 2x memory)
branches.persist(StorageLevel.MEMORY_AND_DISK_2)
# Branches table is 200 rows. Recomputing takes milliseconds.
# Replication wastes memory for no benefit.
What should a senior engineer know?
A senior engineer matches the storage level to each dataset rather than caching everything at the default. They check the Spark UI Storage tab to verify that caches are fully materialized and that the storage level matches the dataset's characteristics.
# Senior practice at the bank:
# Small lookup: MEMORY_ONLY (fast, fits easily)
branches.persist(StorageLevel.MEMORY_ONLY)
# Medium shared dataset: MEMORY_AND_DISK (safe default)
cleaned.persist(StorageLevel.MEMORY_AND_DISK)
# Large packed dataset: MEMORY_AND_DISK_SER (tight fit)
large_txns.persist(StorageLevel.MEMORY_AND_DISK_SER)
# Critical ML features: MEMORY_AND_DISK_2 (no recompute risk)
features.persist(StorageLevel.MEMORY_AND_DISK_2)
# GC-heavy cache: OFF_HEAP (no GC pauses)
wide_cache.persist(StorageLevel.OFF_HEAP)
Visual Overview
Storage level decision tree:
Does it fit in memory?
├── YES → Is GC a problem?
│ ├── YES → OFF_HEAP
│ └── NO → Is recompute expensive?
│ ├── YES → MEMORY_ONLY_2
│ └── NO → MEMORY_ONLY
└── NO → Is memory tight?
├── YES → Is recompute expensive?
│ ├── YES → MEMORY_AND_DISK_SER_2
│ └── NO → MEMORY_AND_DISK_SER
└── NO → Is recompute expensive?
├── YES → MEMORY_AND_DISK_2
└── NO → MEMORY_AND_DISK (default)
Memory layout per storage level: MEMORY_ONLY: [Objects in heap] ← fast, GC scans MEMORY_AND_DISK: [Objects in heap] + [Files on disk] MEMORY_AND_DISK_SER: [Byte arrays in heap] + [Files on disk] OFF_HEAP: [Bytes outside heap] ← GC-free DISK_ONLY: [Files on disk only]
Key Takeaway
Spark storage levels control how cached partitions are stored, and each level is a trade-off across speed, memory, durability, and CPU. MEMORY_ONLY is fastest because partitions stay deserialized in heap memory, but overflow is dropped and recomputed, so it only works when data fits comfortably. MEMORY_AND_DISK fixes that by spilling to disk instead of dropping, making it the safe general-purpose default.
Serialized variants pack data into compact byte arrays, using 2x to 5x less memory at the cost of CPU. Replicated variants keep two copies on different nodes so an executor failure does not trigger recomputation. OFF_HEAP stores data outside the JVM heap to eliminate GC pauses on large caches. DISK_ONLY puts everything on disk for datasets too large for any memory allocation.
At the National Bank of India I matched the level to each dataset rather than caching everything at the default. A small branch lookup used MEMORY_ONLY for speed. The shared cleaned transaction dataset used MEMORY_AND_DISK. A large packed history used MEMORY_AND_DISK_SER. An expensive ML feature matrix used a replicated level. A wide cache causing GC pauses moved to OFF_HEAP.
The questions to ask are: does it fit in memory, how expensive is recomputation, is memory tight, and is GC a problem. Answering those gives the right level every time.
Remember one sentence: choose the storage level by asking whether data fits in memory, how expensive recomputation is, and whether GC or fault tolerance is a concern, rather than using the default blindly.
---