Back to Apache Spark: Core
Q15Question 15 of 72

Difference between cache() and persist(StorageLevel)

Key Concepts
  • cache() is just persist() with a fixed default storage level; under the hood they are the same mechanism
  • For DataFrames, cache() defaults to MEMORY_AND_DISK; for RDDs, it defaults to MEMORY_ONLY
  • persist(StorageLevel) lets you explicitly choose memory vs disk, serialized vs deserialized, off-heap, and replication
  • Both are lazy, both store partitions for reuse, both recompute evicted partitions from lineage
  • Serialized storage packs data tighter at the cost of CPU; off-heap reduces GC pressure on large caches
  • Match the storage level to the dataset's size, reuse pattern, and recomputation cost rather than using the default blindly

Definition

cache() is persist() with a fixed default storage level, while persist(StorageLevel) lets you explicitly choose memory vs disk, serialized vs deserialized, off-heap, and replication.
  • cache() – a convenience method that calls persist() with a fixed default: MEMORY_AND_DISK for DataFrames, MEMORY_ONLY for RDDs.
  • persist(StorageLevel) – the general method that accepts an explicit storage level controlling location (memory, disk, or both), format (serialized or deserialized), off-heap usage, and replication factor.
  • Same mechanism – both are lazy, both store computed partitions for reuse, both are released with unpersist(), and both recompute evicted partitions from lineage.
  • Key difference – cache fixes the storage level choice; persist exposes it.

Think of storing a finished summary at the National Bank of India. Calling cache() is telling the clerk "keep this handy" and letting him use the branch's standard spot. Calling persist() is handing him a detailed instruction card: keep it on your desk if there is room, otherwise in the filing cabinet; keep it folded tightly to save space or flat for instant reading; and keep a backup copy on another floor in case your desk is cleared.

Understanding the Concepts

What is the difference between cache() and persist()?

Under the hood, cache() calls persist() with a default storage level. They are the same mechanism. The only difference is that cache() fixes the storage level while persist() lets you choose it.

PYTHON
from pyspark.sql import SparkSession, functions as F
from pyspark import StorageLevel

spark = SparkSession.builder.appName("NBI-CacheVsPersist").getOrCreate()

txns = spark.read.parquet("s3://nbi-data-lake/transactions/")
accounts = spark.read.parquet("s3://nbi-data-lake/accounts/")

cleaned = txns.filter(F.col("status") == "SETTLED") \
    .join(accounts, "accountId")

# These two are IDENTICAL:
cleaned.cache()                                    # Default: MEMORY_AND_DISK
cleaned.persist(StorageLevel.MEMORY_AND_DISK)       # Explicit same level

print(f"Storage level: {cleaned.storageLevel}")
TEXT
Storage level: Disk Memory Deserialized 1x Replicated

What are the default storage levels?

The default differs by API, which is a detail interviewers like because it shows you understand that cache() is not one single behavior.

| API | cache() default | Description |
|---|---|---|
| DataFrame / Dataset | MEMORY_AND_DISK | Memory preferred, spills to disk |
| RDD | MEMORY_ONLY | Memory only, drops overflow |

PYTHON
# DataFrame: MEMORY_AND_DISK
df = spark.read.parquet("s3://nbi-data-lake/transactions/")
df.cache()
print(f"DataFrame: {df.storageLevel}")

# RDD: MEMORY_ONLY
rdd = df.rdd
rdd.cache()
print(f"RDD: {rdd.getStorageLevel()}")
TEXT
DataFrame: Disk Memory Deserialized 1x Replicated
RDD: Memory Deserialized 1x Replicated

What storage levels are available?

Persist exposes the full set of choices. Each level is a combination of location, format, and replication.

| Storage Level | Memory | Disk | Serialized | Copies |
|---|---|---|---|---|
| MEMORY_ONLY | Yes | No | No | 1 |
| MEMORY_AND_DISK | Yes | Yes | No | 1 |
| MEMORY_ONLY_SER | Yes | No | Yes | 1 |
| MEMORY_AND_DISK_SER | Yes | Yes | Yes | 1 |
| DISK_ONLY | No | Yes | Yes | 1 |
| MEMORY_ONLY_2 | Yes | No | No | 2 |
| MEMORY_AND_DISK_2 | Yes | Yes | No | 2 |
| OFF_HEAP | Yes (off-heap) | No | Yes | 1 |

PYTHON
# Choose based on the dataset's characteristics

# Small, reused many times: fastest access
cleaned.persist(StorageLevel.MEMORY_ONLY)

# Large, does not fit in memory: safe fallback to disk
cleaned.persist(StorageLevel.MEMORY_AND_DISK)

# Very large, tight memory: pack tightly with serialization
cleaned.persist(StorageLevel.MEMORY_AND_DISK_SER)

# Critical, expensive recomputation: replicate across nodes
cleaned.persist(StorageLevel.MEMORY_AND_DISK_2)

# Large cache causing GC pauses: move outside JVM heap
cleaned.persist(StorageLevel.OFF_HEAP)

When should you use cache() vs persist()?

Use cache() when the default storage level is appropriate for your dataset. Use persist() when you need a specific level that differs from the default.

PYTHON
# CASE 1: Medium dataset, fits in memory, reused a few times
# cache() is fine (default MEMORY_AND_DISK for DataFrames)
branch_lookup = spark.read.parquet("s3://nbi-data-lake/branches/")
branch_lookup.cache()
branch_lookup.count()

# CASE 2: Very large dataset, does not fit in memory
# Use persist with serialized level to pack tightly
large_txns = spark.read.parquet("s3://nbi-data-lake/transactions/all-years/")
large_txns.persist(StorageLevel.MEMORY_AND_DISK_SER)
large_txns.count()

# CASE 3: Critical ML feature matrix, expensive to recompute
# Use replicated level so executor loss does not trigger recompute
feature_matrix = compute_features(large_txns)  # Takes 2 hours
feature_matrix.persist(StorageLevel.MEMORY_AND_DISK_2)
feature_matrix.count()

How does serialized storage save memory?

Serialized storage converts each partition into a compact byte array instead of keeping Java/Python objects in memory. This uses significantly less memory (often 2x to 5x less) but requires CPU to serialize on write and deserialize on read.

PYTHON
# Deserialized (default): fast access, more memory
# Each row is a full Java/Python object in memory
cleaned.persist(StorageLevel.MEMORY_AND_DISK)       # ~2.4 GB

# Serialized: slower access, much less memory
# Rows packed into compact byte arrays
cleaned.persist(StorageLevel.MEMORY_AND_DISK_SER)   # ~0.8 GB

# The trade-off: 3x less memory but CPU cost on every read

What does replication do?

Storage levels ending in _2 keep two copies of each partition on different executor nodes. If one executor dies, the second copy is used immediately without recomputing from lineage.

PYTHON
# Without replication: executor dies → recompute from lineage
cleaned.persist(StorageLevel.MEMORY_AND_DISK)

# With replication: executor dies → read from backup copy (instant)
cleaned.persist(StorageLevel.MEMORY_AND_DISK_2)

# Cost: 2x memory usage
# Benefit: no recompute for expensive datasets
# Use case: ML feature matrix that took 2 hours to compute

What is off-heap storage?

Off-heap storage keeps cached data outside the JVM heap, which means the garbage collector does not scan it. This reduces GC pauses for large caches.

PYTHON
# Enable off-heap memory
spark.conf.set("spark.memory.offHeap.enabled", "true")
spark.conf.set("spark.memory.offHeap.size", "10g")

# Persist off-heap
cleaned.persist(StorageLevel.OFF_HEAP)
cleaned.count()

# The GC no longer needs to scan this 2.4 GB cache,
# reducing GC pause times significantly.

What should a senior engineer know?

A senior engineer matches the storage level to the dataset rather than using the default blindly. They consider four questions: does it fit in memory, how expensive is recomputation, is GC a problem, and can I afford any recompute.

PYTHON
# Decision tree:
# 1. Fits in memory + cheap recompute → MEMORY_ONLY
# 2. May not fit + moderate recompute → MEMORY_AND_DISK (default)
# 3. Large + tight memory → MEMORY_AND_DISK_SER (serialized)
# 4. Large + GC pain → OFF_HEAP
# 5. Expensive + cannot afford recompute → _2 (replicated)
# 6. Huge + only disk feasible → DISK_ONLY

# Always verify in Storage tab that the level matches expectations

Visual Overview

DIAGRAM
  cache() vs persist() under the hood:

  cache()
    │
    └──→ persist(default_storage_level)
              │
              ├── DataFrame → MEMORY_AND_DISK
              └── RDD       → MEMORY_ONLY

  persist(StorageLevel)
    │
    └──→ persist(your_chosen_level)
              │
              ├── MEMORY_ONLY           Fast, drops overflow
              ├── MEMORY_AND_DISK       Safe default
              ├── MEMORY_AND_DISK_SER   Packed tight
              ├── DISK_ONLY             All on disk
              ├── MEMORY_ONLY_2         Replicated
              ├── MEMORY_AND_DISK_2     Replicated + disk
              └── OFF_HEAP              Outside GC
DIAGRAM
  Storage level trade-offs:

  Speed ◄──────────────────────────────► Memory savings
  MEMORY_ONLY    MEMORY_AND_DISK   MEMORY_AND_DISK_SER   DISK_ONLY
  (fastest)      (safe default)    (compact)             (most memory-efficient)

  GC pressure ◄────────────────────────► GC-free
  On-heap deserialized              OFF_HEAP

  No fault tolerance ◄─────────────────► Fault tolerant
  _1 (single copy)                  _2 (replicated)

Key Takeaway

The difference between cache() and persist() is about control over the storage level. Under the hood they are the same mechanism: cache() literally calls persist() with a default. Both are lazy, both store partitions for reuse, both release with unpersist(), and both recompute evicted partitions from lineage.

The default differs by API. For DataFrames, cache() uses MEMORY_AND_DISK, which stores partitions in memory and spills to disk when memory is full. For RDDs, it uses MEMORY_ONLY, which drops partitions that do not fit. persist() lets you choose the level explicitly: memory vs disk, serialized vs deserialized, off-heap, and replication.

At the National Bank of India I used cache() for most medium datasets that fit comfortably in memory. But for a very large cleaned transaction table, I used persist() with MEMORY_AND_DISK_SER to pack data tightly and spill gracefully. For critical ML feature matrices that took hours to compute, I used a replicated level so an executor failure would not trigger a costly recompute.

The principle is to match the storage level to the dataset's size, reuse pattern, and recomputation cost. Use cache() when the default is right, and persist() when you need a specific strategy.

Remember one sentence: cache() is persist() with a fixed default storage level; use persist() when you need explicit control over memory, disk, serialization, or replication.

---

Share
Use ← → arrow keys to navigate