Back to Apache Spark: Core
Q14Question 14 of 72

Caching in Spark and why cache() is lazy

Key Concepts
  • Caching stores computed partitions on executors so later actions skip recomputation of the entire lineage
  • cache() is lazy: it only marks the dataset for caching; actual computation and storage happen on the first action
  • The first action after caching is NOT faster because it does the full computation plus storage overhead
  • Benefits appear from the second action onward, which reads cached partitions instead of recomputing
  • cache() is shorthand for persist() with the default storage level (MEMORY_AND_DISK for DataFrames)
  • Cached partitions can be evicted under memory pressure and are recomputed transparently from lineage

Definition

Caching stores computed partitions on executors so later actions skip recomputation, and cache() is lazy because it only marks intent and the data is stored on the first action.
  • Purpose – keep computed partitions available on executors so subsequent actions reuse them instead of recomputing the entire lineage from source data.
  • Lazy – calling cache() only marks the dataset with a storage level; no computation or storage happens until an action triggers execution.
  • First action – computes the data AND stores it in the cache, so it is not faster; the speedup appears from the second action onward.
  • Default level – cache() uses MEMORY_AND_DISK for DataFrames and MEMORY_ONLY for RDDs.
  • Eviction – cached partitions can be evicted under memory pressure and are transparently recomputed from lineage.

Imagine a clerk at the National Bank of India spends an hour assembling a complicated summary of Rajesh's year of transactions. Without caching, every question about that summary makes him redo the whole hour of work. Caching is keeping the finished summary on his desk so the next ten questions are answered instantly. But saying "keep this handy" does not build the summary. It puts a sticky note saying "when you do build it, keep a copy." The summary is built and kept only when someone first asks a question that needs it.

Understanding the Concepts

What is caching in Spark?

Caching keeps the computed partitions of an RDD or DataFrame in executor memory (and optionally on disk) so that subsequent actions reuse them instead of recomputing the whole lineage from scratch. Without caching, every action triggers a full recomputation from the source data.

PYTHON
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("NBI-CachingDemo").getOrCreate()

# Expensive pipeline: read, filter, join, aggregate
txns = spark.read.parquet("s3://nbi-data-lake/transactions/")
accounts = spark.read.parquet("s3://nbi-data-lake/accounts/")

cleaned = txns \
    .filter(F.col("status") == "SETTLED") \
    .join(accounts, "accountId") \
    .select("accountId", "branch", "amount", "txnDate", "accountType")

# Without caching, every action below recomputes the entire pipeline

Why is cache() lazy?

Because Spark's fundamental rule is that computation only happens on an action. Calling cache() is a transformation, not an action. It marks the dataset with a storage level and tells the block manager "when these partitions are computed, keep them." No data is read and nothing is stored.

PYTHON
# cache() just marks intent. Nothing runs yet.
cleaned.cache()

# No data has been computed or stored at this point.
# The block manager has registered the intent, that is all.

# THIS action triggers computation + caching
print(f"Total rows: {cleaned.count()}")  # Full computation + store

# THIS action reads from cache (fast!)
print(f"Branch count: {cleaned.select('branch').distinct().count()}")
TEXT
Total rows: 48000000
Branch count: 156

When does caching help and when does it not?

Caching helps when a dataset is both expensive to produce and reused multiple times. It does not help for single-use datasets.

PYTHON
# GOOD: expensive dataset reused by multiple actions
cleaned.cache()

# Action 1: branch report
branch_report = cleaned.groupBy("branch").agg(F.sum("amount"))
branch_report.show()

# Action 2: account type breakdown
type_report = cleaned.groupBy("accountType").agg(F.count("*"))
type_report.show()

# Action 3: daily trend
daily_report = cleaned.groupBy("txnDate").agg(F.sum("amount"))
daily_report.show()

# Without caching: pipeline computed 3 times
# With caching: computed once, read from cache twice

How do you verify caching is working?

Use the Spark UI Storage tab to see cached datasets, the number of partitions cached, and the memory and disk used. A fully cached dataset shows 100% fraction cached.

PYTHON
# Force caching with an action
cleaned.cache()
cleaned.count()  # Triggers computation + caching

# Check the Spark UI at http://driver:4040/storage/
# You should see:
# - RDD Name: cleaned
# - Storage Level: Disk Memory Deserialized
# - Cached Partitions: 200 (or however many)
# - Fraction Cached: 100%
# - Size in Memory: 2.4 GB
# - Size on Disk: 0 B

# Programmatic check
print(f"Is cached: {cleaned.is_cached}")
print(f"Storage level: {cleaned.storageLevel}")
TEXT
Is cached: True
Storage level: Disk Memory Deserialized 1x Replicated

What is the default storage level for cache()?

The default depends on the API. DataFrames use MEMORY_AND_DISK, which stores partitions in memory when possible and spills to disk when memory is full. RDDs use MEMORY_ONLY, which drops partitions that do not fit.

PYTHON
# DataFrame cache: MEMORY_AND_DISK (safe default)
df = spark.read.parquet("s3://nbi-data-lake/transactions/")
df.cache()
print(f"DataFrame storage: {df.storageLevel}")

# RDD cache: MEMORY_ONLY (drops overflow)
rdd = df.rdd
rdd.cache()
print(f"RDD storage: {rdd.getStorageLevel()}")
TEXT
DataFrame storage: Disk Memory Deserialized 1x Replicated
RDD storage: Memory Deserialized 1x Replicated

What happens when cached data is evicted?

If executor memory runs low, Spark can evict cached partitions. With MEMORY_AND_DISK, evicted partitions spill to disk. With MEMORY_ONLY, they are simply dropped and must be recomputed from lineage the next time they are needed.

PYTHON
# MEMORY_AND_DISK: evicted partitions go to disk (safe)
from pyspark import StorageLevel
cleaned.persist(StorageLevel.MEMORY_AND_DISK)

# MEMORY_ONLY: evicted partitions are LOST, recomputed from lineage
cleaned.persist(StorageLevel.MEMORY_ONLY)

# Check the Storage tab for partially cached datasets.
# If fraction cached is below 100%, partitions were evicted.

How do you release a cache?

Call unpersist() to remove cached partitions and free executor memory. This is important in long-running applications where cached data accumulates.

PYTHON
# Release the cache when you are done with the dataset
cleaned.unpersist()
print(f"Is cached after unpersist: {cleaned.is_cached}")

# In a long pipeline, unpersist each phase's cache before the next
cleaned.cache()
cleaned.count()  # Compute + cache
# ... use cleaned for reports ...
cleaned.unpersist()  # Free memory before next phase
TEXT
Is cached after unpersist: False

What are common mistakes with caching?

The most common mistake is expecting cache() to run eagerly. The second is caching everything, which fills executor memory and causes eviction.

PYTHON
# WRONG: expecting cache to compute immediately
cleaned.cache()
# Developer thinks data is cached, but no action has run!
# Nothing is in memory yet.

# WRONG: caching everything
txns.cache()      # 500M rows
accounts.cache()  # 10M rows
branches.cache()  # 200 rows
products.cache()  # 50 rows
# Over-caching fills memory and causes useful caches to be evicted!

# RIGHT: cache only the expensive, reused dataset
cleaned.cache()  # Expensive join result used by 5 reports
cleaned.count()  # Force materialization

What should a senior engineer know?

A senior engineer knows that caching is a performance optimization, not durable storage. Cached data can be evicted at any time, and the lineage ensures transparent recovery. They also know that cache() is shorthand for persist() with a default storage level, and they choose specific storage levels (serialized, replicated, off-heap) based on the dataset's characteristics.

PYTHON
# Senior practice: cache deliberately, measure in Storage tab
cleaned.cache()
cleaned.count()  # Materialize

# Verify: Spark UI → Storage tab
# If fraction cached < 100%, the data does not fit.
# Switch to a serialized level to pack more data:
from pyspark import StorageLevel
cleaned.unpersist()
cleaned.persist(StorageLevel.MEMORY_AND_DISK_SER)
cleaned.count()

Visual Overview

DIAGRAM
  Without caching (3 actions = 3 full computations):

  Action 1: read → filter → join → select → groupBy → show
  Action 2: read → filter → join → select → groupBy → show
  Action 3: read → filter → join → select → groupBy → show
  Total: 3x full pipeline


  With caching (1 computation + 2 cache reads):

  cache() ← marks intent only, nothing happens

  Action 1: read → filter → join → select → STORE in cache → groupBy → show
  Action 2: READ from cache → groupBy → show   (fast!)
  Action 3: READ from cache → groupBy → show   (fast!)
  Total: 1x full pipeline + 2x cache reads
DIAGRAM
  Executor memory with caching:

  ┌─────────────────────────────────┐
  │          Executor Memory         │
  │                                   │
  │  ┌───────────────────────────┐   │
  │  │  Storage Memory (cache)   │   │
  │  │  cleaned: 200 partitions  │   │
  │  │  2.4 GB in memory         │   │
  │  └───────────────────────────┘   │
  │                                   │
  │  ┌───────────────────────────┐   │
  │  │  Execution Memory         │   │
  │  │  (shuffles, sorts, joins) │   │
  │  └───────────────────────────┘   │
  └─────────────────────────────────┘

Key Takeaway

Caching in Spark means keeping computed partitions on executors so subsequent actions reuse them instead of recomputing the entire lineage from scratch. Without caching, every action triggers a full recomputation from the source data, which is wasteful when an expensive pipeline feeds multiple reports.

The reason cache() is lazy is that it follows Spark's fundamental rule: computation only happens on an action. Calling cache() just marks the dataset with a storage level and registers intent with the block manager. The partitions are actually computed and stored on the first action. A direct consequence is that the first action after caching is not faster: it does the full computation plus the storage overhead. The benefit appears from the second action onward.

At the National Bank of India, we cached a cleaned, joined transaction dataset that fed five different daily reports. Without caching, Spark recomputed that expensive pipeline five times. With caching, it computed once and read from the cache four more times, cutting the total runtime dramatically.

The nuances to remember are that cache() is shorthand for persist() with a default storage level (MEMORY_AND_DISK for DataFrames), that cached data can be evicted under memory pressure, and that you should call unpersist() in long-running applications to free memory when a cache is no longer needed.

Remember one sentence: cache() is lazy and only marks intent; the data is computed and stored on the first action, and every subsequent action benefits by reading from the cache instead of recomputing.

---

Share
Use ← → arrow keys to navigate