Back to Advanced Data Engineering
Q2Question 2 of 24

Common caching mistakes

Key Concepts
  • Caching everything fills executor storage and forces eviction, often making jobs slower than no caching at all.
  • cache() is lazy; it marks the dataset but materializes only when an action triggers computation.
  • Caching data used only once wastes memory for no reuse benefit.
  • Forgetting unpersist() in long-running applications accumulates dead caches until storage is exhausted.
  • collect() pulls all data to the Driver and can crash it; cache() keeps partitions distributed on executors.
  • Caching does not eliminate a shuffle; it only avoids recomputing the lineage (including the shuffle) on subsequent actions.

Definition

Do not cache everything, do not assume cache is eager, do not cache single-use data, do not forget unpersist, do not confuse cache with collect, and do not expect cache to remove a shuffle.
  • Caching everything – executor storage is finite, so indiscriminate caching fills it and forces eviction, often making the job slower.
  • Assuming cache is eager – cache() is lazy; it only marks the dataset and does nothing until an action triggers computation.
  • Caching single-use data – with no second read, there is nothing to accelerate, so the storage cost is pure waste.
  • Forgetting unpersist – caches live until evicted or released, so long-running apps accumulate data until storage is exhausted.
  • Confusing cache with collect – cache() keeps data distributed; collect() pulls everything to the Driver and can cause OOM.
  • Thinking cache removes shuffle – caching a post-shuffle result avoids re-shuffling on later actions, but the first computation still pays the shuffle.

These are the predictable mistakes a clerk at the National Bank of India makes with his desk. Keeping every scrap of paper so the desk overflows and important ones get lost. Saying "keep this handy" and assuming it is done when it only happens the first time he actually looks for it. Keeping a figure he will use once. Never clearing old summaries during a long shift until the desk is buried. Confusing "keep a copy on your desk" with "bring every original file to my office," which crushes the office. And believing that keeping a sorted pile saves him from the sorting he already did – it does not undo the sort, it just saves re-sorting next time.

Understanding the Concepts

Why is caching everything a mistake?

Executor storage is finite. Caching indiscriminately fills it and forces eviction, where useful partitions are pushed out and, under MEMORY_ONLY, dropped and recomputed.

PYTHON
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("CacheMistakes").getOrCreate()

txn = spark.read.parquet("/lake/silver/transactions")
accounts = spark.read.parquet("/lake/silver/accounts")
branches = spark.read.parquet("/lake/dim/branches")

# Wrong: cache everything "just in case"
txn.cache()
accounts.cache()
branches.cache()

# Right: cache only expensive, reused datasets
# branches is small and reused in multiple joins – cache it
branches.cache()
branches.count()  # materialize

The result of caching everything is often a job that is slower than no caching at all, plus heavier garbage collection.

Why is assuming cache is eager a mistake?

Cache is lazy. Calling cache() or persist() only marks the dataset. The data is not computed or stored until an action triggers it.

PYTHON
txn_cached = txn.filter(F.col("amount") > 10000).cache()

# At this point, nothing has happened – no data is cached
# The first action materializes the cache
txn_cached.count()  # THIS is when the cache is built

# Now the second action benefits from the cache
txn_cached.groupBy("account_id").agg(F.sum("amount")).show()

People sometimes call cache() and expect the next action to be instant, but that first action is the one that actually computes and stores the data.

Why is caching single-use data a waste?

With no second read, there is nothing to accelerate. You pay the memory cost for zero benefit.

PYTHON
# Wrong: cache data used only once
one_time = txn.filter(F.col("branch_code") == "NBI-MUM-001").cache()
one_time.write.parquet("/lake/gold/mumbai_report")
# Cache was stored but never read again – pure waste

# Right: do not cache single-use datasets
one_time = txn.filter(F.col("branch_code") == "NBI-MUM-001")
one_time.write.parquet("/lake/gold/mumbai_report")

Cache only when a dataset is expensive to compute and reused more than once.

Why is forgetting unpersist dangerous?

Caches live until evicted or explicitly released. In long-running applications or streaming jobs, caching in each phase without unpersisting accumulates data until storage is exhausted.

PYTHON
# Long pipeline with multiple phases
phase1 = txn.groupBy("account_id").agg(F.sum("amount")).cache()
phase1.count()

# Phase 2 uses phase1, then builds phase2
phase2 = phase1.join(accounts, "account_id").cache()
phase2.count()

# Wrong: phase1 is still cached even though it is no longer needed
# Right: release it as soon as its reuse is finished
phase1.unpersist()

The rule is to call unpersist() as soon as the cached dataset's reuse is over.

Why is confusing cache with collect dangerous?

These are completely different operations.

PYTHON
# cache() – stores partitions distributed across executors
# Data stays on the cluster, safe at any size
txn.cache()

# collect() – pulls ALL rows back to the Driver as a local list
# On a large dataset, this crashes the Driver with OOM
all_rows = txn.collect()  # DANGEROUS on large data
DIAGRAM
# cache(): partitions stay on executors
  Executor 1: [partition 0, partition 1]
  Executor 2: [partition 2, partition 3]

# collect(): everything funnels to the Driver
  Driver: [ALL rows from ALL partitions]  → OOM

Reaching for collect() when you meant cache() is a serious error that can crash your application.

Why does cache not eliminate a shuffle?

A shuffle happens when a wide operation redistributes data. Caching the result does not undo the shuffle that already occurred – it only avoids recomputing the whole lineage on subsequent actions.

PYTHON
# The groupBy triggers a shuffle
spend = txn.groupBy("account_id").agg(
    F.sum("amount").alias("total_spend")
).cache()

# First action: computes AND shuffles AND caches
spend.count()

# Second action: reads from cache, no recomputation, no re-shuffle
spend.filter(F.col("total_spend") > 100000).show()

# But the first count() still paid the full shuffle cost
# Caching never removes a shuffle from a single computation

What is the correct caching strategy?

Cache deliberately: only datasets that are expensive to compute and reused more than once.

PYTHON
# Good pattern: expensive join result reused in multiple downstream queries
enriched = txn.join(accounts, "account_id").join(branches, "branch_code")
enriched.cache()
enriched.count()  # materialize the cache

# Reuse 1: fraud report
fraud = enriched.filter(F.col("amount") > 500000)
fraud.write.parquet("/lake/gold/fraud_candidates")

# Reuse 2: branch summary
branch_totals = enriched.groupBy("branch_code").agg(
    F.sum("amount").alias("branch_total")
)
branch_totals.write.parquet("/lake/gold/branch_summary")

# Done with enriched – release it
enriched.unpersist()

What should a senior engineer know?

Every caching mistake comes from misunderstanding what caching is: a lazy, distributed, best-effort reuse optimization that preserves lineage.

  • It is not eager – materialize with a cheap action like count().
  • It is not durable – a lost cached partition is recomputed from lineage.
  • It is not collection – data stays distributed on executors.
  • It is not shuffle removal – it avoids re-shuffling on later actions, not the first time.
PYTHON
# Checklist before caching:
# 1. Is this dataset reused more than once?  → yes → cache
# 2. Is it expensive to recompute?            → yes → cache
# 3. Will I unpersist when done?              → yes → proceed
# 4. Does it fit in executor memory?          → yes → MEMORY_ONLY
#                                             → no  → MEMORY_AND_DISK_SER

What are common interview follow-up questions?

  • Does caching guarantee the data stays in memory? No, under MEMORY_ONLY Spark drops partitions that do not fit; use MEMORY_AND_DISK to spill to local disk.
  • Is there a way to force eager materialization? Call a cheap action like count() right after cache().
  • What happens if you call cache() twice on the same DataFrame? The second call is a no-op if the first has already been materialized.

Visual Overview

DIAGRAM
  Caching Decision Flowchart
  ──────────────────────────

  Is this dataset reused more than once?
       │
       ├── No  → Do NOT cache
       │
       └── Yes
            │
            Is it expensive to recompute?
            │
            ├── No  → Do NOT cache (cheap to recompute)
            │
            └── Yes
                 │
                 Does it fit in memory?
                 │
                 ├── Yes → persist(MEMORY_ONLY)
                 │
                 └── No  → persist(MEMORY_AND_DISK_SER)
                          │
                          Always unpersist() when done

  cache() vs collect()
  ────────────────────
  cache():   [Exec1: P0,P1]  [Exec2: P2,P3]  → distributed
  collect(): [Driver: ALL ROWS]               → OOM risk

Key Takeaway

The common caching mistakes all come from misunderstanding what caching actually is. Caching everything fills executor storage and forces eviction, often making the job slower than no caching. Assuming cache is eager leads to surprise when the first post-cache action is slow, because that action is the one that actually computes and stores the data. Caching data used only once wastes memory for zero benefit.

Forgetting unpersist() in long-running applications lets caches accumulate until storage is exhausted. Confusing cache() with collect() is dangerous because cache keeps data distributed on executors while collect pulls everything to the single Driver and can crash it with an out-of-memory error. And believing cache eliminates shuffles is wrong: caching a post-shuffle result avoids recomputing that lineage on later actions, but the first computation still pays the full shuffle cost.

At the National Bank of India these were exactly the issues most often corrected in code reviews: engineers caching single-use extracts, forgetting unpersist in the long nightly pipelines, and expecting the first post-cache action to be instant. The fix was establishing a simple checklist: cache only if the dataset is expensive to compute and reused more than once, materialize with a cheap action, unpersist when done, and never confuse cache with collect.

Remember one sentence: caching is a lazy, distributed, best-effort reuse optimization – not eager, not durable, not a way to pull data to the Driver, and not a shuffle remover.

---

Share
Use ← → arrow keys to navigate