Definition
- Cache when – a dataset is expensive to compute AND reused by multiple actions, so the one-time storage cost is repaid by avoiding repeated recomputation.
- Classic cases – iterative ML that loops over the same data, one cleaned dataset feeding several reports, interactive exploration with many queries.
- Do NOT cache – single-use datasets (no reuse), tiny datasets (recompute is near-free), or everything indiscriminately (causes memory pressure and eviction).
- Unpersist – release caches when their reuse phase is done to keep memory available for later stages.
Think of a clerk at the National Bank of India deciding what to keep on his desk. He should keep the hard-won summary that ten reports all need, because building it once and reusing it saves hours. He should not clutter his desk with a figure he uses only once, or with a trivial total he can redo in a second, or with so many papers that the desk overflows and important ones fall off.
Understanding the Concepts
When should you cache?
Cache when a dataset meets two criteria: it is expensive to compute AND it is reused by multiple actions. Caching has a cost: it computes and stores data on the first action and consumes executor memory thereafter. That cost is only recovered if later actions read from the cache instead of recomputing.
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("NBI-CacheDecision").getOrCreate()
# EXPENSIVE pipeline: read + filter + join + derive columns
txns = spark.read.parquet("s3://nbi-data-lake/transactions/")
accounts = spark.read.parquet("s3://nbi-data-lake/accounts/")
cleaned = txns \
.filter(F.col("status") == "SETTLED") \
.join(accounts, "accountId") \
.withColumn("fee", F.col("amount") * F.lit(0.001))
# REUSED by 5 different reports → CACHE!
cleaned.cache()
cleaned.count() # Materialize the cache
What are the classic good cases for caching?
Three patterns almost always benefit from caching.
# CASE 1: Iterative ML training (same data, many passes)
from pyspark.ml.feature import VectorAssembler
features = VectorAssembler(inputCols=["amount", "fee"], outputCol="features") \
.transform(cleaned)
features.cache() # Used by every training iteration
features.count()
# Training loop reads features 100+ times
# Without cache: 100 full recomputations
# With cache: 1 computation + 99 cache reads
# CASE 2: One dataset feeding multiple downstream branches
cleaned.cache()
cleaned.count()
branch_report = cleaned.groupBy("branch").agg(F.sum("amount"))
type_report = cleaned.groupBy("accountType").agg(F.count("*"))
daily_report = cleaned.groupBy("txnDate").agg(F.sum("amount"))
fee_report = cleaned.groupBy("branch").agg(F.sum("fee"))
risk_report = cleaned.filter(F.col("amount") > 100000).count()
# 5 reports, 1 computation instead of 5
# CASE 3: Interactive exploration (many queries on same base)
# In a notebook, cache the base and run many ad-hoc queries:
cleaned.cache()
cleaned.count()
# Query 1: cleaned.filter(...).show()
# Query 2: cleaned.groupBy(...).agg(...).show()
# Query 3: cleaned.describe().show()
When should you NOT cache?
Three situations where caching wastes resources or hurts performance.
# BAD CASE 1: Dataset used only once (no reuse, no benefit)
single_use = txns.filter(F.col("branch") == "Mumbai") \
.groupBy("accountId").agg(F.sum("amount"))
# single_use.cache() ← WRONG: no second read to benefit from
single_use.write.parquet("s3://nbi-data-lake/mumbai-totals/")
# Cache just wasted memory for a dataset used once
# BAD CASE 2: Tiny dataset that is trivially cheap to recompute
branches = spark.read.parquet("s3://nbi-data-lake/branches/")
# branches.cache() ← WASTEFUL: 200 rows, recompute takes milliseconds
# Caching costs more (memory management) than it saves
# BAD CASE 3: Caching everything (over-caching)
txns.cache() # 500M rows, 4 GB
accounts.cache() # 10M rows, 800 MB
cleaned.cache() # 500M rows, 3 GB
features.cache() # 500M rows, 2 GB
# Total: ~10 GB in cache! Executors have 8 GB storage.
# Result: eviction, where useful caches are pushed out to
# make room, and MEMORY_ONLY drops them for recompute.
How does over-caching hurt performance?
Executor memory is finite and shared between storage (caching) and execution (shuffles, joins). When too much is cached, useful partitions get evicted. With MEMORY_ONLY, evicted partitions are dropped and recomputed. The job can become SLOWER than with no caching at all.
# Scenario: 8 GB executor storage memory, 10 GB of cached data
# Spark starts evicting older cached partitions to make room.
# If the evicted partitions are needed again → recompute.
# If execution memory is squeezed → shuffle spills to disk.
# WRONG: cache everything
txns.cache()
accounts.cache()
cleaned.cache()
# RIGHT: cache only the reused, expensive dataset
cleaned.cache() # Only this one is reused by multiple reports
cleaned.count()
How do you decide what to cache in a pipeline?
Look for the point where an expensive lineage branches into multiple consumers. Cache at that branch point.
# Pipeline structure:
#
# source → filter → join → cleaned → report1
# → report2
# → report3
# → write to lake
#
# 'cleaned' is the branch point. Cache it.
cleaned.cache()
cleaned.count()
# All four consumers read from cache instead of recomputing
report1 = cleaned.groupBy("branch").agg(F.sum("amount"))
report2 = cleaned.groupBy("accountType").agg(F.count("*"))
report3 = cleaned.filter(F.col("amount") > 50000)
cleaned.write.parquet("s3://nbi-data-lake/cleaned/")
# After all consumers are done:
cleaned.unpersist()
Why is unpersist important?
In long-running applications, cached data lingers until explicitly released or evicted. Calling unpersist() frees memory for later stages.
# Multi-phase pipeline: unpersist between phases
# Phase 1: daily settlement
cleaned.cache()
cleaned.count()
# ... generate 5 settlement reports ...
cleaned.unpersist() # Free memory
# Phase 2: risk analysis (needs memory for different cache)
risk_data = compute_risk_features(txns)
risk_data.cache()
risk_data.count()
# ... run risk reports ...
risk_data.unpersist()
# Without unpersist: phase 1's cache starves phase 2
How do you monitor caching effectiveness?
Use the Spark UI Storage tab to verify caches are fully materialized and actually reused.
# After caching and running actions, check:
# Spark UI → Storage tab
# - Fraction Cached: should be 100%
# - Size in Memory / Size on Disk
# - If fraction < 100%: data does not fit, change level or free caches
# Spark UI → Stages tab
# - Tasks reading cached data show fast completion
# - Tasks recomputing show slow completion
cleaned.cache()
cleaned.count()
print(f"Is cached: {cleaned.is_cached}")
print(f"Storage level: {cleaned.storageLevel}")
Is cached: True
Storage level: Disk Memory Deserialized 1x Replicated
What should a senior engineer know?
A senior engineer caches deliberately at branch points, unpersists when reuse ends, and verifies in the Storage tab. They never cache indiscriminately and they always measure whether caching actually helped.
# Senior caching checklist:
# 1. Is the dataset expensive to compute? (joins, aggregations)
# 2. Is it reused by multiple actions? (>1 consumer)
# 3. Does it fit in available storage memory?
# 4. Am I unpersisting when reuse ends?
# 5. Am I checking the Storage tab for 100% cached?
# If any answer is NO, do not cache.
Visual Overview
Caching decision flowchart:
Is the dataset expensive to compute?
├── NO → Do NOT cache (recomputation is cheap)
└── YES → Is it reused by multiple actions?
├── NO → Do NOT cache (single use)
└── YES → Does it fit in storage memory?
├── NO → Use serialized level or free other caches
└── YES → CACHE it!
│
├── Materialize with an action
├── Use in multiple downstream actions
└── Unpersist when reuse is done
Pipeline branch point (cache here):
source → filter → join → select
│
├──→ cache() ←── branch point
│
┌────────┼────────┬────────┐
▼ ▼ ▼ ▼
Report1 Report2 Report3 Write
(cache) (cache) (cache) (cache)
Key Takeaway
The decision to cache rests on two factors: how expensive the dataset is to compute, and how many times it is reused. Caching pays off only when both are high, because caching itself has a cost: it computes and stores data on the first action and consumes executor memory. When a dataset is both expensive and reused many times, the first action pays the full cost and every later action reads for free.
The classic good cases are iterative ML that loops over the same data many times, a cleaned dataset feeding several downstream reports, and interactive exploration where an analyst runs many queries against the same base. Knowing when NOT to cache is equally important: single-use datasets get no benefit, tiny datasets are trivially cheap to recompute, and caching everything fills memory and causes eviction that can make the job slower than no caching.
At the National Bank of India I cached the shared cleaned transaction dataset that fed five daily reports. Building it once instead of five times was a big win. But I never cached single-pass extracts or tiny reference tables, and I always called unpersist() when a phase's cache was no longer needed.
The discipline is to cache at the point where an expensive lineage branches into multiple consumers, unpersist when reuse ends, and verify in the Storage tab that caches are fully materialized and genuinely hit.
Remember one sentence: cache when a dataset is both expensive and reused multiple times, never cache single-use or trivially cheap data, and always unpersist when reuse is done.
---