Definition
- Caching – stores partitions on executors (memory/disk) for fast reuse; lineage is preserved so evicted partitions are recomputed from parents.
- Checkpointing – writes data to reliable, replicated storage (HDFS, S3) and truncates lineage, making the checkpoint the new starting point.
- Key difference – caching keeps lineage (fast but fragile); checkpointing cuts lineage (durable but expensive to write).
- Use cases – cache for reuse optimization; checkpoint for long lineage chains (iterative ML) and fault recovery.
Both are ways a clerk at the National Bank of India keeps a finished summary. Caching is keeping it on his desk: fast to grab, but if the desk is cleared he still has the recipe to rebuild it. Checkpointing is taking the summary to the bank's fireproof vault and tearing up the long recipe, because the vault copy is the new starting point. Caching is fast and keeps the recipe; checkpointing is durable and throws it away.
Understanding the Concepts
What is the difference between caching and checkpointing?
Both materialize a dataset to avoid recomputation, but they differ in purpose, storage location, and how they treat lineage. Caching is a performance optimization that stores data on executors. Checkpointing is a durability mechanism that writes data to reliable storage and cuts the lineage.
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("NBI-CacheVsCheckpoint").getOrCreate()
sc = spark.sparkContext
sc.setCheckpointDir("s3://nbi-data-lake/checkpoints/")
txns = spark.read.parquet("s3://nbi-data-lake/transactions/")
cleaned = txns.filter(F.col("status") == "SETTLED")
How does caching preserve lineage?
When you cache a dataset, Spark still remembers the full chain of transformations. If a cached partition is lost (executor dies, memory eviction), Spark transparently recomputes it from the lineage. Caching is fast but not durable.
# Caching: lineage preserved
cleaned.cache()
cleaned.count()
# If Executor 3 dies, cached partitions on that executor are lost.
# Spark replays: read parquet → filter(SETTLED) → recompute
# Only the lost partitions are recomputed, not the whole dataset.
# The lineage is still intact:
cleaned.explain("extended")
== Optimized Logical Plan ==
Filter (status = SETTLED)
+- Relation [accountId,amount,status,...] parquet
How does checkpointing truncate lineage?
Checkpointing writes the dataset to reliable storage (HDFS, S3) and then discards the lineage. The checkpoint files become the new source. Recovery starts from the checkpoint, not from the original transformations.
# Checkpointing: lineage truncated
cleaned_cp = cleaned.checkpoint() # Writes to s3://nbi-data-lake/checkpoints/
# The lineage is now GONE. The checkpoint files are the source.
cleaned_cp.explain("extended")
== Optimized Logical Plan ==
LogicalRDD [accountId,amount,status,...], false
Why does lineage truncation matter?
In iterative algorithms, each iteration adds to the lineage. After many iterations, the chain becomes enormous. Planning over this giant lineage becomes slow and memory-heavy, sometimes causing stack overflows in the scheduler. Checkpointing periodically cuts the chain.
# Iterative risk model: 100 iterations
rdd = sc.parallelize(range(1000000), 100)
for i in range(100):
rdd = rdd.map(lambda x: x + 1) # Each iteration adds to lineage
if i % 10 == 0:
# Without checkpoint: lineage has 100 map operations
# With checkpoint: lineage resets every 10 iterations
rdd.checkpoint()
rdd.count() # Force materialization
# Without checkpointing at iteration 100:
# Lineage: map → map → map → ... → map (100 steps)
# Failure at iteration 99 replays ALL 100 steps.
# With checkpointing every 10:
# Lineage: checkpoint → map → ... → map (at most 10 steps)
# Failure at iteration 99 replays at most 10 steps.
How do caching and checkpointing compare?
| Aspect | Caching | Checkpointing |
|---|---|---|
| Purpose | Speed (reuse optimization) | Durability + lineage management |
| Storage | Executor memory/disk | Reliable storage (HDFS, S3) |
| Lineage | Preserved | Truncated |
| Lost data | Recomputed from lineage | Read from checkpoint |
| Write cost | Cheap (local executor) | Expensive (reliable storage) |
| Read speed | Very fast (local memory) | Slower (network/disk) |
| Use case | Reusing data across actions | Iterative algorithms, long lineage |
# Caching: fast reads, lineage intact
cleaned.cache() # Stored on executors
# Checkpointing: durable, lineage cut
cleaned.checkpoint() # Written to HDFS/S3
What is the cache-then-checkpoint pattern?
Because checkpointing triggers a computation to write data, a common pattern is to cache first and then checkpoint. The checkpoint write reads from the cache (fast) instead of recomputing the entire lineage. After checkpointing, the cached copy still serves fast reuse.
# Cache-then-checkpoint: best of both worlds
expensive = compute_risk_features(txns) # Very expensive computation
# Step 1: Cache (so checkpoint reads from cache, not recomputing)
expensive.cache()
expensive.count() # Materialize cache
# Step 2: Checkpoint (writes from cache to reliable storage, truncates lineage)
checkpointed = expensive.checkpoint()
# Step 3: Future reads use the cache (fast) or checkpoint (durable)
# Step 4: After reuse, release the cache
expensive.unpersist()
# Without caching first: checkpoint triggers a FULL recomputation
# With caching first: checkpoint reads from the already-computed cache
What is local checkpointing?
Local checkpointing truncates lineage by writing to local executor disk instead of reliable storage. It is faster but not fault-tolerant: if the executor dies, the local checkpoint is lost.
# Reliable checkpoint: writes to HDFS/S3, survives node failure
sc.setCheckpointDir("s3://nbi-data-lake/checkpoints/")
reliable = rdd.checkpoint()
# Local checkpoint: writes to executor local disk, faster but fragile
local = rdd.localCheckpoint()
# Use local checkpoint when:
# - You only need lineage truncation (not fault tolerance)
# - Speed matters more than durability
# - The lineage can be rebuilt if the executor dies
When should you use each?
The decision rule is simple: cache for speed when a dataset is reused, and checkpoint when the lineage is dangerously long or when the dataset must survive failures durably.
# Decision tree:
# 1. Reused multiple times, lineage is short → cache()
# 2. Reused multiple times, lineage is long → cache() + checkpoint()
# 3. Iterative algorithm (100+ iterations) → checkpoint every N iterations
# 4. Must survive cluster failure → checkpoint to reliable storage
# 5. Only need lineage truncation → localCheckpoint()
# At the bank:
cleaned.cache() # Short lineage, reused by 5 reports
# Risk model: 100 iterations, checkpoint every 10
for i in range(100):
model_rdd = model_rdd.map(update_weights)
if i % 10 == 0:
model_rdd.checkpoint()
model_rdd.count()
What should a senior engineer know?
A senior engineer uses caching and checkpointing together for long iterative jobs. They also know that DataFrame checkpointing is eager (triggers computation immediately), while RDD checkpointing is lazy (needs an action to materialize). They cache before checkpointing to avoid redundant computation.
# DataFrame checkpoint is EAGER:
cleaned_df = cleaned.checkpoint() # Runs immediately!
# RDD checkpoint is LAZY:
rdd.checkpoint() # Marks intent
rdd.count() # Must trigger manually
# Senior pattern for iterative Spark jobs:
# cache + checkpoint every N iterations
# unpersist the previous iteration's cache
Visual Overview
Caching (lineage preserved):
source → filter → join → select → [CACHE on executors]
│
┌──────────┤
▼ ▼
report1 report2
If executor dies: replay source → filter → join → select
(lineage still exists)
Checkpointing (lineage truncated):
source → filter → join → select → [WRITE to HDFS/S3]
│
│ lineage cut!
▼
checkpoint files (new source)
│
┌──────────┤
▼ ▼
report1 report2
If executor dies: read from checkpoint files
(no lineage to replay)
Lineage growth in iterative algorithms: Without checkpoint: iter1 → iter2 → iter3 → ... → iter100 (100-step lineage!) With checkpoint every 10: iter1-10 → [checkpoint] → iter11-20 → [checkpoint] → ... At any point, lineage is at most 10 steps.
Key Takeaway
Caching and checkpointing both materialize a dataset to avoid recomputation, but they serve different purposes and treat lineage differently. Caching stores data on executors for fast reuse and preserves the lineage, so if a partition is lost, it is recomputed from the chain of transformations. Checkpointing writes data to reliable storage like HDFS or S3 and truncates the lineage, making the checkpoint files the new starting point.
The key difference is that caching keeps the lineage while checkpointing cuts it. This matters most in iterative algorithms where each iteration extends the lineage chain. After many iterations, the chain becomes enormous and can cause stack overflows in the scheduler. Checkpointing periodically cuts the chain so it cannot grow unboundedly.
The common pattern is cache-then-checkpoint: cache the dataset first so the checkpoint write reads from the cache instead of recomputing the entire lineage, and the cached copy still serves fast reuse afterward. At the National Bank of India we cached shared reporting datasets for speed and checkpointed long iterative risk-model computations to HDFS so a failure did not replay hundreds of iterations.
The decision rule is simple: cache for speed when data is reused, and checkpoint when lineage is long or the data must survive failures durably.
Remember one sentence: caching preserves lineage for fast reuse on executors, while checkpointing truncates lineage by writing to reliable storage for durable fault recovery.
---