What "Stage Skipped" means in the Spark UI

• Skipped stage – a stage whose output the DAG scheduler found already available, so it reused the result and marked the stage skipped with zero tasks re-executed.
• Shuffle output reuse – when a stage produces shuffle files, those files persist on disk; if a later job needs the same shuffle output, Spark reuses it.
• Cache reuse – when a DataFrame is cached or persisted, subsequent actions read from the cache and skip the stages that would have recomputed it.
• Checkpoint reuse – checkpointing saves results and truncates lineage, so downstream work reads the checkpointed data.
• Good sign – skipped stages mean Spark avoided redundant computation; unexpected recomputation often means missing cache or evicted blocks.

What does "Stage Skipped" mean in the Spark UI?

When you see a stage marked as "skipped" in the Spark UI, it means Spark's DAG scheduler determined that the stage's output was already available and did not need to be recomputed. No tasks were launched for that stage.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum as spark_sum

spark = SparkSession.builder.appName("SkippedStageDemo").getOrCreate()

# Read transaction data at the National Bank of India
txns = spark.read.parquet("/data/transactions")

# Cache the aggregated result
account_totals = txns.groupBy("account_id") \
    .agg(spark_sum("amount").alias("total_amount"))
account_totals.cache()

# Action 1: materializes the cache
count = account_totals.count()
print(f"Total accounts: {count}")

# Action 2: reuses the cache -- earlier stages are SKIPPED
account_totals.show(5)
# In the Spark UI, the groupBy stage shows as "Skipped" for this job

Why does this happen with shuffle output?

Spark breaks jobs into stages at shuffle boundaries. When a stage produces shuffle output, that output is written to disk by the map tasks. If a later job needs the same shuffle output because it shares a common ancestor in the DAG, Spark detects the existing files and skips the stage.

# Two actions that share the same shuffle stage
result = txns.groupBy("account_id").agg(spark_sum("amount"))

# Action 1: triggers the groupBy shuffle
result.count()
# Spark UI: Job 0 -> Stage 0 (scan) + Stage 1 (groupBy shuffle)

# Action 2: shuffle output from Stage 1 still exists on disk
result.show(5)
# Spark UI: Job 1 -> Stage 0 (SKIPPED) + Stage 1 (SKIPPED)
# Both stages skipped because their output is already available!

Why does caching cause stage skipping?

When you cache a DataFrame and run the first action, the cached blocks are materialized in executor memory. Every subsequent action reads from the cache instead of recomputing, so the stages that produced the cached data are skipped.

# Demonstrate cache-driven stage skipping
expensive = txns.filter(col("amount") > 10000) \
    .groupBy("account_id", "txn_type") \
    .agg(spark_sum("amount").alias("total"))

expensive.cache()

# Action 1: full computation, populates the cache
expensive.count()  # All stages run

# Action 2: reads from cache, stages skipped
expensive.filter(col("total") > 100000).count()  # Shared stages SKIPPED

# Action 3: also reads from cache
expensive.orderBy(col("total").desc()).show(10)  # Shared stages SKIPPED

# Clean up
expensive.unpersist()

How does checkpointing cause stage skipping?

Checkpointing saves the DataFrame's result to reliable storage and truncates its lineage. Subsequent jobs start from the checkpointed data and skip all earlier stages.

spark.sparkContext.setCheckpointDir("/checkpoints")

# Checkpoint truncates lineage
checkpointed = txns.groupBy("account_id") \
    .agg(spark_sum("amount").alias("total"))
checkpointed.checkpoint()

# Subsequent actions read from checkpoint, skip all prior stages
checkpointed.filter(col("total") > 50000).count()
# Spark UI: scan and groupBy stages are SKIPPED

How do you read skipped stages in the Spark UI?

In the Spark UI Jobs tab, a skipped stage appears with its task count showing zero or a note saying "skipped." The stage is listed for transparency but no work was done.

Spark UI -> Jobs Tab
┌────────────────────────────────────────────────┐
│ Job 0 (first action on cached DataFrame)       │
│   Stage 0: Scan (100 tasks, 3.2s)              │
│   Stage 1: GroupBy shuffle (200 tasks, 8.5s)   │
│   Stage 2: Result (1 task, 0.1s)               │
├────────────────────────────────────────────────┤
│ Job 1 (second action on same cached DataFrame) │
│   Stage 0: SKIPPED (0 tasks, 0s)               │
│   Stage 1: SKIPPED (0 tasks, 0s)               │
│   Stage 2: Result (1 task, 0.1s)               │
└────────────────────────────────────────────────┘

What if stages are NOT skipped when you expected them to be?

Unexpected recomputation usually means one of these problems:

• Cache not called – you forgot to call .cache() or .persist() on the DataFrame.
• Cache evicted – executor memory was tight and Spark evicted the cached blocks; check the Storage tab for fraction cached.
• Different lineage – the two actions actually reference different DataFrames that happen to look similar but have different lineage.
• Shuffle files cleaned – Spark's shuffle cleanup removed the shuffle files before the second action ran.

# Debugging unexpected recomputation
df = txns.groupBy("account_id").agg(spark_sum("amount"))

# WRONG: forgot to cache
df.count()   # Computes
df.show(5)   # Recomputes from scratch!

# CORRECT: cache first
df.cache()
df.count()   # Computes and populates cache
df.show(5)   # Reads from cache, stages skipped

How does implicit shuffle reuse differ from explicit caching?

Implicit shuffle reuse happens when Spark detects that shuffle files from a prior job are still on disk. It is convenient but fragile because shuffle files can be cleaned up at any time.

# Implicit reuse: works but fragile
df = txns.groupBy("account_id").agg(spark_sum("amount"))
df.count()   # Shuffle files written
df.show(5)   # Shuffle files reused IF they still exist

# Explicit caching: guaranteed reuse
df.cache()
df.count()   # Cache populated
df.show(5)   # Cache read, always works until unpersist or eviction

# For critical reuse across multiple actions, always cache or checkpoint
# explicitly rather than relying on implicit shuffle file reuse

What should a senior engineer know?

• Skipped stages are a positive diagnostic signal: they confirm Spark is reusing work.
• Use .cache() or .persist() deliberately when a DataFrame is used by multiple downstream actions.
• Check the Storage tab to verify cache occupancy and detect eviction.
• For guaranteed reuse that survives executor failures, use .checkpoint() which writes to reliable storage.
• At the National Bank of India, the nightly pipeline cached the expensive account aggregation once and ran five different metric computations on it, with the Spark UI confirming all five jobs skipped the aggregation stages.
• Do not rely on implicit shuffle file reuse for correctness; it is an optimization that can disappear if shuffle files are cleaned up.