Definition
- SparkSession – the Driver starts a session that holds the program, DAG, and schedulers.
- Executor provisioning – the Cluster Manager (YARN, Kubernetes, Standalone) allocates executors on worker nodes.
- DAG and Catalyst – transformations build a lazy DAG; the Catalyst Optimizer produces an optimized physical plan.
- Stages and Tasks – the DAGScheduler splits the plan at shuffle boundaries into stages; the TaskScheduler creates one task per partition per stage.
- Execution and Shuffle – executors run tasks in parallel; shuffle exchanges data between stages across the network.
- Results and Cleanup – results are returned to the Driver or written to storage, and stopping the session releases all resources.
Picture the National Bank of India launching a big reporting project. The manager opens the project office (SparkSession). HR assigns desks and clerks (Executors). The manager writes the plan of steps (the DAG). A consultant reorders and streamlines the plan to be efficient (Catalyst). The plan is cut into phases wherever paperwork must be re-sorted between teams (stages at shuffle boundaries). Each phase is broken into one task per box of files (tasks). Clerks do the tasks in parallel. Between phases, files are re-sorted and handed over (shuffle). The finished report is handed to the manager or filed in the vault (results). Finally the project office is closed and desks are freed (stop).
Understanding the Concepts
What are the 10 steps of the Spark execution lifecycle?
The Spark execution lifecycle is the ordered sequence by which a Spark application runs from session start to resource cleanup. Understanding each step lets you read the Spark UI, diagnose slow stages, and explain Spark's architecture in interviews.
- Driver starts SparkSession.
- Cluster Manager provisions Executors.
- Transformations build a lazy DAG.
- Catalyst Optimizer creates the physical plan.
- DAGScheduler splits the plan into Stages.
- TaskScheduler creates Tasks (one per partition per stage).
- Executors run Tasks in parallel.
- Shuffle exchanges data between stages.
- Results returned to Driver or written to storage.
- SparkSession.stop() releases resources.
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
# Step 1: Driver starts SparkSession
spark = SparkSession.builder \
.appName("NBI-Lifecycle-Demo") \
.config("spark.sql.shuffle.partitions", "8") \
.getOrCreate()
# Steps 2-10 happen when we run a query
# (Cluster Manager provisions executors automatically)
print("SparkSession created:", spark.sparkContext.applicationId)
SparkSession created: local-1719849600000
How do Steps 1 and 2 set up the application?
Creating a SparkSession starts the Driver, which is the master JVM process holding your program, the DAG, and the schedulers. The Driver then negotiates with the Cluster Manager, which can be YARN, Kubernetes, Standalone, or a managed cloud service. The Cluster Manager provisions Executors on worker nodes with the requested cores and memory. These executors are long-lived JVM processes that run all the tasks for the application.
# Step 1: SparkSession is the unified entry point
spark = SparkSession.builder \
.appName("NBI-Daily-Settlement") \
.config("spark.executor.memory", "4g") \
.config("spark.executor.cores", "4") \
.config("spark.dynamicAllocation.enabled", "true") \
.getOrCreate()
# Step 2: Cluster Manager provisions executors
sc = spark.sparkContext
print("App name:", sc.appName)
print("Master:", sc.master)
print("Default parallelism:", sc.defaultParallelism)
App name: NBI-Daily-Settlement
Master: yarn
Default parallelism: 16
How do Steps 3 and 4 build and optimize the plan?
As you call transformations like filter, select, join, and groupBy, Spark records them into a logical DAG rather than executing them. Transformations are lazy. When an action triggers execution, the Catalyst Optimizer takes over for DataFrames and SQL: it parses the logical plan, analyzes it against the schema, applies optimizations like predicate pushdown and column pruning, and produces a physical plan with whole-stage code generation.
# Step 3: Build the lazy DAG with transformations
txn = spark.read.parquet("hdfs:///bank/transactions/2026-01/")
accounts = spark.read.parquet("hdfs:///bank/accounts/")
# These transformations are lazy: no computation yet
filtered = txn.filter(F.col("amount") > 50000)
joined = filtered.join(accounts, "account_id")
result = joined.groupBy("branch").agg(F.sum("amount").alias("total"))
# Step 4: Catalyst optimizes when we call explain (or an action)
result.explain(True)
== Parsed Logical Plan ==
Aggregate [branch], [branch, sum(amount) AS total]
+- Join Inner, (account_id = account_id)
:- Filter (amount > 50000)
: +- Relation [parquet] hdfs:///bank/transactions/2026-01/
+- Relation [parquet] hdfs:///bank/accounts/
== Optimized Logical Plan ==
Aggregate [branch], [branch, sum(amount) AS total]
+- Project [branch, amount]
+- Join Inner, (account_id = account_id)
:- Filter (amount > 50000)
: +- Relation [parquet] (column pruning applied)
+- Relation [parquet] (column pruning applied)
== Physical Plan ==
*(3) HashAggregate(keys=[branch], functions=[sum(amount)])
+- Exchange hashpartitioning(branch, 8)
+- *(2) HashAggregate(keys=[branch], functions=[partial_sum(amount)])
+- *(2) BroadcastHashJoin [account_id], [account_id]
:- *(2) Filter (amount > 50000)
: +- *(2) ColumnarToRow
: +- FileScan parquet
+- BroadcastExchange
+- *(1) FileScan parquet
How do Steps 5 and 6 schedule stages and tasks?
The DAGScheduler walks the physical plan and cuts it into stages wherever there is a wide dependency (a shuffle). Within each stage, the TaskScheduler creates one task per partition and assigns them to executor cores, respecting data locality where possible.
# The physical plan above has two shuffle boundaries:
# 1. BroadcastExchange (broadcast the small accounts table)
# 2. Exchange hashpartitioning (shuffle for groupBy)
#
# This creates roughly 3 stages:
# Stage 1: Scan accounts, broadcast
# Stage 2: Scan transactions, filter, join, partial aggregate
# Stage 3: Final aggregate after shuffle
# Each stage has one task per partition:
print("Shuffle partitions:", spark.conf.get("spark.sql.shuffle.partitions"))
# Stage 3 will have 8 tasks (one per shuffle partition)
Shuffle partitions: 8
How do Steps 7 and 8 execute tasks and shuffle data?
Executors run the tasks in parallel, each task processing one partition. Between stages, a shuffle exchanges data across the network. Each task in the earlier stage writes bucketed shuffle output to local disk. Each task in the later stage reads the shuffle buckets it needs from the producing executors.
# Step 7: Trigger execution with an action
# Step 8: Shuffle happens automatically between stages
result_df = result.collect()
for row in result_df:
print(f"Branch: {row['branch']}, Total: {row['total']}")
Branch: Mumbai, Total: 45750000.00
Branch: Delhi, Total: 32100000.00
Branch: Chennai, Total: 28900000.00
How do Steps 9 and 10 return results and clean up?
When all stages complete, the action's result either returns to the Driver (like count returning a number or collect returning a list) or is written to storage (like write.parquet saving files). Finally, stopping the SparkSession tells the Cluster Manager to release the executors and frees all resources.
# Step 9: Write results to storage instead of collecting
result.write.mode("overwrite").parquet(
"hdfs:///bank/reports/branch-totals/2026-01/"
)
print("Results written to HDFS")
# Step 10: Release resources
spark.stop()
print("SparkSession stopped, executors released")
Results written to HDFS
SparkSession stopped, executors released
How does each action trigger its own pass through the lifecycle?
A critical nuance is that each action triggers its own run through steps 3 to 9 against the current plan. A job with multiple actions goes through the scheduling phases multiple times unless intermediate results are cached. This is why caching matters for reused DataFrames.
spark = SparkSession.builder.appName("NBI-Multi-Action").getOrCreate()
txn = spark.read.parquet("hdfs:///bank/transactions/2026-01/")
clean = txn.filter(F.col("status") == "COMPLETED")
# Without cache: each action re-reads and re-filters
count1 = clean.count() # Action 1: full pass
count2 = clean.agg(F.sum("amount")).collect() # Action 2: another full pass
# With cache: first action materializes, second reuses
clean.cache()
count1 = clean.count() # Action 1: reads, filters, caches
count2 = clean.agg(F.sum("amount")).collect() # Action 2: reads from cache
print(f"Completed transactions: {count1}")
print(f"Total amount: {count2[0][0]}")
Completed transactions: 1250000
Total amount: 9875000000.00
How does the Spark UI map to the lifecycle?
The Spark UI is structured around the lifecycle. Jobs map to actions, stages map to shuffle boundaries, and tasks map to partitions. When a banking job is slow, I open the UI and walk the lifecycle backward to find the bottleneck.
Spark UI Mapping: Jobs tab → one job per action (count, collect, write) Stages tab → one stage per shuffle boundary Tasks tab → one task per partition per stage SQL tab → the Catalyst physical plan Storage tab → cached DataFrames and their sizes Executors tab → GC time, memory usage, task counts
# Example: diagnosing a slow stage
# In the Spark UI:
# Job 0 (the count action) has 3 stages
# Stage 0: FileScan (fast, 16 tasks)
# Stage 1: Shuffle + Join (slow, check task duration spread)
# Stage 2: Final aggregate (8 tasks)
#
# If Stage 1 is slow and one task takes 10x longer than others:
# → Data skew on the join key
# Fix: enable AQE skew join or salt the hot key
What should a senior engineer know about the lifecycle?
A senior engineer uses the lifecycle to explain every performance question. Each action is a separate job. Each shuffle boundary creates a new stage. Each partition creates a task. Understanding this mapping lets you pinpoint whether a problem is in the plan (Catalyst), the scheduling (too few or too many partitions), the shuffle (skew or excessive data movement), or the execution (GC pressure, spills, or stragglers).
# Senior checklist for every slow Spark job:
# 1. How many jobs? (one per action: are you calling collect multiple times?)
# 2. How many stages? (one per shuffle: are there unnecessary shuffles?)
# 3. How many tasks per stage? (one per partition: right partition count?)
# 4. Task duration spread? (even = good, skewed = data skew)
# 5. GC time? (high = memory pressure, likely over-caching)
# 6. Spill metrics? (high = partitions too large or memory too small)
# 7. Physical plan? (broadcast vs sort-merge join, pushdown applied?)
Visual Overview
┌─────────────────────────────────────────────────────────────┐ │ SPARK 10-STEP EXECUTION LIFECYCLE │ ├─────────────────────────────────────────────────────────────┤ │ │ │ Step 1: SparkSession.builder.getOrCreate() │ │ ┌──────────┐ │ │ │ DRIVER │ (JVM process: program + schedulers) │ │ └────┬─────┘ │ │ │ │ │ Step 2: ▼ Request executors │ │ ┌──────────────┐ │ │ │CLUSTER MGR │ (YARN / K8s / Standalone) │ │ │ provisions │ │ │ └──┬───┬───┬───┘ │ │ ▼ ▼ ▼ │ │ ┌───┐┌───┐┌───┐ │ │ │E1 ││E2 ││E3 │ Executors on worker nodes │ │ └───┘└───┘└───┘ │ │ │ │ Step 3: Lazy DAG built from transformations │ │ filter → join → groupBy → agg (no execution yet) │ │ │ │ Step 4: Catalyst Optimizer │ │ Logical Plan → Optimized Plan → Physical Plan │ │ (predicate pushdown, column pruning, codegen) │ │ │ │ Step 5: DAGScheduler splits into Stages │ │ ┌────────┐ shuffle ┌────────┐ shuffle ┌────────┐ │ │ │Stage 0 │──────────→│Stage 1 │─────────→│Stage 2 │ │ │ │(scan) │ boundary │(join) │ boundary │(agg) │ │ │ └────────┘ └────────┘ └────────┘ │ │ │ │ Step 6: TaskScheduler creates Tasks │ │ Stage 0: [T0][T1][T2][T3] (1 task per partition) │ │ Stage 1: [T0][T1][T2][T3] │ │ Stage 2: [T0][T1] │ │ │ │ Step 7: Executors run Tasks in parallel │ │ E1: [T0,T1] E2: [T2,T3] E3: [T0,T1] │ │ │ │ Step 8: Shuffle exchanges data between stages │ │ Map tasks write → shuffle files → Reduce tasks read │ │ │ │ Step 9: Results │ │ → Driver (collect, count) OR → Storage (write) │ │ │ │ Step 10: spark.stop() │ │ Executors released, resources freed │ └─────────────────────────────────────────────────────────────┘
Key Takeaway
The Spark execution lifecycle is a clear ten-step sequence, and being able to walk through it shows you understand how Spark actually runs. It begins when the Driver starts a SparkSession, which brings up the master process that holds the program, the DAG, and the schedulers. Next, the Cluster Manager provisions executors on worker nodes with the requested cores and memory. As you write transformations, Spark builds a logical DAG lazily, and when an action fires, the Catalyst Optimizer turns that logical plan into an optimized physical plan, applying predicate pushdown, column pruning, and whole-stage code generation.
From there, the DAGScheduler splits the job into stages, cutting a new stage wherever there is a wide dependency, because a shuffle is a boundary the next stage must wait for. The TaskScheduler then creates tasks following the one-task-per-partition-per-stage rule and assigns them to executor cores with data locality in mind. The executors run the tasks in parallel, and between stages a shuffle exchanges data across the network, with earlier-stage tasks writing bucketed output and later-stage tasks reading the buckets they need. Finally, the action's result is either returned to the Driver or written to storage, and stopping the SparkSession tells the Cluster Manager to release the executors.
In my project at the National Bank of India this lifecycle was exactly how I read the Spark UI to diagnose slow jobs. Jobs in the UI correspond to actions, stages correspond to shuffle boundaries, and tasks correspond to partitions, so when a nightly report was slow I could pinpoint whether the cost was in a particular shuffle-heavy stage or in skewed tasks within a stage. One nuance I always mention is that each action triggers its own pass through the planning and execution steps, so a job with several actions repeats the scheduling phases unless I cache the shared intermediate result.
Knowing this lifecycle end to end is what turned vague performance complaints into targeted fixes. Whether the problem is in the Catalyst plan, the stage scheduling, the shuffle, or the task execution, the lifecycle tells you exactly where to look.
Remember one sentence: the Spark lifecycle flows from SparkSession start through DAG build, Catalyst optimization, stage splitting at shuffles, one-task-per-partition execution, shuffle exchange, results out, and session stop.
---