Spark master-worker architecture: Driver, Cluster Manager, Workers, Executors, Tasks | Data Engineering

Definition

The Driver plans and schedules work, the Cluster Manager allocates resources, Executors on Worker Nodes run Tasks in parallel, and one Task processes one partition.

Driver – the master process that hosts the SparkSession, records transformations into a DAG, and contains the DAGScheduler (breaks jobs into stages) and TaskScheduler (dispatches tasks to executors).
Cluster Manager – a pluggable component (Standalone, YARN, Kubernetes, or Databricks) that allocates CPU and memory for executors across the cluster.
Worker Node – a physical or virtual machine that provides compute resources to the cluster.
Executor – a long-lived JVM process launched on a worker node that runs tasks concurrently and holds cached partitions in memory.
Task – the smallest unit of work; Spark creates one task per partition per stage, and tasks run in parallel across executor cores.

Picture the National Bank of India processing year-end statements for millions of accounts. The Driver is the branch manager who holds the master plan and decides who does what. The Cluster Manager is the HR department that assigns desks and staff. The Worker Nodes are the office floors with desks. The Executors are the clerks sitting at those desks who actually do the counting and keep their working papers in front of them. A Task is one clerk processing one box of account files. If there are two hundred boxes, the manager splits the job into two hundred tasks and hands them to the clerks, who work in parallel.

Understanding the Concepts

What is the Driver?

The Driver is the master process of a Spark application. When you create a SparkSession, you start the Driver. It runs your main program, records transformations into a logical plan, and contains two schedulers. The DAGScheduler breaks the job into stages at shuffle boundaries, and the TaskScheduler dispatches individual tasks to executors and tracks their completion.

The Driver also holds metadata about the application, collects final results from actions like collect, and serves the Spark UI on port 4040. If the Driver dies, the entire application fails.

PYTHON

from pyspark.sql import SparkSession

# Creating a SparkSession starts the Driver process
spark = SparkSession.builder \
    .appName("NBI-YearEndStatements") \
    .config("spark.executor.instances", "10") \
    .config("spark.executor.cores", "4") \
    .config("spark.executor.memory", "8g") \
    .getOrCreate()

# The Driver is now running and ready to schedule work
print(f"App ID: {spark.sparkContext.applicationId}")
print(f"Master: {spark.sparkContext.master}")

TEXT

App ID: app-20240115-001
Master: yarn

What is the Cluster Manager?

The Cluster Manager is a pluggable component that allocates compute resources. Spark does not insist on one resource manager. It works with Standalone (Spark's built-in scheduler), Hadoop YARN, Kubernetes, or a managed platform like Databricks.

When the Driver requests executors with a certain number of cores and memory, the Cluster Manager finds room on worker nodes and launches those JVM processes.

PYTHON

# Submit to YARN cluster manager
# spark-submit --master yarn --deploy-mode cluster \
#   --num-executors 10 --executor-cores 4 --executor-memory 8g \
#   nbi_year_end.py

# Submit to Kubernetes
# spark-submit --master k8s://https://k8s-api:6443 \
#   --deploy-mode cluster \
#   --conf spark.executor.instances=10 \
#   nbi_year_end.py

# Submit to Standalone
# spark-submit --master spark://master-node:7077 \
#   --deploy-mode cluster \
#   nbi_year_end.py

What are Worker Nodes and Executors?

Worker Nodes are the physical or virtual machines in the cluster that provide CPU and memory. The Cluster Manager launches Executors on these nodes. Each Executor is a long-lived JVM process with a fixed number of cores and a memory allocation.

An Executor runs multiple tasks concurrently (one task per core) and keeps cached partitions in memory for later reuse. Executors report their status back to the Driver with heartbeats.

PYTHON

# Check executor details from the Driver
sc = spark.sparkContext
print(f"Default parallelism: {sc.defaultParallelism}")

# With 10 executors x 4 cores each = 40 total cores
# Spark can run up to 40 tasks in parallel

TEXT

Default parallelism: 40

What is a Task and how does parallelism work?

A Task is the smallest unit of execution. The rule is: one task per partition per stage. If a DataFrame has 200 partitions and the current stage runs a filter, Spark creates 200 tasks for that stage. These tasks are distributed across executor cores.

The actual parallelism you get is min(partition_count, total_executor_cores). Too few partitions leave cores idle; too many create scheduling overhead.

PYTHON

from pyspark.sql import functions as F

txns = spark.read.parquet("s3://nbi-data-lake/transactions/")
print(f"Partitions: {txns.rdd.getNumPartitions()}")

# With 200 partitions and 40 cores, Spark runs 40 tasks at a time
# and completes all 200 in 5 waves.

# Repartition to match cluster capacity
txns_balanced = txns.repartition(400)  # 400 partitions, 10 tasks per core wave
print(f"New partitions: {txns_balanced.rdd.getNumPartitions()}")

TEXT

Partitions: 200
New partitions: 400

How does the DAGScheduler break a job into stages?

The DAGScheduler examines the lineage of transformations and draws stage boundaries wherever a shuffle is needed. Operations like filter and select that work partition-by-partition (narrow transformations) go into the same stage. Operations like groupBy and join that require data to move across partitions (wide transformations) trigger a new stage.

PYTHON

# Stage 1: read + filter + select (narrow, no shuffle)
# Stage 2: groupBy + sum (wide, shuffle boundary)
# Stage 3: orderBy (wide, another shuffle)

result = txns \
    .filter(F.col("status") == "SETTLED") \
    .select("branch", "amount") \
    .groupBy("branch") \
    .agg(F.sum("amount").alias("total")) \
    .orderBy(F.desc("total"))

result.explain("formatted")

DIAGRAM

== Physical Plan ==
AdaptiveSparkPlan
+- Sort [total DESC]
   +- Exchange rangepartitioning(total DESC, 200)    ← Stage boundary
      +- HashAggregate(keys=[branch], functions=[sum(amount)])
         +- Exchange hashpartitioning(branch, 200)    ← Stage boundary
            +- HashAggregate(keys=[branch], functions=[partial_sum(amount)])
               +- Filter (status = SETTLED)
                  +- FileScan parquet [branch,amount,status]

How does the TaskScheduler dispatch tasks?

Once the DAGScheduler creates stages, the TaskScheduler sends individual tasks to executors. It uses data locality to prefer executors that already have the relevant data. Locality levels from best to worst are: PROCESS_LOCAL (data in the same executor), NODE_LOCAL (same machine), RACK_LOCAL (same network rack), and ANY.

PYTHON

# The TaskScheduler considers locality when placing tasks.
# In the Spark UI, you can see the locality level for each task.

# Force a job to run so we can inspect the Spark UI
result.write.mode("overwrite").parquet("s3://nbi-data-lake/branch-totals/")

# Check Spark UI at http://driver-host:4040
# Stages tab → click a stage → see locality level per task

What happens when the Driver or an Executor fails?

If the Driver dies, the entire application fails because it holds the plan and the schedulers. This is why production jobs run in cluster mode where the Cluster Manager can restart the Driver on another node. In client mode, the Driver runs on the submitter machine, and if that machine disconnects, the job dies.

If an Executor dies, the TaskScheduler detects the missing heartbeat and reschedules the lost tasks on other executors. Cached partitions on the dead executor are recomputed from lineage. The application continues.

PYTHON

# Production: submit in cluster mode so the Driver runs inside the cluster
# spark-submit --master yarn --deploy-mode cluster \
#   --conf spark.yarn.maxAppAttempts=2 \
#   nbi_year_end.py

# If the Driver container fails, YARN retries up to maxAppAttempts times.
# Executor failures are handled automatically by the TaskScheduler.

How do you tune the architecture for performance?

The key tuning parameters are executor count, cores per executor, and memory per executor. A common rule of thumb is 4-5 cores per executor (to balance parallelism against thread contention) and to leave 1 core and 1 GB per node for the OS and Cluster Manager.

PYTHON

# Example cluster: 10 nodes, each with 16 cores and 64 GB memory
# Leave 1 core + 1 GB for OS/YARN per node.
# Remaining: 15 cores and 63 GB per node.
# 3 executors per node x 5 cores each = 15 cores used.
# Memory: 63 GB / 3 = 21 GB per executor.

# spark-submit --master yarn --deploy-mode cluster \
#   --num-executors 30 \
#   --executor-cores 5 \
#   --executor-memory 21g \
#   --driver-memory 4g \
#   nbi_year_end.py

# Total parallelism: 30 executors x 5 cores = 150 concurrent tasks

What should a senior engineer know?

A senior engineer understands that the Driver is a single point of failure and always runs production jobs in cluster mode. They know that executor memory is split into execution memory (shuffles, joins, sorts) and storage memory (caching), and the two can borrow from each other under Spark's unified memory manager.

They also know that dynamic allocation lets the Cluster Manager add or remove executors based on load, which is critical for shared clusters at a bank where many jobs compete for resources.

PYTHON

# Enable dynamic allocation for shared clusters
# spark-submit --master yarn --deploy-mode cluster \
#   --conf spark.dynamicAllocation.enabled=true \
#   --conf spark.dynamicAllocation.minExecutors=5 \
#   --conf spark.dynamicAllocation.maxExecutors=50 \
#   --conf spark.shuffle.service.enabled=true \
#   nbi_year_end.py

Visual Overview

DIAGRAM

  ┌──────────────────────────────────────────────────────┐
  │                  Spark Application                    │
  │                                                        │
  │  ┌──────────────────────────┐                         │
  │  │       DRIVER              │                         │
  │  │  SparkSession             │                         │
  │  │  DAGScheduler             │                         │
  │  │  TaskScheduler            │                         │
  │  │  Spark UI (:4040)         │                         │
  │  └──────────┬───────────────┘                         │
  │             │ requests resources                       │
  │             ▼                                          │
  │  ┌──────────────────────────┐                         │
  │  │   CLUSTER MANAGER        │                         │
  │  │  (YARN / K8s / Standalone)│                         │
  │  └──────────┬───────────────┘                         │
  │             │ launches executors                       │
  │             ▼                                          │
  │  ┌────────────┐  ┌────────────┐  ┌────────────┐      │
  │  │ Worker Node │  │ Worker Node │  │ Worker Node │     │
  │  │ ┌────────┐  │  │ ┌────────┐  │  │ ┌────────┐  │    │
  │  │ │Executor│  │  │ │Executor│  │  │ │Executor│  │    │
  │  │ │ Core 1 │  │  │ │ Core 1 │  │  │ │ Core 1 │  │    │
  │  │ │ Core 2 │  │  │ │ Core 2 │  │  │ │ Core 2 │  │    │
  │  │ │ Cache  │  │  │ │ Cache  │  │  │ │ Cache  │  │    │
  │  │ └────────┘  │  │ └────────┘  │  │ └────────┘  │    │
  │  └────────────┘  └────────────┘  └────────────┘      │
  └──────────────────────────────────────────────────────┘

DIAGRAM

  Task scheduling flow:

  Job (triggered by action)
    │
    ▼
  DAGScheduler
    │  splits at shuffle boundaries
    ▼
  Stage 1 ──→ Stage 2 ──→ Stage 3
    │              │              │
    ▼              ▼              ▼
  Tasks        Tasks         Tasks
  [T1,T2,..]   [T1,T2,..]   [T1,T2,..]
    │              │              │
    ▼              ▼              ▼
  Executors    Executors     Executors

Key Takeaway

Spark uses a master-worker architecture with five distinct roles. The Driver is the master process that hosts the SparkSession, runs your main program, and contains the DAGScheduler and TaskScheduler. The Cluster Manager, which can be Standalone, YARN, Kubernetes, or Databricks, allocates resources by launching Executors on Worker Nodes when the Driver requests them.

Executors are long-lived JVM processes that do the real work. Each executor runs tasks concurrently, one per core, and holds cached partitions in memory for later reuse. The smallest unit of work is a Task, and Spark creates one task per partition per stage. The actual parallelism is the minimum of the partition count and the total executor cores.

At the National Bank of India we ran our nightly reconciliation on YARN with 30 executors of 5 cores each, giving us 150 concurrent tasks. The Driver planned the job, YARN allocated the executors, and each executor processed its partitions in parallel. Tuning the partition count to match the cluster capacity was the single biggest performance lever.

The key operational fact is that if the Driver dies, the application dies. This is why production jobs run in cluster mode, where the Cluster Manager restarts the Driver on a stable node, and why we always set maxAppAttempts greater than one for critical banking jobs.

Remember one sentence: the Driver plans and schedules, the Cluster Manager allocates resources, Executors run Tasks in parallel on Worker Nodes, and one Task processes one partition.

---