Back to Apache Spark: Core
Q21Question 21 of 72

Spark execution modes: cluster, client, local

Key Concepts
  • Cluster mode runs the Driver inside the cluster on a worker node, so the job survives if the submitter disconnects
  • Client mode runs the Driver on the submitting machine, giving interactive feedback but making the launcher a single point of failure
  • Local mode runs Driver and executors in a single JVM using threads, ideal for development and testing
  • Production batch jobs should always use cluster mode for robustness and proper resource allocation
  • Client mode is the default for spark-shell and notebooks where immediate output matters
  • The data logic is identical across all three modes; only the Driver location changes

Definition

Cluster mode puts the Driver inside the cluster for robust production runs, client mode keeps it on your machine for interactive work, and local mode runs everything in one JVM for development and testing.
  • Cluster mode – the Cluster Manager launches the Driver on a worker node inside the cluster, so the job runs independently of the submitting machine and is the production standard.
  • Client mode – the Driver runs on the machine that submitted the job, with output streaming back to the user, ideal for interactive exploration and notebooks.
  • Local mode – the Driver and executors run together in a single JVM using threads, with no cluster or Cluster Manager involved, perfect for development and testing.
  • Key distinction – only the Driver location differs; the data logic and Spark APIs are identical across all three modes.

Think of launching a reporting project at the National Bank of India. Cluster mode is like sending the project manager to sit permanently inside the data center with the clerks, so even if your laptop is switched off the project keeps running. Client mode is like the manager staying at their own desk in the branch and phoning instructions to the data center, so they see every update live, but if their desk shuts down the project dies. Local mode is like doing a tiny pilot entirely at your own desk with no data center at all.

Understanding the Concepts

What is cluster mode?

In cluster mode, the Cluster Manager launches the Driver on one of the worker nodes inside the cluster. The submitting machine sends the application jar or Python file to the Cluster Manager and then can disconnect. The Driver lives on the cluster network close to the executors, which reduces scheduling latency and makes the job robust.

BASH
# Submit a job in cluster mode on YARN
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --num-executors 20 \
  --executor-cores 4 \
  --executor-memory 8g \
  --driver-memory 4g \
  --conf spark.yarn.maxAppAttempts=2 \
  s3://nbi-jars/nbi_daily_settlement.py

# The Driver now runs inside a YARN container.
# The submitter can disconnect safely.

What is client mode?

In client mode, the Driver runs on the machine that issued the submit command. The executors still run on the cluster. Output streams back to the terminal because the Driver is local. This is the default for spark-shell, pyspark, and most Jupyter notebooks.

The trade-off is that the submitting machine must stay connected for the entire job, it must have enough memory for the Driver, and there is extra network latency between your machine and the cluster.

BASH
# Submit in client mode (default for interactive use)
spark-submit \
  --master yarn \
  --deploy-mode client \
  --num-executors 10 \
  --executor-cores 4 \
  --executor-memory 8g \
  nbi_exploration.py

# Or simply start an interactive shell
pyspark --master yarn --num-executors 10

What is local mode?

Local mode runs the entire Spark application in a single JVM on one machine using threads to simulate parallelism. There is no cluster and no Cluster Manager.

PYTHON
from pyspark.sql import SparkSession

# Local mode: all threads in one JVM
spark = SparkSession.builder \
    .master("local[4]") \
    .appName("NBI-UnitTests") \
    .getOrCreate()

# local[4] = 4 threads simulating 4 cores
# local[*] = use all available CPU cores
# local    = 1 thread only

print(f"Master: {spark.sparkContext.master}")
print(f"Default parallelism: {spark.sparkContext.defaultParallelism}")
TEXT
Master: local[4]
Default parallelism: 4

When should you use each mode?

The choice depends on the use case, not on the data logic, which is identical across all modes.

| Mode | When to use | When NOT to use |
|---|---|---|
| Cluster | Scheduled production jobs, Airflow pipelines, long-running batch | Interactive exploration where you need live output |
| Client | Notebooks, spark-shell, ad hoc analysis, development | Production jobs where the launcher may disconnect |
| Local | Unit tests, CI pipelines, debugging on small data | Real workloads that need distributed scale |

PYTHON
# Production: always cluster mode
# spark-submit --deploy-mode cluster ...

# Interactive: client mode gives live feedback
# pyspark --deploy-mode client ...

# Testing: local mode needs no cluster
# spark = SparkSession.builder.master("local[*]").getOrCreate()

How does the Driver location affect reliability?

In cluster mode, if the submitting machine disconnects, the job continues because the Driver is inside the cluster. The Cluster Manager can even restart the Driver container if it fails (for example, YARN's maxAppAttempts).

In client mode, the Driver is the submitting machine. If that machine loses network or shuts down, the job dies immediately.

BASH
# Cluster mode with automatic restart on YARN
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --conf spark.yarn.maxAppAttempts=3 \
  nbi_daily_settlement.py

# If the Driver container fails, YARN restarts it up to 3 times.
# This is NOT available in client mode.

How does the Driver location affect performance?

In cluster mode, the Driver sits on the cluster network alongside the executors, so task scheduling messages and small result collections travel over the fast internal network.

In client mode, every scheduling message and every collect() result must travel between your machine and the cluster, adding latency. For large collect() calls this can be significant.

PYTHON
# Client mode: collecting data crosses the network to your laptop
result = big_dataframe.collect()  # All data travels to your machine

# Cluster mode: collecting happens inside the cluster
# and only the final output is written to storage.
big_dataframe.write.parquet("s3://nbi-data-lake/output/")

What are common mistakes with execution modes?

The most common mistake is running a production job in client mode from a laptop that goes to sleep or loses VPN. The second most common is trying to debug a cluster-mode job without access to the Driver logs, which are inside a YARN container.

BASH
# Wrong: production job in client mode from a laptop
spark-submit --deploy-mode client nbi_nightly_batch.py
# If you close your laptop, the job dies.

# Right: production job in cluster mode
spark-submit --deploy-mode cluster nbi_nightly_batch.py
# The job runs inside the cluster, independent of your laptop.

# To read Driver logs in cluster mode on YARN:
yarn logs -applicationId app-20240115-001

How does local mode help with testing?

Local mode runs Spark without any cluster, so unit tests can create a SparkSession, run transformations, and assert results without external dependencies. This keeps CI fast and self-contained.

PYTHON
# Unit test using local mode (pytest example)
import pytest
from pyspark.sql import SparkSession, functions as F

@pytest.fixture(scope="session")
def spark():
    return SparkSession.builder \
        .master("local[2]") \
        .appName("NBI-Tests") \
        .getOrCreate()

def test_settlement_total(spark):
    data = [("SAV-1001", 1000.00, "SETTLED"),
            ("SAV-1002", 2000.00, "SETTLED"),
            ("SAV-1003", 500.00, "PENDING")]
    df = spark.createDataFrame(data, ["accountId", "amount", "status"])

    result = df.filter(F.col("status") == "SETTLED") \
               .agg(F.sum("amount").alias("total")) \
               .collect()[0]["total"]

    assert result == 3000.00

What should a senior engineer know about execution modes?

A senior engineer knows that the mode choice is an operational decision about reliability and feedback, not about data logic. They also know about Kubernetes as an alternative to YARN, where the Driver runs as a Kubernetes pod and executors are dynamically created pods, which is increasingly common for cloud-native deployments.

BASH
# Kubernetes cluster mode
spark-submit \
  --master k8s://https://k8s-api:6443 \
  --deploy-mode cluster \
  --conf spark.kubernetes.container.image=nbi-spark:3.5.0 \
  --conf spark.executor.instances=20 \
  local:///opt/spark/nbi_daily_settlement.py

Visual Overview

DIAGRAM
  Cluster Mode:                   Client Mode:
  ┌───────────┐                  ┌───────────┐
  │ Submitter │ submit           │ Submitter │ submit + DRIVER
  │ (edge)    │──────┐           │ (laptop)  │──────┐
  └───────────┘      │           └─────┬─────┘      │
      can disconnect │                 │ must stay   │
                     ▼                 │ connected   ▼
  ┌──────────────────────┐      ┌──────▼───────────────┐
  │  Cluster              │      │  Cluster              │
  │  ┌────────┐           │      │                       │
  │  │ DRIVER │           │      │  Executors only       │
  │  └────────┘           │      │  (Driver on laptop)   │
  │  Executors            │      │                       │
  └──────────────────────┘      └───────────────────────┘

  Local Mode:
  ┌──────────────────────┐
  │  Single JVM          │
  │  Driver + Executors  │
  │  (threads only)      │
  │  No cluster needed   │
  └──────────────────────┘

Key Takeaway

Spark has three deployment modes, and the single thing that distinguishes them is where the Driver process runs. In cluster mode, the Cluster Manager launches the Driver on a worker node inside the cluster, so the job runs independently of the submitting machine and survives disconnections. In client mode, the Driver stays on the submitter and streams output live, which is perfect for notebooks and interactive exploration but makes the launcher a single point of failure.

Local mode runs the entire application in one JVM using threads, with no cluster at all. It is meant for development, testing, and CI pipelines where you want a fast Spark environment without external dependencies.

At the National Bank of India we matched each mode to its purpose. Airflow-scheduled nightly reconciliation always ran in cluster mode so a flaky launcher never killed a running pipeline. Data scientists explored in client-mode notebooks where seeing each result immediately mattered. The automated test suite ran in local mode so CI needed no cluster.

The key insight is that the data logic is identical across all three modes. You write the same transformations, the same SQL, the same DataFrame operations. Only the Driver location and operational characteristics change.

Remember one sentence: cluster mode for production robustness, client mode for interactive feedback, local mode for testing, and the code stays the same across all three.

---

Share
Use ← → arrow keys to navigate