Definition
- No installation on workers – Spark runs on top of YARN as an application; worker nodes only need a YARN NodeManager and a JVM.
- Edge node only – install Spark on the submitter/edge node where
spark-submit, client libraries, and configuration live. - Jar distribution – Spark either uploads its jars from the edge node or localizes them from a staged HDFS archive (
spark.yarn.archive). - Upgrade path – update the edge node and the staged archive; no changes to worker software.
Think of YARN as the bank's building manager who hands out empty meeting rooms (containers). Spark is a consultant who brings their own laptop and materials (jars) into whatever room they are given. You do not pre-stock every room; the consultant carries everything in. Only the consultant's home office (edge node) needs to be fully equipped.
Understanding the Concepts
Do you need Spark on every YARN node?
No. Spark runs on top of YARN as an application. YARN provides containers (JVM processes with allocated CPU and memory), and Spark fills those containers with its own runtime. Worker nodes only need a YARN NodeManager and a JVM.
# The edge node has Spark installed
which spark-submit
# /opt/spark/bin/spark-submit
spark-submit --version
# version 3.5.0
# Worker nodes do NOT have spark-submit
# They only have the YARN NodeManager and Java
How does Spark get its jars to YARN containers?
When you run spark-submit --master yarn, Spark provides its own jars to the containers. There are two ways.
# Way 1: Upload from edge node (slow, happens every run)
spark-submit --master yarn --deploy-mode cluster \
nbi_daily_settlement.py
# Spark uploads its jars from the edge node to each container.
# This happens on every job submission.
# Way 2: Stage archive in HDFS (fast, cached across runs)
# First, stage the archive once:
hdfs dfs -put /opt/spark/jars/spark-libs.jar \
hdfs:///spark-archives/spark-3.5.0-libs.jar
# Then configure spark-submit:
spark-submit --master yarn --deploy-mode cluster \
--conf spark.yarn.archive=hdfs:///spark-archives/spark-3.5.0-libs.jar \
nbi_daily_settlement.py
# YARN localizes from HDFS (fast, cached)
What is the submission flow on YARN?
The flow has four steps. The edge node's Spark client contacts the ResourceManager. YARN starts an ApplicationMaster container. The ApplicationMaster requests executor containers. NodeManagers launch executor JVMs with Spark's jars.
# Step 1: Edge node submits to YARN ResourceManager
# spark-submit --master yarn --deploy-mode cluster ...
# Step 2: ResourceManager allocates an ApplicationMaster container
# (This is the Spark Driver in cluster mode)
# Step 3: ApplicationMaster requests executor containers
# (e.g., 10 executors with 4 cores and 8 GB each)
# Step 4: NodeManagers launch executor JVMs
# Spark jars are localized from HDFS to each container
# No Spark pre-installed on worker nodes!
How does staging the archive speed up startup?
Staging a Spark archive in HDFS means YARN localizes it once and caches it. Subsequent jobs reuse the cached archive, which dramatically reduces startup time for clusters running many Spark jobs per day.
# Create a compressed archive of Spark jars
jar cv0f spark-libs.jar -C /opt/spark/jars/ .
# Upload to HDFS
hdfs dfs -put spark-libs.jar hdfs:///spark-archives/
# Configure in spark-defaults.conf on the edge node
# spark.yarn.archive hdfs:///spark-archives/spark-libs.jar
# Every job now localizes from HDFS instead of uploading
# from the edge node. Startup drops from minutes to seconds.
How do you upgrade Spark on a YARN cluster?
Upgrading is simple because Spark is only on the edge node and in the HDFS archive. No worker changes needed.
# Step 1: Install new Spark on the edge node
sudo tar -xzf spark-3.6.0-bin-hadoop3.tgz -C /opt/
sudo ln -sfn /opt/spark-3.6.0-bin-hadoop3 /opt/spark
# Step 2: Update the staged archive in HDFS
jar cv0f spark-libs-3.6.0.jar -C /opt/spark/jars/ .
hdfs dfs -put -f spark-libs-3.6.0.jar hdfs:///spark-archives/
# Step 3: Update spark-defaults.conf
# spark.yarn.archive hdfs:///spark-archives/spark-libs-3.6.0.jar
# Workers are unchanged! They just run YARN containers.
What is the difference between client and cluster mode on YARN?
Client mode runs the Driver on the edge node. Cluster mode runs the Driver inside a YARN container on a worker node.
# Cluster mode: Driver in YARN container (production)
spark-submit --master yarn --deploy-mode cluster \
--conf spark.yarn.maxAppAttempts=2 \
nbi_daily_settlement.py
# Edge node can disconnect. Job survives.
# Client mode: Driver on edge node (interactive)
pyspark --master yarn --deploy-mode client
# Output streams to your terminal.
# Edge node must stay connected.
What do worker nodes actually need?
Worker nodes need only three things: a YARN NodeManager, a JVM (Java 8 or 11+), and network connectivity to the ResourceManager and HDFS.
# On a worker node:
java -version
# openjdk version "17.0.9"
ps aux | grep NodeManager
# yarn NodeManager running
which spark-submit
# Not found! Spark is NOT installed on workers.
What if you use Kubernetes instead of YARN?
The same principle applies. Spark on Kubernetes creates executor pods with the Spark runtime baked into the Docker image. You do not install Spark on Kubernetes worker nodes.
# Kubernetes: Spark runtime is in the container image
spark-submit --master k8s://https://k8s-api:6443 \
--deploy-mode cluster \
--conf spark.kubernetes.container.image=nbi-spark:3.5.0 \
nbi_daily_settlement.py
# Worker nodes run generic Kubernetes. The Spark image is pulled.
What should a senior engineer know?
A senior engineer knows that Spark is a YARN application, not a YARN service. They stage the archive in HDFS for fast startup, run production jobs in cluster mode, and upgrade Spark without touching worker nodes.
# Senior checklist for Spark on YARN:
# 1. Install Spark only on edge node(s)
# 2. Stage spark.yarn.archive in HDFS
# 3. Production: --deploy-mode cluster
# 4. Interactive: --deploy-mode client
# 5. Upgrade: edge node + HDFS archive only
# 6. Worker nodes: YARN NodeManager + JVM only
Visual Overview
Spark on YARN deployment:
┌──────────────────┐
│ Edge Node │ ← Spark installed here
│ spark-submit │
│ spark-defaults │
└────────┬─────────┘
│ submit
▼
┌──────────────────┐
│ ResourceManager │ ← YARN scheduler
└────────┬─────────┘
│ allocates containers
▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ Worker 1 │ │ Worker 2 │ │ Worker 3 │
│ NodeManager│ │ NodeManager│ │ NodeManager│
│ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │
│ │Executor│ │ │ │Executor│ │ │ │Executor│ │
│ │(JVM + │ │ │ │(JVM + │ │ │ │(JVM + │ │
│ │ Spark │ │ │ │ Spark │ │ │ │ Spark │ │
│ │ jars) │ │ │ │ jars) │ │ │ │ jars) │ │
│ └────────┘ │ │ └────────┘ │ │ └────────┘ │
└────────────┘ └────────────┘ └────────────┘
No Spark pre-installed! Jars arrive with the application.
Key Takeaway
No, you do not need Spark installed on every YARN node. Spark runs on top of YARN as an application: the edge node submits the job, YARN allocates containers on worker nodes, and Spark ships its own jars to those containers. Worker nodes only need a YARN NodeManager and a JVM.
In practice, install Spark only on the edge node where spark-submit and your configuration live. Stage a Spark archive in HDFS with spark.yarn.archive so YARN localizes and caches it, which dramatically speeds up startup for clusters running many jobs per day. Upgrading Spark means updating the edge node and the staged archive, with no changes to worker software.
At the National Bank of India, the platform team initially feared they had to install and patch Spark on all 200 worker nodes. I showed them that only the edge node needed Spark and that staging the archive in HDFS gave fast startup. We ran production in cluster mode so the Driver survived edge node disconnections, and reserved client mode for interactive notebook work.
Remember one sentence: Spark is a YARN application that ships its own jars to containers, so install it on the edge node, stage the archive in HDFS, and leave the workers generic.
---