Back to Apache Spark: Core
Q70Question 70 of 72

The Spark Web UI and debugging with it

Key Concepts
  • The Spark UI is served at port 4040 on the driver during a running app; the History Server preserves it for completed apps.
  • Jobs tab shows overall progress and which job is slow; Stages tab is the workhorse for per-task metrics.
  • Stages tab shows task duration spread, shuffle read/write, GC time, and spill to disk.
  • SQL tab shows the query plan, AQE decisions, and confirms predicate pushdown and broadcast joins.
  • Storage tab shows cached data, partition count, and memory/disk occupancy.
  • Executors tab shows per-executor memory, cores, task counts, and GC time.
  • Three diagnostic signatures: one task far slower = skew, high GC time = memory pressure, large shuffle write = expensive wide transformation.

Definition

The Spark UI (port 4040 live, History Server for completed apps) debugs through Jobs, Stages, SQL, Storage, and Executors tabs – one task far slower means skew, high GC time means memory pressure, and large shuffle write means an expensive wide transformation.
  • Spark Web UI – built-in monitoring interface served at port 4040 on the driver; History Server preserves completed apps.
  • Jobs tab – overall progress, job-to-stage breakdown, which job is slow or failed.
  • Stages tab – per-task metrics: duration spread, shuffle read/write, GC time, spill to disk.
  • SQL tab – query plan graph, AQE decisions, confirmation of pushdown and broadcast.
  • Storage tab – cached data, partition count, memory/disk occupancy.
  • Executors tab – per-executor memory, cores, task counts, failures, GC time.

At the National Bank of India, the Spark UI is like the live operations dashboard in the branch manager's office. Jobs is the "how far along" board, Stages is the per-clerk breakdown showing who is slow, SQL is the execution plan, Storage shows what is kept on the shelf, and Executors shows each clerk's desk size and workload.

Understanding the Concepts

What is the Spark Web UI?

The Spark Web UI is served by the driver at port 4040 while the application runs. For completed applications, the History Server reads event logs and renders the same tabs.

PYTHON
from pyspark.sql import SparkSession

# Enable event logging for the History Server
spark = SparkSession.builder \
    .appName("UIDemo") \
    .config("spark.eventLog.enabled", "true") \
    .config("spark.eventLog.dir", "/spark-events") \
    .getOrCreate()

# Access the live UI at http://localhost:4040
# After the app finishes, use the History Server

How do you use the Jobs tab?

The Jobs tab gives the top-level view. Each action triggers a job, and the tab shows status, duration, and stages.

PYTHON
from pyspark.sql.functions import col, sum as spark_sum

txns = spark.read.parquet("/data/transactions")

# Each action creates a job visible in the Jobs tab
txns.count()  # Job 0
txns.groupBy("account_id").agg(spark_sum("amount")).show()  # Job 1

# Jobs tab shows:
# Job 0: 1 stage, 100 tasks, duration 2.5s (succeeded)
# Job 1: 2 stages, 300 tasks, duration 45s (succeeded) <-- investigate

How do you use the Stages tab?

The Stages tab is the workhorse for performance debugging. It shows per-task metrics that reveal bottlenecks.

DIAGRAM
Spark UI -> Stages -> Click on slow Stage 1

Summary Metrics (quantiles across tasks):
┌────────────────┬────────┬────────┬────────┬────────┬────────┐
│ Metric         │  Min   │  25th  │ Median │  75th  │  Max   │
├────────────────┼────────┼────────┼────────┼────────┼────────┤
│ Duration       │  0.1s  │  0.3s  │  0.5s  │  0.8s  │  42.0s │ <-- SKEW!
│ Shuffle Read   │  10 KB │  50 KB │  80 KB │ 120 KB │  1.5 GB│ <-- SKEW!
│ GC Time        │    0ms │   50ms │  100ms │  200ms │ 5000ms │ <-- check
│ Spill (Disk)   │    0 B │    0 B │    0 B │    0 B │ 800 MB │ <-- memory
└────────────────┴────────┴────────┴────────┴────────┴────────┘
PYTHON
# After seeing skew in the Stages tab, investigate the hot key:
txns.groupBy("account_id").count().orderBy(col("count").desc()).show(5)
# Likely shows one account with disproportionately many transactions

How do you use the SQL tab?

The SQL tab shows the query plan as a visual graph with metrics at each operator.

PYTHON
# Run a query and check the SQL tab
result = txns.filter(col("amount") > 50000) \
    .join(spark.read.parquet("/data/accounts"), "account_id")
result.count()

# SQL tab shows:
# - Scan operator with PushedFilters (predicate pushdown confirmed)
# - BroadcastHashJoin or SortMergeJoin (join strategy)
# - AQE decisions: partition coalescing, broadcast switch
# - Row counts at each operator (helps find where data explodes)

What are the three diagnostic signatures?

Three patterns cover most performance problems. Learn to recognize them instantly.

| Signature | Where to Look | Root Cause | Fix |
|---|---|---|---|
| One task 10-100x slower | Stages: max vs median duration | Data skew | AQE, salting, broadcast |
| High GC time fraction | Executors: GC Time column | Memory pressure | More memory, fewer partitions, reduce cache |
| Large shuffle write | Stages: Shuffle Write column | Expensive wide op | Pre-aggregate, prune, tune partitions |

PYTHON
# Diagnostic workflow:
# 1. Jobs tab: find the slow job
# 2. Stages tab: find the slow stage
# 3. Check task duration spread: skew?
# 4. Check GC time: memory pressure?
# 5. Check shuffle write: expensive shuffle?
# 6. SQL tab: verify plan (pushdown? broadcast?)
# 7. Executors tab: executor-level health

How do you use the Storage and Executors tabs?

The Storage tab shows cached data. The Executors tab shows per-executor health.

PYTHON
# Storage tab: verify caching
txns.cache()
txns.count()  # Populate cache
# Storage tab shows:
# - RDD Name: transactions
# - Partitions Cached: 100 of 100
# - Memory Used: 2.5 GB
# - Disk Used: 0 B

# If partitions cached < total, cache is being evicted (memory pressure)

# Executors tab: per-executor health
# Columns to check:
# - Memory Used / Max Memory: utilization
# - GC Time: should be < 10% of total task time
# - Failed Tasks: should be 0 or very low
# - Shuffle Read/Write: distribution across executors

How do you correlate tabs for diagnosis?

The real skill is reading tabs together rather than in isolation.

PYTHON
# Example investigation at the National Bank of India:
# 1. Jobs tab: Job 5 took 45 minutes (expected: 5 minutes)
# 2. Stages tab: Stage 3 (groupBy) has one task at 42 minutes,
#    median 0.5s -> DATA SKEW
# 3. Executors tab: executor-4 has high GC time during that stage
#    -> the skewed partition is causing memory pressure on that executor
# 4. SQL tab: SortMergeJoin on account_id, no broadcast
#    -> could the small side be broadcast to avoid the skewed shuffle?
# 5. Fix: enable AQE skew join + broadcast the small table
# Result: Job drops from 45 minutes to 3 minutes

What should a senior engineer know?

  • Always enable event logging so the History Server captures everything for after-the-fact investigation.
  • The three signatures to memorize: skew, memory pressure, expensive shuffle.
  • Correlate tabs rather than reading one in isolation: a slow job leads to a stage, whose task distribution reveals skew, corroborated by executor GC time and the SQL plan.
  • At the National Bank of India, the Spark UI was the standard first-stop for diagnosing every slow nightly job.
  • The SQL tab is especially valuable for confirming that AQE decisions, predicate pushdown, and broadcast joins happened as expected.

Visual Overview

DIAGRAM
  Spark UI Debugging Workflow

  Step 1: Jobs Tab
  ┌────────────────────────────┐
  │ Job 5: 45 min (SLOW!)      │ --> Drill into stages
  └────────────┬───────────────┘
               ▼
  Step 2: Stages Tab
  ┌────────────────────────────┐
  │ Stage 3: groupBy           │
  │ Max task: 42 min           │ --> SKEW detected
  │ Median task: 0.5s          │
  │ Shuffle Read Max: 1.5 GB   │
  └────────────┬───────────────┘
               ▼
  Step 3: Executors Tab
  ┌────────────────────────────┐
  │ Executor 4: high GC time   │ --> Memory pressure
  │ (hosting the skewed task)  │
  └────────────┬───────────────┘
               ▼
  Step 4: SQL Tab
  ┌────────────────────────────┐
  │ SortMergeJoin on acct_id   │ --> Could broadcast?
  │ No AQE skew handling       │
  └────────────┬───────────────┘
               ▼
  Fix: Enable AQE + broadcast small table
  Result: 45 min -> 3 min

Key Takeaway

The Spark Web UI is the built-in monitoring interface, served at port 4040 during a running application and preserved by the History Server for completed apps. The Jobs tab gives overall progress and identifies which job is slow. The Stages tab is the workhorse, showing per-task metrics like duration spread, shuffle read/write, GC time, and spill. The SQL tab confirms the query plan and AQE decisions. Storage shows cached data, and Executors shows per-executor health.

The real skill is recognizing three diagnostic signatures. In the Stages tab, when the maximum task duration is many times the median, it is data skew. In the Executors tab, when GC time is a large fraction of task time, it is memory pressure. A large shuffle write signals an expensive wide transformation. These three patterns cover most performance problems.

At the National Bank of India, the standard triage was: Jobs tab to localize the slow job, Stages tab for task skew and shuffle metrics, Executors for GC and failures, SQL for the plan and AQE, and Storage for caching. Correlating tabs rather than reading one in isolation told a coherent story: a slow job led to a skewed stage, confirmed by high GC on one executor and an unexpected join strategy in the SQL plan.

The production practice is to always enable event logging so the History Server captures everything for after-the-fact investigation, because the most important debugging happens after the job finishes.

Remember one sentence: Jobs to localize, Stages for task skew and shuffle, Executors for GC, SQL for the plan – and the three signatures are skew, memory pressure, and expensive shuffle.

---

Share
Use ← → arrow keys to navigate