Back to Apache Spark: Core
Q68Question 68 of 72

The Spark / PySpark DataFrame programming model (vs Pandas)

Key Concepts
  • Spark DataFrames are distributed, immutable, and lazily evaluated: transformations build a plan, actions trigger execution.
  • Pandas DataFrames are single-machine, in-memory, and eager: every operation executes immediately.
  • Lazy evaluation lets Catalyst see the entire chain and optimize it before data moves (predicate pushdown, column pruning, join reordering).
  • Each action re-runs the plan from the start unless you cache or persist the intermediate DataFrame.
  • Wide transformations (join, groupBy) require a shuffle; narrow transformations (filter, select) do not.
  • Null handling in Spark uses dropna, fillna, and when/otherwise for conditional column logic.
  • Use Pandas for small interactive work and Spark when data outgrows one machine.

Definition

Spark DataFrames are distributed and lazy, building an optimized plan that only runs on an action; Pandas DataFrames are single-machine and eager, running every step immediately.
  • Spark DataFrame – a distributed, immutable, lazily-evaluated table of rows partitioned across a cluster.
  • Transformations – select, filter, withColumn, join are lazy; they build a logical plan without executing.
  • Actions – show, collect, count, write trigger the Catalyst optimizer to compile and execute the plan.
  • Pandas DataFrame – a single-machine, in-memory, eagerly-evaluated table limited by one machine's RAM.
  • Null handling – dropna, fillna, and when/otherwise for conditional logic.

Analysing the National Bank of India's transactions: a Pandas DataFrame is like one accountant with one desk who does everything immediately, but the whole ledger must fit on that desk. A Spark DataFrame is like a team across many desks, where a manager writes down the whole plan first, optimizes it, and only sends the team to work when you say "show me the result."

Understanding the Concepts

What is the Spark DataFrame programming model?

The core of the model is lazy evaluation. Transformations record operations into a logical plan without executing anything. Actions trigger the Catalyst optimizer to compile and run the plan.

PYTHON
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum as spark_sum, when

spark = SparkSession.builder.appName("DataFrameDemo").getOrCreate()

# Read transactions at the National Bank of India
txns = spark.read.parquet("/data/transactions")

# These are ALL lazy: nothing executes yet
filtered = txns.filter(col("amount") > 0)
selected = filtered.select("account_id", "amount", "txn_date")
enriched = selected.withColumn("fee", col("amount") * 0.001)

# Catalyst has built a logical plan but no data has moved
# Only an ACTION triggers execution:
enriched.show(5)   # Action: executes the full plan
enriched.count()   # Action: executes again from the start

What are transformations vs actions?

Transformations return a new DataFrame describing the operation. Actions produce a concrete result.

| Transformations (lazy) | Actions (trigger execution) |
|---|---|
| select, filter, where | show, display |
| withColumn, drop | count, first |
| join, crossJoin | collect, take |
| groupBy, agg | write, save |
| orderBy, sort | foreach, foreachPartition |
| union, intersect | toPandas |
| distinct, dropDuplicates | explain (shows plan, no data) |

PYTHON
# Chaining transformations: nothing runs until an action
result = txns \
    .filter(col("amount") > 10000) \
    .select("account_id", "amount") \
    .groupBy("account_id") \
    .agg(spark_sum("amount").alias("total"))

# Still lazy! View the optimized plan:
result.explain(True)
# Catalyst optimizes: pushes filter to source, prunes unused columns

# NOW execute:
result.show(5)  # Triggers the full pipeline

Why does lazy evaluation matter?

Laziness lets Catalyst see the entire chain and optimize it as a unit before any data moves.

PYTHON
# Without laziness (eager, like Pandas):
# Step 1: read ALL columns -> 10 GB in memory
# Step 2: filter -> 5 GB
# Step 3: select 2 columns -> 1 GB
# Total memory touched: 10 + 5 + 1 = 16 GB

# With laziness (Spark):
# Catalyst sees the entire chain and optimizes:
# - Pushes filter to the Parquet reader (predicate pushdown)
# - Reads only needed columns (column pruning)
# - Only 1 GB actually enters memory
# Total memory touched: ~1 GB

result = txns.filter(col("amount") > 10000) \
    .select("account_id", "amount")
result.explain(True)
# ReadSchema shows only account_id and amount (pruned!)
# PushedFilters shows amount > 10000 (pushed!)

How does Spark differ from Pandas?

The differences are fundamental and affect when you use each.

| Feature | Spark DataFrame | Pandas DataFrame |
|---|---|---|
| Execution | Lazy | Eager |
| Distribution | Across cluster | Single machine |
| Mutability | Immutable | Mutable |
| Data size | Terabytes | Gigabytes (RAM limit) |
| Optimization | Catalyst optimizer | None (manual) |
| Null handling | dropna, fillna, when/otherwise | dropna, fillna, np.where |
| Speed on small data | Slower (cluster overhead) | Faster (no overhead) |

PYTHON
import pandas as pd

# Pandas: eager, single-machine
pdf = pd.read_csv("/data/small_transactions.csv")  # Runs immediately
pdf_filtered = pdf[pdf["amount"] > 10000]  # Runs immediately

# Spark: lazy, distributed
sdf = spark.read.parquet("/data/transactions")  # Returns a handle
sdf_filtered = sdf.filter(col("amount") > 10000)  # Builds plan
sdf_filtered.count()  # NOW it runs

How do you handle nulls in Spark?

Spark provides dropna, fillna, and when/otherwise for null handling and conditional column logic.

PYTHON
from pyspark.sql.functions import when, col, lit

txns = spark.read.parquet("/data/transactions")

# Drop rows with any null
cleaned = txns.dropna()

# Drop rows where specific columns are null
cleaned_specific = txns.dropna(subset=["account_id", "amount"])

# Fill nulls with a default value
filled = txns.fillna({"amount": 0, "account_id": "UNKNOWN"})

# Conditional column with when/otherwise
flagged = txns.withColumn("status",
    when(col("balance") < 0, lit("OVERDRAWN"))
    .when(col("balance") < 1000, lit("LOW_BALANCE"))
    .otherwise(lit("OK")))
flagged.show(5)
TEXT
+----------+--------+-------+-----------+
|account_id|  amount|balance|     status|
+----------+--------+-------+-----------+
|   SAV2001|50000.00| 125000|         OK|
|   CUR3001|75000.00|   -500|  OVERDRAWN|
|   SAV2002|25000.00|    800|LOW_BALANCE|
+----------+--------+-------+-----------+

Why does each action re-run the plan?

Spark does not store intermediate results by default. Each action triggers a fresh execution of the full plan. Cache to avoid recomputation.

PYTHON
# Without caching: each action recomputes from scratch
expensive = txns.filter(col("amount") > 10000) \
    .groupBy("account_id").agg(spark_sum("amount"))

expensive.count()   # Full computation
expensive.show(5)   # Full computation AGAIN

# With caching: compute once, reuse
expensive.cache()
expensive.count()   # Computes and caches
expensive.show(5)   # Reads from cache (fast)
expensive.unpersist()

What are narrow vs wide transformations?

Narrow transformations process each partition independently (no shuffle). Wide transformations require redistributing data across the network.

PYTHON
# Narrow (no shuffle): filter, select, withColumn, map
narrow = txns.filter(col("amount") > 0).select("account_id", "amount")

# Wide (shuffle): groupBy, join, distinct, orderBy, repartition
wide = narrow.groupBy("account_id").agg(spark_sum("amount"))

# Minimize wide transformations to reduce shuffle cost

When should you use Spark vs Pandas?

The choice depends on data size and the use case.

PYTHON
# Small interactive analysis: Pandas
pdf = spark.read.parquet("/data/sample").limit(10000).toPandas()
# Work interactively with pdf in Pandas

# Production on full dataset: Spark
result = spark.read.parquet("/data/transactions") \
    .filter(col("amount") > 10000) \
    .groupBy("account_id").sum("amount")
result.write.parquet("/output/high_value_totals")

# PySpark Pandas API: best of both worlds
import pyspark.pandas as ps
psdf = ps.read_parquet("/data/transactions")
# Pandas-like syntax, Spark execution

What should a senior engineer know?

  • Lazy evaluation is not just a design choice; it enables optimizations that eager execution cannot achieve.
  • At the National Bank of India, prototyping on Pandas samples and scaling to Spark for production was the standard workflow.
  • Cache aggressively when a DataFrame is reused by multiple actions to avoid recomputation.
  • Minimize wide transformations to reduce shuffle cost; each one creates a stage boundary.
  • The PySpark Pandas API lets teams with Pandas skills scale to big data without learning the DataFrame API from scratch.

Visual Overview

DIAGRAM
  Lazy Evaluation Flow

  txns.filter(amount > 10000)    -> Logical Plan: Filter
    .select("account_id")        -> Logical Plan: Project
    .groupBy("account_id")       -> Logical Plan: Aggregate
    .sum("amount")               -> Logical Plan: Sum

  Nothing executed yet!

  .show()  <-- ACTION triggers execution
       │
       ▼
  Catalyst Optimizer:
    1. Push filter to Parquet reader
    2. Prune unused columns
    3. Choose optimal join strategy
       │
       ▼
  Physical Execution:
    Stage 0: Scan + Filter + Select (pipelined)
    Stage 1: Shuffle + GroupBy + Sum
    Stage 2: Show result
DIAGRAM
  Spark vs Pandas

  Pandas (single machine, eager):
  ┌──────────────────────────┐
  │  One desk, one accountant │
  │  Runs each step NOW       │
  │  Limited by desk size     │
  │  (machine RAM)            │
  └──────────────────────────┘

  Spark (distributed, lazy):
  ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
  │Desk 1│ │Desk 2│ │Desk 3│ │Desk 4│
  │      │ │      │ │      │ │      │
  └──────┘ └──────┘ └──────┘ └──────┘
  Manager writes plan first, optimizes,
  THEN sends team to work in parallel

Key Takeaway

A Spark DataFrame is a distributed, immutable, lazily-evaluated table of rows spread across a cluster. Transformations like select, filter, withColumn, and join do not compute anything; they record operations into a logical plan. Execution happens only when an action like count, show, or write is called, at which point Catalyst sees the entire chain and optimizes it with predicate pushdown, column pruning, and join reordering.

The contrast with Pandas is fundamental. Pandas is single-machine, in-memory, and eager, executing each operation immediately. Spark distributes across a cluster and defers execution, scaling to terabytes but adding cluster overhead that makes it slower for small data. For nulls, Spark uses dropna, fillna, and when/otherwise for conditional column logic.

At the National Bank of India, exploratory analysis on a sample used Pandas, but every production job over the full transaction history used Spark DataFrames. A typical pipeline chained fillna, filter, and when/otherwise for flagging, and because the transformations were lazy, Catalyst optimized the entire chain and only executed on the final write. Caching was used whenever multiple actions reused the same intermediate result.

The practical rules are: use lazy transformations to let Catalyst optimize, cache when reusing across actions, minimize wide transformations to reduce shuffle cost, and choose Pandas for small interactive work and Spark when data outgrows one machine.

Remember one sentence: Spark DataFrames are lazy and distributed so Catalyst can optimize the whole plan, while Pandas DataFrames are eager and single-machine so they execute each step immediately.

---

Share
Use ← → arrow keys to navigate