Definition
- Transformation – a lazy operation that takes an RDD or DataFrame and returns a new one, building the logical plan without executing (e.g., filter, map, join, select, distinct).
- Action – an eager operation that triggers execution of the accumulated plan and returns a result or writes output (e.g., collect, count, show, write).
- Lazy evaluation – the principle that transformations only record intent until an action forces the optimized plan to run.
- Catalyst optimizer – Spark's query planner that benefits from lazy evaluation to push filters, prune columns, and reorder joins.
- Narrow transformation – no data movement needed (map, filter); each output partition depends on one input partition.
- Wide transformation – requires a shuffle across the network (groupByKey, join).
Imagine a manager at the National Bank of India writing a to-do list for processing statements: filter Rajesh's transactions, convert amounts to rupees, group by month. Writing each item on the list is a transformation: it costs nothing, the clerks are still sitting idle. Only when the manager says "now go and total it up and print the report", that command is an action, do the clerks spring into motion and do all the listed steps at once. Because the manager wrote the full list before anyone moved, the steps could be reordered and merged for efficiency.
Understanding the Concepts
What is a transformation?
A transformation never runs on its own. When you call filter, map, flatMap, groupByKey, join, select, sorted, or distinct, Spark extends the lineage graph with a new node and hands you back a new RDD or DataFrame. Nothing is read, nothing is computed, no executor does any work.
# Transformations: nothing executes yet
df = spark.read.parquet(
"s3://national-bank-lake/curated/transactions/")
filtered = df.filter("amount > 10000") # lazy
selected = filtered.select("account_id", "amount") # lazy
grouped = selected.groupBy("account_id") # lazy
summed = grouped.sum("amount") # lazy
print("No executor has done any work yet.")
No executor has done any work yet.
What is an action?
An action forces the whole lineage to execute. When you call collect, count, show, saveAsTextFile, or write, Spark hands the plan to the Catalyst optimizer, builds stages and tasks, and runs them across executors.
# Action: triggers execution of the entire chain
summed.show(5) # THIS is where Spark actually runs
+----------+-----------+
|account_id|sum(amount)|
+----------+-----------+
| CUR3002| 1250000.00|
| SAV2001| 450000.00|
+----------+-----------+
package com.bank.spark;
public class TransformVsAction {
public static void main(String[] args) {
System.out.println("Transformations (lazy):");
System.out.println(" filter, map, flatMap, join, select,"
+ " groupByKey, sorted, distinct");
System.out.println(" Return type: DataFrame / RDD");
System.out.println();
System.out.println("Actions (eager, trigger execution):");
System.out.println(" collect, count, show, write,"
+ " saveAsTextFile");
System.out.println(" Return type: value, collection, or void");
}
}
Transformations (lazy):
filter, map, flatMap, join, select, groupByKey, sorted, distinct
Return type: DataFrame / RDD
Actions (eager, trigger execution):
collect, count, show, write, saveAsTextFile
Return type: value, collection, or void
Why does lazy evaluation exist?
Lazy evaluation exists for optimization. Because Spark sees the full chain before executing, the Catalyst optimizer can push a filter down so less data is read from disk, prune columns you never use, combine adjacent maps into a single pass, and reorder joins.
# Lazy evaluation allows Catalyst to optimize
df = spark.read.parquet("s3://bank-lake/transactions/")
result = df.select("account_id", "amount") \
.filter("amount > 10000") \
.groupBy("account_id").sum("amount")
result.explain(True)
# Catalyst pushes filter BEFORE the select
# and prunes unused columns at the scan
== Optimized Logical Plan ==
Aggregate [account_id], [sum(amount)]
+- Filter (amount > 10000)
+- FileScan parquet [account_id, amount] <-- only 2 columns, filter pushed
What is the difference between narrow and wide transformations?
Narrow transformations like map and filter need no data movement because each output partition depends on one input partition. Wide transformations like groupByKey and join require a shuffle that moves data across the network.
package com.bank.spark;
public class NarrowVsWide {
public static void main(String[] args) {
System.out.println("Narrow (no shuffle):");
System.out.println(" map, filter, flatMap, mapPartitions");
System.out.println(" Partition 0 -> Partition 0");
System.out.println();
System.out.println("Wide (shuffle):");
System.out.println(" groupByKey, reduceByKey, join, distinct");
System.out.println(" All partitions -> hash -> new partitions");
System.out.println(" Data crosses the network.");
System.out.println(" Stage boundary in the DAG.");
}
}
Narrow (no shuffle):
map, filter, flatMap, mapPartitions
Partition 0 -> Partition 0
Wide (shuffle):
groupByKey, reduceByKey, join, distinct
All partitions -> hash -> new partitions
Data crosses the network.
Stage boundary in the DAG.
What happens when you call multiple actions?
Each action re-triggers the entire lineage from the beginning. If you call two actions on the same uncached chain, Spark recomputes everything twice. This is a common performance mistake.
# WRONG: two actions, double computation
df = spark.read.parquet("s3://bank-lake/transactions/")
filtered = df.filter("amount > 10000")
print(filtered.count()) # Action 1: full scan
filtered.show(5) # Action 2: full scan AGAIN
# CORRECT: cache before multiple actions
filtered.cache()
print(filtered.count()) # Action 1: full scan, cached
filtered.show(5) # Action 2: reads from cache
filtered.unpersist() # Clean up
package com.bank.spark;
public class CacheBeforeMultipleActions {
public static void main(String[] args) {
System.out.println("Without cache:");
System.out.println(" count() -> full scan");
System.out.println(" show() -> full scan AGAIN");
System.out.println();
System.out.println("With cache:");
System.out.println(" cache()");
System.out.println(" count() -> scan, result cached");
System.out.println(" show() -> reads from cache");
System.out.println(" unpersist() -> clean up");
}
}
Without cache:
count() -> full scan
show() -> full scan AGAIN
With cache:
cache()
count() -> scan, result cached
show() -> reads from cache
unpersist() -> clean up
What are common mistakes with lazy evaluation?
Wrong: counting after every step for debugging
# WRONG: unnecessary actions slow the pipeline
df = spark.read.parquet("s3://bank-lake/transactions/")
print(df.count()) # Action 1: full scan
filtered = df.filter("amount > 10000")
print(filtered.count()) # Action 2: full scan + filter
enriched = filtered.join(customers, "account_id")
print(enriched.count()) # Action 3: full scan + filter + join
enriched.write.parquet("s3://bank-lake/output/") # Action 4
# 4 actions = 4 full scans!
Correct: one write at the end
# CORRECT: chain transformations, one action
df = spark.read.parquet("s3://bank-lake/transactions/")
result = df.filter("amount > 10000") \
.join(customers, "account_id")
result.write.parquet("s3://bank-lake/output/")
# 1 action = 1 scan
How do you identify transformations vs actions?
A quick rule: if the return type is another RDD or DataFrame, it is a transformation. If the return type is a value, a collection in the Driver, or Unit from writing, it is an action.
package com.bank.spark;
public class IdentifyRule {
public static void main(String[] args) {
System.out.println("Return type -> category:");
System.out.println(" DataFrame -> transformation (lazy)");
System.out.println(" Long (count) -> action (eager)");
System.out.println(" Array (collect) -> action (eager)");
System.out.println(" Unit (write) -> action (eager)");
}
}
Return type -> category:
DataFrame -> transformation (lazy)
Long (count) -> action (eager)
Array (collect) -> action (eager)
Unit (write) -> action (eager)
What are common interview follow-up questions?
- What is the difference between map and flatMap?
- What happens if you call collect on a very large dataset?
- How does caching interact with lazy evaluation?
- What is the difference between cache and persist?
- How do you force Spark to evaluate a transformation without returning data (e.g., for side effects)?
Visual Overview
Lazy Evaluation Flow
────────────────────
Transformations (lazy: build plan) Action (eager: execute)
┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
│ filter │─▶│ select │─▶│groupBy │──────▶│ show() │
│ (lazy) │ │ (lazy) │ │ (lazy) │ │(ACTION)│
└────────┘ └────────┘ └────────┘ └───┬────┘
│
Nothing executes until show() is called. │
▼
┌──────────────────┐
│ Catalyst Optimizer│
│ Push filter down │
│ Prune columns │
│ Fuse maps │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Execute on cluster│
│ (stages + tasks) │
└──────────────────┘
Key Takeaway
Spark operations come in two categories, and knowing the difference is fundamental to writing efficient jobs. Transformations are lazy: operations like filter, map, join, select, and distinct return a new dataset but execute nothing, just extending the logical plan. Actions are eager: operations like collect, count, show, and write trigger execution and either return a result or write output. A simple rule is that if the return type is a DataFrame, it is a transformation; if it returns a value or writes, it is an action.
Lazy evaluation exists for optimization. Because Spark sees the entire chain before executing, the Catalyst optimizer can push filters down, prune unused columns, fuse adjacent operations, and reorder joins. If every transformation ran eagerly, Spark would materialize huge intermediate results and lose all optimizations. The subtle consequence is that every action re-triggers the whole lineage, so calling several actions on the same uncached chain recomputes everything each time.
Within transformations, narrow ones (map, filter) need no shuffle, while wide ones (groupByKey, join) require a shuffle across the network, creating a stage boundary in the DAG.
In a banking context at the National Bank of India, this understanding directly shaped performance. Long chains of transformations were chained freely because they cost nothing, but actions were counted deliberately: ideally one write at the end rather than a count or show after every step. When multiple actions were genuinely needed, the intermediate result was cached first so Spark computed it once.
Remember one sentence: transformations are lazy and build the plan, actions are eager and run it, and lazy evaluation lets Catalyst optimize the whole chain before any work happens.
---