Definition
- Apache Spark – an open-source distributed computing engine for large-scale data processing, created at UC Berkeley to overcome the performance limits of Hadoop MapReduce.
- In-memory processing – keeps intermediate data in memory across stages, spilling to disk only when memory is tight, making iterative jobs dramatically faster.
- Lazy evaluation – transformations build a logical plan; the Catalyst optimizer optimizes it as a whole before an action triggers execution.
- Fault tolerance via lineage – records the sequence of transformations so a lost partition is recomputed, not restored from a saved copy.
- Four unified libraries – Spark SQL (structured queries), Structured Streaming (near-real-time), MLlib (machine learning), GraphX (graph processing).
Imagine the National Bank of India processing mountains of data with the old MapReduce method, where after every step the clerks had to file everything back into slow filing cabinets and fetch it again. Spark keeps the working papers on the desk in memory and only files to the cabinet when truly necessary, so multi-step jobs fly. And Spark is a four-in-one toolkit on the same desk: one tool for SQL questions, one for live streaming data, one for machine learning, and one for relationship graphs.
Understanding the Concepts
What is Apache Spark?
Apache Spark is an open-source distributed computing engine for large-scale data processing. It was created at UC Berkeley's AMPLab to solve a specific performance problem with Hadoop MapReduce.
from pyspark.sql import SparkSession
# Create a Spark session at the National Bank of India
spark = SparkSession.builder \
.appName("NationalBankAnalytics") \
.config("spark.sql.adaptive.enabled", "true") \
.getOrCreate()
# Read and process millions of transactions in parallel
txns = spark.read.parquet("/data/transactions")
print(f"Total transactions: {txns.count()}")
Why was Spark created?
MapReduce writes the output of every map and reduce stage to disk. For a single-pass job this is acceptable, but for iterative algorithms that pass over the data repeatedly – like machine learning training – the disk I/O cost is enormous.
MapReduce: disk I/O after every step
Step 1 -> Write to Disk -> Read from Disk -> Step 2 -> Write to Disk -> ...
Each iteration pays full disk I/O cost
Spark: in-memory across stages
Step 1 -> In Memory -> Step 2 -> In Memory -> Step 3
Only spills to disk when memory is tight
# Example: iterative ML training that benefits from in-memory caching
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import VectorAssembler
# Prepare features for fraud detection model
assembler = VectorAssembler(
inputCols=["amount", "hour_of_day", "merchant_risk_score"],
outputCol="features")
train_data = assembler.transform(txns).select("features", "is_fraud")
train_data.cache() # Keep in memory for iterative training
# Logistic regression iterates over the data many times
# With MapReduce: each iteration reads from disk (slow)
# With Spark: each iteration reads from memory (fast)
lr = LogisticRegression(maxIter=100, labelCol="is_fraud")
model = lr.fit(train_data)
train_data.unpersist()
How does lazy evaluation work?
Transformations do not execute immediately. They build a logical plan that the Catalyst optimizer can analyze and optimize as a whole before any data moves.
from pyspark.sql.functions import col, sum as spark_sum
# These transformations are lazy: no data moves yet
filtered = txns.filter(col("amount") > 10000)
selected = filtered.select("account_id", "amount")
totals = selected.groupBy("account_id").agg(
spark_sum("amount").alias("total"))
# Nothing has executed yet! Catalyst builds and optimizes the plan:
# - Pushes the filter to the Parquet reader (predicate pushdown)
# - Prunes unused columns (only reads account_id and amount)
# - Chooses the optimal join strategy
# Only an ACTION triggers execution:
totals.show(5) # NOW Spark reads, filters, groups, and shows
How does fault tolerance through lineage work?
Spark records the sequence of transformations that built each dataset. If a node fails and a partition is lost, Spark recomputes that partition from its lineage rather than restoring from a checkpoint.
# Lineage example
raw = spark.read.parquet("/data/raw_transactions")
cleaned = raw.filter(col("amount").isNotNull())
enriched = cleaned.withColumn("fee", col("amount") * 0.001)
# If a node fails while processing partition 5 of 'enriched',
# Spark rebuilds it by:
# 1. Re-reading partition 5 of /data/raw_transactions
# 2. Applying the filter
# 3. Applying the fee calculation
# No checkpoint needed; lineage provides the recipe
# View the lineage:
enriched.explain(extended=True)
What are the four unified libraries?
All four libraries run on the same engine and share the same data abstractions, so you use one runtime and one skill set for very different workloads.
# 1. Spark SQL: structured data processing
account_summary = spark.sql("""
SELECT account_id, SUM(amount) as total, COUNT(*) as txn_count
FROM transactions
GROUP BY account_id
HAVING total > 100000
""")
# 2. Structured Streaming: near-real-time processing
stream = spark.readStream \
.format("kafka") \
.option("subscribe", "bank-transactions") \
.load()
stream.writeStream \
.format("delta") \
.option("checkpointLocation", "/checkpoints/txns") \
.start("/data/lake/transactions")
# 3. MLlib: scalable machine learning
from pyspark.ml.classification import RandomForestClassifier
rf = RandomForestClassifier(numTrees=100, labelCol="is_fraud")
fraud_model = rf.fit(train_data)
# 4. GraphX: graph-parallel computation (Scala API)
# Used for detecting rings of linked accounts in fraud analysis
How does Spark compare to MapReduce?
| Feature | MapReduce | Spark |
|---|---|---|
| Intermediate data | Disk after every step | In memory across stages |
| Iterative jobs | Slow (disk I/O per iteration) | Fast (memory reuse) |
| Optimization | None (manual) | Catalyst optimizer |
| Fault tolerance | Replication | Lineage-based recomputation |
| APIs | Map and Reduce only | SQL, DataFrame, Streaming, ML, Graph |
| Language support | Java | Scala, Python, Java, R, SQL |
# The speed difference on iterative workloads can be 10-100x
# because Spark avoids the disk round-trip that MapReduce requires
# on every iteration
What about the RDD vs DataFrame APIs?
Spark originally used RDDs (Resilient Distributed Datasets) as its core abstraction. The higher-level DataFrame and Dataset APIs added schema awareness and the Catalyst optimizer.
# RDD API (low-level, type-safe but no optimization)
rdd = spark.sparkContext.parallelize([("SAV2001", 50000), ("CUR3001", 75000)])
totals_rdd = rdd.reduceByKey(lambda a, b: a + b)
# DataFrame API (high-level, optimized by Catalyst)
df = spark.createDataFrame([("SAV2001", 50000), ("CUR3001", 75000)],
["account_id", "amount"])
totals_df = df.groupBy("account_id").sum("amount")
# The DataFrame version is usually faster because Catalyst optimizes it
# and Tungsten generates efficient bytecode
What should a senior engineer know?
- Spark runs on cluster managers like YARN, Kubernetes, and Mesos, and reads from HDFS, S3, and cloud storage.
- The unification matters practically: a single pipeline can read with Spark SQL, train with MLlib, and serve with Structured Streaming without stitching separate systems.
- While Flink may beat Spark on very low-latency streaming, Spark's combination of in-memory speed, a mature optimizer, and one unified platform is why it became the default for banking data platforms.
- At the National Bank of India, Spark SQL totalled transactions, Structured Streaming powered real-time enrichment, MLlib trained fraud models, and GraphX detected rings of linked accounts – all on the same cluster and the same transaction data.
Visual Overview
MapReduce vs Spark: Iterative Processing
MapReduce (disk after every step):
┌──────┐ disk ┌──────┐ disk ┌──────┐ disk ┌──────┐
│Step 1│──────>│Step 2│──────>│Step 3│──────>│Step 4│
└──────┘ write └──────┘ write └──────┘ write └──────┘
read read read
Spark (in-memory across stages):
┌──────┐ mem ┌──────┐ mem ┌──────┐ mem ┌──────┐
│Step 1│──────>│Step 2│──────>│Step 3│──────>│Step 4│
└──────┘ └──────┘ └──────┘ └──────┘
No disk I/O between steps!
Spark Unified Ecosystem ┌──────────────────────────────────────────────┐ │ Spark Core Engine │ │ (In-memory, DAG scheduler, fault tolerance)│ ├────────────┬────────────┬──────────┬─────────┤ │ Spark SQL │ Structured │ MLlib │ GraphX │ │ DataFrames │ Streaming │ ML Pipe │ Graph │ │ Catalyst │ Micro-batch│ lines │ Pregel │ │ Optimizer │ + Cont. │ │ │ └────────────┴────────────┴──────────┴─────────┘
Key Takeaway
Apache Spark is an open-source distributed computing engine created to overcome the performance limits of Hadoop MapReduce. MapReduce writes intermediate results to disk after every step, so iterative algorithms like machine learning training pay enormous disk I/O on every iteration. Spark keeps intermediate data in memory across stages, spilling to disk only under pressure, making multi-stage workloads dramatically faster.
Two further design choices power this. Lazy evaluation means transformations build a logical plan that the Catalyst optimizer can analyze and optimize as a whole before an action triggers execution, enabling predicate pushdown, column pruning, and join reordering automatically. Fault tolerance through lineage records the sequence of transformations so a lost partition is recomputed rather than restored from a saved copy.
The other half of Spark's value is its unified ecosystem of four libraries on one engine. Spark SQL handles structured data with DataFrames and the Catalyst optimizer. Structured Streaming handles near-real-time unbounded data. MLlib provides scalable machine learning. GraphX provides graph-parallel computation. All four share the same data abstractions, so a single pipeline can read, train, and serve without stitching separate systems.
At the National Bank of India, Spark anchored the data platform because one engine handled batch analytics, real-time enrichment, fraud model training, and linked-account graph detection over the same transaction data.
Remember one sentence: Spark is a fast, in-memory, unified engine that replaced MapReduce's disk-bound approach with lazy evaluation, lineage-based fault tolerance, and four integrated libraries for SQL, streaming, ML, and graphs.
---