What is Apache Spark

• Distributed engine – Spark splits data into partitions and processes them in parallel across many machines in a cluster.
• In-memory computation – intermediate results stay in memory between stages instead of being written to disk, which is why Spark is up to 100 times faster than MapReduce for iterative workloads.
• Lazy evaluation – transformations like filter and select only build a logical plan; nothing runs until an action like count or write is called.
• DAG scheduler – Spark compiles the plan into a directed acyclic graph of stages and optimizes it before execution.
• Fault tolerance via lineage – every dataset records how it was derived, so a lost partition is recomputed from its parents rather than restored from a backup.
• Unified platform – Spark SQL, Structured Streaming, MLlib, and GraphX all run on the same engine.

What is Apache Spark?

Apache Spark is an open-source distributed computing engine designed to process very large datasets in parallel across a cluster of machines. It was created at UC Berkeley's AMPLab in 2009 and donated to the Apache Software Foundation. Spark provides a unified analytics engine with libraries for SQL, streaming, machine learning, and graph processing, all running on one runtime.

At the National Bank of India, we use Spark to process billions of daily transactions, compute risk aggregates, and train fraud-detection models. A single machine cannot handle this volume, so Spark splits the work across hundreds of executor cores.

# PySpark: create a SparkSession for the bank's analytics
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("NationalBankDailySettlement") \
    .config("spark.sql.shuffle.partitions", "200") \
    .getOrCreate()

# Read the day's transactions from the data lake
txns = spark.read.parquet("s3://nbi-data-lake/transactions/2024-01-15/")
print(f"Partitions: {txns.rdd.getNumPartitions()}")
print(f"Row count: {txns.count()}")

Partitions: 200
Row count: 48000000

Why is Spark faster than Hadoop MapReduce?

MapReduce writes the output of every map and reduce step to HDFS disk, so a multi-step job pays a heavy I/O penalty at each stage boundary. Spark keeps intermediate results in memory between stages and only spills to disk when memory is tight. For iterative algorithms that pass over the same data many times, such as training a fraud model, this in-memory approach can be 10 to 100 times faster.

Spark also uses a DAG (directed acyclic graph) scheduler instead of the rigid two-phase map-reduce model. The DAG lets Spark fuse multiple transformations into a single stage and avoid redundant data passes.

from pyspark.sql import functions as F

# MapReduce would write to disk after each step.
# Spark chains these lazily and runs them in one fused stage.
settled = txns \
    .filter(F.col("status") == "SETTLED") \
    .filter(F.col("amount") > 0) \
    .select("accountId", "amount", "branch")

# Nothing has executed yet. Spark only built a plan.
# The action below triggers the optimized DAG.
total = settled.groupBy("branch").agg(F.sum("amount").alias("branchTotal"))
total.show(5)

+--------+-------------+
|  branch|  branchTotal|
+--------+-------------+
|  Mumbai| 245000000.00|
|   Delhi| 198000000.00|
| Chennai| 167000000.00|
|Kolkata | 143000000.00|
|   Pune |  98000000.00|
+--------+-------------+

How does lazy evaluation work?

Transformations like filter, select, groupBy, and join do not run immediately. They build a logical plan, which is a description of what to compute. Only when you call an action like count, show, collect, or write does Spark hand the plan to the Catalyst optimizer, compile it into a physical plan, and execute it as a DAG of stages.

This laziness lets Spark see the entire computation before running anything, so it can push filters down to the data source, prune unneeded columns, and fuse operations to avoid materializing intermediate datasets.

# All transformations below are lazy: no data moves yet.
plan = txns \
    .filter(F.col("txnDate") == "2024-01-15") \
    .filter(F.col("amount") > 1000) \
    .select("accountId", "amount")

# Inspect the optimized logical plan before running.
plan.explain("formatted")

# Now trigger execution with an action.
plan.count()

== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet [accountId,amount]
     PushedFilters: [IsNotNull(amount), GreaterThan(amount,1000)]
     ReadSchema: struct<accountId:string,amount:decimal(18,2)>

How does Spark achieve fault tolerance through lineage?

Every RDD and DataFrame records the sequence of transformations that produced it, called the lineage. If an executor dies and takes a cached partition with it, Spark does not fail the job. Instead, it finds the lineage for the lost partition and recomputes only that partition from its parent data.

This is far cheaper than replicating every intermediate result to multiple nodes the way traditional databases do. At the bank, a failed node during a nightly reconciliation causes a small recompute of the lost partitions, not a restart of the entire eight-hour job.

# Spark records lineage automatically.
# If a node fails during this pipeline, only the lost partitions
# are recomputed from the parquet source.
accountTotals = txns \
    .filter(F.col("status") == "SETTLED") \
    .groupBy("accountId") \
    .agg(F.sum("amount").alias("total"))

# Print the lineage (logical plan)
accountTotals.explain("extended")

What are the core abstractions: RDD, DataFrame, Dataset?

Spark's original abstraction is the RDD (Resilient Distributed Dataset), an immutable, partitioned collection of records. RDDs are low-level and give you full control but no schema or optimization.

The DataFrame API adds a schema (column names and types) on top of RDDs and lets the Catalyst optimizer apply SQL-style optimizations. The Dataset API (Scala and Java only) adds compile-time type safety. In PySpark, you work almost exclusively with DataFrames.

• RDD – low-level, untyped, no optimization. Use for custom partitioning or legacy code.
• DataFrame – structured, schema-aware, optimized by Catalyst. The standard API.
• Dataset – typed DataFrame (Scala/Java). Compile-time safety plus Catalyst optimization.

# DataFrame: the standard way to work with structured bank data
df = spark.read.parquet("s3://nbi-data-lake/accounts/")
df.printSchema()

root
 |-- accountId: string
 |-- customerName: string
 |-- accountType: string
 |-- balance: decimal(18,2)
 |-- branch: string

What is a shuffle and why is it expensive?

A shuffle happens when an operation like groupBy, join, or repartition requires data to move across the network so that rows with the same key land on the same partition. Spark writes shuffle data to local disk, transfers it over the network, and reads it back on the receiving executors. This is the most expensive operation in Spark.

At the bank, joining the billion-row transactions table to the accounts table on accountId requires a shuffle of both sides unless one is small enough to broadcast. Minimizing and tuning shuffles is the single most impactful performance lever.

# This groupBy triggers a shuffle: rows with the same accountId
# must land on the same partition to be summed together.
perAccount = txns.groupBy("accountId").agg(
    F.sum("amount").alias("totalAmount"),
    F.count("*").alias("txnCount")
)
perAccount.show(3)

+----------+-----------+--------+
| accountId|totalAmount|txnCount|
+----------+-----------+--------+
|SAV-100001|   45230.50|      12|
|CUR-200042|  892100.00|      47|
|SAV-100099|   12890.75|       5|
+----------+-----------+--------+

What does Spark's unified platform include?

Spark is not just a batch engine. It ships four libraries on one runtime, so the same cluster, the same code style, and the same skills cover multiple workloads.

• Spark SQL – run SQL queries and work with DataFrames on structured data.
• Structured Streaming – process real-time data streams as unbounded tables.
• MLlib – train and serve machine learning models at scale.
• GraphX – analyze graph-structured data like social networks or transaction graphs.

# Spark SQL: register a DataFrame as a SQL table
txns.createOrReplaceTempView("transactions")

result = spark.sql("""
    SELECT branch, SUM(amount) AS total, COUNT(*) AS cnt
    FROM transactions
    WHERE status = 'SETTLED'
    GROUP BY branch
    ORDER BY total DESC
    LIMIT 5
""")
result.show()

+--------+--------------+------+
|  branch|         total|   cnt|
+--------+--------------+------+
|  Mumbai|245000000.00  |980000|
|   Delhi|198000000.00  |820000|
| Chennai|167000000.00  |710000|
|Kolkata |143000000.00  |590000|
|    Pune| 98000000.00  |430000|
+--------+--------------+------+

How is Spark different from Hadoop MapReduce?

| Aspect | Apache Spark | Hadoop MapReduce |
|---|---|---|
| Intermediate data | In-memory between stages | Written to HDFS disk after each step |
| Execution model | DAG of stages, many operators | Rigid two-phase map then reduce |
| Speed (iterative) | 10x to 100x faster | Slow due to repeated disk I/O |
| APIs | DataFrame, SQL, Streaming, ML | Java MapReduce API |
| Fault tolerance | Lineage-based recomputation | HDFS replication |
| Real-time | Structured Streaming (micro-batch) | Not designed for streaming |
| Ease of use | Python, Scala, Java, R, SQL | Primarily Java |

# In MapReduce, this would be two separate jobs with disk I/O between them.
# In Spark, the entire pipeline runs as one optimized DAG.
result = txns \
    .filter(F.col("status") == "SETTLED") \
    .groupBy("accountId") \
    .agg(F.sum("amount").alias("total")) \
    .filter(F.col("total") > 100000) \
    .orderBy(F.desc("total"))

result.write.parquet("s3://nbi-data-lake/high-value-accounts/")

When should you use Spark and when should you not?

Spark excels at large-scale batch processing, iterative machine learning, interactive SQL analytics, and near-real-time streaming. At the bank, we use it for nightly settlement reconciliation, daily risk aggregation, and training fraud models.

Spark is not the right tool for single-row transactional lookups (use a database), for sub-millisecond latency requirements (use Apache Flink or a custom streaming system), or for small datasets that fit on one machine (use pandas or plain SQL).

# Good Spark use case: aggregate billions of rows across a cluster
daily_risk = txns \
    .filter(F.col("txnDate") == "2024-01-15") \
    .groupBy("riskCategory") \
    .agg(
        F.sum("amount").alias("totalExposure"),
        F.count("*").alias("txnCount"),
        F.max("amount").alias("largestTxn")
    )
daily_risk.show()

+------------+--------------+--------+----------+
|riskCategory| totalExposure|txnCount|largestTxn|
+------------+--------------+--------+----------+
|         LOW|1200000000.00 | 4200000| 50000.00 |
|      MEDIUM| 340000000.00 |  980000|250000.00 |
|        HIGH|  45000000.00 |   12000|900000.00 |
+------------+--------------+--------+----------+

What should a senior engineer know about Spark?

A senior engineer understands that the expensive part of any Spark job is the shuffle, and tunes partition counts, join strategies, and caching to minimize it. They know the Catalyst optimizer's query plan, read it with explain, and verify the physical plan matches expectations.

They also know that Spark's memory model splits executor memory into execution memory (for shuffles, joins, sorts) and storage memory (for caching), and that the two can borrow from each other under the unified memory manager. Tuning executor count, cores, and memory per executor to match the workload is the difference between a 20-minute job and a 2-hour job.

# Senior practice: inspect the physical plan before running
txns.join(accounts, "accountId") \
    .groupBy("branch") \
    .agg(F.sum("amount").alias("total")) \
    .explain("formatted")