Definition
- Partitions – the list of chunks the data is split into, setting the unit of parallelism.
- Compute function – tells Spark how to produce each partition on demand, either by reading storage or applying a transformation.
- Dependencies – the lineage chain of parent RDDs, either narrow (no shuffle) or wide (shuffle boundary).
- Partitioner – an optional key-to-partition mapping such as
HashPartitionerorRangePartitionerfor key-value RDDs. - Preferred locations – optional hints per partition for data locality, guiding the scheduler to place tasks where data already lives.
Picture the National Bank of India's archive of transaction records. The list of partitions is how the archive is divided into boxes. The computation function is the instruction slip on each box telling the clerk how to produce that box's contents. The dependencies are the chain of previous boxes a box was derived from, so you can rebuild it if lost. The Partitioner is the filing rule that decides which box a given account's records go into, so all of one account's records land together. And the preferred locations are notes saying 'this box is already on the third floor, so send a clerk who is already there' instead of hauling it across the building.
Understanding the Concepts
What are the five internal properties of an RDD?
Every RDD in Spark is defined by exactly five properties. These properties are not hidden internals reserved for framework developers. They are the contract that every RDD must implement, and they explain how Spark schedules, executes, and recovers distributed computations. When you understand these five properties, you can reason about parallelism, shuffles, data locality, and fault tolerance from first principles.
- Partitions – the chunks the data is split into.
- Compute function – how to produce each partition.
- Dependencies – the lineage of parent RDDs.
- Partitioner – optional key-to-partition mapping.
- Preferred locations – optional data-locality hints.
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
txn_rdd = sc.textFile("hdfs:///bank/transactions/2026-01-15.csv")
# Property 1: list of partitions
print("Partition count:", txn_rdd.getNumPartitions())
# Property 2: compute function (internal, not called directly)
# Each partition reads its HDFS block via the compute function.
# Property 3: dependencies
filtered = txn_rdd.filter(lambda line: "HDFC" in line) # narrow dependency
print("Dependencies:", filtered.dependencies)
# Property 4: partitioner (None for non-key-value RDDs)
print("Partitioner:", txn_rdd.partitioner)
# Property 5: preferred locations
print("Preferred locs for partition 0:", txn_rdd.preferredLocations(txn_rdd.partitions()[0]))
Partition count: 8
Dependencies: [<pyspark.rdd.NarrowDependency ...>]
Partitioner: None
Preferred locs for partition 0: ['datanode1.bank.local']
How does the list of partitions drive parallelism?
The first property is the list of partitions. Each partition is an independent chunk of data, and Spark creates exactly one task per partition per stage. So the partition count directly controls the degree of parallelism. Too few partitions under-utilise the cluster; too many create scheduling overhead with tiny tasks.
For a base RDD created from HDFS, the number of partitions typically equals the number of HDFS blocks. For a parallelized collection, you control it explicitly.
# Reading a large CSV from HDFS: one partition per block
account_rdd = sc.textFile("hdfs:///bank/accounts/master.csv")
print("Partitions from HDFS:", account_rdd.getNumPartitions())
# Parallelizing a local list with explicit partition count
branch_codes = ["MUM001", "DEL002", "CHN003", "BLR004", "HYD005"]
branch_rdd = sc.parallelize(branch_codes, numSlices=3)
print("Partitions from parallelize:", branch_rdd.getNumPartitions())
# Checking partition contents
for i, part in enumerate(branch_rdd.glom().collect()):
print(f" Partition {i}: {part}")
Partitions from HDFS: 16
Partitions from parallelize: 3
Partition 0: ['MUM001', 'DEL002']
Partition 1: ['CHN003']
Partition 2: ['BLR004', 'HYD005']
How does the compute function work?
The second property is the compute function. It tells Spark how to produce the records of a given partition on demand. For a base RDD, the compute function reads from storage, such as an HDFS block or an S3 object. For a derived RDD, it applies the transformation to the parent's partition. This is what makes an RDD lazy and describable rather than materialized: it knows how to compute itself but does not do so until an action triggers it.
# The compute function is internal, but you see its effect:
# map creates a new RDD whose compute function applies the lambda to each parent record.
account_balances = sc.parallelize([
("SAV2001", 150000.00),
("SAV2002", 320000.00),
("CUR3001", 980000.00)
])
# Derived RDD: compute function = apply the interest calculation per record
with_interest = account_balances.mapValues(lambda bal: bal * 1.04)
# Nothing computed yet (lazy). Action triggers compute:
print(with_interest.collect())
[('SAV2001', 156000.0), ('SAV2002', 332800.0), ('CUR3001', 1019200.0)]
What are narrow vs wide dependencies?
The third property is the list of dependencies on parent RDDs, forming the lineage. Dependencies come in two kinds, and this distinction is the most important one for performance.
A narrow dependency means each parent partition feeds at most one child partition. Operations like map, filter, and flatMap produce narrow dependencies. Spark can pipeline them within one stage without any data movement across the network.
A wide dependency means one parent partition feeds many child partitions. Operations like groupByKey, reduceByKey, join, and repartition produce wide dependencies. They require a shuffle that redistributes data across executors and creates a stage boundary.
transactions = sc.parallelize([
("SAV2001", 5000), ("SAV2001", 12000),
("SAV2002", 8000), ("SAV2002", 3000),
("CUR3001", 25000)
])
# Narrow dependency: filter does not shuffle
high_value = transactions.filter(lambda x: x[1] > 10000)
print("Filter deps (narrow):", high_value.dependencies)
# Wide dependency: reduceByKey triggers a shuffle
totals = transactions.reduceByKey(lambda a, b: a + b)
print("ReduceByKey deps (wide):", totals.dependencies)
print("Totals:", totals.collect())
Filter deps (narrow): [<pyspark.rdd.NarrowDependency ...>]
ReduceByKey deps (wide): [<pyspark.rdd.ShuffleDependency ...>]
Totals: [('SAV2001', 17000), ('SAV2002', 11000), ('CUR3001', 25000)]
How does the Partitioner control key placement?
The fourth property is the optional Partitioner, which applies to key-value RDDs. It controls how keys are mapped to partitions. Spark provides two built-in partitioners:
- HashPartitioner – assigns a key to partition
hash(key) % numPartitions, giving even distribution. - RangePartitioner – assigns contiguous key ranges to partitions, useful for sorting and range queries.
A well-chosen Partitioner can eliminate shuffles entirely. If two RDDs share the same Partitioner, a join between them can proceed without moving data.
from pyspark import SparkContext
from pyspark.rdd import portable_hash
txn = sc.parallelize([
("SAV2001", 5000), ("SAV2002", 8000),
("SAV2001", 12000), ("CUR3001", 25000)
])
# No partitioner yet
print("Before partitionBy:", txn.partitioner)
# Apply HashPartitioner with 4 partitions
partitioned_txn = txn.partitionBy(4)
print("After partitionBy:", partitioned_txn.partitioner)
# Same-account records now co-located in same partition
for i, part in enumerate(partitioned_txn.glom().collect()):
print(f" Partition {i}: {part}")
Before partitionBy: None
After partitionBy: <pyspark.rdd.Partitioner object: numPartitions=4>
Partition 0: [('SAV2002', 8000)]
Partition 1: []
Partition 2: [('SAV2001', 5000), ('SAV2001', 12000)]
Partition 3: [('CUR3001', 25000)]
How do preferred locations enable data locality?
The fifth property is the optional list of preferred locations per partition. Spark uses these hints to schedule tasks close to where their data already resides. The scheduler tries locality levels in order:
- PROCESS_LOCAL – data is in the same executor's memory (cached block).
- NODE_LOCAL – data is on the same machine's disk (HDFS block on local datanode).
- RACK_LOCAL – data is in the same rack but on a different machine.
- ANY – no locality preference; data must be fetched over the network.
# Reading from HDFS: preferred locations come from HDFS block locations
rdd = sc.textFile("hdfs:///bank/transactions/2026-01.csv")
partitions = rdd.partitions()
# Check preferred location for the first partition
for p in partitions[:3]:
locs = rdd.preferredLocations(p)
print(f"Partition {p.index}: preferred hosts = {locs}")
Partition 0: preferred hosts = ['datanode1.bank.local', 'datanode3.bank.local']
Partition 1: preferred hosts = ['datanode2.bank.local', 'datanode1.bank.local']
Partition 2: preferred hosts = ['datanode3.bank.local', 'datanode2.bank.local']
How does the Partitioner eliminate shuffles in joins?
When two RDDs share the same Partitioner, a join between them on the same key can proceed without a shuffle. Spark recognizes that matching keys are already co-located in the same partition on both sides. This is a powerful optimization for banking workloads where you repeatedly join transactions to accounts.
# Both RDDs partitioned by account_id with same partitioner
accounts = sc.parallelize([
("SAV2001", "Rajesh Kumar"),
("SAV2002", "Priya Sharma"),
("CUR3001", "National Bank Pool")
]).partitionBy(4)
transactions = sc.parallelize([
("SAV2001", 50000), ("SAV2002", 12000),
("SAV2001", 75000), ("CUR3001", 300000)
]).partitionBy(4) # same partitioner, same count
# This join is shuffle-free because both share the partitioner
joined = accounts.join(transactions)
print(joined.collect())
print("Join partitioner:", joined.partitioner)
[('SAV2001', ('Rajesh Kumar', 50000)), ('SAV2001', ('Rajesh Kumar', 75000)),
('SAV2002', ('Priya Sharma', 12000)), ('CUR3001', ('National Bank Pool', 300000))]
Join partitioner: <pyspark.rdd.Partitioner object: numPartitions=4>
How do these five properties relate to fault tolerance?
The combination of the compute function and the dependencies (lineage) gives Spark fault tolerance without replication. If a partition is lost because an executor fails, Spark traces the lineage back to the parent partitions and recomputes only the lost partition using the compute function. For narrow dependencies, only the single lost partition is recomputed. For wide dependencies, the recomputation may require re-reading shuffle data from other executors.
# Lineage enables recomputation:
# If partition 2 of 'with_interest' is lost, Spark traces:
# with_interest.partition(2)
# -> account_balances.partition(2) (via mapValues compute function)
# -> original data source (via textFile compute function)
# Only partition 2 is recomputed, not the entire RDD.
balances = sc.parallelize([
("SAV2001", 150000), ("SAV2002", 320000)
], 2)
with_interest = balances.mapValues(lambda b: b * 1.04) # narrow dep
totals = with_interest.reduceByKey(lambda a, b: a + b) # wide dep
# View the full lineage
print(totals.toDebugString().decode("utf-8"))
(2) PythonRDD[6] at collect
| MapPartitionsRDD[5] at reduceByKey
| ShuffledRDD[4] at reduceByKey
+-(2) MapPartitionsRDD[3] at mapValues
| ParallelCollectionRDD[2] at parallelize
What should a senior engineer know about these five properties?
A senior engineer uses these five properties as a mental checklist for every performance investigation.
- Partition count – is it balanced against executor cores? Too few means under-utilisation; too many means scheduling overhead.
- Dependencies – where are the wide dependencies? Each one is a shuffle and a stage boundary. Minimize shuffles by choosing narrow transformations or pre-partitioning.
- Partitioner – can you share a Partitioner between two RDDs to eliminate a join shuffle? Partitioning transactions by account_id before a per-account aggregation avoids a second shuffle.
- Preferred locations – are tasks running with good locality? The Spark UI shows locality level per task.
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("NBI-RDD-Properties").getOrCreate()
sc = spark.sparkContext
# Senior pattern: pre-partition to eliminate downstream shuffles
txn = sc.textFile("hdfs:///bank/transactions/2026-01.csv") \
.map(lambda line: (line.split(",")[0], float(line.split(",")[2]))) \
.partitionBy(64) # partition by account_id
# All subsequent operations on the same key avoid shuffles
account_totals = txn.reduceByKey(lambda a, b: a + b) # no extra shuffle
account_max = txn.reduceByKey(lambda a, b: max(a, b)) # no extra shuffle
print("Totals partitioner:", account_totals.partitioner)
print("Max partitioner:", account_max.partitioner)
Totals partitioner: <pyspark.rdd.Partitioner object: numPartitions=64>
Max partitioner: <pyspark.rdd.Partitioner object: numPartitions=64>
What are common interview follow-up questions?
Interviewers often follow up with these questions after you list the five properties:
- "Which property explains why
groupByKeyis expensive?" – Dependencies: it creates a wide (shuffle) dependency. - "How does Spark recover a lost partition?" – Lineage: trace the compute function back through dependencies and recompute.
- "Can you avoid a shuffle in a join?" – Partitioner: if both RDDs share the same Partitioner on the join key, no shuffle is needed.
- "What happens if you set too few partitions?" – Under-utilisation: tasks are too large and the cluster sits idle.
- "What is PROCESS_LOCAL?" – Preferred locations: the data is already in the same executor's memory, the best locality level.
# Quick reference: operations and their dependency types
# Narrow (no shuffle): map, filter, flatMap, mapValues, union
# Wide (shuffle): groupByKey, reduceByKey, join, repartition, sortByKey
rdd = sc.parallelize(range(100), 10)
# Narrow chain: all pipelined in one stage
result = rdd.filter(lambda x: x % 2 == 0).map(lambda x: x * 10)
print("Narrow chain partitions:", result.getNumPartitions()) # still 10
# Wide operation: creates a new stage
sorted_rdd = rdd.sortBy(lambda x: x, numPartitions=5)
print("After sortBy partitions:", sorted_rdd.getNumPartitions()) # 5
Narrow chain partitions: 10
After sortBy partitions: 5
Visual Overview
┌─────────────────────────────────────────────────────────────────┐ │ FIVE PROPERTIES OF AN RDD │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ 1. PARTITIONS 2. COMPUTE FUNCTION │ │ ┌──────┐ ┌──────┐ Each partition knows how │ │ │ P0 │ │ P1 │ to produce its records: │ │ │(block│ │(block│ - Base RDD: read from HDFS │ │ │ 0) │ │ 1) │ - Derived: apply transform │ │ └──────┘ └──────┘ to parent partition │ │ │ │ 3. DEPENDENCIES (LINEAGE) │ │ ┌────────────────────────────────────────────┐ │ │ │ Narrow (no shuffle) Wide (shuffle) │ │ │ │ Parent P0 → Child P0 Parent P0 → Child * │ │ │ │ map, filter, union groupByKey, join │ │ │ │ Pipeline in 1 stage Stage boundary │ │ │ └────────────────────────────────────────────┘ │ │ │ │ 4. PARTITIONER (optional, key-value RDDs) │ │ ┌─────────────────────────────────────┐ │ │ │ HashPartitioner: hash(key) % n │ │ │ │ RangePartitioner: key ranges → parts│ │ │ │ Same partitioner = shuffle-free join│ │ │ └─────────────────────────────────────┘ │ │ │ │ 5. PREFERRED LOCATIONS (data locality) │ │ ┌──────────────────────────────────────────┐ │ │ │ PROCESS_LOCAL → NODE_LOCAL → RACK_LOCAL │ │ │ │ (best) (good) (okay) │ │ │ │ → ANY │ │ │ │ (network) │ │ │ └──────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ Narrow vs Wide Dependency: NARROW (map, filter) WIDE (groupByKey, join) Parent Child Parent Child ┌──┐ ┌──┐ ┌──┐ ┌──┐ │P0│─────→│P0│ │P0│──┬──→│P0│ └──┘ └──┘ └──┘ │ └──┘ ┌──┐ ┌──┐ ┌──┐ ├──→┌──┐ │P1│─────→│P1│ │P1│──┤ │P1│ └──┘ └──┘ └──┘ │ └──┘ ┌──┐ ┌──┐ ┌──┐ └──→┌──┐ │P2│─────→│P2│ │P2│──────→│P2│ └──┘ └──┘ └──┘ └──┘ (1-to-1, no shuffle) (all-to-all, shuffle)
Key Takeaway
Every RDD in Spark is defined internally by five properties, and listing them shows you understand how Spark works from the ground up. The first is the list of partitions, the chunks the data is divided into, which directly set the degree of parallelism because Spark creates one task per partition per stage. The second is the compute function per partition, which tells Spark how to produce the records of any given partition on demand, either by reading storage for a base RDD or by applying a transformation to the parent for a derived RDD. Together with the third property, the dependencies, the compute function enables fault tolerance: if a partition is lost, Spark traces the lineage and recomputes only what was lost.
Dependencies come in two kinds, and this is the most important distinction for performance. Narrow dependencies mean each parent partition feeds at most one child, as with map and filter, so no data moves across the network and Spark pipelines them in a single stage. Wide dependencies mean one parent feeds many children, as with groupByKey, reduceByKey, and join, forcing a shuffle that redistributes data and creates a stage boundary. Knowing which operations are wide is the key to predicting where expensive shuffles happen.
The fourth property is the optional Partitioner for key-value RDDs. A HashPartitioner distributes keys evenly; a RangePartitioner sorts them into contiguous ranges. If two RDDs share the same Partitioner on the join key, the join is shuffle-free, which is one of the most impactful optimizations in banking workloads where transactions and accounts are repeatedly joined. The fifth property is the optional list of preferred locations, the data-locality hints Spark uses to schedule a task where its data already lives, trying PROCESS_LOCAL, then NODE_LOCAL, then RACK_LOCAL, then ANY.
At the National Bank of India, these five properties were my mental checklist whenever a job was slow. I tuned the partition count against executor cores for parallelism, I traced the dependency chain to find where expensive shuffles happened, and most impactfully I partitioned transaction data by account_id so that later joins and aggregations ran shuffle-free. Understanding that an RDD is nothing more than these five properties is what lets you reason about and fix real performance problems rather than guessing.
Remember one sentence: an RDD is defined by its partitions, compute function, dependencies, optional Partitioner, and preferred locations, and together these five properties explain parallelism, shuffles, fault tolerance, and data locality.
---