Definition
- SparkContext – the original Spark 1.x entry point representing the connection to the cluster and the low-level RDD API; only one active per JVM.
- SparkSession – the Spark 2.0+ unified entry point wrapping SparkContext, SQLContext, and HiveContext into one object.
- Access pattern – use SparkSession for DataFrames, SQL, and catalog; reach SparkContext via
spark.sparkContextfor broadcasts, accumulators, and RDD operations. - Modern standard – SparkSession unifies all entry points, runs through the Catalyst optimizer, and supports
getOrCreatefor safe reuse.
Think of SparkContext as the old single phone line into the National Bank's data-processing cluster: essential but low level, and you can only have one line per building. SparkSession is the modern reception desk that still has that phone line behind it but also handles SQL requests, table lookups, and DataFrame work through one friendly front door.
Understanding the Concepts
What is SparkContext?
SparkContext is the original Spark 1.x entry point. It represents the connection to the cluster manager and the low-level API for creating and manipulating RDDs. There can be only one active SparkContext per JVM because it owns the cluster connection and scheduler.
from pyspark import SparkContext, SparkConf
# Spark 1.x style: create SparkContext directly
conf = SparkConf().setAppName("NBI-Legacy-Job").setMaster("yarn")
sc = SparkContext(conf=conf)
# Low-level RDD operations
txn_rdd = sc.textFile("hdfs:///bank/transactions/2026-01.csv")
count = txn_rdd.filter(lambda line: "Mumbai" in line).count()
print(f"Mumbai transactions: {count}")
sc.stop()
Mumbai transactions: 45230
What is SparkSession?
SparkSession, introduced in Spark 2.0, is the unified entry point. It wraps SparkContext plus the old SQLContext and HiveContext, giving one object for DataFrames, Datasets, SQL, and catalog access.
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
# Spark 2.0+ style: SparkSession is the entry point
spark = SparkSession.builder \
.appName("NBI-Modern-Job") \
.config("spark.sql.shuffle.partitions", "200") \
.getOrCreate()
# DataFrames, SQL, and catalog through one object
txn = spark.read.parquet("hdfs:///bank/transactions/2026-01/")
mumbai = txn.filter(F.col("branch") == "Mumbai")
print(f"Mumbai count: {mumbai.count()}")
# SQL
txn.createOrReplaceTempView("transactions")
result = spark.sql("SELECT branch, SUM(amount) FROM transactions GROUP BY branch")
result.show()
# Catalog
print(spark.catalog.listTables())
Mumbai count: 45230
+--------+-----------+
| branch|sum(amount)|
+--------+-----------+
| Mumbai| 45750000.0|
| Delhi| 32100000.0|
| Chennai| 28900000.0|
+--------+-----------+
How do SparkContext and SparkSession compare?
| Feature | SparkContext | SparkSession |
|---|---|---|
| Introduced | Spark 1.0 | Spark 2.0 |
| Primary API | RDDs | DataFrames, Datasets, SQL |
| Optimizer | None (manual) | Catalyst + Tungsten |
| SQL support | Requires separate SQLContext | Built-in (spark.sql) |
| Hive support | Requires separate HiveContext | Built-in (enableHiveSupport) |
| Catalog | Not directly | spark.catalog |
| Per JVM | Only one active | Wraps the one SparkContext |
| Creation | SparkContext(conf) | SparkSession.builder.getOrCreate() |
# SparkSession wraps SparkContext internally
print("SparkContext via SparkSession:", spark.sparkContext.appName)
print("Same object:", spark.sparkContext is spark.sparkContext) # True
SparkContext via SparkSession: NBI-Modern-Job
Same object: True
How do you access SparkContext from SparkSession?
When you need the low-level API (broadcast, accumulators, parallelize, or RDD operations), reach the underlying SparkContext through spark.sparkContext.
sc = spark.sparkContext
# Broadcast a lookup table
fee_rates = {"savings": 0.005, "current": 0.01, "loan": 0.02}
fee_bc = sc.broadcast(fee_rates)
print("Broadcast value:", fee_bc.value)
# Accumulator for counting bad records
bad_record_count = sc.accumulator(0)
# Parallelize a local collection
branches = sc.parallelize(["Mumbai", "Delhi", "Chennai"])
print("Branches:", branches.collect())
Broadcast value: {'savings': 0.005, 'current': 0.01, 'loan': 0.02}
Branches: ['Mumbai', 'Delhi', 'Chennai']
Why is SparkSession the modern standard?
SparkSession unifies fragmented contexts, exposes the optimized DataFrame/SQL API through Catalyst and Tungsten, supports getOrCreate for safe reuse, and allows multiple isolated sessions.
# Reason 1: One object instead of three
# Old: SparkContext + SQLContext + HiveContext
# New: SparkSession (wraps all three)
# Reason 2: Catalyst optimizer for DataFrames and SQL
# DataFrame operations are optimized automatically
txn.filter(F.col("amount") > 50000).select("account_id", "amount").explain()
# Shows optimized plan with predicate pushdown and column pruning
# Reason 3: getOrCreate avoids duplicate contexts
spark1 = SparkSession.builder.appName("NBI").getOrCreate()
spark2 = SparkSession.builder.appName("NBI").getOrCreate()
print("Same session:", spark1 is spark2) # True
# Reason 4: Multiple isolated sessions sharing one context
session_a = spark.newSession()
session_b = spark.newSession()
# Each has its own SQL configurations but shares the SparkContext
Same session: True
When do you still need SparkContext directly?
You drop to SparkContext for operations not exposed through SparkSession: broadcasts, accumulators, custom RDD operations, and setting the log level.
sc = spark.sparkContext
# Set log level
sc.setLogLevel("WARN")
# Broadcast a large lookup
account_types = spark.read.parquet("hdfs:///bank/ref/account-types/") \
.rdd.collectAsMap()
type_bc = sc.broadcast(account_types)
# Custom RDD operation
raw = sc.textFile("hdfs:///bank/legacy/fixed-width.txt")
parsed = raw.map(lambda line: (line[0:10].strip(), float(line[10:25].strip())))
print(parsed.take(3))
[('SAV2001', 50000.0), ('SAV2002', 12000.0), ('CUR3001', 300000.0)]
What are common mistakes?
The most common mistake is creating multiple SparkContexts, which throws an error. Another is using RDDs when DataFrames would be faster.
# WRONG: trying to create a second SparkContext
# sc2 = SparkContext(conf=SparkConf()) # Error: only one SC per JVM
# CORRECT: use getOrCreate
spark = SparkSession.builder.getOrCreate()
sc = spark.sparkContext # reuses existing
# WRONG: using RDD operations when DataFrames are better
# RDDs bypass Catalyst optimizer
rdd_result = sc.textFile("hdfs:///bank/txn.csv") \
.map(lambda l: l.split(",")) \
.filter(lambda r: float(r[2]) > 50000)
# CORRECT: use DataFrames for optimizer benefits
df_result = spark.read.csv("hdfs:///bank/txn.csv", header=True, inferSchema=True) \
.filter(F.col("amount") > 50000)
What should a senior engineer know?
A senior engineer writes all new code against SparkSession and DataFrames, drops to SparkContext only for low-level operations, and understands that SparkSession is built on top of SparkContext.
# Senior pattern: SparkSession for everything, SparkContext for low-level
spark = SparkSession.builder \
.appName("NBI-Production") \
.config("spark.sql.adaptive.enabled", "true") \
.enableHiveSupport() \
.getOrCreate()
# DataFrames and SQL through SparkSession
txn = spark.table("bank.transactions")
result = txn.groupBy("branch").agg(F.sum("amount"))
# Low-level only when needed
sc = spark.sparkContext
sc.setLogLevel("WARN")
rate_bc = sc.broadcast({"savings": 0.04})
Visual Overview
SparkContext vs SparkSession:
Spark 1.x (fragmented):
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ SparkContext │ │ SQLContext │ │ HiveContext │
│ (RDDs, │ │ (DataFrames, │ │ (Hive tables,│
│ cluster) │ │ SQL) │ │ HiveQL) │
└──────────────┘ └──────────────┘ └──────────────┘
↓ ↓ ↓
3 separate objects, confusing
Spark 2.0+ (unified):
┌──────────────────────────────────────────────────┐
│ SparkSession │
│ ┌──────────────────────────────────────────┐ │
│ │ DataFrames, Datasets, SQL, Catalog │ │
│ │ Catalyst Optimizer + Tungsten Engine │ │
│ └──────────────────────────────────────────┘ │
│ ┌──────────────────────────────────────────┐ │
│ │ SparkContext (wrapped) │ │
│ │ RDDs, broadcast, accumulators, low-level │ │
│ │ Access: spark.sparkContext │ │
│ └──────────────────────────────────────────┘ │
└──────────────────────────────────────────────────┘
1 unified entry point
Key Takeaway
SparkContext is the original Spark 1.x entry point. It represents the connection to the cluster and the low-level API for creating and manipulating RDDs, and there can be only one active SparkContext per JVM. In that era, to do SQL you needed a separate SQLContext, and to touch Hive tables you needed a HiveContext, so applications juggled several context objects.
SparkSession, introduced in Spark 2.0, is the unified entry point that wraps a SparkContext together with the old SQLContext and HiveContext behind one object. It is the handle for DataFrames, Datasets, SQL, and catalog access, all of which run through the Catalyst optimizer and Tungsten engine. When I still need the low-level API I reach the underlying context through spark.sparkContext.
SparkSession is the modern standard for concrete reasons. It unifies the fragmented contexts into one handle, so there is no boilerplate. It is the entry point for the optimized DataFrame API, which is far faster than hand-written RDD code. It supports getOrCreate for safe reuse and newSession for multiple isolated sessions sharing one SparkContext.
At the National Bank of India I modernized a legacy Spark 1.x job that mixed a SparkContext for RDDs with a separate SQLContext for queries. I replaced all of that with a single SparkSession, moved the logic to DataFrames so the optimizer could accelerate it, and only dropped to spark.sparkContext when I genuinely needed broadcasts or accumulators.
Remember one sentence: SparkContext is the low-level cluster connection, SparkSession is the unified modern entry point wrapping it with DataFrames, SQL, and catalog, and new code should always use SparkSession.
---