Definition
- UnsafeRow – Spark SQL's compact binary row format stored as contiguous bytes (often off-heap), with a null-tracking bitset and fixed-offset fields, enabling direct field access and raw byte-copy movement.
- Project Tungsten – Spark's execution engine initiative that improves performance through explicit binary memory management, direct binary processing, cache-aware data layouts, and whole-stage code generation.
- Key benefit – serialization/deserialization shrinks to byte copies, and GC pressure drops because the collector manages a few large buffers instead of millions of small objects.
Storing each transaction as a normal Java object is like keeping every record in its own labeled folder scattered around the office. To move or read them the clerk must open each folder (deserialize) and reseal it (serialize). UnsafeRow packs all fields of a row tightly onto a single index card with fixed positions, so the clerk reads any field by looking at a known spot and hands the whole card to another clerk instantly, no unpacking needed. Tungsten is the whole philosophy of working with these tight cards and writing custom one-pass routines to process them.
Understanding the Concepts
What is the problem with JVM objects?
A normal JVM representation of a row is a collection of boxed objects with headers, pointers, and padding. This wastes memory and forces expensive serialization when rows move between operators or executors.
# Conceptual: a JVM object for one transaction
# Object header: 12-16 bytes
# account_id (Long wrapper): 24 bytes
# amount (BigDecimal): 40+ bytes
# txn_date (String): 50+ bytes
# Total: ~130 bytes for ~30 bytes of actual data
# Multiply by 100 million rows = massive overhead + GC pressure
What is UnsafeRow?
UnsafeRow stores a row as a single contiguous block of bytes: a null-tracking bitset, then a fixed-length region where each field sits at a known offset.
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("TungstenDemo").getOrCreate()
txn = spark.read.parquet("/lake/silver/transactions")
# Internally, each row is stored as UnsafeRow:
# [null bitset][account_id at offset 0][amount at offset 8][txn_date at offset 16]...
# Variable-length fields (strings) are stored as offset+length pointers into a trailing region
# You can see the physical representation in the plan
txn.select("account_id", "amount").explain(True)
== Physical Plan ==
*(1) Project [account_id, amount]
+- *(1) FileScan parquet
ReadSchema: struct<account_id:bigint,amount:decimal(18,2)>
Because the layout is fixed and flat, Spark reads a field by jumping to its offset and copies a whole row as bytes without converting to objects.
How does UnsafeRow eliminate serialization overhead?
When rows move between operators in a pipeline or across the network in a shuffle, there is no object-to-bytes conversion. The data is already bytes.
# Traditional RDD: each row is a JVM object
# Shuffle: serialize to bytes -> network -> deserialize to objects
# DataFrame with UnsafeRow:
# Shuffle: copy bytes -> network -> read bytes directly
# No serialize/deserialize step
result = txn.groupBy("account_id").agg(F.sum("amount"))
result.explain()
== Physical Plan ==
*(2) HashAggregate(keys=[account_id], functions=[sum(amount)])
+- Exchange hashpartitioning(account_id, 200)
+- *(1) HashAggregate(keys=[account_id], functions=[partial_sum(amount)])
+- *(1) FileScan parquet
The Exchange (shuffle) moves UnsafeRow bytes directly. No object creation on either side.
What is Project Tungsten?
Project Tungsten is the broader engine initiative that UnsafeRow enables. It has four themes.
# Theme 1: Explicit memory management
# Uses off-heap binary buffers managed by Spark, not the JVM GC
# Result: no GC pauses even on terabytes of data
# Theme 2: Binary processing
# Operations like hashing, sorting, and aggregation work directly
# on the byte format without materializing objects
# Theme 3: Cache-aware computation
# Data layouts are designed to fit CPU cache lines
# Sort buffers use cache-friendly access patterns
# Theme 4: Whole-stage code generation (codegen)
# Fuses a chain of operators into a single generated Java method
spark.conf.get("spark.sql.codegen.wholeStage") # true by default
How does whole-stage codegen work?
Instead of a chain of virtual function calls (filter.next() -> project.next() -> aggregate.next()), Spark generates a single Java method that loops over UnsafeRows and does all operations inline.
result = txn.filter(F.col("amount") > 10000) \
.select("account_id", "amount") \
.groupBy("account_id") \
.agg(F.sum("amount").alias("total"))
result.explain()
== Physical Plan ==
*(2) HashAggregate(keys=[account_id], functions=[sum(amount)])
+- Exchange hashpartitioning(account_id, 200)
+- *(1) HashAggregate(keys=[account_id], functions=[partial_sum(amount)])
+- *(1) Project [account_id, amount]
+- *(1) Filter (amount > 10000)
+- *(1) FileScan parquet
The *(1) prefix means all those operators are fused into one generated method by WholeStageCodegen. The filter, project, and partial aggregate run as one tight loop.
Why is DataFrame code faster than RDD code?
RDDs hold JVM objects and cannot use UnsafeRow or codegen. DataFrames and Spark SQL operate on UnsafeRows with full Tungsten optimization.
# RDD approach: JVM objects, no codegen, no binary processing
rdd_result = txn.rdd \
.filter(lambda r: r["amount"] > 10000) \
.map(lambda r: (r["account_id"], r["amount"])) \
.reduceByKey(lambda a, b: a + b)
# DataFrame approach: UnsafeRow, codegen, binary processing
df_result = txn.filter(F.col("amount") > 10000) \
.groupBy("account_id") \
.agg(F.sum("amount"))
# The DataFrame version is typically 5-10x faster
How do you verify Tungsten and codegen are active?
Check the physical plan for WholeStageCodegen nodes (shown as * prefixes).
result.explain(True)
# Look for *(n) prefixes on operators – these are inside codegen stages
# Operators without * are not codegen'd (e.g., some complex expressions)
# You can also check the generated code:
result.queryExecution.debug.codegen()
What should a senior engineer know?
- UnsafeRow is why DataFrames are faster than RDDs even for the same logical computation.
- Whole-stage codegen eliminates virtual dispatch and intermediate objects, running queries like hand-written C-style loops.
- Off-heap memory avoids GC pressure, which matters enormously at terabyte scale.
- Some operations fall out of codegen (complex UDFs, certain join types); check the plan to confirm.
- Tungsten is automatic for DataFrame and SQL workloads; no configuration needed beyond the defaults.
# Confirm codegen is enabled
print(spark.conf.get("spark.sql.codegen.wholeStage")) # true
print(spark.conf.get("spark.sql.codegen.fallback")) # true (fallback if codegen fails)
Visual Overview
JVM Objects vs UnsafeRow ──────────────────────── JVM Object (per row): ┌────────┬──────────┬────────┬───────┬─────┐ │ header │ ref→Long │ ref→BD │ ref→S │ pad │ ~130 bytes └────────┴──────────┴────────┴───────┴─────┘ × millions = huge memory + heavy GC UnsafeRow (per row): ┌──────────┬────────────┬────────┬──────────┬─────────┐ │ null bits│ account_id │ amount │ txn_date │ var-len │ ~40 bytes └──────────┴────────────┴────────┴──────────┴─────────┘ Flat bytes, read by offset, copy as-is Project Tungsten Themes: ┌───────────────────────────────────────────┐ │ 1. Off-heap memory (bypass GC) │ │ 2. Binary processing (no object creation) │ │ 3. Cache-aware layouts (CPU-friendly) │ │ 4. Whole-stage codegen (fused loops) │ └───────────────────────────────────────────┘
Key Takeaway
UnsafeRow is Spark SQL's compact binary representation of a row. Instead of storing each row as a graph of boxed JVM objects with headers and pointers, Tungsten stores it as a single contiguous block of bytes: a null-tracking bitset, then a fixed-length region where each field sits at a known offset. Because the layout is flat and fixed, Spark reads any field by jumping to its offset and moves rows between operators or across the network as raw bytes with no serialization.
UnsafeRow is one pillar of Project Tungsten, which has four themes: explicit off-heap memory management that bypasses the garbage collector, binary processing that operates directly on bytes for hashing, sorting, and aggregation, cache-aware data layouts for CPU-friendly access, and whole-stage code generation that fuses a chain of operators into a single generated Java method running like hand-written code.
At the National Bank of India, moving an old RDD job to DataFrames transformed its performance. The RDD version held every transaction as a JVM object, thrashing the garbage collector and serializing rows repeatedly across shuffles. The DataFrame version ran on UnsafeRows with whole-stage codegen, so a filter-then-aggregate over hundreds of millions of transactions executed as one tight loop over binary rows with off-heap aggregation buffers. The WholeStageCodegen nodes in the physical plan confirmed the fast path.
Remember one sentence: UnsafeRow is the flat binary row that eliminates serialization overhead, and Tungsten processes those rows with off-heap memory, binary operations, cache-aware layouts, and generated code, which is why DataFrames are far faster than RDDs.
---