Back to Advanced Data Engineering
Q1Question 1 of 24

Serialized vs deserialized storage

Key Concepts
  • Deserialized storage keeps cached data as live JVM objects for the fastest access but the highest memory footprint.
  • Serialized storage packs each partition into a compact byte array, using roughly a third of the memory at the cost of CPU for ser/deser.
  • Deserialized is the default for MEMORY_ONLY; serialized is selected with MEMORY_ONLY_SER or MEMORY_AND_DISK_SER.
  • Serialized storage also reduces GC pressure because a single byte array is cheaper for the collector to track than millions of objects.
  • The Kryo serializer is far more compact and faster than default Java serialization and should always be preferred.
  • Choose deserialized when data fits comfortably in memory; switch to serialized when memory or GC is the bottleneck.

Definition

Deserialized storage is fastest but memory-hungry because it keeps live objects, while serialized storage packs data into compact bytes to save memory at the cost of serialization CPU.
  • Deserialized storage – keeps cached records as live JVM objects in the heap, giving the fastest access because no conversion is needed, but consuming the most memory due to per-object overhead (headers, references, padding).
  • Serialized storage – converts each partition into a compact byte array using a serializer (typically Kryo), reducing memory by roughly two-thirds but adding CPU cost to serialize on store and deserialize on every read.
  • GC benefit – a single byte array is almost free for the garbage collector to track, while millions of live objects create significant GC pressure.
  • Storage level selection – MEMORY_ONLY is deserialized; MEMORY_ONLY_SER is serialized; the _SER suffix indicates serialized in all storage levels.

Imagine a clerk at the National Bank of India keeping finished transaction summaries on his desk. Deserialized storage is keeping each summary fully laid out and open, ready to read instantly, but each one takes a lot of desk space. Serialized storage is folding each summary into a tight compact packet that takes a fraction of the space, so many more fit on the desk, but every time the clerk needs one he must unfold it first and refold it after. If the desk is big, keep them open for speed; if space is tight, fold them to fit more.

Understanding the Concepts

What is deserialized storage?

Deserialized storage keeps cached records as ordinary JVM objects in the heap. Access is as fast as possible because Spark just references the objects with no conversion needed.

PYTHON
from pyspark import StorageLevel
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("StorageDemo").getOrCreate()

txn = spark.read.parquet("/lake/silver/transactions")

# MEMORY_ONLY is deserialized by default
txn.persist(StorageLevel.MEMORY_ONLY)
txn.count()  # materializes the cache

The downside is memory footprint. A JVM object carries overhead beyond its raw data: object headers (12-16 bytes), references (4-8 bytes each), alignment padding, and boxed values. A record that is 50 bytes of actual data can occupy 200+ bytes as a live object.

What is serialized storage?

Serialized storage converts each partition into a single compact byte array using a serializer. This removes the per-object overhead because data is packed densely.

PYTHON
# MEMORY_ONLY_SER stores partitions as serialized byte arrays
txn.persist(StorageLevel.MEMORY_ONLY_SER)
txn.count()

# The same data now uses roughly 1/3 of the memory
# but every read pays deserialization CPU

The cost is CPU: Spark serializes when storing the partition and deserializes back into objects every time a task reads it.

How do you configure the serializer?

The default Java serializer is slow and verbose. Kryo is far more compact and faster, and should always be preferred for serialized storage.

PYTHON
spark = SparkSession.builder \
    .appName("KryoDemo") \
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
    .config("spark.kryo.registrationRequired", "false") \
    .getOrCreate()

# With Kryo, serialized partitions are 2-10x smaller than with Java serialization
txn.persist(StorageLevel.MEMORY_ONLY_SER)
txn.count()
TEXT
# Spark UI Storage tab shows the difference:
# Java serialization: 4.2 GB cached
# Kryo serialization: 1.1 GB cached (same data)

How does serialized storage reduce GC pressure?

A byte array is a single object that the garbage collector barely has to scan. Millions of live record objects are expensive for the collector to track, causing long GC pauses.

PYTHON
# Deserialized: 10 million records = 10 million+ JVM objects
# GC must scan every object, causing frequent pauses
txn.persist(StorageLevel.MEMORY_ONLY)

# Serialized: 10 million records = a few hundred byte arrays (one per partition)
# GC scans only the byte arrays, almost no pause
txn.persist(StorageLevel.MEMORY_ONLY_SER)

On large caches, serialized storage can improve stability as much as it saves memory.

When should you choose deserialized storage?

Choose deserialized when the dataset comfortably fits in memory and access speed matters most, because you pay no conversion cost.

PYTHON
# Small reference table that fits easily in memory
branch_ref = spark.read.parquet("/lake/dim/branches")  # 50 MB
branch_ref.persist(StorageLevel.MEMORY_ONLY)
branch_ref.count()

# Fast: every subsequent join reads live objects with zero overhead
enriched = txn.join(branch_ref, "branch_code")

When should you switch to serialized storage?

Switch to serialized when memory is the bottleneck – when a deserialized cache would not fit and would start dropping or spilling partitions.

PYTHON
# Large feature table that overflows memory when deserialized
features = spark.read.parquet("/lake/silver/account_features")

# Deserialized: 12 GB, exceeds executor memory, partitions dropped
# Serialized:   4 GB, fits comfortably, no drops
features.persist(StorageLevel.MEMORY_AND_DISK_SER)
features.count()

# Even though each read pays deserialization, that is much cheaper
# than recomputing dropped partitions from an expensive lineage

What are the available storage levels?

Spark provides several storage levels that combine serialization, replication, and disk spillover.

| Storage Level | Serialized | Disk Spill | Replicated |
|---|---|---|---|
| MEMORY_ONLY | No | No | No |
| MEMORY_ONLY_SER | Yes | No | No |
| MEMORY_AND_DISK | No | Yes | No |
| MEMORY_AND_DISK_SER | Yes | Yes | No |
| MEMORY_ONLY_2 | No | No | Yes (2x) |
| DISK_ONLY | Yes | Disk only | No |

PYTHON
# Choose based on your constraints
from pyspark import StorageLevel

# Default for .cache() is MEMORY_AND_DISK (deserialized)
txn.cache()  # equivalent to persist(StorageLevel.MEMORY_AND_DISK)

# Explicit serialized with disk spill
txn.persist(StorageLevel.MEMORY_AND_DISK_SER)

What are common mistakes?

The most frequent mistakes are using Java serialization instead of Kryo, caching a huge dataset as deserialized and wondering why partitions get dropped, and forgetting to check the Storage tab in the Spark UI.

PYTHON
# Mistake: default Java serializer wastes space
# Fix: always configure Kryo
spark.conf.set("spark.serializer",
    "org.apache.spark.serializer.KryoSerializer")

# Mistake: assuming cache() uses serialized storage
# Reality: cache() is MEMORY_AND_DISK (deserialized)
# Fix: use persist(StorageLevel.MEMORY_AND_DISK_SER) explicitly

What should a senior engineer know?

  • The trade-off is memory vs. CPU: deserialized is fast but large; serialized is compact but costs CPU.
  • Serialized storage also improves GC stability, which can matter more than the memory savings on large caches.
  • Kryo is mandatory for production; never use the default Java serializer for serialized storage.
  • Monitor the Spark UI Storage tab to see actual cache size and whether partitions are being dropped.
  • In DataFrames, Spark's Tungsten engine already uses a binary format (UnsafeRow) internally, so the benefit of serialized storage is less dramatic than with RDDs.
PYTHON
# Check cache efficiency in the Spark UI
# Storage tab shows: Size in Memory, Size on Disk, Fraction Cached
# If Fraction Cached < 100%, consider switching to serialized storage

Visual Overview

DIAGRAM
  Deserialized Storage             Serialized Storage
  ────────────────────             ──────────────────
  ┌─────────────────┐              ┌─────────────────┐
  │  JVM Object 1   │              │                 │
  │  [hdr][ref][pad] │              │  ┌───────────┐  │
  │  JVM Object 2   │              │  │ Byte Array │  │
  │  [hdr][ref][pad] │     vs.      │  │ (compact)  │  │
  │  JVM Object 3   │              │  └───────────┘  │
  │  [hdr][ref][pad] │              │                 │
  │  ...millions...  │              │  One array per  │
  │                  │              │  partition       │
  └─────────────────┘              └─────────────────┘
  Fast access                      Compact storage
  High memory                      CPU on read/write
  Heavy GC                         Light GC

Key Takeaway

Serialized versus deserialized storage is a trade-off between memory on one side and CPU plus access speed on the other. Deserialized storage keeps cached records as live JVM objects, so access is instant because no conversion is needed. The cost is memory footprint: every JVM object carries headers, references, and padding that can multiply the raw data size several times over, and millions of objects create heavy garbage-collection pressure.

Serialized storage converts each partition into a single compact byte array using a serializer, ideally Kryo rather than the default Java serializer. Packing the data densely removes the per-object overhead, so the same partition often takes roughly a third of the memory. The price is CPU: Spark serializes when storing and deserializes on every read. But serialized storage also reduces GC pressure because a single byte array is almost free for the collector to scan, which on large caches can improve stability as much as it saves memory.

At the National Bank of India this mattered on a large cached account-features table that overflowed memory as live objects and kept dropping partitions. Moving it from the deserialized default to MEMORY_AND_DISK_SER with Kryo roughly tripled how much fit in memory and smoothed out GC pauses, at a CPU cost that was well worth it. The guidance is to start deserialized for speed when data fits comfortably, switch to serialized when memory or GC is the bottleneck, and always pair serialization with Kryo.

Remember one sentence: deserialized is fastest but memory-hungry, serialized is compact but costs CPU, so choose deserialized when data fits and serialized when memory or GC is the bottleneck.

---

Share
Use ← → arrow keys to navigate