Back to Apache Spark: Core
Q64Question 64 of 72

Spark serialization: Java vs Kryo

Key Concepts
  • Serialization converts in-memory objects to bytes for transfer between JVMs or writing to disk; it is pervasive in Spark.
  • All shuffle data, broadcast variables, task closures, and serialized cache data go through the serializer.
  • Java serialization is the default: works with any Serializable class but is slow and produces large byte streams.
  • Kryo serialization is faster and several times more compact, enabled with spark.serializer=KryoSerializer.
  • Registering classes with Kryo lets it write a small integer id instead of the full class name, giving the best size.
  • In the DataFrame world, Tungsten binary encoders handle most heavy data; the serializer choice matters most for RDD operations and closures.
  • Enable Kryo by default, register domain classes, and raise the buffer max for large objects.

Definition

Spark serializes all shuffle, broadcast, closure, and serialized cache data; Java serialization is the slow, bulky default, while Kryo (spark.serializer=KryoSerializer) is faster and smaller, especially after registering your classes.
  • Serialization – converting in-memory objects to bytes for transfer between JVMs or writing to disk.
  • Where it applies – all shuffle data, broadcast variables, task closures, and data cached with serialized storage levels.
  • Java serialization – default, works with any Serializable class, but slow and produces large byte streams.
  • Kryo serialization – significantly faster and more compact; reaches best size by registering classes (writes integer ids instead of class names).
  • Tungsten – in the DataFrame world, Spark's own binary encoders handle most heavy data, so the serializer matters most for RDD and closure paths.

At the National Bank of India, moving records between branches means packing them into boxes (serialization) and unpacking at the other end (deserialization). Java serialization is packing with lots of bubble wrap and a long label on every box – reliable but bulky. Kryo is using tight, standardized boxes with a short code number – smaller and quicker, especially when you pre-register a code book of document types.

Understanding the Concepts

Why does serialization matter in Spark?

Serialization is pervasive. Every time data moves between JVM processes in Spark, it must be serialized. A faster, smaller serializer reduces network traffic, disk I/O, memory footprint, and CPU time across the whole job.

PYTHON
from pyspark.sql import SparkSession

# Where serialization happens:
# 1. Shuffle: data crosses the network between stages
# 2. Broadcast: variables sent to every executor
# 3. Closures: functions and captured variables sent to executors
# 4. Cache: data stored with serialized storage levels (MEMORY_ONLY_SER)

spark = SparkSession.builder \
    .appName("SerializationDemo") \
    .config("spark.serializer",
        "org.apache.spark.serializer.KryoSerializer") \
    .getOrCreate()

How does Java serialization work?

Java serialization is the default because it works with any class implementing Serializable. Its drawback is performance: it uses reflection and embeds full class descriptors in the output.

PYTHON
# Java serialization (default)
# Pros: works out of the box with any Serializable class
# Cons: slow (reflection), large output (full class metadata per object)

# Example: serializing a custom Transaction class
# Java serialization writes:
#   - Full class name: "com.bank.model.Transaction" (40+ bytes per object)
#   - Field metadata for each field
#   - The actual data values
# This overhead adds up over millions of objects in a shuffle

How does Kryo serialization work?

Kryo is faster and more compact. You enable it by setting spark.serializer and optionally registering your classes for optimal size.

PYTHON
# Enable Kryo serialization
spark.conf.set("spark.serializer",
    "org.apache.spark.serializer.KryoSerializer")

# Register classes for best performance
spark.conf.set("spark.kryo.classesToRegister",
    "com.bank.model.Transaction,com.bank.model.Account")

# With registration, Kryo writes:
#   - Class id: 1 (1-2 bytes instead of 40+ bytes)
#   - The actual data values
# Several times smaller than Java serialization
PYTHON
# Kryo configuration options
spark.conf.set("spark.serializer",
    "org.apache.spark.serializer.KryoSerializer")

# Force registration: catch unregistered classes as errors
spark.conf.set("spark.kryo.registrationRequired", "true")

# Raise buffer max for large objects (default 64 MB)
spark.conf.set("spark.kryoserializer.buffer.max", "128m")

# Initial buffer size
spark.conf.set("spark.kryoserializer.buffer", "64k")

What is the performance difference?

Kryo is consistently faster and smaller than Java serialization, especially for custom objects.

| Metric | Java Serialization | Kryo (registered) |
|---|---|---|
| Serialized size | 100% (baseline) | 30-50% of Java |
| Serialization speed | 100% (baseline) | 2-5x faster |
| Deserialization speed | 100% (baseline) | 2-5x faster |
| Setup required | None | Register classes |

PYTHON
# Impact on shuffle: smaller serialized size means less data moves
# A groupBy that shuffles 10 GB with Java serialization
# may shuffle only 3-5 GB with Kryo (registered)
# That directly reduces stage time and network pressure

When does the serializer choice matter most?

In the DataFrame and Spark SQL world, Tungsten binary encoders handle most heavy data, so the generic serializer matters most for specific paths.

PYTHON
# Paths where serializer choice matters:
# 1. RDD operations with custom objects
rdd = spark.sparkContext.parallelize([
    ("SAV2001", {"amount": 50000, "type": "credit"}),
    ("CUR3001", {"amount": 75000, "type": "debit"})
])
shuffled = rdd.groupByKey()  # Shuffles custom objects

# 2. Broadcast variables with non-trivial objects
lookup = spark.sparkContext.broadcast(
    {"SAV2001": "Rajesh", "CUR3001": "Priya"})

# 3. Task closures with captured variables
large_dict = {"config": "value" * 10000}
rdd.map(lambda x: (x[0], large_dict)).collect()

# 4. Cached data with serialized storage level
from pyspark import StorageLevel
rdd.persist(StorageLevel.MEMORY_ONLY_SER)

# For pure DataFrame operations, Tungsten handles the heavy path
# But Kryo still helps the closure and broadcast paths

How do you measure serialization impact?

Check the Spark UI for shuffle sizes and task serialization time to determine if serialization is a bottleneck.

PYTHON
# In the Spark UI Stages tab:
# - Shuffle Write Size: how many bytes are serialized and written
# - Task Serialization Time: time spent serializing the task closure
# - Shuffle Read Size: bytes deserialized on the reduce side

# Compare before and after enabling Kryo:
# Before (Java): Shuffle Write = 10 GB, Task Ser Time = 15s
# After (Kryo):  Shuffle Write = 4 GB,  Task Ser Time = 5s

What are common mistakes with serialization?

The most common mistake is not enabling Kryo when working with RDD-heavy or custom-object-heavy jobs.

PYTHON
# WRONG: forgetting to register classes, losing size benefit
spark.conf.set("spark.serializer",
    "org.apache.spark.serializer.KryoSerializer")
# Without registration, Kryo writes full class names (better than Java,
# but not as good as registered)

# CORRECT: register the application's domain classes
spark.conf.set("spark.kryo.classesToRegister",
    "com.bank.model.Transaction,com.bank.model.Account,"
    "com.bank.model.Transfer")

# WRONG: not raising buffer max for large objects
# Results in: com.esotericsoftware.kryo.KryoException: Buffer overflow
# CORRECT:
spark.conf.set("spark.kryoserializer.buffer.max", "256m")

What should a senior engineer know?

  • Enable Kryo by default on every Spark application; it costs nothing for DataFrame pipelines and helps RDD and closure paths.
  • At the National Bank of India, switching to Kryo and registering domain classes cut shuffle bytes several-fold on RDD-based transaction processing jobs.
  • For DataFrame pipelines, the gain from Kryo is smaller because Tungsten already handles the bulk, but it is still recommended.
  • Serialization is one of the first levers to check when shuffle or broadcast time dominates a job.
  • Use spark.kryo.registrationRequired=true in strict environments to guarantee full registration benefit.

Visual Overview

DIAGRAM
  Java Serialization (default)
  ┌─────────────────────────────────────────┐
  │ Object: Transaction                     │
  │ Output: [class=com.bank.model.Transaction│
  │          field=account_id:SAV2001        │
  │          field=amount:50000.00           │
  │          field=type:credit]              │
  │ Size: ~200 bytes per object              │
  └─────────────────────────────────────────┘

  Kryo Serialization (registered)
  ┌─────────────────────────────────────────┐
  │ Object: Transaction                     │
  │ Output: [classId=1                       │
  │          SAV2001                          │
  │          50000.00                         │
  │          credit]                          │
  │ Size: ~60 bytes per object               │
  └─────────────────────────────────────────┘

  Impact on Shuffle:
  Java: 10 GB shuffled  ->  Kryo: 3-4 GB shuffled

Key Takeaway

Serialization in Spark converts in-memory objects to bytes so they can move between JVMs or be written to disk. It matters because Spark serializes an enormous amount of data: all shuffle data, broadcast variables, task closures, and cached data in serialized storage levels. A faster, more compact serializer reduces network traffic, disk I/O, memory footprint, and CPU across the whole job.

Java serialization is the default because it works with any Serializable class, but it is slow and produces large output because it uses reflection and embeds full class descriptors. Kryo is faster and several times more compact. Registering classes lets Kryo write a small integer id instead of the full class name, giving the best size. In the DataFrame world, Tungsten binary encoders handle most heavy data, so the serializer matters most for RDD operations, custom objects in closures, and broadcast variables.

At the National Bank of India, switching to Kryo and registering transaction domain classes cut shuffle bytes several-fold on RDD-based processing jobs and shrank broadcast objects. For pure DataFrame pipelines the gain was smaller but still worthwhile because Kryo helped the remaining closure and broadcast paths at no cost.

The practice for production is: enable Kryo by default, register the application's domain classes, raise the buffer max for large objects, and treat serialization as one of the first levers to check when shuffle or broadcast time dominates a job.

Remember one sentence: Kryo is faster and smaller than Java serialization, and you should enable it by default and register your classes for the best size.

---

Share
Use ← → arrow keys to navigate