Back to Kafka Fundamentals: Topics, Partitions, Offsets
Q8Question 8 of 18

What are the trade-offs between the Kafka compression codecs?

Key Concepts
  • gzip gives the highest compression ratio but the highest CPU cost; best for cold data or archival.
  • snappy is fast with a moderate ratio; a long-time default because it balances speed and size.
  • lz4 is very fast with low CPU; excellent for latency-sensitive high-throughput feeds.
  • zstd offers near-gzip ratios with good speed and tunable levels; the modern default for many teams.
  • Compression is applied to the whole batch, not individual records, so larger batches compress better.
  • The broker stores and forwards compressed batches end to end; one compression step saves network, disk, and replication.
  • Producer and broker codecs should match to avoid costly recompression on the broker.

Definition

gzip squeezes hardest but burns CPU, lz4 and snappy are fast but looser, zstd is the balanced modern default, and compression pays off across network and disk because the batch stays compressed until the consumer opens it.
  • gzip -- highest compression ratio, highest CPU cost; best for cold data or when network/disk is the bottleneck.
  • snappy -- fast with a moderate ratio; a long-time default balancing speed and size.
  • lz4 -- very fast with low CPU; excellent for latency-sensitive, high-throughput feeds.
  • zstd -- strong ratio with good speed and tunable compression levels; the modern default for many teams.
  • Batch-level compression -- Kafka compresses the entire producer batch, not individual records, so larger batches compress better.
  • End-to-end savings -- the broker stores, replicates, and forwards the compressed batch; only the consumer decompresses.
  • Codec compatibility -- the broker setting compression.type=producer keeps the batch as-is; a different broker codec causes costly recompression.

Think of National Bank of India mailing a stack of statements. Compression is like vacuum-packing the stack. gzip squeezes it the tightest but takes the longest at the packing machine. snappy and lz4 pack quickly but less tightly. zstd is the modern machine that packs almost as tight as gzip but much faster. Because the vacuum-packed parcel stays sealed through the whole postal system and is only opened by the final recipient, it saves space on every truck and in every warehouse along the way.

Understanding the Concepts

What are the four compression codecs?

Kafka's compression.type producer setting accepts none, gzip, snappy, lz4, or zstd.

JAVA
package com.bank.config;

import org.apache.kafka.clients.producer.ProducerConfig;
import java.util.Properties;

public class CompressionCodecs {
    public static Properties withCompression(String codec) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, codec);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
            "org.apache.kafka.common.serialization.StringSerializer");
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
            "org.apache.kafka.common.serialization.StringSerializer");
        return props;
    }

    public static void main(String[] args) {
        System.out.println("Compression codecs:");
        System.out.println("  none   : no compression");
        System.out.println("  gzip   : highest ratio, highest CPU");
        System.out.println("  snappy : fast, moderate ratio");
        System.out.println("  lz4    : very fast, low CPU");
        System.out.println("  zstd   : strong ratio + good speed (modern default)");
    }
}
TEXT
Compression codecs:
  none   : no compression
  gzip   : highest ratio, highest CPU
  snappy : fast, moderate ratio
  lz4    : very fast, low CPU
  zstd   : strong ratio + good speed (modern default)

How do the codecs compare? (comparison table)

| Codec | Ratio | Compression speed | Decompression speed | CPU cost | Best for |
|---|---|---|---|---|---|
| gzip | Highest | Slow | Moderate | High | Cold/archival data |
| snappy | Moderate | Fast | Fast | Low | Legacy default |
| lz4 | Moderate-good | Very fast | Very fast | Very low | Latency-sensitive feeds |
| zstd | Near-gzip | Fast | Fast | Moderate | Modern default, tunable |

JAVA
package com.bank.benchmark;

public class CompressionBenchmark {
    public static void main(String[] args) {
        System.out.println("Benchmark on 10000 JSON transfer events:");
        System.out.println("  Codec    Ratio   Compress   Decompress");
        System.out.println("  none     1.0x    -          -");
        System.out.println("  gzip     8.5x    45 ms      12 ms");
        System.out.println("  snappy   5.2x    8 ms       5 ms");
        System.out.println("  lz4      5.8x    6 ms       3 ms");
        System.out.println("  zstd     7.9x    10 ms      5 ms");
        System.out.println();
        System.out.println("zstd: near-gzip ratio at lz4-like speed");
    }
}
TEXT
Benchmark on 10000 JSON transfer events:
  Codec    Ratio   Compress   Decompress
  none     1.0x    -          -
  gzip     8.5x    45 ms      12 ms
  snappy   5.2x    8 ms       5 ms
  lz4      5.8x    6 ms       3 ms
  zstd     7.9x    10 ms      5 ms

zstd: near-gzip ratio at lz4-like speed

Why does batch-level compression matter?

Kafka compresses the entire batch, not individual records. A larger batch has more repetitive content, so it compresses better. This is why linger.ms and compression reinforce each other.

JAVA
package com.bank.compression;

public class BatchCompression {
    public static void main(String[] args) {
        System.out.println("Batch of 10 records (small linger):");
        System.out.println("  zstd ratio: ~5x");
        System.out.println();
        System.out.println("Batch of 500 records (larger linger):");
        System.out.println("  zstd ratio: ~9x (more repetition to exploit)");
        System.out.println();
        System.out.println("linger.ms + compression = multiplicative savings");
    }
}
TEXT
Batch of 10 records (small linger):
  zstd ratio: ~5x

Batch of 500 records (larger linger):
  zstd ratio: ~9x (more repetition to exploit)

linger.ms + compression = multiplicative savings

How does end-to-end compression work?

The producer compresses once. The broker stores and replicates the compressed batch. The consumer decompresses. One compression step saves bandwidth and disk at every stage.

JAVA
package com.bank.compression;

public class EndToEndSavings {
    public static void main(String[] args) {
        System.out.println("End-to-end compression path:");
        System.out.println("  1. Producer compresses batch (lz4)");
        System.out.println("  2. Producer -> Broker network: COMPRESSED");
        System.out.println("  3. Broker disk storage: COMPRESSED");
        System.out.println("  4. Broker -> Follower replication: COMPRESSED");
        System.out.println("  5. Broker -> Consumer network: COMPRESSED");
        System.out.println("  6. Consumer decompresses");
        System.out.println();
        System.out.println("One compression step saves at EVERY stage");
    }
}
TEXT
End-to-end compression path:
  1. Producer compresses batch (lz4)
  2. Producer -> Broker network: COMPRESSED
  3. Broker disk storage: COMPRESSED
  4. Broker -> Follower replication: COMPRESSED
  5. Broker -> Consumer network: COMPRESSED
  6. Consumer decompresses

One compression step saves at EVERY stage

Why must producer and broker codecs be compatible?

The broker setting compression.type defaults to producer, meaning keep whatever the producer sent. If you force a different broker codec, the broker must decompress and recompress every batch, wasting significant CPU.

JAVA
package com.bank.config;

public class CodecCompatibility {
    public static void main(String[] args) {
        System.out.println("Broker compression.type = producer (default):");
        System.out.println("  Stores batch AS-IS, zero extra CPU -- IDEAL");
        System.out.println();
        System.out.println("Broker compression.type = gzip (forced):");
        System.out.println("  Producer sends lz4 -> broker decompresses");
        System.out.println("  Broker recompresses with gzip");
        System.out.println("  Significant CPU wasted -- AVOID");
    }
}
TEXT
Broker compression.type = producer (default):
  Stores batch AS-IS, zero extra CPU -- IDEAL

Broker compression.type = gzip (forced):
  Producer sends lz4 -> broker decompresses
  Broker recompresses with gzip
  Significant CPU wasted -- AVOID

When should you use each codec?

  • lz4: latency-sensitive, high-throughput feeds where speed matters most.
  • zstd: the modern default for most topics; near-gzip ratio with good speed.
  • gzip: cold or archival data where ratio matters more than latency.
  • snappy: legacy systems that already use it.
  • none: already-compressed data (images, encrypted blobs) where compression wastes CPU.

What are common mistakes?

JAVA
package com.bank.mistake;

public class CompressionMistakes {
    public static void main(String[] args) {
        System.out.println("WRONG: compressing already-encrypted blobs");
        System.out.println("  Encrypted data is random -- no compression gain");
        System.out.println();
        System.out.println("WRONG: forcing a different codec on the broker");
        System.out.println("  Broker decompresses + recompresses every batch");
        System.out.println();
        System.out.println("CORRECT: set codec on producer, broker keeps as-is");
    }
}
TEXT
WRONG: compressing already-encrypted blobs
  Encrypted data is random -- no compression gain

WRONG: forcing a different codec on the broker
  Broker decompresses + recompresses every batch

CORRECT: set codec on producer, broker keeps as-is

What should a senior engineer know?

All modern brokers support all codecs, but old consumers might not support zstd. For truly incompressible data, use none to avoid wasting CPU. The tunable compression levels in zstd let you pick a point on the ratio-vs-speed curve for your workload.

Visual Overview

DIAGRAM
  End-to-End Compression

  Producer                Broker              Consumer
  ┌──────────────┐       ┌──────────────┐    ┌──────────────┐
  │ Batch records│       │ Store        │    │ Fetch        │
  │ (64 KB raw)  │       │ compressed   │    │ compressed   │
  │              │       │ batch on disk│    │ batch        │
  │ Compress     │       │              │    │              │
  │ with lz4     │       │ Replicate    │    │ Decompress   │
  │ (8 KB)       │──────▶│ compressed   │───▶│ with lz4     │
  │              │       │ to followers │    │ (64 KB)      │
  └──────────────┘       └──────────────┘    └──────────────┘

  Savings at every hop:
    Producer->Broker: 8 KB instead of 64 KB
    Broker disk     : 8 KB instead of 64 KB
    Replication     : 8 KB instead of 64 KB
    Broker->Consumer: 8 KB instead of 64 KB

Key Takeaway

Kafka compresses the entire producer batch before sending, and the codec choice is a trade-off between ratio and CPU. gzip gives the highest ratio but highest CPU, lz4 is very fast with low CPU, and zstd is the balanced modern default with near-gzip ratios at good speed. Compression pays off end to end because the broker stores, replicates, and forwards the compressed batch, and only the consumer decompresses.

Producer and broker codec compatibility matters: leave the broker on compression.type=producer so it stores batches as-is with zero extra CPU. Forcing a different broker codec causes costly decompression and recompression on every batch.

At National Bank of India, the transaction JSON was highly repetitive, so it compressed 5-10x smaller. Pankaj standardized on zstd on the producers and left the brokers on keep-as-produced, cutting network and storage costs across the whole pipeline while keeping latency low.

Remember one sentence: zstd is the modern default because it compresses nearly as well as gzip but much faster, and one compression step saves network, disk, and replication end to end.

---

Share
Use ← → arrow keys to navigate