Definition
- Tiered storage (KIP-405) -- lets a Kafka broker keep only recent log segments on local disk and offload older, closed segments to cheaper remote storage such as S3, while they still appear as part of the same topic.
- Storage-compute decoupling -- separates how much data you retain from how much local disk each broker needs, so long retention becomes affordable.
- Local retention -- controlled by
local.retention.msorlocal.retention.bytes, determines how much stays on fast local disk. - Remote tier -- pluggable through a
RemoteStorageManagerinterface, with implementations for S3, GCS, and Azure Blob. - Transparent reads -- consumers reading old offsets transparently fetch from the remote tier with slightly higher latency.
Imagine National Bank of India keeps this month's transaction ledgers on the clerk's desk for instant access, but moves last year's ledgers to a cheap off-site warehouse. The ledgers are still part of the official records and anyone can request an old one, it just comes from the warehouse instead of the desk. The desk stays small and tidy, the bank keeps seven years of history cheaply, and a new clerk can start work immediately because the desk has very little on it.
Understanding the Concepts
What is tiered storage?
A Kafka partition is stored as a sequence of log segment files on a broker's local disk. Traditionally, every byte you retain lives on that local disk, so if you want ninety days of retention for a high-volume topic, you must provision and pay for a lot of fast local disk on every broker, and that disk is coupled to the compute. Tiered storage, introduced by KIP-405, decouples storage from compute.
package com.bank.kafka;
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.clients.admin.Config;
import org.apache.kafka.clients.admin.ConfigEntry;
import org.apache.kafka.common.config.TopicConfig;
import java.util.*;
public class TieredStorageTopicCreator {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put("bootstrap.servers", "kafka-1:9092");
try (AdminClient admin = AdminClient.create(props)) {
NewTopic topic = new NewTopic("transactions", 12, (short) 3);
Map<String, String> configs = new HashMap<>();
configs.put("remote.storage.enable", "true");
configs.put("local.retention.ms", "86400000"); // 1 day local
configs.put("retention.ms", "220752000000"); // 7 years total
topic.configs(configs);
admin.createTopics(Collections.singleton(topic)).all().get();
System.out.println("Created topic with tiered storage:");
System.out.println(" Local retention: 1 day");
System.out.println(" Total retention: 7 years (remote tier)");
}
}
}
Expected output:
Created topic with tiered storage:
Local retention: 1 day
Total retention: 7 years (remote tier)
What problem does tiered storage solve?
The core problem is that traditionally every retained byte lives on fast, expensive local disk coupled to the broker's compute. Tiered storage gives four big wins.
| Problem | Without Tiered Storage | With Tiered Storage |
|---|---|---|
| Cost of long retention | Huge local disks on every broker | Cheap object storage (S3) |
| Scaling elasticity | Rebalancing moves lots of data | Brokers hold little local data |
| Recovery speed | Catching-up replica copies everything | Less local data to replicate |
| Retention flexibility | Size disks for worst case | Keep years of data cheaply |
# Topic configuration for tiered storage
kafka-topics.sh --alter --topic transactions \
--config remote.storage.enable=true \
--config local.retention.ms=86400000 \
--config retention.ms=220752000000 \
--bootstrap-server kafka-1:9092
How does tiered storage work internally?
With tiered storage enabled, a broker keeps only the most recent segments locally. Once a segment is closed and older than the local retention, it is uploaded to the configured remote store and removed from local disk. The broker tracks which segments live remotely in remote log metadata.
When a consumer requests an offset that is only in the remote tier, the broker fetches that segment from the remote store and serves it. From the consumer's point of view nothing changes -- it is the same topic and the same offsets, just a slightly higher latency for very old data.
package com.bank.kafka;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
// Consumer reading from a tiered storage topic -- no code change needed
public class AuditConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "audit-replay-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(Collections.singletonList("transactions"));
// Reading 3-year-old data transparently fetches from remote tier
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(10));
for (ConsumerRecord<String, String> record : records) {
System.out.println("Audit record: offset=" + record.offset()
+ " key=" + record.key() + " value=" + record.value());
}
}
}
}
Expected output:
Audit record: offset=0 key=SAV2001 value=DEBIT 50000
Audit record: offset=1 key=SAV3001 value=CREDIT 50000
How is tiered storage different from just adding more disk?
| Aspect | More Local Disk | Tiered Storage |
|---|---|---|
| Cost per GB | High (SSD/NVMe) | Low (S3: ~$0.023/GB/month) |
| Broker recovery | Slow (copies all data) | Fast (only local data) |
| Scaling | Moving lots of data on rebalance | Minimal local data to move |
| Retention | Limited by disk budget | Practically unlimited |
| Read latency (recent) | Fast (local) | Fast (local) |
| Read latency (old) | Fast (local) | Slightly higher (remote fetch) |
What are the configuration options?
You control tiered storage with per-topic configurations.
remote.storage.enable=true-- enables tiered storage for the topiclocal.retention.ms-- how long segments stay on local disk (e.g., 1 day)local.retention.bytes-- alternative size-based local retentionretention.ms-- total retention including remote tier (e.g., 7 years)retention.bytes-- alternative size-based total retention
# Enable tiered storage on existing topic
kafka-topics.sh --alter --topic transactions \
--config remote.storage.enable=true \
--config local.retention.ms=86400000 \
--bootstrap-server kafka-1:9092
# Verify configuration
kafka-configs.sh --describe --topic transactions \
--bootstrap-server kafka-1:9092
What are the limitations and tradeoffs?
The latency tradeoff is the main consideration. Reads of recent data stay fast from local disk, but reads of very old data pay the remote fetch latency, so tiered storage suits workloads where most reads are recent and only occasional backfills or audits touch old data.
The remote store becomes part of your durability and security surface, so you encrypt it, control access with policies, and account for remote-store availability.
package com.bank.kafka;
public class TieredStorageTradeoffs {
public static void main(String[] args) {
System.out.println("Tiered Storage Tradeoffs:");
System.out.println(" + Cost: S3 is ~100x cheaper than SSD per GB");
System.out.println(" + Recovery: broker restarts faster with less local data");
System.out.println(" + Retention: 7 years is affordable");
System.out.println(" - Latency: old data reads are slower (remote fetch)");
System.out.println(" - Security: remote store needs encryption and access control");
System.out.println(" - Availability: depends on remote store uptime");
}
}
Expected output:
Tiered Storage Tradeoffs:
+ Cost: S3 is ~100x cheaper than SSD per GB
+ Recovery: broker restarts faster with less local data
+ Retention: 7 years is affordable
- Latency: old data reads are slower (remote fetch)
- Security: remote store needs encryption and access control
- Availability: depends on remote store uptime
What should a senior engineer know?
Tiered storage matured across Kafka versions and is production-ready in recent releases. Some managed Kafka offerings provide it transparently. The remote store is pluggable through a RemoteStorageManager interface. For compliance-heavy banking environments, encryption of the remote tier and key rotation are mandatory.
How do you monitor tiered storage?
Monitor the upload lag between local segments and the remote tier, the remote fetch latency for old reads, and the local disk usage to ensure it stays within the local retention bounds.
# Monitor local vs remote segment distribution
kafka-log-dirs.sh --describe --bootstrap-server kafka-1:9092 \
--topic-list transactions
Visual Overview
Tiered Storage Architecture
============================
┌─────────────────────────────────────────┐
│ Kafka Broker │
│ │
│ ┌─────────────────────────────────┐ │
│ │ Local Disk (SSD/NVMe) │ │
│ │ │ │
│ │ Segment 8 (active, appending) │ │
│ │ Segment 7 (closed, < 1 day) │ │
│ │ Segment 6 (closed, < 1 day) │ │
│ └─────────────────────────────────┘ │
│ │ │
│ upload when │
│ older than │
│ local.retention.ms │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────┐ │
│ │ Remote Log Metadata │ │
│ │ (tracks remote segments) │ │
│ └─────────────────────────────────┘ │
└─────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Remote Object Storage (S3) │
│ │
│ Segment 0 (3 years old) │
│ Segment 1 (2 years old) │
│ Segment 2 (1 year old) │
│ Segment 3 (6 months old) │
│ Segment 4 (3 months old) │
│ Segment 5 (1 month old) │
│ │
│ Total retention: 7 years │
│ Cost: ~$0.023/GB/month │
└─────────────────────────────────────────┘
Consumer reads:
Recent offset ──► Local disk (fast)
Old offset ──► Remote S3 (slightly slower)
Key Takeaway
Tiered storage, introduced by KIP-405, lets a Kafka broker keep only recent log segments on its local disk and offload older closed segments to cheaper remote object storage such as S3, while they remain part of the same topic. A local retention setting controls how much stays on local disk, and the overall retention governs the total including the remote tier, so you might keep one day locally and seven years in object storage.
The core problem it solves is that traditionally every retained byte lived on fast, expensive local disk coupled to the broker's compute. Tiered storage decouples storage from compute, making long retention affordable, scaling elastic, recovery faster, and retention flexible for compliance requirements.
In my National Bank of India project, regulators required seven years of transaction history, but the fraud and analytics systems almost always read only the last few days. We enabled tiered storage with one day local and seven years in S3, so the brokers stayed small and fast, we scaled out quickly during surges, and an auditor could still pull a three-year-old transaction through the same topic.
Remember one sentence: tiered storage keeps hot data on local disk and cold data in cheap object storage, so Kafka retains years of history without expensive local disks on every broker.
---