Definition
- GlobalKTable stores the complete data from all partitions of a topic on every application instance, unlike KTable which stores only the data for assigned partitions.
- Co-partitioning not required: because every instance has all the data, you can join a KStream with a GlobalKTable even if the topics have different partition counts or different key types.
- KeyValueMapper: when joining KStream with GlobalKTable, you provide a mapper function that extracts the join key from the KStream record, which can be different from the KStream's actual key.
- Read-only: GlobalKTable cannot be modified from within the topology; it is populated exclusively from the source topic.
- Full replication cost: every instance reads every partition of the GlobalKTable source topic, so network usage and memory scale with the total topic size times instance count.
Think of National Bank of India's branch code directory. Every bank branch needs a complete copy of this directory to look up any branch code during transactions. That is a GlobalKTable: a small reference dataset replicated everywhere. In contrast, each branch keeps only its own customer accounts (KTable, partitioned). If the branch code directory had 50 million entries and changed every second, replicating it everywhere would be wasteful. That is why GlobalKTable is for small, stable reference data, not for large, frequently changing datasets.
Understanding the Concepts
What is GlobalKTable?
GlobalKTable is a special table abstraction in Kafka Streams that replicates the complete data from a source topic to every instance of the application. Unlike KTable, which is partitioned (each instance holds only its assigned partitions), GlobalKTable gives every instance a full copy.
You create a GlobalKTable with builder.globalTable("topic-name").
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.GlobalKTable;
import org.apache.kafka.streams.kstream.Materialized;
public class GlobalKTableBasics {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// GlobalKTable: every instance gets ALL branch data
GlobalKTable<String, String> branches = builder.globalTable(
"branch-directory",
Materialized.as("branch-store"));
System.out.println("GlobalKTable created from 'branch-directory'");
System.out.println(" Every instance has the complete branch directory");
System.out.println(" No co-partitioning needed for joins");
System.out.println(" Suitable for small reference data");
Topology topology = builder.build();
System.out.println();
System.out.println(topology.describe());
}
}
Expected output:
GlobalKTable created from 'branch-directory'
Every instance has the complete branch directory
No co-partitioning needed for joins
Suitable for small reference data
Topologies:
Sub-topology: 0 for global store (will not generate tasks)
Source: KSTREAM-SOURCE-0000000000 (topics: [branch-directory])
--> KTABLE-SOURCE-0000000001
Processor: KTABLE-SOURCE-0000000001 (stores: [branch-store])
<-- KSTREAM-SOURCE-0000000000
Notice the topology says "for global store (will not generate tasks)". GlobalKTable processing is separate from the main task assignment.
How does GlobalKTable differ from KTable?
| Feature | KTable | GlobalKTable |
|---|---|---|
| Data per instance | Only assigned partitions | Entire topic (all partitions) |
| Co-partitioning for join | Required | Not required |
| Join key | Must be the record key | Can be any field (KeyValueMapper) |
| Memory usage | Proportional to partitions/instance | Proportional to total topic size |
| Network usage | Reads only assigned partitions | Every instance reads all partitions |
| Suitable data size | Any size | Small reference data only |
| Update frequency | Any frequency | Low frequency recommended |
| Task assignment | Part of normal task distribution | Separate global consumer |
| Created by | builder.table() | builder.globalTable() |
| Changelog topic | Created automatically | Not created (source topic is the log) |
package com.bank.streams;
public class KTableVsGlobalKTable {
public static void main(String[] args) {
System.out.println("KTable (partitioned):");
System.out.println(" Instance A sees: partitions 0, 1 -> accounts SAV2001, SAV2002");
System.out.println(" Instance B sees: partitions 2, 3 -> accounts CUR3001, CUR3002");
System.out.println(" Instance A cannot look up CUR3001");
System.out.println();
System.out.println("GlobalKTable (replicated):");
System.out.println(" Instance A sees: ALL partitions -> all branches");
System.out.println(" Instance B sees: ALL partitions -> all branches");
System.out.println(" Both instances can look up any branch code");
System.out.println();
System.out.println("Rule:");
System.out.println(" Small lookup data (branches, currencies) -> GlobalKTable");
System.out.println(" Large transactional data (accounts, orders) -> KTable");
}
}
Expected output:
KTable (partitioned):
Instance A sees: partitions 0, 1 -> accounts SAV2001, SAV2002
Instance B sees: partitions 2, 3 -> accounts CUR3001, CUR3002
Instance A cannot look up CUR3001
GlobalKTable (replicated):
Instance A sees: ALL partitions -> all branches
Instance B sees: ALL partitions -> all branches
Both instances can look up any branch code
Rule:
Small lookup data (branches, currencies) -> GlobalKTable
Large transactional data (accounts, orders) -> KTable
Why does KTable require co-partitioning but GlobalKTable does not?
KTable is partitioned. Instance A holds state for partitions 0 and 1. Instance B holds state for partitions 2 and 3. If a KStream record on Instance A needs to join with a KTable entry that lives on Instance B, the join fails silently because the data is not local.
Co-partitioning ensures that records with the same key in the KStream and KTable land on the same partition and therefore the same instance.
GlobalKTable avoids this problem because every instance has all the data. No matter which instance processes a KStream record, it can find any key in the GlobalKTable's local store.
package com.bank.streams;
public class CoPartitioningExplained {
public static void main(String[] args) {
System.out.println("Problem with KTable join without co-partitioning:");
System.out.println();
System.out.println(" transactions topic: 4 partitions (keyed by account ID)");
System.out.println(" branch-info topic: 6 partitions (keyed by branch code)");
System.out.println();
System.out.println(" KStream-KTable join on branch code:");
System.out.println(" Transaction SAV2001 is on partition 2 (Instance B)");
System.out.println(" Branch MUMBAI01 is on partition 5 (Instance C in KTable)");
System.out.println(" Instance B cannot see partition 5 data -> join fails");
System.out.println();
System.out.println("Solution with GlobalKTable:");
System.out.println(" builder.globalTable('branch-info')");
System.out.println(" Every instance has ALL branch data");
System.out.println(" Instance B can look up MUMBAI01 locally");
System.out.println(" No co-partitioning needed");
System.out.println(" Different partition counts are fine");
}
}
Expected output:
Problem with KTable join without co-partitioning:
transactions topic: 4 partitions (keyed by account ID)
branch-info topic: 6 partitions (keyed by branch code)
KStream-KTable join on branch code:
Transaction SAV2001 is on partition 2 (Instance B)
Branch MUMBAI01 is on partition 5 (Instance C in KTable)
Instance B cannot see partition 5 data -> join fails
Solution with GlobalKTable:
builder.globalTable('branch-info')
Every instance has ALL branch data
Instance B can look up MUMBAI01 locally
No co-partitioning needed
Different partition counts are fine
What is a complete banking project using GlobalKTable?
This project enriches transactions with branch information. Transactions are keyed by account ID. The branch directory is keyed by branch code. Since the keys differ, a KTable join would require repartitioning. GlobalKTable avoids that.
Project structure
branch-enrichment-service/
├── pom.xml
└── src/
├── main/
│ └── java/
│ └── com/
│ └── bank/
│ └── streams/
│ ├── BranchEnrichmentApp.java
│ └── BranchEnrichmentTopology.java
└── test/
└── java/
└── com/
└── bank/
└── streams/
└── BranchEnrichmentTest.java
BranchEnrichmentTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.GlobalKTable;
import org.apache.kafka.streams.kstream.KStream;
public class BranchEnrichmentTopology {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
// KStream: transactions keyed by account ID
// Value format: accountId|amount|type|branchCode
KStream<String, String> transactions = builder.stream("transactions");
// GlobalKTable: branch directory keyed by branch code
// Value format: branchName|city|region
GlobalKTable<String, String> branches = builder.globalTable("branch-directory");
// Join: extract branchCode from transaction value (not the key)
KStream<String, String> enriched = transactions.join(
branches,
// KeyValueMapper: extract the join key (branch code) from the transaction
(accountId, transaction) -> transaction.split("\\|")[3],
// ValueJoiner: combine transaction with branch info
(transaction, branchInfo) -> {
String branchName = branchInfo.split("\\|")[0];
String city = branchInfo.split("\\|")[1];
return transaction + "|BRANCH:" + branchName + "|CITY:" + city;
}
);
enriched.to("enriched-transactions");
return builder.build();
}
}
BranchEnrichmentTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class BranchEnrichmentTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(BranchEnrichmentTopology.build(), props)) {
TestInputTopic<String, String> branchInput = driver.createInputTopic(
"branch-directory",
new StringSerializer(), new StringSerializer());
TestInputTopic<String, String> txnInput = driver.createInputTopic(
"transactions",
new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> output = driver.createOutputTopic(
"enriched-transactions",
new StringDeserializer(), new StringDeserializer());
// Load branch directory (GlobalKTable)
branchInput.pipeInput("MUMBAI01", "Mumbai Main|Mumbai|West");
branchInput.pipeInput("DELHI01", "Delhi Central|Delhi|North");
branchInput.pipeInput("CHENNAI01", "Chennai Fort|Chennai|South");
// Transactions keyed by account ID, branchCode in value
txnInput.pipeInput("SAV2001", "SAV2001|50000|DEPOSIT|MUMBAI01");
txnInput.pipeInput("CUR3001", "CUR3001|75000|NEFT|DELHI01");
txnInput.pipeInput("SAV2002", "SAV2002|25000|UPI|CHENNAI01");
System.out.println("Enriched transactions:");
while (!output.isEmpty()) {
var record = output.readKeyValue();
System.out.println(" " + record.key + ": " + record.value);
}
System.out.println();
System.out.println("Key observation:");
System.out.println(" Transaction key = account ID (SAV2001)");
System.out.println(" GlobalKTable key = branch code (MUMBAI01)");
System.out.println(" Different keys, different partition counts = OK with GlobalKTable");
}
}
}
Expected output:
Enriched transactions:
SAV2001: SAV2001|50000|DEPOSIT|MUMBAI01|BRANCH:Mumbai Main|CITY:Mumbai
CUR3001: CUR3001|75000|NEFT|DELHI01|BRANCH:Delhi Central|CITY:Delhi
SAV2002: SAV2002|25000|UPI|CHENNAI01|BRANCH:Chennai Fort|CITY:Chennai
Key observation:
Transaction key = account ID (SAV2001)
GlobalKTable key = branch code (MUMBAI01)
Different keys, different partition counts = OK with GlobalKTable
How to run
# Step 1: Create topics (branch-directory should be compacted and small)
kafka-topics.sh --create --topic branch-directory --partitions 2 --replication-factor 1 --config cleanup.policy=compact --bootstrap-server localhost:9092
kafka-topics.sh --create --topic transactions --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic enriched-transactions --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
# Step 2: Build and test
cd branch-enrichment-service
mvn clean package
java -cp target/branch-enrichment-service-1.0.0.jar com.bank.streams.BranchEnrichmentTest
# Step 3: Run
java -cp target/branch-enrichment-service-1.0.0.jar com.bank.streams.BranchEnrichmentApp
# Step 4: Produce branch data and transactions
kafka-console-producer.sh --topic branch-directory --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# MUMBAI01:Mumbai Main|Mumbai|West
# DELHI01:Delhi Central|Delhi|North
kafka-console-producer.sh --topic transactions --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# SAV2001:SAV2001|50000|DEPOSIT|MUMBAI01
What are common mistakes with GlobalKTable?
Wrong: using GlobalKTable for a large, high-throughput topic
package com.bank.streams;
public class WrongLargeGlobalKTable {
public static void main(String[] args) {
System.out.println("WRONG: builder.globalTable('bank-transactions')");
System.out.println();
System.out.println(" bank-transactions: 10 million records per day");
System.out.println(" 3 instances running");
System.out.println();
System.out.println(" Problem:");
System.out.println(" - Each instance reads ALL 10M records (not just its partitions)");
System.out.println(" - 3 instances x 10M = 30M reads per day (3x network traffic)");
System.out.println(" - Each instance stores 10M records in RocksDB");
System.out.println(" - Startup rebuilds from all partitions (slow)");
System.out.println();
System.out.println(" Fix: use KTable (partitioned) for large topics");
System.out.println(" Each instance reads only its partitions");
System.out.println(" Co-partition the topics for joins");
}
}
Expected output:
WRONG: builder.globalTable('bank-transactions')
bank-transactions: 10 million records per day
3 instances running
Problem:
- Each instance reads ALL 10M records (not just its partitions)
- 3 instances x 10M = 30M reads per day (3x network traffic)
- Each instance stores 10M records in RocksDB
- Startup rebuilds from all partitions (slow)
Fix: use KTable (partitioned) for large topics
Each instance reads only its partitions
Co-partition the topics for joins
Correct: use GlobalKTable for small reference data
package com.bank.streams;
public class CorrectSmallGlobalKTable {
public static void main(String[] args) {
System.out.println("CORRECT: Use GlobalKTable for small, stable reference data");
System.out.println();
System.out.println(" Good candidates:");
System.out.println(" - Branch directory: ~5,000 branches, rarely changes");
System.out.println(" - Currency codes: ~180 currencies, stable");
System.out.println(" - Product catalog: ~500 products, monthly updates");
System.out.println(" - Country codes: ~250 countries, stable");
System.out.println();
System.out.println(" Bad candidates:");
System.out.println(" - Customer accounts: millions of records, frequent updates");
System.out.println(" - Transactions: millions per day");
System.out.println(" - Audit logs: append-only, grows forever");
}
}
Expected output:
CORRECT: Use GlobalKTable for small, stable reference data
Good candidates:
- Branch directory: ~5,000 branches, rarely changes
- Currency codes: ~180 currencies, stable
- Product catalog: ~500 products, monthly updates
- Country codes: ~250 countries, stable
Bad candidates:
- Customer accounts: millions of records, frequent updates
- Transactions: millions per day
- Audit logs: append-only, grows forever
What should a senior or principal engineer know?
At the senior level, you should know when GlobalKTable is appropriate (small reference data, non-key joins) and when KTable is better (large datasets, co-partitioned joins).
At the principal level, consider:
- Memory sizing: GlobalKTable on 3 instances with 5,000 branches at 200 bytes each = 1 MB per instance. Manageable. But 10 million customer records at 500 bytes = 5 GB per instance. Too large for GlobalKTable.
- Bootstrap time: GlobalKTable reads the entire topic from offset 0 on every restart. For a compacted topic with 5,000 entries, this takes seconds. For millions of entries, it takes minutes.
- No task parallelism: GlobalKTable is processed by a single global consumer thread, not distributed across tasks. High update rates on the source topic can create a bottleneck.
- Consistency: GlobalKTable lags behind the main stream processing. A transaction may be processed before the GlobalKTable has received the latest branch update. For most reference data this lag is acceptable.
- Alternative for large lookups: if you need to join with a large dataset on a non-primary key, consider repartitioning the KStream to match the KTable key, or use an external lookup (database query via a custom processor).
package com.bank.streams;
public class GlobalKTablePrincipalKnowledge {
public static void main(String[] args) {
System.out.println("Principal-level GlobalKTable decisions:");
System.out.println();
System.out.println("Decision tree:");
System.out.println(" Is the lookup data < 100K records? -> GlobalKTable");
System.out.println(" Is the lookup data > 1M records? -> KTable + co-partition");
System.out.println(" Is the join key different from the stream key? -> GlobalKTable");
System.out.println(" Can you repartition the stream? -> KTable is cheaper");
System.out.println(" Is update frequency > 1000/sec? -> KTable");
System.out.println();
System.out.println("Monitoring:");
System.out.println(" global-consumer-lag: how far behind the global store is");
System.out.println(" global-store-size: bytes used by the GlobalKTable store");
System.out.println(" bootstrap-duration: time to rebuild on restart");
}
}
Expected output:
Principal-level GlobalKTable decisions:
Decision tree:
Is the lookup data < 100K records? -> GlobalKTable
Is the lookup data > 1M records? -> KTable + co-partition
Is the join key different from the stream key? -> GlobalKTable
Can you repartition the stream? -> KTable is cheaper
Is update frequency > 1000/sec? -> KTable
Monitoring:
global-consumer-lag: how far behind the global store is
global-store-size: bytes used by the GlobalKTable store
bootstrap-duration: time to rebuild on restart
How do you decide between KTable, GlobalKTable, and an external database lookup?
When you need to enrich a KStream with reference data, you have three options. The choice depends on data size, key alignment, and latency requirements.
package com.bank.streams;
public class LookupDecisionTree {
public static void main(String[] args) {
System.out.println("Decision tree for lookup data:");
System.out.println();
System.out.println(" Data < 100K rows, different key from stream:");
System.out.println(" -> GlobalKTable (replicated, non-key join)");
System.out.println();
System.out.println(" Data any size, same key as stream, co-partitioned:");
System.out.println(" -> KTable (partitioned, efficient, scalable)");
System.out.println();
System.out.println(" Data in external database, needs real-time freshness:");
System.out.println(" -> Custom Processor API with database lookup");
System.out.println(" -> Warning: adds latency, blocks stream thread");
System.out.println();
System.out.println(" Data > 1M rows, different key, cannot repartition:");
System.out.println(" -> External lookup or repartition the stream first");
System.out.println();
System.out.println(" Best practice: prefer KTable when possible");
System.out.println(" GlobalKTable for small reference data only");
}
}
Expected output:
Decision tree for lookup data:
Data < 100K rows, different key from stream:
-> GlobalKTable (replicated, non-key join)
Data any size, same key as stream, co-partitioned:
-> KTable (partitioned, efficient, scalable)
Data in external database, needs real-time freshness:
-> Custom Processor API with database lookup
-> Warning: adds latency, blocks stream thread
Data > 1M rows, different key, cannot repartition:
-> External lookup or repartition the stream first
Best practice: prefer KTable when possible
GlobalKTable for small reference data only
What are common interview follow-up questions?
- Can you write to a GlobalKTable from within the topology? (No. GlobalKTable is read-only. Updates come only from the source topic. You must produce to the source topic externally.)
- Does GlobalKTable participate in task rebalancing? (No. It runs on a separate global consumer thread. It is not assigned to stream tasks.)
- Can you aggregate a KStream into a GlobalKTable? (No. Aggregation produces a KTable, not a GlobalKTable. To get a GlobalKTable, you must write the aggregated results to a topic and then read that topic as a GlobalKTable.)
- What happens if a GlobalKTable source topic has many partitions? (Every instance reads all partitions. More partitions mean more connections and higher fan-out. Keep the partition count low for GlobalKTable sources.)
- How do you ensure the GlobalKTable is fully loaded before processing starts? (Kafka Streams loads the GlobalKTable to the end of the topic before starting the main topology processing. This is automatic.)
- Can two GlobalKTables exist in the same topology? (Yes. You can have multiple GlobalKTables reading from different topics. Each one runs its own global consumer.)
Visual Overview
KTable (partitioned) vs GlobalKTable (replicated)
===================================================
Topic: branch-directory (3 partitions)
KTable (partitioned):
┌───────────┐ ┌───────────┐ ┌───────────┐
│Instance A │ │Instance B │ │Instance C │
│ P0 data │ │ P1 data │ │ P2 data │
│ MUMBAI01 │ │ DELHI01 │ │ CHENNAI01 │
└───────────┘ └───────────┘ └───────────┘
Instance A cannot see DELHI01 or CHENNAI01
GlobalKTable (replicated):
┌───────────┐ ┌───────────┐ ┌───────────┐
│Instance A │ │Instance B │ │Instance C │
│ MUMBAI01 │ │ MUMBAI01 │ │ MUMBAI01 │
│ DELHI01 │ │ DELHI01 │ │ DELHI01 │
│ CHENNAI01 │ │ CHENNAI01 │ │ CHENNAI01 │
└───────────┘ └───────────┘ └───────────┘
Every instance has ALL data
KStream + GlobalKTable Join (non-key join)
==========================================
KStream (transactions) GlobalKTable (branches)
key=SAV2001 key=MUMBAI01
value=...|MUMBAI01 value=Mumbai Main|Mumbai
│ │
└──── KeyValueMapper ──────────┘
extract branchCode
from transaction value
│
▼
Enriched: ...|BRANCH:Mumbai Main
Key Takeaway
GlobalKTable replicates the entire source topic to every Kafka Streams instance. Unlike KTable, which is partitioned so each instance sees only its assigned data, GlobalKTable ensures every instance has a complete copy. This eliminates the co-partitioning requirement and enables non-key joins.
The trade-off is cost. Every instance reads all partitions of the GlobalKTable source topic, stores the complete data in its local state, and processes every update. For small reference data like the 5,000 branches in our National Bank of India directory, this is fine. For millions of customer records, use a partitioned KTable instead.
In our branch enrichment project, transactions were keyed by account ID and the branch directory was keyed by branch code. The keys were different, and the partition counts did not match. A KTable join would have failed silently. GlobalKTable solved this: the KeyValueMapper extracted the branch code from the transaction value, looked it up in the replicated branch store, and enriched the output with branch name and city.
At the principal level, use GlobalKTable only for small, slowly changing reference data. If the lookup data exceeds 100,000 records or updates more than 1,000 times per second, repartition the KStream and use a regular KTable join instead.
Remember one sentence: GlobalKTable replicates the full topic to every instance for non-key joins on small reference data, while KTable partitions the data for scalable joins on co-partitioned keys.
---