Definition
- Topic – a named, logical category of events in Kafka, like a folder that holds all records of a particular type such as
transfersortrade-events. - Partition – an ordered, append-only, immutable log within a topic stored as segment files on a broker's disk, serving as the unit of parallelism and ordering.
- Offset – a monotonically increasing 64-bit integer assigned by the broker that uniquely identifies a record's position within a specific partition, starting at zero.
- Message key – a value attached to a record that Kafka hashes to determine which partition the record is written to, ensuring records with the same key always land in the same partition.
__consumer_offsets– an internal Kafka topic where consumer groups store their committed offsets, enabling consumers to resume from exactly where they left off.
Think of National Bank of India's record-keeping. A topic is a labeled filing cabinet called "transfers" that holds all transfer records. A partition is one drawer inside that cabinet; the cabinet has several drawers so many clerks can work in parallel. An offset is the sequential page number written on each sheet in a drawer: page 0, page 1, page 2. To find any single record you say the cabinet name (topic), the drawer number (partition), and the page number (offset).
Understanding the Concepts
What is a Kafka topic?
A topic is purely a logical name that categorizes a stream of events. It does not store anything by itself. When you create a topic you specify how many partitions it has, because partitions are where the real data lives. In a banking system you might have topics named transfers, transactions, and trade-settled, each representing a distinct event category.
package com.bank.admin;
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.NewTopic;
import java.util.Collections;
import java.util.Properties;
public class TopicCreator {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
try (AdminClient admin = AdminClient.create(props)) {
NewTopic transfersTopic = new NewTopic("transfers", 6, (short) 3);
admin.createTopics(Collections.singleton(transfersTopic)).all().get();
System.out.println("Topic 'transfers' created: 6 partitions, RF=3");
}
}
}
Expected output:
Topic 'transfers' created: 6 partitions, RF=3
What is a Kafka partition?
A partition is an ordered, append-only sequence of records. New records are always appended to the end, and records are never modified once written. Kafka guarantees ordering only within a single partition, not across partitions. Each partition is stored as a set of segment files on a broker's disk, and it can be replicated to other brokers for fault tolerance. The number of partitions determines the maximum parallelism of a consumer group, because one partition is consumed by exactly one consumer in a group at a time.
package com.bank.producer;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class TransferProducer {
private final KafkaTemplate<String, String> kafkaTemplate;
public TransferProducer(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void publishTransfer(String accountId, BigDecimal amount) {
String event = "{\"accountId\":\"" + accountId
+ "\",\"amount\":" + amount + "}";
// Key = accountId ensures all transfers for SAV2001
// go to the SAME partition and stay ordered
kafkaTemplate.send("transfers", accountId, event);
System.out.println("Published transfer for " + accountId
+ " (partition chosen by key hash)");
}
}
Expected output:
Published transfer for SAV2001 (partition chosen by key hash)
-> All SAV2001 events land in partition 2 (determined by hash)
-> All SAV3002 events land in partition 5 (different hash)
What is a Kafka offset?
An offset is a 64-bit integer assigned by the broker when a record is appended. Offsets start at 0 and increase by one for each new record in that partition. Offsets are per-partition, so partition 0 and partition 1 each have their own offset 0. A consumer reads records in offset order within a partition and periodically commits the offset of the last processed record. The committed offset is stored in the internal __consumer_offsets topic.
package com.bank.consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class TransferConsumer {
@KafkaListener(topics = "transfers", groupId = "settlement-group")
public void processTransfer(ConsumerRecord<String, String> record) {
System.out.println("Received transfer:"
+ " topic=" + record.topic()
+ " partition=" + record.partition()
+ " offset=" + record.offset()
+ " key=" + record.key()
+ " value=" + record.value());
}
}
Expected output:
Received transfer: topic=transfers partition=2 offset=0 key=SAV2001 value={...}
Received transfer: topic=transfers partition=2 offset=1 key=SAV2001 value={...}
Received transfer: topic=transfers partition=5 offset=0 key=SAV3002 value={...}
How does the message key determine the partition?
If you produce with a key, Kafka hashes the key and uses the hash to pick the partition, so the same key always lands in the same partition. For banking, keying by account id means all events for one account are ordered in the same partition. If you produce without a key, Kafka distributes records across partitions in a round-robin or sticky fashion for balanced load.
package com.bank.partitioning;
import org.apache.kafka.clients.producer.internals.DefaultPartitioner;
import org.apache.kafka.common.utils.Utils;
public class PartitionCalculator {
public static int calculatePartition(String key, int numPartitions) {
// Kafka's default: murmur2 hash of key, mod number of partitions
int hash = Utils.toPositive(Utils.murmur2(key.getBytes()));
int partition = hash % numPartitions;
return partition;
}
public static void main(String[] args) {
int partitions = 6;
System.out.println("SAV2001 -> partition "
+ calculatePartition("SAV2001", partitions));
System.out.println("SAV3002 -> partition "
+ calculatePartition("SAV3002", partitions));
System.out.println("SAV2001 -> partition "
+ calculatePartition("SAV2001", partitions)
+ " (same key = same partition, always)");
}
}
Expected output:
SAV2001 -> partition 2
SAV3002 -> partition 5
SAV2001 -> partition 2 (same key = same partition, always)
How do offsets enable replay?
When a consumer restarts, it resumes from the last committed offset. A consumer can also seek to an older offset to replay data. This is how the fraud team at National Bank of India re-reads last week's events to retrain a detection model without disturbing other consumer groups.
package com.bank.replay;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.TopicPartition;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class OffsetReplayExample {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "fraud-replay-group");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
TopicPartition tp = new TopicPartition("transfers", 2);
consumer.assign(Collections.singleton(tp));
// Seek to offset 100 to replay from that point
consumer.seek(tp, 100);
System.out.println("Seeking to offset 100 on partition 2");
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
System.out.println("Replayed: offset=" + record.offset()
+ " key=" + record.key());
}
consumer.close();
}
}
Expected output:
Seeking to offset 100 on partition 2
Replayed: offset=100 key=SAV2001
Replayed: offset=101 key=SAV2001
Replayed: offset=102 key=SAV2001
How is a topic different from a partition? (comparison table)
A topic is logical; a partition is physical. A topic is a name; a partition stores the actual data.
| Feature | Topic | Partition |
|---|---|---|
| Nature | Logical name (category) | Physical log (data storage) |
| Stores data | No (just a grouping) | Yes (append-only log on disk) |
| Ordering | No ordering guarantee | Strict ordering within partition |
| Number | One per event category | Many per topic (configurable) |
| Parallelism | N/A | One partition = one consumer max |
| Replication | N/A | Each partition replicated to N brokers |
package com.bank.comparison;
public class TopicVsPartition {
public static void main(String[] args) {
System.out.println("Topic 'transfers':");
System.out.println(" Logical name, no data storage itself");
System.out.println();
System.out.println("Partitions of 'transfers' (6 total):");
System.out.println(" Partition 0: [offset 0][offset 1][offset 2]...");
System.out.println(" Partition 1: [offset 0][offset 1]...");
System.out.println(" Partition 2: [offset 0][offset 1][offset 2][offset 3]...");
System.out.println(" Each partition is an independent ordered log");
}
}
Expected output:
Topic 'transfers':
Logical name, no data storage itself
Partitions of 'transfers' (6 total):
Partition 0: [offset 0][offset 1][offset 2]...
Partition 1: [offset 0][offset 1]...
Partition 2: [offset 0][offset 1][offset 2][offset 3]...
Each partition is an independent ordered log
What are the edge cases with topics, partitions, and offsets?
One edge case is producing without a key. Without a key, Kafka uses a sticky partitioner that sends batches to random partitions, so you lose per-entity ordering. Another edge case is increasing partitions later: it changes the key-to-partition mapping, so a key that was on partition 2 may now land on partition 4, breaking ordering for existing keys.
package com.bank.edge;
import org.springframework.kafka.core.KafkaTemplate;
public class NoKeyProducer {
private final KafkaTemplate<String, String> kafkaTemplate;
public NoKeyProducer(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void publishWithoutKey(String event) {
// No key: partition is chosen by sticky partitioner (round-robin batches)
// SAV2001 events may land on ANY partition, breaking ordering
kafkaTemplate.send("transfers", event);
System.out.println("Published without key (no ordering guarantee)");
}
public void publishWithKey(String accountId, String event) {
// With key: hash(accountId) determines partition
// SAV2001 events always land on the SAME partition
kafkaTemplate.send("transfers", accountId, event);
System.out.println("Published with key " + accountId
+ " (ordering guaranteed per key)");
}
}
Expected output:
Published without key (no ordering guarantee)
-> Event could land on partition 0, 3, 5 (random)
Published with key SAV2001 (ordering guaranteed per key)
-> Event always lands on partition 2 (hash of SAV2001)
What should a senior engineer know?
Offsets are the key to Kafka's replay capability and its delivery guarantees. Committing the offset after processing gives at-least-once delivery, while committing before processing gives at-most-once. Because offsets are durable and per-partition, a brand-new consumer group can read the entire history of a topic from offset 0 while another group reads only new records from the latest offset, both at the same time without interference.
package com.bank.senior;
public class OffsetSemantics {
public static void main(String[] args) {
System.out.println("Offset Commit Timing and Delivery Semantics:");
System.out.println();
System.out.println("At-most-once: commit offset BEFORE processing");
System.out.println(" Risk: crash after commit, message lost");
System.out.println();
System.out.println("At-least-once: commit offset AFTER processing");
System.out.println(" Risk: crash after processing, message redelivered");
System.out.println(" Solution: idempotent consumer");
System.out.println();
System.out.println("auto.offset.reset:");
System.out.println(" earliest -> new group reads ALL retained records");
System.out.println(" latest -> new group reads only NEW records");
}
}
Expected output:
Offset Commit Timing and Delivery Semantics:
At-most-once: commit offset BEFORE processing
Risk: crash after commit, message lost
At-least-once: commit offset AFTER processing
Risk: crash after processing, message redelivered
Solution: idempotent consumer
auto.offset.reset:
earliest -> new group reads ALL retained records
latest -> new group reads only NEW records
Visual Overview
Topic, Partition, and Offset Structure
======================================
Topic: "transfers"
┌───────────────────────────────────────────────────┐
│ │
│ Partition 0: [0] [1] [2] [3] [4] [5] │
│ ▲ ▲ │
│ │ │ │
│ oldest offset latest offset │
│ │
│ Partition 1: [0] [1] [2] [3] │
│ │
│ Partition 2: [0] [1] [2] [3] [4] [5] [6] [7] │
│ │
└───────────────────────────────────────────────────┘
Record Location = (topic, partition, offset)
Example: ("transfers", 2, 5) = Rajesh's 6th transfer
Key Routing:
┌──────────────┐ hash("SAV2001") % 6 = 2
│ Producer │────────────────────────────▶ Partition 2
│ key=SAV2001 │
└──────────────┘
┌──────────────┐ hash("SAV3002") % 6 = 5
│ Producer │────────────────────────────▶ Partition 5
│ key=SAV3002 │
└──────────────┘
Consumer Progress:
Partition 2: [0] [1] [2] [3] [4] [5] [6] [7]
▲ ▲
│ │
committed offset latest offset
(resume here) (newest record)
Key Takeaway
In Kafka, a topic, a partition, and an offset are three levels of the same storage model. A topic is a logical category of events, like a named folder: for example a transfers topic that holds all transfer events at National Bank of India. A topic by itself stores nothing; it is split into one or more partitions, and partitions are where the data actually lives as ordered, append-only, immutable logs.
A partition is the unit of both ordering and parallelism. Kafka guarantees strict ordering only within a single partition, and one partition is consumed by exactly one consumer in a group at a time. The record's partition is chosen by hashing its key, so Rajesh's account SAV2001 always maps to the same partition and his transfers stay perfectly ordered, while Priya's account SAV3002 maps to a different partition and processes in parallel.
An offset is a sequential integer that identifies a record's position within one partition, starting at zero. Consumers commit the offset of the last record they processed, so they can resume from exactly that point after a restart or crash. Offsets also enable replay: the fraud team can seek back to an earlier offset to re-read historical events without disturbing other consumer groups.
Remember one sentence: a topic is the category, a partition is the ordered log, and an offset is the position, and together the triple of topic, partition, and offset locates any record in Kafka.
---