Definition
Spring Kafka is an abstraction over the raw Apache Kafka client that integrates Kafka into the Spring programming model.
- KafkaTemplate – a thread-safe wrapper around a single shared
KafkaProducer;send(topic, key, value)returns aCompletableFuture. - @KafkaListener – a declarative annotation that replaces the manual poll loop; the listener container manages polling, offset commits, error handling, and lifecycle.
- ConcurrentKafkaListenerContainerFactory – creates the container; its
concurrencyproperty sets how many consumer threads run in the same group. - Partition-thread mapping – one partition is assigned to exactly one consumer in a group; effective parallelism is capped by partition count.
- Ordering guarantee – keying by account ID routes all of one account's events to the same partition and the same thread, preserving order.
The raw Kafka client is like hiring a bank clerk at National Bank of India and personally teaching them every step: open the counter, pull the next token, stamp it, close for the night. Spring Kafka is like hiring through an agency that already trained the clerk: you just say what to do with each customer, and the agency handles opening, closing, rotating staff, and covering sick leave. KafkaTemplate is the one shared outgoing-mail clerk everyone uses, and @KafkaListener with concurrency is the agency assigning several clerks to several counters, one clerk per counter.
Understanding the Concepts
What does KafkaTemplate provide over the raw producer?
With the raw client you must construct a KafkaProducer, call send, deal with the returned Future, flush, and close it yourself. Spring Kafka gives you KafkaTemplate, which wraps one shared producer and is fully thread-safe, so every thread in the application can call it.
package com.bank.kafka;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.SendResult;
import org.springframework.stereotype.Service;
import java.util.concurrent.CompletableFuture;
@Service
public class TransferEventPublisher {
private final KafkaTemplate<String, String> kafkaTemplate;
public TransferEventPublisher(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void publishTransfer(String accountId, String event) {
CompletableFuture<SendResult<String, String>> future =
kafkaTemplate.send("transfer.completed", accountId, event);
future.whenComplete((result, ex) -> {
if (ex == null) {
System.out.println("Sent to partition "
+ result.getRecordMetadata().partition());
} else {
System.err.println("Send failed: " + ex.getMessage());
}
});
}
}
Sent to partition 3
What does @KafkaListener provide over the raw consumer?
The raw client forces you to write an infinite poll loop with manual deserialization, offset commits, and shutdown handling. Spring Kafka replaces all of that with a declarative annotation. The container deserializes each record, invokes your method, commits offsets, catches exceptions, and manages start and stop.
package com.bank.kafka;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class SettlementListener {
@KafkaListener(topics = "transfer.completed", groupId = "settlement-group")
public void onTransfer(String event) {
System.out.println("Settling: " + event);
// Business logic only -- container handles poll, commit, errors
}
}
Settling: {"accountId":"SAV2001","amount":50000}
How does the concurrency setting map threads to partitions?
Setting factory.setConcurrency(3) creates three child containers, each with its own consumer, all in the same consumer group. Kafka's rebalance distributes partitions across them. The rule is that a single partition is only ever assigned to one consumer in the group at a time.
package com.bank.kafka;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import java.util.Map;
import org.apache.kafka.clients.consumer.ConsumerConfig;
@Configuration
public class KafkaConsumerConfig {
@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(3); // 3 consumer threads
return factory;
}
@Bean
public ConsumerFactory<String, String> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(Map.of(
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092",
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class,
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class
));
}
}
# Topic with 12 partitions, concurrency=3:
# Thread-0 owns partitions [0, 1, 2, 3]
# Thread-1 owns partitions [4, 5, 6, 7]
# Thread-2 owns partitions [8, 9, 10, 11]
# Topic with 2 partitions, concurrency=3:
# Thread-0 owns partition [0]
# Thread-1 owns partition [1]
# Thread-2 is IDLE (no partition to assign)
Why must you not create a KafkaTemplate per request?
A Kafka producer is a heavy object that batches and compresses across threads for throughput. Creating one per request wastes resources and prevents batching. KafkaTemplate is designed to be shared.
package com.bank.kafka;
// WRONG: creating a new producer per request
public class BadProducerUsage {
public void send(String event) {
// KafkaTemplate template = new KafkaTemplate<>(new DefaultKafkaProducerFactory<>(...));
// template.send("topic", event);
// This creates a new producer every call -- expensive, no batching!
System.out.println("WRONG: new producer per request");
}
}
// CORRECT: inject the shared KafkaTemplate
// @Service
// public class GoodProducerUsage {
// private final KafkaTemplate<String, String> template; // injected, shared
// public void send(String event) { template.send("topic", event); }
// }
WRONG: new producer per request
How does keying preserve ordering with concurrency?
Keying by account ID routes all of Rajesh's events to the same partition. Since one partition is assigned to one thread, all of Rajesh's events are processed in order by a single thread, even with high concurrency.
package com.bank.kafka;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
@Service
public class OrderPreservingPublisher {
private final KafkaTemplate<String, String> template;
public OrderPreservingPublisher(KafkaTemplate<String, String> template) {
this.template = template;
}
public void publish(String accountId, String event) {
// Key = accountId: all events for SAV2001 go to the same partition
template.send("transfer.completed", accountId, event);
System.out.println("Event for " + accountId + " routed to consistent partition");
}
}
Event for SAV2001 routed to consistent partition
How do you configure Spring Kafka with application.properties?
Spring Boot auto-configures most settings from properties, so you often do not need a manual configuration class.
# application.properties
spring.kafka.bootstrap-servers=kafka-1:9092
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer
spring.kafka.consumer.group-id=settlement-group
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.concurrency=3
How do you scale across multiple application instances?
The concurrency number is per instance. If you run three instances each with concurrency=4, that is twelve consumers in the group. Kafka distributes the topic's partitions across all twelve. Total parallelism is still capped by partition count across the whole group.
# Topic: transfer.completed (12 partitions)
# Instance-1 (concurrency=4): threads own partitions [0,1,2,3]
# Instance-2 (concurrency=4): threads own partitions [4,5,6,7]
# Instance-3 (concurrency=4): threads own partitions [8,9,10,11]
# Perfect utilization: 12 threads for 12 partitions
# If you add Instance-4 (concurrency=4):
# 16 threads but only 12 partitions -- 4 threads idle
What should a senior engineer know?
Choose a partition count that matches peak desired parallelism, then set concurrency per instance so instances times concurrency does not exceed partitions. KafkaTemplate being one shared producer is intentional because producers batch across threads. Ordering per partition is the foundation of per-account ordering, which is critical in banking.
# Sizing rule of thumb
# Peak parallelism needed: 12
# Create topic with 12 partitions
# Deploy 3 instances with concurrency=4 each (3 x 4 = 12)
# Or deploy 4 instances with concurrency=3 each (4 x 3 = 12)
Visual Overview
Spring Kafka Architecture Producing: ┌───────────────────────────────────────────────────┐ │ Application Threads │ │ Thread-A ──┐ │ │ Thread-B ──┤──→ KafkaTemplate (shared, thread-safe)│ │ Thread-C ──┘ │ │ │ ▼ │ │ KafkaProducer (one instance) │ │ Batches + compresses across threads │ │ │ │ │ ▼ │ │ Kafka Broker │ └───────────────────────────────────────────────────┘ Consuming (concurrency=3, 12 partitions): ┌───────────────────────────────────────────────────┐ │ ConcurrentKafkaListenerContainerFactory │ │ │ │ Thread-0 ──→ Consumer-0 ──→ Partitions [0,1,2,3] │ │ Thread-1 ──→ Consumer-1 ──→ Partitions [4,5,6,7] │ │ Thread-2 ──→ Consumer-2 ──→ Partitions [8,9,10,11]│ │ │ │ Each thread calls @KafkaListener method │ │ Container handles: poll, commit, errors, lifecycle │ └───────────────────────────────────────────────────┘
Key Takeaway
Spring Kafka is a productive abstraction over the raw Apache Kafka client. On the producing side, KafkaTemplate wraps a single shared, thread-safe producer, and you publish with send(topic, key, value) returning a CompletableFuture. On the consuming side, @KafkaListener replaces the hand-written poll loop: the container manages polling, deserialization, offset commits, error handling, and lifecycle so you write only business logic.
Parallelism comes from the concurrency setting on the listener container factory. It creates that many consumer threads in the same group, and Kafka's rebalance distributes partitions across them. The rule is that a partition is owned by exactly one consumer in the group, so effective parallelism is capped by partition count. Extra threads sit idle.
In the settlement service at National Bank of India, Pankaj keyed every transaction by account ID so all of Rajesh's events landed on one partition and stayed strictly ordered even under high concurrency. The partition count was set to match peak parallelism, and concurrency per instance was configured so instances times concurrency equaled the partition count.
Remember one sentence: KafkaTemplate is your shared sender, @KafkaListener is your declarative consumer, and concurrency gives you one thread per partition up to the partition count.
---