Definition
- Apache Kafka – a distributed event streaming platform that stores ordered, durable streams of records in partitioned topics, serving as the central nervous system for event-driven architectures.
- Real-time streaming – consuming and processing events continuously as they arrive, enabling sub-second reactions like fraud detection on live transactions.
- Log aggregation – centralizing logs from many services into Kafka topics, then sinking them to Elasticsearch, Splunk, or a data lake for analysis.
- Event-driven microservices – an architecture where services communicate by publishing and reacting to events on Kafka topics instead of calling each other synchronously.
- Change Data Capture (CDC) – capturing row-level insert, update, and delete changes from a database and streaming them into Kafka using connectors like Debezium.
- MirrorMaker 2 – a Kafka tool that replicates topics between clusters for cross-datacenter disaster recovery and multi-region synchronization.
Think of Kafka as the central nervous system of National Bank of India. Every time something happens: Rajesh swipes a card, Priya pays rent, an ATM dispenses cash, a tiny signal (event) is sent down the nervous system. Different organs react: the fraud brain watches for danger in real time, the memory keeps a permanent log, the reporting organ builds dashboards, and the backup body in another city copies every signal. One event, many reactions, all from the same reliable stream.
Understanding the Concepts
What are the main use cases of Kafka?
Kafka serves as the event backbone for a wide range of use cases in modern distributed systems. Each use case takes advantage of the same core properties: partitioned topics for horizontal scale, configurable retention for replay, and consumer groups for independent parallel consumption. In banking, a single transfer event can simultaneously feed real-time fraud detection, audit logging, customer notifications, data warehousing, and disaster-recovery replication.
package com.bank.events;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class TransferEventPublisher {
private final KafkaTemplate<String, String> kafkaTemplate;
public TransferEventPublisher(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void publishTransfer(String accountId, BigDecimal amount) {
String event = "{\"accountId\":\"" + accountId
+ "\",\"amount\":" + amount + ",\"type\":\"TRANSFER\"}";
kafkaTemplate.send("transactions", accountId, event);
System.out.println("Published transfer event for: " + accountId);
}
}
Expected output:
Published transfer event for: SAV2001
-> Fraud service consumes (fraud-group)
-> Audit service consumes (audit-group)
-> Notification service consumes (notification-group)
-> Analytics pipeline consumes (analytics-group)
How does Kafka power real-time streaming?
Real-time streaming is Kafka's headline use case. Producers emit events and stream processors such as Kafka Streams or Apache Flink compute results continuously. In banking, a fraud engine reads the transaction stream and flags anomalies within milliseconds. The consumer processes each event as it arrives, maintaining a rolling state per account.
package com.bank.fraud;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
import java.math.BigDecimal;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
@Component
public class RealTimeFraudDetector {
private final Map<String, BigDecimal> rollingSum = new ConcurrentHashMap<>();
private static final BigDecimal VELOCITY_LIMIT = new BigDecimal("500000.00");
@KafkaListener(topics = "transactions", groupId = "fraud-group")
public void detectFraud(String eventJson) {
// Parse accountId and amount from event
String accountId = "SAV2001"; // parsed from JSON
BigDecimal amount = new BigDecimal("50000.00"); // parsed from JSON
BigDecimal total = rollingSum.merge(accountId, amount, BigDecimal::add);
if (total.compareTo(VELOCITY_LIMIT) > 0) {
System.out.println("ALERT: Velocity limit exceeded for " + accountId
+ " total=" + total);
} else {
System.out.println("Transaction OK: " + accountId
+ " running total=" + total);
}
}
}
Expected output:
Transaction OK: SAV2001 running total=50000.00
Transaction OK: SAV2001 running total=100000.00
...
ALERT: Velocity limit exceeded for SAV2001 total=550000.00
How does Kafka handle log aggregation?
Log aggregation centralizes logs from hundreds of microservices into Kafka topics, from where they flow to Elasticsearch or a data lake. Instead of each microservice writing to its own file, every service produces structured log events to a shared app-logs topic. This decouples log producers from log storage and lets you switch backends without changing any service.
package com.bank.logging;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;
import java.time.Instant;
@Component
public class CentralizedLogger {
private final KafkaTemplate<String, String> kafkaTemplate;
public CentralizedLogger(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void log(String serviceName, String level, String message) {
String logEvent = "{\"service\":\"" + serviceName
+ "\",\"level\":\"" + level
+ "\",\"message\":\"" + message
+ "\",\"timestamp\":\"" + Instant.now() + "\"}";
kafkaTemplate.send("app-logs", serviceName, logEvent);
}
}
Expected output (Kafka -> Elasticsearch):
transfer-service -> app-logs topic -> Elasticsearch (logs-group)
fraud-service -> app-logs topic -> Elasticsearch (logs-group)
payment-service -> app-logs topic -> Elasticsearch (logs-group)
How does Kafka enable event-driven microservices?
Event-driven microservices use Kafka as the integration backbone. Services do not call each other synchronously; they publish events and react to events. When the transfer service completes a transfer, it publishes a TransferCompleted event, and the notification and ledger services react independently. This removes tight coupling and allows services to evolve separately.
package com.bank.notification;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class NotificationService {
@KafkaListener(topics = "transfer-completed", groupId = "notification-group")
public void onTransferCompleted(String eventJson) {
// Parse event and send SMS/email to customer
System.out.println("Sending notification for transfer: " + eventJson);
}
}
package com.bank.ledger;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class LedgerService {
@KafkaListener(topics = "transfer-completed", groupId = "ledger-group")
public void onTransferCompleted(String eventJson) {
// Update the general ledger
System.out.println("Updating ledger for transfer: " + eventJson);
}
}
Expected output:
Transfer service publishes: transfer-completed
-> NotificationService: Sending notification for transfer: {...}
-> LedgerService: Updating ledger for transfer: {...}
How does Change Data Capture (CDC) work with Kafka?
CDC streams row-level changes from a database into Kafka using tools like Debezium. Every insert, update, and delete on the accounts table becomes an event in a Kafka topic. This keeps caches, search indexes, and downstream systems in sync with the source database without dual writes, which avoids the consistency problems of writing to both a database and Kafka in application code.
package com.bank.cdc;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class AccountCdcConsumer {
@KafkaListener(topics = "dbserver1.bank.accounts", groupId = "cache-sync-group")
public void syncAccountCache(String cdcEventJson) {
// Debezium CDC event contains before/after state of the row
System.out.println("CDC event received: " + cdcEventJson);
// Update Redis cache with latest account state
System.out.println("Cache updated for account from CDC");
}
}
Expected output:
Database: UPDATE accounts SET balance = 75000 WHERE id = 'SAV2001'
Debezium captures change -> Kafka topic: dbserver1.bank.accounts
CDC event received: {"before":{"balance":50000},"after":{"balance":75000}}
Cache updated for account from CDC
How does Kafka feed big-data pipelines?
Big-data pipelines use Kafka as the ingestion layer. Raw events land in Kafka, then Kafka Connect sinks them into HDFS, S3, or a warehouse like Snowflake for batch analytics. Kafka buffers massive spikes so the slower warehouse is not overwhelmed. This decouples the real-time event producers from the batch analytics consumers.
package com.bank.pipeline;
public class BigDataPipelineOverview {
public static void main(String[] args) {
System.out.println("Big Data Pipeline Architecture:");
System.out.println("1. Services produce events -> Kafka topics");
System.out.println("2. Kafka Connect S3 Sink -> S3 bucket (Parquet)");
System.out.println("3. Snowflake reads from S3 -> analytics queries");
System.out.println();
System.out.println("Kafka buffers 1M events/sec during peak hours.");
System.out.println("S3 sink writes in 5-minute batches.");
System.out.println("No events lost, warehouse never overwhelmed.");
}
}
Expected output:
Big Data Pipeline Architecture:
1. Services produce events -> Kafka topics
2. Kafka Connect S3 Sink -> S3 bucket (Parquet)
3. Snowflake reads from S3 -> analytics queries
Kafka buffers 1M events/sec during peak hours.
S3 sink writes in 5-minute batches.
No events lost, warehouse never overwhelmed.
How does Kafka handle cross-datacenter replication?
Replication and synchronization across datacenters use MirrorMaker 2 to copy topics from one Kafka cluster to another. This enables disaster recovery and multi-region active-active setups. If the Mumbai datacenter fails, the Chennai datacenter already has every event replicated and can take over processing.
package com.bank.replication;
public class MirrorMakerOverview {
public static void main(String[] args) {
System.out.println("Cross-Datacenter Replication with MirrorMaker 2:");
System.out.println("Mumbai cluster (primary):");
System.out.println(" transfer-events: 6 partitions, RF=3");
System.out.println();
System.out.println("Chennai cluster (DR replica):");
System.out.println(" mumbai.transfer-events: mirrored topic");
System.out.println();
System.out.println("MirrorMaker 2 copies records in near-real-time.");
System.out.println("If Mumbai fails, Chennai consumers switch topics.");
}
}
Expected output:
Cross-Datacenter Replication with MirrorMaker 2:
Mumbai cluster (primary):
transfer-events: 6 partitions, RF=3
Chennai cluster (DR replica):
mumbai.transfer-events: mirrored topic
MirrorMaker 2 copies records in near-real-time.
If Mumbai fails, Chennai consumers switch topics.
What about content delivery and fan-out?
Content delivery and fan-out push the same content or notification to many consumers efficiently. A single "new statement available" event fans out to email, SMS, and push-notification consumers. Each consumer group processes the event independently, so adding a new notification channel is just adding a new consumer group.
package com.bank.notification;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class MultiChannelNotifier {
@KafkaListener(topics = "statement-available", groupId = "email-group")
public void sendEmail(String event) {
System.out.println("Email sent for statement: " + event);
}
@KafkaListener(topics = "statement-available", groupId = "sms-group")
public void sendSms(String event) {
System.out.println("SMS sent for statement: " + event);
}
@KafkaListener(topics = "statement-available", groupId = "push-group")
public void sendPush(String event) {
System.out.println("Push notification sent: " + event);
}
}
Expected output (one event, three channels):
Email sent for statement: {"accountId":"SAV2001","month":"2025-01"}
SMS sent for statement: {"accountId":"SAV2001","month":"2025-01"}
Push notification sent: {"accountId":"SAV2001","month":"2025-01"}
What should a senior engineer know about Kafka use cases?
The senior insight is that all these use cases share the same three Kafka properties applied differently. Partitioning gives horizontal scale, retention gives replay, and consumer groups give independent parallel consumption. Once you internalize that, every Kafka use case is the same tool pointed at a different problem. The key design decision is choosing the right topic structure, key strategy, and retention policy for each use case.
package com.bank.architecture;
public class KafkaUseCaseSummary {
public static void main(String[] args) {
System.out.println("All Kafka use cases share three core properties:");
System.out.println("1. Partitioning -> horizontal scale");
System.out.println("2. Retention -> replay capability");
System.out.println("3. Consumer groups -> independent consumption");
System.out.println();
System.out.println("Use Case | Key Property Used");
System.out.println("Real-time streaming | Partitioning (parallel processing)");
System.out.println("Log aggregation | Retention (long-term storage)");
System.out.println("Event-driven micro | Consumer groups (independence)");
System.out.println("CDC | All three (sync, replay, parallel)");
System.out.println("Big-data pipeline | Retention (buffering spikes)");
System.out.println("Replication (DR) | Retention + MirrorMaker 2");
System.out.println("Content fan-out | Consumer groups (multi-channel)");
}
}
Expected output:
All Kafka use cases share three core properties:
1. Partitioning -> horizontal scale
2. Retention -> replay capability
3. Consumer groups -> independent consumption
Use Case | Key Property Used
Real-time streaming | Partitioning (parallel processing)
Log aggregation | Retention (long-term storage)
Event-driven micro | Consumer groups (independence)
CDC | All three (sync, replay, parallel)
Big-data pipeline | Retention (buffering spikes)
Replication (DR) | Retention + MirrorMaker 2
Content fan-out | Consumer groups (multi-channel)
Visual Overview
Kafka Use Cases in a Banking Platform
=====================================
┌──────────────────┐
│ Producers │
│ (all services) │
└────────┬─────────┘
│
▼
┌──────────────────────────────────┐
│ Kafka Cluster │
│ ┌────────────────────────────┐ │
│ │ transactions topic │ │
│ │ [P0] [P1] [P2] [P3] [P4] │ │
│ └────────────────────────────┘ │
└──────────────┬───────────────────┘
│
┌──────────┬───────┼───────┬──────────┐
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌──────────┐ ┌────────┐ ┌────┐ ┌──────┐ ┌────────┐
│ Fraud │ │ Audit │ │Log │ │Notify│ │Pipeline│
│ Detector │ │Service │ │Agg │ │Svc │ │(S3) │
│(stream) │ │(comply)│ │(ELK)│ │(fan) │ │(batch) │
└──────────┘ └────────┘ └────┘ └──────┘ └────────┘
MirrorMaker 2
┌──────────────┐ ┌──────────────┐
│ Mumbai │──────▶│ Chennai │
│ Cluster │ copy │ Cluster (DR) │
└──────────────┘ └──────────────┘
Key Takeaway
Kafka's main use cases all flow from one strength: it stores ordered, durable, replayable streams that many independent consumers can read in parallel. Real-time streaming lets a fraud detector react to transactions within milliseconds. Log aggregation centralizes logs from all services into a single topic that flows to Elasticsearch. Event-driven microservices publish and react to events instead of calling each other, removing tight coupling between services.
Change Data Capture with Debezium turns every database row change into an event, keeping caches and search indexes in sync without dual writes. Big-data pipelines use Kafka as the ingestion buffer, absorbing millions of events per second and sinking them to warehouses in batches. Cross-datacenter replication with MirrorMaker 2 copies topics between clusters for disaster recovery. Content fan-out lets one event reach email, SMS, and push notification consumers, each running as an independent consumer group.
In National Bank of India's platform, a single transfer event published to the transactions topic simultaneously feeds the real-time fraud engine, the centralized audit log, the notification service, and the nightly data warehouse, and it is mirrored to the disaster-recovery region in Chennai. That is the power of Kafka: one event written once, serving many use cases independently.
Remember one sentence: Kafka's partitioning gives scale, retention gives replay, and consumer groups give independence, and those three properties serve every use case from fraud detection to disaster recovery.
---