Definition
- Stateless operation is a stream processing step that looks at one record at a time without referencing any previously seen record or stored state.
- filter() is a stateless operation that evaluates a predicate on each record and passes through only records that match.
- map() and mapValues() are transformation operations; map() can change both key and value while mapValues() changes only the value.
- flatMap() is a one-to-many transformation that takes one input record and produces zero, one, or many output records.
- selectKey() is a re-keying operation that changes the record key, which marks the stream for repartitioning.
- branch() is a routing operation that splits one stream into multiple named sub-streams based on predicates.
- peek() is a side-effect operation that runs an action on each record (like logging) without changing the record.
Think of a payment processing assembly line at National Bank of India. Each workstation does one job to the payment slip passing through. The fraud check station (filter) passes clean payments and rejects suspicious ones. The fee calculation station (mapValues) adds a processing fee to the amount. The multi-beneficiary desk (flatMap) splits one payment into three separate slips for three recipients. The routing desk (branch) sends payments above one lakh to the priority counter and smaller ones to the regular counter. The audit camera (peek) photographs every slip without touching it. None of these stations remember the previous slip; each slip is handled on its own.
Understanding the Concepts
What are stateless operations in Kafka Streams?
Stateless operations process each record independently. They do not maintain any local state store, do not reference previously seen records, and do not require changelog topics for fault tolerance. This makes them lightweight and fast.
The stateless operations available in Kafka Streams are: filter, filterNot, map, mapValues, flatMap, flatMapValues, selectKey, branch, merge, peek, foreach, and print. Each operation takes one record as input and produces zero or more records as output.
package com.bank.streams;
public class StatelessOperationsOverview {
public static void main(String[] args) {
System.out.println("Stateless operations in Kafka Streams:");
System.out.println(" filter / filterNot : keep or discard records");
System.out.println(" map / mapValues : transform key+value or value only");
System.out.println(" flatMap / flatMapValues : one-to-many transformation");
System.out.println(" selectKey : change the record key");
System.out.println(" branch : split stream by predicates");
System.out.println(" merge : combine multiple streams");
System.out.println(" peek : side-effect without modification");
System.out.println();
System.out.println("Key property: no state store, no changelog topic.");
System.out.println("Each record is processed independently.");
}
}
Expected output:
Stateless operations in Kafka Streams:
filter / filterNot : keep or discard records
map / mapValues : transform key+value or value only
flatMap / flatMapValues : one-to-many transformation
selectKey : change the record key
branch : split stream by predicates
merge : combine multiple streams
peek : side-effect without modification
Key property: no state store, no changelog topic.
Each record is processed independently.
How does filter work?
filter() evaluates a predicate on each record. Records that match the predicate pass through; records that do not match are discarded. filterNot() does the opposite: it keeps records that do not match the predicate.
At National Bank of India, Rajesh uses filter to detect suspicious transactions. Any payment above 500,000 rupees is routed to the fraud review queue.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
import java.math.BigDecimal;
public class FilterExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> payments = builder.stream("payments");
BigDecimal fraudThreshold = new BigDecimal("500000");
// filter: keep only high-value payments
KStream<String, String> suspicious = payments
.filter((accountId, value) -> {
BigDecimal amount = new BigDecimal(value.split("\\|")[2]);
return amount.compareTo(fraudThreshold) > 0;
});
suspicious.to("fraud-review");
// filterNot: keep only normal payments (below threshold)
KStream<String, String> normal = payments
.filterNot((accountId, value) -> {
BigDecimal amount = new BigDecimal(value.split("\\|")[2]);
return amount.compareTo(fraudThreshold) > 0;
});
normal.to("normal-processing");
Topology topology = builder.build();
System.out.println("Filter topology built.");
System.out.println(" Input: payments");
System.out.println(" Output: fraud-review (> 500000)");
System.out.println(" Output: normal-processing (<= 500000)");
}
}
Expected output:
Filter topology built.
Input: payments
Output: fraud-review (> 500000)
Output: normal-processing (<= 500000)
Example records:
SAV2001|NEFT|250000|transfer to Priya -> normal-processing
CUR3001|RTGS|750000|vendor payment -> fraud-review
SAV2002|IMPS|5000|mobile recharge -> normal-processing
How do map and mapValues work?
map() transforms both the key and the value of each record. Because the key can change, Kafka Streams marks the resulting stream for repartitioning. mapValues() transforms only the value and leaves the key unchanged, so no repartitioning is needed.
Use mapValues() when you only need to change the value. Use map() when you need to change the key, but be aware of the repartitioning cost.
At National Bank of India, Rajesh uses mapValues to calculate processing fees. The payment amount stays the same, but a fee field is appended to the value.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.Topology;
import java.math.BigDecimal;
import java.math.RoundingMode;
public class MapExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> payments = builder.stream("payments");
// mapValues: add processing fee (0.5% of amount)
KStream<String, String> withFee = payments.mapValues(value -> {
String[] parts = value.split("\\|");
BigDecimal amount = new BigDecimal(parts[2]);
BigDecimal fee = amount.multiply(new BigDecimal("0.005"))
.setScale(2, RoundingMode.HALF_UP);
return value + "|FEE:" + fee;
});
withFee.to("payments-with-fee");
// map: rekey by branch code extracted from description
KStream<String, String> rekeyed = payments.map((accountId, value) -> {
String branchCode = accountId.substring(0, 3);
return KeyValue.pair(branchCode, value);
});
rekeyed.to("payments-by-branch");
Topology topology = builder.build();
System.out.println("Map examples:");
System.out.println(" mapValues: adds fee without repartition");
System.out.println(" map: rekeys by branch, triggers repartition");
}
}
Expected output:
Map examples:
mapValues: adds fee without repartition
map: rekeys by branch, triggers repartition
Example mapValues result:
Input: SAV2001|NEFT|250000|transfer to Priya
Output: SAV2001|NEFT|250000|transfer to Priya|FEE:1250.00
Example map result:
Input key: SAV2001 Input value: SAV2001|NEFT|250000|transfer to Priya
Output key: SAV Output value: SAV2001|NEFT|250000|transfer to Priya
How does flatMap work?
flatMap() takes one input record and produces zero, one, or many output records. Each output record can have a different key and value. Because the key can change, flatMap triggers repartitioning. flatMapValues() produces multiple values for the same key and does not trigger repartitioning.
At National Bank of India, a corporate customer sends a single bulk payment instruction with multiple beneficiaries. Rajesh uses flatMap to split it into individual transfers.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.Topology;
import java.util.ArrayList;
import java.util.List;
public class FlatMapExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Input: bulk payment with multiple beneficiaries separated by semicolons
// Key: sender account, Value: senderAcc|type|benef1:amt1;benef2:amt2;benef3:amt3
KStream<String, String> bulkPayments = builder.stream("bulk-payments");
KStream<String, String> individualTransfers = bulkPayments.flatMap(
(senderAccount, bulkValue) -> {
String[] parts = bulkValue.split("\\|");
String sender = parts[0];
String type = parts[1];
String[] beneficiaries = parts[2].split(";");
List<KeyValue<String, String>> transfers = new ArrayList<>();
for (String beneficiary : beneficiaries) {
String[] bParts = beneficiary.split(":");
String benefAccount = bParts[0];
String amount = bParts[1];
transfers.add(KeyValue.pair(
benefAccount,
sender + "|" + type + "|" + amount + "|to:" + benefAccount
));
}
return transfers;
}
);
individualTransfers.to("individual-transfers");
Topology topology = builder.build();
System.out.println("FlatMap topology: bulk payment -> individual transfers");
System.out.println(" Input: 1 bulk payment with 3 beneficiaries");
System.out.println(" Output: 3 individual transfer records");
}
}
Expected output:
FlatMap topology: bulk payment -> individual transfers
Input: 1 bulk payment with 3 beneficiaries
Output: 3 individual transfer records
Example:
Input: CUR3001|NEFT|SAV2001:50000;SAV2002:30000;SAV2003:20000
Output record 1: key=SAV2001, value=CUR3001|NEFT|50000|to:SAV2001
Output record 2: key=SAV2002, value=CUR3001|NEFT|30000|to:SAV2002
Output record 3: key=SAV2003, value=CUR3001|NEFT|20000|to:SAV2003
How does selectKey work?
selectKey() changes the key of each record without modifying the value. The resulting stream is marked for repartitioning. Any downstream key-based operation (join, groupByKey, aggregate) will trigger creation of an internal repartition topic.
At National Bank of India, transactions arrive keyed by account ID. Rajesh uses selectKey to rekey them by branch code so he can aggregate transactions per branch.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
public class SelectKeyExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Payments keyed by account ID: SAV2001, CUR3001, SAV2002
KStream<String, String> payments = builder.stream("payments");
// Rekey by branch code (first 3 characters of account ID)
KStream<String, String> byBranch = payments.selectKey(
(accountId, value) -> accountId.substring(0, 3)
);
// Now key is branch code: SAV, CUR
byBranch.to("payments-by-branch");
Topology topology = builder.build();
System.out.println("SelectKey: rekey from account ID to branch code");
System.out.println(" SAV2001 -> SAV");
System.out.println(" CUR3001 -> CUR");
System.out.println(" SAV2002 -> SAV");
System.out.println("Stream is marked for repartitioning.");
}
}
Expected output:
SelectKey: rekey from account ID to branch code
SAV2001 -> SAV
CUR3001 -> CUR
SAV2002 -> SAV
Stream is marked for repartitioning.
How does branch work?
branch() splits one stream into multiple named sub-streams based on predicates. In Kafka Streams 3.x, branch() returns a BranchedKStream where you define named branches with conditions. Each record goes to the first branch whose predicate matches. If no predicate matches, the record is dropped unless you add a defaultBranch().
At National Bank of India, Rajesh routes payments to different processing tiers: premium (above 1,00,000), standard (10,000 to 1,00,000), and micro (below 10,000).
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.Branched;
import org.apache.kafka.streams.Topology;
import java.math.BigDecimal;
import java.util.Map;
public class BranchExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> payments = builder.stream("payments");
BigDecimal premiumThreshold = new BigDecimal("100000");
BigDecimal microThreshold = new BigDecimal("10000");
Map<String, KStream<String, String>> branches = payments
.split()
.branch((key, value) -> {
BigDecimal amount = new BigDecimal(value.split("\\|")[2]);
return amount.compareTo(premiumThreshold) > 0;
}, Branched.as("premium"))
.branch((key, value) -> {
BigDecimal amount = new BigDecimal(value.split("\\|")[2]);
return amount.compareTo(microThreshold) < 0;
}, Branched.as("micro"))
.defaultBranch(Branched.as("standard"));
branches.get("premium").to("premium-payments");
branches.get("micro").to("micro-payments");
branches.get("standard").to("standard-payments");
Topology topology = builder.build();
System.out.println("Branch routing:");
System.out.println(" > 100000 INR -> premium-payments");
System.out.println(" < 10000 INR -> micro-payments");
System.out.println(" 10000-100000 -> standard-payments");
}
}
Expected output:
Branch routing:
> 100000 INR -> premium-payments
< 10000 INR -> micro-payments
10000-100000 -> standard-payments
Example routing:
SAV2001|NEFT|250000|transfer -> premium-payments
SAV2002|IMPS|5000|recharge -> micro-payments
CUR3001|NEFT|50000|vendor pay -> standard-payments
How does merge work?
merge() combines two or more KStream instances into a single KStream. The merged stream contains all records from all input streams in the order they arrive. Records are not deduplicated or sorted.
At National Bank of India, payments arrive through three channels: NEFT, IMPS, and UPI. Rajesh merges all three into a single stream for unified processing.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
public class MergeExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> neft = builder.stream("neft-payments");
KStream<String, String> imps = builder.stream("imps-payments");
KStream<String, String> upi = builder.stream("upi-payments");
// Merge all payment channels into one stream
KStream<String, String> allPayments = neft
.merge(imps)
.merge(upi);
// Process all payments uniformly
allPayments.to("all-payments");
Topology topology = builder.build();
System.out.println("Merge: 3 payment channels -> 1 unified stream");
System.out.println(" neft-payments ─┐");
System.out.println(" imps-payments ─┼─> all-payments");
System.out.println(" upi-payments ─┘");
}
}
Expected output: Merge: 3 payment channels -> 1 unified stream neft-payments ─┐ imps-payments ─┼─> all-payments upi-payments ─┘ Records in merged stream (interleaved by arrival order): SAV2001|NEFT|250000|transfer SAV2002|UPI|500|grocery CUR3001|IMPS|15000|bill payment
How does peek work?
peek() performs a side-effect action on each record without modifying it. The record passes through unchanged. Common uses are logging, metrics collection, and audit trail writing.
peek() should not modify the record. If you need to transform the record, use map or mapValues instead. In production, Rajesh uses peek to log every transaction for debugging without affecting the processing pipeline.
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;
public class PeekExample {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> payments = builder.stream("payments");
KStream<String, String> processed = payments
.peek((key, value) ->
System.out.println("AUDIT: account=" + key + ", payment=" + value))
.mapValues(value -> {
String[] parts = value.split("\\|");
return parts[0] + "|" + parts[1] + "|" + parts[2] + "|PROCESSED";
})
.peek((key, value) ->
System.out.println("POST-PROCESS: account=" + key + ", result=" + value));
processed.to("processed-payments");
Topology topology = builder.build();
System.out.println("Peek: audit logging at two points in the topology");
}
}
Expected output:
Peek: audit logging at two points in the topology
When SAV2001|NEFT|250000|transfer arrives:
AUDIT: account=SAV2001, payment=SAV2001|NEFT|250000|transfer
POST-PROCESS: account=SAV2001, result=SAV2001|NEFT|250000|PROCESSED
The record passes through peek unchanged.
How do all stateless operations compare?
This table summarizes every stateless operation, what it does, whether it changes the key, and whether it triggers repartitioning.
| Operation | What it does | Changes key? | Triggers repartition? | Banking example |
|---|---|---|---|---|
| filter | Keeps records matching predicate | No | No | Keep payments > 100000 |
| filterNot | Keeps records NOT matching predicate | No | No | Remove blocked accounts |
| map | Transforms key and value | Yes | Yes | Rekey by branch code |
| mapValues | Transforms value only | No | No | Add processing fee |
| flatMap | One record to zero or many, new keys | Yes | Yes | Split bulk payment |
| flatMapValues | One record to zero or many, same key | No | No | Expand line items |
| selectKey | Changes key only | Yes | Yes | Rekey by currency |
| branch / split | Route to named sub-streams | No | No | Route by amount tier |
| merge | Combine streams into one | No | No | Merge NEFT + IMPS + UPI |
| peek | Side effect, no modification | No | No | Audit logging |
| foreach | Terminal side effect | No | No | Send SMS notification |
package com.bank.streams;
public class StatelessComparisonSummary {
public static void main(String[] args) {
System.out.println("Operations that trigger repartitioning:");
System.out.println(" map, flatMap, selectKey");
System.out.println(" Reason: they change the record key");
System.out.println();
System.out.println("Operations that do NOT trigger repartitioning:");
System.out.println(" filter, filterNot, mapValues, flatMapValues,");
System.out.println(" branch, merge, peek, foreach");
System.out.println(" Reason: they keep the original key");
System.out.println();
System.out.println("Rule: prefer mapValues over map, flatMapValues over flatMap");
System.out.println("when you do not need to change the key.");
}
}
Expected output:
Operations that trigger repartitioning:
map, flatMap, selectKey
Reason: they change the record key
Operations that do NOT trigger repartitioning:
filter, filterNot, mapValues, flatMapValues,
branch, merge, peek, foreach
Reason: they keep the original key
Rule: prefer mapValues over map, flatMapValues over flatMap
when you do not need to change the key.
What is a complete banking project using stateless operations?
This project is a Payment Routing Service for National Bank of India. It reads payments from a single topic and applies a chain of stateless operations: peek for audit, filter for fraud, mapValues for fee calculation, flatMap for multi-beneficiary splitting, and branch for tier-based routing.
Project structure
payment-routing-service/
├── pom.xml
└── src/
└── main/
└── java/
└── com/
└── bank/
└── streams/
├── PaymentRoutingApp.java
├── PaymentRoutingTopology.java
├── Payment.java
└── PaymentRoutingTopologyTest.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>payment-routing-servicesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
Payment.java
package com.bank.streams;
import java.math.BigDecimal;
public class Payment {
private String accountId;
private String type;
private BigDecimal amount;
private String description;
public Payment() {}
public Payment(String accountId, String type,
BigDecimal amount, String description) {
this.accountId = accountId;
this.type = type;
this.amount = amount;
this.description = description;
}
public String getAccountId() { return accountId; }
public String getType() { return type; }
public BigDecimal getAmount() { return amount; }
public String getDescription() { return description; }
@Override
public String toString() {
return accountId + "|" + type + "|" + amount + "|" + description;
}
public static Payment fromString(String value) {
String[] parts = value.split("\\|");
return new Payment(
parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
}
}
PaymentRoutingTopology.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.Branched;
import org.apache.kafka.streams.Topology;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.util.Map;
public class PaymentRoutingTopology {
private static final BigDecimal FRAUD_THRESHOLD = new BigDecimal("1000000");
private static final BigDecimal PREMIUM_THRESHOLD = new BigDecimal("100000");
private static final BigDecimal FEE_RATE = new BigDecimal("0.005");
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> payments = builder.stream("incoming-payments");
// Step 1: peek for audit logging
KStream<String, String> audited = payments
.peek((key, value) ->
System.out.println("AUDIT-IN: " + key + " -> " + value));
// Step 2: filter out suspected fraud (amount >= 10 lakh)
KStream<String, String> legitimate = audited
.filter((key, value) -> {
Payment p = Payment.fromString(value);
boolean isFraud = p.getAmount().compareTo(FRAUD_THRESHOLD) >= 0;
if (isFraud) {
System.out.println("FRAUD-BLOCK: " + key);
}
return !isFraud;
});
// Step 3: mapValues to add processing fee
KStream<String, String> withFee = legitimate.mapValues(value -> {
Payment p = Payment.fromString(value);
BigDecimal fee = p.getAmount().multiply(FEE_RATE)
.setScale(2, RoundingMode.HALF_UP);
return value + "|FEE:" + fee;
});
// Step 4: branch by amount tier
Map<String, KStream<String, String>> tiers = withFee
.split()
.branch((key, value) -> {
BigDecimal amount = new BigDecimal(value.split("\\|")[2]);
return amount.compareTo(PREMIUM_THRESHOLD) > 0;
}, Branched.as("premium"))
.defaultBranch(Branched.as("standard"));
tiers.get("premium").to("premium-payments");
tiers.get("standard").to("standard-payments");
return builder.build();
}
}
PaymentRoutingApp.java
package com.bank.streams;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class PaymentRoutingApp {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
"payment-routing-service");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
KafkaStreams streams = new KafkaStreams(
PaymentRoutingTopology.build(), props);
CountDownLatch latch = new CountDownLatch(1);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close();
latch.countDown();
}));
try {
streams.start();
System.out.println("Payment routing service started");
System.out.println(" Input: incoming-payments");
System.out.println(" Output: premium-payments, standard-payments");
System.out.println(" Fraud: blocked if >= 1000000 INR");
latch.await();
} catch (Exception e) {
System.exit(1);
}
}
}
Expected output (on startup):
Payment routing service started
Input: incoming-payments
Output: premium-payments, standard-payments
Fraud: blocked if >= 1000000 INR
PaymentRoutingTopologyTest.java
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Properties;
public class PaymentRoutingTopologyTest {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
try (TopologyTestDriver driver =
new TopologyTestDriver(PaymentRoutingTopology.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"incoming-payments", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> premium = driver.createOutputTopic(
"premium-payments", new StringDeserializer(), new StringDeserializer());
TestOutputTopic<String, String> standard = driver.createOutputTopic(
"standard-payments", new StringDeserializer(), new StringDeserializer());
// Premium payment (250000 > 100000)
input.pipeInput("SAV2001", "SAV2001|NEFT|250000|transfer to Priya");
// Standard payment (50000 <= 100000)
input.pipeInput("SAV2002", "SAV2002|IMPS|50000|bill payment");
// Fraud (1500000 >= 1000000, blocked)
input.pipeInput("CUR3001", "CUR3001|RTGS|1500000|suspicious transfer");
System.out.println("=== Premium payments ===");
System.out.println("Count: " + premium.getQueueSize());
System.out.println("Record: " + premium.readValue());
System.out.println();
System.out.println("=== Standard payments ===");
System.out.println("Count: " + standard.getQueueSize());
System.out.println("Record: " + standard.readValue());
System.out.println();
System.out.println("Fraud payment was blocked by filter. Not in any output.");
}
}
}
Expected output:
AUDIT-IN: SAV2001 -> SAV2001|NEFT|250000|transfer to Priya
AUDIT-IN: SAV2002 -> SAV2002|IMPS|50000|bill payment
AUDIT-IN: CUR3001 -> CUR3001|RTGS|1500000|suspicious transfer
FRAUD-BLOCK: CUR3001
=== Premium payments ===
Count: 1
Record: SAV2001|NEFT|250000|transfer to Priya|FEE:1250.00
=== Standard payments ===
Count: 1
Record: SAV2002|IMPS|50000|bill payment|FEE:250.00
Fraud payment was blocked by filter. Not in any output.
How to run the project
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper
# Step 2: Create topics
kafka-topics.sh --create --topic incoming-payments --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic premium-payments --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic standard-payments --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Step 3: Build and run
cd payment-routing-service
mvn clean package
java -cp target/payment-routing-service-1.0.0.jar com.bank.streams.PaymentRoutingApp
# Step 4: Produce test messages
kafka-console-producer.sh --topic incoming-payments --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# SAV2001:SAV2001|NEFT|250000|transfer to Priya
# SAV2002:SAV2002|IMPS|50000|bill payment
# CUR3001:CUR3001|RTGS|1500000|suspicious transfer
What are common mistakes with stateless operations?
The most common mistake is using map() when mapValues() is sufficient. This triggers an unnecessary repartition.
Wrong: using map when only the value changes
package com.bank.streams;
public class WrongMapForValueOnly {
public static void main(String[] args) {
System.out.println("WRONG: Using map() to change only the value");
System.out.println();
System.out.println("Code:");
System.out.println(" payments.map((key, value) ->");
System.out.println(" KeyValue.pair(key, value.toUpperCase()));");
System.out.println();
System.out.println("Problem: map() marks the stream for repartitioning");
System.out.println("even though the key did not change.");
System.out.println("An internal repartition topic is created unnecessarily.");
System.out.println("This doubles network traffic for downstream key-based operations.");
}
}
Expected output:
WRONG: Using map() to change only the value
Code:
payments.map((key, value) ->
KeyValue.pair(key, value.toUpperCase()));
Problem: map() marks the stream for repartitioning
even though the key did not change.
An internal repartition topic is created unnecessarily.
This doubles network traffic for downstream key-based operations.
Correct: using mapValues when only the value changes
package com.bank.streams;
public class CorrectMapValuesForValueOnly {
public static void main(String[] args) {
System.out.println("CORRECT: Using mapValues() to change only the value");
System.out.println();
System.out.println("Code:");
System.out.println(" payments.mapValues(value -> value.toUpperCase());");
System.out.println();
System.out.println("mapValues() does not change the key.");
System.out.println("No repartition topic is created.");
System.out.println("Same result, less network overhead.");
}
}
Expected output:
CORRECT: Using mapValues() to change only the value
Code:
payments.mapValues(value -> value.toUpperCase());
mapValues() does not change the key.
No repartition topic is created.
Same result, less network overhead.
Wrong: using peek to modify the record
package com.bank.streams;
public class WrongPeekModification {
public static void main(String[] args) {
System.out.println("WRONG: Modifying state inside peek()");
System.out.println();
System.out.println("Code:");
System.out.println(" List<String> auditList = new ArrayList<>();");
System.out.println(" payments.peek((key, value) -> auditList.add(value));");
System.out.println();
System.out.println("Problem: peek runs on stream threads.");
System.out.println("Mutating shared state (auditList) is not thread-safe.");
System.out.println("This causes ConcurrentModificationException or data corruption.");
System.out.println();
System.out.println("CORRECT: Use peek only for stateless side effects like logging.");
System.out.println(" payments.peek((key, value) ->");
System.out.println(" logger.info(\"Payment: {} -> {}\", key, value));");
}
}
Expected output:
WRONG: Modifying state inside peek()
Code:
List<String> auditList = new ArrayList<>();
payments.peek((key, value) -> auditList.add(value));
Problem: peek runs on stream threads.
Mutating shared state (auditList) is not thread-safe.
This causes ConcurrentModificationException or data corruption.
CORRECT: Use peek only for stateless side effects like logging.
payments.peek((key, value) ->
logger.info("Payment: {} -> {}", key, value));
What should a senior or principal engineer know?
At the senior level, you should understand the performance implications of stateless operations and how to build efficient topologies.
- Repartition cost: Every map(), flatMap(), or selectKey() that changes the key creates an internal repartition topic. This adds network hops, storage, and latency. In a topology with multiple key changes, each one creates a separate repartition topic.
- Operation ordering: Place filter() early in the topology to reduce the number of records processed by downstream operations. If 80% of records are filtered out, the fee calculation and routing steps process only 20% of the traffic.
- Null key handling: If selectKey() returns null, the record is dropped silently during any downstream groupByKey() or join. Always validate keys before re-keying.
- flatMap cardinality: flatMap() can produce millions of output records from one input record. In banking, a bulk payment with 10,000 beneficiaries produces 10,000 output records. Monitor output rate and backpressure.
- Topology testing: Use TopologyTestDriver to test each stateless operation in isolation. Write separate tests for filter logic, fee calculation, and routing predicates.
- Idempotency: Stateless operations are naturally idempotent because they have no state. If Kafka Streams reprocesses a record after a failure, the operation produces the same result.
package com.bank.streams;
public class SeniorStatelessKnowledge {
public static void main(String[] args) {
System.out.println("Senior checklist for stateless operations:");
System.out.println(" 1. Place filter early to reduce downstream volume");
System.out.println(" 2. Prefer mapValues over map to avoid repartition");
System.out.println(" 3. Prefer flatMapValues over flatMap for same reason");
System.out.println(" 4. Monitor flatMap output cardinality in production");
System.out.println(" 5. Validate keys in selectKey to avoid null drops");
System.out.println(" 6. Test each operation independently with TopologyTestDriver");
System.out.println(" 7. Stateless ops are idempotent: safe to reprocess");
}
}
Expected output:
Senior checklist for stateless operations:
1. Place filter early to reduce downstream volume
2. Prefer mapValues over map to avoid repartition
3. Prefer flatMapValues over flatMap for same reason
4. Monitor flatMap output cardinality in production
5. Validate keys in selectKey to avoid null drops
6. Test each operation independently with TopologyTestDriver
7. Stateless ops are idempotent: safe to reprocess
What are common interview follow-up questions?
- What is the difference between foreach and peek? (foreach is a terminal operation that ends the stream; peek is an intermediate operation that passes the record through. You cannot chain more operations after foreach.)
- Does filter create a new topic? (No. filter is an in-memory operation within the topology. No topic is created. The filtered records continue in the same sub-topology.)
- Can you call map on a KTable? (Yes. KTable has mapValues() but not map(). If you need to change the key, convert to KStream first with toStream(), then call map().)
- What happens if flatMap returns an empty list? (The input record is dropped. Zero output records are produced. This is a valid way to conditionally suppress records.)
- Does branch guarantee ordering? (Each branch preserves the ordering of the original stream for records that match that branch. Records in different branches are independent.)
- Can you merge a KStream with a KTable? (No. merge() works only between KStream instances. To combine a KStream with a KTable, use a join operation instead.)
Visual Overview
Payment Routing Topology (Stateless Operations)
==================================================
┌───────────────────────────┐
│ incoming-payments │
└────────────┬──────────────┘
│
▼
┌───────────────────────────┐
│ peek (audit log) │ ← logs each payment
└────────────┬──────────────┘
│
▼
┌───────────────────────────┐
│ filter (fraud check) │ ← blocks >= 10 lakh
└────────────┬──────────────┘
│ legitimate only
▼
┌───────────────────────────┐
│ mapValues (add fee 0.5%) │ ← no repartition
└────────────┬──────────────┘
│
▼
┌───────────────────────────┐
│ branch (amount tier) │
└──────┬──────────┬─────────┘
│ │
▼ ▼
┌──────────┐ ┌──────────────┐
│ premium │ │ standard │
│ payments │ │ payments │
│ > 1 lakh │ │ <= 1 lakh │
└──────────┘ └──────────────┘
Repartition Decision Tree
===========================
Need to change the key?
├── No → use mapValues, flatMapValues, filter, peek
│ (no repartition, no extra topic)
└── Yes → use map, flatMap, selectKey
(triggers repartition topic)
└── Is repartition worth it?
├── Yes: downstream join/aggregation needs new key
└── No: reconsider your design
Key Takeaway
Stateless operations are the building blocks of every Kafka Streams topology. They process each record independently, with no local state store and no changelog topic. This makes them lightweight, fast, and naturally idempotent.
The most commonly used stateless operations are filter, mapValues, and peek. filter removes records that do not meet a condition, like blocking suspected fraud payments above ten lakh rupees. mapValues transforms the value without changing the key, like adding a 0.5% processing fee to each payment. peek logs or inspects each record without modifying it, which is useful for audit trails at National Bank of India.
The operations that change the record key, including map, flatMap, and selectKey, trigger repartitioning. This means Kafka Streams creates an internal topic, writes the rekeyed records to it, and reads them back. This adds network overhead and latency. The rule is to prefer mapValues over map and flatMapValues over flatMap whenever you do not need to change the key.
flatMap is the most powerful stateless operation. It takes one record and produces zero or more output records, each with its own key and value. In banking, this is how you split a bulk payment instruction with three beneficiaries into three individual transfer records. branch and merge handle the routing side: branch splits a stream into named sub-streams by predicate (premium vs standard tiers), and merge combines multiple channel streams (NEFT, IMPS, UPI) into one.
At the senior level, the key insight is topology ordering. Place filter early to reduce downstream volume. Avoid unnecessary repartitioning by using value-only variants. Monitor flatMap output cardinality to avoid unexpected volume spikes. Test each operation independently with TopologyTestDriver.
Remember one sentence: stateless operations transform, filter, split, merge, and observe records one at a time with no memory of what came before, and the critical design choice is whether you change the key, because that triggers repartitioning.
---