Back to Kafka Streams
Q8Question 8 of 27

What is KStream in Kafka Streams

Answer: KStream is a continuously flowing, unbounded sequence of key-value records from a Kafka topic where every record is an independent event, not an update to a previous value.

Key Concepts
  • KStream is an abstraction for an unbounded, continuously arriving sequence of key-value records from a Kafka topic.
  • Every record in a KStream is treated as an independent event: a new record with the same key does not replace the previous one.
  • KStream is created by calling builder.stream("topic-name") and supports stateless operations like filter, map, mapValues, flatMap, peek, and branch.
  • KStream also supports stateful operations through groupByKey().aggregate(), groupByKey().count(), and groupByKey().reduce(), which produce a KTable.
  • Records flow through a KStream one at a time in the order they arrive within each partition.
  • KStream is analogous to an INSERT-only log: every record is appended, nothing is updated or deleted.
  • You terminate a KStream processing path by calling .to("output-topic") to write results or .foreach() to perform a side effect.

Definition

KStream is a continuously flowing, unbounded sequence of key-value records from a Kafka topic where every record is an independent event, not an update to a previous value.
  • KStream represents an event stream: each record is a fact that happened (a transaction occurred, a payment was made, a login was recorded), and duplicate keys mean multiple events, not updates.
  • Unbounded means the stream has no end: records keep arriving as long as producers publish to the topic.
  • Record independence means each record is processed on its own; a new record with key SAV2001 does not overwrite the previous SAV2001 record.
  • Stateless operations on KStream (filter, map, mapValues, flatMap, peek, branch, merge) process each record without needing memory of prior records.
  • Stateful transition: calling groupByKey() on a KStream produces a KGroupedStream, and then calling aggregate(), reduce(), or count() produces a KTable with accumulated state.

Think of the transaction log at National Bank of India. Every time Rajesh deposits money, withdraws money, or transfers money, a new line is written in the log. The log never erases old entries. If Rajesh deposits 50,000 rupees at 10:00 AM and another 30,000 rupees at 11:00 AM, both lines exist. That is a KStream: a continuous log of events. The bank does not overwrite the 10:00 AM entry with the 11:00 AM entry. Each deposit is a separate fact. If Priya at the counter wants to know Rajesh's current balance, she must add up all the events. That summation is a stateful operation that produces a KTable.

Understanding the Concepts

What is KStream?

KStream is the core abstraction in Kafka Streams for representing a stream of records. Each record is a key-value pair. Records arrive continuously from a Kafka topic, one at a time, in partition order.

The key concept is that KStream treats every record as an independent event. If you receive three records with key SAV2001, KStream sees three separate events. It does not replace or update earlier records.

You create a KStream by calling builder.stream("topic-name").

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;

public class KStreamBasics {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        // Create a KStream from the bank-transactions topic
        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Each record is an independent event
        // Key: account ID (e.g., SAV2001)
        // Value: transaction detail (e.g., DEPOSIT|50000|Mumbai)
        transactions.peek((key, value) ->
            System.out.println("Event: key=" + key + ", value=" + value));

        transactions.to("transactions-log");

        Topology topology = builder.build();
        System.out.println("KStream topology:");
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
KStream topology:
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
      --> KSTREAM-PEEK-0000000001
    Processor: KSTREAM-PEEK-0000000001 (stores: [])
      --> KSTREAM-SINK-0000000002
      <-- KSTREAM-SOURCE-0000000000
    Sink: KSTREAM-SINK-0000000002 (topic: transactions-log)
      <-- KSTREAM-PEEK-0000000001

How does KStream handle multiple records with the same key?

KStream does not merge, update, or deduplicate records. Every record is processed independently regardless of its key. This is the fundamental difference between KStream and KTable.

If three deposit events arrive for account SAV2001, the KStream processes all three as separate events.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.util.Properties;

public class KStreamSameKeyDemo {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();
        builder.<String, String>stream("bank-transactions")
            .mapValues(value -> "PROCESSED: " + value)
            .to("processed-transactions");

        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(builder.build(), props)) {

            TestInputTopic<String, String> input = driver.createInputTopic(
                "bank-transactions",
                new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> output = driver.createOutputTopic(
                "processed-transactions",
                new StringDeserializer(), new StringDeserializer());

            // Three events with the SAME key: SAV2001
            input.pipeInput("SAV2001", "DEPOSIT|50000|Mumbai|10:00");
            input.pipeInput("SAV2001", "DEPOSIT|30000|Mumbai|11:00");
            input.pipeInput("SAV2001", "WITHDRAWAL|10000|Delhi|12:00");

            // KStream processes ALL THREE, not just the latest
            System.out.println("Records in output: " + output.getQueueSize());
            System.out.println("Record 1: " + output.readValue());
            System.out.println("Record 2: " + output.readValue());
            System.out.println("Record 3: " + output.readValue());
            System.out.println();
            System.out.println("KStream behavior: all 3 records processed");
            System.out.println("  No replacement, no dedup, no merge");
            System.out.println("  Each record is an independent event");
        }
    }
}
TEXT
Expected output:
Records in output: 3
Record 1: PROCESSED: DEPOSIT|50000|Mumbai|10:00
Record 2: PROCESSED: DEPOSIT|30000|Mumbai|11:00
Record 3: PROCESSED: WITHDRAWAL|10000|Delhi|12:00

KStream behavior: all 3 records processed
  No replacement, no dedup, no merge
  Each record is an independent event

What stateless operations does KStream support?

Stateless operations process each record independently without remembering any previous records. KStream provides these stateless operations:

OperationWhat it doesChanges key?Repartition?
filter(predicate)Keep records matching conditionNoNo
filterNot(predicate)Remove records matching conditionNoNo
map(mapper)Transform both key and valueYesYes (if key changes)
mapValues(mapper)Transform value onlyNoNo
flatMap(mapper)One record produces zero or more records (key+value)YesYes
flatMapValues(mapper)One value produces zero or more valuesNoNo
selectKey(mapper)Change the key onlyYesYes
peek(action)Observe each record (for logging), pass through unchangedNoNo
branch(predicates)Split into multiple streams by conditionNoNo
merge(other)Combine two streams into oneNoNo
JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;

import java.math.BigDecimal;

public class KStreamStatelessOps {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();
        KStream<String, String> txns = builder.stream("bank-transactions");

        // filter: keep only deposits
        KStream<String, String> deposits = txns
            .filter((key, value) -> value.startsWith("DEPOSIT"));

        // mapValues: add a processing tag
        KStream<String, String> tagged = deposits
            .mapValues(value -> "VERIFIED|" + value);

        // peek: log each record for monitoring
        KStream<String, String> logged = tagged
            .peek((key, value) ->
                System.out.println("Verified deposit: " + key + " -> " + value));

        // flatMapValues: split multi-item transactions into individual records
        KStream<String, String> expanded = txns
            .flatMapValues(value -> {
                String[] items = value.split(";");
                return java.util.Arrays.asList(items);
            });

        logged.to("verified-deposits");
        expanded.to("expanded-transactions");

        System.out.println("Stateless operations used:");
        System.out.println("  filter()        -> keep deposits only");
        System.out.println("  mapValues()     -> add VERIFIED tag");
        System.out.println("  peek()          -> log for monitoring");
        System.out.println("  flatMapValues() -> split multi-item records");
    }
}
TEXT
Expected output:
Stateless operations used:
  filter()        -> keep deposits only
  mapValues()     -> add VERIFIED tag
  peek()          -> log for monitoring
  flatMapValues() -> split multi-item records

How does KStream transition to stateful processing?

KStream itself is stateless. To perform stateful operations like counting, summing, or aggregating, you first group records by key using groupByKey() or groupBy(). This produces a KGroupedStream. Then you call count(), reduce(), or aggregate() to produce a KTable.

The KTable holds the latest aggregated result per key. As new records arrive on the KStream, the KTable updates.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KGroupedStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.state.KeyValueStore;
import org.apache.kafka.streams.state.ValueAndTimestamp;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class KStreamToStateful {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Step 1: groupByKey produces KGroupedStream
        KGroupedStream<String, String> grouped = transactions.groupByKey();

        // Step 2: count() produces KTable<String, Long>
        KTable<String, Long> txnCounts = grouped.count(
            Materialized.as("transaction-counts"));

        // The KTable holds the latest count per account
        txnCounts.toStream().peek((key, count) ->
            System.out.println("Account " + key + " has " + count + " transactions"))
            .to("transaction-count-output");

        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(builder.build(), props)) {

            TestInputTopic<String, String> input = driver.createInputTopic(
                "bank-transactions",
                new StringSerializer(), new StringSerializer());

            // SAV2001: 3 transactions
            input.pipeInput("SAV2001", "DEPOSIT|50000");
            input.pipeInput("SAV2001", "DEPOSIT|30000");
            input.pipeInput("SAV2001", "WITHDRAWAL|10000");

            // SAV2002: 1 transaction
            input.pipeInput("SAV2002", "DEPOSIT|25000");

            // Read state store directly
            KeyValueStore<String, ValueAndTimestamp<Long>> store =
                driver.getTimestampedKeyValueStore("transaction-counts");

            System.out.println();
            System.out.println("State store contents:");
            System.out.println("  SAV2001: " + store.get("SAV2001").value() + " transactions");
            System.out.println("  SAV2002: " + store.get("SAV2002").value() + " transaction");
            System.out.println();
            System.out.println("KStream (events) -> groupByKey -> count -> KTable (state)");
        }
    }
}
TEXT
Expected output:
Account SAV2001 has 1 transactions
Account SAV2001 has 2 transactions
Account SAV2001 has 3 transactions
Account SAV2002 has 1 transactions

State store contents:
  SAV2001: 3 transactions
  SAV2002: 1 transaction

KStream (events) -> groupByKey -> count -> KTable (state)

Notice that the KTable emits an updated count after every input record. The state store holds the final accumulated value.

What is a complete banking project using KStream?

This project builds a payment notification service for National Bank of India. It reads payment events from a KStream, classifies them by type (UPI, NEFT, RTGS), enriches each with a processing fee, and routes notifications to type-specific topics.

Project structure

DIAGRAM
payment-notification-service/
├── pom.xml
└── src/
    ├── main/
    │   └── java/
    │       └── com/
    │           └── bank/
    │               └── streams/
    │                   ├── PaymentNotificationApp.java
    │                   ├── PaymentNotificationTopology.java
    │                   └── PaymentEvent.java
    └── test/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        └── PaymentNotificationTest.java

PaymentEvent.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class PaymentEvent {
    private String paymentId;
    private String fromAccount;
    private String toAccount;
    private BigDecimal amount;
    private String type;  // UPI, NEFT, RTGS
    private String timestamp;

    public PaymentEvent() {}

    public PaymentEvent(String paymentId, String fromAccount, String toAccount,
                        BigDecimal amount, String type, String timestamp) {
        this.paymentId = paymentId;
        this.fromAccount = fromAccount;
        this.toAccount = toAccount;
        this.amount = amount;
        this.type = type;
        this.timestamp = timestamp;
    }

    public String getPaymentId() { return paymentId; }
    public String getFromAccount() { return fromAccount; }
    public String getToAccount() { return toAccount; }
    public BigDecimal getAmount() { return amount; }
    public String getType() { return type; }
    public String getTimestamp() { return timestamp; }

    @Override
    public String toString() {
        return paymentId + "|" + fromAccount + "|" + toAccount + "|"
               + amount + "|" + type + "|" + timestamp;
    }

    public static PaymentEvent fromString(String value) {
        String[] p = value.split("\\|");
        return new PaymentEvent(p[0], p[1], p[2],
            new BigDecimal(p[3]), p[4], p[5]);
    }
}

PaymentNotificationTopology.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;

import java.math.BigDecimal;
import java.math.RoundingMode;

public class PaymentNotificationTopology {

    // Fee schedule: UPI free, NEFT 2.50, RTGS 25.00
    private static final BigDecimal NEFT_FEE = new BigDecimal("2.50");
    private static final BigDecimal RTGS_FEE = new BigDecimal("25.00");

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> payments = builder.stream("payment-events");

        // Enrich with processing fee
        KStream<String, String> enriched = payments.mapValues(value -> {
            PaymentEvent evt = PaymentEvent.fromString(value);
            BigDecimal fee;
            switch (evt.getType()) {
                case "RTGS": fee = RTGS_FEE; break;
                case "NEFT": fee = NEFT_FEE; break;
                default:     fee = BigDecimal.ZERO; break;
            }
            BigDecimal total = evt.getAmount().add(fee)
                .setScale(2, RoundingMode.HALF_UP);
            return evt.toString() + "|FEE:" + fee + "|TOTAL:" + total;
        });

        // Route by payment type to separate notification topics
        KStream<String, String> upi = enriched
            .filter((key, value) -> value.contains("|UPI|"));
        KStream<String, String> neft = enriched
            .filter((key, value) -> value.contains("|NEFT|"));
        KStream<String, String> rtgs = enriched
            .filter((key, value) -> value.contains("|RTGS|"));

        upi.to("notifications-upi");
        neft.to("notifications-neft");
        rtgs.to("notifications-rtgs");

        return builder.build();
    }
}

PaymentNotificationApp.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class PaymentNotificationApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG,
                  "payment-notification-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        Topology topology = PaymentNotificationTopology.build();
        KafkaStreams streams = new KafkaStreams(topology, props);
        CountDownLatch latch = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            latch.countDown();
            System.out.println("Payment notification service stopped");
        }));

        try {
            streams.start();
            System.out.println("Payment notification service started");
            System.out.println("  Input:  payment-events");
            System.out.println("  Output: notifications-upi, notifications-neft, notifications-rtgs");
            latch.await();
        } catch (Exception e) {
            System.exit(1);
        }
    }
}

PaymentNotificationTest.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.util.Properties;

public class PaymentNotificationTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(PaymentNotificationTopology.build(), props)) {

            TestInputTopic<String, String> input = driver.createInputTopic(
                "payment-events",
                new StringSerializer(), new StringSerializer());

            TestOutputTopic<String, String> upiOut = driver.createOutputTopic(
                "notifications-upi",
                new StringDeserializer(), new StringDeserializer());
            TestOutputTopic<String, String> neftOut = driver.createOutputTopic(
                "notifications-neft",
                new StringDeserializer(), new StringDeserializer());
            TestOutputTopic<String, String> rtgsOut = driver.createOutputTopic(
                "notifications-rtgs",
                new StringDeserializer(), new StringDeserializer());

            // UPI payment: Rajesh pays Priya 500 rupees, no fee
            input.pipeInput("PAY001",
                "PAY001|SAV2001|SAV2002|500|UPI|2026-11-13T10:00:00");

            // NEFT payment: 75,000 rupees, fee 2.50
            input.pipeInput("PAY002",
                "PAY002|SAV2001|CUR3001|75000|NEFT|2026-11-13T10:15:00");

            // RTGS payment: 5,00,000 rupees, fee 25.00
            input.pipeInput("PAY003",
                "PAY003|CUR3001|SAV2003|500000|RTGS|2026-11-13T10:30:00");

            System.out.println("UPI notifications: " + upiOut.getQueueSize());
            System.out.println("  " + upiOut.readValue());
            System.out.println();
            System.out.println("NEFT notifications: " + neftOut.getQueueSize());
            System.out.println("  " + neftOut.readValue());
            System.out.println();
            System.out.println("RTGS notifications: " + rtgsOut.getQueueSize());
            System.out.println("  " + rtgsOut.readValue());
            System.out.println();
            System.out.println("Test passed: 3 payments routed to correct type topics with fees");
        }
    }
}
TEXT
Expected output:
UPI notifications: 1
  PAY001|SAV2001|SAV2002|500|UPI|2026-11-13T10:00:00|FEE:0|TOTAL:500.00

NEFT notifications: 1
  PAY002|SAV2001|CUR3001|75000|NEFT|2026-11-13T10:15:00|FEE:2.50|TOTAL:75002.50

RTGS notifications: 1
  PAY003|CUR3001|SAV2003|500000|RTGS|2026-11-13T10:30:00|FEE:25.00|TOTAL:500025.00

Test passed: 3 payments routed to correct type topics with fees

How to run

BASH
# Step 1: Create topics
kafka-topics.sh --create --topic payment-events --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic notifications-upi --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic notifications-neft --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic notifications-rtgs --partitions 4 --replication-factor 1 --bootstrap-server localhost:9092

# Step 2: Build
cd payment-notification-service
mvn clean package

# Step 3: Run test (no broker needed)
java -cp target/payment-notification-service-1.0.0.jar com.bank.streams.PaymentNotificationTest

# Step 4: Run app
java -cp target/payment-notification-service-1.0.0.jar com.bank.streams.PaymentNotificationApp

# Step 5: Produce
kafka-console-producer.sh --topic payment-events --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# PAY001:PAY001|SAV2001|SAV2002|500|UPI|2026-11-13T10:00:00
# PAY002:PAY002|SAV2001|CUR3001|75000|NEFT|2026-11-13T10:15:00
# PAY003:PAY003|CUR3001|SAV2003|500000|RTGS|2026-11-13T10:30:00

# Step 6: Consume each output topic
kafka-console-consumer.sh --topic notifications-upi --from-beginning --bootstrap-server localhost:9092
kafka-console-consumer.sh --topic notifications-neft --from-beginning --bootstrap-server localhost:9092
kafka-console-consumer.sh --topic notifications-rtgs --from-beginning --bootstrap-server localhost:9092

What are common mistakes when working with KStream?

Wrong: treating KStream as a table (expecting latest-value semantics)

JAVA
package com.bank.streams;

public class WrongKStreamAsTable {
    public static void main(String[] args) {
        System.out.println("WRONG: Expecting KStream to keep only the latest value per key");
        System.out.println();
        System.out.println("  Input (3 records, same key SAV2001):");
        System.out.println("    SAV2001 -> DEPOSIT|50000");
        System.out.println("    SAV2001 -> DEPOSIT|30000");
        System.out.println("    SAV2001 -> WITHDRAWAL|10000");
        System.out.println();
        System.out.println("  Developer expects: only WITHDRAWAL|10000 in output");
        System.out.println("  Actual result:     ALL THREE records in output");
        System.out.println();
        System.out.println("  KStream is an event log, not a key-value store");
        System.out.println("  Every record is processed independently");
        System.out.println();
        System.out.println("  Fix: if you need latest-value-per-key, use KTable");
        System.out.println("    builder.table('topic') instead of builder.stream('topic')");
    }
}
TEXT
Expected output:
WRONG: Expecting KStream to keep only the latest value per key

  Input (3 records, same key SAV2001):
    SAV2001 -> DEPOSIT|50000
    SAV2001 -> DEPOSIT|30000
    SAV2001 -> WITHDRAWAL|10000

  Developer expects: only WITHDRAWAL|10000 in output
  Actual result:     ALL THREE records in output

  KStream is an event log, not a key-value store
  Every record is processed independently

  Fix: if you need latest-value-per-key, use KTable
    builder.table('topic') instead of builder.stream('topic')

Correct: use KStream for events, KTable for state

JAVA
package com.bank.streams;

public class CorrectKStreamVsKTable {
    public static void main(String[] args) {
        System.out.println("CORRECT: Choose the right abstraction");
        System.out.println();
        System.out.println("  Use KStream (builder.stream) when:");
        System.out.println("    - Every event matters: transactions, payments, logins");
        System.out.println("    - You process each record independently");
        System.out.println("    - Duplicate keys mean multiple events");
        System.out.println();
        System.out.println("  Use KTable (builder.table) when:");
        System.out.println("    - You need the latest value per key: balance, status, profile");
        System.out.println("    - A new record with the same key replaces the old one");
        System.out.println("    - You need lookup/reference data for joins");
    }
}
TEXT
Expected output:
CORRECT: Choose the right abstraction

  Use KStream (builder.stream) when:
    - Every event matters: transactions, payments, logins
    - You process each record independently
    - Duplicate keys mean multiple events

  Use KTable (builder.table) when:
    - You need the latest value per key: balance, status, profile
    - A new record with the same key replaces the old one
    - You need lookup/reference data for joins

What should a senior or principal engineer know?

At the senior level, you should understand that KStream is append-only and every record is independent. You should know when to use KStream versus KTable, and how groupByKey transitions a KStream into stateful processing.

At the principal level, consider:

  • Event time vs processing time: KStream records carry a timestamp. By default, Kafka Streams uses the record's event time (when the producer created it). You can override this with a custom TimestampExtractor.
  • Out-of-order records: within a partition, records are ordered. Across partitions, order is not guaranteed. Design your KStream processing to handle late or out-of-order events, especially for windowed operations.
  • Repartitioning cost: operations that change the key (map, selectKey, flatMap) cause repartitioning. Kafka Streams writes records to an internal repartition topic and reads them back, doubling the I/O. Use mapValues or flatMapValues when you do not need to change the key.
  • Serialization: KStream uses Serdes (serializer-deserializer pairs) for keys and values. Mismatched serdes cause deserialization exceptions at runtime. Set default serdes globally or override per operation.
  • Backpressure: KStream does not have built-in backpressure. If the downstream sink topic is slow, records accumulate in the processing buffer. Monitor records-lag-max and process-rate metrics.
JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsConfig;
import java.util.Properties;

public class KStreamPrincipalKnowledge {
    public static void main(String[] args) {
        System.out.println("Principal-level KStream knowledge:");
        System.out.println();
        System.out.println("1. Event time vs processing time:");
        System.out.println("   Default: event time from producer timestamp");
        System.out.println("   Custom: implement TimestampExtractor to parse from value");
        System.out.println();
        System.out.println("2. Repartitioning cost:");
        System.out.println("   map() / selectKey() / flatMap() -> repartition topic -> 2x I/O");
        System.out.println("   mapValues() / flatMapValues()   -> no repartition -> fast");
        System.out.println();
        System.out.println("3. Serialization:");
        System.out.println("   Mismatch = SerializationException at runtime");
        System.out.println("   Always set default.key.serde and default.value.serde");
        System.out.println();
        System.out.println("4. Monitoring KStream throughput:");
        System.out.println("   records-lag-max: how far behind the consumer is");
        System.out.println("   process-rate:   records processed per second");
        System.out.println("   process-latency: time per record in the topology");
    }
}
TEXT
Expected output:
Principal-level KStream knowledge:

1. Event time vs processing time:
   Default: event time from producer timestamp
   Custom: implement TimestampExtractor to parse from value

2. Repartitioning cost:
   map() / selectKey() / flatMap() -> repartition topic -> 2x I/O
   mapValues() / flatMapValues()   -> no repartition -> fast

3. Serialization:
   Mismatch = SerializationException at runtime
   Always set default.key.serde and default.value.serde

4. Monitoring KStream throughput:
   records-lag-max: how far behind the consumer is
   process-rate:   records processed per second
   process-latency: time per record in the topology

What are common interview follow-up questions?

  • Is KStream bounded or unbounded? (Unbounded. Records keep arriving as long as producers publish to the topic. There is no end-of-stream marker.)
  • Can KStream handle deletes? (Not directly. A null value is a tombstone and is typically filtered out by KStream. KTable handles tombstones by removing the key from the state store.)
  • What is the difference between KStream and Java Streams? (KStream is for distributed, real-time processing of Kafka records. Java Streams is for in-memory processing of collections. They share similar method names like filter and map, but are completely different abstractions.)
  • Can you read from multiple topics into one KStream? (Yes. Pass a list or a regex pattern: builder.stream(Arrays.asList("topic1", "topic2")) or builder.stream(Pattern.compile("bank-.*")).)
  • What happens if the producer changes the value format? (Deserialization fails. Use a DeserializationExceptionHandler to log and skip bad records instead of crashing the stream thread.)
  • Can a KStream write to multiple output topics? (Yes. Branch the stream and call .to() on each branch. Or use .to(TopicNameExtractor) to dynamically choose the topic per record.)

Visual Overview

DIAGRAM
 KStream: Unbounded Event Stream
 =================================

 Producer publishes continuously:

   10:00  SAV2001 -> DEPOSIT|50000
   10:01  SAV2002 -> DEPOSIT|25000
   10:02  SAV2001 -> DEPOSIT|30000
   10:03  SAV2001 -> WITHDRAWAL|10000
   10:04  CUR3001 -> NEFT|75000
     ...       (never ends)

 KStream sees EVERY record:

   ┌──────────────────────────┐
   │       KStream              │
   │                            │
   │  SAV2001 DEPOSIT  50000    │  event 1
   │  SAV2002 DEPOSIT  25000    │  event 2
   │  SAV2001 DEPOSIT  30000    │  event 3
   │  SAV2001 WITHDRAWAL 10000  │  event 4
   │  CUR3001 NEFT     75000    │  event 5
   │  ...                       │  unbounded
   └──────────────────────────┘

   3 records for SAV2001 = 3 separate events
   (not 1 updated record)


 KStream Stateless vs Stateful
 ==============================

 KStream ──> filter/map/peek ──> KStream   (stateless)

 KStream ──> groupByKey ──> KGroupedStream
                 │
                 ├──> count()     ──> KTable
                 ├──> reduce()    ──> KTable
                 └──> aggregate() ──> KTable

Key Takeaway

KStream is the fundamental abstraction in Kafka Streams for representing a continuous, unbounded flow of events from a Kafka topic. Every record is treated as an independent event. Three deposits for account SAV2001 mean three separate events, not one updated balance. This is the core difference from KTable, which keeps only the latest value per key.

Stateless operations on KStream like filter, mapValues, flatMapValues, and peek process each record independently with no memory of previous records. When you need to aggregate, count, or reduce, you call groupByKey() to get a KGroupedStream, then count(), reduce(), or aggregate() to produce a KTable with accumulated state.

In our National Bank of India payment notification project, we read payment events as a KStream, enriched each with a processing fee based on type (UPI free, NEFT 2.50, RTGS 25.00), and routed notifications to type-specific topics. The unit test verified all three payment types with correct fees and totals. A junior developer can clone the project, run the test without a broker, and see the exact output.

At the principal level, remember that repartitioning happens when you change the key, doubling I/O. Use mapValues and flatMapValues when you do not need to change the key. Monitor records-lag-max and process-rate to detect throughput problems early.

Remember one sentence: KStream is an unbounded log of independent events where every record is processed as a new fact, not an update to a previous value.

---

Share
Use ← → arrow keys to navigate