Back to Kafka Streams
Q3Question 3 of 27

What is Kafka Streams, and how does it differ from the Consumer API

Answer: Kafka Streams is a lightweight Java library that reads from Kafka topics, processes records through a topology of operations, and writes results back to Kafka, while the Consumer API only reads messages and leaves all processing logic to you.

Key Concepts
  • Kafka Streams is a Java client library for real-time stream processing that runs inside your application, not a separate cluster.
  • The Consumer API reads messages and you handle all processing, offsets, error recovery, and state management yourself.
  • Kafka Streams gives you built-in operations like filter, map, groupBy, join, aggregate, and windowing out of the box.
  • Kafka Streams manages consumer offsets, state stores, fault tolerance, and exactly-once processing automatically.
  • The Consumer API is pull-based polling; Kafka Streams builds a topology (source, processor, sink) that processes continuously.
  • Kafka Streams can read from one topic, process, and write to another topic in a single atomic pipeline.
  • Use the Consumer API for simple consume-and-act patterns; use Kafka Streams when you need transformations, joins, aggregations, or windowing.

Definition

Kafka Streams is a lightweight Java library that reads from Kafka topics, processes records through a topology of operations, and writes results back to Kafka, while the Consumer API only reads messages and leaves all processing logic to you.
  • Kafka Streams is a client library bundled with Apache Kafka for building stream-processing applications; it runs inside your JVM process, not on a separate cluster like Flink or Spark.
  • Consumer API is the low-level Kafka client that polls records from topics; you write your own processing loop, manage offsets, handle errors, and coordinate state yourself.
  • Topology is the processing plan in Kafka Streams: a directed graph of source nodes (read), processor nodes (transform), and sink nodes (write).
  • State store is the local storage Kafka Streams uses for stateful operations like aggregations; it is backed by a changelog topic for fault tolerance.
  • Exactly-once processing is built into Kafka Streams via Kafka transactions; with the Consumer API you must implement idempotency yourself.

Think of National Bank of India processing daily transactions. The Consumer API is like a clerk who picks up one paper slip at a time from a pile, reads it, and manually decides what to do: calculate the running balance, check for fraud, file the result. The clerk tracks which slips have been read and handles errors alone. Kafka Streams is like an automated conveyor belt system in the same bank: slips enter at one end, pass through a series of stations (filter suspicious amounts, calculate running balance, route to the right department), and exit at the other end. The conveyor handles slip tracking, station recovery, and parallel lanes automatically.

Understanding the Concepts

What is Kafka Streams?

Kafka Streams is a Java library provided by Apache Kafka for processing data from Kafka topics in real time. You add it as a Maven or Gradle dependency, write your processing logic, and run it as a normal Java application. There is no separate cluster to deploy or manage.

The core idea is a topology: you define a pipeline of operations (read from a topic, filter, transform, aggregate, write to another topic), and Kafka Streams executes that pipeline continuously.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;

public class WhatIsKafkaStreams {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        // Read transactions from a Kafka topic
        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Filter only high-value transactions
        KStream<String, String> highValue = transactions
            .filter((accountId, txn) -> {
                // In a real app, parse the value and check amount
                System.out.println("Filtering transaction for account: " + accountId);
                return txn.contains("HIGH");
            });

        // Write filtered transactions to another topic
        highValue.to("high-value-alerts");

        Topology topology = builder.build();
        System.out.println("Topology description:");
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
Topology description:
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [bank-transactions])
      --> KSTREAM-FILTER-0000000001
    Processor: KSTREAM-FILTER-0000000001 (stores: [])
      --> KSTREAM-SINK-0000000002
      <-- KSTREAM-SOURCE-0000000000
    Sink: KSTREAM-SINK-0000000002 (topic: high-value-alerts)
      <-- KSTREAM-FILTER-0000000001

What is the Kafka Consumer API?

The Consumer API is the low-level client that reads records from Kafka topics. You create a KafkaConsumer, subscribe to topics, call poll() in a loop, and process each record yourself. You manage offsets, error handling, and any state.

JAVA
package com.bank.consumer;

import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class TransactionConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "bank-transaction-processor");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("bank-transactions"));

        System.out.println("Consumer started, polling bank-transactions...");

        try {
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
                for (ConsumerRecord<String, String> record : records) {
                    // All processing logic is manual
                    String accountId = record.key();
                    String txn = record.value();
                    if (txn.contains("HIGH")) {
                        System.out.println("HIGH value alert: account=" + accountId + ", txn=" + txn);
                        // Manually write to another topic or call a service
                    }
                }
                consumer.commitSync();
            }
        } finally {
            consumer.close();
            System.out.println("Consumer closed");
        }
    }
}
TEXT
Expected output:
Consumer started, polling bank-transactions...
HIGH value alert: account=SAV2001, txn=DEBIT 500000 HIGH
Consumer closed

How does Kafka Streams differ from the Consumer API?

The difference is in what the library handles for you versus what you handle yourself.

FeatureKafka Consumer APIKafka Streams
Processing modelManual poll loopDeclarative topology
Offset managementYou call commitSync/commitAsyncAutomatic
State managementYou build and maintain stateBuilt-in state stores (RocksDB)
Fault toleranceYou implement retry and recoveryAutomatic via changelog topics
Exactly-onceYou implement idempotencyBuilt-in (processing.guarantee)
TransformationsYou write filter/map/join logicfilter(), map(), join(), aggregate()
WindowingYou implement time bucketingBuilt-in tumbling, hopping, session windows
Writing outputYou create a separate KafkaProducer.to() writes directly in the topology
ScalingYou manage consumer group rebalancingAutomatic partition assignment
TestingYou mock the consumerTopologyTestDriver (no broker needed)
JAVA
package com.bank.streams;

public class ConsumerVsStreamsComparison {
    public static void main(String[] args) {
        System.out.println("=== Consumer API ===");
        System.out.println("1. Create KafkaConsumer with properties");
        System.out.println("2. Subscribe to topic");
        System.out.println("3. Poll in a loop");
        System.out.println("4. Process each record manually");
        System.out.println("5. Commit offsets manually");
        System.out.println("6. Create KafkaProducer to write output");
        System.out.println("7. Handle errors, retries, state yourself");
        System.out.println();
        System.out.println("=== Kafka Streams ===");
        System.out.println("1. Create StreamsBuilder");
        System.out.println("2. Define topology: stream -> filter -> map -> to");
        System.out.println("3. Start KafkaStreams instance");
        System.out.println("4. Offsets, state, fault tolerance are automatic");
        System.out.println("5. Output writing is part of the topology");
        System.out.println();
        System.out.println("Consumer API: 7 manual steps");
        System.out.println("Kafka Streams: 3 steps, rest is automatic");
    }
}
TEXT
Expected output:
=== Consumer API ===
1. Create KafkaConsumer with properties
2. Subscribe to topic
3. Poll in a loop
4. Process each record manually
5. Commit offsets manually
6. Create KafkaProducer to write output
7. Handle errors, retries, state yourself

=== Kafka Streams ===
1. Create StreamsBuilder
2. Define topology: stream -> filter -> map -> to
3. Start KafkaStreams instance
4. Offsets, state, fault tolerance are automatic
5. Output writing is part of the topology

Consumer API: 7 manual steps
Kafka Streams: 3 steps, rest is automatic

When should you use the Consumer API instead of Kafka Streams?

The Consumer API is the right choice when you only need to read messages and call an external service, without writing results back to Kafka. For example, reading a payment notification and sending an email does not need a topology.

Another case is when your processing requires calling a database or REST API for every record. Kafka Streams is designed for Kafka-in, Kafka-out pipelines. If you need to call an external system in the middle, the Consumer API gives you more control over error handling and retries.

JAVA
package com.bank.consumer;

public class WhenToUseConsumerApi {
    public static void main(String[] args) {
        System.out.println("Use Consumer API when:");
        System.out.println("  1. You read from Kafka and call an external service (email, SMS, REST API)");
        System.out.println("  2. You need fine-grained control over offset commits");
        System.out.println("  3. Your processing is simple: read, act, done");
        System.out.println("  4. You do not write results back to Kafka");
        System.out.println();
        System.out.println("Use Kafka Streams when:");
        System.out.println("  1. You read from Kafka, process, and write back to Kafka");
        System.out.println("  2. You need filter, map, join, aggregate, or windowing");
        System.out.println("  3. You need stateful processing with automatic fault tolerance");
        System.out.println("  4. You need exactly-once guarantees in the pipeline");
    }
}
TEXT
Expected output:
Use Consumer API when:
  1. You read from Kafka and call an external service (email, SMS, REST API)
  2. You need fine-grained control over offset commits
  3. Your processing is simple: read, act, done
  4. You do not write results back to Kafka

Use Kafka Streams when:
  1. You read from Kafka, process, and write back to Kafka
  2. You need filter, map, join, aggregate, or windowing
  3. You need stateful processing with automatic fault tolerance
  4. You need exactly-once guarantees in the pipeline

What is a complete banking project using Kafka Streams?

This project reads payment transactions from a Kafka topic, filters transactions above 100,000 rupees, normalizes the description to uppercase, and writes the results to an alerts topic. A junior developer can run this project from start to finish.

Project structure

DIAGRAM
payment-alert-service/
├── pom.xml
└── src/
    └── main/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        ├── PaymentAlertApp.java          (main class)
                        ├── PaymentAlertTopology.java     (topology definition)
                        └── PaymentTransaction.java       (model)

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>

    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>payment-alert-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
    syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>

    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
        syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
        syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
        syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
    syn-kw"></propertiessyn-kw">>

    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

PaymentTransaction.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class PaymentTransaction {
    private String accountId;
    private String description;
    private BigDecimal amount;
    private String status;

    public PaymentTransaction() {}

    public PaymentTransaction(String accountId, String description,
                              BigDecimal amount, String status) {
        this.accountId = accountId;
        this.description = description;
        this.amount = amount;
        this.status = status;
    }

    public String getAccountId() { return accountId; }
    public void setAccountId(String accountId) { this.accountId = accountId; }
    public String getDescription() { return description; }
    public void setDescription(String description) { this.description = description; }
    public BigDecimal getAmount() { return amount; }
    public void setAmount(BigDecimal amount) { this.amount = amount; }
    public String getStatus() { return status; }
    public void setStatus(String status) { this.status = status; }

    @Override
    public String toString() {
        return accountId + "|" + description + "|" + amount + "|" + status;
    }

    public static PaymentTransaction fromString(String value) {
        String[] parts = value.split("\\|");
        return new PaymentTransaction(
            parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
    }
}

PaymentAlertTopology.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;

import java.math.BigDecimal;

public class PaymentAlertTopology {

    private static final BigDecimal ALERT_THRESHOLD = new BigDecimal("100000");

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        // Source: read raw payment events
        KStream<String, String> payments = builder.stream("payments");

        // Step 1: Parse the payment string into fields
        // Step 2: Filter transactions above 1,00,000 rupees
        // Step 3: Normalize description to uppercase
        // Step 4: Write to alerts topic
        payments
            .filter((key, value) -> {
                PaymentTransaction txn = PaymentTransaction.fromString(value);
                return txn.getAmount().compareTo(ALERT_THRESHOLD) > 0;
            })
            .mapValues(value -> {
                PaymentTransaction txn = PaymentTransaction.fromString(value);
                txn.setDescription(txn.getDescription().toUpperCase());
                return txn.toString();
            })
            .to("high-value-alerts");

        return builder.build();
    }
}

PaymentAlertApp.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class PaymentAlertApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "payment-alert-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        KafkaStreams streams = new KafkaStreams(
            PaymentAlertTopology.build(), props);
        CountDownLatch latch = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            latch.countDown();
            System.out.println("Payment alert service shut down");
        }));

        try {
            streams.start();
            System.out.println("Payment alert service started");
            System.out.println("  Input topic:  payments");
            System.out.println("  Output topic: high-value-alerts");
            System.out.println("  Threshold:    100000 INR");
            latch.await();
        } catch (Exception e) {
            System.exit(1);
        }
    }
}
TEXT
Expected output (on startup):
Payment alert service started
  Input topic:  payments
  Output topic: high-value-alerts
  Threshold:    100000 INR

How to run the project

BASH
# Step 1: Start Kafka (using Docker Compose or local install)
docker-compose up -d kafka zookeeper

# Step 2: Create the input and output topics
kafka-topics.sh --create --topic payments --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic high-value-alerts --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

# Step 3: Build the project
cd payment-alert-service
mvn clean package

# Step 4: Run the Kafka Streams app
java -cp target/payment-alert-service-1.0.0.jar com.bank.streams.PaymentAlertApp

# Step 5: In a new terminal, produce test messages
kafka-console-producer.sh --topic payments --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# Type these messages (key:value format):
# SAV2001:SAV2001|neft transfer to Priya|250000|SUCCESS
# SAV2002:SAV2002|atm withdrawal|5000|SUCCESS
# CUR3001:CUR3001|rtgs payment to vendor|500000|SUCCESS

# Step 6: In another terminal, consume the output topic
kafka-console-consumer.sh --topic high-value-alerts --from-beginning --bootstrap-server localhost:9092
TEXT
Expected output on the high-value-alerts consumer:
SAV2001|NEFT TRANSFER TO PRIYA|250000|SUCCESS
CUR3001|RTGS PAYMENT TO VENDOR|500000|SUCCESS

Notice that SAV2002 (5,000 rupees) did not appear because it is below the 100,000 threshold. Only the two high-value payments passed through, and their descriptions are now uppercase.

What are common mistakes when choosing between Consumer API and Kafka Streams?

The most common mistake is using the Consumer API for complex processing that Kafka Streams handles better.

Wrong: manual aggregation with Consumer API

JAVA
package com.bank.consumer;

import java.math.BigDecimal;
import java.util.HashMap;
import java.util.Map;

public class WrongManualAggregation {
    public static void main(String[] args) {
        // Simulating what developers do wrong: manual state in a HashMap
        Map<String, BigDecimal> balances = new HashMap<>();

        // Problem 1: This state is lost if the app restarts
        balances.merge("SAV2001", new BigDecimal("50000"), BigDecimal::add);
        balances.merge("SAV2001", new BigDecimal("30000"), BigDecimal::add);
        balances.merge("SAV2002", new BigDecimal("10000"), BigDecimal::add);

        System.out.println("Manual aggregation (WRONG approach):");
        balances.forEach((account, balance) ->
            System.out.println("  " + account + ": " + balance));
        System.out.println();
        System.out.println("Problems with this approach:");
        System.out.println("  1. State is in-memory only, lost on restart");
        System.out.println("  2. No fault tolerance, no changelog backup");
        System.out.println("  3. No exactly-once guarantee");
        System.out.println("  4. Scaling requires manual partition handling");
    }
}
TEXT
Expected output:
Manual aggregation (WRONG approach):
  SAV2001: 80000
  SAV2002: 10000

Problems with this approach:
  1. State is in-memory only, lost on restart
  2. No fault tolerance, no changelog backup
  3. No exactly-once guarantee
  4. Scaling requires manual partition handling

Correct: use Kafka Streams aggregate with state store

JAVA
package com.bank.streams;

public class CorrectStreamsAggregation {
    public static void main(String[] args) {
        System.out.println("Kafka Streams aggregation (CORRECT approach):");
        System.out.println("  KStream -> groupByKey() -> aggregate()");
        System.out.println();
        System.out.println("What Kafka Streams handles automatically:");
        System.out.println("  1. State stored in RocksDB, survives restarts");
        System.out.println("  2. Changelog topic backs up state for recovery");
        System.out.println("  3. exactly_once_v2 makes read-process-write atomic");
        System.out.println("  4. Scaling: just start more instances, partitions rebalance");
    }
}
TEXT
Expected output:
Kafka Streams aggregation (CORRECT approach):
  KStream -> groupByKey() -> aggregate()

What Kafka Streams handles automatically:
  1. State stored in RocksDB, survives restarts
  2. Changelog topic backs up state for recovery
  3. exactly_once_v2 makes read-process-write atomic
  4. Scaling: just start more instances, partitions rebalance

What about testing the topology without a Kafka broker?

Kafka Streams provides TopologyTestDriver for unit testing. You create the topology, feed test records into a TestInputTopic, and read results from a TestOutputTopic. No running broker is needed.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.util.Properties;

public class PaymentAlertTopologyTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(PaymentAlertTopology.build(), props)) {

            TestInputTopic<String, String> input = driver.createInputTopic(
                "payments", new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> output = driver.createOutputTopic(
                "high-value-alerts", new StringDeserializer(), new StringDeserializer());

            // Test 1: High value transaction (should pass through)
            input.pipeInput("SAV2001", "SAV2001|neft transfer to Priya|250000|SUCCESS");

            // Test 2: Low value transaction (should be filtered out)
            input.pipeInput("SAV2002", "SAV2002|atm withdrawal|5000|SUCCESS");

            // Test 3: Another high value transaction
            input.pipeInput("CUR3001", "CUR3001|rtgs payment|500000|SUCCESS");

            System.out.println("Records in output topic: " + output.getQueueSize());
            System.out.println("Record 1: " + output.readValue());
            System.out.println("Record 2: " + output.readValue());
            System.out.println("Queue empty: " + output.isEmpty());
            System.out.println();
            System.out.println("Test passed: 2 high-value, 0 low-value in output");
        }
    }
}
TEXT
Expected output:
Records in output topic: 2
Record 1: SAV2001|NEFT TRANSFER TO PRIYA|250000|SUCCESS
Record 2: CUR3001|RTGS PAYMENT|500000|SUCCESS
Queue empty: true

Test passed: 2 high-value, 0 low-value in output

What should a senior or principal engineer know?

At the senior level, you should understand when Kafka Streams is the right tool and when it is not. Kafka Streams is designed for Kafka-to-Kafka pipelines. If your output is a database, REST call, or email, the Consumer API with a service layer is often simpler.

At the principal level, you need to think about:

  • Topology isolation: each Kafka Streams application should own its topology. Do not put unrelated processing into one app.
  • Partition count determines parallelism: the maximum number of stream threads doing useful work equals the number of input partitions. Plan partition count at topic creation time.
  • State store sizing: stateful operations create local RocksDB stores. On restart, state is rebuilt from changelog topics, which takes time proportional to state size.
  • Exactly-once overhead: exactly_once_v2 uses Kafka transactions, which reduce throughput by 10-30%. Measure before enabling.
  • Upgrade strategy: Kafka Streams apps are just JVM processes, so rolling restarts work. But topology changes (adding a new processor) require careful planning because stream thread rebalancing happens.
  • Monitoring: expose JMX metrics for consumer lag, state store size, and stream thread state. Alert on REBALANCING state lasting more than 60 seconds.
JAVA
package com.bank.streams;

public class SeniorKnowledge {
    public static void main(String[] args) {
        System.out.println("Senior/Principal Kafka Streams checklist:");
        System.out.println("  1. Partition count = max parallelism");
        System.out.println("  2. State stores use RocksDB, rebuilt from changelog on restart");
        System.out.println("  3. exactly_once_v2 reduces throughput 10-30%");
        System.out.println("  4. One topology per app, do not mix unrelated logic");
        System.out.println("  5. Monitor consumer lag, stream-thread-state, state-store-size");
        System.out.println("  6. Rolling restart for upgrades, plan for rebalancing time");
        System.out.println("  7. Use TopologyTestDriver for unit tests, EmbeddedKafka for integration");
    }
}
TEXT
Expected output:
Senior/Principal Kafka Streams checklist:
  1. Partition count = max parallelism
  2. State stores use RocksDB, rebuilt from changelog on restart
  3. exactly_once_v2 reduces throughput 10-30%
  4. One topology per app, do not mix unrelated logic
  5. Monitor consumer lag, stream-thread-state, state-store-size
  6. Rolling restart for upgrades, plan for rebalancing time
  7. Use TopologyTestDriver for unit tests, EmbeddedKafka for integration

What are common interview follow-up questions?

  • Can Kafka Streams call a REST API or database during processing? (Yes, but it blocks the stream thread and is discouraged. Use the Consumer API for external calls.)
  • What happens if a Kafka Streams app crashes mid-processing? (With exactly_once_v2, uncommitted work is rolled back. On restart, processing resumes from the last committed offset.)
  • Can you run Kafka Streams in a Spring Boot application? (Yes. Spring Kafka provides auto-configuration for Kafka Streams with @EnableKafkaStreams.)
  • How many stream threads should you configure? (One thread processes one or more tasks, where each task corresponds to a partition. Default is one thread. Set num.stream.threads to match CPU cores, but never more than input partitions.)
  • Is Kafka Streams the same as KSQL? (No. KSQL/ksqlDB is a SQL layer on top of Kafka Streams. Kafka Streams is the Java library underneath.)

Visual Overview

DIAGRAM
 Kafka Consumer API Flow
 ========================

 ┌──────────────┐
 │ Kafka Topic   │
 │ (payments)    │
 └──────┬───────┘
        │ poll()
        ▼
 ┌──────────────────────┐
 │ KafkaConsumer        │
 │  for each record:    │
 │   parse              │
 │   filter manually    │
 │   process manually   │
 │   produce manually   │
 │  commitSync()        │
 └──────────────────────┘


 Kafka Streams Flow
 ===================

 ┌──────────────┐
 │ Source Topic  │
 │ (payments)    │
 └──────┬───────┘
        │
        ▼
 ┌──────────────────────┐
 │ Filter Node          │
 │ amount > 100,000     │
 └──────┬───────────────┘
        │
        ▼
 ┌──────────────────────┐
 │ MapValues Node       │
 │ uppercase description│
 └──────┬───────────────┘
        │
        ▼
 ┌──────────────────────┐
 │ Sink Topic           │
 │ (high-value-alerts)  │
 └──────────────────────┘

 Offsets, state, and recovery are automatic.

Key Takeaway

Kafka Streams is a Java library that runs inside your application and processes data from Kafka topics in real time. Unlike the Consumer API, where you write a polling loop and manage offsets, state, and error handling yourself, Kafka Streams lets you define a topology of operations and handles the rest.

The Consumer API is the right choice when you need to read a message and call an external service like a REST API, send an email, or write to a database. Kafka Streams is the right choice when your pipeline is Kafka-in, Kafka-out: read from a topic, filter, transform, aggregate, and write to another topic.

In our National Bank of India payment-alert-service project, we read payment transactions, filtered amounts above one lakh rupees, normalized descriptions to uppercase, and wrote alerts to a separate topic. The entire pipeline was three Kafka Streams operations: filter, mapValues, and to. We tested it with TopologyTestDriver without starting a broker.

At the senior level, remember that partition count determines parallelism, state stores are backed by changelog topics, and exactly-once processing uses Kafka transactions with a measurable throughput cost. At the principal level, design one topology per application, monitor consumer lag and stream-thread state, and plan for rebalancing time during rolling upgrades.

Remember one sentence: Kafka Streams is a library, not a server, that turns a topology of filter, map, join, and aggregate into a fault-tolerant, scalable, exactly-once pipeline running inside your own JVM.

---

Share
Use ← → arrow keys to navigate