Back to Kafka Streams
Q27Question 27 of 27

How does Kafka Streams compare to Flink and Spark Streaming

Answer: Kafka Streams is a lightweight library embedded in your JVM for Kafka-to-Kafka processing, Flink is a distributed cluster framework for complex event processing with advanced event-time semantics, and Spark Structured Streaming is a micro-batch engine for combined batch and stream workloads.

Key Concepts
  • Kafka Streams is a Java library that runs inside your application process, while Flink and Spark require a separate cluster.
  • Flink provides true event-time processing with watermarks and is designed for complex event processing at scale.
  • Spark Structured Streaming uses micro-batching, processing data in small batches rather than record-by-record.
  • Kafka Streams reads only from Kafka topics, while Flink and Spark can read from Kafka, HDFS, S3, databases, and other sources.
  • Kafka Streams is the simplest to deploy: just run a JAR file with no cluster manager needed.
  • Flink offers the most advanced windowing, event-time handling, and exactly-once across external sinks.
  • Choose Kafka Streams for Kafka-centric pipelines, Flink for complex event processing, and Spark for combined batch and stream workloads.

Definition

Kafka Streams is a lightweight library embedded in your JVM for Kafka-to-Kafka processing, Flink is a distributed cluster framework for complex event processing with advanced event-time semantics, and Spark Structured Streaming is a micro-batch engine for combined batch and stream workloads.
  • Kafka Streams is a client library that runs inside your application as a regular Java process; it reads from Kafka, processes, and writes back to Kafka without a separate cluster.
  • Apache Flink is a distributed stream processing framework that runs on its own cluster (JobManager + TaskManagers) with advanced event-time processing, watermarks, and exactly-once guarantees across external sinks.
  • Spark Structured Streaming is Apache Spark's stream processing engine that uses micro-batching to process streams as a series of small batch jobs; it shares Spark's batch engine, SQL support, and ML libraries.
  • Micro-batching means collecting records for a short interval (e.g., 100ms) and processing them as a batch, which adds latency compared to record-at-a-time processing.
  • Watermark is Flink's mechanism for tracking event-time progress and handling late events; it defines how far behind event-time the system expects records to be.

Think of National Bank of India processing transactions across three different setups. Kafka Streams is like a clerk sitting at a desk with a paper tray: transactions come in, the clerk processes them one by one, and puts results in an outgoing tray. No special infrastructure needed, just a desk. Flink is like a factory floor with conveyor belts, specialized stations, and a foreman (JobManager): it handles complex assembly lines, branching, and merging at high volume. Spark Streaming is like a mail room that opens a bag of letters every minute, sorts and processes each bag, then sends results: efficient for volume but each bag adds a small delay.

Understanding the Concepts

What is the architecture difference between the three frameworks?

The fundamental difference is deployment model. Kafka Streams is a library, not a framework. Flink and Spark are distributed systems that require cluster management.

FeatureKafka StreamsApache FlinkSpark Streaming
DeploymentLibrary (runs in your JVM)Distributed cluster (JobManager + TaskManagers)Distributed cluster (Driver + Executors)
Processing modelRecord-at-a-time, true streamingRecord-at-a-time, true streamingMicro-batch (Structured Streaming)
State managementRocksDB local store + changelog topicRocksDB + distributed snapshots (Savepoints)In-memory or external (HDFS checkpoints)
Exactly-onceKafka transactions (processing.guarantee)Distributed snapshots (Chandy-Lamport)WAL + idempotent sinks
ScalingBy Kafka partition countBy task parallelism (independent of Kafka)By executor count
LatencyMillisecondsMillisecondsSeconds (micro-batch interval)
DependencyKafka onlyStandalone, works with any sourceStandalone, works with any source
SQL supportksqlDB (separate component)Flink SQL (built-in)Spark SQL (built-in)
Cluster managementNone needed (library)YARN, Kubernetes, standaloneYARN, Kubernetes, standalone
  • Kafka Streams: add a Maven dependency, write your topology, run your JAR. No cluster, no YARN, no Kubernetes operator needed (though you can use Kubernetes for scaling). The library runs inside your JVM.
  • Flink: deploy a Flink cluster (JobManager + TaskManagers) on YARN, Kubernetes, or standalone. Submit your job to the cluster. The cluster manages parallelism, checkpointing, and failover.
  • Spark: deploy a Spark cluster (Driver + Executors) on YARN, Kubernetes, or standalone. Submit your streaming job. The driver schedules micro-batches on executors.
JAVA
package com.bank.streams;

public class ArchitectureComparison {
    public static void main(String[] args) {
        System.out.println("Architecture comparison:");
        System.out.println();
        System.out.println("Kafka Streams:");
        System.out.println("  Type: Library (JAR dependency)");
        System.out.println("  Deploy: java -jar your-app.jar");
        System.out.println("  Cluster: None required");
        System.out.println("  Scaling: Start more JVM instances");
        System.out.println();
        System.out.println("Apache Flink:");
        System.out.println("  Type: Distributed framework");
        System.out.println("  Deploy: flink run your-job.jar");
        System.out.println("  Cluster: JobManager + TaskManagers");
        System.out.println("  Scaling: Add TaskManagers, adjust parallelism");
        System.out.println();
        System.out.println("Spark Streaming:");
        System.out.println("  Type: Distributed framework");
        System.out.println("  Deploy: spark-submit your-job.jar");
        System.out.println("  Cluster: Driver + Executors");
        System.out.println("  Scaling: Add executors, adjust partitions");
    }
}
TEXT
Expected output:
Architecture comparison:

Kafka Streams:
  Type: Library (JAR dependency)
  Deploy: java -jar your-app.jar
  Cluster: None required
  Scaling: Start more JVM instances

Apache Flink:
  Type: Distributed framework
  Deploy: flink run your-job.jar
  Cluster: JobManager + TaskManagers
  Scaling: Add TaskManagers, adjust parallelism

Spark Streaming:
  Type: Distributed framework
  Deploy: spark-submit your-job.jar
  Cluster: Driver + Executors
  Scaling: Add executors, adjust partitions

How do the three frameworks compare on key features?

FeatureKafka StreamsApache FlinkSpark Structured Streaming
DeploymentLibrary (JAR)Cluster (JobManager)Cluster (Driver)
Data sourceKafka onlyKafka, HDFS, S3, DB, filesKafka, HDFS, S3, DB, files
Processing modelRecord-at-a-timeRecord-at-a-timeMicro-batch
LatencyLow (ms)Very low (ms)Higher (100ms+ per batch)
Exactly-onceKafka-to-Kafka onlyEnd-to-end (with sinks)End-to-end (with sinks)
WindowingTumbling, hopping, sliding, sessionAll + custom windowsTumbling, sliding
Event-time handlingStream timeWatermarks (advanced)Watermarks (basic)
State managementRocksDB + changelogRocksDB + checkpointsIn-memory + checkpoints
SQL supportNo (use ksqlDB separately)Flink SQL (built-in)Spark SQL (built-in)
Batch processingNoYes (unified batch + stream)Yes (unified batch + stream)
Language supportJava, ScalaJava, Scala, Python, SQLJava, Scala, Python, R, SQL
Cluster managementNot neededYARN, K8s, standaloneYARN, K8s, standalone
Interactive queriesBuilt-in (state store)Not built-inNot built-in
Learning curveLowMedium-HighMedium
JAVA
package com.bank.streams;

public class FeatureComparisonDemo {
    public static void main(String[] args) {
        System.out.println("Key differentiators:");
        System.out.println();
        System.out.println("1. Deployment complexity:");
        System.out.println("   Kafka Streams < Spark < Flink");
        System.out.println();
        System.out.println("2. Latency (best case):");
        System.out.println("   Flink < Kafka Streams < Spark");
        System.out.println();
        System.out.println("3. Feature richness:");
        System.out.println("   Flink > Spark > Kafka Streams");
        System.out.println();
        System.out.println("4. Data source variety:");
        System.out.println("   Spark = Flink > Kafka Streams (Kafka only)");
        System.out.println();
        System.out.println("5. Batch + stream unified:");
        System.out.println("   Flink = Spark (yes), Kafka Streams (no)");
    }
}
TEXT
Expected output:
Key differentiators:

1. Deployment complexity:
   Kafka Streams < Spark < Flink

2. Latency (best case):
   Flink < Kafka Streams < Spark

3. Feature richness:
   Flink > Spark > Kafka Streams

4. Data source variety:
   Spark = Flink > Kafka Streams (Kafka only)

5. Batch + stream unified:
   Flink = Spark (yes), Kafka Streams (no)

When should you choose Kafka Streams?

Kafka Streams is the right choice when your pipeline is Kafka-centric: read from Kafka, process, write to Kafka. It is the simplest option for teams that already use Kafka and do not want to manage a separate processing cluster.

  • Good use cases for Kafka Streams at a bank:
  • Real-time transaction filtering and routing.
  • Maintaining account balances in a state store.
  • Fraud detection based on transaction counts per window.
  • Enriching payment events with account data from a KTable.
JAVA
package com.bank.streams;

public class WhenToChooseKafkaStreams {
    public static void main(String[] args) {
        System.out.println("Choose Kafka Streams when:");
        System.out.println("  1. Input and output are Kafka topics");
        System.out.println("  2. You want zero cluster management overhead");
        System.out.println("  3. Processing is filter, map, join, aggregate, window");
        System.out.println("  4. Team already knows Kafka and Java");
        System.out.println("  5. You need interactive queries (state store REST API)");
        System.out.println("  6. Sub-second latency is needed but not sub-millisecond");
        System.out.println();
        System.out.println("Banking examples:");
        System.out.println("  Transaction routing: NEFT/RTGS/IMPS per channel");
        System.out.println("  Balance maintenance: deposit/withdrawal aggregation");
        System.out.println("  Fraud alerts: count ATM withdrawals per 5-min window");
    }
}
TEXT
Expected output:
Choose Kafka Streams when:
  1. Input and output are Kafka topics
  2. You want zero cluster management overhead
  3. Processing is filter, map, join, aggregate, window
  4. Team already knows Kafka and Java
  5. You need interactive queries (state store REST API)
  6. Sub-second latency is needed but not sub-millisecond

Banking examples:
  Transaction routing: NEFT/RTGS/IMPS per channel
  Balance maintenance: deposit/withdrawal aggregation
  Fraud alerts: count ATM withdrawals per 5-min window

Flink is the right choice when you need advanced event-time processing, complex event patterns, or exactly-once writes to external systems (databases, file systems). Flink's watermark mechanism is more sophisticated than Kafka Streams' stream time.

  • Good use cases for Flink at a bank:
  • Complex event processing: detect patterns like "3 failed logins followed by a large withdrawal within 10 minutes."
  • Multi-source pipelines: join Kafka streams with database change data capture (CDC) and file ingestion.
  • Exactly-once writes to a database (using Flink's two-phase commit sinks).
JAVA
package com.bank.streams;

public class WhenToChooseFlink {
    public static void main(String[] args) {
        System.out.println("Choose Flink when:");
        System.out.println("  1. Complex event processing (CEP) is needed");
        System.out.println("  2. Multiple data sources: Kafka + DB + files");
        System.out.println("  3. Exactly-once writes to external sinks (JDBC, S3)");
        System.out.println("  4. Advanced event-time with watermarks and late handling");
        System.out.println("  5. Flink SQL for ad-hoc stream queries");
        System.out.println("  6. Very high throughput (millions of events/sec)");
        System.out.println();
        System.out.println("Banking examples:");
        System.out.println("  CEP: 3 failed logins + large withdrawal = block account");
        System.out.println("  Multi-source: join Kafka payments with Oracle CDC");
        System.out.println("  Sink: exactly-once insert into PostgreSQL ledger");
    }
}
TEXT
Expected output:
Choose Flink when:
  1. Complex event processing (CEP) is needed
  2. Multiple data sources: Kafka + DB + files
  3. Exactly-once writes to external sinks (JDBC, S3)
  4. Advanced event-time with watermarks and late handling
  5. Flink SQL for ad-hoc stream queries
  6. Very high throughput (millions of events/sec)

Banking examples:
  CEP: 3 failed logins + large withdrawal = block account
  Multi-source: join Kafka payments with Oracle CDC
  Sink: exactly-once insert into PostgreSQL ledger

When should you choose Spark Streaming?

Spark Structured Streaming is the right choice when you need unified batch and stream processing, or when your team already uses Spark for analytics, ML, and data engineering. The micro-batch model adds latency but provides strong consistency.

  • Good use cases for Spark at a bank:
  • Unified pipeline: same code processes historical batch data and real-time streams.
  • ML scoring: score transactions against a fraud model trained in Spark MLlib.
  • Data lake ingestion: stream transactions into Delta Lake or Iceberg for analytics.
JAVA
package com.bank.streams;

public class WhenToChooseSpark {
    public static void main(String[] args) {
        System.out.println("Choose Spark Structured Streaming when:");
        System.out.println("  1. Unified batch + stream processing");
        System.out.println("  2. Team already uses Spark for analytics/ML");
        System.out.println("  3. Data lake ingestion (Delta Lake, Iceberg)");
        System.out.println("  4. SQL-first development approach");
        System.out.println("  5. ML model serving in the streaming pipeline");
        System.out.println("  6. Latency of 100ms+ is acceptable");
        System.out.println();
        System.out.println("Banking examples:");
        System.out.println("  ML scoring: real-time fraud score per transaction");
        System.out.println("  Data lake: stream to Delta Lake for BI dashboards");
        System.out.println("  Reconciliation: batch reprocess + real-time matching");
    }
}
TEXT
Expected output:
Choose Spark Structured Streaming when:
  1. Unified batch + stream processing
  2. Team already uses Spark for analytics/ML
  3. Data lake ingestion (Delta Lake, Iceberg)
  4. SQL-first development approach
  5. ML model serving in the streaming pipeline
  6. Latency of 100ms+ is acceptable

Banking examples:
  ML scoring: real-time fraud score per transaction
  Data lake: stream to Delta Lake for BI dashboards
  Reconciliation: batch reprocess + real-time matching

What is a banking scenario comparison project?

This project shows how you would approach the same fraud detection problem in all three frameworks.

Project structure

DIAGRAM
framework-comparison/
├── pom.xml
└── src/
    └── main/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        ├── KafkaStreamsFraud.java
                        ├── FlinkFraudPseudocode.java
                        ├── SparkFraudPseudocode.java
                        ├── FrameworkChooser.java
                        └── AtmWithdrawal.java

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>framework-comparisonsyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
    syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
        syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
        syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
        syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
    syn-kw"></propertiessyn-kw">>
    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

AtmWithdrawal.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class AtmWithdrawal {
    private String accountId;
    private String atmId;
    private BigDecimal amount;
    private String city;

    public AtmWithdrawal() {}

    public AtmWithdrawal(String accountId, String atmId,
                         BigDecimal amount, String city) {
        this.accountId = accountId;
        this.atmId = atmId;
        this.amount = amount;
        this.city = city;
    }

    public String getAccountId() { return accountId; }
    public String getAtmId() { return atmId; }
    public BigDecimal getAmount() { return amount; }
    public String getCity() { return city; }

    @Override
    public String toString() {
        return accountId + "|" + atmId + "|" + amount + "|" + city;
    }

    public static AtmWithdrawal fromString(String value) {
        String[] parts = value.split("\\|");
        return new AtmWithdrawal(parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
    }
}

KafkaStreamsFraud.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.TimeWindows;
import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;

import java.time.Duration;

public class KafkaStreamsFraud {

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();
        KStream<String, String> withdrawals = builder.stream("atm-withdrawals");

        withdrawals
            .groupByKey()
            .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
            .count(Materialized.as("withdrawal-counts"))
            .toStream()
            .filter((windowedKey, count) -> count >= 3)
            .selectKey((windowedKey, count) -> windowedKey.key())
            .mapValues((key, count) -> "FRAUD_ALERT|" + key + "|count=" + count)
            .to("fraud-alerts");

        return builder.build();
    }

    public static void main(String[] args) {
        System.out.println("Kafka Streams fraud detection:");
        System.out.println("  stream(atm-withdrawals)");
        System.out.println("  .groupByKey()");
        System.out.println("  .windowedBy(TimeWindows.ofSizeWithNoGrace(5 min))");
        System.out.println("  .count()");
        System.out.println("  .filter(count >= 3)");
        System.out.println("  .to(fraud-alerts)");
        System.out.println();
        System.out.println("Lines of topology code: ~15");
        System.out.println("Cluster required: No");
        System.out.println("Deploy: java -jar fraud-detector.jar");
    }
}
TEXT
Expected output:
Kafka Streams fraud detection:
  stream(atm-withdrawals)
  .groupByKey()
  .windowedBy(TimeWindows.ofSizeWithNoGrace(5 min))
  .count()
  .filter(count >= 3)
  .to(fraud-alerts)

Lines of topology code: ~15
Cluster required: No
Deploy: java -jar fraud-detector.jar

FlinkFraudPseudocode.java

JAVA
package com.bank.streams;

public class FlinkFraudPseudocode {
    public static void main(String[] args) {
        System.out.println("Flink fraud detection (pseudocode):");
        System.out.println("  StreamExecutionEnvironment env = ...");
        System.out.println("  DataStream<String> withdrawals = env");
        System.out.println("    .addSource(new FlinkKafkaConsumer<>(\"atm-withdrawals\", ...))");
        System.out.println("    .assignTimestampsAndWatermarks(strategy)");
        System.out.println("    .keyBy(w -> w.getAccountId())");
        System.out.println("    .window(TumblingEventTimeWindows.of(Time.minutes(5)))");
        System.out.println("    .aggregate(new CountAggregate())");
        System.out.println("    .filter(count -> count >= 3)");
        System.out.println("    .addSink(new FlinkKafkaProducer<>(\"fraud-alerts\", ...))");
        System.out.println("  env.execute(\"fraud-detection\")");
        System.out.println();
        System.out.println("Lines of processing code: ~20");
        System.out.println("Cluster required: Yes (JobManager + TaskManagers)");
        System.out.println("Deploy: flink run fraud-detection.jar");
        System.out.println();
        System.out.println("Advantage: watermarks for precise event-time handling");
        System.out.println("Advantage: CEP library for complex pattern detection");
    }
}
TEXT
Expected output:
Flink fraud detection (pseudocode):
  StreamExecutionEnvironment env = ...
  DataStream<String> withdrawals = env
    .addSource(new FlinkKafkaConsumer<>("atm-withdrawals", ...))
    .assignTimestampsAndWatermarks(strategy)
    .keyBy(w -> w.getAccountId())
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .aggregate(new CountAggregate())
    .filter(count -> count >= 3)
    .addSink(new FlinkKafkaProducer<>("fraud-alerts", ...))
  env.execute("fraud-detection")

Lines of processing code: ~20
Cluster required: Yes (JobManager + TaskManagers)
Deploy: flink run fraud-detection.jar

Advantage: watermarks for precise event-time handling
Advantage: CEP library for complex pattern detection

SparkFraudPseudocode.java

JAVA
package com.bank.streams;

public class SparkFraudPseudocode {
    public static void main(String[] args) {
        System.out.println("Spark Structured Streaming fraud detection (pseudocode):");
        System.out.println("  SparkSession spark = SparkSession.builder()...");
        System.out.println("  Dataset<Row> withdrawals = spark.readStream()");
        System.out.println("    .format(\"kafka\")");
        System.out.println("    .option(\"subscribe\", \"atm-withdrawals\")");
        System.out.println("    .load()");
        System.out.println("    .withWatermark(\"timestamp\", \"1 minute\")");
        System.out.println("    .groupBy(window(col(\"timestamp\"), \"5 minutes\"),");
        System.out.println("             col(\"accountId\"))");
        System.out.println("    .count()");
        System.out.println("    .filter(col(\"count\").geq(3))");
        System.out.println("    .writeStream()");
        System.out.println("    .format(\"kafka\")");
        System.out.println("    .option(\"topic\", \"fraud-alerts\")");
        System.out.println("    .start()");
        System.out.println();
        System.out.println("Lines of processing code: ~15");
        System.out.println("Cluster required: Yes (Driver + Executors)");
        System.out.println("Deploy: spark-submit fraud-detection.jar");
        System.out.println();
        System.out.println("Advantage: SQL syntax for stream processing");
        System.out.println("Advantage: same code for batch and stream");
        System.out.println("Disadvantage: micro-batch adds 100ms+ latency");
    }
}
TEXT
Expected output:
Spark Structured Streaming fraud detection (pseudocode):
  SparkSession spark = SparkSession.builder()...
  Dataset<Row> withdrawals = spark.readStream()
    .format("kafka")
    .option("subscribe", "atm-withdrawals")
    .load()
    .withWatermark("timestamp", "1 minute")
    .groupBy(window(col("timestamp"), "5 minutes"),
             col("accountId"))
    .count()
    .filter(col("count").geq(3))
    .writeStream()
    .format("kafka")
    .option("topic", "fraud-alerts")
    .start()

Lines of processing code: ~15
Cluster required: Yes (Driver + Executors)
Deploy: spark-submit fraud-detection.jar

Advantage: SQL syntax for stream processing
Advantage: same code for batch and stream
Disadvantage: micro-batch adds 100ms+ latency

FrameworkChooser.java (test class)

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Instant;
import java.util.Properties;

public class FrameworkChooser {
    public static void main(String[] args) {
        // Test the Kafka Streams implementation
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

        try (TopologyTestDriver driver = new TopologyTestDriver(KafkaStreamsFraud.build(), props)) {
            TestInputTopic<String, String> input = driver.createInputTopic(
                "atm-withdrawals", new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> output = driver.createOutputTopic(
                "fraud-alerts", new StringDeserializer(), new StringDeserializer());

            Instant base = Instant.parse("2024-01-15T10:00:00Z");

            input.pipeInput("SAV2001", "SAV2001|ATM001|10000|Mumbai", base);
            input.pipeInput("SAV2001", "SAV2001|ATM002|15000|Mumbai", base.plusSeconds(60));
            input.pipeInput("SAV2001", "SAV2001|ATM003|20000|Delhi", base.plusSeconds(120));

            System.out.println("Kafka Streams test results:");
            while (!output.isEmpty()) {
                System.out.println("  " + output.readValue());
            }
            System.out.println();
            System.out.println("Decision guide:");
            System.out.println("  Kafka-in, Kafka-out, no cluster -> Kafka Streams");
            System.out.println("  CEP, multi-source, DB sink      -> Flink");
            System.out.println("  Batch + stream, ML, data lake   -> Spark");
        }
    }
}
TEXT
Expected output:
Kafka Streams test results:
  FRAUD_ALERT|SAV2001|count=3

Decision guide:
  Kafka-in, Kafka-out, no cluster -> Kafka Streams
  CEP, multi-source, DB sink      -> Flink
  Batch + stream, ML, data lake   -> Spark

How to run the project

BASH
# Build
cd framework-comparison
mvn clean package

# Run the comparison (test class)
java -cp target/framework-comparison-1.0.0.jar com.bank.streams.FrameworkChooser

# Run individual pseudocode demos
java -cp target/framework-comparison-1.0.0.jar com.bank.streams.KafkaStreamsFraud
java -cp target/framework-comparison-1.0.0.jar com.bank.streams.FlinkFraudPseudocode
java -cp target/framework-comparison-1.0.0.jar com.bank.streams.SparkFraudPseudocode

What are common mistakes when choosing a framework?

The most common mistake is choosing Flink or Spark when Kafka Streams is sufficient.

Wrong: deploying a Flink cluster for simple Kafka-to-Kafka filtering

JAVA
package com.bank.streams;

public class WrongOverEngineered {
    public static void main(String[] args) {
        System.out.println("WRONG: Deploying Flink for simple Kafka-to-Kafka filtering");
        System.out.println();
        System.out.println("Task: filter high-value transactions from topic A to topic B");
        System.out.println();
        System.out.println("Over-engineered approach:");
        System.out.println("  1. Set up Flink cluster (JobManager + 3 TaskManagers)");
        System.out.println("  2. Configure YARN or Kubernetes for Flink");
        System.out.println("  3. Write Flink job with FlinkKafkaConsumer/Producer");
        System.out.println("  4. Submit job to cluster");
        System.out.println("  5. Monitor Flink dashboard");
        System.out.println();
        System.out.println("Operational cost: 4 JVMs, cluster management, Flink upgrades");
        System.out.println("The task needs 10 lines of Kafka Streams code and 1 JVM.");
    }
}
TEXT
Expected output:
WRONG: Deploying Flink for simple Kafka-to-Kafka filtering

Task: filter high-value transactions from topic A to topic B

Over-engineered approach:
  1. Set up Flink cluster (JobManager + 3 TaskManagers)
  2. Configure YARN or Kubernetes for Flink
  3. Write Flink job with FlinkKafkaConsumer/Producer
  4. Submit job to cluster
  5. Monitor Flink dashboard

Operational cost: 4 JVMs, cluster management, Flink upgrades
The task needs 10 lines of Kafka Streams code and 1 JVM.

Correct: match the framework to the complexity

JAVA
package com.bank.streams;

public class CorrectFrameworkChoice {
    public static void main(String[] args) {
        System.out.println("CORRECT: Match framework to complexity");
        System.out.println();
        System.out.println("Simple Kafka-to-Kafka pipeline:");
        System.out.println("  -> Kafka Streams (library, no cluster, 1 JVM)");
        System.out.println();
        System.out.println("Complex event processing with DB sinks:");
        System.out.println("  -> Flink (cluster, advanced event-time, CEP)");
        System.out.println();
        System.out.println("Unified batch + stream with ML scoring:");
        System.out.println("  -> Spark (cluster, SQL, MLlib, data lake)");
        System.out.println();
        System.out.println("Rule: start with the simplest tool that meets requirements.");
        System.out.println("Upgrade to a more powerful framework only when needed.");
    }
}
TEXT
Expected output:
CORRECT: Match framework to complexity

Simple Kafka-to-Kafka pipeline:
  -> Kafka Streams (library, no cluster, 1 JVM)

Complex event processing with DB sinks:
  -> Flink (cluster, advanced event-time, CEP)

Unified batch + stream with ML scoring:
  -> Spark (cluster, SQL, MLlib, data lake)

Rule: start with the simplest tool that meets requirements.
Upgrade to a more powerful framework only when needed.

What should a senior or principal engineer know?

At the senior level, framework choice is an architecture decision with long-term consequences.

  • Operational cost: Kafka Streams has near-zero operational overhead. Flink requires a dedicated cluster, monitoring, upgrades, and a Flink-savvy team. Spark requires a Spark cluster and often YARN/K8s expertise.
  • Team skills: if your team knows Spark for batch, Spark Streaming is the lowest-friction choice for streaming. If your team is Kafka-native, Kafka Streams is the natural fit.
  • Vendor lock-in: Kafka Streams requires Kafka. Flink and Spark can read from many sources. If you might migrate away from Kafka, Flink gives more flexibility.
  • Hybrid architectures: many banks use Kafka Streams for real-time pipelines and Spark for batch analytics. Flink bridges both worlds but adds cluster complexity.
  • Cost: Kafka Streams runs on your existing Kafka infrastructure. Flink and Spark require additional compute resources for their clusters.
  • Exactly-once scope: Kafka Streams provides exactly-once within Kafka only. Flink provides exactly-once to external sinks via two-phase commit. This is critical for writing to databases.
JAVA
package com.bank.streams;

public class SeniorFrameworkKnowledge {
    public static void main(String[] args) {
        System.out.println("Senior/Principal framework choice checklist:");
        System.out.println("  1. Start simple: Kafka Streams first");
        System.out.println("  2. Upgrade to Flink only if CEP or DB sinks needed");
        System.out.println("  3. Use Spark if batch + stream unification is required");
        System.out.println("  4. Consider team skills and hiring market");
        System.out.println("  5. Kafka Streams: zero cluster cost");
        System.out.println("  6. Flink: exactly-once to external sinks");
        System.out.println("  7. Spark: strongest ML and SQL ecosystem");
        System.out.println("  8. Hybrid is common: Kafka Streams + Spark at same bank");
    }
}
TEXT
Expected output:
Senior/Principal framework choice checklist:
  1. Start simple: Kafka Streams first
  2. Upgrade to Flink only if CEP or DB sinks needed
  3. Use Spark if batch + stream unification is required
  4. Consider team skills and hiring market
  5. Kafka Streams: zero cluster cost
  6. Flink: exactly-once to external sinks
  7. Spark: strongest ML and SQL ecosystem
  8. Hybrid is common: Kafka Streams + Spark at same bank

What are common interview follow-up questions?

  • Can Kafka Streams replace Flink entirely? (No. Kafka Streams cannot read from non-Kafka sources, does not have a CEP library, and provides exactly-once only within Kafka. Flink is needed for multi-source and external-sink scenarios.)
  • Can Flink replace Kafka Streams? (Technically yes, but at the cost of cluster complexity. For simple Kafka-to-Kafka pipelines, Kafka Streams is simpler and cheaper.)
  • What about ksqlDB? (ksqlDB is a SQL engine built on Kafka Streams. It lets you write stream processing in SQL without Java code. It is an alternative to Kafka Streams DSL for teams that prefer SQL.)
  • How does latency compare in practice? (Kafka Streams: 1-10ms per record. Flink: sub-millisecond to 5ms. Spark micro-batch: 100ms-1s per batch. These are approximate; actual numbers depend on workload.)
  • Can you mix frameworks in one pipeline? (Yes. A common pattern is Kafka Streams for real-time enrichment, Flink for CEP, and Spark for batch analytics, all reading from the same Kafka topics.)

Visual Overview

DIAGRAM
 Framework Deployment Comparison
 ================================

 Kafka Streams:
 ┌────────────────────────┐
 │ Your Application (JVM) │
 │ ┌────────────────────┐ │
 │ │ Kafka Streams Lib  │ │
 │ │ Topology + State   │ │
 │ └────────────────────┘ │
 └────────────────────────┘
         │ reads/writes
         ▼
 ┌────────────────────────┐
 │ Kafka Cluster          │
 └────────────────────────┘


 Apache Flink:
 ┌────────────────────────┐
 │ Flink Cluster          │
 │ ┌──────────────┐       │
 │ │ JobManager   │       │
 │ └──────┬───────┘       │
 │   ┌────┴────┐          │
 │ ┌─┤TaskMgr 1├─┐        │
 │ │ └─────────┘ │        │
 │ │ ┌─────────┐ │        │
 │ └─┤TaskMgr 2├─┘        │
 │   └─────────┘          │
 └────────────────────────┘
         │ reads/writes
         ▼
 ┌────────────────────────┐
 │ Kafka + HDFS + DB      │
 └────────────────────────┘


 Spark Streaming:
 ┌────────────────────────┐
 │ Spark Cluster          │
 │ ┌──────────────┐       │
 │ │ Driver       │       │
 │ └──────┬───────┘       │
 │   ┌────┴────┐          │
 │ ┌─┤Executor1├─┐        │
 │ │ └─────────┘ │        │
 │ │ ┌─────────┐ │        │
 │ └─┤Executor2├─┘        │
 │   └─────────┘          │
 └────────────────────────┘
         │ micro-batches
         ▼
 ┌────────────────────────┐
 │ Kafka + S3 + Delta     │
 └────────────────────────┘

Key Takeaway

Kafka Streams, Apache Flink, and Spark Structured Streaming serve different points on the stream processing spectrum. Kafka Streams is a library that runs inside your JVM with no separate cluster, reading only from Kafka. Flink is a distributed framework with advanced event-time processing, watermarks, and exactly-once writes to external sinks. Spark uses micro-batching and provides unified batch and stream processing with strong SQL and ML support.

For National Bank of India, the choice depends on the use case. Simple Kafka-to-Kafka pipelines like transaction filtering, balance maintenance, and windowed fraud counts are best served by Kafka Streams. Complex event processing like detecting multi-step fraud patterns, or writing exactly-once to a database, requires Flink. Unified batch-and-stream workloads like ML scoring and data lake ingestion are best served by Spark.

The most common mistake is over-engineering: deploying a Flink cluster for a task that Kafka Streams handles in 10 lines. Start with the simplest tool that meets your requirements. Upgrade when you genuinely need features that the simpler tool does not provide.

At the senior level, framework choice has long-term consequences for team skills, operational cost, and architectural flexibility. Many organizations use a hybrid approach: Kafka Streams for real-time microservice pipelines and Spark for batch analytics, both reading from the same Kafka topics.

Remember one sentence: Kafka Streams is a library for Kafka-to-Kafka pipelines, Flink is a cluster framework for complex event processing and external sinks, and Spark is a cluster framework for unified batch-stream workloads with SQL and ML.

---

Share
Use ← → arrow keys to navigate