Definition
- Kafka Streams is a client library that runs inside your application as a regular Java process; it reads from Kafka, processes, and writes back to Kafka without a separate cluster.
- Apache Flink is a distributed stream processing framework that runs on its own cluster (JobManager + TaskManagers) with advanced event-time processing, watermarks, and exactly-once guarantees across external sinks.
- Spark Structured Streaming is Apache Spark's stream processing engine that uses micro-batching to process streams as a series of small batch jobs; it shares Spark's batch engine, SQL support, and ML libraries.
- Micro-batching means collecting records for a short interval (e.g., 100ms) and processing them as a batch, which adds latency compared to record-at-a-time processing.
- Watermark is Flink's mechanism for tracking event-time progress and handling late events; it defines how far behind event-time the system expects records to be.
Think of National Bank of India processing transactions across three different setups. Kafka Streams is like a clerk sitting at a desk with a paper tray: transactions come in, the clerk processes them one by one, and puts results in an outgoing tray. No special infrastructure needed, just a desk. Flink is like a factory floor with conveyor belts, specialized stations, and a foreman (JobManager): it handles complex assembly lines, branching, and merging at high volume. Spark Streaming is like a mail room that opens a bag of letters every minute, sorts and processes each bag, then sends results: efficient for volume but each bag adds a small delay.
Understanding the Concepts
What is the architecture difference between the three frameworks?
The fundamental difference is deployment model. Kafka Streams is a library, not a framework. Flink and Spark are distributed systems that require cluster management.
| Feature | Kafka Streams | Apache Flink | Spark Streaming |
|---|---|---|---|
| Deployment | Library (runs in your JVM) | Distributed cluster (JobManager + TaskManagers) | Distributed cluster (Driver + Executors) |
| Processing model | Record-at-a-time, true streaming | Record-at-a-time, true streaming | Micro-batch (Structured Streaming) |
| State management | RocksDB local store + changelog topic | RocksDB + distributed snapshots (Savepoints) | In-memory or external (HDFS checkpoints) |
| Exactly-once | Kafka transactions (processing.guarantee) | Distributed snapshots (Chandy-Lamport) | WAL + idempotent sinks |
| Scaling | By Kafka partition count | By task parallelism (independent of Kafka) | By executor count |
| Latency | Milliseconds | Milliseconds | Seconds (micro-batch interval) |
| Dependency | Kafka only | Standalone, works with any source | Standalone, works with any source |
| SQL support | ksqlDB (separate component) | Flink SQL (built-in) | Spark SQL (built-in) |
| Cluster management | None needed (library) | YARN, Kubernetes, standalone | YARN, Kubernetes, standalone |
- Kafka Streams: add a Maven dependency, write your topology, run your JAR. No cluster, no YARN, no Kubernetes operator needed (though you can use Kubernetes for scaling). The library runs inside your JVM.
- Flink: deploy a Flink cluster (JobManager + TaskManagers) on YARN, Kubernetes, or standalone. Submit your job to the cluster. The cluster manages parallelism, checkpointing, and failover.
- Spark: deploy a Spark cluster (Driver + Executors) on YARN, Kubernetes, or standalone. Submit your streaming job. The driver schedules micro-batches on executors.
package com.bank.streams;
public class ArchitectureComparison {
public static void main(String[] args) {
System.out.println("Architecture comparison:");
System.out.println();
System.out.println("Kafka Streams:");
System.out.println(" Type: Library (JAR dependency)");
System.out.println(" Deploy: java -jar your-app.jar");
System.out.println(" Cluster: None required");
System.out.println(" Scaling: Start more JVM instances");
System.out.println();
System.out.println("Apache Flink:");
System.out.println(" Type: Distributed framework");
System.out.println(" Deploy: flink run your-job.jar");
System.out.println(" Cluster: JobManager + TaskManagers");
System.out.println(" Scaling: Add TaskManagers, adjust parallelism");
System.out.println();
System.out.println("Spark Streaming:");
System.out.println(" Type: Distributed framework");
System.out.println(" Deploy: spark-submit your-job.jar");
System.out.println(" Cluster: Driver + Executors");
System.out.println(" Scaling: Add executors, adjust partitions");
}
}
Expected output:
Architecture comparison:
Kafka Streams:
Type: Library (JAR dependency)
Deploy: java -jar your-app.jar
Cluster: None required
Scaling: Start more JVM instances
Apache Flink:
Type: Distributed framework
Deploy: flink run your-job.jar
Cluster: JobManager + TaskManagers
Scaling: Add TaskManagers, adjust parallelism
Spark Streaming:
Type: Distributed framework
Deploy: spark-submit your-job.jar
Cluster: Driver + Executors
Scaling: Add executors, adjust partitions
How do the three frameworks compare on key features?
| Feature | Kafka Streams | Apache Flink | Spark Structured Streaming |
|---|---|---|---|
| Deployment | Library (JAR) | Cluster (JobManager) | Cluster (Driver) |
| Data source | Kafka only | Kafka, HDFS, S3, DB, files | Kafka, HDFS, S3, DB, files |
| Processing model | Record-at-a-time | Record-at-a-time | Micro-batch |
| Latency | Low (ms) | Very low (ms) | Higher (100ms+ per batch) |
| Exactly-once | Kafka-to-Kafka only | End-to-end (with sinks) | End-to-end (with sinks) |
| Windowing | Tumbling, hopping, sliding, session | All + custom windows | Tumbling, sliding |
| Event-time handling | Stream time | Watermarks (advanced) | Watermarks (basic) |
| State management | RocksDB + changelog | RocksDB + checkpoints | In-memory + checkpoints |
| SQL support | No (use ksqlDB separately) | Flink SQL (built-in) | Spark SQL (built-in) |
| Batch processing | No | Yes (unified batch + stream) | Yes (unified batch + stream) |
| Language support | Java, Scala | Java, Scala, Python, SQL | Java, Scala, Python, R, SQL |
| Cluster management | Not needed | YARN, K8s, standalone | YARN, K8s, standalone |
| Interactive queries | Built-in (state store) | Not built-in | Not built-in |
| Learning curve | Low | Medium-High | Medium |
package com.bank.streams;
public class FeatureComparisonDemo {
public static void main(String[] args) {
System.out.println("Key differentiators:");
System.out.println();
System.out.println("1. Deployment complexity:");
System.out.println(" Kafka Streams < Spark < Flink");
System.out.println();
System.out.println("2. Latency (best case):");
System.out.println(" Flink < Kafka Streams < Spark");
System.out.println();
System.out.println("3. Feature richness:");
System.out.println(" Flink > Spark > Kafka Streams");
System.out.println();
System.out.println("4. Data source variety:");
System.out.println(" Spark = Flink > Kafka Streams (Kafka only)");
System.out.println();
System.out.println("5. Batch + stream unified:");
System.out.println(" Flink = Spark (yes), Kafka Streams (no)");
}
}
Expected output:
Key differentiators:
1. Deployment complexity:
Kafka Streams < Spark < Flink
2. Latency (best case):
Flink < Kafka Streams < Spark
3. Feature richness:
Flink > Spark > Kafka Streams
4. Data source variety:
Spark = Flink > Kafka Streams (Kafka only)
5. Batch + stream unified:
Flink = Spark (yes), Kafka Streams (no)
When should you choose Kafka Streams?
Kafka Streams is the right choice when your pipeline is Kafka-centric: read from Kafka, process, write to Kafka. It is the simplest option for teams that already use Kafka and do not want to manage a separate processing cluster.
- Good use cases for Kafka Streams at a bank:
- Real-time transaction filtering and routing.
- Maintaining account balances in a state store.
- Fraud detection based on transaction counts per window.
- Enriching payment events with account data from a KTable.
package com.bank.streams;
public class WhenToChooseKafkaStreams {
public static void main(String[] args) {
System.out.println("Choose Kafka Streams when:");
System.out.println(" 1. Input and output are Kafka topics");
System.out.println(" 2. You want zero cluster management overhead");
System.out.println(" 3. Processing is filter, map, join, aggregate, window");
System.out.println(" 4. Team already knows Kafka and Java");
System.out.println(" 5. You need interactive queries (state store REST API)");
System.out.println(" 6. Sub-second latency is needed but not sub-millisecond");
System.out.println();
System.out.println("Banking examples:");
System.out.println(" Transaction routing: NEFT/RTGS/IMPS per channel");
System.out.println(" Balance maintenance: deposit/withdrawal aggregation");
System.out.println(" Fraud alerts: count ATM withdrawals per 5-min window");
}
}
Expected output:
Choose Kafka Streams when:
1. Input and output are Kafka topics
2. You want zero cluster management overhead
3. Processing is filter, map, join, aggregate, window
4. Team already knows Kafka and Java
5. You need interactive queries (state store REST API)
6. Sub-second latency is needed but not sub-millisecond
Banking examples:
Transaction routing: NEFT/RTGS/IMPS per channel
Balance maintenance: deposit/withdrawal aggregation
Fraud alerts: count ATM withdrawals per 5-min window
When should you choose Flink?
Flink is the right choice when you need advanced event-time processing, complex event patterns, or exactly-once writes to external systems (databases, file systems). Flink's watermark mechanism is more sophisticated than Kafka Streams' stream time.
- Good use cases for Flink at a bank:
- Complex event processing: detect patterns like "3 failed logins followed by a large withdrawal within 10 minutes."
- Multi-source pipelines: join Kafka streams with database change data capture (CDC) and file ingestion.
- Exactly-once writes to a database (using Flink's two-phase commit sinks).
package com.bank.streams;
public class WhenToChooseFlink {
public static void main(String[] args) {
System.out.println("Choose Flink when:");
System.out.println(" 1. Complex event processing (CEP) is needed");
System.out.println(" 2. Multiple data sources: Kafka + DB + files");
System.out.println(" 3. Exactly-once writes to external sinks (JDBC, S3)");
System.out.println(" 4. Advanced event-time with watermarks and late handling");
System.out.println(" 5. Flink SQL for ad-hoc stream queries");
System.out.println(" 6. Very high throughput (millions of events/sec)");
System.out.println();
System.out.println("Banking examples:");
System.out.println(" CEP: 3 failed logins + large withdrawal = block account");
System.out.println(" Multi-source: join Kafka payments with Oracle CDC");
System.out.println(" Sink: exactly-once insert into PostgreSQL ledger");
}
}
Expected output:
Choose Flink when:
1. Complex event processing (CEP) is needed
2. Multiple data sources: Kafka + DB + files
3. Exactly-once writes to external sinks (JDBC, S3)
4. Advanced event-time with watermarks and late handling
5. Flink SQL for ad-hoc stream queries
6. Very high throughput (millions of events/sec)
Banking examples:
CEP: 3 failed logins + large withdrawal = block account
Multi-source: join Kafka payments with Oracle CDC
Sink: exactly-once insert into PostgreSQL ledger
When should you choose Spark Streaming?
Spark Structured Streaming is the right choice when you need unified batch and stream processing, or when your team already uses Spark for analytics, ML, and data engineering. The micro-batch model adds latency but provides strong consistency.
- Good use cases for Spark at a bank:
- Unified pipeline: same code processes historical batch data and real-time streams.
- ML scoring: score transactions against a fraud model trained in Spark MLlib.
- Data lake ingestion: stream transactions into Delta Lake or Iceberg for analytics.
package com.bank.streams;
public class WhenToChooseSpark {
public static void main(String[] args) {
System.out.println("Choose Spark Structured Streaming when:");
System.out.println(" 1. Unified batch + stream processing");
System.out.println(" 2. Team already uses Spark for analytics/ML");
System.out.println(" 3. Data lake ingestion (Delta Lake, Iceberg)");
System.out.println(" 4. SQL-first development approach");
System.out.println(" 5. ML model serving in the streaming pipeline");
System.out.println(" 6. Latency of 100ms+ is acceptable");
System.out.println();
System.out.println("Banking examples:");
System.out.println(" ML scoring: real-time fraud score per transaction");
System.out.println(" Data lake: stream to Delta Lake for BI dashboards");
System.out.println(" Reconciliation: batch reprocess + real-time matching");
}
}
Expected output:
Choose Spark Structured Streaming when:
1. Unified batch + stream processing
2. Team already uses Spark for analytics/ML
3. Data lake ingestion (Delta Lake, Iceberg)
4. SQL-first development approach
5. ML model serving in the streaming pipeline
6. Latency of 100ms+ is acceptable
Banking examples:
ML scoring: real-time fraud score per transaction
Data lake: stream to Delta Lake for BI dashboards
Reconciliation: batch reprocess + real-time matching
What is a banking scenario comparison project?
This project shows how you would approach the same fraud detection problem in all three frameworks.
Project structure
framework-comparison/
├── pom.xml
└── src/
└── main/
└── java/
└── com/
└── bank/
└── streams/
├── KafkaStreamsFraud.java
├── FlinkFraudPseudocode.java
├── SparkFraudPseudocode.java
├── FrameworkChooser.java
└── AtmWithdrawal.java
pom.xml
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>framework-comparisonsyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
syn-kw"><propertiessyn-kw">>
syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
syn-kw"></propertiessyn-kw">>
syn-kw"><dependenciessyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>
AtmWithdrawal.java
package com.bank.streams;
import java.math.BigDecimal;
public class AtmWithdrawal {
private String accountId;
private String atmId;
private BigDecimal amount;
private String city;
public AtmWithdrawal() {}
public AtmWithdrawal(String accountId, String atmId,
BigDecimal amount, String city) {
this.accountId = accountId;
this.atmId = atmId;
this.amount = amount;
this.city = city;
}
public String getAccountId() { return accountId; }
public String getAtmId() { return atmId; }
public BigDecimal getAmount() { return amount; }
public String getCity() { return city; }
@Override
public String toString() {
return accountId + "|" + atmId + "|" + amount + "|" + city;
}
public static AtmWithdrawal fromString(String value) {
String[] parts = value.split("\\|");
return new AtmWithdrawal(parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
}
}
KafkaStreamsFraud.java
package com.bank.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.TimeWindows;
import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import java.time.Duration;
public class KafkaStreamsFraud {
public static Topology build() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> withdrawals = builder.stream("atm-withdrawals");
withdrawals
.groupByKey()
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
.count(Materialized.as("withdrawal-counts"))
.toStream()
.filter((windowedKey, count) -> count >= 3)
.selectKey((windowedKey, count) -> windowedKey.key())
.mapValues((key, count) -> "FRAUD_ALERT|" + key + "|count=" + count)
.to("fraud-alerts");
return builder.build();
}
public static void main(String[] args) {
System.out.println("Kafka Streams fraud detection:");
System.out.println(" stream(atm-withdrawals)");
System.out.println(" .groupByKey()");
System.out.println(" .windowedBy(TimeWindows.ofSizeWithNoGrace(5 min))");
System.out.println(" .count()");
System.out.println(" .filter(count >= 3)");
System.out.println(" .to(fraud-alerts)");
System.out.println();
System.out.println("Lines of topology code: ~15");
System.out.println("Cluster required: No");
System.out.println("Deploy: java -jar fraud-detector.jar");
}
}
Expected output:
Kafka Streams fraud detection:
stream(atm-withdrawals)
.groupByKey()
.windowedBy(TimeWindows.ofSizeWithNoGrace(5 min))
.count()
.filter(count >= 3)
.to(fraud-alerts)
Lines of topology code: ~15
Cluster required: No
Deploy: java -jar fraud-detector.jar
FlinkFraudPseudocode.java
package com.bank.streams;
public class FlinkFraudPseudocode {
public static void main(String[] args) {
System.out.println("Flink fraud detection (pseudocode):");
System.out.println(" StreamExecutionEnvironment env = ...");
System.out.println(" DataStream<String> withdrawals = env");
System.out.println(" .addSource(new FlinkKafkaConsumer<>(\"atm-withdrawals\", ...))");
System.out.println(" .assignTimestampsAndWatermarks(strategy)");
System.out.println(" .keyBy(w -> w.getAccountId())");
System.out.println(" .window(TumblingEventTimeWindows.of(Time.minutes(5)))");
System.out.println(" .aggregate(new CountAggregate())");
System.out.println(" .filter(count -> count >= 3)");
System.out.println(" .addSink(new FlinkKafkaProducer<>(\"fraud-alerts\", ...))");
System.out.println(" env.execute(\"fraud-detection\")");
System.out.println();
System.out.println("Lines of processing code: ~20");
System.out.println("Cluster required: Yes (JobManager + TaskManagers)");
System.out.println("Deploy: flink run fraud-detection.jar");
System.out.println();
System.out.println("Advantage: watermarks for precise event-time handling");
System.out.println("Advantage: CEP library for complex pattern detection");
}
}
Expected output:
Flink fraud detection (pseudocode):
StreamExecutionEnvironment env = ...
DataStream<String> withdrawals = env
.addSource(new FlinkKafkaConsumer<>("atm-withdrawals", ...))
.assignTimestampsAndWatermarks(strategy)
.keyBy(w -> w.getAccountId())
.window(TumblingEventTimeWindows.of(Time.minutes(5)))
.aggregate(new CountAggregate())
.filter(count -> count >= 3)
.addSink(new FlinkKafkaProducer<>("fraud-alerts", ...))
env.execute("fraud-detection")
Lines of processing code: ~20
Cluster required: Yes (JobManager + TaskManagers)
Deploy: flink run fraud-detection.jar
Advantage: watermarks for precise event-time handling
Advantage: CEP library for complex pattern detection
SparkFraudPseudocode.java
package com.bank.streams;
public class SparkFraudPseudocode {
public static void main(String[] args) {
System.out.println("Spark Structured Streaming fraud detection (pseudocode):");
System.out.println(" SparkSession spark = SparkSession.builder()...");
System.out.println(" Dataset<Row> withdrawals = spark.readStream()");
System.out.println(" .format(\"kafka\")");
System.out.println(" .option(\"subscribe\", \"atm-withdrawals\")");
System.out.println(" .load()");
System.out.println(" .withWatermark(\"timestamp\", \"1 minute\")");
System.out.println(" .groupBy(window(col(\"timestamp\"), \"5 minutes\"),");
System.out.println(" col(\"accountId\"))");
System.out.println(" .count()");
System.out.println(" .filter(col(\"count\").geq(3))");
System.out.println(" .writeStream()");
System.out.println(" .format(\"kafka\")");
System.out.println(" .option(\"topic\", \"fraud-alerts\")");
System.out.println(" .start()");
System.out.println();
System.out.println("Lines of processing code: ~15");
System.out.println("Cluster required: Yes (Driver + Executors)");
System.out.println("Deploy: spark-submit fraud-detection.jar");
System.out.println();
System.out.println("Advantage: SQL syntax for stream processing");
System.out.println("Advantage: same code for batch and stream");
System.out.println("Disadvantage: micro-batch adds 100ms+ latency");
}
}
Expected output:
Spark Structured Streaming fraud detection (pseudocode):
SparkSession spark = SparkSession.builder()...
Dataset<Row> withdrawals = spark.readStream()
.format("kafka")
.option("subscribe", "atm-withdrawals")
.load()
.withWatermark("timestamp", "1 minute")
.groupBy(window(col("timestamp"), "5 minutes"),
col("accountId"))
.count()
.filter(col("count").geq(3))
.writeStream()
.format("kafka")
.option("topic", "fraud-alerts")
.start()
Lines of processing code: ~15
Cluster required: Yes (Driver + Executors)
Deploy: spark-submit fraud-detection.jar
Advantage: SQL syntax for stream processing
Advantage: same code for batch and stream
Disadvantage: micro-batch adds 100ms+ latency
FrameworkChooser.java (test class)
package com.bank.streams;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Instant;
import java.util.Properties;
public class FrameworkChooser {
public static void main(String[] args) {
// Test the Kafka Streams implementation
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
try (TopologyTestDriver driver = new TopologyTestDriver(KafkaStreamsFraud.build(), props)) {
TestInputTopic<String, String> input = driver.createInputTopic(
"atm-withdrawals", new StringSerializer(), new StringSerializer());
TestOutputTopic<String, String> output = driver.createOutputTopic(
"fraud-alerts", new StringDeserializer(), new StringDeserializer());
Instant base = Instant.parse("2024-01-15T10:00:00Z");
input.pipeInput("SAV2001", "SAV2001|ATM001|10000|Mumbai", base);
input.pipeInput("SAV2001", "SAV2001|ATM002|15000|Mumbai", base.plusSeconds(60));
input.pipeInput("SAV2001", "SAV2001|ATM003|20000|Delhi", base.plusSeconds(120));
System.out.println("Kafka Streams test results:");
while (!output.isEmpty()) {
System.out.println(" " + output.readValue());
}
System.out.println();
System.out.println("Decision guide:");
System.out.println(" Kafka-in, Kafka-out, no cluster -> Kafka Streams");
System.out.println(" CEP, multi-source, DB sink -> Flink");
System.out.println(" Batch + stream, ML, data lake -> Spark");
}
}
}
Expected output:
Kafka Streams test results:
FRAUD_ALERT|SAV2001|count=3
Decision guide:
Kafka-in, Kafka-out, no cluster -> Kafka Streams
CEP, multi-source, DB sink -> Flink
Batch + stream, ML, data lake -> Spark
How to run the project
# Build
cd framework-comparison
mvn clean package
# Run the comparison (test class)
java -cp target/framework-comparison-1.0.0.jar com.bank.streams.FrameworkChooser
# Run individual pseudocode demos
java -cp target/framework-comparison-1.0.0.jar com.bank.streams.KafkaStreamsFraud
java -cp target/framework-comparison-1.0.0.jar com.bank.streams.FlinkFraudPseudocode
java -cp target/framework-comparison-1.0.0.jar com.bank.streams.SparkFraudPseudocode
What are common mistakes when choosing a framework?
The most common mistake is choosing Flink or Spark when Kafka Streams is sufficient.
Wrong: deploying a Flink cluster for simple Kafka-to-Kafka filtering
package com.bank.streams;
public class WrongOverEngineered {
public static void main(String[] args) {
System.out.println("WRONG: Deploying Flink for simple Kafka-to-Kafka filtering");
System.out.println();
System.out.println("Task: filter high-value transactions from topic A to topic B");
System.out.println();
System.out.println("Over-engineered approach:");
System.out.println(" 1. Set up Flink cluster (JobManager + 3 TaskManagers)");
System.out.println(" 2. Configure YARN or Kubernetes for Flink");
System.out.println(" 3. Write Flink job with FlinkKafkaConsumer/Producer");
System.out.println(" 4. Submit job to cluster");
System.out.println(" 5. Monitor Flink dashboard");
System.out.println();
System.out.println("Operational cost: 4 JVMs, cluster management, Flink upgrades");
System.out.println("The task needs 10 lines of Kafka Streams code and 1 JVM.");
}
}
Expected output:
WRONG: Deploying Flink for simple Kafka-to-Kafka filtering
Task: filter high-value transactions from topic A to topic B
Over-engineered approach:
1. Set up Flink cluster (JobManager + 3 TaskManagers)
2. Configure YARN or Kubernetes for Flink
3. Write Flink job with FlinkKafkaConsumer/Producer
4. Submit job to cluster
5. Monitor Flink dashboard
Operational cost: 4 JVMs, cluster management, Flink upgrades
The task needs 10 lines of Kafka Streams code and 1 JVM.
Correct: match the framework to the complexity
package com.bank.streams;
public class CorrectFrameworkChoice {
public static void main(String[] args) {
System.out.println("CORRECT: Match framework to complexity");
System.out.println();
System.out.println("Simple Kafka-to-Kafka pipeline:");
System.out.println(" -> Kafka Streams (library, no cluster, 1 JVM)");
System.out.println();
System.out.println("Complex event processing with DB sinks:");
System.out.println(" -> Flink (cluster, advanced event-time, CEP)");
System.out.println();
System.out.println("Unified batch + stream with ML scoring:");
System.out.println(" -> Spark (cluster, SQL, MLlib, data lake)");
System.out.println();
System.out.println("Rule: start with the simplest tool that meets requirements.");
System.out.println("Upgrade to a more powerful framework only when needed.");
}
}
Expected output:
CORRECT: Match framework to complexity
Simple Kafka-to-Kafka pipeline:
-> Kafka Streams (library, no cluster, 1 JVM)
Complex event processing with DB sinks:
-> Flink (cluster, advanced event-time, CEP)
Unified batch + stream with ML scoring:
-> Spark (cluster, SQL, MLlib, data lake)
Rule: start with the simplest tool that meets requirements.
Upgrade to a more powerful framework only when needed.
What should a senior or principal engineer know?
At the senior level, framework choice is an architecture decision with long-term consequences.
- Operational cost: Kafka Streams has near-zero operational overhead. Flink requires a dedicated cluster, monitoring, upgrades, and a Flink-savvy team. Spark requires a Spark cluster and often YARN/K8s expertise.
- Team skills: if your team knows Spark for batch, Spark Streaming is the lowest-friction choice for streaming. If your team is Kafka-native, Kafka Streams is the natural fit.
- Vendor lock-in: Kafka Streams requires Kafka. Flink and Spark can read from many sources. If you might migrate away from Kafka, Flink gives more flexibility.
- Hybrid architectures: many banks use Kafka Streams for real-time pipelines and Spark for batch analytics. Flink bridges both worlds but adds cluster complexity.
- Cost: Kafka Streams runs on your existing Kafka infrastructure. Flink and Spark require additional compute resources for their clusters.
- Exactly-once scope: Kafka Streams provides exactly-once within Kafka only. Flink provides exactly-once to external sinks via two-phase commit. This is critical for writing to databases.
package com.bank.streams;
public class SeniorFrameworkKnowledge {
public static void main(String[] args) {
System.out.println("Senior/Principal framework choice checklist:");
System.out.println(" 1. Start simple: Kafka Streams first");
System.out.println(" 2. Upgrade to Flink only if CEP or DB sinks needed");
System.out.println(" 3. Use Spark if batch + stream unification is required");
System.out.println(" 4. Consider team skills and hiring market");
System.out.println(" 5. Kafka Streams: zero cluster cost");
System.out.println(" 6. Flink: exactly-once to external sinks");
System.out.println(" 7. Spark: strongest ML and SQL ecosystem");
System.out.println(" 8. Hybrid is common: Kafka Streams + Spark at same bank");
}
}
Expected output:
Senior/Principal framework choice checklist:
1. Start simple: Kafka Streams first
2. Upgrade to Flink only if CEP or DB sinks needed
3. Use Spark if batch + stream unification is required
4. Consider team skills and hiring market
5. Kafka Streams: zero cluster cost
6. Flink: exactly-once to external sinks
7. Spark: strongest ML and SQL ecosystem
8. Hybrid is common: Kafka Streams + Spark at same bank
What are common interview follow-up questions?
- Can Kafka Streams replace Flink entirely? (No. Kafka Streams cannot read from non-Kafka sources, does not have a CEP library, and provides exactly-once only within Kafka. Flink is needed for multi-source and external-sink scenarios.)
- Can Flink replace Kafka Streams? (Technically yes, but at the cost of cluster complexity. For simple Kafka-to-Kafka pipelines, Kafka Streams is simpler and cheaper.)
- What about ksqlDB? (ksqlDB is a SQL engine built on Kafka Streams. It lets you write stream processing in SQL without Java code. It is an alternative to Kafka Streams DSL for teams that prefer SQL.)
- How does latency compare in practice? (Kafka Streams: 1-10ms per record. Flink: sub-millisecond to 5ms. Spark micro-batch: 100ms-1s per batch. These are approximate; actual numbers depend on workload.)
- Can you mix frameworks in one pipeline? (Yes. A common pattern is Kafka Streams for real-time enrichment, Flink for CEP, and Spark for batch analytics, all reading from the same Kafka topics.)
Visual Overview
Framework Deployment Comparison
================================
Kafka Streams:
┌────────────────────────┐
│ Your Application (JVM) │
│ ┌────────────────────┐ │
│ │ Kafka Streams Lib │ │
│ │ Topology + State │ │
│ └────────────────────┘ │
└────────────────────────┘
│ reads/writes
▼
┌────────────────────────┐
│ Kafka Cluster │
└────────────────────────┘
Apache Flink:
┌────────────────────────┐
│ Flink Cluster │
│ ┌──────────────┐ │
│ │ JobManager │ │
│ └──────┬───────┘ │
│ ┌────┴────┐ │
│ ┌─┤TaskMgr 1├─┐ │
│ │ └─────────┘ │ │
│ │ ┌─────────┐ │ │
│ └─┤TaskMgr 2├─┘ │
│ └─────────┘ │
└────────────────────────┘
│ reads/writes
▼
┌────────────────────────┐
│ Kafka + HDFS + DB │
└────────────────────────┘
Spark Streaming:
┌────────────────────────┐
│ Spark Cluster │
│ ┌──────────────┐ │
│ │ Driver │ │
│ └──────┬───────┘ │
│ ┌────┴────┐ │
│ ┌─┤Executor1├─┐ │
│ │ └─────────┘ │ │
│ │ ┌─────────┐ │ │
│ └─┤Executor2├─┘ │
│ └─────────┘ │
└────────────────────────┘
│ micro-batches
▼
┌────────────────────────┐
│ Kafka + S3 + Delta │
└────────────────────────┘
Key Takeaway
Kafka Streams, Apache Flink, and Spark Structured Streaming serve different points on the stream processing spectrum. Kafka Streams is a library that runs inside your JVM with no separate cluster, reading only from Kafka. Flink is a distributed framework with advanced event-time processing, watermarks, and exactly-once writes to external sinks. Spark uses micro-batching and provides unified batch and stream processing with strong SQL and ML support.
For National Bank of India, the choice depends on the use case. Simple Kafka-to-Kafka pipelines like transaction filtering, balance maintenance, and windowed fraud counts are best served by Kafka Streams. Complex event processing like detecting multi-step fraud patterns, or writing exactly-once to a database, requires Flink. Unified batch-and-stream workloads like ML scoring and data lake ingestion are best served by Spark.
The most common mistake is over-engineering: deploying a Flink cluster for a task that Kafka Streams handles in 10 lines. Start with the simplest tool that meets your requirements. Upgrade when you genuinely need features that the simpler tool does not provide.
At the senior level, framework choice has long-term consequences for team skills, operational cost, and architectural flexibility. Many organizations use a hybrid approach: Kafka Streams for real-time microservice pipelines and Spark for batch analytics, both reading from the same Kafka topics.
Remember one sentence: Kafka Streams is a library for Kafka-to-Kafka pipelines, Flink is a cluster framework for complex event processing and external sinks, and Spark is a cluster framework for unified batch-stream workloads with SQL and ML.
---