Definition
- Flink – true streaming engine processing each event as it arrives, with native event-time semantics, rich keyed state (value, list, map), timers, and millisecond latency.
- Spark Structured Streaming – micro-batch by default, treating streams as unbounded tables with sub-second to second latency, and a unified batch-stream API.
- State – Flink has pluggable state backends (RocksDB for large state), fine-grained TTL, and incremental checkpoints; Spark's state store is tied to the micro-batch model.
- Ecosystem – Spark has a larger community, more connectors, and unifies batch, SQL, ML, and streaming on one engine.
- Choice – Flink for lowest latency and complex state; Spark when ecosystem unification matters more.
Flink is a dedicated teller at the National Bank who handles each transaction the instant it arrives and keeps detailed personal notes on every customer. Spark micro-batch is a teller who collects transactions into small stacks every second, processes each stack together, and also happens to run the bank's nightly reports and analytics. The dedicated teller is faster; the versatile teller covers more ground.
Understanding the Concepts
What is the fundamental difference?
The core difference is the processing model. Flink runs a long-lived dataflow graph where operators are always on and each record flows through as it arrives. Spark Structured Streaming treats a stream as an unbounded table and repeatedly runs an incremental batch query every trigger interval.
# Spark Structured Streaming: micro-batch model
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("NBI-Streaming").getOrCreate()
# Read from Kafka as a stream
txn_stream = spark.readStream \
.format("kafka") \
.option("subscribe", "nbi.transactions") \
.load()
# Process as a micro-batch every 1 second
query = txn_stream \
.selectExpr("CAST(value AS STRING)") \
.writeStream \
.trigger(processingTime="1 second") \
.format("parquet") \
.option("path", "s3://nbi-data-lake/streaming-txns/") \
.option("checkpointLocation", "s3://nbi-data-lake/checkpoints/") \
.start()
How does latency compare?
Flink processes events in single-digit to tens of milliseconds because there is no batch boundary to wait for. Spark micro-batch adds the trigger interval as a floor on latency.
| Engine | Latency | Model |
|---|---|---|
| Flink | 1-50 ms | True record-at-a-time |
| Spark (micro-batch) | 100 ms - seconds | Batch per trigger interval |
| Spark (continuous, experimental) | ~10 ms | Record-at-a-time (limited features) |
# Spark micro-batch: latency = trigger interval + processing time
# Minimum practical trigger: 100ms
query = stream.writeStream \
.trigger(processingTime="100 milliseconds") \
.start()
# Spark continuous mode (experimental):
query = stream.writeStream \
.trigger(continuous="1 second") \
.start()
# Lower latency but limited operations supported
How does state management compare?
Flink has rich keyed state primitives (ValueState, ListState, MapState) with pluggable backends including RocksDB for very large state. Spark's state store is tied to the micro-batch model and is less granular.
# Spark stateful aggregation: windowed count per account
from pyspark.sql.functions import window
result = txn_stream \
.withWatermark("txnTime", "10 minutes") \
.groupBy(
window("txnTime", "5 minutes"),
"accountId"
) \
.count()
# Flink equivalent (Java/Scala) would use:
# - KeyedProcessFunction with ValueState
# - Event-time timers for precise window firing
# - RocksDB backend for state larger than memory
# - Fine-grained state TTL per key
How does event-time handling compare?
Both support event time and watermarks, but Flink was built around them from the start with first-class event-time timers and sophisticated windowing including session windows.
# Spark: watermarks and event-time windows
result = txn_stream \
.withWatermark("txnTime", "10 minutes") \
.groupBy(
window("txnTime", "5 minutes", "1 minute"), # sliding window
"branch"
) \
.agg(F.sum("amount").alias("total"))
# Flink adds:
# - Session windows (gap-based, dynamic)
# - Event-time timers for custom logic
# - Native changelog/retract semantics for updates
How does exactly-once compare?
Both achieve end-to-end exactly-once, but through different mechanisms.
# Spark: offset checkpointing + write-ahead log + idempotent sinks
# Each micro-batch records its Kafka offsets in the checkpoint.
# If a batch fails, it is replayed from the saved offsets.
# Flink: asynchronous distributed checkpoints (Chandy-Lamport)
# Barriers flow through the dataflow, aligning state snapshots.
# Combined with transactional sinks (Kafka, JDBC) for e2e exactly-once.
What is Spark's ecosystem advantage?
Spark's streaming API is the same DataFrame/SQL API as batch. A team using Spark for ETL, SQL analytics, and MLlib gets streaming as an extension with no new engine.
# Same code works for batch and streaming!
# Batch:
batch_result = spark.read.parquet("s3://nbi-data-lake/txns/") \
.groupBy("branch").agg(F.sum("amount"))
# Streaming (same transformations):
stream_result = spark.readStream.format("kafka") \
.option("subscribe", "nbi.transactions").load() \
.groupBy("branch").agg(F.sum("amount"))
# One engine, one API, one skill set
When should you choose Flink?
Choose Flink when latency must be in the low tens of milliseconds, when you need native CDC and retract semantics, when state is huge and long-lived, or when you need complex event processing with timers.
# Flink use cases at the bank:
# - Fraud detection: alert within 10ms of suspicious transaction
# - CDC ingestion: native retract/upsert from database changelog
# - Complex event processing: detect patterns across event sequences
# - Large state: per-customer state on RocksDB, terabytes of state
When should you choose Spark Structured Streaming?
Choose Spark when you want one engine for batch, streaming, and ML, when latency of seconds is acceptable, or when you are already invested in the Spark ecosystem.
# Spark Structured Streaming use cases at the bank:
# - Near-real-time dashboard (1-second refresh is fine)
# - Streaming ETL to Delta Lake (unified with batch ETL)
# - Feature engineering for ML (same code as batch training)
# - Team already uses Spark + Databricks for everything
What should a senior engineer know?
A senior engineer chooses by requirements, not by fashion. They weigh latency, state complexity, ecosystem fit, and operational maturity.
# Decision matrix:
# Latency < 50ms required? → Flink
# Native CDC/retract needed? → Flink
# Very large stateful processing? → Flink
# Unified batch + stream + ML? → Spark
# Team already on Spark/Databricks? → Spark
# Latency of 1-5 seconds OK? → Spark
# Both requirements present? → Use both! (common at banks)
Visual Overview
Flink (true streaming): Event → Operator 1 → Operator 2 → Sink (each event flows through immediately, ms latency) State: KeyedState (Value, List, Map) on RocksDB Checkpoints: async distributed (Chandy-Lamport) Spark Structured Streaming (micro-batch): Events accumulate ──→ [Batch 1] ──→ Process ──→ Sink Events accumulate ──→ [Batch 2] ──→ Process ──→ Sink (trigger interval = latency floor) State: StateStore on disk Checkpoints: offset + WAL per micro-batch Decision: ┌─────────────────────────────────────────────┐ │ Low latency + rich state → Flink │ │ Unified batch/stream/ML → Spark │ │ Both needs? → Use both! │ └─────────────────────────────────────────────┘
Key Takeaway
Flink and Spark Structured Streaming are both distributed stream processors, but they come from opposite design philosophies. Flink is streaming-first: each event flows through the operators the instant it arrives, with native event-time semantics and rich keyed state. Spark is batch-first: it collects records into micro-batches and runs each through the Spark engine, with higher latency but a unified batch-stream API and a larger ecosystem.
The trade-offs are clear. Flink reaches single-digit millisecond latency and has richer state management with pluggable backends like RocksDB. Spark's micro-batch is typically hundreds of milliseconds to seconds but gives you one engine for batch ETL, SQL analytics, ML, and streaming, which is a major operational advantage.
At the National Bank of India we used both deliberately. For fraud detection where an alert must fire within milliseconds, we used Flink. For a near-real-time transaction dashboard where one-second latency was fine and the team already used Spark for everything else, we used Structured Streaming.
The senior engineer's approach is to choose by requirements: latency, state complexity, ecosystem fit, and operational maturity. It is not about one being universally better.
Remember one sentence: Flink is true streaming with millisecond latency and rich state; Spark is micro-batch with a unified ecosystem; choose by latency requirements and operational fit.
---