Definition
- Batch processing : runs a job over a large, bounded dataset collected over a period, optimized for throughput; Apache Spark is the dominant engine.
- Stream processing : computes results incrementally over an unbounded, continuous flow of events with low latency; Apache Flink and Kafka Streams are common engines.
- Windowing : grouping events by time intervals (tumbling, hopping, sliding) for stream aggregation.
- Watermark : a marker that tracks event-time progress so the system knows when a window can be finalized.
- Checkpointing : periodically saving stream state for fault-tolerant recovery without reprocessing everything.
At the end of each day, National Bank of India gathers every transaction and runs one big job to compute interest and reports : that is batch processing, done on a fixed pile of data overnight. But the fraud team needs to catch a suspicious transaction within seconds, so they watch the live stream and react immediately : that is stream processing. Batch is the overnight accountant; streaming is the real-time security guard.
Understanding the Concepts
What is batch processing?
Batch processing takes a bounded dataset and runs a job over all of it. It optimizes for throughput: it can crunch enormous volumes because it sees the whole dataset. The latency is high because results are available only after the job completes.
package com.bank.processing;
// Spark-style: process a full day's transactions at once
public class BatchInterestCalculation {
public void run() {
Dataset<Row> txns = spark.read().parquet("/data/transactions/2026-10-31");
Dataset<Row> interest = txns.groupBy("accountId")
.agg(sum("amount").multiply(lit(0.0001)).as("dailyInterest"));
interest.write().parquet("/data/interest/2026-10-31");
}
}
What is stream processing?
Stream processing consumes an unbounded flow of events and computes results incrementally as each event arrives. It maintains state over time, uses windows to group events, and must handle late data.
package com.bank.processing;
// Kafka Streams-style: react to each transaction within seconds
public class StreamFraudDetection {
public void build(StreamsBuilder builder) {
builder.stream("transactions")
.filter((key, txn) -> isSuspicious(txn)) // react immediately
.to("fraud.alerts"); // emit alert
}
}
When should you use batch?
Use batch when you need to analyze large historical datasets, when results can wait minutes or hours, and when correctness over the complete dataset matters: end-of-day interest, regulatory reports, reconciliation.
When should you use streaming?
Use streaming when you need to react within seconds: fraud detection, real-time dashboards, live alerts, and event-driven triggers.
How do they compare?
| Aspect | Batch | Stream |
|---|---|---|
| Data | Bounded (fixed dataset) | Unbounded (continuous flow) |
| Latency | High (job must complete) | Low (seconds) |
| Throughput | Very high | High |
| Correctness | Exact (sees all data) | Handles late data with watermarks |
| Engine | Spark | Flink, Kafka Streams |
| Banking use | Interest, reports | Fraud detection, alerts |
What is windowing in stream processing?
Windowing groups events by time intervals. A tumbling window is non-overlapping fixed intervals. A hopping window overlaps. A sliding window moves continuously.
package com.bank.processing;
public class WindowingExample {
// 5-minute tumbling window: count transactions per account
// Hopping window: 5 min window, 1 min hop
// Used for: "transactions in last 5 minutes" dashboards
}
What about event-time and watermarks?
Event-time is when the event actually happened; processing-time is when the system processes it. Watermarks track event-time progress so the system knows when to finalize a window, allowing bounded lateness for late arrivals.
How do batch and stream coexist?
Modern architectures run both. Streaming handles real-time reactions while batch handles periodic recomputation. At National Bank of India, a fraud alert fires in seconds on the stream, while overnight a Spark batch job recomputes complete, correct interest totals.
What should a senior engineer know?
Batch is increasingly understood as a special case of streaming over a bounded source. Engines are converging toward this unified model. The key decision is matching each workload to the right path rather than choosing one paradigm exclusively.
Visual Overview
Batch vs Stream Processing
┌─────────────────────────────────────────────────┐
│ Data Sources │
│ Transactions, Events, Logs │
└──────────────┬──────────────────┬────────────────┘
│ │
┌───────▼───────┐ ┌──────▼────────┐
│ Stream Path │ │ Batch Path │
│ (Flink/KS) │ │ (Spark) │
│ │ │ │
│ Fraud detect │ │ Interest calc │
│ Live alerts │ │ Reconciliation│
│ RT dashboard │ │ Reg. reports │
│ │ │ │
│ Latency: secs │ │ Latency: hours│
│ Continuous │ │ Periodic │
└───────┬───────┘ └──────┬────────┘
│ │
┌───────▼───────┐ ┌──────▼────────┐
│ Real-time │ │ Data warehouse│
│ alerts topic │ │ / reports │
└───────────────┘ └───────────────┘
Stream windowing:
Tumbling: |──5min──|──5min──|──5min──| (non-overlapping)
Hopping: |──5min──|
|──5min──|
|──5min──| (overlapping)
Key Takeaway
Batch and stream processing are two paradigms for computing over data, and I choose by latency need and data boundedness. Batch processing runs a job over a bounded dataset in one high-throughput pass, optimized for throughput, with results available only after completion. Spark is the dominant engine, and in banking I use batch for end-of-day interest, regulatory reports, and reconciliation.
Stream processing computes incrementally over an unbounded flow with low latency as each event arrives. Flink and Kafka Streams are the common engines. In banking I use streaming for fraud detection, real-time dashboards, and live alerts, where reacting within seconds is the entire value.
The two are complements, not competitors. Modern architectures run both: the streaming path handles real-time reactions while the batch path handles periodic, authoritative recomputation. At National Bank of India, a fraud alert fires in seconds on the stream, while overnight a Spark batch job recomputes complete interest totals and reconciles.
The senior insight is that batch is increasingly understood as a special case of streaming over a bounded source. The key decision is matching each workload to the right path rather than choosing one paradigm exclusively.
Remember one sentence: use streaming for low-latency reactions like fraud detection and batch for high-throughput historical analysis like interest calculation, and run both together in a unified architecture.
---