Definition
Batch processing and stream processing are two models for handling data, differing in data shape and latency.
- Batch processing – operates on a bounded dataset collected over a window, processed all at once on a schedule; optimized for throughput and completeness.
- Stream processing – operates on an unbounded, continuously arriving sequence of events, processing each record as it arrives; optimized for low latency and freshness.
- Bounded vs unbounded – batch data has a known start and end; streaming data never stops arriving.
- Event time vs processing time – stream processing must distinguish when an event happened from when it is processed.
- Windowing – stream processing groups unbounded events into finite windows (tumbling, sliding, session) for aggregation.
Think of National Bank of India at the end of the day. Batch processing is the nightly settlement run: the bank waits until midnight, gathers every transaction Rajesh and Priya made all day, and processes the whole pile in one scheduled job. It is efficient because it handles millions of rows together, but you only see the result the next morning. Stream processing is the fraud alert that fires the instant Rajesh swipes his card in a strange city: the bank cannot wait until midnight, it must react within milliseconds while the event is still happening.
Understanding the Concepts
What is batch processing?
Batch processing assumes the data is bounded, meaning it has a known start and end. You run a job over a file, a day's worth of rows, or a table snapshot, and when the last record is processed the job finishes. Classic banking batch jobs are end-of-day interest accrual, nightly reconciliation, and monthly statement generation.
package com.bank.batch;
import java.math.BigDecimal;
import java.util.List;
public class NightlySettlementBatch {
public static void main(String[] args) {
// Bounded dataset: all transactions from today
List<String> todaysTransactions = loadFromDatabase("2024-01-15");
System.out.println("Loaded " + todaysTransactions.size() + " transactions");
BigDecimal totalSettled = BigDecimal.ZERO;
for (String txn : todaysTransactions) {
totalSettled = totalSettled.add(new BigDecimal("1000.00")); // simplified
}
System.out.println("Total settled: " + totalSettled);
System.out.println("Job complete -- results available next morning");
}
static List<String> loadFromDatabase(String date) {
return List.of("TXN-1", "TXN-2", "TXN-3"); // simplified
}
}
Loaded 3 transactions
Total settled: 3000.00
Job complete -- results available next morning
What is stream processing?
Stream processing assumes the data is unbounded: events keep arriving forever and there is no natural end. A consumer polls Kafka continuously and processes each record within milliseconds. Because events never stop, the program runs forever, and correctness depends on handling late events, out-of-order events, and windowing.
package com.bank.stream;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.List;
import java.util.Properties;
public class FraudScorerStream {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "fraud-scorer");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("card.authorizations"));
// Unbounded: runs forever, processes each event immediately
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
double riskScore = scoreFraud(record.value());
if (riskScore > 0.8) {
System.out.println("ALERT: High risk for " + record.key()
+ " score=" + riskScore);
}
}
}
}
}
static double scoreFraud(String txn) {
return 0.85; // simplified
}
}
ALERT: High risk for SAV2001 score=0.85
How do batch and stream differ in handling time and state?
Batch computes aggregates over a complete, frozen dataset, so a daily total is exact. Stream must compute aggregates over windows of an endless feed, introducing event time vs processing time, watermarks to decide when a window is complete, and incremental state stores.
| Aspect | Batch | Stream |
|---|---|---|
| Data shape | Bounded (start + end) | Unbounded (never ends) |
| Latency | Hours (scheduled) | Milliseconds (continuous) |
| State | Full dataset in memory/disk | Incremental state stores |
| Time model | Processing time only | Event time + watermarks |
| Late events | Not an issue (all data present) | Must handle with allowed lateness |
| Complexity | Simple (one pass over data) | Complex (windowing, ordering, state) |
| Banking example | Nightly reconciliation | Real-time fraud detection |
# Batch: Spark job scheduled at midnight
spark-submit --class com.bank.batch.NightlyReconciliation reconciliation.jar
# Runs once, finishes, result: exact daily totals
# Stream: Kafka Streams application running 24/7
java -jar fraud-scorer.jar
# Runs forever, processes each event in milliseconds
When do you choose batch?
Choose batch when the data is naturally bounded, the result is needed periodically, and throughput matters more than freshness. Examples: nightly settlement, regulatory reports, data warehouse loads, monthly statements.
package com.bank.batch;
import java.math.BigDecimal;
public class MonthlyStatementBatch {
public static void main(String[] args) {
// Bounded: all transactions for account SAV2001 in January
BigDecimal openingBalance = new BigDecimal("100000.00");
BigDecimal totalDebits = new BigDecimal("35000.00");
BigDecimal totalCredits = new BigDecimal("20000.00");
BigDecimal closingBalance = openingBalance.subtract(totalDebits).add(totalCredits);
System.out.println("Account: SAV2001 (Rajesh)");
System.out.println("Opening: " + openingBalance);
System.out.println("Debits: " + totalDebits);
System.out.println("Credits: " + totalCredits);
System.out.println("Closing: " + closingBalance);
}
}
Account: SAV2001 (Rajesh)
Opening: 100000.00
Debits: 35000.00
Credits: 20000.00
Closing: 85000.00
When do you choose stream?
Choose stream when events arrive continuously and the business value decays in seconds: fraud detection, real-time balance updates, live dashboards, instant notifications.
package com.bank.stream;
import java.math.BigDecimal;
public class RealTimeBalanceUpdate {
private BigDecimal balance = new BigDecimal("100000.00");
public void processTransaction(String accountId, BigDecimal amount) {
balance = balance.subtract(amount);
System.out.println("Account " + accountId + " new balance: " + balance);
if (balance.compareTo(new BigDecimal("10000.00")) < 0) {
System.out.println("LOW BALANCE ALERT for " + accountId);
}
}
public static void main(String[] args) {
RealTimeBalanceUpdate updater = new RealTimeBalanceUpdate();
updater.processTransaction("SAV2001", new BigDecimal("95000.00"));
}
}
Account SAV2001 new balance: 5000.00
LOW BALANCE ALERT for SAV2001
How do most banks run both paths?
A common pattern is a fast streaming path for immediate reactions and a slower batch path for the authoritative, reconciled end-of-day numbers. This is the essence of the Lambda architecture.
# Dual path over the same Kafka topic:
# Stream path: fraud scorer reads card.authorizations in real-time
# Batch path: nightly Spark job reads card.authorizations for daily totals
# Both consume from the same topic but serve different latency needs
What are common mistakes when choosing between batch and stream?
The most common mistake is building a streaming pipeline for a use case that only needs daily results, adding complexity for no latency benefit. The reverse mistake is using a nightly batch for something that needs sub-second reaction.
# WRONG: streaming pipeline for a monthly report
# Adds windowing, state management, late-event handling complexity
# A simple batch job run once a month would be far simpler
# WRONG: nightly batch for fraud detection
# Rajesh's stolen card was used 1000 times before the batch ran at midnight
# A streaming consumer would have blocked the card in milliseconds
What should a senior engineer know?
The two models are converging: micro-batch engines like Spark Structured Streaming process streams as tiny batches, and Kafka Streams processes batches as a bounded stream. The real decision is about required latency and the shape of the input. Most mature banks run both paths and reconcile them, using the stream for speed and the batch for authority.
# Kafka Streams: process unbounded stream with exactly-once
java -jar settlement-streams.jar
# Spark Structured Streaming: micro-batch over Kafka topic
spark-submit --class com.bank.spark.StreamingReconciliation streaming.jar
# Processes in 100ms micro-batches -- almost real-time
Visual Overview
Batch vs Stream Processing Batch (nightly settlement): ┌──────────────────────────────────────────┐ │ Data collected all day │ │ [TXN-1] [TXN-2] ... [TXN-1M] │ │ │ │ │ ▼ Midnight job │ │ ┌────────────────────────┐ │ │ │ Process entire dataset │ │ │ │ Result: exact daily │ │ │ │ totals for Rajesh/Priya │ │ │ └────────────────────────┘ │ │ Latency: hours. Throughput: high. │ └──────────────────────────────────────────┘ Stream (fraud detection): ┌──────────────────────────────────────────┐ │ Events arrive continuously │ │ [TXN] → score() → alert? │ │ [TXN] → score() → alert? │ │ [TXN] → score() → BLOCK CARD! │ │ ... never stops ... │ │ Latency: ms. Throughput: per-record. │ └──────────────────────────────────────────┘ Dual path (Lambda): ┌────────────┐ │ Kafka Topic │───→ Stream: fraud alerts (ms) │ card.auth │───→ Batch: daily totals (hours) └────────────┘
Key Takeaway
Batch processing and stream processing differ in whether the data is bounded and how quickly you need the result. Batch works on a bounded dataset processed all at once on a schedule, optimized for throughput, perfect for nightly settlement, regulatory reports, and statement generation. Stream works on an unbounded feed processed continuously, optimized for low latency, the right choice for fraud detection, real-time balance updates, and instant notifications.
At National Bank of India, Pankaj's team runs both paths over the same transaction feed. A streaming consumer scores every card authorization as it arrives and can block Rajesh's stolen card within a second. A nightly batch job drains the whole day's transactions and produces the exact per-account totals posted to the general ledger the next morning. The streaming path gives immediate reactions; the batch path gives reconciled, authoritative figures.
The senior nuance is that these models are converging. Micro-batch engines like Spark Structured Streaming treat a stream as a sequence of tiny batches, and Kafka Streams treats a batch as a bounded stream. The real decision is driven by latency requirements and input shape. Most mature banks run a fast streaming path for instant decisions alongside a slower batch path for the trusted end-of-day truth.
Remember one sentence: batch processes bounded data on a schedule for throughput, stream processes unbounded data continuously for freshness, and most banks run both.
---