Definition
- Event time – The timestamp embedded in each record describing when the event actually occurred at its source, such as the moment a card was swiped at the ATM.
- Processing time – The wall-clock time of the machine running the operator at the instant it handles the record, requiring no timestamp extraction or watermarks.
- Ingestion time – The timestamp assigned by the Flink source operator when the record first enters the pipeline, behaving like event time downstream but ignoring true source delays.
- Watermarks – Special markers that flow with the stream in event-time mode, declaring that all events up to a certain time have been seen, enabling windows to fire correctly despite out-of-order data.
- Determinism – Event time results are deterministic and replayable because grouping depends on the data, not on when or how fast you process it.
Imagine Rajesh swipes his card at an ATM at exactly 10:00:00. Event time is the clock printed on the receipt at the ATM, the true moment it happened. Ingestion time is the time stamped by the National Bank of India's front-gate security the moment the transaction message walks through the bank's door. Processing time is whatever time the back-office clerk's wall clock shows when the clerk finally gets around to processing the slip. If the network was slow, all three can differ by minutes.
Understanding the Concepts
What is event time?
Event time is the timestamp carried inside the event itself, for example the point-of-sale time embedded in a transaction. You tell Flink how to extract it with a timestamp assigner inside a watermark strategy. Results are deterministic and replayable because the grouping depends on the data, not on when you run the job.
package com.bank.streaming;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
import java.time.Duration;
public class EventTimeDemo {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.addSource(new KafkaSource<>("nbi-transactions"))
.assignTimestampsAndWatermarks(
WatermarkStrategy.<Transaction>forBoundedOutOfOrderness(
Duration.ofSeconds(5))
.withTimestampAssigner(
(txn, ts) -> txn.getPosTimeMillis()))
.keyBy(Transaction::getAccountId)
.window(TumblingEventTimeWindows.of(Time.minutes(1)))
.sum("amount")
.print("Per-minute total");
env.execute("NBI Event-Time Settlement Report");
}
}
Per-minute total> Transaction{accountId=ACC-001, amount=125000.00}
Per-minute total> Transaction{accountId=ACC-002, amount=45000.00}
// Replaying the same data a week later produces identical results
What is processing time?
Processing time uses the operator machine's system clock. A processing-time window of one minute groups whatever records arrive in the next sixty seconds of wall-clock time. It needs no watermarks and no timestamps, so it has the lowest latency and simplest setup.
package com.bank.streaming;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
public class ProcessingTimeDemo {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
// No watermarks, no timestamp extraction
env.addSource(new KafkaSource<>("nbi-transactions"))
.keyBy(Transaction::getAccountId)
.window(TumblingProcessingTimeWindows.of(Time.minutes(1)))
.sum("amount")
.print("Ops dashboard");
// WARNING: non-deterministic results
// Replaying the same data at different speed gives different totals
env.execute("NBI Ops Dashboard");
}
}
What is ingestion time?
Ingestion time assigns the timestamp at the source operator when the record enters Flink, then behaves like event time downstream with automatic watermarks. It is more stable than processing time because all downstream operators agree on one timestamp, but it ignores the true delay between when the event happened and when it reached Flink.
package com.bank.streaming;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import java.time.Duration;
public class IngestionTimeDemo {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
// Simulate ingestion time by assigning current time at source
env.addSource(new KafkaSource<>("nbi-transactions"))
.assignTimestampsAndWatermarks(
WatermarkStrategy.<Transaction>forBoundedOutOfOrderness(
Duration.ZERO)
.withTimestampAssigner(
(txn, ts) -> System.currentTimeMillis())) // ingestion time
.keyBy(Transaction::getAccountId)
.window(TumblingEventTimeWindows.of(Time.minutes(1)))
.sum("amount")
.print("Ingestion-time total");
env.execute("NBI Ingestion-Time Demo");
}
}
How are the three time semantics different?
| Feature | Event Time | Processing Time | Ingestion Time |
|---|---|---|---|
| Source | Timestamp in the record | Operator machine clock | Flink source operator clock |
| Deterministic | Yes (replayable) | No (depends on speed) | Partially (stable within run) |
| Watermarks needed | Yes | No | Automatic |
| Latency | Higher (waits for watermark) | Lowest | Medium |
| Out-of-order handling | Yes (configurable) | No | No |
| Use case | Financial reports, audits | Rough dashboards | Rarely used in modern Flink |
// Same transaction, three different time semantics:
// Transaction: ACC-001, Rs5000, POS time = 10:00:00
// Arrived at Flink source at 10:00:02
// Processed by operator at 10:00:05
// Event time window (10:00 - 10:01): includes this transaction
// Processing time window (10:00 - 10:01): includes at 10:00:05
// Ingestion time window (10:00 - 10:01): includes at 10:00:02
// If replayed at 11:00:
// Event time: still in 10:00 window (deterministic)
// Processing time: now in 11:00 window (non-deterministic!)
// Ingestion time: depends on replay speed
How do watermarks work with event time?
Watermarks are special markers that flow with the stream saying: I believe I have seen all events up to time T. When a watermark crosses a window's end time, the window fires. The forBoundedOutOfOrderness strategy generates watermarks assuming events can be at most a certain duration late.
package com.bank.streaming;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.eventtime.WatermarkGenerator;
import org.apache.flink.api.common.eventtime.WatermarkOutput;
import org.apache.flink.api.common.eventtime.Watermark;
public class CustomWatermarkDemo {
// Custom watermark generator for NBI transactions
public static class NbiWatermarkGenerator
implements WatermarkGenerator<Transaction> {
private long maxTimestamp = Long.MIN_VALUE;
private static final long MAX_DELAY = 5000L; // 5 seconds
@Override
public void onEvent(Transaction txn, long eventTimestamp,
WatermarkOutput output) {
maxTimestamp = Math.max(maxTimestamp, eventTimestamp);
}
@Override
public void onPeriodicEmit(WatermarkOutput output) {
// Watermark = max seen timestamp minus allowed delay
output.emitWatermark(
new Watermark(maxTimestamp - MAX_DELAY));
}
}
}
Watermark flow: Event(10:00:03) ──▶ Watermark(09:59:58) Event(10:00:07) ──▶ Watermark(10:00:02) Event(10:00:10) ──▶ Watermark(10:00:05) // Window [10:00:00 - 10:01:00) fires when watermark >= 10:01:00
What are common mistakes with time semantics?
The most common mistake is using processing time for financial aggregations that must be reconciled or audited. The same data replayed at a different speed produces different window results, which breaks reconciliation against the card network.
// WRONG: processing time for settlement report
stream.keyBy(Transaction::getAccountId)
.window(TumblingProcessingTimeWindows.of(Time.minutes(1)))
.sum("amount"); // non-deterministic: replay changes results!
// CORRECT: event time for settlement report
stream.assignTimestampsAndWatermarks(
WatermarkStrategy.<Transaction>forBoundedOutOfOrderness(
Duration.ofSeconds(5))
.withTimestampAssigner((txn, ts) -> txn.getPosTimeMillis()))
.keyBy(Transaction::getAccountId)
.window(TumblingEventTimeWindows.of(Time.minutes(1)))
.sum("amount"); // deterministic: same data = same result always
When should you use each time semantic?
Use event time when the result must be correct, auditable, or replayable, because the business cares about when the thing actually happened. Use processing time when you only need a rough, low-latency operational metric and nobody will audit the exact grouping. Ingestion time is rarely the explicit choice today.
Decision guide:
Financial settlement report? --> Event time (must reconcile)
Regulatory audit feed? --> Event time (must be replayable)
Fraud detection windows? --> Event time (event order matters)
Ops health dashboard? --> Processing time (rough is fine)
"How many TPS right now?" --> Processing time (simplest)
General streaming job? --> Event time (safe default)
What should a senior engineer know about Flink time semantics?
A senior engineer should know that event time is almost always the right default for financial workloads, that watermark generation must account for source-specific delays (like mobile network latency), that idle sources can stall watermarks and need withIdleness(), and that the allowed-lateness and side-output mechanisms handle events that arrive after the watermark has passed.
package com.bank.streaming;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import java.time.Duration;
public class SeniorTimeConfigDemo {
public static WatermarkStrategy<Transaction> nbiWatermarkStrategy() {
return WatermarkStrategy
.<Transaction>forBoundedOutOfOrderness(Duration.ofSeconds(10))
.withTimestampAssigner((txn, ts) -> txn.getPosTimeMillis())
.withIdleness(Duration.ofMinutes(1)); // handle idle partitions
}
}
Visual Overview
Three Time Semantics for a Transaction
Real world Flink Source Flink Operator
────────── ──────────── ──────────────
ATM Swipe Enters Pipeline Processes Record
10:00:00 10:00:02 10:00:05
│ │ │
▼ ▼ ▼
Event Time Ingestion Time Processing Time
(in the record) (at source entry) (operator clock)
Watermark-Driven Window Firing (Event Time)
Events: [10:00:03] [10:00:01] [10:00:07] [10:00:05]
│ │ │ │
▼ ▼ ▼ ▼
Stream: ──▶ E1 ──▶ E2 ──▶ E3 ──▶ E4 ──▶ ...
Watermarks (5s allowed lateness):
After E1: W(09:59:58)
After E3: W(10:00:02)
After En: W(10:00:55) ──▶ Window [10:00-10:01) FIRES
Processing Time vs Event Time Results
Same data, replayed at different times:
Event Time: Window [10:00] = Rs125,000 (always)
Processing Time: Window [10:00] = Rs125,000 (first run)
Window [11:00] = Rs125,000 (replay at 11:00)
Results differ by window placement!
Key Takeaway
Flink gives you three ways to reason about time, and picking the right one is one of the most important decisions in a streaming job. Event time is the timestamp carried inside the record itself, describing when the event actually happened at its source. Processing time is the wall-clock time of the machine running the operator at the instant it handles the record. Ingestion time is the timestamp the source operator stamps the moment the record first enters Flink.
Event time is deterministic and replayable because the grouping depends only on the data, so reprocessing the same stream later gives identical results. But it needs watermarks to decide when a window is complete, and you must decide how long to wait for stragglers. Processing time has the lowest latency and needs no watermarks, but it is non-deterministic because the same records replayed at a different speed land in different windows. Ingestion time sits in between but is rarely the explicit choice today.
At the National Bank of India, our end-of-day settlement report absolutely had to use event time because it is reconciled against the card network by the business minute when money moved. During peak hours the queue backed up, so processing time would have placed transactions in the wrong minute and broken reconciliation. In parallel, our operations health dashboard used processing-time windows because it only answered how many transactions the servers were handling right now, and simplicity mattered more than source-true grouping.
The rule is simple: if the result must be correct, auditable, or replayable, use event time. If you only need a rough operational metric, processing time is fine.
Remember one sentence: Event time is when the event happened and gives deterministic replayable results; processing time is when the operator sees it and is fast but non-deterministic.
---