Definition
- Fixed-delay – Restarts the job up to N times with a constant delay between attempts, suitable for transient issues.
- Failure-rate – Allows unlimited restarts as long as the failure count within a sliding time interval stays below a threshold.
- Exponential-delay – Increases the wait after each consecutive failure up to a maximum, giving overloaded dependencies time to recover.
- No-restart – Fails the job immediately on the first failure, appropriate for batch jobs or when an external orchestrator handles retries.
- Checkpoint interaction – With checkpointing enabled, each restart restores from the last completed checkpoint, so it is a recovery, not data loss.
When a processing clerk at the National Bank of India stumbles and drops the whole tray, the branch manager must decide how to react. Fixed-delay is like saying "pick it up and try again, wait thirty seconds, and you get three tries." Failure-rate is like saying "keep trying as long as you do not stumble more than three times in any ten-minute window." Exponential-delay is like saying "try again after one minute, then two, then four, giving the jammed machine time to recover." No-restart is like saying "if you drop the tray even once, stop everything and call me."
Understanding the Concepts
What is a restart strategy in Flink?
When a task fails, perhaps because of an exception, a lost TaskManager, or a transient dependency error, Flink consults the configured restart strategy to decide what to do. With checkpointing enabled, a restart means Flink restores the job from the last completed checkpoint and resumes, so a restart is not a data-loss event. Without checkpointing, the default strategy is no-restart.
package com.bank.flink;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.CheckpointingMode;
public class RestartStrategyBasics {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// Checkpointing must be enabled for restarts to restore state
env.enableCheckpointing(30000, CheckpointingMode.EXACTLY_ONCE);
// Without explicit restart strategy + with checkpointing,
// Flink uses a default fixed-delay strategy
env.fromElements("txn-001", "txn-002")
.map(txn -> "Processed: " + txn)
.print();
env.execute("NBI-Restart-Basics");
}
}
How does fixed-delay restart work?
Fixed-delay is the simplest strategy. You configure it with a maximum number of attempts and a constant delay between each. If the job keeps failing past the attempt count, Flink gives up and the job fails terminally.
package com.bank.flink;
import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.api.common.time.Time;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
public class FixedDelayDemo {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(30000);
// Up to 3 restart attempts with 10-second delay between each
env.setRestartStrategy(
RestartStrategies.fixedDelayRestart(3, Time.seconds(10))
);
env.fromElements("loan-payment-001", "loan-payment-002")
.map(payment -> {
if (payment.contains("002")) {
throw new RuntimeException("Settlement API timeout");
}
return "Settled: " + payment;
})
.print();
env.execute("NBI-Fixed-Delay");
}
}
Behavior: Attempt 1: fails on loan-payment-002 → wait 10s Attempt 2: fails on loan-payment-002 → wait 10s Attempt 3: fails on loan-payment-002 → wait 10s Attempt 4: exhausted → job fails terminally
How does failure-rate restart work?
Failure-rate is more forgiving for long-running jobs. The job may restart freely as long as the failure count within a sliding interval stays below a threshold. This recognizes that an occasional failure over a long lifetime is normal, while a burst of failures signals a real problem.
package com.bank.flink;
import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.api.common.time.Time;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
public class FailureRateDemo {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(30000);
// Max 3 failures within any 5-minute window, 10s delay between restarts
env.setRestartStrategy(
RestartStrategies.failureRateRestart(
3, // max failures per interval
Time.minutes(5), // measuring interval
Time.seconds(10) // delay between restarts
)
);
env.fromElements("fraud-check-001")
.map(check -> "Scored: " + check)
.print();
env.execute("NBI-Failure-Rate");
}
}
Behavior: Hour 1: 1 failure → restart, OK (1 in 5 min) Hour 3: 1 failure → restart, OK (1 in 5 min) Hour 5: 3 failures in 2 minutes → job fails terminally (burst signals a real defect, not a transient blip)
How does exponential-delay restart work?
Exponential-delay increases the wait after each consecutive failure up to a cap and resets once the job has run healthily for a while. This avoids hammering a struggling dependency with a constant retry storm.
package com.bank.flink;
import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.api.common.time.Time;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
public class ExponentialDelayDemo {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(30000);
// Initial delay 1s, multiplier 2x, max delay 60s,
// reset after 10 minutes of healthy running
env.setRestartStrategy(
RestartStrategies.exponentialDelayRestart(
Time.seconds(1), // initial backoff
Time.seconds(60), // max backoff
2.0, // backoff multiplier
Time.minutes(10), // reset backoff threshold
0.1 // jitter factor
)
);
env.fromElements("settlement-batch-001")
.map(batch -> "Settled: " + batch)
.print();
env.execute("NBI-Exponential-Delay");
}
}
Behavior:
Failure 1: wait 1s
Failure 2: wait 2s
Failure 3: wait 4s
Failure 4: wait 8s
...
Failure N: wait 60s (capped)
After 10 min healthy: backoff resets to 1s
When should you use no-restart?
No-restart fails the job immediately on the first failure. This is appropriate for batch jobs where a failure should surface at once, or when an external orchestrator like Airflow handles retries.
package com.bank.flink;
import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
public class NoRestartDemo {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// Fail immediately: let Airflow handle retries
env.setRestartStrategy(RestartStrategies.noRestart());
env.fromElements("eod-report-001")
.map(report -> "Generated: " + report)
.print();
env.execute("NBI-EOD-Batch");
}
}
How do you compare the four strategies?
| Strategy | Retries | Delay Pattern | Best For | Risk |
|---|---|---|---|---|
| Fixed-delay | N times | Constant | Transient blips | Hammers slow deps |
| Failure-rate | Unlimited (within rate) | Constant | Long-lived streaming | None if tuned |
| Exponential-delay | Unlimited | Growing | Overloaded dependencies | Slow recovery |
| No-restart | Zero | N/A | Batch / external orchestrator | Fast failure |
How do you configure strategies globally vs per job?
You can set the strategy per job in the execution environment or globally in flink-conf.yaml. The per-job setting overrides the cluster default.
// Per-job configuration (overrides global)
env.setRestartStrategy(
RestartStrategies.failureRateRestart(3, Time.minutes(5), Time.seconds(10))
);
// Global configuration in flink-conf.yaml:
// restart-strategy: failure-rate
// restart-strategy.failure-rate.max-failures-per-interval: 3
// restart-strategy.failure-rate.failure-rate-interval: 5 min
// restart-strategy.failure-rate.delay: 10 s
What should a senior engineer know?
A senior engineer chooses the strategy based on the failure's nature and monitors restart counts to avoid masking real bugs.
- The restart strategy governs full-job or region restarts, while the failover strategy decides how much of the graph to restart.
- A job that silently restarts every minute is as broken as one that is down, so alert on restart counts in your monitoring.
- Exponential-delay prevents retry storms that pile load onto a struggling settlement API.
- Failure-rate is the safest default for long-lived streaming jobs because it tolerates rare blips but fails on sustained bursts.
- Test your restart strategy by deliberately killing a TaskManager and observing the recovery behavior.
// Monitor restarts: check Flink metrics
// flink.job.numRestarts → should be low and stable
// flink.job.lastCheckpointDuration → should be consistent
// Alert if numRestarts > 5 in 10 minutes
Visual Overview
Restart Strategy Decision
Task Failure Detected
│
▼
┌──────────────────────┐
│ Consult restart │
│ strategy config │
└──────────┬───────────┘
│
┌─────────┼─────────┬────────────┐
▼ ▼ ▼ ▼
Fixed Failure Exponential No-Restart
Delay Rate Delay
│ │ │ │
▼ ▼ ▼ ▼
attempts failures backoff FAIL
< max? < rate? < max? immediately
│ │ │
Yes→wait Yes→wait Yes→wait(growing)
No→FAIL No→FAIL reset after healthy
Exponential-Delay Backoff Timeline Time ──────────────────────────────────────────▶ F1 F2 F3 F4 F5 │ │ │ │ │ ├─1s──┤─2s────┤───4s──────┤─────8s────────┤──16s──▶ │ │ │ After 10 min healthy running: reset to 1s │
Key Takeaway
A Flink restart strategy defines whether and how the job recovers when a failure occurs. Because a restart with checkpointing enabled means Flink restores from the last completed checkpoint and resumes, the choice is about handling repeated trouble sensibly, not about data loss. Fixed-delay restarts a configured number of times with a constant wait, which suits transient issues but can hammer a slow-recovering dependency. Failure-rate lets the job restart freely as long as failures stay below a threshold within a sliding interval, making it ideal for long-lived streaming jobs.
Exponential-delay increases the backoff after each consecutive failure up to a maximum, which is exactly what you want when failures come from an overloaded external system. No-restart fails immediately, fitting batch jobs where an external orchestrator owns retries. You configure these per job through the RestartStrategies API or globally in flink-conf.yaml.
At the National Bank of India, our long-lived fraud-scoring stream used failure-rate so that a brief TaskManager loss recovered automatically from the last checkpoint, but a burst of failures paged the on-call engineer. A settlement API writer used exponential-delay so it backed off with increasing delays instead of piling load onto a struggling dependency. We also alerted on restart counts, because a job quietly restarting every minute is just as broken as one that is down.
Remember one sentence: choose fixed-delay for transient blips, failure-rate for long-lived streams, exponential-delay for overloaded dependencies, and no-restart for batch jobs, and always monitor restart counts so recovery does not mask a real bug.
---