What is Flink's failover strategy | Data Engineering

Q: What should a senior engineer know?

Generally leave the strategy at the default region. Choose full only in rare cases where region boundaries cannot guarantee a consistent recovery. Region failover depends on checkpointing to restore state consistently. In streaming jobs that form one big region, there is no practical difference, but

Definition

Region failover restarts only the failed pipelined region and its dependents, not the whole job, which is why it is the default and more efficient than full restart.

Failover strategy – Decides how much of the running job graph must be restarted when a task fails, distinct from the restart strategy that decides how often.
Region failover – Restarts only the failed pipelined region and regions connected by the failure, leaving unrelated regions running.
Pipelined region – A set of tasks connected by streaming, record-by-record data exchanges, bounded by blocking exchanges.
Full restart – Cancels and restarts every task in the entire job, simple but wasteful when independent regions exist.
Default – Region failover has been the default since Flink 1.9 because it limits the blast radius of a single failure.

The National Bank of India runs several independent processing lines in the same hall. If one clerk on one line stumbles, the full-restart approach evacuates the entire hall and restarts every line from scratch, even the lines that were working perfectly. The region-failover approach tells only the affected line to stop and restart from its last saved state, while the other independent lines keep working undisturbed.

Understanding the Concepts

What is a failover strategy?

When a task fails, Flink must restart tasks to recover. The failover strategy, set by jobmanager.execution.failover-strategy, controls the scope of the restart. It is distinct from the restart strategy, which controls the frequency and timing of retries.

JAVA

package com.bank.flink;

import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.JobManagerOptions;

public class FailoverStrategyConfig {
    public static void main(String[] args) {
        Configuration config = new Configuration();
        // Region failover is the default since Flink 1.9
        config.setString("jobmanager.execution.failover-strategy", "region");

        // Alternative: full restart (rarely used)
        // config.setString("jobmanager.execution.failover-strategy", "full");

        System.out.println("Failover strategy: "
            + config.getString("jobmanager.execution.failover-strategy", "region"));
    }
}

What is a pipelined region?

A pipelined region is a maximal set of tasks connected by pipelined (streaming) data exchanges. Within a region, tasks continuously produce and consume each other's records, so they are mutually dependent. Regions are bounded by blocking exchanges, where one stage fully produces its output before the next consumes it.

DIAGRAM

Streaming job (one pipelined region):
  Source[0..3] ──pipelined──▶ Filter[0..3] ──pipelined──▶ Sink[0..3]
  All tasks form ONE region because all exchanges are pipelined.

Batch job (multiple regions):
  Source[0..3] ──blocking──▶ Aggregate[0..3] ──blocking──▶ Sink[0..3]
  Three separate regions because blocking exchanges bound them.

JAVA

package com.bank.flink;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.datastream.DataStream;

public class PipelinedRegionDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(4);

        // All operators form one pipelined region in streaming mode
        DataStream<String> transactions = env
            .socketTextStream("kafka.bank.internal", 9092);

        transactions
            .filter(txn -> txn.contains("DEPOSIT"))
            .map(txn -> "Processed: " + txn)
            .print();

        env.execute("NBI-Single-Region");
    }
}

How does region failover work?

When a task in a region fails, Flink restarts that entire region because the tasks within it are mutually dependent. Regions that were not involved keep running. If a failed region had consumed data from an upstream region whose intermediate result is no longer available, that upstream region may also need to restart, but truly independent regions are untouched.

DIAGRAM

Job with two independent pipelines:

  Pipeline A: Source_A → Filter_A → Sink_A  (Region 1)
  Pipeline B: Source_B → Agg_B    → Sink_B  (Region 2)

  If Filter_A fails:
    Region 1 restarts from checkpoint → Source_A, Filter_A, Sink_A restart
    Region 2 keeps running → Source_B, Agg_B, Sink_B unaffected

JAVA

package com.bank.flink;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.datastream.DataStream;

public class TwoPipelineDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.enableCheckpointing(30000);

        // Pipeline A: fraud detection
        DataStream<String> fraudStream = env
            .socketTextStream("kafka.bank.internal", 9092)
            .filter(txn -> txn.contains("SUSPECT"))
            .map(txn -> "ALERT: " + txn);
        fraudStream.print("fraud");

        // Pipeline B: interest calculation (independent)
        DataStream<String> interestStream = env
            .socketTextStream("kafka.bank.internal", 9093)
            .map(acct -> "Interest: " + acct);
        interestStream.print("interest");

        // If fraud pipeline fails, interest pipeline keeps running
        env.execute("NBI-Two-Pipelines");
    }
}

How does full restart compare?

Full restart cancels and restarts every task in the entire job and restores all of them from the last checkpoint. It is simple and always correct, but it wastes time and work because even tasks far from the failure are needlessly restarted.

TEXT

Same job with full restart:

  If Filter_A fails:
    BOTH Region 1 AND Region 2 restart
    Source_B, Agg_B, Sink_B are needlessly stopped and restored
    All work done by Pipeline B since the last checkpoint is repeated

Why is region failover the default?

Region failover is the default because it limits the blast radius of a single failure. The real advantage appears in jobs with multiple independent pipelines or in batch and hybrid jobs with blocking exchanges, where a failure in one branch leaves the other branches running.

In a streaming job that is one big pipelined region, region failover and full restart restart the same tasks.
In multi-pipeline or batch jobs, region failover restarts one region out of many instead of all of them.
Blocking exchanges and batch shuffle services make regions smaller, amplifying the benefit.

JAVA

// The benefit is proportional to the number of independent regions:
// 1 region:  region failover == full restart (no benefit)
// 3 regions: region failover restarts 1/3 instead of 3/3
// 10 regions: region failover restarts 1/10 instead of 10/10

How do failover and restart strategies interact?

The failover strategy decides scope (what to restart), while the restart strategy decides frequency and timing (how often and with what delay). They work together.

DIAGRAM

Failover strategy: region   → restart only the failed region
Restart strategy: failure-rate(3 per 5 min, 10s delay)

  Task fails → region failover identifies the affected region
             → restart strategy decides: attempt or give up?
             → if within rate: wait 10s, restore region from checkpoint
             → if rate exceeded: job fails terminally

What are the edge cases?

Region failover can cascade when a failed region's input comes from an upstream region whose output has been lost. The upstream region must also restart to reproduce the data.

DIAGRAM

Cascading failover:
  Region 1 (batch stage) ──blocking──▶ Region 2 (streaming stage)

  If Region 2 fails and Region 1's output has been cleaned up:
    Region 1 must also restart to reproduce the blocking output
    But Region 3 (independent) still runs undisturbed

What should a senior engineer know?

Generally leave the strategy at the default region. Choose full only in rare cases where region boundaries cannot guarantee a consistent recovery.
Region failover depends on checkpointing to restore state consistently.
In streaming jobs that form one big region, there is no practical difference, but the model is still correct.
Smaller regions, created by blocking exchanges, make failover cheaper.
Monitor which regions restart and how often to detect systemic issues in specific pipelines.

JAVA

// Check failover behavior in Flink Web UI:
// Jobs → select job → Exceptions tab
// Shows which TaskManager failed and which tasks restarted
// Region failover shows only the affected subtasks restarting

Visual Overview

DIAGRAM

  Failover Strategy Comparison

  Full Restart:
  ┌─────────────────────────────────────────┐
  │  Region 1     Region 2     Region 3     │
  │  [Source_A]   [Source_B]   [Source_C]   │
  │  [Filter_A]   [Agg_B]     [Join_C]     │
  │  [Sink_A]     [Sink_B]    [Sink_C]     │
  │                                         │
  │  Filter_A fails → ALL regions restart   │
  │  ████████████████████████████████████   │
  └─────────────────────────────────────────┘

  Region Failover:
  ┌─────────────────────────────────────────┐
  │  Region 1     Region 2     Region 3     │
  │  [Source_A]   [Source_B]   [Source_C]   │
  │  [Filter_A]   [Agg_B]     [Join_C]     │
  │  [Sink_A]     [Sink_B]    [Sink_C]     │
  │                                         │
  │  Filter_A fails → only Region 1 restarts│
  │  ████████████   (running)   (running)   │
  └─────────────────────────────────────────┘

DIAGRAM

  Pipelined Region Boundaries

  Streaming job (one region):
  ┌──────────────────────────────────────┐
  │  Source ──pipe──▶ Filter ──pipe──▶ Sink │
  └──────────────────────────────────────┘

  Batch job (three regions):
  ┌────────┐  blocking  ┌────────┐  blocking  ┌──────┐
  │ Source  │──────────▶│  Agg   │──────────▶│ Sink  │
  │Region 1│           │Region 2│           │Region 3│
  └────────┘           └────────┘           └────────┘

Key Takeaway

A Flink failover strategy decides how much of the running job graph has to restart when a task fails, distinct from the restart strategy that decides how often to retry. There are two failover strategies. Full restart cancels and restarts every task in the whole job, which is simple but wastes work on unrelated tasks. Region failover, the default since Flink 1.9, restarts only the failed pipelined region and any regions whose intermediate results were lost, leaving independent regions running.

The real advantage appears in jobs with multiple independent pipelines or in batch jobs with blocking exchanges. A failure in one pipeline restarts only that pipeline's region while the others keep working. In a streaming job that forms one big pipelined region, both strategies restart the same tasks, but region failover is still the correct default.

At the National Bank of India we hosted three independent pipelines in one Flink application: fraud scoring, interest calculation, and regulatory reporting. With region failover, when the interest pipeline hit a failure, only the interest region restarted while fraud detection and reporting kept running, which was critical because stopping fraud detection during an unrelated recovery would have been unacceptable.

Remember one sentence: region failover restarts only the failed pipelined region and its dependents, leaving unrelated pipelines running, which is why it is the default and more efficient than restarting the entire job.

---