Definition
- Flaky test – Passes and fails intermittently without any code change, making results unreliable.
- Common causes – Poor synchronization (hard sleeps), unstable locators, shared data, environment instability, race conditions, network dependencies, order dependence, inadequate cleanup.
- Diagnosis first – Measure flake rate, pull traces from failures, classify by cause, then fix the root.
- Blanket retrying – Automatically re-running failures until they pass. Masks the cause, hides real bugs, slows the suite, erodes trust.
- Acceptable retries – Only as a visible, tracked, temporary safety net while the root is being fixed.
A flaky test is like a smoke detector at the National Bank of India that beeps randomly. Staff stop trusting it, and the day there is a real fire they ignore it. Wrapping tape over it so it only counts if it beeps three times in a row is blanket retrying: it hides the problem and might silence a real fire. The right response is to find the loose wire and fix it.
Understanding the Concepts
What makes a test flaky?
A flaky test produces different results across runs with no code or test change. The intermittent nature makes it uniquely damaging because it erodes trust.
Run 1: PASS
Run 2: PASS
Run 3: FAIL <- same code, same test
Run 4: PASS
Run 5: FAIL
Developer: "It's just flaky, ignore it."
Later, Run 6: FAIL <- this time it's a REAL bug
Developer: "It's just flaky, ignore it." <- regression ships
How do you diagnose the cause?
Gather evidence from the failed runs: trace, screenshot, video, logs. Classify failures by cause.
Cause Classification: Cause │ Signature │ Fix ───────────────────────┼─────────────────────────┼────────────────── Hard-coded sleep │ Timeout after fixed wait │ Auto-waiting Unstable locator │ Element not found │ Role/label locator Shared data │ Wrong value, parallel │ Unique keys/worker Environment instability│ Timeout, slow response │ Stabilize env Race condition │ Intermittent state error │ Wait on readiness Network dependency │ External service error │ Mock/stub Order dependency │ Fails when reordered │ Self-contained Inadequate cleanup │ Stale data from prior run│ Reliable teardown
How do you fix hard-coded sleeps?
Replace fixed waits with auto-waiting assertions that wait on the real condition.
// WRONG: hard-coded sleep (most common flakiness cause)
await page.getByRole('button', { name: 'Transfer' }).click();
await page.waitForTimeout(3000); // Guess: 3 seconds enough?
const text = await page.getByTestId('result').textContent();
// CORRECT: auto-waiting assertion
await page.getByRole('button', { name: 'Transfer' }).click();
await expect(
page.getByText('Transfer Successful')
).toBeVisible(); // Waits until visible or timeout
How do you fix shared data interference?
Isolate data per test or per worker with unique keys.
// WRONG: all tests use the same account
const account = 'ACC-001'; // Worker 1 and Worker 2 both touch this
// CORRECT: unique account per test
const account = `ACC-${testInfo.workerIndex}-${Date.now()}`;
await apiHelper.createAccount(account, 50000);
How do you fix unstable locators?
Replace brittle CSS with role-based or label-based locators.
// WRONG: brittle CSS locator
await page.locator('.btn-primary.transfer-submit').click();
// Breaks when CSS class is renamed
// CORRECT: role-based locator
await page.getByRole('button', { name: 'Transfer' }).click();
// Survives CSS refactors
How do you fix external network dependencies?
Mock or stub external calls to make the test deterministic.
// Mock a flaky third-party API
await page.route('**/api/external/credit-check', (route) => {
route.fulfill({
status: 200,
contentType: 'application/json',
body: JSON.stringify({ approved: true, score: 750 }),
});
});
Why is blanket retrying a bad fix?
Four concrete harms from blanket retrying.
1. MASKS ROOT CAUSE
The test still has the defect. It persists and spreads.
2. HIDES REAL BUGS
Some intermittent failures are real product bugs.
Retrying to green silently passes over them.
3. SLOWS THE SUITE
Every retried test re-runs, inflating wall-clock time.
4. ERODES TRUST
"The suite only passes after retries."
-> People stop believing failures are real.
-> A genuine regression slips through.
// WRONG: blanket retry in config
export default defineConfig({
retries: 3, // Every failing test retries 3 times
// Hides: real bugs, flaky tests, data issues
});
// ACCEPTABLE: temporary, tracked, while fixing root cause
export default defineConfig({
retries: process.env.CI ? 1 : 0, // 1 retry in CI only
// Track retry rate as a metric
// Target: 0 retries needed
});
What is the right approach to flakiness?
- Detect : Track flake rate across runs. A test that fails more than 2% of runs is flaky.
- Quarantine : Move known-flaky tests out of the blocking suite so they do not erode trust.
- Diagnose : Pull traces from failures, classify by cause.
- Fix : Address the root cause (sleep, locator, data, etc.).
- Restore : Move the fixed test back into the blocking suite.
- Prevent : Use auto-waiting, stable locators, isolated data, and mocked externals by default.
What are common interview follow-up questions?
- "Is any retry acceptable?" Yes, a single retry as a temporary safety net while fixing the root, with tracking.
- "How do you measure flakiness?" Track the ratio of failures to runs for each test. Dashboard the flake rate.
- "What is the most common cause?" Hard-coded sleeps, followed by shared test data.
Visual Overview
Flaky Test Resolution Process
─────────────────────────────
┌────────────────┐
│ 1. DETECT │ Track flake rate per test
│ flakiness │ > 2% failure rate = flaky
└───────┬────────┘
│
┌───────▼────────┐
│ 2. QUARANTINE │ Move out of blocking suite
│ the test │ So it does not erode trust
└───────┬────────┘
│
┌───────▼────────┐
│ 3. DIAGNOSE │ Pull traces from failures
│ root cause │ Classify: sleep? data? locator?
└───────┬────────┘
│
┌───────▼────────┐
│ 4. FIX the │ Auto-waiting, unique data,
│ root cause │ stable locators, mocks
└───────┬────────┘
│
┌───────▼────────┐
│ 5. RESTORE │ Move back to blocking suite
│ the test │ Verified deterministic
└───────┬────────┘
│
┌───────▼────────┐
│ 6. PREVENT │ Standards: auto-wait, isolate,
│ recurrence │ mock externals by default
└────────────────┘
Blanket Retrying: Why It Is Bad ──────────────────────────────── Blanket retry: Correct approach: ┌──────────────┐ ┌──────────────┐ │ Test fails │ │ Test fails │ │ Retry 1 fails│ │ Diagnose root│ │ Retry 2 PASS │ │ Fix the cause│ │ Report: PASS │ │ Test passes │ │ │ │ deterministic│ │ Root cause: │ │ │ │ STILL THERE │ │ Root cause: │ │ Real bugs: │ │ ELIMINATED │ │ HIDDEN │ │ Trust: HIGH │ │ Trust: LOW │ │ │ └──────────────┘ └──────────────┘
Key Takeaway
A flaky test passes and fails intermittently without code changes, and I treat flakiness as a defect to diagnose and fix, not to paper over. My process is: detect by tracking flake rates, quarantine so it does not erode trust, diagnose by pulling traces and classifying the cause, fix the root cause, restore to the blocking suite, and prevent recurrence with standards.
The common causes and their fixes: hard-coded sleeps become auto-waiting assertions, unstable locators become role-based locators, shared data becomes per-worker unique keys, external dependencies become mocks, and order dependencies become self-contained tests.
Blanket retrying is a bad fix because it masks the root cause, can hide real product bugs, slows the suite, and erodes trust. Limited retries are acceptable only as a visible, tracked temporary safety net while the root is being fixed.
At the National Bank of India a transfer test flaked because it used a fixed sleep before checking an asynchronously updated balance. I pulled the trace, saw the balance was still loading, and replaced the sleep with an auto-waiting assertion. It became deterministic. The team trusted the suite again.
Remember one sentence: flakiness is a defect to diagnose and fix at the root, not to mask with retries, because a suite people do not trust is worth nothing.
---