Which test cases to automate vs leave manual, and why "automate everything" is a mistake

• Good automation candidates – Repetitive tests that run on every build, stable features with rarely changing behavior, business-critical paths where a regression is catastrophic, time-consuming manual checks, data-driven tests with many input rows, and regression tests that protect existing behavior.
• Poor automation candidates – Exploratory testing where value is human adaptability, one-off checks that run once, features with volatile requirements that change every sprint, and tests requiring human judgement like visual aesthetics or usability.
• "Automate everything" trap – Automating unstable features creates brittle tests that break on legitimate changes, inflating maintenance cost until the team fixes tests instead of finding bugs, and eroding trust when failures are ignored.

What makes a test a good automation candidate?

Automation has an up-front cost to write and an ongoing cost to maintain, and it returns value each time it runs without human effort. So the decision is economic: automate where the lifetime savings exceed the lifetime cost.

The strongest candidates share traits:

• Repetitive : Tests that run on every build, such as smoke and regression, amortize their cost over thousands of runs.
• Stable : Tests against features whose behavior and interface rarely change stay green for the right reasons.
• Business-critical : Paths like login, transfer, and payment justify automation because a regression is catastrophic.
• Time-consuming : Manual tests that take hours become cheap automated loops.
• Data-driven : Same steps with many input rows are naturally automatable.
• Regression-suitable : Tests that exist precisely to protect existing behavior are the backbone of the suite.

// Good candidate: stable, data-driven, critical
package com.bank.service;

import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvFileSource;
import java.math.BigDecimal;
import static org.junit.jupiter.api.Assertions.assertEquals;

public class InterestCalculationRegressionTest {

    private final InterestService service = new InterestService();

    @ParameterizedTest(name = "Balance={0} Rate={1} Days={2} -> {3}")
    @CsvFileSource(resources = "/test-data/interest-cases.csv")
    void interestCalculation(String balance, String rate,
                             int days, String expected) {
        BigDecimal result = service.calculate(
            new BigDecimal(balance),
            new BigDecimal(rate),
            days
        );
        assertEquals(new BigDecimal(expected), result);
    }
}

// test-data/interest-cases.csv - 200 rows, runs in seconds
balance,rate,days,expected
0.00,7.0,365,0.00
1.00,7.0,365,0.07
100000.00,7.0,365,7000.00
100000.00,7.0,1,19.18
5000000.00,4.5,365,225000.00

What makes a test a poor automation candidate?

Poor candidates share opposite traits:

• Exploratory testing : Unscripted investigation where a skilled tester follows hunches. The value is human adaptability that automation cannot replicate.
• One-off checks : A check that will run once for a specific data migration rarely earns back the automation cost.
• Volatile features : When design and rules change every sprint, automated tests break on every legitimate change, so maintenance swamps the benefit.
• Judgement-based checks : Whether a layout looks trustworthy or an error message reads clearly cannot be meaningfully automated.

// Poor candidate: volatile feature changing weekly
// This test breaks every sprint and costs more to maintain than it saves
test('Rewards dashboard shows correct points', async ({ page }) => {
  // Week 1: points are in the sidebar
  // Week 2: points moved to the header
  // Week 3: points are now a modal
  // Week 4: points replaced by a chart
  // STOP AUTOMATING THIS until the design stabilizes
});

Why is "automate everything" a mistake?

Automation value is not uniform. Automating unstable or low-value tests produces two harms.

First, the maintenance burden grows until the team spends more time fixing tests than finding bugs. When a volatile feature changes, every test touching it breaks, and someone must update them, which is pure cost with no new coverage.

Second, when tests break for the wrong reasons, people start ignoring failures. This destroys the suite's credibility exactly when a real failure appears.

The trust curve:

  Trust in  │          Healthy suite
  the suite │    ●────────────────────●
            │   /                      \
            │  /                        \ Brittle suite
            │ /                          \────────●
            │/                          Failures ignored
            └──────────────────────────────────────
                      Time / test count

  A healthy suite stays trusted because tests break
  only for real bugs. A brittle suite erodes trust
  because tests break for every UI tweak.

How do you decide the automation boundary?

A balanced strategy automates the stable high-value core aggressively, defers automation of volatile features until they settle, and preserves human time for exploration and judgement.

// Decision framework in code
package com.bank.qa;

public class AutomationDecisionFramework {

    public enum Decision {
        AUTOMATE_NOW, DEFER_AUTOMATION, KEEP_MANUAL
    }

    public Decision evaluate(TestCandidate candidate) {
        // Business-critical + stable = automate immediately
        if (candidate.isBusinessCritical() && candidate.isStable()) {
            return Decision.AUTOMATE_NOW;
        }
        // Business-critical + volatile = defer until stable
        if (candidate.isBusinessCritical() && !candidate.isStable()) {
            return Decision.DEFER_AUTOMATION;
        }
        // Requires human judgement = always manual
        if (candidate.requiresHumanJudgement()) {
            return Decision.KEEP_MANUAL;
        }
        // Low-value + one-off = not worth automating
        if (!candidate.isRepetitive() && candidate.runCount() < 5) {
            return Decision.KEEP_MANUAL;
        }
        return Decision.AUTOMATE_NOW;
    }
}

How do you measure automation ROI?

Automation ROI is the time saved by automated runs minus the time spent writing and maintaining the tests. You can estimate it as follows.

  ROI = (manual_time_per_run x number_of_runs) - (write_time + maintenance_time)

  Example: interest calculation regression
  ─────────────────────────────────────────
  Manual: 4 hours per run x 50 runs/year = 200 hours
  Automation: 8 hours to write + 4 hours maintenance/year = 12 hours
  Savings: 200 - 12 = 188 hours/year

  Example: volatile rewards screen
  ────────────────────────────────
  Manual: 30 min per run x 10 runs/year = 5 hours
  Automation: 4 hours to write + 20 hours maintenance/year = 24 hours
  Savings: 5 - 24 = -19 hours/year (net LOSS)

What should a senior engineer know?

• Treat the suite as a product with its own cost of ownership. Prioritize by risk and run-frequency, not by vanity metrics like raw test count.
• Measure flakiness and fix or quarantine flaky tests immediately, because a flaky test is worse than no test.
• Retire tests that no longer earn their keep, for example a test for a feature that was removed or a test that duplicates unit-level coverage.
• Resist the coverage percentage vanity . Eighty percent coverage of the wrong things is less valuable than fifty percent coverage of the critical money paths.

// Flakiness tracking
package com.bank.qa;

public class FlakinessDashboard {

    public void reportFlaky(String testName, int failCount, int runCount) {
        double flakyRate = (double) failCount / runCount;
        if (flakyRate > 0.05) {
            System.out.println("QUARANTINE: " + testName
                + " flaky rate " + (flakyRate * 100) + "%");
        }
    }
}

What are the automation levels?

Not all automation is UI automation. The automation pyramid applies here too.

| Level | Speed | Stability | When to Use |
|---|---|---|---|
| Unit tests | Milliseconds | Very stable | Business logic, calculations |
| API tests | Seconds | Stable | Service contracts, integrations |
| UI tests | Seconds-minutes | Less stable | Critical user journeys only |

  Automation Pyramid
  ──────────────────
       /\
      / UI\        Few: critical journeys only
     /──────\
    /  API   \     Moderate: service contracts
   /──────────\
  / Unit Tests \   Many: business logic
 /──────────────\

What are common interview follow-up questions?

• "What percentage of tests should be automated?" There is no universal number. Aim for high automation of stable, critical, repetitive tests. The percentage is an outcome, not a target.
• "How do you handle a test that keeps breaking?" First determine if it breaks for real bugs or for UI changes. If for UI changes, the locators need to be more stable or the test needs refactoring. If it is genuinely flaky, quarantine it and investigate.
• "What is the biggest automation mistake you have seen?" Automating everything on day one, including volatile features, creating a massive brittle suite that nobody trusts.