Difference between smoke, sanity, and regression testing and where each runs in CI/CD

• Smoke testing – A shallow but wide check of the most critical paths of a freshly deployed build, answering "is this build even worth testing further?" and gating every deployment.
• Sanity testing – A narrow but deep check that a specific bug fix or small feature change actually behaves correctly, without re-testing unrelated areas.
• Regression testing – A broad suite re-run across existing features to confirm that a change did not break previously working behavior.
• Pipeline placement – Smoke runs as a fast gate on every build and deployment, sanity runs immediately after a targeted change, and the full regression suite runs before a release to production.
• Scope not tool – These are testing scopes, not separate tools; the same automation framework can hold all three, organized with tags and pipeline stages.

What is smoke testing?

Smoke testing, sometimes called build verification testing, is the first thing you run against a new build. It touches many critical features but only lightly, confirming that the application starts, the main screens load, authentication works, and the core happy-path flows do not immediately fail. The purpose is a fast reject: if smoke fails, there is no point running the expensive full suite, so you fail the pipeline early and send the build back.

Smoke suites are intentionally small and fast, often a handful of tests that run in a minute or two. In Playwright you tag them so the runner can select only them.

// tests/smoke/login.smoke.spec.ts
import { test, expect } from '@playwright/test';

test('@smoke Login and view dashboard', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Username').fill(process.env.TEST_USER!);
  await page.getByLabel('Password').fill(process.env.TEST_PASS!);
  await page.getByRole('button', { name: 'Sign In' }).click();
  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
  await expect(page.getByText('Account Balance')).toBeVisible();
});

test('@smoke Transfer page loads', async ({ page }) => {
  await page.goto('/transfer');
  await expect(page.getByLabel('Beneficiary Account')).toBeVisible();
});

# Run only smoke-tagged tests as a deploy gate
npx playwright test --grep @smoke

What is sanity testing?

Sanity testing is narrow and deep. After a specific bug fix or a small feature change, you do not want to re-run everything; you want to confirm that the changed area now behaves correctly and that its immediate neighbors still make sense.

For example, if a developer fixed a rounding bug in interest calculation, a sanity check exercises several interest scenarios in depth, boundary amounts, zero balance, large balance, but does not re-test the card module. Sanity testing is often unscripted or lightly scripted and is a subset chosen by judgement.

// SanityInterestCalculationTest.java
package com.bank.service;

import org.junit.jupiter.api.Test;
import java.math.BigDecimal;
import static org.junit.jupiter.api.Assertions.assertEquals;

public class SanityInterestCalculationTest {

    private final InterestService service = new InterestService();

    @Test
    void interestOnZeroBalance() {
        BigDecimal result = service.calculateQuarterlyInterest(BigDecimal.ZERO);
        assertEquals(new BigDecimal("0.00"), result);
    }

    @Test
    void interestOnBoundaryBalance() {
        // Fix was for rounding at exactly 100000.00
        BigDecimal result = service.calculateQuarterlyInterest(
            new BigDecimal("100000.00")
        );
        assertEquals(new BigDecimal("1750.00"), result);
    }

    @Test
    void interestOnLargeBalance() {
        BigDecimal result = service.calculateQuarterlyInterest(
            new BigDecimal("5000000.00")
        );
        assertEquals(new BigDecimal("87500.00"), result);
    }
}

What is regression testing?

Regression testing is the broad safety net. Any change, even a correct one, can have side effects elsewhere, because code is interconnected. Regression re-runs a large suite of previously passing tests across many features to catch those unintended side effects.

Because this suite is large and slow, you automate as much of it as possible and run it at the right pipeline stage, typically before a release or nightly, rather than on every commit.

# Run the full regression suite before release
npx playwright test --project=regression
mvn test -Dgroups=regression

// RegressionTransferTest.java
package com.bank.service;

import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import java.math.BigDecimal;
import static org.junit.jupiter.api.Assertions.*;

@Tag("regression")
public class RegressionTransferTest {

    private final TransferService transferService = new TransferService();

    @Test
    void transferDebitsAndCreditsCorrectly() {
        Account sender = new Account("ACC-001", new BigDecimal("50000.00"));
        Account receiver = new Account("ACC-002", new BigDecimal("10000.00"));

        transferService.transfer(sender, receiver, new BigDecimal("5000.00"));

        assertEquals(new BigDecimal("45000.00"), sender.getBalance());
        assertEquals(new BigDecimal("15000.00"), receiver.getBalance());
    }

    @Test
    void transferRejectsInsufficientBalance() {
        Account sender = new Account("ACC-001", new BigDecimal("1000.00"));
        Account receiver = new Account("ACC-002", new BigDecimal("10000.00"));

        assertThrows(InsufficientBalanceException.class, () ->
            transferService.transfer(sender, receiver, new BigDecimal("5000.00"))
        );
    }
}

How is smoke different from sanity?

The distinction interviewers want is that smoke is breadth-first and shallow across the whole system, while sanity is depth-first and narrow on one change.

| Aspect | Smoke Testing | Sanity Testing |
|---|---|---|
| Breadth | Wide: touches many features | Narrow: focuses on one area |
| Depth | Shallow: only happy paths | Deep: exercises edge cases |
| Trigger | Every build or deployment | After a specific fix or change |
| Purpose | Decide if the build is worth testing | Confirm the fix actually works |
| Suite size | Small and fixed | Subset chosen by judgement |
| Duration | 1-2 minutes | Varies, usually 5-15 minutes |
| Who decides scope | Predefined critical paths | Tester's judgement per change |

// Smoke: touches login, transfer, balance, statement (wide, shallow)
test.describe('@smoke Critical paths', () => {
  test('login works', async ({ page }) => { /* ... */ });
  test('transfer page loads', async ({ page }) => { /* ... */ });
  test('balance is visible', async ({ page }) => { /* ... */ });
  test('statement downloads', async ({ page }) => { /* ... */ });
});

// Sanity: deep on one area after a fix (narrow, deep)
test.describe('Sanity: interest rounding fix', () => {
  test('zero balance', async ({ page }) => { /* ... */ });
  test('boundary 100000', async ({ page }) => { /* ... */ });
  test('large balance 5000000', async ({ page }) => { /* ... */ });
  test('negative balance rejected', async ({ page }) => { /* ... */ });
});

Where does each fit in a CI/CD pipeline?

In a CI/CD pipeline the placement matters because each scope has a different cost and purpose.

• On every commit or pull request : Run unit tests and lint. These are the fastest feedback.
• On every build or deploy to an environment : Run the smoke suite as a gate. If smoke fails, reject the build immediately.
• After a targeted change : Run the sanity subset chosen for the changed area. This often happens in the feature-branch pipeline.
• Before release to production : Run the full regression suite against the staging environment. This is the final safety net.
• Nightly or scheduled : Run the full regression plus non-functional tests like performance, if they are too slow for pre-release.

# .github/workflows/pipeline.yml
jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - run: mvn test -Dgroups=unit

  deploy-staging:
    needs: unit-tests
    steps:
      - run: ./deploy.sh staging

  smoke-gate:
    needs: deploy-staging
    steps:
      - run: npx playwright test --grep @smoke

  regression-before-release:
    needs: smoke-gate
    steps:
      - run: npx playwright test --project=regression
      - run: mvn test -Dgroups=regression

What are common mistakes?

The most frequent mistake is conflating smoke and sanity. Smoke is a fixed predefined set of broad checks; sanity is a judgement-chosen subset of deep checks on one area. Another mistake is running the full regression suite on every commit, which makes the pipeline unbearably slow and wastes resources. The correct approach is to layer: fast feedback first, expensive checks later.

Wrong: Run 2000 regression tests on every commit
       Pipeline takes 45 minutes, developers stop waiting

Correct: Commit  -> unit tests (seconds)
         Deploy  -> smoke (1-2 minutes)
         Release -> regression (30 minutes, acceptable)

Another mistake is having no smoke suite at all and discovering after a 30-minute regression run that the application did not even start. A two-minute smoke check at the gate would have caught that immediately.

What should a senior engineer know?

A senior engineer knows these are scopes, not tools. The same automation framework holds all three, and you organize them with tags, projects, and pipeline stages rather than separate codebases.

• Tag-based selection : Use @smoke, @sanity, @regression tags or JUnit @Tag annotations so the CI pipeline can select the right scope at each stage.
• Flakiness monitoring : Smoke tests that flake destroy trust in the gate. Track flake rates and fix or quarantine flaky tests immediately.
• Parallel execution : The regression suite runs in parallel shards to keep the total wall-clock time acceptable.
• Test data isolation : Each scope uses its own test data so tests do not interfere with each other.
• Environment parity : Smoke and regression should run against an environment that closely mirrors production to catch environment-specific issues.

// Using JUnit @Tag for pipeline selection
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;

@Tag("smoke")
public class SmokeLoginTest {
    @Test
    void loginWithValidCredentials() {
        // Quick check: login works
    }
}

@Tag("regression")
public class RegressionLoanTest {
    @Test
    void loanApprovalForEligibleCustomer() {
        // Detailed: full loan flow
    }
}

How is this different from the test pyramid?

Smoke, sanity, and regression are testing scopes that describe when and why you run a set of tests. The test pyramid describes the distribution of tests by level: unit at the base, integration in the middle, and end-to-end at the top. They are orthogonal.

| Dimension | Smoke / Sanity / Regression | Test Pyramid |
|---|---|---|
| Describes | When and why to run tests | How many tests at each level |
| Categories | Scope of execution | Level of isolation |
| Example | Smoke suite runs on deploy | Unit tests form the base |
| Overlap | A smoke suite may contain E2E tests | E2E tests may be tagged as smoke |

A smoke suite is typically composed of a few end-to-end tests, but it could also include integration or even unit checks. The regression suite spans all pyramid levels. The pyramid tells you the healthy shape; the scopes tell you when each slice runs.

What are common interview follow-up questions?

• "If your smoke suite takes ten minutes, is that still smoke?" No, a true smoke suite should be one to two minutes. If it grows, prune it back to the absolute critical paths.
• "Can a single test be both smoke and regression?" Yes. A smoke test is also part of the regression suite. Tags are additive.
• "How do you decide what goes in the smoke suite?" Pick the paths whose failure means the build is completely unusable: login, main navigation, the single most important transaction.
• "What if sanity finds a new bug outside the changed area?" Log it, but do not expand the sanity scope mid-run. The regression suite is where broad coverage catches neighboring issues.