Back to SDET Scenarios
Q3Question 3 of 12

(Scenario) The entire automation suite starts failing right after a deployment

Key Concepts
  • On a mass post-deploy failure, first classify failures by symptom using traces, screenshots, and health checks
  • Distinguish a real product regression from an environmental or deployment issue before blaming the code
  • Check the application health endpoint, deployment status, and compare the failing tests to the changes in the deploy
  • If the app is down or misconfigured, fix the deployment; if a code change broke functionality, the suite correctly caught it
  • Communicate the triage to stakeholders with evidence: 'the failures are caused by X, here is the trace showing Y'
  • Never blame the tests first; a mass failure after a deploy is most often a real issue the tests correctly detected

Definition

On a mass post-deploy failure, classify the failures by symptom using logs, traces, and health checks to distinguish a real product regression from a contract, locator, auth, data, or environment change, because assuming one root cause leads to the wrong action, crying wolf or missing a real defect.
  • Classify first – Use traces, screenshots, and health checks to classify failures by symptom before taking action.
  • App health – Check if the application is even running and responding to health endpoints.
  • Deployment status – Verify the deploy completed successfully and the correct version is running.
  • Correlation – Compare failing tests to the changes in the deployment to identify the specific regression.
  • Environment vs code – Distinguish environmental issues (config, data, network) from real code regressions.
  • Never blame tests first – A mass failure right after a deploy is most often a real issue the tests correctly detected.

If every check in the National Bank of India suite goes red right after a release, panicking and assuming the product is broken is as wrong as assuming the tests are just stale. It could be that the bank genuinely broke transfers, a real defect, or that the login endpoint moved so none of the tests can even sign in, or that the deploy reset the test accounts, or that the environment URL changed. Treating all the red as one thing leads to the wrong action, either crying wolf about a product bug that is not there, or dismissing a real fire as a flaky alarm. The skilled responder sorts the failures into buckets by what the evidence shows, so they can say with confidence which reds are the product breaking and which are the tests needing an update.

Understanding the Concepts

What is the first response to a mass failure?

Do not panic and do not blame the tests. A mass failure right after a deploy is most often a real issue.

TEXT
  Step 1: Take a breath. Gather evidence before acting.
  Step 2: Open the test report and check the failure patterns.
  Step 3: Check the application health endpoint.

How do you classify failures by symptom?

Group the failures to determine the category of the problem.

DIAGRAM
  Pattern                    │ Likely Cause
  ───────────────────────────┼──────────────────────────
  ALL tests fail on login    │ App is down or auth broken
  ALL tests timeout          │ Environment unreachable
  A GROUP of tests fail      │ Specific feature regression
  (related to deploy changes)│
  RANDOM scattered fails     │ Environment config issue
  Tests that passed before   │ Real code regression
  the exact deploy           │

How do you check application health?

Before investigating tests, verify the application is even running.

BASH
# Check health endpoint
curl -s https://staging.nationalbank.in/health | jq .

# Check deployment status
kubectl get pods -n staging
kubectl describe deployment banking-app -n staging

# Check the deployed version
curl -s https://staging.nationalbank.in/api/version

How do you correlate failures with the deployment?

Compare the failing tests to the changes in the deploy.

BASH
# What changed in this deploy?
git log --oneline previous-deploy..current-deploy

# Which tests are failing?
npx playwright test --list | grep FAIL

# Do the failing tests cover the changed areas?
# If transfer tests fail and the deploy changed transfer code -> regression
# If loan tests fail but the deploy only changed auth -> environment issue

How do you distinguish environmental vs code issues?

Check the traces and logs for clues.

TEXT
  Environmental signs:
  - Connection refused / timeout errors
  - Wrong base URL or missing secrets
  - Database migration not run
  - Feature flag misconfigured

  Code regression signs:
  - Assertion failures on specific values
  - New error messages from the application
  - Tests that directly cover the changed code

How do you communicate the triage?

Provide evidence-based communication to stakeholders.

TEXT
  Bad: "Tests are broken, we're looking into it."

  Good: "17 of 200 tests fail. All 17 are transfer tests.
         The traces show the transfer API returns 500 with
         'NullPointerException in TransferService line 42.'
         This correlates with commit abc123 which refactored
         TransferService. This is a real regression caught
         by the suite. Rolling back the deploy."

What should you never do?

Never blame the tests first. Never disable the suite to unblock the pipeline. Never skip the investigation.

TEXT
  WRONG responses to mass failure:
  ✗ "The tests are flaky, let's skip them"
  ✗ "Just retry everything until it passes"
  ✗ "Disable the test suite so we can deploy"

  CORRECT responses:
  ✓ "Let me check the app health first"
  ✓ "The traces show the API is returning 500"
  ✓ "This correlates with the deploy changes"
  ✓ "Roll back the deploy while we fix the regression"

What are common interview follow-up questions?

  • "What if the tests fail but the health check passes?" Check the specific feature area. The app may be up but a specific function broken.
  • "How long should triage take?" 15-30 minutes maximum with the right artifacts. If longer, escalate.
  • "What if it was actually a test environment issue?" Fix the environment and document the root cause so it does not recur.

Key Takeaway

When the entire automation suite goes red right after a deployment, my first discipline is to refuse to assume a single cause, because a mass failure can come from several very different roots that demand opposite actions: a genuine product regression, a changed API contract, changed locators from a UI update, broken authentication, missing or changed test data, or an environment and configuration change. The worst mistake is assuming: if I assume the tests are just brittle and the cause was a real regression, I ship a defect to customers; if I assume a real regression and the cause was a test or environment change, I block a good release and waste everyone's time. So I stabilize, gather the grouped error messages and the traces and videos, and classify before concluding.

I classify by grouping failures by symptom and mapping each group to a likely cause. If every test fails at the very first step, at login or landing, that points to authentication or environment, the auth endpoint changed, the secret is wrong, or the base URL is pointing at the wrong or a down environment, and the product is probably fine. If elements are not found across the board, that points to changed locators from a UI refactor, so the page objects need updating but users are fine. If API tests fail with schema or status mismatches, that points to a changed contract, which I then judge as either an intended breaking change to version and update tests for, or an accidental break to fix in the product. If tests fail because expected data is missing, that points to the deploy resetting the database, so I restore seeding. And if a coherent set of functional tests run with correct steps but produce wrong business outcomes, like a transfer completing with the wrong balance, that is the signature of a real product regression, and I raise a high-priority defect and consider a rollback. I corroborate with an independent smoke or health probe, can I load the app and log in manually, and with the deployment diff, which often names the cause directly.

The reason classifying is mandatory is that the correct action is entirely cause-dependent, so lumping all the red together means taking one action for problems that need five different ones, and it costs the team credibility either way. At the National Bank of India a release once turned the whole suite red, and the pressure was to declare a product outage, but I classified the failures and found that most were a moved login endpoint and a reset test database, pure automation and environment issues, while exactly one group, a wrong-balance assertion, was a genuine regression. I escalated that one as a P1 defect and fixed the config and seeding for the rest, so we neither cried wolf about the whole release nor missed the real bug. Being able to describe classifying by symptom, using multiple signals, and distinguishing brittle automation from a real product defect, rather than assuming one root cause, is exactly what shows an interviewer I respond to a broken release like a calm senior engineer.

Remember one sentence: a mass post-deploy failure is most often a real issue the tests correctly detected, so classify by symptom and check the app before blaming the tests.

---

Visual Overview

DIAGRAM
  Mass Post-Deploy Failure Triage
  ────────────────────────────────

  Suite fails after deploy
       │
  ┌────▼──────────────────────┐
  │ 1. Check app health       │ Health endpoint? Up or down?
  └────┬──────────────────────┘
       │
  ┌────▼──────────────────────┐
  │ 2. Classify failures      │ All same? Or varied symptoms?
  │    Read traces + logs     │
  └────┬──────────────────────┘
       │
       ├── All tests fail on login ──> App or env is down
       │
       ├── Group of tests fail ──> Specific feature broken
       │   (matches deploy changes)    (real regression!)
       │
       └── Random scattered fails ──> Environment issue
                                      (config, data, network)
Share
Use ← → arrow keys to navigate