Definition
- Incident lifecycle – Follow detect, stabilize, investigate, diagnose, fix, verify, communicate, and prevent.
- Stabilize first – A safe, reversible action (rollback, circuit breaker) that reduces harm before you know the root cause.
- Evidence-based investigation – Use logs, metrics, traces, and thread dumps, not guesses.
- Communication cadence – Keep stakeholders updated throughout so the business can make decisions.
- Permanent prevention – Add a new alert, a test, or a config limit so the class of bug cannot recur silently.
- Blameless postmortem – Focus on the system and the process, not on who made a mistake.
Imagine a water pipe bursts in the vault at National Bank of India during business hours. A panicked person runs around; a professional first shuts the main valve to stop the flooding, then finds the exact broken joint, then repairs it, then checks for leaks, then adds a pressure sensor so it cannot happen silently again.
Understanding the Concepts
What is this question really testing?
The interviewer wants your incident-response discipline under pressure. They care about the method, not just the fix. A calm, step-by-step account with evidence and prevention signals a senior engineer. A panicked narrative with no prevention signals a junior one.
package com.bank.interview;
public class IncidentDiscipline {
enum Level { SENIOR, JUNIOR }
public static Level assess(boolean stabilizedFirst, boolean usedEvidence, boolean addedPrevention) {
return (stabilizedFirst && usedEvidence && addedPrevention) ? Level.SENIOR : Level.JUNIOR;
}
public static void main(String[] args) {
System.out.println(assess(true, true, true)); // SENIOR
System.out.println(assess(false, false, false)); // JUNIOR
}
}
How do you detect and triage?
Detection comes from an alert, a customer report, or a metric anomaly. Triage quantifies impact: "every transfer was failing, affecting all customers at National Bank of India."
package com.bank.service;
import java.util.logging.Logger;
public class IncidentDetection {
private static final Logger log = Logger.getLogger(IncidentDetection.class.getName());
public static void main(String[] args) {
double errorRate = 95.0;
int affectedCustomers = 50000;
log.severe(() -> "ALERT: Transfer error rate=" + errorRate + "% affecting " + affectedCustomers + " customers");
log.info("Triage: SEV1 - all transfers blocked");
}
}
How do you stabilize before investigating?
Stabilization is a safe, reversible action that reduces harm. A feature flag or circuit breaker lets you cut off a failing path fast, even before you know the root cause.
package com.bank.service;
import java.util.logging.Logger;
public class Stabilization {
private static final Logger log = Logger.getLogger(Stabilization.class.getName());
private static boolean newTransferPathEnabled = true;
public static void stabilize() {
newTransferPathEnabled = false; // flip feature flag
log.info("Stabilized: disabled new transfer path, using legacy fallback");
log.info("Customers can transfer again while we investigate");
}
public static void main(String[] args) {
stabilize();
}
}
Stabilized: disabled new transfer path, using legacy fallback
Customers can transfer again while we investigate
How do you investigate with evidence?
Use logs, metrics, traces, and thread dumps to narrow from symptom to cause. Rising latency plus growing database connections points to connection-pool exhaustion. A thread dump with many threads BLOCKED on the same lock points to a deadlock.
package com.bank.service;
import java.util.logging.Logger;
public class EvidenceInvestigation {
private static final Logger log = Logger.getLogger(EvidenceInvestigation.class.getName());
public static void main(String[] args) {
// Simulated investigation
log.info("Metrics: DB connection pool = 50/50 (exhausted)");
log.info("Logs: retry count per request = unbounded");
log.info("Diagnosis: retry storm exhausted connection pool");
log.info("Root cause: no retry budget; transient DB slowdown triggered unbounded retries");
}
}
How do you fix and verify?
The fix targets the root cause. Verification uses the same metrics you used to diagnose. Both must happen before you declare the incident resolved.
package com.bank.service;
import java.util.logging.Logger;
public class FixAndVerify {
private static final Logger log = Logger.getLogger(FixAndVerify.class.getName());
public static void main(String[] args) {
log.info("Fix: capped retries at 3 with exponential backoff");
log.info("Fix: drained and recovered connection pool");
log.info("Verify: pool usage back to 12/50 (healthy)");
log.info("Verify: transfer success rate back to 99.8%");
}
}
How do you communicate throughout?
Send clear status updates every few minutes so stakeholders can make business decisions. This is as important as the technical fix.
package com.bank.service;
import java.util.logging.Logger;
public class IncidentComms {
private static final Logger log = Logger.getLogger(IncidentComms.class.getName());
public static void main(String[] args) {
log.info("[12:01] Incident detected: transfers failing at 95% error rate");
log.info("[12:03] Stabilized: legacy fallback active, transfers resuming");
log.info("[12:15] Root cause identified: retry storm on DB pool");
log.info("[12:25] Fix deployed: retries capped, pool recovering");
log.info("[12:30] Resolved: error rate back to normal");
}
}
How do you add prevention?
A blameless postmortem leads to concrete preventive actions: a new alert, a test, a config limit. "This class of problem can never again build up silently."
package com.bank.service;
import java.util.logging.Logger;
public class Prevention {
private static final Logger log = Logger.getLogger(Prevention.class.getName());
public static void main(String[] args) {
log.info("Prevention 1: Alert on pool usage > 80%");
log.info("Prevention 2: Retry budget monitor with circuit breaker");
log.info("Prevention 3: Load test covering retry storms in CI");
log.info("Class of problem: can no longer build up silently");
}
}
What should a senior engineer emphasize about the postmortem?
Focus on the system and the process, not on who made a mistake. A blameless postmortem improves the system; a blame-heavy one only makes people hide mistakes.
package com.bank.interview;
public class BlamelessPostmortem {
public static void main(String[] args) {
System.out.println("Blameless: 'The system allowed unbounded retries' -> fix the system");
System.out.println("Blame: 'Developer X forgot to add a limit' -> people hide mistakes");
System.out.println("Always focus on systemic fixes, not individual blame.");
}
}
Visual Overview
┌────────────────────────────────────────────────────────┐ │ Incident Response Lifecycle │ ├────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────┐ │ │ │ 1. DETECT │ Alert fires: error rate spike │ │ └──────┬───────┘ │ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ 2. STABILIZE │ Flip feature flag to legacy path │ │ │ (stop the │ Customers can transfer again │ │ │ bleeding) │ │ │ └──────┬───────┘ │ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ 3. INVESTIGATE│ Logs + metrics + thread dumps │ │ │ (evidence) │ Pool exhausted + unbounded retries │ │ └──────┬───────┘ │ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ 4. FIX │ Cap retries, drain pool │ │ └──────┬───────┘ │ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ 5. VERIFY │ Pool healthy, error rate normal │ │ └──────┬───────┘ │ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ 6. PREVENT │ Alert on 80% pool, retry monitor │ │ │ (postmortem) │ Blameless: fix systems, not people │ │ └──────────────┘ │ │ │ │ Throughout: COMMUNICATE status every few minutes │ └────────────────────────────────────────────────────────┘
Key Takeaway
The critical production bug I handled at National Bank of India was a payment outage where every transfer for customers like Rajesh and Priya started failing during a busy period. The business impact was immediate : customers could not move money at all : so the urgency was very high and the pressure to just restart everything was intense. I was on call and I owned the resolution.
I worked through it in a deliberate sequence. First I stabilized: I opened the circuit breaker on the failing path so we stopped the bleeding and protected customers while I investigated. Then I diagnosed from evidence : the pool saturation plus a flood of retries told me the real cause was a retry storm. Our retry logic was unbounded, so when the database slowed briefly, retries piled up, exhausted every connection, and turned a small hiccup into a full outage.
The fix was targeted at that root cause: I capped the retries with a sensible budget so a storm could no longer saturate the pool. I verified using the exact same metrics I had used to diagnose. Throughout, I kept stakeholders updated with clear status. Afterwards I added an alert on pool usage crossing eighty percent and a retry-budget monitor.
What I learned is that under pressure the discipline is everything: stabilize first to protect customers, diagnose with evidence, fix the actual root cause, verify, and add prevention so it cannot recur. The distinction between stopping the bleeding and curing the disease is what I always carry into the next incident.
Remember one sentence: Stabilize first to protect customers, diagnose from evidence not guesses, fix the root cause, verify, and add prevention so the class of bug never recurs.
---