Definition
- On-call rotation -- A scheduling practice where team members take turns being the first responder for production issues, typically for a one-week shift.
- Escalation policy -- If the primary does not acknowledge within minutes, the alert escalates to the secondary, then to a manager, so no alert is dropped.
- Alerting tools -- PagerDuty or Opsgenie route alerts to the current on-call person via phone, SMS, and push, repeating until acknowledged.
- Communication tools -- Slack or Teams channels coordinate the response, creating a timeline for the postmortem.
- Runbooks -- Step-by-step guides for diagnosing and fixing common problems, linked directly from the alert.
Think of the National Bank of India needing someone reachable at all hours in case the systems break at 3 AM. Rather than exhausting one person, team members take turns wearing the pager for a week each, so everyone knows exactly who is responsible this week. A dispatcher (PagerDuty) rings their phone when an alarm fires, and if they do not answer, it rings the backup. The responder follows a prepared instruction manual (runbook) that says step by step how to handle each common problem, so they are not improvising at 3 AM.
Understanding the Concepts
What is on-call rotation?
The core of on-call is a schedule that assigns a primary responder for each shift, usually about a week. Most teams also assign a secondary responder as backup. The rotation is published so everyone knows who is on call now and next, and handovers between shifts pass along ongoing issues.
# On-call schedule for National Bank of India transfer-api team:
# Week of Jan 15: Primary: Rajesh, Secondary: Priya
# Week of Jan 22: Primary: Priya, Secondary: Pankaj
# Week of Jan 29: Primary: Pankaj, Secondary: Rajesh
# Rotation size: 3 engineers (minimum recommended: 4-5 for sustainability)
How does the alerting tool work?
PagerDuty or Opsgenie integrates with your monitoring (Prometheus, CloudWatch, Datadog), receives alerts, and routes them to the current on-call person through phone call, SMS, push notification, and email, repeating and escalating until acknowledged.
# PagerDuty escalation policy (simplified)
escalation_policy:
name: transfer-api-escalation
rules:
- escalation_delay_in_minutes: 5
targets:
- type: schedule_reference
id: transfer-api-primary # primary on-call
- escalation_delay_in_minutes: 10
targets:
- type: schedule_reference
id: transfer-api-secondary # secondary backup
- escalation_delay_in_minutes: 15
targets:
- type: user_reference
id: pankaj # engineering manager
# Alert flow:
# 1. Prometheus fires TransferAPIHighErrorRate alert
# 2. Alertmanager sends to PagerDuty integration
# 3. PagerDuty looks up the schedule: Rajesh is on-call
# 4. PagerDuty calls Rajesh's phone
# 5. If Rajesh does not acknowledge in 5 minutes:
# -> PagerDuty calls Priya (secondary)
# 6. If Priya does not acknowledge in 10 minutes:
# -> PagerDuty calls Pankaj (manager)
# 7. Alert is NEVER silently dropped.
How do communication tools support incident response?
When an incident is declared, teams use a dedicated Slack or Teams channel to coordinate. This creates a timeline that later feeds the postmortem.
# Automated incident channel creation:
# PagerDuty triggers a Slack workflow:
# 1. Creates channel: #incident-2024-01-15-transfer-api
# 2. Invites: Rajesh (on-call), Priya (secondary), Pankaj (IC)
# 3. Posts: alert details, dashboard link, runbook link
# 4. Pins: status template for regular updates
# Channel timeline becomes the postmortem's raw material:
# 10:30 Rajesh: "Error rate at 40%. Checking deploy history."
# 10:33 Rajesh: "Deploy at 10:28 is the prime suspect."
# 10:35 Pankaj: "Confirmed SEV1. I am IC. Rajesh investigate, Priya comms."
# 10:40 Rajesh: "Rolling back now."
# 10:42 Rajesh: "Rollback complete. Error rate dropping."
What are runbooks and why do they matter?
Runbooks are step-by-step guides for diagnosing and resolving known problems, linked directly from the alert. A responder woken at 3 AM has concrete instructions instead of improvising under stress.
# Runbook: TransferAPIHighErrorRate
# Linked from: PagerDuty alert "TransferAPIHighErrorRate"
## Step 1: Confirm impact
- Open Grafana dashboard: https://grafana.nationalbank.in/d/transfer-api
- Check: error rate, latency, tx success rate
- Size: what is broken, how many users, since when
## Step 2: Check recent changes
- kubectl rollout history deployment/transfer-api -n banking
- argocd app history transfer-api
- Check feature-flag changes in LaunchDarkly
## Step 3: Mitigate
- If deploy is suspect: kubectl rollout undo deployment/transfer-api -n banking
- If flag is suspect: disable the flag in LaunchDarkly
- If load spike: kubectl scale deployment/transfer-api --replicas=10 -n banking
## Step 4: Verify
- Watch error rate in Grafana for 5 minutes
- Confirm rate returns to pre-incident baseline
## Step 5: Escalate if not resolved
- Page the secondary on-call
- Open an incident channel: #incident-YYYY-MM-DD-transfer-api
- Notify engineering manager: Pankaj
What makes alert quality important?
Alerts must be actionable and tied to user impact. Alert fatigue from noisy, non-actionable pages trains responders to ignore the pager, which is dangerous.
# BAD alert: pages on internal metrics
# "Pod CPU > 80%" -> may not affect users at all
# Result: 15 pages per week, most are noise. Responder learns to ignore.
# GOOD alert: pages on user-facing symptoms
# "Transfer API error rate > 5% for 2 minutes"
# Result: every page means customers are affected. Responder takes action.
# Refinement:
# Page on: symptoms that affect users, SLO burn rate
# Ticket on: internal degradation that does not yet affect users
# Do not alert on: auto-recoverable conditions (brief CPU spike)
What about on-call sustainability?
On-call must be sustainable. Track metrics like pages per shift and after-hours pages to detect and fix unhealthy load.
# On-call health metrics:
# Pages per shift: target < 5 (actionable, not noisy)
# After-hours pages: target < 2 per week
# Mean time to acknowledge: target < 5 minutes
# Recurring alerts: track and eliminate root causes
# If pages per shift > 10:
# -> Review alert thresholds for noise
# -> Fix root causes of recurring alerts
# -> Consider adding more people to the rotation
What are common mistakes?
# WRONG: On-call rotation with only 2 people
# Each person is on-call every other week = burnout in 2 months.
# CORRECT: Minimum 4-5 people in the rotation.
# Each person is on-call ~1 week per month = sustainable.
# WRONG: Stale runbooks that reference deleted dashboards
# Responder opens the runbook at 3 AM. Links are broken.
# They have no guidance. Time is wasted.
# CORRECT: Test runbooks quarterly. Update after every incident.
# Runbooks are living documents, not write-once artifacts.
What should a senior engineer know?
On-call is a feedback loop for reliability: every page is a signal, and recurring pages should drive engineering work to eliminate the underlying cause so the pager gets quieter over time. A healthy on-call system is one where alert volume decreases as the team fixes root causes. Follow-the-sun rotations across time zones can avoid night shifts entirely for global teams. In banking, on-call carries clear severity definitions and communication protocols because production issues can directly affect customers' money.
# On-call improvement loop:
# 1. Track recurring alerts: "TransferAPITimeout" fired 12 times this month
# 2. Investigate root cause: database connection pool too small
# 3. Fix: increase pool size, add connection pooling metrics
# 4. Result: alert stops firing. Pager gets quieter.
# 5. Repeat for the next most frequent alert.
What are common interview follow-up questions?
- "How do you handle on-call for multiple services?" -- Each service has its own rotation, or a platform team handles cross-cutting infrastructure while service teams handle their own services.
- "What if someone is on-call and gets sick?" -- PagerDuty supports schedule overrides so another team member can take over the shift.
- "Should on-call be compensated?" -- Yes. Organizations increasingly compensate on-call time and provide time off after particularly heavy shifts.
Visual Overview
On-Call Alert Flow
===================
Monitoring (Prometheus / CloudWatch)
│ alert fires
▼
┌─────────────────────────┐
│ PagerDuty / Opsgenie │ Routes to current on-call
│ (alerting backbone) │
└──────────┬──────────────┘
│ phone / SMS / push
▼
┌─────────────────────────┐
│ Primary On-Call │ Acknowledges within 5 min?
│ (Rajesh, this week) │
└──────────┬──────────────┘
/ \
yes no (escalate)
│ │
▼ ▼
┌──────────┐ ┌─────────────────────┐
│ ACK │ │ Secondary On-Call │
│ Opens │ │ (Priya) │
│ runbook │ └──────────┬──────────┘
│ Opens │ / \
│ Slack │ yes no (escalate to manager)
│ channel │
└──────────┘
Runbook: step-by-step guide linked from the alert
Slack: #incident channel for coordination + timeline
Key Takeaway
On-call rotation is a scheduling practice where team members take turns being the designated first responder, typically for a one-week shift, so there is always one clear person responsible for acknowledging alerts. A secondary backup and an escalation policy ensure that no alert is ever silently dropped. The alerting tool (PagerDuty or Opsgenie) is the backbone: it integrates with monitoring, routes alerts to the current on-call, and repeats until acknowledged. Communication tools like Slack coordinate the response in a dedicated incident channel, and runbooks give step-by-step guidance so a responder at 3 AM is not improvising.
Alert quality matters enormously: alerts must be actionable and tied to user impact, because noisy pages train responders to ignore the pager. On-call must be sustainable: the rotation should be large enough and alert volume low enough that nobody burns out, with metrics like pages per shift tracked to detect unhealthy load. At the National Bank of India, when a transfer-5xx alert fires, PagerDuty pages the week's primary, escalates if needed, opens an incident channel, and attaches the runbook whose first steps are confirm impact, check what changed, and mitigate first.
Remember one sentence: on-call assigns a clear first responder backed by alerting, escalation, runbooks, and a communication channel, and every recurring page should drive engineering work to make the pager quieter over time.
---