Definition
- Before-vs-after comparison – Overlay the deploy timestamp on every dashboard and compare each signal just before and just after the release.
- Signals to compare – Application latency percentiles, distributed traces, database query performance, CPU and memory, GC activity, external dependency latency, config and feature-flag changes, and error logs.
- Isolation test – If old-version instances stay fast and new-version instances are slow, the deploy is confirmed as the cause.
- Restore safely – Roll back to the last known-good version (blue-green flip, rollout undo), verify the baseline is restored, then re-release behind a canary.
- Root-cause later – Investigate the specific cause only after service is restored, because every minute of high latency costs customers.
Imagine the National Bank of India's transfer service suddenly serves customers slowly right after a new process was rolled out. You line up yesterday's performance against today's, metric by metric: how long each transfer took, which step in the journey got slow, whether the ledger database is struggling, whether the servers are overloaded, whether the garbage collector is thrashing. If everything was normal until the exact moment of the rollout and only the new counters are slow, you have your culprit, and you switch customers back to the old process while you investigate.
Understanding the Concepts
What is the before-vs-after comparison method?
The core method is to pin the deploy timestamp and overlay it on every dashboard, because the whole diagnosis is a before-and-after comparison anchored on that moment. Every signal is examined just before the deploy and just after, and the question is: did this metric change at exactly that time, and does it affect only the new version?
# Pin the deploy timestamp
# In Grafana, add an annotation at the deploy time:
# Annotations > Add > "banking-api v5.0 deployed at 2024-01-15 10:30 UTC"
# In Prometheus, query latency before and after:
# BEFORE (10:00-10:29):
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{app="banking-api"}[5m]))
# AFTER (10:31-11:00):
# Same query, look for the jump at the deploy annotation
How do you check application metrics?
Start with p50, p95, and p99 latency and error rate for the affected endpoint. Check whether the jump coincides with the deploy and whether it affects only Pods running the new version. If old-version instances stay fast and new-version instances are slow, the release is implicated rather than external load.
# PromQL: Compare latency by version
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket{
app="banking-api",
version="v5.0" # new version
}[5m])
)
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket{
app="banking-api",
version="v4.9" # old version (if still running during canary)
}[5m])
)
# Expected finding:
# v4.9 (old): p99 = 120ms (steady, normal)
# v5.0 (new): p99 = 850ms (jumped at deploy time)
# Conclusion: the deploy caused the regression.
How do you use distributed traces to find the slow span?
A trace shows the full journey of one request across services. Compare a slow trace after the deploy with a fast trace before it, and see exactly which hop grew. This narrows a vague "the API is slow" into a precise "the account-lookup span went from 10ms to 400ms."
# In Jaeger or your tracing backend:
# 1. Find a slow trace from after the deploy (tag: version=v5.0)
# 2. Find a normal trace from before the deploy (tag: version=v4.9)
# 3. Compare span durations side by side
# Trace comparison:
# BEFORE (v4.9):
# gateway 5ms
# auth-service 12ms
# banking-api 45ms
# -> db-query 10ms <- normal
# Total: 62ms
# AFTER (v5.0):
# gateway 5ms
# auth-service 12ms
# banking-api 780ms
# -> db-query 400ms <- 40x slower!
# Total: 797ms
# Root cause: the account-lookup query in v5.0 is 40x slower.
How do you check database performance?
A common regression is a new or changed query that lost an index or started doing a full table scan. Check slow-query logs, query execution plans, lock waits, and connection-pool saturation, comparing timings before and after.
-- Check for slow queries that appeared after the deploy
SELECT query, calls, mean_time, max_time
FROM pg_stat_statements
WHERE mean_time > 100 -- milliseconds
ORDER BY mean_time DESC
LIMIT 10;
-- Check the execution plan for the suspect query
EXPLAIN ANALYZE
SELECT balance, account_type
FROM accounts
WHERE customer_id = 'SAV2001';
# Before deploy: Index Scan using idx_customer_id (0.05ms)
# After deploy: Seq Scan on accounts (380ms)
# Root cause: a migration in v5.0 dropped the index on customer_id.
How do you check CPU, memory, and GC activity?
A release can increase CPU through a hot loop or raise memory so high that the process is paging. GC pauses are a classic cause of tail latency: a version that allocates far more objects causes longer GC pauses that inflate p99.
# Check CPU and memory per Pod version
kubectl top pods -l app=banking-api -n banking
# Check GC metrics in Prometheus (for JVM services)
# GC pause time:
jvm_gc_pause_seconds_sum{app="banking-api", version="v5.0"}
# GC pause count:
jvm_gc_pause_seconds_count{app="banking-api", version="v5.0"}
# Before deploy: GC pause avg = 15ms, frequency = 2/min
# After deploy: GC pause avg = 120ms, frequency = 12/min
# Conclusion: v5.0 allocates far more objects, causing 8x more GC pauses.
# This explains the p99 latency spike.
How do you rule out external dependencies and config changes?
The slowdown might be a coincidence: a payment gateway or credit bureau got slow at the same time and your deploy is innocent. Compare external call latency to rule this in or out. Also review config and feature-flag changes, since the release may have shipped a config change that is the real cause.
# Check external dependency latency
# PromQL: latency to the payment gateway
histogram_quantile(0.99,
rate(http_client_duration_seconds_bucket{
target="payment-gateway.icici.in"
}[5m])
)
# If this jumped at the same time, the external dependency is the cause,
# and your deploy is innocent.
# Check config changes
kubectl get configmap banking-api-config -n banking -o yaml
# Look for: smaller thread pool, disabled cache, new feature flag enabled
How do you restore service safely after confirming the deploy?
The fastest safe action is to roll back to the last known-good version. If the cause was a feature flag or config, flip the flag off without a full rollback.
# Kubernetes: Roll back to the previous version
kubectl rollout undo deployment/banking-api -n banking
kubectl rollout status deployment/banking-api -n banking
# Verify latency returned to the pre-deploy baseline
# Watch p99 in Grafana for 5 minutes to confirm recovery
# If the cause was a feature flag:
kubectl set env deployment/banking-api \
FEATURE_NEW_ACCOUNT_LOOKUP=false -n banking
# Even faster: no image change, just env change
What are common mistakes when diagnosing latency regressions?
# WRONG: Assuming the deploy is innocent because "the code change was small"
# Small code changes can drop indexes, change query plans, or enable hot paths.
# Always compare metrics before and after.
# WRONG: Blaming external dependencies without checking
# "The payment gateway must be slow."
# But the external latency metrics are unchanged.
# The deploy caused the regression.
# CORRECT: Compare every signal, isolate by version, confirm at the timestamp.
What are common interview follow-up questions?
- "What if both old and new versions are slow?" – Then the cause is external (dependency slowdown, load spike, or infrastructure issue), not the deploy.
- "How do you prevent this in the future?" – Re-release behind a canary with automatic rollback on p99 regression, and add database migration review to the CI pipeline.
- "What if the rollback does not fix latency?" – Then the deploy was not the cause. Check external dependencies, load changes, and infrastructure issues.
Visual Overview
Before vs. After Diagnosis Flow
================================
Deploy Timestamp (anchor)
│
▼
┌──────────────────────────────────────┐
│ Application Metrics (p50/p95/p99) │
│ Old version fast? New version slow? │
└──────────┬───────────────────────────┘
│ yes: deploy implicated
▼
┌──────────────────────────────────────┐
│ Distributed Traces │
│ Which span got slower? │
└──────────┬───────────────────────────┘
│ e.g., db-query 10ms -> 400ms
▼
┌──────────────────────────────────────┐
│ DB / CPU / Memory / GC / External │
│ Confirm the root cause │
└──────────┬───────────────────────────┘
│ confirmed: deploy is cause
▼
┌──────────────────────────────────────┐
│ ROLLBACK to last known-good version │
│ Verify baseline restored │
└──────────────────────────────────────┘
Key Takeaway
When a deployment suddenly increases API latency, diagnose it as a disciplined before-and-after comparison anchored on the exact deploy timestamp, which you overlay on every dashboard. Start with application metrics: p50, p95, and p99 latency and error rate, checking whether the jump coincides with the deploy and affects only instances running the new version. Then use distributed traces to find which specific span got slower, turning a vague latency complaint into a precise finding like "the account-lookup query went from 10ms to 400ms."
From there, check database performance for a query that lost an index, CPU and memory for a hot loop, and GC activity for a version that allocates far more objects. Also compare external dependency latency to rule out coincidences, and review config or feature-flag changes that the release may have shipped. When the evidence lines up: the regression starts at the deploy, affects only the new version, and a specific span or query changed, the deploy is confirmed as the cause. Roll back to the last known-good version immediately, verify latency returns to baseline, and only then do the deep root-cause analysis and plan a re-release behind a canary.
Remember one sentence: pin the deploy timestamp, compare every signal before and after, isolate by version, and roll back to restore service before diagnosing the cause.
---