Definition
- Infrastructure ≠ business health -- CPU, memory, pods, and blended error rate can all be green while customers are failing.
- Aggregation hides failures -- a blended 0.5% error rate can mask one large tenant at 100% failure.
- Disaggregate -- slice by tenant, region, endpoint, version, dependency, workflow step.
- HTTP 200 lie -- the service returns success but the downstream settlement silently fails.
- Business-outcome metrics -- 'did the transfer settle?' not 'did the HTTP call succeed?'
- Reconciliation -- a periodic job comparing intended vs actual state.
- Synthetic transactions -- automated end-to-end operations that verify the final result.
National Bank of India head office sees green dials: branches open, lights on, total footfall normal. But Priya cannot complete a single transfer. A teller stamping 'received' and dropping the form in the bin would also keep those dials green. The head office must measure whether real transactions finished, broken down by customer and branch.
Understanding the Concepts
Why can dashboards be green while customers fail?
Infrastructure metrics tell you machines are alive. Aggregate error rate tells you the blended failure rate. Neither tells you whether a specific customer's operation succeeded.
How does aggregation hide failures?
// 10,000 req/s total; BigCorp sends 50 req/s, all failing (100%)
double blendedError = 50.0 / 10000; // = 0.5% → dashboard is GREEN
// BigCorp is COMPLETELY DOWN but invisible in the aggregate
// Fix: disaggregate by tenant
Map<String, Double> errorByTenant = computeErrorByTenant();
// errorByTenant.get("BigCorp") = 1.0 → clearly broken
What is the HTTP 200 lie?
// The service returns 200 but the settlement silently fails downstream
@PostMapping("/api/transfers")
public ResponseEntity<Transfer> create(@RequestBody TransferRequest req) {
Transfer t = transferService.initiate(req); // returns 200
kafkaTemplate.send("transfer.initiated", t); // event published
return ResponseEntity.ok(t);
// But the settlement consumer fails → money never moves
// Error rate: 0% (all HTTP 200!). Business outcome: FAILING.
}
How do you fix it?
1. Disaggregated request metrics
// Tag every metric with tenant, region, endpoint
meterRegistry.counter("transfer.initiated",
"tenant", request.getTenantId(),
"region", request.getRegion(),
"endpoint", "/api/transfers")
.increment();
// Grafana dashboard can filter by any dimension
2. Business-outcome metrics
// Measure what the customer cares about
boolean settled = ledger.isSettled(transferId);
meterRegistry.counter("transfer.settled",
"tenant", tenantId,
"result", settled ? "success" : "failure")
.increment();
// THIS is what "healthy" means, not HTTP status
3. Reconciliation job
// Nightly: compare initiated transfers vs settled transfers
public void reconcile() {
for (Transfer t : transferRepo.findInitiatedBefore(Duration.ofHours(1))) {
if (!ledger.isSettled(t.getId())) {
alertService.fire("Transfer " + t.getId() + " not settled");
}
}
}
4. Synthetic transactions
Automated test transfers that run every minute and verify the money actually arrived.
Three layers of observability
| Layer | Measures | Example |
|---|---|---|
| Infrastructure | Machines alive | CPU, memory, pods |
| Request (disaggregated) | HTTP success per dimension | Error rate by tenant/region/endpoint |
| Business outcome | Customer's operation completed | Transfer settled, funds arrived, statement generated |
What are common mistakes?
// WRONG: only infrastructure metrics
// "CPU 30%, memory 60%, pods 5, error rate 0.3%" → all green
// But BigCorp's international transfers from Mumbai are 100% failing
// CORRECT: three layers of observability
// + disaggregate by every meaningful dimension
// + measure the business outcome, not just HTTP status
What should a senior engineer know?
- When customers report failures that dashboards do not show, the gap is almost always that dashboards measure infrastructure, not business outcomes.
- High-cardinality dimensioned metrics and structured logs with correlation IDs are essential.
- A payment returning HTTP 200 while failing to settle is NOT healthy.
Visual Overview
Aggregation Hides Failures Total: 10,000 req/s ┌──────────────────────────────┐ │ 9,950 small tenants: 0 err │ │ 50 BigCorp req/s: 50 errors │ ← 100% failure for BigCorp └──────────────────────────────┘ Blended error: 50/10000 = 0.5% → dashboard GREEN BigCorp: COMPLETELY DOWN Three Observability Layers ┌──────────────────────────────────┐ │ Layer 3: Business Outcome │ "Did the transfer settle?" ├──────────────────────────────────┤ │ Layer 2: Request (disaggregated) │ Error by tenant/region/endpoint ├──────────────────────────────────┤ │ Layer 1: Infrastructure │ CPU, memory, pods └──────────────────────────────────┘ All three needed; Layer 1 alone is BLIND to business failures
Key Takeaway
When dashboards are green but customers report failures, the dashboards are measuring infrastructure health, not business health. CPU, memory, pods, and aggregate error rate prove the machines are alive and traffic is flowing; they do not prove that a specific customer's operation completed. The most common hiding place is aggregation: one large tenant at 100% failure is invisible in a blended 0.5% error rate.
The deeper problem is the HTTP 200 lie: a service can return success while the downstream settlement silently fails, so even the error metric is wrong. Observability must extend to business outcomes: did the transfer settle, did the funds arrive, not just did the HTTP call succeed.
At National Bank of India, a large corporate tenant's international transfers from Mumbai were all returning 200 but never settling. The blended error rate was 0%. Pankaj fixed it by disaggregating metrics by tenant and adding a business-outcome metric on settlement, plus a reconciliation job that flagged initiated-but-not-settled transfers.
Remember one sentence: green infrastructure dashboards prove the building has power, not that the customer's transaction completed, so measure business outcomes disaggregated by tenant, region, endpoint, and workflow.
---