Why all services can be green while the product is broken: business metrics and synthetic journeys

Scenario: Feature flag accidentally turned off.

Payment Service:
  Health check: 200 OK (green)
  CPU: 15% (normal)
  5xx rate: 0% (perfect)
  What it does: accepts every transfer, returns HTTP 200,
    writes status as "INITIATED" but the settlement flag is OFF.
    No payment ever moves to "COMPLETED".

Result: every dashboard is green, but no customer's money moves.

package com.bank.seam;

import java.math.BigDecimal;
import java.util.logging.Logger;

public class SeamFailureExamples {

    private static final Logger LOG = Logger.getLogger(SeamFailureExamples.class.getName());

    // Scenario 1: Feature flag flipped off.
    // Payment Service returns 200 but marks as "INITIATED" forever.
    public void scenarioFeatureFlag(String paymentId) {
        boolean settlementEnabled = false; // accidentally turned off
        if (settlementEnabled) {
            // settle(paymentId);
        }
        LOG.info("paymentId=" + paymentId + " status=INITIATED"); // no error logged
    }

    // Scenario 2: Wrong exchange rate from pricing service.
    // Transfer "succeeds" but with incorrect amounts.
    public void scenarioWrongData(String paymentId, BigDecimal rate) {
        BigDecimal amount = new BigDecimal("50000").multiply(rate); // rate is wrong
        LOG.info("paymentId=" + paymentId + " converted amount=" + amount); // looks fine
    }

    // Scenario 3: Schema change causes silent field drop.
    // Notification service consumes events but drops them because
    // a required field is now null after a schema change.
    public void scenarioSchemaChange(String paymentId, String email) {
        if (email == null) {
            LOG.info("paymentId=" + paymentId + " skipping notification (no email)");
            return; // silently does nothing, no error
        }
    }
}

In all three scenarios:
- Health checks: 200 OK (green)
- CPU/memory: normal
- Error rate: 0%
- Per-service dashboard: all green
- Customer experience: broken

package com.bank.metrics;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Component;

@Component
public class TransferOutcomeMetrics {

    private final Counter initiated;
    private final Counter completed;
    private final Counter failed;

    public TransferOutcomeMetrics(MeterRegistry registry) {
        this.initiated = registry.counter("transfers.initiated");
        this.completed = registry.counter("transfers.completed");
        this.failed = registry.counter("transfers.failed");
    }

    public void recordInitiated() { initiated.increment(); }
    public void recordCompleted() { completed.increment(); }
    public void recordFailed()    { failed.increment(); }
}

Prometheus metrics:
transfers_initiated_total  1523
transfers_completed_total  0     <- ALERT: completions dropped to zero
transfers_failed_total     0

Payment-success rate = completed / initiated = 0 / 1523 = 0%
Alert fires: "Transfer completion rate below 95% SLO"
All per-service dashboards are still green.

SLI: Transfer completion rate = transfers.completed / transfers.initiated
SLO: Transfer completion rate >= 99.5% over 5-minute window

Prometheus alerting rule:
alert: TransferCompletionRateLow
expr: rate(transfers_completed_total[5m]) / rate(transfers_initiated_total[5m]) < 0.995
for: 2m
labels:
  severity: critical
annotations:
  summary: "Transfer completion rate below 99.5% SLO"

package com.bank.synthetic;

import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestTemplate;
import java.math.BigDecimal;
import java.util.logging.Logger;

@Component
public class SyntheticTransferProbe {

    private static final Logger LOG = Logger.getLogger(SyntheticTransferProbe.class.getName());
    private final RestTemplate restTemplate;

    public SyntheticTransferProbe(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @Scheduled(fixedRate = 60000) // every 60 seconds
    public void runSyntheticTransfer() {
        try {
            // Use dedicated synthetic accounts that never affect real customers
            var request = new TransferRequest(
                "SYNTH-FROM-001", "SYNTH-TO-001", new BigDecimal("1.00"));

            var response = restTemplate.postForEntity(
                "http://api-gateway/api/v1/transfers", request, TransferResponse.class);

            // Assert the transfer reaches COMPLETED within 30 seconds
            String paymentId = response.getBody().getPaymentId();
            boolean completed = pollForCompletion(paymentId, 30_000);

            if (completed) {
                LOG.info("Synthetic transfer PASSED: " + paymentId);
            } else {
                LOG.severe("Synthetic transfer FAILED: " + paymentId
                    + " did not complete within 30s");
                // Fire alert
            }
        } catch (Exception e) {
            LOG.severe("Synthetic transfer ERROR: " + e.getMessage());
        }
    }

    private boolean pollForCompletion(String paymentId, long timeoutMs) {
        // Poll payment status until COMPLETED or timeout
        return true; // simplified
    }
}

Expected behavior:
Every 60 seconds: create test transfer from SYNTH-FROM-001 to SYNTH-TO-001
Assert: transfer reaches COMPLETED within 30 seconds

Settlement flag OFF -> synthetic transfer stays INITIATED -> alert fires
All per-service dashboards still green, but synthetic probe caught it.

package com.bank.antipattern;

// WRONG: Only per-service health checks. Misses composition-level failures.
public class WrongMonitoring {
    // readinessProbe: /actuator/health -> 200 OK
    // Conclusion: "Everything is fine."
    // Reality: No transfer has completed in 2 hours.
}

// CORRECT: Business metrics + synthetic journeys + per-service probes.
public class CorrectMonitoring {
    // 1. transfers.completed / transfers.initiated -> alert if < 99.5%
    // 2. Synthetic probe every 60s -> alert if transfer does not complete
    // 3. Per-service probes -> localize the root cause
}