Back to Observability: Logging, Tracing, Metrics
Q9Question 9 of 9

How do you test a microservice's failure behavior as a failure contract?

Key Concepts
  • A failure contract is an explicit specification of how the service behaves when each dependency or its own process fails.
  • Each test injects ONE specific failure and asserts ONE named hypothesis about the outcome.
  • Dependency latency injection: assert the service abandons at its deadline and takes the correct fallback.
  • Duplicate delivery: deliver the same command twice and assert the side effect happened exactly once.
  • Process kill between steps: crash in the exact window and assert the system converges to a consistent state.
  • Assert the INVARIANT, not just the absence of an exception (e.g., 'total money unchanged').
  • Use failure-contract tests in CI for known guarantees; use chaos game-days to discover unknown weaknesses.

Definition

Test failure as a contract: inject one specific failure and assert one named guarantee, rather than breaking things randomly and hoping to spot trouble.
  • Failure contract – an explicit specification of how the service behaves when each dependency fails.
  • Targeted hypothesis – each test injects one failure and asserts one named outcome.
  • Dependency latency – inject slow response; assert deadline + fallback behavior.
  • Duplicate delivery – deliver same message twice; assert side effect happened once (idempotency).
  • Process kill – crash between two durable steps; assert recovery converges to consistent state.
  • Cache loss – flush cache; assert correct answers + DB stays within connection budget.
  • Broker unavailability – make Kafka refuse; assert event is not lost (outbox persists it).

National Bank of India does not test safety by randomly shutting off power. It writes a rulebook: 'If the vault jams, no cash is lost. If two tellers process the same cheque, customer is debited once. If power dies mid-transfer, the ledger is fully updated or not at all.' Then auditors stage each exact scenario and check the exact promise.

Understanding the Concepts

What is a failure contract?

Before testing, enumerate what the service promises for each failure mode.

JAVA
// Failure contract for TransferService:
// 1. If fraud service times out → transfer HELD_FOR_REVIEW, never silently approved
// 2. If same transfer delivered twice → balance changed by exactly one transfer amount
// 3. If process dies after debit but before credit → recovery completes or reverses
// 4. If cache is cold → balances correct, DB within connection budget
// 5. If Kafka is down → event persisted to outbox, published on recovery

How do you test dependency latency?

JAVA
@Test
void transferHeldWhenFraudTimesOut() {
    fraudServiceStub.setLatency(Duration.ofSeconds(5)); // inject latency
    TransferResult result = transferService.execute("SAV2001", "SAV3002",
        new BigDecimal("50000"));
    assertEquals("HELD_FOR_REVIEW", result.getStatus());
    assertTrue(result.getDuration().toMillis() < 1500); // abandoned within deadline
}

How do you test duplicate delivery (idempotency)?

JAVA
@Test
void duplicateTransferDebitedOnce() {
    String txnId = "TXN-001";
    TransferEvent event = new TransferEvent(txnId, "SAV2001", new BigDecimal("50000"));
    BigDecimal before = accountRepo.findBalance("SAV2001");

    consumer.handle(event); // first delivery
    consumer.handle(event); // duplicate delivery

    BigDecimal after = accountRepo.findBalance("SAV2001");
    assertEquals(before.subtract(new BigDecimal("50000")), after);
    // Debited exactly ONCE, not twice
}

How do you test crash recovery?

JAVA
@Test
void moneyConservedAfterMidTransferCrash() {
    BigDecimal totalBefore = totalMoneyInSystem();

    // Simulate crash after debit, before credit
    transferService.executeWithCrashAfterDebit("SAV2001", "SAV3002",
        new BigDecimal("50000"));

    // Run recovery
    recoveryJob.run();

    BigDecimal totalAfter = totalMoneyInSystem();
    assertEquals(totalBefore, totalAfter); // INVARIANT: total money unchanged
}

How do you test broker unavailability?

JAVA
@Test
void eventPersistedToOutboxWhenKafkaDown() {
    kafkaBroker.shutdown();
    transferService.execute("SAV2001", "SAV3002", new BigDecimal("50000"));
    assertTrue(outboxRepo.existsByAggregateId("SAV2001")); // event safe in outbox

    kafkaBroker.start();
    outboxRelay.run();
    assertTrue(kafkaConsumer.hasEvent("SAV2001")); // published on recovery
}

What separates this from chaos engineering?

| Approach | Injects | Asserts | Runs in |
|---|---|---|---|
| Failure-contract test | One known fault | One named guarantee | CI (deterministic) |
| Chaos engineering | Random faults | Discovers unknowns | Staging/prod |

Use both: contract tests lock in known guarantees; chaos game-days discover the ones you did not think of.

What are common mistakes?

JAVA
// WRONG: asserting only "no exception thrown"
assertDoesNotThrow(() -> transferService.execute(...));
// This is WEAK: the service could silently lose money

// CORRECT: asserting the invariant
assertEquals(totalBefore, totalAfter); // total money unchanged

What should a senior engineer know?

  • Each failure-contract test asserts the invariant, not just the absence of an exception.
  • Tests are deterministic and run in CI; a regression fails the build.
  • When a chaos game-day reveals a new weakness, promote it into a deterministic contract test.
  • The contract documents the guarantee; the test enforces it.

Visual Overview

DIAGRAM
  Failure Contract Testing

  ┌─────────────────────────────────────────┐
  │ Write the contract:                      │
  │ 1. Fraud timeout → HELD_FOR_REVIEW      │
  │ 2. Duplicate → debited once              │
  │ 3. Crash mid-transfer → money conserved  │
  │ 4. Cache cold → correct + DB safe        │
  │ 5. Kafka down → event in outbox          │
  └───────────────────┬─────────────────────┘
                      │
                      ▼
  ┌─────────────────────────────────────────┐
  │ For each: inject ONE fault              │
  │           assert ONE hypothesis          │
  │           run in CI (deterministic)      │
  └───────────────────┬─────────────────────┘
                      │
                      ▼
  ┌─────────────────────────────────────────┐
  │ Passing = guarantee PROVEN              │
  │ Failing = precise bug EXPOSED           │
  └─────────────────────────────────────────┘

Key Takeaway

A failure contract is an explicit specification of how the service behaves when each dependency fails. Testing it means injecting one specific failure and asserting one precise hypothesis. Dependency latency injection tests deadline and fallback behavior. Duplicate delivery tests idempotency. Process kills between durable steps test crash recovery. Cache loss tests correctness without the cache. Broker unavailability tests outbox durability.

The discipline that separates this from chaos is the targeted hypothesis. Each test asserts the invariant ('total money unchanged'), not just the absence of an exception. Tests are deterministic and run in CI, so guarantees are locked in and regressions fail the build. Chaos game-days discover failure modes you did not anticipate, and when they reveal a new weakness, promote it into a contract test.

At National Bank of India, duplicate messages once double-debited a customer, a crash left money temporarily missing, and a broker outage dropped events. Each incident became a named failure-contract test so the same failure could never reach production unnoticed again.

Remember one sentence: test failure as a contract by injecting one specific failure and asserting one named invariant, because a targeted test proves a guarantee when it passes and pinpoints a bug when it fails.

---

Share
Use ← → arrow keys to navigate