Chaos engineering during load tests?

• Chaos engineering – deliberately injecting faults into a running system to verify graceful degradation and automatic recovery.
• CPU throttling – simulates a noisy neighbour or misbehaving GC stealing cycles, revealing unbounded-queue growth.
• Network latency injection – adds delay between services, draining connection pools and exposing timeout misconfiguration.
• SLIs – latency (p50, p95, p99), error rate, throughput, and saturation (CPU, heap, queue depth, active connections).
• Phased playbook – step (find capacity knee), spike (test burst handling), MTTR (measure recovery), soak (expose leaks over 24 hours).

What is chaos engineering?

Chaos engineering is the disciplined practice of deliberately injecting faults into a running system to verify that the system degrades gracefully and recovers automatically. Every experiment starts with a written hypothesis that predicts what will happen. Without a hypothesis you are vandalising; with one, you are running a controlled experiment.

How does CPU throttling reveal queue issues?

CPU throttling simulates a noisy neighbour or a misbehaving GC cycle that steals CPU cycles. A throttled JVM's thread pool processes tasks slower, so the queue grows. If the queue is unbounded, memory climbs until OutOfMemoryError crashes the process.

package com.bank.resilience;

import java.util.concurrent.*;

public class BoundedPoolExample {
    // Bounded queue with CallerRunsPolicy applies natural backpressure
    private static final ThreadPoolExecutor pool = new ThreadPoolExecutor(
        10, 10, 0L, TimeUnit.MILLISECONDS,
        new ArrayBlockingQueue<>(100),
        new ThreadPoolExecutor.CallerRunsPolicy()
    );

    public void submitSettlement(Runnable task) {
        pool.execute(task);
        // When queue is full, CallerRunsPolicy makes the caller execute the task
        // This slows the caller, providing natural backpressure
    }
}

How does network latency injection drain pools?

Adding 200 ms between the settlement service and PostgreSQL makes each database call hold its connection longer. The HikariCP pool of ten connections drains fast under load, producing Connection is not available, request timed out.

# Chaos Mesh: inject 200ms delay to PostgreSQL
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: db-latency
spec:
  action: delay
  mode: all
  selector:
    labelSelectors:
      app: settlement-service
  delay:
    latency: "200ms"
  direction: to
  externalTargets:
    - "postgresql.bank.svc"

What SLIs do you watch?

| SLI | What It Shows |
|-----|---------------|
| p50, p95, p99, p99.9 latency | Request duration distribution |
| Error rate (5xx, timeouts) | Failure percentage |
| Throughput (RPS) | Requests actually completed |
| CPU usage | Compute saturation |
| Heap used, GC pause time | Memory pressure |
| Queue depth | Work backing up |
| Active connection count | Pool exhaustion |

What are the four test phases?

The step phase ramps load in plateaus (1K, 2K, 4K RPS) to find the capacity knee where latency rises sharply. The spike phase slams a sudden surge to test burst handling and auto-scaling triggers. The MTTR phase removes the fault and measures how long until SLIs return to baseline: healthy services recover in seconds, sick services need restarts. The soak phase runs steady moderate load for 24 hours to expose slow leaks such as a thread pool that never shrinks, a ThreadLocal that is never cleaned, or direct byte buffers that accumulate.

How do you write a chaos hypothesis for a banking system?

A well-written hypothesis follows this format: "Given the settlement service under 2000 RPS steady load, when we inject 200ms latency to the database for 60 seconds, we expect error rate to stay below 1% because the circuit breaker will open after 5 failed calls and the service will return cached results."

package com.bank.chaos;

import java.math.BigDecimal;

public class ChaosHypothesis {
    // Hypothesis: circuit breaker opens within 5 seconds
    // Steady state: p99 < 500ms, error rate < 0.1%
    // Fault: 200ms DB latency injection
    // Expected: p99 rises to ~700ms, error rate stays < 1%
    // Abort condition: error rate > 5% or p99 > 2000ms

    public BigDecimal getSettlementAmount(String tradeId) {
        return circuitBreaker.executeSupplier(() ->
            settlementRepo.findAmount(tradeId)
        );
    }
}

What tools are available?

Chaos Mesh is Kubernetes-native: you declare chaos as YAML custom resources and the operator injects faults. Gremlin is a hosted SaaS with a UI, API, and "halt all attacks" button with audit trails that matter in banking compliance. AWS Fault Injection Simulator (FIS) integrates with CloudWatch for managed chaos on AWS infrastructure.

What are common mistakes?

The wrong approach injects faults without a hypothesis, without monitoring SLIs, or without an abort condition. This risks turning a chaos experiment into a real outage.

The correct approach follows a strict checklist: write the hypothesis, define the abort condition, inject one fault at a time, watch SLIs in real time, remove the fault to measure MTTR, and document findings.

What are common interview follow-up questions?

• How do you decide the blast radius for a chaos experiment? Start with a single pod, expand to a percentage of replicas, never start at 100%.
• How do circuit breakers interact with chaos testing? Circuit breakers are the primary defense mechanism that chaos testing validates.
• What is the difference between chaos engineering and fault injection testing? Chaos engineering runs on production or production-like systems under real load. Fault injection testing can be done in isolation.