If the application suddenly becomes very slow, how do you debug it systematically

package com.bank.monitoring;

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import org.springframework.stereotype.Component;

@Component
public class GoldenSignalMetrics {

    private final Timer requestTimer;

    public GoldenSignalMetrics(MeterRegistry registry) {
        this.requestTimer = Timer.builder("http.server.requests")
            .description("Request duration")
            .tag("endpoint", "all")
            .register(registry);
    }

    // Golden signals to check first:
    // 1. Latency: P50, P95, P99 per endpoint
    // 2. Traffic: requests/sec (did it spike?)
    // 3. Errors: error rate (did it jump?)
    // 4. Saturation: CPU, memory, pool utilization
}

Scoping questions:
  When did it start?           -> correlate with deploy, cron, spike
  Which endpoints are slow?    -> all or just /api/transfers?
  Which instances are slow?    -> all or just pod-3?
  Did traffic increase?        -> latency rose at same throughput?
  Is it getting worse?         -> gradual (leak) or sudden (deploy)?

# Check CPU per pod
kubectl top pods -n banking

# If CPU is high: use async-profiler to find hot methods
java -agentpath:/path/to/async-profiler/libasyncProfiler.so=start,event=cpu,file=profile.html -jar app.jar

# If CPU is low but latency is high:
# The system is WAITING, not computing
# Look at I/O, locks, database, external APIs

CPU diagnostic:
  High CPU (90%+):  code doing too much work
    -> profile with async-profiler to find hot method
    -> common causes: tight loop, heavy serialization

  Low CPU (5-10%) + high latency:
    -> threads are WAITING, not working
    -> look at: database, external APIs, thread locks
    -> thread dump will show what they are waiting for

# Check GC metrics via JMX or Actuator
curl http://localhost:8080/actuator/metrics/jvm.gc.pause
curl http://localhost:8080/actuator/metrics/jvm.memory.used

# Enable GC logging
java -Xlog:gc*:file=gc.log:time -jar app.jar

# Take a heap dump for analysis
jmap -dump:format=b,file=heapdump.hprof $(pgrep -f app.jar)
# Analyze with Eclipse MAT: find what retains memory

Memory diagnostic:
  Heap usage trending upward, never dropping -> memory leak
  GC pause time increasing -> GC thrashing
  After deploy with rising old-gen -> classic new leak

  Action:
  1. Check GC logs for pause time and frequency
  2. Take heap dump BEFORE restarting (restart erases evidence)
  3. Open in Eclipse MAT -> find largest retained objects
  4. Common culprits: growing cache, unclosed streams, static maps

-- PostgreSQL: find slow queries
SELECT pid, now() - pg_stat_activity.query_start AS duration,
       query, state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 seconds'
ORDER BY duration DESC;

-- Check for lock waits
SELECT blocked.pid AS blocked_pid,
       blocking.pid AS blocking_pid,
       blocked.query AS blocked_query
FROM pg_stat_activity blocked
JOIN pg_locks blocked_locks ON blocked.pid = blocked_locks.pid
JOIN pg_locks blocking_locks ON blocked_locks.locktype = blocking_locks.locktype
JOIN pg_stat_activity blocking ON blocking_locks.pid = blocking.pid
WHERE NOT blocked_locks.granted;

package com.bank.monitoring;

import com.zaxxer.hikari.HikariDataSource;
import com.zaxxer.hikari.HikariPoolMXBean;
import org.springframework.stereotype.Component;

@Component
public class ConnectionPoolMonitor {

    private final HikariDataSource dataSource;

    public ConnectionPoolMonitor(HikariDataSource dataSource) {
        this.dataSource = dataSource;
    }

    public void logPoolStatus() {
        HikariPoolMXBean pool = dataSource.getHikariPoolMXBean();
        System.out.println("Active connections: " +
            pool.getActiveConnections());
        System.out.println("Idle connections:   " +
            pool.getIdleConnections());
        System.out.println("Waiting threads:    " +
            pool.getThreadsAwaitingConnection());
        System.out.println("Total connections:  " +
            pool.getTotalConnections());

        // DANGER: active == total AND waiting > 0
        // means pool is exhausted!
    }
}

Database diagnostic:
  Slow query log shows query taking 10s -> missing index?
  Active connections == max pool size -> pool exhaustion
  Threads waiting for connection -> everything queues

  Key insight: a query that got slow holds connections longer,
  which cascades into pool exhaustion, which starves all requests.
  Chase the ROOT (slow query), not the SYMPTOM (pool exhaustion).

Distributed trace waterfall:
  [Transfer API]        total: 5200ms
    [Validate]          10ms
    [Fraud Check]       200ms
    [Account Service]   4800ms  <-- THIS is the bottleneck
    [Notification]      190ms

  Account service latency spike caused transfer slowness.
  Fix: add timeout + circuit breaker to account service call.

# Take thread dump
jstack $(pgrep -f app.jar) > threaddump.txt

# Take 3 dumps 5 seconds apart to see if threads are stuck
for i in 1 2 3; do
  jstack $(pgrep -f app.jar) > threaddump_$i.txt
  sleep 5
done

# Look for patterns:
grep -c "BLOCKED" threaddump.txt
grep -c "WAITING" threaddump.txt
grep -c "RUNNABLE" threaddump.txt

Thread dump analysis:
  180 threads WAITING on database connection pool
  15 threads RUNNABLE (serving requests)
  5 threads BLOCKED on a lock

  Interpretation: pool exhaustion!
  180 threads waiting for connections because a slow query
  holds each connection too long.

  Two dumps 5s apart with same threads in same wait:
  -> threads are STUCK, not progressing
  -> confirms the starvation diagnosis

package com.bank.logging;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import jakarta.servlet.FilterChain;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.util.UUID;

@Component
public class RequestTracingFilter extends OncePerRequestFilter {

    private static final Logger log =
        LoggerFactory.getLogger(RequestTracingFilter.class);

    @Override
    protected void doFilterInternal(HttpServletRequest request,
                                     HttpServletResponse response,
                                     FilterChain chain)
            throws Exception {
        String correlationId = UUID.randomUUID().toString();
        MDC.put("correlationId", correlationId);
        long start = System.currentTimeMillis();

        try {
            chain.doFilter(request, response);
        } finally {
            long duration = System.currentTimeMillis() - start;
            log.info("endpoint={} status={} duration={}ms",
                request.getRequestURI(),
                response.getStatus(),
                duration);
            MDC.remove("correlationId");
        }
    }
}

Structured log output:
  correlationId=abc-123 endpoint=/api/transfers status=200 duration=5200ms
  correlationId=abc-123 db.query=SELECT balance duration=4800ms
  -> 92% of time spent in database query -> investigate that query

WRONG:
  Service is slow -> restart it -> slowness gone temporarily
  Root cause unknown -> happens again next week
  No evidence to investigate

CORRECT:
  Service is slow -> capture thread dump + heap dump + GC logs
  -> analyze -> find slow query with missing index
  -> add index -> verify latency drops
  -> root cause fixed permanently

  Always capture evidence BEFORE restarting!