Definition
- Spring Boot Actuator -- exposes live metrics at /actuator/metrics for request latency, connection pool usage, JVM memory, and garbage collection pauses.
- Thread dump -- a snapshot of every thread showing what each one is doing; reveals blocked threads waiting for database connections or locks.
- HikariCP connection pool -- the default database connection pool; exhaustion (all connections busy) causes requests to queue and eventually time out.
- N+1 query problem -- loading a list of accounts triggers a separate query for each account's transactions; fix with JOIN FETCH or @EntityGraph.
- Circuit breaker -- Resilience4j pattern that stops calling a failing downstream service, letting it recover instead of cascading the failure.
- Horizontal auto-scaling -- Kubernetes HPA adds more pod instances behind the load balancer when CPU or memory crosses a threshold.
When the National Bank of India branch is slow, the manager does not guess. She checks: are tellers idle (thread dump)? Is the vault door stuck (database)? Is the queue too long (pool exhaustion)? Is one customer holding up everyone (slow query)? The fix: open more windows (scale), speed up vault access (cache), and skip unnecessary steps (async). Rajesh follows the same process with a Spring Boot service.
Understanding the Concepts
How do you debug a running Spring Boot app?
The golden rule is: never guess, always measure. Rajesh, the developer at National Bank of India, follows a systematic process when the account service starts responding slowly in production.
The first step is enabling Spring Boot Actuator and exposing metrics. Actuator gives live numbers from the running application without restarting it. The single most useful metric for a database-heavy banking service is hikaricp.connections.active, which shows how many database connections are currently in use.
# application.yml -- expose Actuator metrics
management:
endpoints:
web:
exposure:
include: health,metrics,prometheus,threaddump
endpoint:
health:
show-details: always
# Check active HikariCP connections
curl http://localhost:8080/actuator/metrics/hikaricp.connections.active
# Check per-endpoint latency percentiles
curl http://localhost:8080/actuator/metrics/http.server.requests
# Get a thread dump to see blocked threads
curl http://localhost:8080/actuator/threaddump
Expected output (connection pool metric):
{
"name": "hikaricp.connections.active",
"measurements": [{"statistic": "VALUE", "value": 48.0}],
"availableTags": [{"tag": "pool", "values": ["HikariPool-1"]}]
}
If value equals maximum-pool-size, the pool is exhausted.
If the CPU is the bottleneck rather than the database, Rajesh attaches Java Flight Recorder to capture a CPU profile.
# Capture a 60-second CPU profile with Java Flight Recorder
jcmd <pid> JFR.start duration=60s filename=profile.jfr
# Open the recording in JDK Mission Control to find hot methods
Why is production slower than UAT?
Pankaj, the architect, tells Rajesh that when production is slower than UAT, the difference is almost always one of five things:
- JVM arguments differ. UAT might have a smaller heap that forces more frequent but shorter GC pauses, while production has a larger heap with longer full-GC pauses.
- Connection pool sizes differ. UAT with 5 concurrent users never exhausts a pool of 10. Production with 500 concurrent users exhausts it instantly.
- Missing database indexes. UAT has a small dataset where a full table scan is fast. Production has millions of rows.
- Caching is cold. A freshly restarted production instance has an empty cache, so every request hits the database.
- Network latency. UAT services run on the same network. Production calls go across availability zones.
package com.bank.config;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
import javax.sql.DataSource;
import java.sql.Connection;
@Component
public class DatabaseHealthCheck implements HealthIndicator {
private final DataSource dataSource;
public DatabaseHealthCheck(DataSource dataSource) {
this.dataSource = dataSource;
}
@Override
public Health health() {
try (Connection conn = dataSource.getConnection()) {
long start = System.currentTimeMillis();
conn.createStatement().execute("SELECT 1");
long elapsed = System.currentTimeMillis() - start;
return Health.up()
.withDetail("dbLatencyMs", elapsed)
.build();
} catch (Exception e) {
return Health.down(e).build();
}
}
}
GET /actuator/health
{
"status": "UP",
"components": {
"database": { "status": "UP", "details": { "dbLatencyMs": 3 } }
}
}
How do you reduce API response time?
Rajesh reduced the /api/accounts/{id}/statement endpoint from 500ms to 40ms at National Bank of India by fixing three things: N+1 queries, missing cache, and synchronous audit logging.
The N+1 problem happens when loading one account triggers a separate query for each transaction. The fix is a JOIN FETCH that loads everything in a single query.
package com.bank.repository;
import com.bank.model.Account;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import java.util.Optional;
public interface AccountRepository extends JpaRepository<Account, String> {
// WRONG: triggers N+1 -- each account loads transactions separately
// Optional<Account> findById(String id);
// CORRECT: JOIN FETCH loads account + transactions in one query
@Query("SELECT a FROM Account a JOIN FETCH a.transactions WHERE a.id = :id")
Optional<Account> findWithTransactions(@Param("id") String id);
}
Redis caching eliminates repeated calls for hot, slow-changing data like exchange rates and holiday calendars.
package com.bank.service;
import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class ExchangeRateService {
@Cacheable(value = "exchangeRates", key = "#currency")
public BigDecimal getRate(String currency) {
// This slow external API call is cached after the first hit
return externalApi.fetchRate(currency);
}
}
First call: getRate("USD") -> calls external API, takes 200ms, caches result
Second call: getRate("USD") -> returns from Redis cache, takes 2ms
How do you use @Async to move work off the request thread?
Non-critical work like sending notifications and writing audit logs should not block the API response. Rajesh moves them to a background thread pool with @Async.
package com.bank.config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import java.util.concurrent.Executor;
@Configuration
@EnableAsync
public class AsyncConfig {
@Bean(name = "notificationExecutor")
public Executor notificationExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5);
executor.setMaxPoolSize(20);
executor.setQueueCapacity(100);
executor.setThreadNamePrefix("notification-");
executor.initialize();
return executor;
}
}
package com.bank.service;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;
@Service
public class NotificationService {
@Async("notificationExecutor")
public void sendTransferAlert(String accountId, String message) {
// This runs on a background thread -- does not block the API response
emailGateway.send(accountId, message);
smsGateway.send(accountId, message);
}
}
Before @Async: POST /api/transfers takes 500ms (200ms transfer + 300ms notification)
After @Async: POST /api/transfers takes 200ms (notification runs in background)
How do you cut Spring Boot startup time?
Startup time matters for serverless deployments and auto-scaling. A service that takes 30 seconds to start cannot handle traffic spikes quickly. Rajesh uses these techniques:
- Lazy initialization defers bean creation until first use.
- Exclude unused auto-configuration prevents Spring from setting up features you do not use.
- Flyway instead of ddl-auto avoids schema generation on every startup.
- GraalVM native image eliminates JVM startup entirely.
# application.yml -- startup optimization
spring:
main:
lazy-initialization: true
autoconfigure:
exclude:
- org.springframework.boot.autoconfigure.mail.MailAutoConfiguration
- org.springframework.boot.autoconfigure.thymeleaf.ThymeleafAutoConfiguration
jpa:
hibernate:
ddl-auto: validate # Flyway handles migrations, not Hibernate
# Build a GraalVM native image for instant startup
./mvnw -Pnative native:compile
# Startup time comparison
# JVM: Started in 4.2 seconds
# GraalVM: Started in 0.08 seconds
How do you handle connection pool exhaustion (503 errors)?
When all HikariCP connections are busy, new requests wait in a queue. If the wait exceeds connection-timeout, HikariCP throws a SQLTransientConnectionException and the request returns 503. Pankaj configures HikariCP with sensible limits and short timeouts so requests fail fast instead of queuing indefinitely.
# application.yml -- HikariCP tuning
spring:
datasource:
hikari:
maximum-pool-size: 50
minimum-idle: 10
connection-timeout: 3000 # fail fast after 3 seconds
idle-timeout: 600000 # close idle connections after 10 minutes
max-lifetime: 1800000 # recycle connections every 30 minutes
leak-detection-threshold: 30000 # warn if connection held > 30 seconds
package com.bank.config;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class MetricsConfig {
@Bean
public MeterRegistryCustomizer<MeterRegistry> commonTags() {
return registry -> registry.config()
.commonTags("application", "nbi-account-service");
}
}
Alert rule (Prometheus):
hikaricp_connections_active / hikaricp_connections_max > 0.8
Message: "HikariCP pool usage above 80% -- investigate slow queries"
How do you handle sudden 10x traffic surges?
During a National Bank of India salary credit day, traffic spikes 10x within minutes. Rajesh protects the account service with three layers: rate limiting, circuit breakers, and horizontal auto-scaling.
package com.bank.service;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.ratelimiter.annotation.RateLimiter;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class TransferService {
@RateLimiter(name = "transferApi")
@CircuitBreaker(name = "coreBank", fallbackMethod = "fallback")
public String transfer(String fromAccount, String toAccount, BigDecimal amount) {
return coreBankingClient.executeTransfer(fromAccount, toAccount, amount);
}
public String fallback(String from, String to, BigDecimal amt, Throwable t) {
return "Transfer queued for retry -- core banking temporarily unavailable";
}
}
# application.yml -- Resilience4j configuration
resilience4j:
ratelimiter:
instances:
transferApi:
limit-for-period: 100
limit-refresh-period: 1s
timeout-duration: 0s # reject immediately if rate exceeded
circuitbreaker:
instances:
coreBank:
sliding-window-size: 10
failure-rate-threshold: 50
wait-duration-in-open-state: 30s
Behavior during traffic surge:
- First 100 requests per second: processed normally.
- Requests 101+: rejected immediately with 429 Too Many Requests.
- If core banking fails 50% of calls: circuit opens, fallback returns queued message.
- Kubernetes HPA detects CPU > 70%: scales from 3 to 10 pods in 60 seconds.
What are common mistakes in performance tuning?
The most common mistake is optimizing without measuring. Rajesh once spent a week caching a query that took 2ms, while the real bottleneck was a 400ms external API call.
// WRONG: caching a fast query while ignoring the slow external call
@Cacheable("accounts")
public Account findById(String id) {
return accountRepository.findById(id).orElseThrow(); // 2ms -- not the bottleneck
}
public BigDecimal getExchangeRate(String currency) {
return externalApi.fetchRate(currency); // 400ms -- this is the bottleneck
}
// CORRECT: cache the slow call, not the fast one
public Account findById(String id) {
return accountRepository.findById(id).orElseThrow(); // 2ms, no cache needed
}
@Cacheable(value = "exchangeRates", key = "#currency")
public BigDecimal getExchangeRate(String currency) {
return externalApi.fetchRate(currency); // cached, subsequent calls take 2ms
}
Another mistake is using the default SimpleAsyncTaskExecutor for @Async, which creates unlimited threads. Always configure a bounded ThreadPoolTaskExecutor.
What should a senior engineer know?
- Garbage collection impact. Long GC pauses freeze every request. Enable GC logging with
-Xlog:gc*and check for pauses that coincide with latency spikes. Consider switching from G1GC to ZGC for sub-millisecond pauses on large heaps. - Open Session in View (OSIV). Spring Boot enables OSIV by default, which keeps the Hibernate session open for the entire HTTP request. This hides N+1 problems and holds database connections longer than necessary. Disable with
spring.jpa.open-in-view=false. - Connection pool formula. PostgreSQL recommends
connections = (core_count * 2) + effective_spindle_count. For a 4-core server, start with 10 connections per instance, not 50. - Structured logging for latency tracing. Use Micrometer Tracing (formerly Sleuth) to propagate trace IDs across services, making it easy to find the slow service in a distributed call chain.
- Graceful shutdown. Configure
server.shutdown=gracefulso the service finishes in-flight requests before stopping, preventing 503 errors during deployments.
What are common interview follow-up questions?
- How do you find N+1 queries in production? (Enable Hibernate SQL logging or use a JDBC proxy like P6Spy to count queries per request.)
- What is the difference between fixedRate and fixedDelay in @Scheduled? (fixedRate fires at a constant interval regardless of task duration; fixedDelay waits after the previous execution completes.)
- How does Kubernetes HPA decide when to scale? (It monitors CPU/memory utilization or custom metrics from Prometheus and adds pods when the target threshold is exceeded.)
- What is the difference between a thread dump and a heap dump? (Thread dump shows what every thread is doing; heap dump shows every object in memory for finding memory leaks.)
Visual Overview
Debugging Workflow
══════════════════
Slow API detected
│
▼
┌──────────────────────┐
│ 1. Check Actuator │
│ /health │
│ /metrics │
└──────────┬───────────┘
│
┌─────┴──────┐
│ │
▼ ▼
┌──────────┐ ┌──────────────┐
│ DB issue │ │ CPU issue │
│ Pool │ │ GC pauses │
│ N+1 │ │ Hot methods │
└────┬─────┘ └──────┬───────┘
│ │
▼ ▼
┌──────────┐ ┌──────────────┐
│ Fix: │ │ Fix: │
│ Pool sz │ │ JFR profile │
│ JOIN │ │ Switch GC │
│ FETCH │ │ Tune heap │
│ Index │ │ │
└──────────┘ └──────────────┘
Traffic Surge Protection Layers
═══════════════════════════════
Client request
│
▼
┌────────────────┐
│ Rate Limiter │── 429 Too Many Requests
│ (Resilience4j) │
└───────┬────────┘
│
▼
┌────────────────┐
│ Circuit Breaker│── Fallback response
│ (open/half) │
└───────┬────────┘
│
▼
┌────────────────┐
│ Load Balancer │── K8s HPA auto-scales
│ (3→10 pods) │
└───────┬────────┘
│
▼
┌────────────────┐
│ HikariCP Pool │── Fail fast (3s timeout)
│ (50 conns) │
└───────┬────────┘
│
▼
┌────────────────┐
│ Database │
└────────────────┘
Key Takeaway
Debugging a slow Spring Boot app starts with measurement, not guessing. Check Actuator health for dependency status, metrics for pool usage and latency percentiles, and thread dumps for blocked threads. The most common causes at National Bank of India were missing database indexes, N+1 queries from lazy loading, HikariCP connection pool exhaustion, slow downstream calls without timeouts, and synchronous work that should have been async.
Rajesh reduced the account statement API from 500ms to 40ms through three changes: JOIN FETCH to eliminate N+1 queries, Redis caching for exchange rate lookups, and @Async for notification delivery. For traffic surges during salary credit day, the team uses Kubernetes HPA to auto-scale pods, Resilience4j rate limiting at the API level, circuit breakers on the core banking downstream call, and HikariCP timeouts that fail fast rather than queue indefinitely.
Remember one sentence: measure the bottleneck first with Actuator and profilers, then fix the one real problem with caching, async, or pool tuning.
---