Back to Performance & Debugging
Q3Question 3 of 3

What happens when all database connections are busy in Spring Boot

Answer: When all HikariCP connections are in use and a thread requests a new one, it blocks until a connection becomes available or the connectionTimeout expires, at which point HikariCP throws a SQLTransientConnectionException and the request fails.

Key Concepts
  • When all connections in the HikariCP pool are in use and a new thread requests a connection, it blocks for up to connectionTimeout milliseconds before throwing SQLTransientConnectionException.
  • Connection pool exhaustion causes cascading failures: blocked threads pile up, request latency spikes, and eventually the application becomes unresponsive.
  • The most common causes are slow queries holding connections too long, missing @Transactional boundaries that keep connections open, and connection leaks from unclosed resources.
  • HikariCP leak detection (leakDetectionThreshold) logs a warning with a stack trace when a connection is held longer than the configured threshold.
  • Monitoring hikaricp.connections.pending and hikaricp.connections.timeout metrics is the fastest way to detect exhaustion before users are affected.
  • Fixing exhaustion requires identifying the root cause: optimize slow queries, shorten transaction scope, close resources properly, or increase pool size if the database can handle it.

Definition

When all HikariCP connections are in use and a thread requests a new one, it blocks until a connection becomes available or the connectionTimeout expires, at which point HikariCP throws a SQLTransientConnectionException and the request fails.
  • Connection pool exhaustion is the state where every connection in the pool is borrowed and none are available, causing new requests to queue or fail.
  • SQLTransientConnectionException is the exception HikariCP throws when a thread waits longer than connectionTimeout for a connection from the pool.
  • Cascading failure is what happens when pool exhaustion causes thread pile-up: each waiting thread holds resources, more requests arrive, and the entire application grinds to a halt.
  • Leak detection is a HikariCP feature (leakDetectionThreshold) that logs a warning with the full stack trace when a connection is not returned within the configured time.
  • Long-running transaction is the most common cause of pool exhaustion: a @Transactional method that takes seconds to complete holds a connection for the entire duration.
  • Connection leak is a code path that borrows a connection (directly or through a DataSource) but never returns it, permanently reducing the available pool.

Think of the National Bank of India branch with only 5 teller counters. If all 5 tellers are serving customers and a 6th customer arrives, they wait in the queue. If teller 3 is stuck on a complex loan calculation for 10 minutes, the queue grows. If teller 5 went on a tea break and never came back (connection leak), the branch effectively has only 4 tellers. Eventually, the queue extends outside the building, customers start leaving (requests timing out), and the branch manager declares an emergency.

Understanding the Concepts

What exactly happens when the pool runs out of connections?

HikariCP maintains a fixed pool of connections. When your code (through Spring Data JPA, JdbcTemplate, or direct DataSource access) needs a connection, HikariCP tries to provide one from the pool. If all connections are in use, the requesting thread enters a wait state. It blocks for up to connectionTimeout milliseconds (default 30 seconds). If a connection becomes available within that time, the thread proceeds. If not, HikariCP throws a SQLTransientConnectionException.

During this wait, the thread is not doing any useful work. In a web application, that means the HTTP request is hanging. The user sees a loading spinner. If many requests arrive during this period, threads pile up, and the application can run out of HTTP threads too.

JAVA
package com.bank;

import java.sql.Connection;
import java.sql.SQLException;
import java.util.ArrayList;
import javax.sql.DataSource;

import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Component;

@Component
public class PoolExhaustionDemo implements CommandLineRunner {

    private final DataSource dataSource;

    public PoolExhaustionDemo(DataSource dataSource) {
        this.dataSource = dataSource;
    }

    @Override
    public void run(String... args) {
        var borrowedConnections = new ArrayList<Connection>();

        try {
            // Pool size is 5. Borrow all 5 connections.
            for (int i = 1; i <= 5; i++) {
                Connection conn = dataSource.getConnection();
                borrowedConnections.add(conn);
                System.out.println("Borrowed connection " + i);
            }

            System.out.println("All 5 connections are in use. Requesting one more...");

            // This will block for connectionTimeout (3 seconds), then throw
            Connection extra = dataSource.getConnection();
            extra.close();

        } catch (SQLException e) {
            System.out.println("EXCEPTION: " + e.getClass().getSimpleName());
            System.out.println("MESSAGE: " + e.getMessage());
        } finally {
            borrowedConnections.forEach(conn -> {
                try { conn.close(); } catch (SQLException ignored) {}
            });
            System.out.println("Released all connections.");
        }
    }
}
TEXT
Expected output:
Borrowed connection 1
Borrowed connection 2
Borrowed connection 3
Borrowed connection 4
Borrowed connection 5
All 5 connections are in use. Requesting one more...
EXCEPTION: SQLTransientConnectionException
MESSAGE: NationalBankPool - Connection is not available, request timed out after 3000ms
          (total=5, active=5, idle=0, waiting=1)
Released all connections.

What causes connection pool exhaustion?

There are four main causes, and each one needs a different fix.

1. Slow queries: a query that takes 5 seconds holds a connection for 5 seconds. If your pool has 10 connections and you get 3 slow queries per second, the pool drains in under 4 seconds.

2. Long transactions: a @Transactional method that does HTTP calls, file processing, or complex business logic holds a connection for the entire method duration, even during non-database work.

3. Connection leaks: code that borrows a connection (getConnection()) but does not close it in a finally block or try-with-resources permanently removes that connection from the pool.

4. N+1 queries: loading a list of 100 accounts and then lazily fetching each account's transactions generates 101 queries. Each query holds a connection briefly, but the cumulative effect slows down the entire pool.

JAVA
package com.bank;

public class ExhaustionCauseSummary {
    public static void main(String[] args) {
        System.out.println("Connection pool exhaustion causes:");
        System.out.println();
        System.out.println("1. Slow queries (5s per query, pool=10)");
        System.out.println("   3 slow queries/sec -> pool drained in 3.3 seconds");
        System.out.println("   Fix: add indexes, optimize query, add EXPLAIN ANALYZE");
        System.out.println();
        System.out.println("2. Long transactions (@Transactional on 10-second method)");
        System.out.println("   Connection held for 10s even during non-DB work");
        System.out.println("   Fix: shorten transaction scope, move non-DB work outside");
        System.out.println();
        System.out.println("3. Connection leaks (getConnection() without close())");
        System.out.println("   Each leak permanently removes one connection from pool");
        System.out.println("   Fix: use try-with-resources, enable leak detection");
        System.out.println();
        System.out.println("4. N+1 queries (101 queries for 100 accounts + transactions)");
        System.out.println("   Each query is fast, but 101 total floods the pool");
        System.out.println("   Fix: JOIN FETCH, @EntityGraph, or batch fetching");
    }
}
TEXT
Expected output:
Connection pool exhaustion causes:

1. Slow queries (5s per query, pool=10)
   3 slow queries/sec -> pool drained in 3.3 seconds
   Fix: add indexes, optimize query, add EXPLAIN ANALYZE

2. Long transactions (@Transactional on 10-second method)
   Connection held for 10s even during non-DB work
   Fix: shorten transaction scope, move non-DB work outside

3. Connection leaks (getConnection() without close())
   Each leak permanently removes one connection from pool
   Fix: use try-with-resources, enable leak detection

4. N+1 queries (101 queries for 100 accounts + transactions)
   Each query is fast, but 101 total floods the pool
   Fix: JOIN FETCH, @EntityGraph, or batch fetching

How does a long transaction cause pool exhaustion?

When you annotate a method with @Transactional, Spring obtains a database connection at the start of the method and holds it until the method returns. If the method calls an external HTTP service, sends an email, or does heavy computation between database calls, the connection sits idle but unavailable to other threads.

JAVA
package com.bank.service;

import com.bank.model.Account;
import com.bank.repository.AccountRepository;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.math.BigDecimal;

@Service
public class PaymentService {

    private final AccountRepository accountRepository;

    public PaymentService(AccountRepository accountRepository) {
        this.accountRepository = accountRepository;
    }

    // WRONG: entire method is @Transactional, connection held for ~6 seconds
    @Transactional
    public void processPayment(String fromAccount, String toAccount, BigDecimal amount) {
        // Step 1: DB query (10ms) - connection obtained here
        Account sender = accountRepository.findByAccountNumber(fromAccount)
            .orElseThrow();

        // Step 2: HTTP call to fraud check service (3 seconds)
        // Connection is IDLE but held by this transaction
        boolean isFraud = callFraudCheckService(sender, amount);
        if (isFraud) throw new RuntimeException("Fraud detected");

        // Step 3: HTTP call to notification service (2 seconds)
        // Connection still IDLE but held
        sendNotification(sender, amount);

        // Step 4: DB update (10ms) - finally using the connection again
        sender.setBalance(sender.getBalance().subtract(amount));
        accountRepository.save(sender);
        // Connection released here (after ~5.02 seconds total)
    }

    private boolean callFraudCheckService(Account account, BigDecimal amount) {
        try { Thread.sleep(3000); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
        System.out.println("Fraud check completed for " + account.getAccountNumber());
        return false;
    }

    private void sendNotification(Account account, BigDecimal amount) {
        try { Thread.sleep(2000); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
        System.out.println("Notification sent to " + account.getHolderName());
    }
}
TEXT
Expected output:
Fraud check completed for SAV2001
Notification sent to Priya Sharma

Connection held for ~5.02 seconds:
  DB query:          10ms
  Fraud check HTTP:  3000ms (connection idle)
  Notification HTTP: 2000ms (connection idle)
  DB update:         10ms
  Total:             5020ms
  Useful DB time:    20ms (0.4% of total)

How do you fix long transactions holding connections?

The fix is to move non-database work outside the @Transactional boundary. Only the actual database reads and writes should be inside the transaction.

JAVA
package com.bank.service;

import com.bank.model.Account;
import com.bank.repository.AccountRepository;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.math.BigDecimal;

@Service
public class FixedPaymentService {

    private final AccountRepository accountRepository;

    public FixedPaymentService(AccountRepository accountRepository) {
        this.accountRepository = accountRepository;
    }

    // CORRECT: transaction scope is minimal
    public void processPayment(String fromAccount, String toAccount, BigDecimal amount) {
        // Step 1: Read account (short transaction)
        Account sender = readAccount(fromAccount);

        // Step 2: HTTP calls OUTSIDE transaction (no connection held)
        boolean isFraud = callFraudCheckService(sender, amount);
        if (isFraud) throw new RuntimeException("Fraud detected");
        sendNotification(sender, amount);

        // Step 3: Update account (short transaction)
        debitAccount(fromAccount, amount);
    }

    @Transactional(readOnly = true)
    public Account readAccount(String accountNumber) {
        return accountRepository.findByAccountNumber(accountNumber)
            .orElseThrow();
    }

    @Transactional
    public void debitAccount(String accountNumber, BigDecimal amount) {
        Account account = accountRepository.findByAccountNumber(accountNumber)
            .orElseThrow();
        account.setBalance(account.getBalance().subtract(amount));
        accountRepository.save(account);
        System.out.println("Debited " + amount + " from " + accountNumber);
    }

    private boolean callFraudCheckService(Account account, BigDecimal amount) {
        try { Thread.sleep(3000); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
        System.out.println("Fraud check completed for " + account.getAccountNumber());
        return false;
    }

    private void sendNotification(Account account, BigDecimal amount) {
        try { Thread.sleep(2000); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
        System.out.println("Notification sent to " + account.getHolderName());
    }
}
TEXT
Expected output:
Fraud check completed for SAV2001
Notification sent to Priya Sharma
Debited 10000 from SAV2001

Connection usage now:
  Read transaction:  10ms (connection held 10ms)
  Fraud check HTTP:  3000ms (NO connection held)
  Notification HTTP: 2000ms (NO connection held)
  Write transaction:  20ms (connection held 20ms)
  Total connection time: 30ms (was 5020ms)

How do you detect connection leaks with HikariCP?

HikariCP has a built-in leak detection feature. Set leakDetectionThreshold to the number of milliseconds after which a borrowed connection is considered a potential leak. HikariCP logs a warning with the full stack trace of the code that borrowed the connection.

The threshold should be set higher than your longest expected query time. For most banking applications, 30 seconds (30000ms) is a good starting point.

PROPERTIES
# application.properties
spring.datasource.hikari.leak-detection-threshold=30000
JAVA
package com.bank;

import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Component;

import javax.sql.DataSource;
import java.sql.Connection;

@Component
public class LeakDemo implements CommandLineRunner {

    private final DataSource dataSource;

    public LeakDemo(DataSource dataSource) {
        this.dataSource = dataSource;
    }

    @Override
    public void run(String... args) throws Exception {
        // WRONG: connection is never closed (leak)
        Connection leakedConnection = dataSource.getConnection();
        System.out.println("Borrowed connection: " + leakedConnection);
        System.out.println("Simulating work without closing connection...");

        // After leakDetectionThreshold (30s), HikariCP logs a warning
        Thread.sleep(35000);

        // This connection is now flagged as a leak in the logs
        System.out.println("Connection still not returned to pool.");
        // FIX: leakedConnection.close();
    }
}
TEXT
Expected output (after 30 seconds):
Borrowed connection: HikariProxyConnection@1234567
Simulating work without closing connection...

WARN  com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered
  for HikariProxyConnection@1234567 on thread main,
  stack trace follows:
    java.lang.Exception: Apparent connection leak detected
      at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
      at com.bank.LeakDemo.run(LeakDemo.java:22)
      ...

Connection still not returned to pool.

What is a complete banking project simulating pool exhaustion?

This project simulates the National Bank of India payment system under load. It shows how pool exhaustion happens, how to detect it, and how to fix it.

Project structure

DIAGRAM
pool-exhaustion-demo/
├── pom.xml
└── src/
    └── main/
        ├── java/
        │   └── com/
        │       └── bank/
        │           ├── PoolExhaustionApp.java
        │           ├── model/
        │           │   └── Account.java
        │           ├── repository/
        │           │   └── AccountRepository.java
        │           └── service/
        │               ├── SlowPaymentService.java
        │               └── FixedPaymentService.java
        └── resources/
            └── application.properties

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>

    syn-kw"><parentsyn-kw">>
        syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
        syn-kw"><artifactIdsyn-kw">>spring-boot-starter-parentsyn-kw"></artifactIdsyn-kw">>
        syn-kw"><versionsyn-kw">>3.4.7syn-kw"></versionsyn-kw">>
    syn-kw"></parentsyn-kw">>

    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>pool-exhaustion-demosyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>

    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
    syn-kw"></propertiessyn-kw">>

    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-boot-starter-data-jpasyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>com.h2databasesyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>h2syn-kw"></artifactIdsyn-kw">>
            syn-kw"><scopesyn-kw">>runtimesyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

application.properties

PROPERTIES
spring.datasource.url=jdbc:h2:mem:nationalbank
spring.datasource.hikari.maximum-pool-size=3
spring.datasource.hikari.minimum-idle=3
spring.datasource.hikari.connection-timeout=3000
spring.datasource.hikari.pool-name=NationalBankPool
spring.datasource.hikari.leak-detection-threshold=5000
spring.jpa.hibernate.ddl-auto=create-drop

SlowPaymentService.java

JAVA
package com.bank.service;

import com.bank.repository.AccountRepository;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
public class SlowPaymentService {

    private final AccountRepository accountRepository;

    public SlowPaymentService(AccountRepository accountRepository) {
        this.accountRepository = accountRepository;
    }

    @Transactional
    public String processSlowPayment(String accountNumber) {
        // Connection borrowed here
        var account = accountRepository.findByAccountNumber(accountNumber);

        // Simulate slow external call while holding connection
        try { Thread.sleep(5000); } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }

        return account.map(a -> "Processed: " + a.getAccountNumber()).orElse("Not found");
        // Connection returned here after 5+ seconds
    }
}

PoolExhaustionApp.java

JAVA
package com.bank;

import com.bank.model.Account;
import com.bank.repository.AccountRepository;
import com.bank.service.SlowPaymentService;
import com.zaxxer.hikari.HikariDataSource;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

import javax.sql.DataSource;
import java.math.BigDecimal;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

@SpringBootApplication
public class PoolExhaustionApp {

    public static void main(String[] args) {
        SpringApplication.run(PoolExhaustionApp.class, args);
    }

    @Bean
    CommandLineRunner demo(AccountRepository repo, SlowPaymentService service, DataSource ds) {
        return args -> {
            repo.save(new Account("SAV2001", "Priya Sharma", new BigDecimal("50000"), "ACTIVE"));

            System.out.println("Pool size: 3, launching 6 concurrent slow payments...");
            ExecutorService executor = Executors.newFixedThreadPool(6);

            for (int i = 0; i < 6; i++) {
                int threadNum = i + 1;
                executor.submit(() -> {
                    try {
                        System.out.println("Thread-" + threadNum + " requesting connection...");
                        var result = service.processSlowPayment("SAV2001");
                        System.out.println("Thread-" + threadNum + " done: " + result);
                    } catch (Exception e) {
                        System.out.println("Thread-" + threadNum + " FAILED: " + e.getMessage());
                    }
                });
            }

            executor.shutdown();
            executor.awaitTermination(30, TimeUnit.SECONDS);

            if (ds instanceof HikariDataSource hikari) {
                var pool = hikari.getHikariPoolMXBean();
                System.out.println("\nFinal pool state:");
                System.out.println("  Active: " + pool.getActiveConnections());
                System.out.println("  Idle: " + pool.getIdleConnections());
            }
        };
    }
}
TEXT
Expected output:
Pool size: 3, launching 6 concurrent slow payments...
Thread-1 requesting connection...
Thread-2 requesting connection...
Thread-3 requesting connection...
Thread-4 requesting connection...
Thread-5 requesting connection...
Thread-6 requesting connection...
Thread-4 FAILED: NationalBankPool - Connection is not available, request timed out after 3000ms (total=3, active=3, idle=0, waiting=3)
Thread-5 FAILED: NationalBankPool - Connection is not available, request timed out after 3000ms (total=3, active=3, idle=0, waiting=2)
Thread-6 FAILED: NationalBankPool - Connection is not available, request timed out after 3000ms (total=3, active=3, idle=0, waiting=1)
Thread-1 done: Processed: SAV2001
Thread-2 done: Processed: SAV2001
Thread-3 done: Processed: SAV2001

Final pool state:
  Active: 0
  Idle: 3

What are common mistakes when dealing with pool exhaustion?

Wrong: increasing pool size without fixing the root cause

JAVA
package com.bank;

public class WrongFixPoolExhaustion {
    public static void main(String[] args) {
        System.out.println("WRONG: increase pool size to 100 to fix exhaustion");
        System.out.println();
        System.out.println("Scenario: 10 threads each hold a connection for 5 seconds");
        System.out.println("  Pool=10:  exhaustion at 10 concurrent requests");
        System.out.println("  Pool=100: exhaustion at 100 concurrent requests");
        System.out.println();
        System.out.println("You delayed the problem, not fixed it.");
        System.out.println("Now 100 connections compete for 4 DB CPU cores.");
        System.out.println("Each query takes longer, making the problem worse.");
        System.out.println();
        System.out.println("Plus: 100 DB connections use ~1GB memory on the database server.");
    }
}
TEXT
Expected output:
WRONG: increase pool size to 100 to fix exhaustion

Scenario: 10 threads each hold a connection for 5 seconds
  Pool=10:  exhaustion at 10 concurrent requests
  Pool=100: exhaustion at 100 concurrent requests

You delayed the problem, not fixed it.
Now 100 connections compete for 4 DB CPU cores.
Each query takes longer, making the problem worse.

Plus: 100 DB connections use ~1GB memory on the database server.

Correct: fix the root cause (shorten connection hold time)

JAVA
package com.bank;

public class CorrectFixPoolExhaustion {
    public static void main(String[] args) {
        System.out.println("CORRECT: fix the root cause");
        System.out.println();
        System.out.println("Step 1: Enable leak detection");
        System.out.println("  spring.datasource.hikari.leak-detection-threshold=10000");
        System.out.println();
        System.out.println("Step 2: Find slow queries");
        System.out.println("  spring.jpa.properties.hibernate.generate_statistics=true");
        System.out.println("  Look for queries > 1 second in Hibernate stats");
        System.out.println();
        System.out.println("Step 3: Shorten transaction scope");
        System.out.println("  Move HTTP calls, file I/O, computations outside @Transactional");
        System.out.println();
        System.out.println("Step 4: Add database indexes");
        System.out.println("  EXPLAIN ANALYZE the slow query, add missing index");
        System.out.println();
        System.out.println("Step 5: Fix N+1 queries");
        System.out.println("  Use JOIN FETCH or @EntityGraph");
        System.out.println();
        System.out.println("Result: connection hold time drops from 5s to 20ms");
        System.out.println("  Pool=10 can now serve 500 requests/sec instead of 2");
    }
}
TEXT
Expected output:
CORRECT: fix the root cause

Step 1: Enable leak detection
  spring.datasource.hikari.leak-detection-threshold=10000

Step 2: Find slow queries
  spring.jpa.properties.hibernate.generate_statistics=true
  Look for queries > 1 second in Hibernate stats

Step 3: Shorten transaction scope
  Move HTTP calls, file I/O, computations outside @Transactional

Step 4: Add database indexes
  EXPLAIN ANALYZE the slow query, add missing index

Step 5: Fix N+1 queries
  Use JOIN FETCH or @EntityGraph

Result: connection hold time drops from 5s to 20ms
  Pool=10 can now serve 500 requests/sec instead of 2

How do you monitor and alert on pool exhaustion in production?

Spring Boot Actuator with Micrometer exposes HikariCP metrics automatically. The key metrics to watch are hikaricp.connections.pending (threads currently waiting for a connection) and hikaricp.connections.timeout.total (cumulative count of timeouts). Set alerts on these in your monitoring system.

JAVA
package com.bank.monitoring;

import com.zaxxer.hikari.HikariDataSource;
import com.zaxxer.hikari.HikariPoolMXBean;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import javax.sql.DataSource;

@Component
public class PoolHealthChecker {

    private final DataSource dataSource;

    public PoolHealthChecker(DataSource dataSource) {
        this.dataSource = dataSource;
    }

    @Scheduled(fixedRate = 10000)
    public void checkPoolHealth() {
        if (dataSource instanceof HikariDataSource hikariDs) {
            HikariPoolMXBean pool = hikariDs.getHikariPoolMXBean();
            int active = pool.getActiveConnections();
            int total = pool.getTotalConnections();
            int waiting = pool.getThreadsAwaitingConnection();
            int maxSize = hikariDs.getMaximumPoolSize();

            double utilization = (double) active / maxSize * 100;

            System.out.printf("[Pool Health] active=%d/%d (%.0f%%), waiting=%d%n",
                active, maxSize, utilization, waiting);

            if (utilization > 80) {
                System.out.println("  WARNING: Pool utilization above 80%");
            }
            if (waiting > 0) {
                System.out.println("  CRITICAL: " + waiting + " threads waiting for connections");
            }
        }
    }
}
TEXT
Expected output (every 10 seconds):
[Pool Health] active=2/10 (20%), waiting=0
[Pool Health] active=8/10 (80%), waiting=0
  WARNING: Pool utilization above 80%
[Pool Health] active=10/10 (100%), waiting=5
  WARNING: Pool utilization above 80%
  CRITICAL: 5 threads waiting for connections

What should a senior or principal engineer know?

At the senior level, you should be able to diagnose pool exhaustion from logs and metrics, identify the root cause (slow query, long transaction, or leak), and fix it without restarting the application. You should know how to use thread dumps to find which threads are holding connections.

At the principal level, think about:

  • Bulkhead pattern: separate connection pools for critical and non-critical operations. The account balance check (critical) should never be blocked because a report generation (non-critical) is consuming all connections.
  • Circuit breaker on database calls: if the database is slow, open a circuit breaker to fail fast instead of consuming all pool connections waiting for slow responses.
  • Connection pool per read/write split: route read queries to read replicas with their own HikariCP pool, reducing load on the primary.
  • Timeout cascade: set connectionTimeout < HTTP client timeout < API gateway timeout. If the pool times out in 3 seconds, the HTTP response should return an error in 5 seconds, not hang for 30.
  • Load shedding: when pool utilization exceeds 80%, reject new requests with HTTP 503 instead of queuing them. This prevents cascading failures.
JAVA
package com.bank.config;

import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.sql.DataSource;

@Configuration
public class BulkheadPoolConfig {

    @Bean(name = "criticalPool")
    public DataSource criticalPool() {
        var config = new HikariConfig();
        config.setJdbcUrl("jdbc:postgresql://db:5432/accounts");
        config.setMaximumPoolSize(15);
        config.setMinimumIdle(15);
        config.setConnectionTimeout(3000);
        config.setPoolName("CriticalPool");
        return new HikariDataSource(config);
    }

    @Bean(name = "reportPool")
    public DataSource reportPool() {
        var config = new HikariConfig();
        config.setJdbcUrl("jdbc:postgresql://db:5432/accounts");
        config.setMaximumPoolSize(5);
        config.setMinimumIdle(2);
        config.setConnectionTimeout(10000);
        config.setPoolName("ReportPool");
        return new HikariDataSource(config);
    }
}
TEXT
Expected output:
Pool layout for National Bank:
  CriticalPool (balance, transfers): 15 connections, 3s timeout
  ReportPool   (statements, reports): 5 connections, 10s timeout

When report generation consumes all 5 report connections,
balance checks still have 15 dedicated connections available.

What are common interview follow-up questions?

  • How do you take a thread dump to find which code is holding connections? (Use jstack or send SIGQUIT to the JVM. Look for threads in WAITING state on HikariPool.getConnection(). The stack trace shows which method is waiting.)
  • Can you dynamically resize the pool without restarting? (Yes. HikariCP supports setMaximumPoolSize() through its JMX MBean. You can change the pool size at runtime.)
  • What is the difference between connectionTimeout and validationTimeout? (connectionTimeout is how long a thread waits for a connection from the pool. validationTimeout is how long HikariCP waits for a connection validation check, like isValid(), before declaring the connection dead.)
  • How does Spring Boot handle pool exhaustion in a REST controller? (The controller thread blocks waiting for a connection. If connectionTimeout expires, SQLTransientConnectionException propagates up and results in an HTTP 500 unless you handle it with @ExceptionHandler.)
  • Should you catch SQLTransientConnectionException and retry? (Yes, but with a short delay and a maximum retry count. Use exponential backoff. Do not retry indefinitely because that adds more pressure to the pool.)

Visual Overview

DIAGRAM
 Connection Pool Exhaustion Timeline
 =====================================

 Time ──────────────────────────────────────────>

 Pool (max=3):

 T=0s   [idle] [idle] [idle]   3 idle, 0 active
         │      │      │
 T=0.1s [BUSY] [BUSY] [BUSY]   0 idle, 3 active
         │      │      │        Thread-4 arrives, waits...
         │      │      │        Thread-5 arrives, waits...
         │      │      │        Thread-6 arrives, waits...
         │      │      │
 T=3.1s  │      │      │        Thread-4 TIMEOUT (3s connectionTimeout)
         │      │      │        Thread-5 TIMEOUT
         │      │      │        Thread-6 TIMEOUT
         │      │      │
 T=5.1s [idle] [idle] [idle]   Thread-1,2,3 return connections


 Cascading Failure
 ==================

 ┌─────────────┐
 │ Slow Query   │ holds connection 5 seconds
 └──────┬──────┘
        │
        ▼
 ┌─────────────┐
 │ Pool Full    │ all connections in use
 └──────┬──────┘
        │
        ▼
 ┌─────────────┐
 │ Threads Wait │ HTTP threads block
 └──────┬──────┘
        │
        ▼
 ┌─────────────┐
 │ Thread Pool  │ Tomcat threads exhausted
 │ Exhausted    │
 └──────┬──────┘
        │
        ▼
 ┌─────────────┐
 │ App Down     │ no threads to serve requests
 └─────────────┘

Key Takeaway

Connection pool exhaustion is one of the most common production issues in Spring Boot applications. It happens when every connection in the HikariCP pool is borrowed and busy, leaving new requests with no database access. The requesting threads block for up to connectionTimeout milliseconds, and if no connection becomes available, they fail with SQLTransientConnectionException. In a web application, this means HTTP requests hang, latency spikes, and users see errors or timeouts.

The root cause is almost never "the pool is too small." It is almost always that connections are held too long. A @Transactional method that calls an external HTTP service holds a database connection for the duration of that HTTP call, even though the connection is not being used. A slow query without an index holds a connection for seconds. A code path that calls dataSource.getConnection() without closing it in a finally block permanently removes a connection from the pool.

In our National Bank of India pool-exhaustion-demo, we configured a pool of 3 connections and launched 6 concurrent slow payment requests. The first 3 threads got connections and held them for 5 seconds each. The other 3 threads waited 3 seconds (connectionTimeout) and then failed with timeout exceptions. The fix was to move the HTTP calls outside the @Transactional boundary, reducing connection hold time from 5 seconds to 20 milliseconds.

At the senior level, use leak detection (leakDetectionThreshold) to catch code paths that hold connections too long, and monitor hikaricp.connections.pending in production. At the principal level, design bulkhead connection pools that separate critical operations from batch operations, set timeout cascades (pool timeout < HTTP timeout < gateway timeout), and implement load shedding to reject requests when pool utilization exceeds 80%.

Remember one sentence: connection pool exhaustion is caused by holding connections too long, and the fix is always to shorten the time between borrowing and returning a connection, not to increase the pool size.

---

Share
Use ← → arrow keys to navigate