Definition
- REQUIRES_NEW suspension -- when a REQUIRES_NEW method is called, Spring suspends the outer transaction but does NOT release its database connection. The outer connection stays checked out from HikariCP.
- Second connection -- the inner REQUIRES_NEW method needs its own database connection, so HikariCP checks out a second connection for the same thread.
- Pool deadlock -- if all pool connections are held by outer transactions, every inner method waits for a connection that can never be freed, because freeing it requires the outer method to complete, which requires the inner method to complete first.
- Sizing formula --
maxPoolSize >= threads x (maxNestedDepth - 1) + 1. For 20 threads with one level of nesting: 20 x (2-1) + 1 = 21. - HikariCP timeout -- when the pool is exhausted, HikariCP waits for
connectionTimeout(default 30 seconds) and then throwsSQLTransientConnectionException: Connection is not available.
At the National Bank of India, ten tellers each hold one vault key (connection) for their current work. When a teller needs to do a REQUIRES_NEW audit entry, they need a second key. If all ten keys are already held by tellers who each need a second key, nobody can get their second key because all keys are taken. Everyone waits forever. The vault room is full of frozen tellers -- this is a connection pool deadlock.
Understanding the Concepts
What is the REQUIRES_NEW connection pool deadlock?
REQUIRES_NEW tells Spring to suspend the current transaction and start a brand-new, independent one. Suspension does not release the outer transaction's database connection. The connection stays checked out from HikariCP so it can be resumed afterward. Meanwhile, the inner transaction needs a second connection from HikariCP. So for the duration of the inner method, one thread holds two physical connections at once.
This is safe when the pool is large enough, but dangerous when it is not.
package com.bank.service;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class TransferWithAuditService {
private final AccountRepository accountRepository;
private final AuditService auditService; // uses REQUIRES_NEW
public TransferWithAuditService(AccountRepository accountRepository,
AuditService auditService) {
this.accountRepository = accountRepository;
this.auditService = auditService;
}
@Transactional // REQUIRED -- holds connection #1
public void transfer(String fromId, String toId, BigDecimal amount) {
var from = accountRepository.findByIdForUpdate(fromId).orElseThrow();
var to = accountRepository.findByIdForUpdate(toId).orElseThrow();
from.setBalance(from.getBalance().subtract(amount));
to.setBalance(to.getBalance().add(amount));
// REQUIRES_NEW: connection #1 SUSPENDED, connection #2 checked out
auditService.logAttempt(fromId, toId, amount, "SUCCESS");
// connection #2 released, connection #1 RESUMED
}
}
Thread-1 timeline:
1. Check out connection-A from pool (outer TX)
2. Call auditService.logAttempt() [REQUIRES_NEW]
3. Connection-A SUSPENDED (still held)
4. Check out connection-B from pool (inner TX)
5. INSERT audit record using connection-B
6. COMMIT inner TX, release connection-B
7. RESUME outer TX on connection-A
8. COMMIT outer TX, release connection-A
Thread-1 holds TWO connections between steps 3 and 6.
Why does this cause a deadlock?
The failure mode is a connection-pool deadlock, which is not a database-level deadlock but an application-level deadlock caused by resource exhaustion.
Imagine ten worker threads and a HikariCP pool of ten connections. Each thread grabs its first connection for the outer transaction and then calls a REQUIRES_NEW method that asks for a second connection. All ten connections are already held by the ten outer transactions, so every thread waits for a connection that can never be freed, because freeing it requires the inner method to finish, which requires a connection.
package com.bank.config;
import org.springframework.context.annotation.Configuration;
// application.yml -- WRONG configuration
// spring:
// datasource:
// hikari:
// maximum-pool-size: 10 # only 10 connections
// # server.tomcat.threads.max: 10 # 10 request threads
Deadlock scenario with pool = 10, threads = 10:
Thread-1: holds conn-1, waiting for conn-11 (does not exist)
Thread-2: holds conn-2, waiting for conn-11 (does not exist)
Thread-3: holds conn-3, waiting for conn-11 (does not exist)
...
Thread-10: holds conn-10, waiting for conn-11 (does not exist)
All 10 connections are held by outer TXs.
All 10 threads need a second connection for REQUIRES_NEW.
Pool is empty. Nobody can proceed. Nobody can release.
After 30 seconds: SQLTransientConnectionException on all threads.
Application is effectively frozen.
What is the HikariCP sizing formula?
The formula to prevent connection pool deadlock is:
maxPoolSize >= threads x (maxNestedDepth - 1) + 1
Where threads is the maximum number of concurrent threads that can execute the REQUIRES_NEW path, and maxNestedDepth is the maximum nesting depth (1 = no nesting, 2 = one level of REQUIRES_NEW, 3 = two levels).
# application.yml -- CORRECT configuration
spring:
datasource:
hikari:
maximum-pool-size: 21 # 20 threads x (2-1) + 1 = 21
connection-timeout: 5000 # fail fast instead of waiting 30 sec
pool-name: NationalBankPool
server:
tomcat:
threads:
max: 20 # maximum request threads
Sizing examples:
20 threads, 1 level of REQUIRES_NEW (depth = 2):
maxPoolSize >= 20 x (2-1) + 1 = 21
20 threads, 2 levels of REQUIRES_NEW (depth = 3):
maxPoolSize >= 20 x (3-1) + 1 = 41
50 threads, 1 level of REQUIRES_NEW (depth = 2):
maxPoolSize >= 50 x (2-1) + 1 = 51
Rule of thumb: avoid more than 1 level of REQUIRES_NEW nesting.
How do you detect this problem?
HikariCP exposes metrics that reveal pool exhaustion before it becomes a full deadlock. The key metric is hikaricp.connections.pending, which shows how many threads are waiting for a connection.
package com.bank.config;
import com.zaxxer.hikari.HikariDataSource;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
@Component
public class ConnectionPoolHealthIndicator implements HealthIndicator {
private final HikariDataSource dataSource;
public ConnectionPoolHealthIndicator(HikariDataSource dataSource) {
this.dataSource = dataSource;
}
@Override
public Health health() {
var pool = dataSource.getHikariPoolMXBean();
int active = pool.getActiveConnections();
int idle = pool.getIdleConnections();
int waiting = pool.getThreadsAwaitingConnection();
int total = pool.getTotalConnections();
if (waiting > 0) {
return Health.down()
.withDetail("active", active)
.withDetail("idle", idle)
.withDetail("waiting", waiting)
.withDetail("total", total)
.withDetail("warning", "Threads waiting for connections")
.build();
}
return Health.up()
.withDetail("active", active)
.withDetail("idle", idle)
.withDetail("total", total)
.build();
}
}
Healthy pool (no REQUIRES_NEW contention):
{"status":"UP","active":5,"idle":15,"total":20}
Unhealthy pool (pool exhaustion starting):
{"status":"DOWN","active":20,"idle":0,"waiting":8,"total":20,
"warning":"Threads waiting for connections"}
Grafana alert rule:
hikaricp_connections_pending > 0 for 30 seconds -> page on-call
What are common mistakes with REQUIRES_NEW and pool sizing?
The most dangerous mistake is using REQUIRES_NEW on a high-throughput code path without adjusting the pool size. Developers often add REQUIRES_NEW to "be safe" without realizing the connection cost.
package com.bank.service;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class PaymentProcessingService {
private final PaymentRepository paymentRepository;
private final NotificationService notificationService;
public PaymentProcessingService(PaymentRepository paymentRepository,
NotificationService notificationService) {
this.paymentRepository = paymentRepository;
this.notificationService = notificationService;
}
// WRONG: REQUIRES_NEW on every notification in a batch loop
@Transactional
public void processBatch(java.util.List<Payment> payments) {
for (Payment p : payments) {
paymentRepository.save(p);
// Each call holds 2 connections. In a batch of 100,
// this cycles through connections rapidly and risks exhaustion.
notificationService.notifyWrong(p);
}
}
}
@Service
class NotificationService {
// WRONG: REQUIRES_NEW for notification inside a batch loop
@Transactional(propagation = Propagation.REQUIRES_NEW)
public void notifyWrong(Payment p) {
// ...
}
// CORRECT: no REQUIRES_NEW, use event after commit
public void notifyCorrect(Payment p) {
// Not transactional -- just publish an event
}
}
WRONG (REQUIRES_NEW in a loop):
Each iteration: conn-1 suspended, conn-2 acquired, conn-2 released, conn-1 resumed
100 iterations = 200 connection checkouts
Under concurrency: multiple threads doing this = pool exhaustion
CORRECT (event-driven):
Single TX for the batch, events collected
After commit: listener fires all notifications with fresh connections
1 connection per thread during the batch
What are the correct alternatives to REQUIRES_NEW?
There are three alternatives that avoid the two-connection problem:
- @TransactionalEventListener(AFTER_COMMIT) -- publish a domain event inside the transaction. The listener fires after commit in its own transaction, holding only one connection.
- Transactional Outbox -- write an outbox row in the same transaction as the business data. A relay process reads the outbox and performs the side effect in a separate connection.
- Async processing -- use
@Asyncto run the audit or notification in a separate thread after the main transaction commits.
package com.bank.service;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.event.TransactionalEventListener;
import org.springframework.transaction.event.TransactionPhase;
import java.math.BigDecimal;
@Service
public class SafeTransferService {
private final AccountRepository accountRepository;
private final ApplicationEventPublisher eventPublisher;
public SafeTransferService(AccountRepository accountRepository,
ApplicationEventPublisher eventPublisher) {
this.accountRepository = accountRepository;
this.eventPublisher = eventPublisher;
}
@Transactional // holds only ONE connection
public void transfer(String fromId, String toId, BigDecimal amount) {
var from = accountRepository.findByIdForUpdate(fromId).orElseThrow();
var to = accountRepository.findByIdForUpdate(toId).orElseThrow();
from.setBalance(from.getBalance().subtract(amount));
to.setBalance(to.getBalance().add(amount));
// Publish event -- NOT delivered until commit
eventPublisher.publishEvent(
new TransferCompletedEvent(fromId, toId, amount));
}
}
@Service
class AuditEventListener {
private final AuditRepository auditRepository;
AuditEventListener(AuditRepository auditRepository) {
this.auditRepository = auditRepository;
}
@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
@Transactional // runs in its own new TX with ONE connection
public void onTransferCommitted(TransferCompletedEvent event) {
auditRepository.save(new AuditEntry(
event.fromId(), event.toId(), event.amount(), "COMMITTED"));
}
}
With REQUIRES_NEW:
Main TX: conn-1 held
Audit TX: conn-2 held simultaneously
Peak connections per thread: 2
With AFTER_COMMIT:
Main TX: conn-1 held -> commit -> conn-1 released
Audit TX: conn-2 acquired AFTER conn-1 released
Peak connections per thread: 1
What about reproducing the deadlock in a test?
You can reproduce the pool deadlock in an integration test by setting the pool size equal to the thread count and using a CountDownLatch to ensure all threads acquire their first connection before any calls REQUIRES_NEW.
package com.bank.service;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
import java.math.BigDecimal;
import java.util.concurrent.*;
import static org.assertj.core.api.Assertions.assertThat;
@SpringBootTest
class PoolDeadlockReproductionTest {
@Autowired
private TransferWithAuditService transferService;
@DynamicPropertySource
static void configurePool(DynamicPropertyRegistry registry) {
registry.add("spring.datasource.hikari.maximum-pool-size", () -> "5");
registry.add("spring.datasource.hikari.connection-timeout", () -> "3000");
}
@Test
void poolDeadlockWithRequiresNew() throws Exception {
// 5 threads, pool of 5 -- guaranteed deadlock
ExecutorService pool = Executors.newFixedThreadPool(5);
CountDownLatch startGate = new CountDownLatch(5);
CountDownLatch endGate = new CountDownLatch(5);
for (int i = 0; i < 5; i++) {
pool.submit(() -> {
startGate.countDown(); // all threads ready
try {
startGate.await(); // fire at once
transferService.transfer("SAV2001", "CUR3001",
new BigDecimal("100"));
} catch (Exception e) {
// Expected: SQLTransientConnectionException
assertThat(e.getMessage())
.contains("Connection is not available");
} finally {
endGate.countDown();
}
});
}
boolean finished = endGate.await(10, TimeUnit.SECONDS);
assertThat(finished).isTrue(); // All threads hit timeout
}
}
Expected output:
All 5 threads acquire connection-1 (outer TX)
All 5 threads call REQUIRES_NEW audit method
Pool has 0 idle connections
All 5 threads wait for connection-2
After 3 seconds: SQLTransientConnectionException on all threads
Test passes -- deadlock reproduced and detected
What should a senior engineer know?
- HikariCP leak detection -- set
leak-detection-threshold: 10000(10 seconds) so HikariCP logs a warning if a connection is held longer than expected. This catches the two-connection pattern early. - Connection pool metrics -- export
hikaricp.connections.active,hikaricp.connections.pending, andhikaricp.connections.idleto Grafana. Alert onpending > 0. - Nested REQUIRES_NEW -- if method A (REQUIRES_NEW) calls method B (REQUIRES_NEW), one thread holds THREE connections. This escalates the formula to depth 3.
- Database-side limits -- even if HikariCP allows 100 connections, the database has its own
max_connectionslimit (PostgreSQL default: 100). Account for all application instances. - Production rule -- Pankaj, the architect at National Bank of India, enforces a rule: REQUIRES_NEW is allowed only for audit logging, and only at one level of nesting. Everything else uses AFTER_COMMIT events or the Transactional Outbox.
# application.yml -- production configuration
spring:
datasource:
hikari:
maximum-pool-size: 25
minimum-idle: 5
connection-timeout: 5000
leak-detection-threshold: 10000
pool-name: NationalBankPool
# PostgreSQL: max_connections = 100
# 4 application instances x 25 connections = 100 (fits exactly)
Production checklist:
1. Count all REQUIRES_NEW usages in the codebase
2. Determine max nesting depth (should be 1)
3. Apply formula: poolSize >= threads x depth + 1
4. Verify DB max_connections >= instances x poolSize
5. Set leak-detection-threshold to catch held connections
6. Alert on hikaricp.connections.pending > 0
Visual Overview
┌────────────────────────────────────────────────────────┐ │ Thread-1 with REQUIRES_NEW (normal operation) │ │ │ │ ┌─────────────┐ ┌─────────────┐ │ │ │ Connection-A │ │ Connection-B │ │ │ │ (outer TX) │ │ (inner TX) │ │ │ └──────┬──────┘ └──────┬──────┘ │ │ │ │ │ │ BEGIN outer TX │ │ │ debit SAV2001 │ │ │ ─── SUSPEND ───────────────►│ │ │ (conn-A held) BEGIN inner TX │ │ │ INSERT audit │ │ │ COMMIT inner TX │ │ ◄── RESUME ────────────────│ │ │ credit CUR3001 │ (conn-B released) │ │ COMMIT outer TX │ │ │ (conn-A released) │ │ └────────────────────────────────────────────────────────┘ ┌────────────────────────────────────────────────────────┐ │ Pool Deadlock (pool = 10, threads = 10) │ │ │ │ Thread-1: conn-1 HELD ─── waiting for conn-11 │ │ Thread-2: conn-2 HELD ─── waiting for conn-11 │ │ Thread-3: conn-3 HELD ─── waiting for conn-11 │ │ ... │ │ Thread-10: conn-10 HELD ── waiting for conn-11 │ │ │ │ Pool: 10/10 active, 0 idle, 10 pending │ │ Result: ALL threads frozen forever │ │ │ │ Fix: maxPoolSize >= 10 x (2-1) + 1 = 11 │ └────────────────────────────────────────────────────────┘
Key Takeaway
REQUIRES_NEW creates a connection-pool deadlock risk because it suspends the outer transaction's connection and checks out a second connection from HikariCP. One thread holds two physical connections simultaneously. If the pool size is smaller than the number of concurrent threads that can hit the REQUIRES_NEW path, all threads hold their first connection while waiting for a second connection that will never be available. Nobody releases, nobody proceeds, and the application freezes.
In Rajesh's payment processing service at the National Bank of India, 20 concurrent threads each hold connection number one for the payment transaction. When they all call the audit service with REQUIRES_NEW, each needs connection number two. With a pool of 10, all 10 connections are occupied by outer transactions, and every inner method waits for a connection that cannot be freed. The application hangs and HikariCP eventually throws "Connection is not available, request timed out." The fix is the Hikari sizing formula: maxPoolSize >= threads x (maxNestedDepth - 1) + 1. For 20 threads with one level of nesting, that is 21.
Pankaj, the architect, prefers a different approach entirely: avoid REQUIRES_NEW on high-throughput paths by using @TransactionalEventListener(AFTER_COMMIT) for audit logging or writing to an outbox table in the same transaction with an async relay. This way each thread holds only one connection at a time, and the pool pressure drops to the minimum.
Remember one sentence: REQUIRES_NEW means two connections per thread, so size HikariCP with the formula maxPoolSize >= threads x nesting depth or avoid the problem entirely with AFTER_COMMIT events.
---