(Scenario) A race condition causes the SAME payment to be processed twice: how do you prevent it end to end | Microservices

Definition

Stop double charges with an idempotency key the server can recognize, a unique constraint on the transaction ID as the atomic last line of defense, and a single transaction that commits the charge and the key together.

Race condition – Two requests both check whether the payment was processed, both see no, and both charge.
Idempotency key – A client-generated unique key per logical payment, claimed atomically so only the first caller wins.
Unique constraint – A database-enforced uniqueness on the transaction ID that refuses a second row even if application code races.
Single transaction – The charge, the ledger update, and the idempotency record commit together so you never have money moved without a record.
Stored result replay – On retry, the server returns the stored result for the known key instead of charging again.

When Rajesh pays at National Bank of India, the bank stamps his request with a unique ticket number. If the same ticket comes back, whether because Rajesh's phone retried or the card network retried, the clerk looks up the ticket, sees it was already handled, and hands back the exact same receipt instead of charging again. The ledger also physically refuses two entries with the same ticket number, so even if two clerks act at the same instant, only one entry can exist.

Understanding the Concepts

What is the race condition precisely?

Two requests for the same logical payment arrive almost simultaneously. Both check whether the payment was processed, both see no, and both proceed to charge.

JAVA

package com.bank.payment;

public class RaceConditionDemo {
    public static void main(String[] args) {
        System.out.println("The race:");
        System.out.println("  Thread A: reads 'payment not processed' at t=0ms");
        System.out.println("  Thread B: reads 'payment not processed' at t=1ms");
        System.out.println("  Thread A: charges Rajesh at t=5ms");
        System.out.println("  Thread B: charges Rajesh at t=6ms");
        System.out.println("  Result: Rajesh charged TWICE for the same payment");
        System.out.println();
        System.out.println("Root cause: check-then-act is NOT atomic");
        System.out.println("Fix: make the claim atomic (single INSERT or SET NX)");
    }
}

TEXT

Expected output:
The race:
  Thread A: reads 'payment not processed' at t=0ms
  Thread B: reads 'payment not processed' at t=1ms
  Thread A: charges Rajesh at t=5ms
  Thread B: charges Rajesh at t=6ms
  Result: Rajesh charged TWICE for the same payment

Root cause: check-then-act is NOT atomic
Fix: make the claim atomic (single INSERT or SET NX)

How does the idempotency key prevent the first caller from racing?

The client sends a unique key. The server claims it atomically with a single INSERT. If the insert succeeds, this is the first caller. If it fails as a duplicate, someone else owns this operation.

JAVA

package com.bank.payment;

import org.springframework.dao.DuplicateKeyException;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;

@Service
public class IdempotentPaymentService {

    private final IdempotencyKeyRepository keyRepo;
    private final PaymentRepository paymentRepo;

    public IdempotentPaymentService(IdempotencyKeyRepository keyRepo,
                                     PaymentRepository paymentRepo) {
        this.keyRepo = keyRepo;
        this.paymentRepo = paymentRepo;
    }

    @Transactional
    public PaymentResult charge(String idempotencyKey, String accountId, BigDecimal amount) {
        // Step 1: Atomic claim of the idempotency key
        try {
            keyRepo.save(new IdempotencyRecord(idempotencyKey, "IN_PROGRESS"));
        } catch (DuplicateKeyException e) {
            // Someone else owns this key
            IdempotencyRecord existing = keyRepo.findById(idempotencyKey).orElseThrow();
            if ("COMPLETED".equals(existing.getStatus())) {
                System.out.println("Replaying stored result for key: " + idempotencyKey);
                return existing.getStoredResult();
            }
            // Still in progress: return 409 so client retries later
            throw new PaymentInProgressException(idempotencyKey);
        }

        // Step 2: Process the charge
        Payment payment = new Payment(accountId, amount);
        paymentRepo.save(payment);

        // Step 3: Store result against the key (same transaction)
        IdempotencyRecord record = keyRepo.findById(idempotencyKey).orElseThrow();
        record.setStatus("COMPLETED");
        record.setStoredResult(new PaymentResult(payment.getId(), "CHARGED", amount));
        keyRepo.save(record);

        System.out.println("Payment charged: " + amount + " to " + accountId);
        return record.getStoredResult();
    }
}

TEXT

Expected output (first caller):
Payment charged: 50000.00 to SAV2001

Expected output (concurrent second caller):
409 Conflict: Payment in progress for key IK-001

Expected output (retry after completion):
Replaying stored result for key: IK-001

Why is the unique constraint the hard backstop?

Even if application logic races, the database allows only one row with a given transaction ID. The second insert fails atomically.

SQL

CREATE TABLE payments (
    id BIGSERIAL PRIMARY KEY,
    transaction_id VARCHAR(50) NOT NULL,
    account_id VARCHAR(20) NOT NULL,
    amount DECIMAL(15,2) NOT NULL,
    created_at TIMESTAMP DEFAULT NOW(),
    CONSTRAINT uk_transaction_id UNIQUE (transaction_id)
);

JAVA

package com.bank.payment;

public class UniqueConstraintBackstop {
    public static void main(String[] args) {
        System.out.println("Hard backstop: UNIQUE(transaction_id) in database");
        System.out.println();
        System.out.println("Even if two threads both pass the idempotency key check:");
        System.out.println("  Thread A: INSERT payment (TXN-001) -> SUCCESS");
        System.out.println("  Thread B: INSERT payment (TXN-001) -> DUPLICATE KEY ERROR");
        System.out.println("  Thread B catches the error -> returns 'already processed'");
        System.out.println();
        System.out.println("This guarantee lives in the DATABASE, not in application code");
    }
}

TEXT

Expected output:
Hard backstop: UNIQUE(transaction_id) in database

Even if two threads both pass the idempotency key check:
  Thread A: INSERT payment (TXN-001) -> SUCCESS
  Thread B: INSERT payment (TXN-001) -> DUPLICATE KEY ERROR
  Thread B catches the error -> returns 'already processed'

This guarantee lives in the DATABASE, not in application code

Why must the charge and the record commit together?

If they are separate operations, a crash between them leaves money moved but no record, or a record with no money.

JAVA

package com.bank.payment;

public class TransactionManagement {
    // WRONG: two separate operations
    // paymentRepo.save(payment);  // committed
    // --- CRASH HERE ---
    // keyRepo.save(record);       // never committed
    // Result: money moved but no idempotency record
    // Next retry charges AGAIN

    // CORRECT: one @Transactional method
    // @Transactional
    // public PaymentResult charge(...) {
    //     paymentRepo.save(payment);
    //     keyRepo.save(record);
    // }  // BOTH commit together or BOTH roll back

    public static void main(String[] args) {
        System.out.println("WRONG: charge + record in separate transactions");
        System.out.println("  Crash between them -> money moved, no record -> double charge on retry");
        System.out.println();
        System.out.println("CORRECT: charge + record in ONE @Transactional");
        System.out.println("  Both commit or both rollback -> consistent always");
    }
}

TEXT

Expected output:
WRONG: charge + record in separate transactions
  Crash between them -> money moved, no record -> double charge on retry

CORRECT: charge + record in ONE @Transactional
  Both commit or both rollback -> consistent always

When do you use a distributed lock?

Only when the contended resource is NOT a database row you can constrain, like a one-time external coupon or a rate-limited gateway slot.

JAVA

package com.bank.payment;

public class WhenToUseLock {
    public static void main(String[] args) {
        System.out.println("Use distributed lock ONLY when:");
        System.out.println("  - Resource is NOT a DB row (external coupon, API slot)");
        System.out.println("  - Unique constraint is not possible");
        System.out.println();
        System.out.println("For ordinary payments with a transaction_id:");
        System.out.println("  UNIQUE constraint is cheaper and safer than a Redis lock");
        System.out.println("  No TTL hazards, survives crashes, enforced by DB");
    }
}

TEXT

Expected output:
Use distributed lock ONLY when:
  - Resource is NOT a DB row (external coupon, API slot)
  - Unique constraint is not possible

For ordinary payments with a transaction_id:
  UNIQUE constraint is cheaper and safer than a Redis lock
  No TTL hazards, survives crashes, enforced by DB

How do you handle retries from the client or payment provider?

Once the first request completes, store the full response against the idempotency key. Any subsequent request with the same key returns the stored result verbatim.

JAVA

package com.bank.payment;

public class RetryHandling {
    public static void main(String[] args) {
        System.out.println("Retry handling:");
        System.out.println("  1. First request: process charge, store result against key");
        System.out.println("  2. Client retry (same key): return stored result, no new charge");
        System.out.println("  3. Payment provider webhook retry: same idempotency key, same replay");
        System.out.println();
        System.out.println("In-progress handling:");
        System.out.println("  If key exists with IN_PROGRESS status:");
        System.out.println("    Return 409 Conflict -> client retries in 1 second");
        System.out.println("    OR wait on the row lock until first caller finishes");
    }
}

TEXT

Expected output:
Retry handling:
  1. First request: process charge, store result against key
  2. Client retry (same key): return stored result, no new charge
  3. Payment provider webhook retry: same idempotency key, same replay

In-progress handling:
  If key exists with IN_PROGRESS status:
    Return 409 Conflict -> client retries in 1 second
    OR wait on the row lock until first caller finishes

What is the atomic conditional update pattern?

A single SQL statement that succeeds only if the precondition holds, giving single-winner semantics without a separate lock.

SQL

-- Atomic conditional update: debit only if balance >= amount
UPDATE accounts
SET balance = balance - 50000.00
WHERE account_id = 'SAV2001'
  AND balance >= 50000.00;

-- If another thread already debited, balance < 50000
-- -> WHERE clause fails, 0 rows updated, no double charge

JAVA

package com.bank.payment;

public class AtomicConditionalUpdate {
    public static void main(String[] args) {
        System.out.println("Atomic conditional update:");
        System.out.println("  UPDATE accounts SET balance = balance - 50000");
        System.out.println("  WHERE account_id = 'SAV2001' AND balance >= 50000");
        System.out.println();
        System.out.println("Thread A: balance=100000, condition true -> debit succeeds");
        System.out.println("Thread B: balance=50000, condition true -> debit succeeds");
        System.out.println("Thread C: balance=0, condition false -> 0 rows updated");
        System.out.println("  -> No overdraft, no double charge");
    }
}

TEXT

Expected output:
Atomic conditional update:
  UPDATE accounts SET balance = balance - 50000
  WHERE account_id = 'SAV2001' AND balance >= 50000

Thread A: balance=100000, condition true -> debit succeeds
Thread B: balance=50000, condition true -> debit succeeds
Thread C: balance=0, condition false -> 0 rows updated
  -> No overdraft, no double charge

What should a senior engineer know?

Give idempotency records a TTL appropriate to the retry window. Make the key scope correct: it identifies the logical operation, not the HTTP attempt.

TEXT

Senior checklist:
  [x] Atomic idempotency key claim (INSERT or SET NX, not read-then-write)
  [x] UNIQUE constraint on transaction_id as hard backstop
  [x] Charge + record in single @Transactional
  [x] In-progress handling: wait or 409, never double-charge
  [x] Stored result replay for all retry sources
  [x] Key TTL: 24 hours (covers retry window)
  [x] Key scope: per logical operation, not per HTTP attempt
  [x] Async consumers: same idempotency discipline for at-least-once

Visual Overview

DIAGRAM

  Double-Charge Prevention Flow:

  ┌────────────┐  Idempotency-Key: IK-001
  │ Client     │──────────────────────────────┐
  │ (or retry) │                              │
  └────────────┘                              ▼
                                    ┌──────────────────┐
                                    │ Atomic Claim      │
                                    │ INSERT INTO       │
                                    │ idempotency_keys  │
                                    │ (key, IN_PROGRESS)│
                                    └─────┬──────┬──────┘
                                   success│      │ duplicate
                                          │      │
                              ┌───────────▼┐   ┌─▼──────────┐
                              │ Process    │   │ Return     │
                              │ Charge     │   │ stored     │
                              │ + Store    │   │ result     │
                              │ result     │   │ (no charge)│
                              └─────┬──────┘   └────────────┘
                                    │
                              ┌─────▼──────────────────┐
                              │ UNIQUE(transaction_id) │
                              │ Hard backstop in DB    │
                              │ (catches any race)     │
                              └────────────────────────┘

Key Takeaway

The race is between the check and the write: two requests both see "not processed" and both charge. The whole design is about collapsing that window. The outermost defense is an idempotency key claimed atomically with a single INSERT: if it succeeds you are the first, if it fails as a duplicate, someone else owns this operation. The hard backstop is a UNIQUE constraint on the transaction ID, because even if application logic races, the database allows only one row.

The charge, the ledger update, and the idempotency record must commit in one transaction so you never have money moved without a record. A distributed lock is only needed when the contended resource is not a database row. On retry, the stored result is replayed verbatim, and the in-progress case is handled explicitly: a concurrent retry gets a 409 or waits, never a double charge.

At National Bank of India, Rajesh's app retried mid-payment and he was charged twice. Pankaj hardened the flow with an atomic idempotency-key claim as an INSERT, a UNIQUE constraint on the transaction ID, both committed in one transaction, and the full response stored for replay. The in-progress handling, where a concurrent retry waits or gets a 409, was the detail that made the concurrent case safe.

Remember one sentence: prevent a double charge with an atomic idempotency-key claim, a UNIQUE constraint as the hard database backstop, and a single transaction that commits the charge and the record together, so every retry returns the stored result instead of charging again.

---