Definition
- Payment gateway – a system that accepts a payment request, validates it, moves money between accounts reliably, and notifies downstream systems.
- Idempotency – each payment carries a unique key stored in Redis; duplicate requests return the stored result instead of moving money again.
- Saga pattern – a sequence of local transactions where each step has a compensating action, so failure at any step rolls back earlier steps without a distributed transaction.
- Transactional outbox – an event row is inserted in the same database transaction as the payment; a relay publishes it to Kafka, guaranteeing the event exists if and only if the payment committed.
- Observability – Prometheus metrics, structured ELK logs, and distributed tracing with correlation IDs provide full visibility into every request.
Think of the payment gateway as the central money-transfer department at National Bank of India. When Rajesh asks to send money to Priya, a guard at the door (API gateway) checks his ID and blocks flooding. A clerk checks whether this exact request was already processed (idempotency), so if Rajesh clicks Pay twice, the money moves only once. The transfer happens in careful steps with undo capability (Saga), and finally a note is dropped in the mailroom (Kafka) so the SMS team, analytics team, and reconciliation team each pick it up independently.
Understanding the Concepts
What is a payment gateway?
A payment gateway is the system that sits between a client (mobile app, web portal) and the bank's core ledger. It accepts a payment request, validates the sender's identity and balance, moves money between accounts, and notifies downstream systems such as SMS, email, analytics, and reconciliation.
At National Bank of India, when Rajesh transfers money to Priya through the mobile app, the request flows through this gateway.
package com.bank.payment;
import java.math.BigDecimal;
public class PaymentRequest {
private String idempotencyKey;
private String senderAccountId;
private String receiverAccountId;
private BigDecimal amount;
private String currency;
// getters and setters
}
How does the request enter the system?
The request enters through an API gateway that handles TLS termination, JWT authentication, and rate limiting. Unauthenticated or abusive traffic never reaches the core payment service. Behind the API gateway, the payment service is stateless and horizontally scaled, meaning any instance can serve any request.
package com.bank.gateway;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class PaymentController {
private final PaymentService paymentService;
public PaymentController(PaymentService paymentService) {
this.paymentService = paymentService;
}
@PostMapping("/api/v1/payments")
public PaymentResponse initiatePayment(
@RequestHeader("Idempotency-Key") String idempotencyKey,
@RequestBody PaymentRequest request) {
return paymentService.processPayment(idempotencyKey, request);
}
}
How does idempotency guarantee exactly-once processing?
The client sends an idempotency key with each payment. The service checks Redis for that key before doing any work. If the key exists, the stored result is returned immediately. If the key is new, the payment is processed and the result is stored in Redis with a TTL.
This prevents double-charging Rajesh when his phone retries a request it thought failed due to a network timeout.
package com.bank.payment;
import java.util.concurrent.TimeUnit;
public class IdempotencyGuard {
private final RedisTemplate<String, PaymentResponse> redis;
public IdempotencyGuard(RedisTemplate<String, PaymentResponse> redis) {
this.redis = redis;
}
public PaymentResponse checkOrProcess(
String idempotencyKey, PaymentRequest request,
PaymentProcessor processor) {
PaymentResponse cached = redis.opsForValue().get(idempotencyKey);
if (cached != null) {
return cached; // already processed, return stored result
}
PaymentResponse result = processor.execute(request);
redis.opsForValue().set(idempotencyKey, result, 24, TimeUnit.HOURS);
return result;
}
}
How does the Saga pattern keep money consistent?
Moving money touches multiple services: debit sender, credit receiver, record settlement. A single distributed transaction would lock resources and not scale. Instead, the Saga pattern chains local transactions where each step has a compensating action. If crediting Priya fails, the compensation re-credits Rajesh.
package com.bank.payment;
import java.math.BigDecimal;
import java.util.logging.Logger;
public class TransferSaga {
private static final Logger LOG = Logger.getLogger(TransferSaga.class.getName());
private final AccountService accountService;
private final SettlementService settlementService;
public TransferSaga(AccountService accountService,
SettlementService settlementService) {
this.accountService = accountService;
this.settlementService = settlementService;
}
public void execute(String sender, String receiver, BigDecimal amount) {
// Step 1: Debit sender
accountService.debit(sender, amount);
LOG.info("Step 1 done: debited " + sender);
try {
// Step 2: Credit receiver
accountService.credit(receiver, amount);
LOG.info("Step 2 done: credited " + receiver);
} catch (Exception e) {
// Compensation: re-credit sender
accountService.credit(sender, amount);
LOG.warning("Step 2 failed: compensated sender " + sender);
throw new PaymentFailedException("Credit failed, sender refunded", e);
}
try {
// Step 3: Record settlement
settlementService.record(sender, receiver, amount);
LOG.info("Step 3 done: settlement recorded");
} catch (Exception e) {
// Compensation: reverse both accounts
accountService.debit(receiver, amount);
accountService.credit(sender, amount);
LOG.warning("Step 3 failed: both accounts reversed");
throw new PaymentFailedException("Settlement failed, reversed", e);
}
}
}
How does the transactional outbox avoid dual-write problems?
In the same database transaction that records the payment, an event row is inserted into an outbox table. A separate relay process reads the outbox and publishes to a Kafka topic. This guarantees the event is published if and only if the payment committed, avoiding the case where the database succeeds but Kafka fails (or vice versa).
package com.bank.payment;
import org.springframework.transaction.annotation.Transactional;
public class PaymentProcessor {
private final PaymentRepository paymentRepo;
private final OutboxRepository outboxRepo;
public PaymentProcessor(PaymentRepository paymentRepo,
OutboxRepository outboxRepo) {
this.paymentRepo = paymentRepo;
this.outboxRepo = outboxRepo;
}
@Transactional
public PaymentResponse execute(PaymentRequest request) {
Payment payment = paymentRepo.save(toPaymentEntity(request));
// Same transaction: insert outbox event
OutboxEvent event = new OutboxEvent(
"payment-events",
payment.getId(),
"PAYMENT_COMPLETED",
toJson(payment)
);
outboxRepo.save(event);
return toResponse(payment);
}
}
How do you make the system resilient?
Every downstream call is wrapped in a circuit breaker with retry logic. If the credit service is temporarily down, the circuit breaker opens after a threshold of failures and returns a fallback quickly instead of hanging. After a cooldown period, it retries to see if the service has recovered.
package com.bank.resilience;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.retry.annotation.Retry;
public class ResilientAccountService {
private final AccountClient client;
public ResilientAccountService(AccountClient client) {
this.client = client;
}
@CircuitBreaker(name = "accountService", fallbackMethod = "fallback")
@Retry(name = "accountService")
public void credit(String accountId, java.math.BigDecimal amount) {
client.credit(accountId, amount);
}
public void fallback(String accountId, java.math.BigDecimal amount,
Throwable t) {
throw new ServiceUnavailableException(
"Account service unavailable: " + t.getMessage());
}
}
How do you make the system observable?
Three pillars: metrics (Prometheus and Grafana), logs (structured JSON in ELK), and distributed tracing (correlation IDs through Jaeger or Zipkin). Every request gets a correlation ID that flows through all services so a single payment can be traced end to end.
package com.bank.observability;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
public class PaymentMetrics {
private final Counter successCounter;
private final Counter failureCounter;
private final Timer latencyTimer;
public PaymentMetrics(MeterRegistry registry) {
this.successCounter = registry.counter("payment.success");
this.failureCounter = registry.counter("payment.failure");
this.latencyTimer = registry.timer("payment.latency");
}
public void recordSuccess(long durationMs) {
successCounter.increment();
latencyTimer.record(durationMs, java.util.concurrent.TimeUnit.MILLISECONDS);
}
public void recordFailure() {
failureCounter.increment();
}
}
What about data storage and scaling?
PostgreSQL stores the authoritative account and payment tables, sharded by account ID so one database does not become a bottleneck. Read replicas serve heavy reporting queries. Redis stores idempotency keys, rate-limit counters, and session data. Money movement is strongly consistent within each local transaction, while notifications and analytics are eventually consistent through Kafka.
// Sharding strategy: account_id hash determines the shard
// Shard 0: accounts 0-999999
// Shard 1: accounts 1000000-1999999
// application.yml
// spring:
// datasource:
// url: jdbc:postgresql://shard-${accountId.hash % 4}:5432/payments
// redis:
// host: redis-cluster.bank.internal
// ttl-idempotency: 86400 # 24 hours
What should a senior engineer know?
- Consistency trade-off. Money movement must be strongly consistent. Notifications and analytics can be eventually consistent. This is a deliberate choice, not an accident.
- Kafka ordering. Partition by account ID so all events for one account are ordered, enabling correct balance reconstruction.
- Dead letter queue. Events that repeatedly fail processing go to a DLQ for manual review instead of blocking the consumer.
- Container limits. Set JVM heap with
-XX:MaxRAMPercentage=75.0so the JVM respects container memory limits.
// Dead letter queue configuration
// If a Kafka consumer fails 3 times, route to DLQ
@KafkaListener(topics = "payment-events")
public void handlePaymentEvent(PaymentEvent event) {
try {
notificationService.send(event);
} catch (Exception e) {
dlqProducer.send("payment-events-dlq", event);
log.error("Sent to DLQ: " + event.getPaymentId(), e);
}
}
Visual Overview
Client (Mobile App)
│
│ HTTPS + JWT + Idempotency-Key
▼
┌──────────────────┐
│ API Gateway │ TLS, Auth, Rate Limit
└──────────────────┘
│
▼
┌──────────────────┐ ┌───────────┐
│ Payment Service │────>│ Redis │ Idempotency check
│ (stateless, HPA) │ └───────────┘
└──────────────────┘
│
▼ Saga: Debit -> Credit -> Settle
┌──────────┐ ┌──────────┐ ┌──────────────┐
│ Account │ │ Account │ │ Settlement │
│ (Sender) │ │(Receiver)│ │ Service │
└──────────┘ └──────────┘ └──────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────┐
│ PostgreSQL (sharded) │
│ Payment table + Outbox table │
└─────────────────────────────────────────┘
│
│ Outbox relay
▼
┌──────────────────┐
│ Kafka │ payment-events topic
└──────────────────┘
│
┌────┼────────┬──────────┐
│ │ │ │
▼ ▼ ▼ ▼
SMS Email Analytics Reconciliation
Key Takeaway
When I design a payment gateway, I start with the request path and correctness guarantees. Requests enter through an API gateway that handles TLS, authentication, and rate limiting. Behind it the payment service is stateless and horizontally scaled. The most important property is idempotency: the client sends a unique key, I store it in Redis with the result, and if the same key arrives again from a retry I return the stored result instead of moving money twice.
Because moving money spans multiple services, I use the Saga pattern instead of a distributed transaction. I debit the sender, credit the receiver, and record settlement, and if any step fails I run the compensating action for each completed step so the system is never left with money missing. Each downstream call is wrapped in a circuit breaker with retry logic so transient failures do not cascade.
To tell the rest of the system what happened without tight coupling, I use the transactional outbox pattern. In the same database transaction that commits the payment, I insert an event into an outbox table, and a relay publishes it to Kafka. This avoids the dual-write problem because the event is published only if the payment committed. Consumers react asynchronously: one sends SMS and email, one updates analytics, and one does end-of-day reconciliation.
For scale and reliability, I shard PostgreSQL by account ID, use read replicas for reporting, cache idempotency keys in Redis, and make the whole system observable with Prometheus metrics, structured ELK logs, and distributed tracing with correlation IDs. The deliberate trade-off is strong consistency for money movement and eventual consistency for notifications.
Remember one sentence: idempotency keys prevent double charges, Sagas keep money consistent across services, and the transactional outbox guarantees events are published only when the payment commits.
---