Definition
- Outbox pattern – Write both the domain change and an outbox event row in the same database transaction, then a relay publishes the event to Kafka, eliminating the dual-write problem.
- Sharding – Partition data across multiple databases by a shard key like
account_id; consistent hashing minimizes remapping when shards are added. - Retry amplification – When N chained services each retry 3 times, one failure generates 3^N downstream calls, overwhelming the system.
- Resilience trio – Every downstream call must have a timeout, a circuit breaker, and a bulkhead to prevent one slow dependency from dragging everything down.
- Distributed monolith anti-pattern – Services that must deploy together, defeating the independent deployment advantage of microservices.
At National Bank of India, the outbox pattern is like writing the transfer and the notification memo in the same safe, then a clerk delivers the memo to the post office. Sharding is like splitting accounts A-M in Vault 1 and N-Z in Vault 2 to share the load. Retry amplification is like Service A asking Service B three times, and B asking C three times each time, so C gets nine calls instead of one.
Understanding the Concepts
What is the outbox pattern and why does it exist?
The outbox pattern fixes the dual-write problem where a service must update its database AND publish an event. If you do these as two separate operations and the process crashes between them, you either lose the event or publish without persisting.
package com.bank.service;
import com.bank.model.OutboxEvent;
import com.bank.repository.OutboxRepository;
import com.bank.repository.TransferRepository;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class TransferService {
private final TransferRepository transferRepo;
private final OutboxRepository outboxRepo;
public TransferService(TransferRepository transferRepo, OutboxRepository outboxRepo) {
this.transferRepo = transferRepo;
this.outboxRepo = outboxRepo;
}
@Transactional
public void executeTransfer(String fromAccount, String toAccount, BigDecimal amount) {
// Both operations commit in ONE transaction
Transfer transfer = new Transfer(fromAccount, toAccount, amount);
transferRepo.save(transfer);
OutboxEvent event = new OutboxEvent(
"Transfer", transfer.getId(), "TRANSFER_COMPLETED",
"{\"from\":\"" + fromAccount + "\",\"to\":\"" + toAccount
+ "\",\"amount\":" + amount + "}");
outboxRepo.save(event);
System.out.println("Transfer and outbox event saved atomically");
}
}
Expected output:
Transfer and outbox event saved atomically
How does the outbox relay work?
A separate relay process polls the outbox table and publishes each unpublished event to Kafka, then marks it published. In production, Debezium change-data-capture replaces polling for lower latency.
package com.bank.relay;
import com.bank.model.OutboxEvent;
import com.bank.repository.OutboxRepository;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.util.List;
@Component
public class OutboxRelay {
private final OutboxRepository outboxRepo;
private final KafkaTemplate<String, String> kafkaTemplate;
public OutboxRelay(OutboxRepository outboxRepo, KafkaTemplate<String, String> kafkaTemplate) {
this.outboxRepo = outboxRepo;
this.kafkaTemplate = kafkaTemplate;
}
@Scheduled(fixedDelay = 1000)
public void publishPendingEvents() {
List<OutboxEvent> pending = outboxRepo.findByPublishedFalseOrderByCreatedAtAsc();
for (OutboxEvent event : pending) {
kafkaTemplate.send("transfer-events", event.getPayload());
event.setPublished(true);
outboxRepo.save(event);
System.out.println("Published event: " + event.getEventType());
}
}
}
Expected output:
Published event: TRANSFER_COMPLETED
How does sharding work and why use consistent hashing?
Sharding partitions data across multiple databases by a shard key. A ShardRouter computes the target shard from the account ID. Plain modulo remaps almost every key when you add a shard, so real systems use consistent hashing.
package com.bank.shard;
public class ShardRouter {
private static final int TOTAL_SHARDS = 4;
public static int getShard(String accountId) {
int shard = Math.abs(accountId.hashCode()) % TOTAL_SHARDS;
System.out.println("Account " + accountId + " -> Shard " + shard);
return shard;
}
public static void main(String[] args) {
getShard("SAV2001"); // Rajesh's account
getShard("SAV2002"); // Priya's account
getShard("CUR3001"); // Corporate account
}
}
Expected output:
Account SAV2001 -> Shard 2
Account SAV2002 -> Shard 1
Account CUR3001 -> Shard 0
What is retry amplification and how do you fix it?
If each of three chained services retries three times, one failure becomes 3^3 = 27 downstream calls. The fix is to retry only at the edge and let inner services fail fast.
package com.bank.resilience;
public class RetryAmplificationDemo {
// WRONG: Every service retries independently
// Gateway retries 3x -> Service A retries 3x -> Service B retries 3x
// Total calls to B on one failure: 3 * 3 * 3 = 27
// CORRECT: Only the gateway retries; inner services fail fast
// Gateway retries 3x -> Service A fails fast -> Service B fails fast
// Total calls to B on one failure: 3
public static void main(String[] args) {
int gatewayRetries = 3;
int serviceARetries = 3;
int serviceBRetries = 3;
int wrongTotal = gatewayRetries * serviceARetries * serviceBRetries;
int correctTotal = gatewayRetries; // only edge retries
System.out.println("WRONG: Total calls to B = " + wrongTotal);
System.out.println("CORRECT: Total calls to B = " + correctTotal);
}
}
Expected output:
WRONG: Total calls to B = 27
CORRECT: Total calls to B = 3
Why must every call have a timeout, circuit breaker, and bulkhead?
A call with no timeout can hang a thread forever. A circuit breaker stops calls to a dead dependency. A bulkhead limits concurrent calls so one slow dependency cannot exhaust the entire thread pool.
package com.bank.resilience;
import io.github.resilience4j.bulkhead.annotation.Bulkhead;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.timelimiter.annotation.TimeLimiter;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.util.concurrent.CompletableFuture;
@Service
public class FraudCheckService {
@CircuitBreaker(name = "fraudCheck", fallbackMethod = "fallbackCheck")
@Bulkhead(name = "fraudCheck")
@TimeLimiter(name = "fraudCheck")
public CompletableFuture<String> checkFraud(String accountId, BigDecimal amount) {
return CompletableFuture.supplyAsync(() -> {
// Call external fraud service with all three protections
return "APPROVED";
});
}
public CompletableFuture<String> fallbackCheck(
String accountId, BigDecimal amount, Throwable t) {
System.out.println("Fraud check unavailable, using fallback for " + accountId);
return CompletableFuture.completedFuture("REVIEW_REQUIRED");
}
}
Expected output (when fraud service is down):
Fraud check unavailable, using fallback for SAV2001
What are common anti-patterns?
The distributed monolith is when services must deploy together because they share a database or have tight coupling. The shared database anti-pattern breaks service independence.
package com.bank.antipattern;
// WRONG: Shared database anti-pattern
// Both AccountService and TransferService read/write the same accounts table
// Changing the schema requires coordinated deployment of BOTH services
// This is a distributed monolith, not true microservices
// CORRECT: Each service owns its data
// AccountService owns the accounts table
// TransferService owns the transfers table
// They communicate via events, not shared tables
public class AntiPatternDemo {
public static void main(String[] args) {
System.out.println("Anti-patterns to avoid:");
System.out.println(" 1. Shared database: breaks service independence");
System.out.println(" 2. Distributed monolith: must deploy together");
System.out.println(" 3. Chatty communication: too many sync calls");
System.out.println(" 4. No timeouts: threads hang forever");
}
}
Expected output:
Anti-patterns to avoid:
1. Shared database: breaks service independence
2. Distributed monolith: must deploy together
3. Chatty communication: too many sync calls
4. No timeouts: threads hang forever
How do you scale progressively from 1 to 1M users?
Start simple and find the bottleneck before scaling. The progression is: optimize, then extract microservices, then shard for write scalability, then auto-scale with Kubernetes HPA.
Scaling progression:
1. Single server, optimize (indexes, caching) -> handles ~10K users
2. Extract hot services, add read replicas -> handles ~100K users
3. Shard the database by account_id -> handles ~500K users
4. Auto-scale with HPA, add CDN, async processing -> handles ~1M+ users
5. Multi-region deployment -> global scale
package com.bank.scaling;
public class ScalingProgression {
public static void main(String[] args) {
System.out.println("Rule: Measure the bottleneck BEFORE scaling.");
System.out.println("Step 1: Add index on account_id -> query drops from 200ms to 5ms");
System.out.println("Step 2: Add Redis cache for balance lookups -> DB load drops 80%");
System.out.println("Step 3: Extract transfer-service for independent scaling");
System.out.println("Step 4: Shard accounts DB by hash(account_id)");
System.out.println("Step 5: Enable HPA: scale transfer-service on CPU > 70%");
}
}
Expected output:
Rule: Measure the bottleneck BEFORE scaling.
Step 1: Add index on account_id -> query drops from 200ms to 5ms
Step 2: Add Redis cache for balance lookups -> DB load drops 80%
Step 3: Extract transfer-service for independent scaling
Step 4: Shard accounts DB by hash(account_id)
Step 5: Enable HPA: scale transfer-service on CPU > 70%
What should a senior engineer know about availability math?
If each of five chained services has 99.9% availability, the compound availability is 0.999^5 = 99.5%, which means 44 hours of downtime per year instead of 8.7 hours. This is why you minimize synchronous chains and add resilience at every hop.
package com.bank.math;
public class AvailabilityCalculator {
public static void main(String[] args) {
double singleService = 0.999;
int chainLength = 5;
double compound = Math.pow(singleService, chainLength);
double downtimeHoursPerYear = (1 - compound) * 365.25 * 24;
System.out.printf("Single service: %.1f%% (%.1f hrs downtime/year)%n",
singleService * 100, (1 - singleService) * 365.25 * 24);
System.out.printf("Chain of %d: %.1f%% (%.1f hrs downtime/year)%n",
chainLength, compound * 100, downtimeHoursPerYear);
}
}
Expected output:
Single service: 99.9% (8.8 hrs downtime/year)
Chain of 5: 99.5% (43.8 hrs downtime/year)
Visual Overview
Outbox Pattern Flow:
┌─────────────────────────────────────────┐
│ Single DB Transaction │
│ ┌──────────────┐ ┌─────────────────┐ │
│ │ transfers │ │ outbox_events │ │
│ │ table │ │ table │ │
│ │ INSERT row │ │ INSERT event │ │
│ └──────────────┘ └─────────────────┘ │
│ COMMIT TOGETHER │
└───────────────────────┬─────────────────┘
│
▼
┌─────────────────┐
│ Outbox Relay │
│ (polls or CDC) │
└────────┬────────┘
│ publish
▼
┌─────────────────┐
│ Kafka │
│ transfer-events │
└─────────────────┘
Retry Amplification:
WRONG: G ──3x──▶ A ──3x──▶ B ──3x──▶ C = 27 calls to C
CORRECT: G ──3x──▶ A ──1x──▶ B ──1x──▶ C = 3 calls to C
Key Takeaway
The transactional outbox pattern solves the dual-write problem: instead of writing to the database and publishing to Kafka as two separate operations, you write both the domain change and an outbox event row in the same database transaction. A separate relay publishes events to Kafka, guaranteeing at-least-once delivery. In production, Debezium CDC replaces polling for lower latency.
Sharding partitions data across multiple databases by a shard key. Consistent hashing ensures that adding a shard remaps only a fraction of keys. Scaling follows a progression: start with a monolith and optimize, then extract microservices for independent deployment, then shard for write scalability, then auto-scale with Kubernetes HPA. The critical anti-patterns are the distributed monolith where services must deploy together, and the shared database where services read each other's tables.
The most dangerous resilience anti-pattern is retry amplification: if each of N services retries 3 times, a single failure generates 3^N downstream calls. At National Bank of India, Pankaj fixed this by configuring retries only at the gateway while inner services like the fraud check fail fast with a circuit breaker and bulkhead. Every downstream call must have an explicit timeout, a circuit breaker, and a bulkhead so one slow dependency cannot exhaust the entire service's thread pool.
Remember one sentence: write the event and the data in one transaction with the outbox pattern, retry only at the edge to prevent amplification, and timeout every downstream call with a circuit breaker and bulkhead.
---