Back to Microservices Fundamentals
Q13Question 13 of 13

What is a distributed system and why are microservices distributed

Answer: A distributed system is a group of independent computers that communicate over a network and appear as one system, and microservices are inherently distributed because each service is a separate process where every inter-service call can fail due to network timeouts, connection errors, or partial failures.

Key Concepts
  • A distributed system is a collection of independent processes that communicate over a network and appear as a single system to users.
  • Microservices are inherently distributed because each service runs as a separate process and every inter-service call crosses a network boundary.
  • Network failures include connection refused (service down), connection timeout (unreachable), and read timeout (service too slow).
  • Partial failure is the defining challenge: one step succeeds while a later step fails, leaving the system in an inconsistent state.
  • Every outgoing HTTP call must have connect and read timeouts configured to prevent cascading failures.
  • Correlation IDs passed in HTTP headers allow you to trace a single request across multiple services in the logs.
  • Design every write operation to be idempotent so that retries after timeouts do not cause duplicate side effects.

Definition

A distributed system is a group of independent computers that communicate over a network and appear as one system, and microservices are inherently distributed because each service is a separate process where every inter-service call can fail due to network timeouts, connection errors, or partial failures.
  • Distributed system is a collection of independent nodes (computers or processes) that coordinate by exchanging messages over a network, where the network can lose, delay, duplicate, or reorder messages at any time.
  • Partial failure is the situation where one node in the system fails while others continue to operate, making some operations succeed and others fail within the same business transaction.
  • Network partition is when the network link between two nodes is broken so they cannot communicate, even though both nodes are healthy and running.
  • CAP theorem states that a distributed system can guarantee at most two of three properties: Consistency, Availability, and Partition tolerance.
  • Cascading failure is when one slow or failing service causes its callers to fail, which causes their callers to fail, until the entire system is unresponsive.

Think of National Bank of India with branches in Mumbai and Delhi. When a customer in Mumbai (Rajesh) transfers money to a customer in Delhi (Priya), the Mumbai branch debits Rajesh's account and sends a message to the Delhi branch to credit Priya's account. If the courier service between the two cities is delayed, the Mumbai branch has already debited Rajesh but Priya has not received the money yet. If the courier loses the message entirely, the Mumbai branch must detect this and either retry or reverse the debit. This is exactly what happens in microservices: account-service (Mumbai branch) and transfer-service (Delhi branch) communicate over a network that can fail at any time.

Understanding the Concepts

What is a distributed system?

A distributed system is a collection of independent computers or processes that communicate over a network and coordinate their actions to appear as a single coherent system to the end user. Each node runs its own process, has its own memory, and can fail independently. The network connecting the nodes is unreliable: messages can be lost, delayed, duplicated, or delivered out of order.

In Java, you can model the core properties of a distributed system with a sealed interface and records.

JAVA
package com.bank.distributed;

public class DistributedSystemProperties {

    sealed interface NodeState permits Healthy, Degraded, Unreachable {}
    record Healthy(String nodeId, long latencyMs) implements NodeState {}
    record Degraded(String nodeId, long latencyMs, String reason) implements NodeState {}
    record Unreachable(String nodeId, String lastError) implements NodeState {}

    public static void main(String[] args) {
        var nodes = java.util.List.of(
            new Healthy("account-service", 12),
            new Degraded("transfer-service", 450, "high CPU"),
            new Unreachable("notification-service", "Connection refused")
        );

        for (var node : nodes) {
            String status = switch (node) {
                case Healthy h -> h.nodeId() + " is healthy, latency=" + h.latencyMs() + "ms";
                case Degraded d -> d.nodeId() + " is degraded: " + d.reason();
                case Unreachable u -> u.nodeId() + " is unreachable: " + u.lastError();
            };
            System.out.println(status);
        }
    }
}
TEXT
Expected output:
account-service is healthy, latency=12ms
transfer-service is degraded: high CPU
notification-service is unreachable: Connection refused

Why are microservices inherently distributed systems?

In a monolithic application, all modules run in a single JVM process. A method call from TransferModule to AccountModule is a local in-memory call. It cannot fail because of the network, it does not time out, and it either succeeds or throws an exception in microseconds.

In a microservices architecture, TransferService and AccountService are separate processes, often running on different machines. Every call between them crosses a network boundary. That network call can fail in ways that a local method call never can: the connection can time out, the remote service can be down, the response can be lost, or the remote service can process the request but the response never arrives.

This is the fundamental reason microservices are distributed systems. Every inter-service call introduces the possibility of partial failure: one service succeeds while the other fails or never receives the result.

JAVA
package com.bank.distributed;

import java.math.BigDecimal;

public class LocalVsRemoteCall {

    record TransferResult(boolean success, String message) {}

    // Monolith: local method call, never fails due to network
    static TransferResult localTransfer(String from, String to, BigDecimal amount) {
        System.out.println("LOCAL CALL: debit " + from + ", credit " + to + ", amount=" + amount);
        // This call is in-memory. It always completes.
        return new TransferResult(true, "Transfer completed in same JVM");
    }

    // Microservice: remote HTTP call, can fail in many ways
    static TransferResult remoteTransfer(String from, String to, BigDecimal amount) {
        System.out.println("REMOTE CALL: POST http://account-service:8081/accounts/" + from + "/debit");
        // This call goes over the network. It can:
        // 1. Time out (server is slow)
        // 2. Get connection refused (server is down)
        // 3. Succeed on server but response is lost
        // 4. Fail with 500 (server error)
        return new TransferResult(false, "Connection timed out after 3000ms");
    }

    public static void main(String[] args) {
        var amount = new BigDecimal("50000.00");
        System.out.println("=== Monolith (local call) ===");
        var local = localTransfer("SAV2001", "SAV2002", amount);
        System.out.println("Result: " + local);
        System.out.println();
        System.out.println("=== Microservices (remote call) ===");
        var remote = remoteTransfer("SAV2001", "SAV2002", amount);
        System.out.println("Result: " + remote);
        System.out.println();
        System.out.println("Key difference: the remote call can fail due to network.");
        System.out.println("The local call can only fail due to business logic.");
    }
}
TEXT
Expected output:
=== Monolith (local call) ===
LOCAL CALL: debit SAV2001, credit SAV2002, amount=50000.00
Result: TransferResult[success=true, message=Transfer completed in same JVM]

=== Microservices (remote call) ===
REMOTE CALL: POST http://account-service:8081/accounts/SAV2001/debit
Result: TransferResult[success=false, message=Connection timed out after 3000ms]

Key difference: the remote call can fail due to network.
The local call can only fail due to business logic.

What types of failures happen in distributed systems?

Distributed systems face three categories of failures that do not exist in monoliths.

  • Network failures: the connection between two services is broken. This includes connection refused (service is down), connection timeout (service is unreachable), and read timeout (service accepted the request but is too slow to respond).
  • Partial failures: one step in a multi-step operation succeeds but a later step fails. For example, the debit succeeds but the credit fails. The system is now in an inconsistent state.
  • Byzantine failures: a service responds, but the response is incorrect. This is the hardest type to detect. For example, a service returns a success status code but does not persist the data.
JAVA
package com.bank.distributed;

public class FailureTypes {

    sealed interface DistributedFailure permits NetworkFailure, PartialFailure, ByzantineFailure {}

    record NetworkFailure(String service, String type, String detail) implements DistributedFailure {}
    record PartialFailure(String succeededStep, String failedStep, String state) implements DistributedFailure {}
    record ByzantineFailure(String service, String expected, String actual) implements DistributedFailure {}

    public static void main(String[] args) {
        var failures = java.util.List.of(
            new NetworkFailure("account-service", "CONNECTION_TIMEOUT",
                "No response after 3000ms"),
            new NetworkFailure("account-service", "CONNECTION_REFUSED",
                "Service is down, port 8081 not listening"),
            new PartialFailure("debit SAV2001", "credit SAV2002",
                "SAV2001 debited 50000 but SAV2002 never credited"),
            new ByzantineFailure("payment-gateway", "HTTP 200 + persisted",
                "HTTP 200 but data not saved to database")
        );

        for (var failure : failures) {
            String description = switch (failure) {
                case NetworkFailure n -> "[NETWORK] " + n.service() + " " + n.type() + ": " + n.detail();
                case PartialFailure p -> "[PARTIAL] succeeded=" + p.succeededStep()
                    + ", failed=" + p.failedStep() + ", state=" + p.state();
                case ByzantineFailure b -> "[BYZANTINE] " + b.service()
                    + " expected=" + b.expected() + ", actual=" + b.actual();
            };
            System.out.println(description);
        }
    }
}
TEXT
Expected output:
[NETWORK] account-service CONNECTION_TIMEOUT: No response after 3000ms
[NETWORK] account-service CONNECTION_REFUSED: Service is down, port 8081 not listening
[PARTIAL] succeeded=debit SAV2001, failed=credit SAV2002, state=SAV2001 debited 50000 but SAV2002 never credited
[BYZANTINE] payment-gateway expected=HTTP 200 + persisted, actual=HTTP 200 but data not saved to database

How is a monolith different from a distributed system?

FeatureMonolithDistributed (Microservices)
CommunicationIn-process method callsNetwork calls (HTTP, gRPC, messaging)
Failure modesApplication errors onlyNetwork failures, partial failures, timeouts
ConsistencySingle database transactionDistributed transactions or eventual consistency
LatencyMicroseconds (method call)Milliseconds to seconds (network round-trip)
DeploymentOne artifactMultiple artifacts, independent deployment
ScalingScale the entire applicationScale individual services
StateShared memoryEach service has its own database
DebuggingSingle stack traceDistributed tracing with correlation IDs
TestingUnit and integration testsContract tests, chaos engineering
JAVA
package com.bank.distributed;

import java.math.BigDecimal;

public class MonolithVsDistributed {

    // In a monolith: direct method call, single transaction
    static void monolithTransfer(String from, String to, BigDecimal amount) {
        System.out.println("BEGIN TRANSACTION");
        System.out.println("  UPDATE accounts SET balance = balance - " + amount + " WHERE number = '" + from + "'");
        System.out.println("  UPDATE accounts SET balance = balance + " + amount + " WHERE number = '" + to + "'");
        System.out.println("COMMIT");
        System.out.println("Result: both updates in ONE transaction, atomic.");
    }

    // In microservices: two HTTP calls, two databases, no shared transaction
    static void distributedTransfer(String from, String to, BigDecimal amount) {
        System.out.println("Step 1: POST http://account-service:8081/accounts/" + from + "/debit");
        System.out.println("  Result: 200 OK, SAV2001 debited");
        System.out.println("Step 2: POST http://account-service:8081/accounts/" + to + "/credit");
        System.out.println("  Result: 503 Service Unavailable (account-service restarted)");
        System.out.println("Problem: SAV2001 is debited, SAV2002 is NOT credited.");
        System.out.println("  This is a PARTIAL FAILURE. Money disappeared.");
    }

    public static void main(String[] args) {
        var amount = new BigDecimal("50000.00");
        System.out.println("=== Monolith Transfer ===");
        monolithTransfer("SAV2001", "SAV2002", amount);
        System.out.println();
        System.out.println("=== Distributed Transfer ===");
        distributedTransfer("SAV2001", "SAV2002", amount);
    }
}
TEXT
Expected output:
=== Monolith Transfer ===
BEGIN TRANSACTION
  UPDATE accounts SET balance = balance - 50000.00 WHERE number = 'SAV2001'
  UPDATE accounts SET balance = balance + 50000.00 WHERE number = 'SAV2002'
COMMIT
Result: both updates in ONE transaction, atomic.

=== Distributed Transfer ===
Step 1: POST http://account-service:8081/accounts/SAV2001/debit
  Result: 200 OK, SAV2001 debited
Step 2: POST http://account-service:8081/accounts/SAV2002/credit
  Result: 503 Service Unavailable (account-service restarted)
Problem: SAV2001 is debited, SAV2002 is NOT credited.
  This is a PARTIAL FAILURE. Money disappeared.

What is a complete banking project demonstrating distributed failures?

This project has two Spring Boot 3.4 services: account-service (manages account balances) and transfer-service (orchestrates fund transfers by calling account-service). The transfer-service calls account-service via REST. The project demonstrates successful transfers, timeouts, and partial failures.

Project structure

DIAGRAM
distributed-banking/
├── docker-compose.yml
├── init-account-db.sql
├── init-transfer-db.sql
├── account-service/
│   ├── pom.xml
│   └── src/main/
│       ├── java/com/bank/account/
│       │   ├── AccountServiceApp.java
│       │   ├── controller/
│       │   │   └── AccountController.java
│       │   ├── service/
│       │   │   └── AccountService.java
│       │   ├── repository/
│       │   │   └── AccountRepository.java
│       │   ├── model/
│       │   │   └── Account.java
│       │   ├── dto/
│       │   │   ├── DebitRequest.java
│       │   │   ├── CreditRequest.java
│       │   │   └── AccountResponse.java
│       │   └── exception/
│       │       ├── AccountNotFoundException.java
│       │       ├── InsufficientBalanceException.java
│       │       └── GlobalExceptionHandler.java
│       └── resources/
│           └── application.yml
└── transfer-service/
    ├── pom.xml
    └── src/main/
        ├── java/com/bank/transfer/
        │   ├── TransferServiceApp.java
        │   ├── controller/
        │   │   └── TransferController.java
        │   ├── service/
        │   │   └── TransferService.java
        │   ├── repository/
        │   │   └── TransferRepository.java
        │   ├── model/
        │   │   └── Transfer.java
        │   ├── dto/
        │   │   └── TransferRequest.java
        │   ├── client/
        │   │   └── AccountServiceClient.java
        │   ├── config/
        │   │   └── RestClientConfig.java
        │   └── exception/
        │       ├── TransferFailedException.java
        │       └── GlobalExceptionHandler.java
        └── resources/
            └── application.yml

docker-compose.yml

YAML
version: "3.9"
services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: bank
      POSTGRES_PASSWORD: bank123
    ports:
      - "5432:5432"
    volumes:
      - ./init-account-db.sql:/docker-entrypoint-initdb.d/01-account.sql
      - ./init-transfer-db.sql:/docker-entrypoint-initdb.d/02-transfer.sql

  account-service:
    build: ./account-service
    ports:
      - "8081:8081"
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/bank
      SPRING_DATASOURCE_USERNAME: bank
      SPRING_DATASOURCE_PASSWORD: bank123
    depends_on:
      - postgres

  transfer-service:
    build: ./transfer-service
    ports:
      - "8082:8082"
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/bank
      SPRING_DATASOURCE_USERNAME: bank
      SPRING_DATASOURCE_PASSWORD: bank123
      ACCOUNT_SERVICE_URL: http://account-service:8081
    depends_on:
      - account-service

init-account-db.sql

SQL
CREATE TABLE IF NOT EXISTS accounts (
    id              BIGSERIAL PRIMARY KEY,
    account_number  VARCHAR(20) NOT NULL UNIQUE,
    holder_name     VARCHAR(100) NOT NULL,
    balance         NUMERIC(15,2) NOT NULL DEFAULT 0.00,
    created_at      TIMESTAMP NOT NULL DEFAULT NOW()
);

INSERT INTO accounts (account_number, holder_name, balance) VALUES
    ('SAV2001', 'Rajesh Kumar', 500000.00),
    ('SAV2002', 'Priya Sharma', 200000.00),
    ('CUR3001', 'National Bank of India Corporate', 10000000.00);

init-transfer-db.sql

SQL
CREATE TABLE IF NOT EXISTS transfers (
    id              BIGSERIAL PRIMARY KEY,
    from_account    VARCHAR(20) NOT NULL,
    to_account      VARCHAR(20) NOT NULL,
    amount          NUMERIC(15,2) NOT NULL,
    status          VARCHAR(20) NOT NULL DEFAULT 'PENDING',
    correlation_id  VARCHAR(50) NOT NULL,
    error_message   VARCHAR(500),
    created_at      TIMESTAMP NOT NULL DEFAULT NOW()
);

account-service pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>

    syn-kw"><parentsyn-kw">>
        syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
        syn-kw"><artifactIdsyn-kw">>spring-boot-starter-parentsyn-kw"></artifactIdsyn-kw">>
        syn-kw"><versionsyn-kw">>3.4.1syn-kw"></versionsyn-kw">>
    syn-kw"></parentsyn-kw">>

    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>account-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>

    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
    syn-kw"></propertiessyn-kw">>

    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-boot-starter-websyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-boot-starter-data-jpasyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.postgresqlsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>postgresqlsyn-kw"></artifactIdsyn-kw">>
            syn-kw"><scopesyn-kw">>runtimesyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

Account.java

JAVA
package com.bank.account.model;

import jakarta.persistence.Entity;
import jakarta.persistence.Table;
import jakarta.persistence.Id;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Column;

import java.math.BigDecimal;
import java.time.LocalDateTime;

@Entity
@Table(name = "accounts")
public class Account {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "account_number", nullable = false, unique = true)
    private String accountNumber;

    @Column(name = "holder_name", nullable = false)
    private String holderName;

    @Column(nullable = false)
    private BigDecimal balance;

    @Column(name = "created_at", nullable = false)
    private LocalDateTime createdAt = LocalDateTime.now();

    public Long getId() { return id; }
    public String getAccountNumber() { return accountNumber; }
    public String getHolderName() { return holderName; }
    public BigDecimal getBalance() { return balance; }
    public void setBalance(BigDecimal balance) { this.balance = balance; }
}

AccountRepository.java

JAVA
package com.bank.account.repository;

import com.bank.account.model.Account;
import org.springframework.data.jpa.repository.JpaRepository;

import java.util.Optional;

public interface AccountRepository extends JpaRepository<Account, Long> {
    Optional<Account> findByAccountNumber(String accountNumber);
}

DebitRequest.java and CreditRequest.java (Java 25 records)

JAVA
package com.bank.account.dto;

import java.math.BigDecimal;

public record DebitRequest(BigDecimal amount, String description, String correlationId) {}
JAVA
package com.bank.account.dto;

import java.math.BigDecimal;

public record CreditRequest(BigDecimal amount, String description, String correlationId) {}

AccountResponse.java

JAVA
package com.bank.account.dto;

import java.math.BigDecimal;

public record AccountResponse(String accountNumber, String holderName, BigDecimal balance) {}

AccountService.java

JAVA
package com.bank.account.service;

import com.bank.account.dto.AccountResponse;
import com.bank.account.exception.AccountNotFoundException;
import com.bank.account.exception.InsufficientBalanceException;
import com.bank.account.model.Account;
import com.bank.account.repository.AccountRepository;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.math.BigDecimal;
import java.util.logging.Logger;

@Service
public class AccountService {

    private static final Logger LOG = Logger.getLogger(AccountService.class.getName());
    private final AccountRepository repository;

    public AccountService(AccountRepository repository) {
        this.repository = repository;
    }

    public AccountResponse findByNumber(String accountNumber) {
        var account = repository.findByAccountNumber(accountNumber)
            .orElseThrow(() -> new AccountNotFoundException(accountNumber));
        return new AccountResponse(account.getAccountNumber(), account.getHolderName(), account.getBalance());
    }

    @Transactional
    public AccountResponse debit(String accountNumber, BigDecimal amount, String correlationId) {
        LOG.info("Debit request: account=" + accountNumber + ", amount=" + amount + ", correlationId=" + correlationId);
        var account = repository.findByAccountNumber(accountNumber)
            .orElseThrow(() -> new AccountNotFoundException(accountNumber));
        if (account.getBalance().compareTo(amount) < 0) {
            throw new InsufficientBalanceException(accountNumber, account.getBalance(), amount);
        }
        account.setBalance(account.getBalance().subtract(amount));
        repository.save(account);
        LOG.info("Debit success: account=" + accountNumber + ", newBalance=" + account.getBalance());
        return new AccountResponse(account.getAccountNumber(), account.getHolderName(), account.getBalance());
    }

    @Transactional
    public AccountResponse credit(String accountNumber, BigDecimal amount, String correlationId) {
        LOG.info("Credit request: account=" + accountNumber + ", amount=" + amount + ", correlationId=" + correlationId);
        var account = repository.findByAccountNumber(accountNumber)
            .orElseThrow(() -> new AccountNotFoundException(accountNumber));
        account.setBalance(account.getBalance().add(amount));
        repository.save(account);
        LOG.info("Credit success: account=" + accountNumber + ", newBalance=" + account.getBalance());
        return new AccountResponse(account.getAccountNumber(), account.getHolderName(), account.getBalance());
    }
}

AccountController.java

JAVA
package com.bank.account.controller;

import com.bank.account.dto.AccountResponse;
import com.bank.account.dto.CreditRequest;
import com.bank.account.dto.DebitRequest;
import com.bank.account.service.AccountService;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/accounts")
public class AccountController {

    private final AccountService accountService;

    public AccountController(AccountService accountService) {
        this.accountService = accountService;
    }

    @GetMapping("/{accountNumber}")
    public ResponseEntity<AccountResponse> getAccount(@PathVariable String accountNumber) {
        return ResponseEntity.ok(accountService.findByNumber(accountNumber));
    }

    @PostMapping("/{accountNumber}/debit")
    public ResponseEntity<AccountResponse> debit(@PathVariable String accountNumber,
                                                  @RequestBody DebitRequest request) {
        var result = accountService.debit(accountNumber, request.amount(), request.correlationId());
        return ResponseEntity.ok(result);
    }

    @PostMapping("/{accountNumber}/credit")
    public ResponseEntity<AccountResponse> credit(@PathVariable String accountNumber,
                                                   @RequestBody CreditRequest request) {
        var result = accountService.credit(accountNumber, request.amount(), request.correlationId());
        return ResponseEntity.ok(result);
    }
}

AccountNotFoundException.java and InsufficientBalanceException.java

JAVA
package com.bank.account.exception;

public class AccountNotFoundException extends RuntimeException {
    public AccountNotFoundException(String accountNumber) {
        super("Account not found: " + accountNumber);
    }
}
JAVA
package com.bank.account.exception;

import java.math.BigDecimal;

public class InsufficientBalanceException extends RuntimeException {
    public InsufficientBalanceException(String accountNumber, BigDecimal balance, BigDecimal requested) {
        super("Insufficient balance in " + accountNumber + ": balance=" + balance + ", requested=" + requested);
    }
}

GlobalExceptionHandler.java (account-service)

JAVA
package com.bank.account.exception;

import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;

import java.util.Map;

@RestControllerAdvice
public class GlobalExceptionHandler {

    @ExceptionHandler(AccountNotFoundException.class)
    public ResponseEntity<Map<String, String>> handleNotFound(AccountNotFoundException ex) {
        return ResponseEntity.status(HttpStatus.NOT_FOUND)
            .body(Map.of("error", ex.getMessage()));
    }

    @ExceptionHandler(InsufficientBalanceException.class)
    public ResponseEntity<Map<String, String>> handleInsufficientBalance(InsufficientBalanceException ex) {
        return ResponseEntity.status(HttpStatus.UNPROCESSABLE_ENTITY)
            .body(Map.of("error", ex.getMessage()));
    }
}

account-service application.yml

YAML
server:
  port: 8081

spring:
  application:
    name: account-service
  datasource:
    url: jdbc:postgresql://localhost:5432/bank
    username: bank
    password: bank123
  jpa:
    hibernate:
      ddl-auto: none
    show-sql: true

AccountServiceApp.java

JAVA
package com.bank.account;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class AccountServiceApp {
    public static void main(String[] args) {
        SpringApplication.run(AccountServiceApp.class, args);
    }
}

transfer-service pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>

    syn-kw"><parentsyn-kw">>
        syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
        syn-kw"><artifactIdsyn-kw">>spring-boot-starter-parentsyn-kw"></artifactIdsyn-kw">>
        syn-kw"><versionsyn-kw">>3.4.1syn-kw"></versionsyn-kw">>
    syn-kw"></parentsyn-kw">>

    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>transfer-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>

    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
    syn-kw"></propertiessyn-kw">>

    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-boot-starter-websyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.springframework.bootsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>spring-boot-starter-data-jpasyn-kw"></artifactIdsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.postgresqlsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>postgresqlsyn-kw"></artifactIdsyn-kw">>
            syn-kw"><scopesyn-kw">>runtimesyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

Transfer.java

JAVA
package com.bank.transfer.model;

import jakarta.persistence.Entity;
import jakarta.persistence.Table;
import jakarta.persistence.Id;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Column;

import java.math.BigDecimal;
import java.time.LocalDateTime;

@Entity
@Table(name = "transfers")
public class Transfer {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "from_account", nullable = false)
    private String fromAccount;

    @Column(name = "to_account", nullable = false)
    private String toAccount;

    @Column(nullable = false)
    private BigDecimal amount;

    @Column(nullable = false)
    private String status;

    @Column(name = "correlation_id", nullable = false)
    private String correlationId;

    @Column(name = "error_message")
    private String errorMessage;

    @Column(name = "created_at", nullable = false)
    private LocalDateTime createdAt = LocalDateTime.now();

    public Long getId() { return id; }
    public String getFromAccount() { return fromAccount; }
    public void setFromAccount(String fromAccount) { this.fromAccount = fromAccount; }
    public String getToAccount() { return toAccount; }
    public void setToAccount(String toAccount) { this.toAccount = toAccount; }
    public BigDecimal getAmount() { return amount; }
    public void setAmount(BigDecimal amount) { this.amount = amount; }
    public String getStatus() { return status; }
    public void setStatus(String status) { this.status = status; }
    public String getCorrelationId() { return correlationId; }
    public void setCorrelationId(String correlationId) { this.correlationId = correlationId; }
    public String getErrorMessage() { return errorMessage; }
    public void setErrorMessage(String errorMessage) { this.errorMessage = errorMessage; }
}

TransferRepository.java

JAVA
package com.bank.transfer.repository;

import com.bank.transfer.model.Transfer;
import org.springframework.data.jpa.repository.JpaRepository;

public interface TransferRepository extends JpaRepository<Transfer, Long> {}

TransferRequest.java (Java 25 record)

JAVA
package com.bank.transfer.dto;

import java.math.BigDecimal;

public record TransferRequest(
    String fromAccount,
    String toAccount,
    BigDecimal amount,
    String description
) {}

RestClientConfig.java

JAVA
package com.bank.transfer.config;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.web.client.RestClient;

@Configuration
public class RestClientConfig {

    @Value("\${account.service.url:http://localhost:8081}")
    private String accountServiceUrl;

    @Bean
    public RestClient accountRestClient() {
        var factory = new SimpleClientHttpRequestFactory();
        factory.setConnectTimeout(3000);  // 3 seconds to establish connection
        factory.setReadTimeout(5000);     // 5 seconds to read response
        return RestClient.builder()
            .baseUrl(accountServiceUrl)
            .requestFactory(factory)
            .build();
    }
}

AccountServiceClient.java

JAVA
package com.bank.transfer.client;

import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;
import org.springframework.web.client.RestClientException;

import java.math.BigDecimal;
import java.util.Map;
import java.util.logging.Logger;

@Component
public class AccountServiceClient {

    private static final Logger LOG = Logger.getLogger(AccountServiceClient.class.getName());
    private final RestClient restClient;

    public AccountServiceClient(RestClient accountRestClient) {
        this.restClient = accountRestClient;
    }

    public Map<String, Object> debit(String accountNumber, BigDecimal amount, String correlationId) {
        LOG.info("Calling account-service: debit " + accountNumber + ", amount=" + amount
            + ", correlationId=" + correlationId);
        try {
            var body = Map.of("amount", amount, "description", "transfer-debit", "correlationId", correlationId);
            var response = restClient.post()
                .uri("/accounts/{number}/debit", accountNumber)
                .header("X-Correlation-ID", correlationId)
                .body(body)
                .retrieve()
                .body(Map.class);
            LOG.info("Debit success for " + accountNumber);
            return response;
        } catch (RestClientException ex) {
            LOG.severe("Debit failed for " + accountNumber + ": " + ex.getMessage());
            throw ex;
        }
    }

    public Map<String, Object> credit(String accountNumber, BigDecimal amount, String correlationId) {
        LOG.info("Calling account-service: credit " + accountNumber + ", amount=" + amount
            + ", correlationId=" + correlationId);
        try {
            var body = Map.of("amount", amount, "description", "transfer-credit", "correlationId", correlationId);
            var response = restClient.post()
                .uri("/accounts/{number}/credit", accountNumber)
                .header("X-Correlation-ID", correlationId)
                .body(body)
                .retrieve()
                .body(Map.class);
            LOG.info("Credit success for " + accountNumber);
            return response;
        } catch (RestClientException ex) {
            LOG.severe("Credit failed for " + accountNumber + ": " + ex.getMessage());
            throw ex;
        }
    }
}

TransferService.java

JAVA
package com.bank.transfer.service;

import com.bank.transfer.client.AccountServiceClient;
import com.bank.transfer.exception.TransferFailedException;
import com.bank.transfer.model.Transfer;
import com.bank.transfer.repository.TransferRepository;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestClientException;

import java.math.BigDecimal;
import java.util.UUID;
import java.util.logging.Logger;

@Service
public class TransferService {

    private static final Logger LOG = Logger.getLogger(TransferService.class.getName());
    private final TransferRepository repository;
    private final AccountServiceClient accountClient;

    public TransferService(TransferRepository repository, AccountServiceClient accountClient) {
        this.repository = repository;
        this.accountClient = accountClient;
    }

    public Transfer initiateTransfer(String from, String to, BigDecimal amount) {
        var correlationId = UUID.randomUUID().toString();
        LOG.info("Transfer initiated: from=" + from + ", to=" + to
            + ", amount=" + amount + ", correlationId=" + correlationId);

        var transfer = new Transfer();
        transfer.setFromAccount(from);
        transfer.setToAccount(to);
        transfer.setAmount(amount);
        transfer.setStatus("PENDING");
        transfer.setCorrelationId(correlationId);
        repository.save(transfer);

        // Step 1: Debit the source account
        try {
            accountClient.debit(from, amount, correlationId);
            transfer.setStatus("DEBITED");
            repository.save(transfer);
        } catch (RestClientException ex) {
            transfer.setStatus("FAILED");
            transfer.setErrorMessage("Debit failed: " + ex.getMessage());
            repository.save(transfer);
            throw new TransferFailedException("Debit failed for " + from, ex);
        }

        // Step 2: Credit the destination account
        try {
            accountClient.credit(to, amount, correlationId);
            transfer.setStatus("COMPLETED");
            repository.save(transfer);
        } catch (RestClientException ex) {
            // PARTIAL FAILURE: debit succeeded but credit failed
            transfer.setStatus("PARTIAL_FAILURE");
            transfer.setErrorMessage("Credit failed after debit: " + ex.getMessage());
            repository.save(transfer);
            LOG.severe("PARTIAL FAILURE: " + from + " debited but " + to + " not credited. "
                + "correlationId=" + correlationId);
            throw new TransferFailedException(
                "Partial failure: " + from + " debited but " + to + " credit failed", ex);
        }

        LOG.info("Transfer completed: correlationId=" + correlationId);
        return transfer;
    }
}

TransferController.java

JAVA
package com.bank.transfer.controller;

import com.bank.transfer.dto.TransferRequest;
import com.bank.transfer.model.Transfer;
import com.bank.transfer.service.TransferService;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/transfers")
public class TransferController {

    private final TransferService transferService;

    public TransferController(TransferService transferService) {
        this.transferService = transferService;
    }

    @PostMapping
    public ResponseEntity<Transfer> createTransfer(@RequestBody TransferRequest request) {
        var transfer = transferService.initiateTransfer(
            request.fromAccount(), request.toAccount(), request.amount());
        return ResponseEntity.ok(transfer);
    }
}

TransferFailedException.java

JAVA
package com.bank.transfer.exception;

public class TransferFailedException extends RuntimeException {
    public TransferFailedException(String message, Throwable cause) {
        super(message, cause);
    }
}

GlobalExceptionHandler.java (transfer-service)

JAVA
package com.bank.transfer.exception;

import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;

import java.util.Map;

@RestControllerAdvice
public class GlobalExceptionHandler {

    @ExceptionHandler(TransferFailedException.class)
    public ResponseEntity<Map<String, String>> handleTransferFailed(TransferFailedException ex) {
        return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
            .body(Map.of("error", ex.getMessage()));
    }
}

transfer-service application.yml

YAML
server:
  port: 8082

spring:
  application:
    name: transfer-service
  datasource:
    url: jdbc:postgresql://localhost:5432/bank
    username: bank
    password: bank123
  jpa:
    hibernate:
      ddl-auto: none
    show-sql: true

account:
  service:
    url: http://localhost:8081

TransferServiceApp.java

JAVA
package com.bank.transfer;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class TransferServiceApp {
    public static void main(String[] args) {
        SpringApplication.run(TransferServiceApp.class, args);
    }
}

How to run the project

BASH
# Step 1: Start PostgreSQL
docker-compose up -d postgres

# Step 2: Build both services
cd account-service && mvn clean package -DskipTests && cd ..
cd transfer-service && mvn clean package -DskipTests && cd ..

# Step 3: Start account-service
java -jar account-service/target/account-service-1.0.0.jar &

# Step 4: Start transfer-service
java -jar transfer-service/target/transfer-service-1.0.0.jar &

# Step 5: Test successful transfer
curl -s -X POST http://localhost:8082/transfers \
  -H "Content-Type: application/json" \
  -d '{"fromAccount":"SAV2001","toAccount":"SAV2002","amount":10000,"description":"rent payment"}'

# Step 6: Check account balances
curl -s http://localhost:8081/accounts/SAV2001
curl -s http://localhost:8081/accounts/SAV2002
TEXT
Expected output (Step 5, successful transfer):
{
  "id": 1,
  "fromAccount": "SAV2001",
  "toAccount": "SAV2002",
  "amount": 10000.00,
  "status": "COMPLETED",
  "correlationId": "a1b2c3d4-e5f6-...",
  "errorMessage": null
}

Expected output (Step 6, balances after transfer):
SAV2001: {"accountNumber":"SAV2001","holderName":"Rajesh Kumar","balance":490000.00}
SAV2002: {"accountNumber":"SAV2002","holderName":"Priya Sharma","balance":210000.00}

What happens when account-service is down?

When account-service is stopped or unreachable, the RestClient in transfer-service throws an exception. The type of exception depends on the failure mode.

If account-service is completely down (process not running), the RestClient throws a ResourceAccessException wrapping a ConnectException. If account-service is running but slow (overloaded), the RestClient throws a ResourceAccessException wrapping a SocketTimeoutException after the configured read timeout.

JAVA
package com.bank.transfer;

import java.net.ConnectException;
import java.net.SocketTimeoutException;

public class ServiceDownDemo {

    sealed interface DownstreamFailure
        permits ConnectionRefused, ReadTimeout, HttpError {}

    record ConnectionRefused(String service, int port) implements DownstreamFailure {}
    record ReadTimeout(String service, int timeoutMs) implements DownstreamFailure {}
    record HttpError(String service, int statusCode, String body) implements DownstreamFailure {}

    public static void main(String[] args) {
        var failures = java.util.List.of(
            new ConnectionRefused("account-service", 8081),
            new ReadTimeout("account-service", 5000),
            new HttpError("account-service", 500, "Internal Server Error")
        );

        for (var failure : failures) {
            String action = switch (failure) {
                case ConnectionRefused c -> "Service " + c.service() + " is DOWN on port " + c.port()
                    + ". Action: return 503 to caller, mark transfer as FAILED.";
                case ReadTimeout r -> "Service " + r.service() + " timed out after " + r.timeoutMs()
                    + "ms. Action: the request MAY have succeeded. Do NOT retry blindly.";
                case HttpError h -> "Service " + h.service() + " returned HTTP " + h.statusCode()
                    + ". Action: parse error body, decide if retryable.";
            };
            System.out.println(action);
        }
    }
}
TEXT
Expected output:
Service account-service is DOWN on port 8081. Action: return 503 to caller, mark transfer as FAILED.
Service account-service timed out after 5000ms. Action: the request MAY have succeeded. Do NOT retry blindly.
Service account-service returned HTTP 500. Action: parse error body, decide if retryable.
BASH
# Demonstrate: stop account-service and try a transfer
kill $(lsof -ti:8081)  # Stop account-service

curl -s -X POST http://localhost:8082/transfers \
  -H "Content-Type: application/json" \
  -d '{"fromAccount":"SAV2001","toAccount":"SAV2002","amount":5000,"description":"test"}'
TEXT
Expected output (account-service is down):
{
  "error": "Debit failed for SAV2001"
}
HTTP Status: 503 Service Unavailable

How do correlation IDs work across services?

A correlation ID is a unique identifier generated at the entry point of a request and passed through every service in the chain. It allows you to trace a single business operation across multiple services in the logs.

In the banking project above, the TransferService generates a UUID as the correlation ID and passes it in the HTTP header X-Correlation-ID to account-service. Both services log the correlation ID, so you can search logs by this value and see the complete path of a request.

JAVA
package com.bank.distributed;

import java.util.UUID;

public class CorrelationIdDemo {

    record LogEntry(String service, String correlationId, String message) {
        @Override
        public String toString() {
            return "[" + service + "] [" + correlationId + "] " + message;
        }
    }

    public static void main(String[] args) {
        var correlationId = UUID.randomUUID().toString().substring(0, 8);

        var logs = java.util.List.of(
            new LogEntry("api-gateway", correlationId, "Received POST /transfers"),
            new LogEntry("transfer-service", correlationId, "Transfer initiated: SAV2001 -> SAV2002, amount=50000"),
            new LogEntry("transfer-service", correlationId, "Calling account-service: debit SAV2001"),
            new LogEntry("account-service", correlationId, "Debit request: account=SAV2001, amount=50000"),
            new LogEntry("account-service", correlationId, "Debit success: newBalance=450000"),
            new LogEntry("transfer-service", correlationId, "Calling account-service: credit SAV2002"),
            new LogEntry("account-service", correlationId, "Credit request: account=SAV2002, amount=50000"),
            new LogEntry("account-service", correlationId, "Credit success: newBalance=250000"),
            new LogEntry("transfer-service", correlationId, "Transfer completed"),
            new LogEntry("api-gateway", correlationId, "Response 200 OK")
        );

        System.out.println("Distributed trace for correlationId=" + correlationId + ":");
        logs.forEach(System.out::println);
    }
}
TEXT
Expected output:
Distributed trace for correlationId=3f7a91c2:
[api-gateway] [3f7a91c2] Received POST /transfers
[transfer-service] [3f7a91c2] Transfer initiated: SAV2001 -> SAV2002, amount=50000
[transfer-service] [3f7a91c2] Calling account-service: debit SAV2001
[account-service] [3f7a91c2] Debit request: account=SAV2001, amount=50000
[account-service] [3f7a91c2] Debit success: newBalance=450000
[transfer-service] [3f7a91c2] Calling account-service: credit SAV2002
[account-service] [3f7a91c2] Credit request: account=SAV2002, amount=50000
[account-service] [3f7a91c2] Credit success: newBalance=250000
[transfer-service] [3f7a91c2] Transfer completed
[api-gateway] [3f7a91c2] Response 200 OK

What are common mistakes when building distributed microservices?

Wrong: no timeout on REST calls

The most dangerous mistake is calling another service without configuring timeouts. If the downstream service hangs, your service hangs too. This is called cascading failure: one slow service makes every caller slow, and eventually the entire system is unresponsive.

JAVA
package com.bank.transfer.wrong;

import org.springframework.web.client.RestClient;

public class WrongNoTimeout {
    public static void main(String[] args) {
        // WRONG: no timeout configured
        var client = RestClient.builder()
            .baseUrl("http://account-service:8081")
            .build();  // Default timeout is INFINITE in some implementations

        System.out.println("WRONG: RestClient with no timeout");
        System.out.println("  If account-service hangs, this thread blocks FOREVER.");
        System.out.println("  All threads fill up, transfer-service becomes unresponsive.");
        System.out.println("  This is a cascading failure.");
    }
}
TEXT
Expected output:
WRONG: RestClient with no timeout
  If account-service hangs, this thread blocks FOREVER.
  All threads fill up, transfer-service becomes unresponsive.
  This is a cascading failure.

Correct: configure connect and read timeouts

JAVA
package com.bank.transfer.correct;

import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.web.client.RestClient;

public class CorrectWithTimeout {
    public static void main(String[] args) {
        var factory = new SimpleClientHttpRequestFactory();
        factory.setConnectTimeout(3000);  // 3 seconds to connect
        factory.setReadTimeout(5000);     // 5 seconds to read response

        var client = RestClient.builder()
            .baseUrl("http://account-service:8081")
            .requestFactory(factory)
            .build();

        System.out.println("CORRECT: RestClient with timeouts");
        System.out.println("  Connect timeout: 3000ms (fail fast if service is down)");
        System.out.println("  Read timeout: 5000ms (fail fast if service is slow)");
        System.out.println("  Thread is released after timeout, no cascading failure.");
    }
}
TEXT
Expected output:
CORRECT: RestClient with timeouts
  Connect timeout: 3000ms (fail fast if service is down)
  Read timeout: 5000ms (fail fast if service is slow)
  Thread is released after timeout, no cascading failure.

How do you test distributed failure scenarios?

Use WireMock to simulate a downstream service that is slow, returns errors, or drops connections. This lets you test your error handling without running the actual downstream service.

JAVA
package com.bank.transfer;

import com.github.tomakehurst.wiremock.WireMockServer;
import com.github.tomakehurst.wiremock.client.WireMock;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.web.client.RestClient;
import org.springframework.web.client.RestClientException;

import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
import static com.github.tomakehurst.wiremock.client.WireMock.post;
import static com.github.tomakehurst.wiremock.client.WireMock.urlPathMatching;

public class WireMockFailureTest {
    public static void main(String[] args) {
        var wireMock = new WireMockServer(9999);
        wireMock.start();

        // Simulate: account-service responds with 2-second delay
        wireMock.stubFor(post(urlPathMatching("/accounts/.*/debit"))
            .willReturn(aResponse()
                .withStatus(200)
                .withFixedDelay(2000)
                .withBody("{\"accountNumber\":\"SAV2001\",\"balance\":450000}")));

        var factory = new SimpleClientHttpRequestFactory();
        factory.setConnectTimeout(1000);
        factory.setReadTimeout(1000);  // 1 second timeout, but response takes 2 seconds

        var client = RestClient.builder()
            .baseUrl("http://localhost:9999")
            .requestFactory(factory)
            .build();

        try {
            client.post()
                .uri("/accounts/SAV2001/debit")
                .body(java.util.Map.of("amount", 10000))
                .retrieve()
                .body(String.class);
            System.out.println("Unexpected success");
        } catch (RestClientException ex) {
            System.out.println("Test PASSED: caught timeout exception");
            System.out.println("Exception: " + ex.getClass().getSimpleName());
            System.out.println("Message: " + ex.getMessage());
        } finally {
            wireMock.stop();
        }
    }
}
TEXT
Expected output:
Test PASSED: caught timeout exception
Exception: ResourceAccessException
Message: I/O error on POST request for "http://localhost:9999/accounts/SAV2001/debit": Read timed out

What should a senior or principal engineer know?

At the senior level, you must understand the CAP theorem and its practical implications. The CAP theorem states that a distributed system can provide at most two of three guarantees: Consistency (every read returns the most recent write), Availability (every request receives a response), and Partition tolerance (the system continues to operate despite network partitions). Since network partitions always happen in production, you must choose between consistency and availability.

At the principal level, you need to design for observability from day one. Every inter-service call must be traceable. Use structured logging with correlation IDs. Emit metrics for request rate, error rate, and latency (the RED method). Set up alerts on error rates and latency percentiles (p99), not averages.

  • Key areas of expertise:
  • Fallacies of distributed computing: the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology does not change. All of these are false in production.
  • Idempotency: every write operation that crosses a network must be idempotent. If a response is lost, the client will retry. If the operation is not idempotent, you get double writes.
  • Circuit breaker: when a downstream service fails repeatedly, stop sending requests to it. Give it time to recover. This prevents cascading failures.
  • Bulkhead isolation: isolate thread pools per downstream service. If account-service is slow, it should not exhaust the threads used for notification-service.
JAVA
package com.bank.distributed;

public class SeniorChecklist {
    public static void main(String[] args) {
        System.out.println("Senior/Principal distributed systems checklist:");
        System.out.println("  1. Configure timeouts on EVERY outgoing HTTP call");
        System.out.println("  2. Add correlation IDs to every inter-service request");
        System.out.println("  3. Make all write operations idempotent (use idempotency keys)");
        System.out.println("  4. Use circuit breakers to prevent cascading failures");
        System.out.println("  5. Design for partial failure: what if step 2 of 3 fails?");
        System.out.println("  6. Use structured logging with JSON format for log aggregation");
        System.out.println("  7. Monitor RED metrics: Request rate, Error rate, Duration");
        System.out.println("  8. Test failure scenarios with WireMock and chaos engineering");
        System.out.println("  9. Understand CAP theorem: choose CP or AP per use case");
        System.out.println("  10. Use the outbox pattern for reliable event publishing");
    }
}
TEXT
Expected output:
Senior/Principal distributed systems checklist:
  1. Configure timeouts on EVERY outgoing HTTP call
  2. Add correlation IDs to every inter-service request
  3. Make all write operations idempotent (use idempotency keys)
  4. Use circuit breakers to prevent cascading failures
  5. Design for partial failure: what if step 2 of 3 fails?
  6. Use structured logging with JSON format for log aggregation
  7. Monitor RED metrics: Request rate, Error rate, Duration
  8. Test failure scenarios with WireMock and chaos engineering
  9. Understand CAP theorem: choose CP or AP per use case
  10. Use the outbox pattern for reliable event publishing

What are common interview follow-up questions?

  • What is the CAP theorem and which tradeoff would you choose for a banking application? (CP: consistency over availability. A bank cannot show a wrong balance, even if it means the service is temporarily unavailable.)
  • How do you handle a partial failure where the debit succeeds but the credit fails? (Record the transfer state as PARTIAL_FAILURE. Run a reconciliation job that detects partial failures and either completes or reverses them. Use the Saga pattern for complex workflows.)
  • What is the difference between a timeout and a connection refused? (Connection refused means the service is not running at all. Timeout means the service is running but too slow to respond. With a timeout, the request may have already been processed on the server side.)
  • How do you prevent cascading failures? (Use timeouts, circuit breakers, bulkhead isolation, and rate limiting. If one downstream service is slow, it should not bring down the entire system.)
  • What is the difference between synchronous and asynchronous communication in microservices? (Synchronous: the caller waits for a response, like HTTP REST. Asynchronous: the caller sends a message and moves on, like Kafka or RabbitMQ. Asynchronous decouples services and improves resilience.)

Visual Overview

DIAGRAM
 Distributed Banking System Architecture
 =========================================

 ┌─────────────────┐         ┌─────────────────┐
 │  transfer-service│         │  account-service │
 │  (port 8082)     │         │  (port 8081)     │
 │                  │  HTTP   │                  │
 │  TransferService │────────>│  AccountService  │
 │    1. debit()    │  REST   │    debit()       │
 │    2. credit()   │────────>│    credit()      │
 │                  │         │                  │
 └────────┬────────┘         └────────┬─────────┘
          │                           │
          │ JDBC                      │ JDBC
          ▼                           ▼
 ┌─────────────────┐         ┌─────────────────┐
 │   transfers      │         │   accounts       │
 │   table          │         │   table          │
 └─────────────────┘         └─────────────────┘
          └────────────┬──────────────┘
                       │
                ┌──────┴──────┐
                │  PostgreSQL  │
                │  (port 5432) │
                └─────────────┘

 Failure Scenarios
 ==================

 Scenario 1: Connection Refused
 transfer-service ──X──> account-service (DOWN)
 Result: TransferFailedException, status=FAILED

 Scenario 2: Read Timeout
 transfer-service ──────> account-service (SLOW, 10s)
 Result: SocketTimeoutException after 5s, status=FAILED

 Scenario 3: Partial Failure
 transfer-service ──────> debit SAV2001 (OK)
 transfer-service ──X──> credit SAV2002 (FAIL)
 Result: SAV2001 debited, SAV2002 NOT credited, status=PARTIAL_FAILURE

Key Takeaway

A distributed system is a collection of independent processes that communicate over a network. In a monolithic application, every function call is local and reliable. In a microservices architecture, every inter-service call crosses a network boundary, introducing the possibility of timeouts, connection failures, and partial failures.

The banking project in this question demonstrates this clearly. The transfer-service calls account-service twice: once to debit the sender and once to credit the receiver. If the first call succeeds but the second fails, the system is in an inconsistent state. The sender lost money, but the receiver never got it. This is a partial failure, and it is the defining challenge of distributed systems.

To handle these failures, you must configure timeouts on every outgoing HTTP call, use correlation IDs for end-to-end tracing, and design every write operation to be idempotent. At the senior level, you add circuit breakers to prevent cascading failures, bulkhead isolation to protect thread pools, and the Saga pattern for multi-step workflows that need compensating actions.

In our National Bank of India transfer project, Rajesh's account SAV2001 was debited successfully, but Priya's account SAV2002 was never credited because account-service restarted at the wrong moment. The transfer record shows status PARTIAL_FAILURE with the correlation ID. The operations team can search logs by this correlation ID and see exactly where the failure occurred across both services.

Remember one sentence: microservices are distributed systems, and every network call can fail, time out, or succeed on the server but lose the response, so you must design for partial failure from day one.

---

Share
Use ← → arrow keys to navigate