Definition
- Process isolation – each instance is a separate JVM with its own heap; local state does not propagate.
- Shared database – the source of truth every instance reads; provides strong consistency and durability.
- Distributed cache (Redis) – a shared in-memory store all instances talk to; provides fast reads with a TTL.
- Kafka event – App1 publishes a change and other instances consume it to update local state or invalidate caches.
- Layered approach – combine database for truth, Redis for speed, and events for freshness.
- Rule – never store mutable shared state in instance-local memory.
Imagine National Bank of India has two identical counters, each with its own private notepad. If the clerk at counter one writes "Rajesh's new balance is 950000" on their notepad, the clerk at counter two has no idea: their notepad still says 1000000. For counter two to know, they must either both read from the central ledger in the vault (shared database), both check a shared whiteboard on the wall (Redis), or counter one must shout an announcement (Kafka event). Private notepads never sync by themselves.
Understanding the Concepts
Why does process isolation matter?
Each instance is a distinct OS process with its own JVM and heap. A field, a static map, a Spring singleton's state, or an in-process cache like Caffeine all live only in that instance's heap. Another instance has its own separate copy and never sees changes. This is why keeping mutable shared state in instance memory is a classic scaling bug: it works in development with one instance and silently breaks in production behind a load balancer.
package com.bank.cache;
import java.math.BigDecimal;
import java.util.concurrent.ConcurrentHashMap;
// WRONG: local cache that does not sync across instances
public class LocalBalanceCache {
private final ConcurrentHashMap<String, BigDecimal> cache
= new ConcurrentHashMap<>();
public void update(String accountId, BigDecimal balance) {
cache.put(accountId, balance); // Only this instance sees it!
}
public BigDecimal get(String accountId) {
return cache.get(accountId); // Stale on other instances!
}
}
The bug:
Instance 1: updates Rajesh's balance to 950000 in local cache
Instance 2: still has 1000000 in its local cache
Load balancer sends next request to Instance 2
Result: Rajesh sees wrong (stale) balance!
Works in dev (1 instance), breaks in prod (N instances)
How does a shared database solve this?
Every instance writes the authoritative value to a shared database and reads it back. This is the correct default for anything that must be consistent and durable, like money.
package com.bank.service;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class AccountService {
private final AccountRepository repository;
public AccountService(AccountRepository repository) {
this.repository = repository;
}
@Transactional
public void updateBalance(String accountId, BigDecimal amount) {
// Writes to shared database: all instances see it
repository.updateBalance(accountId, amount);
}
public BigDecimal getBalance(String accountId) {
// Reads from shared database: always current
return repository.findBalanceByAccountId(accountId);
}
}
Shared database flow:
Instance 1: UPDATE accounts SET balance=950000 WHERE id='SAV2001'
Instance 2: SELECT balance FROM accounts WHERE id='SAV2001'
Result: Instance 2 reads 950000 (correct, current)
Cost: database round trip on every read
How does Redis provide fast cross-instance visibility?
Redis is an external, shared, in-memory store that all instances talk to. A value written by App1 is immediately readable by App2. It is used both as a fast cache in front of the database and for shared transient state like sessions and rate-limit counters.
package com.bank.cache;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.time.Duration;
@Service
public class RedisBalanceCache {
private final StringRedisTemplate redis;
private final AccountRepository repository;
public RedisBalanceCache(StringRedisTemplate redis,
AccountRepository repository) {
this.redis = redis;
this.repository = repository;
}
// Cache-aside pattern: check Redis, then DB
public BigDecimal getBalance(String accountId) {
String cached = redis.opsForValue()
.get("balance:" + accountId);
if (cached != null) {
return new BigDecimal(cached); // Shared cache hit
}
// Miss: read from DB, populate Redis
BigDecimal balance = repository.findBalance(accountId);
redis.opsForValue().set("balance:" + accountId,
balance.toPlainString(), Duration.ofSeconds(30));
return balance;
}
public void invalidate(String accountId) {
redis.delete("balance:" + accountId);
}
}
Redis flow:
Instance 1: writes balance to DB + invalidates Redis key
Instance 2: reads from Redis (miss) -> reads from DB -> populates Redis
Instance 3: reads from Redis (hit) -> returns cached value
All instances see the same value via shared Redis
How do Kafka events propagate changes?
When App1 makes a change, it publishes an event. All other instances consume it and react by updating their local state or invalidating their local caches. This is push instead of pull: App1 tells everyone when it changes, rather than everyone polling.
package com.bank.events;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
@Service
public class BalanceEventPublisher {
private final KafkaTemplate<String, BalanceChangedEvent> kafka;
public BalanceEventPublisher(
KafkaTemplate<String, BalanceChangedEvent> kafka) {
this.kafka = kafka;
}
public void publishBalanceChanged(String accountId,
BigDecimal newBalance) {
kafka.send("balance-events",
accountId,
new BalanceChangedEvent(accountId, newBalance));
}
}
package com.bank.events;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;
@Service
public class BalanceEventConsumer {
private final RedisBalanceCache cache;
public BalanceEventConsumer(RedisBalanceCache cache) {
this.cache = cache;
}
@KafkaListener(topics = "balance-events")
public void onBalanceChanged(BalanceChangedEvent event) {
// Invalidate local and Redis cache so next read is fresh
cache.invalidate(event.getAccountId());
}
}
Kafka event flow:
Instance 1: updates DB -> publishes BalanceChangedEvent
Instance 2: consumes event -> invalidates Redis cache
Instance 3: consumes event -> invalidates Redis cache
Next read on any instance: cache miss -> reads fresh from DB
Trade-off: brief eventual consistency delay between publish and consume
How do the three mechanisms compare?
| Mechanism | Consistency | Latency | Durability | Best For |
|---|---|---|---|---|
| Shared DB | Strong | High (disk) | Yes | Source of truth, money |
| Redis | Eventual (TTL) | Low (memory) | No (cache) | Hot reads, sessions |
| Kafka event | Eventual (async) | Medium | Yes (log) | Cache invalidation, reactions |
How do you layer them in practice?
Combine all three: the database as source of truth, Redis for fast reads, and Kafka events for cache invalidation.
package com.bank.service;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
@Service
public class LayeredAccountService {
private final AccountRepository repository;
private final RedisBalanceCache cache;
private final BalanceEventPublisher publisher;
public LayeredAccountService(AccountRepository repository,
RedisBalanceCache cache,
BalanceEventPublisher publisher) {
this.repository = repository;
this.cache = cache;
this.publisher = publisher;
}
@Transactional
public void updateBalance(String accountId, BigDecimal newBalance) {
// 1. Write to DB (source of truth)
repository.updateBalance(accountId, newBalance);
// 2. Invalidate Redis (so no stale cache)
cache.invalidate(accountId);
// 3. Publish event (so other instances invalidate too)
publisher.publishBalanceChanged(accountId, newBalance);
}
public BigDecimal getBalance(String accountId) {
// Read path: Redis -> DB (cache-aside)
return cache.getBalance(accountId);
}
}
Layered approach:
Write path: DB (truth) -> invalidate Redis -> publish event
Read path: Redis (fast) -> DB (on miss)
Other instances: consume event -> invalidate their Redis keys
Result: all instances converge quickly to the correct value
What are common mistakes?
The most common mistake is storing mutable shared state in a local ConcurrentHashMap or Caffeine cache and assuming all instances see updates.
WRONG: local in-memory cache
Instance 1 updates local cache -> other instances stale
Works in dev (1 instance), breaks in prod (N instances)
CORRECT: externalized state
DB for truth + Redis for speed + Kafka events for freshness
All instances see the same data through shared external stores
What should a senior engineer know?
A senior engineer decides per value whether strong consistency (always read the database) or eventual consistency (cache plus events with a brief stale window) is acceptable. For Rajesh's authoritative balance, the database is the source of truth and every write must go there. For a dashboard counter or a leaderboard, eventual consistency via Redis and events is fine. The layered approach (DB, Redis, Kafka) is not one-size-fits-all; each value has a freshness requirement.
package com.bank.architecture;
public class ConsistencyDecision {
public static void main(String[] args) {
System.out.println("Senior decisions per data type:");
System.out.println(" Balance (critical): always read DB, never local cache");
System.out.println(" Rate display (non-critical): Redis with 30s TTL is fine");
System.out.println(" Session token: Redis is the source of truth");
System.out.println(" Notification badge count: eventual via Kafka events");
System.out.println();
System.out.println("Never store mutable shared state in local JVM memory.");
System.out.println("Local caches are read-only and must be invalidated.");
}
}
Expected output:
Senior decisions per data type:
Balance (critical): always read DB, never local cache
Rate display (non-critical): Redis with 30s TTL is fine
Session token: Redis is the source of truth
Notification badge count: eventual via Kafka events
Never store mutable shared state in local JVM memory.
Local caches are read-only and must be invalidated.
Visual Overview
The Problem: Process Isolation
┌──────────────┐ ┌──────────────┐
│ Instance 1 │ │ Instance 2 │
│ JVM Heap │ │ JVM Heap │
│ balance=950K │ │ balance=1M │ <- STALE!
└──────────────┘ └──────────────┘
No shared memory between JVMs.
The Solution: Externalized State
┌──────────────┐ ┌──────────────┐
│ Instance 1 │ │ Instance 2 │
└──────┬───────┘ └──────┬───────┘
│ write │ read
▼ ▼
┌──────────────────────────────────────────────┐
│ Redis (shared cache) │
└──────────────────┬───────────────────────────┘
│ cache miss
▼
┌──────────────────────────────────────────────┐
│ PostgreSQL (source of truth) │
└──────────────────────────────────────────────┘
┌──────────────┐ event ┌───────┐ event ┌──────────────┐
│ Instance 1 │ ──────> │ Kafka │ ──────> │ Instance 2 │
│ (publisher) │ │ │ │ (invalidate) │
└──────────────┘ └───────┘ └──────────────┘
Key Takeaway
The starting point is process isolation: each instance is a separate JVM with its own heap, so a value updated in one instance is invisible to others. A field, a static map, or an in-process cache all live only in that one instance. This is why keeping mutable shared state in instance memory is a classic scaling bug that works in development and breaks in production.
To make App2 observe App1's change, you externalize the state through three mechanisms. The shared database is the source of truth for consistent, durable data like Rajesh's balance. Redis is a distributed cache all instances share for fast reads with a TTL. Kafka events push changes so other instances invalidate their caches promptly rather than waiting for a TTL to expire. In practice you layer all three: the database for correctness, Redis for speed, and events for freshness.
At National Bank of India, the balance service initially kept balances in a local cache, and a follow-up read routed to another instance returned the stale value. The fix was to write to the database as the source of truth, invalidate Redis, and publish a BalanceChangedEvent that all instances consume to evict their caches, so the next read on any instance returns the correct value.
Remember one sentence: separate instances share no memory, so externalize state through a shared database for truth, Redis for speed, and Kafka events for freshness, and never rely on local memory for state that must be shared.
---