Definition
- Business invariant – a correctness constraint whose violation causes real financial or legal harm; phrased as "must never happen."
- Hard invariant – demands strong consistency: linearizable writes, ACID transactions, row locking (e.g., balance must never go negative).
- Cosmetic data – has no hard invariant and can lag; permits eventual consistency and high availability (e.g., activity feed, profile picture).
- Requirements-driven – classify each data item by its invariant first, then pick the technology that enforces it.
- Anti-pattern – picking a database (Cassandra, DynamoDB) first and hoping the business rules fit its consistency model.
- Split-consistency – even within one transaction, the money core is strongly consistent while downstream notifications are eventually consistent.
At the National Bank of India, some rules can never be broken: Rajesh's available balance must never go negative, because that is real money that does not exist. Other things are cosmetic: his profile photo or his activity feed can be a few seconds out of date and nobody is harmed. You do not pick a database first and then hope it fits. You first ask "what must never go wrong here?" and then choose the technology that enforces exactly that. Money rules get the strict vault; cosmetic data gets the fast, relaxed store.
Understanding the Concepts
Why is "pick consistency or availability" an oversimplification?
Real systems are a mix. A single application has many data items and operations, each with a different tolerance for staleness and conflict. Labeling the whole system with one CAP letter forces the wrong trade-off somewhere.
Oversimplification:
"Our system is AP." ──► Every piece of data is eventually consistent
Problem: balance goes negative during concurrent withdrawals
Correct approach:
Per-data classification:
Balance ledger ──► Strong consistency (CP)
Activity feed ──► Eventual consistency (AP)
Like counter ──► Eventual consistency (AP)
Profile photo ──► Eventual consistency (AP)
How do you start from the business invariant?
Phrase each rule as a "must never" statement. If violation causes financial or legal harm, the data demands strong consistency. If violation is merely cosmetic, eventual consistency is safe.
package com.bank.model;
/**
* Business invariants for National Bank of India.
* Each one determines the consistency model for its data.
*/
public class BusinessInvariants {
// HARD INVARIANTS (strong consistency required)
// "The available balance must never go negative."
// "A loan must not be disbursed twice."
// "The last item in stock must not be sold to two customers."
// SOFT/COSMETIC (eventual consistency acceptable)
// "The activity feed can lag a few seconds."
// "The profile picture can be stale briefly after an update."
// "The like count can be approximate."
// "Recently viewed items can be eventually consistent."
}
What happens when you violate a hard invariant?
Violating a hard invariant causes real loss. If "balance must never go negative" is violated, the bank loses money.
package com.bank.service;
import java.math.BigDecimal;
import java.sql.*;
import java.util.logging.Logger;
/**
* Wrong: using eventual consistency for balance.
* Two concurrent withdrawals can both succeed,
* driving the balance negative.
*/
public class UnsafeWithdrawalService {
private static final Logger LOG = Logger.getLogger(
UnsafeWithdrawalService.class.getName());
public void withdraw(Connection conn, String accountId, BigDecimal amount)
throws SQLException {
// Read from a (potentially stale) replica
BigDecimal balance = readFromReplica(accountId);
if (balance.compareTo(amount) >= 0) {
// Both threads pass this check with stale balance
String sql = "UPDATE accounts SET balance = balance - ? "
+ "WHERE account_id = ?";
try (PreparedStatement ps = conn.prepareStatement(sql)) {
ps.setBigDecimal(1, amount);
ps.setString(2, accountId);
ps.executeUpdate();
}
}
// BUG: balance can go negative under concurrent access
}
private BigDecimal readFromReplica(String accountId) {
// Simulated stale read from an AP replica
return new BigDecimal("50000.00");
}
}
package com.bank.service;
import java.math.BigDecimal;
import java.sql.*;
import java.util.logging.Logger;
/**
* Correct: strong consistency for balance.
* Uses the primary with a CHECK constraint.
*/
public class SafeWithdrawalService {
private static final Logger LOG = Logger.getLogger(
SafeWithdrawalService.class.getName());
public boolean withdraw(Connection conn, String accountId, BigDecimal amount)
throws SQLException {
// Read-and-write on the primary with FOR UPDATE lock
String sql = "UPDATE accounts SET balance = balance - ? "
+ "WHERE account_id = ? AND balance >= ?";
try (PreparedStatement ps = conn.prepareStatement(sql)) {
ps.setBigDecimal(1, amount);
ps.setString(2, accountId);
ps.setBigDecimal(3, amount);
int rows = ps.executeUpdate();
if (rows == 0) {
LOG.warning("Insufficient balance for " + accountId);
return false;
}
return true;
}
// The CHECK (balance >= 0) constraint on the table
// is the final safety net even if the WHERE is bypassed.
}
}
How do you classify data and pick technology?
After classifying each data item by its invariant, the technology choice becomes obvious.
Classification Table:
Data Item Invariant Consistency Technology
───────────────── ────────────────── ──────────── ─────────────────
Account balance Must never go < 0 Strong (CP) PostgreSQL primary
Loan disbursement Must not disburse Strong (CP) PostgreSQL + lock
twice
Activity feed Can lag seconds Eventual (AP) Cassandra / Redis
Profile picture Can be stale Eventual (AP) S3 + CDN
Like counter Can be approximate Eventual (AP) Redis counter
Notifications Can delay Eventual (AP) Kafka + consumer
-- Strong consistency: balance ledger on PostgreSQL
CREATE TABLE accounts (
account_id VARCHAR(20) PRIMARY KEY,
holder_name VARCHAR(100) NOT NULL,
balance NUMERIC(15,2) NOT NULL DEFAULT 0.00,
CONSTRAINT chk_balance_non_negative CHECK (balance >= 0)
);
-- The CHECK constraint is the database-level invariant enforcement.
-- No amount of application logic can bypass it.
How do you split consistency within one transaction?
Even within one business transaction, you split the work. The money-moving core runs strongly consistent and synchronous. The downstream effects fire as asynchronous events that are allowed to be eventually consistent.
package com.bank.service;
import java.math.BigDecimal;
import java.sql.*;
import java.util.logging.Logger;
public class TransferWithSplitConsistency {
private static final Logger LOG = Logger.getLogger(
TransferWithSplitConsistency.class.getName());
private final Connection primaryConn; // CP: strong consistency
public TransferWithSplitConsistency(Connection primaryConn) {
this.primaryConn = primaryConn;
}
public void transfer(String sourceId, String targetId, BigDecimal amount)
throws SQLException {
// STEP 1: Strongly consistent (CP)
// Debit and credit in one ACID transaction on the primary
primaryConn.setAutoCommit(false);
try {
debit(sourceId, amount);
credit(targetId, amount);
insertOutboxEvent(sourceId, targetId, amount);
primaryConn.commit();
LOG.info("Transfer committed: " + amount);
} catch (SQLException e) {
primaryConn.rollback();
throw e;
}
// STEP 2: Eventually consistent (AP)
// Outbox event is picked up by a Kafka publisher
// Feed, notification, and analytics services consume it
// A few seconds of lag is acceptable
}
private void debit(String accountId, BigDecimal amount) throws SQLException {
String sql = "UPDATE accounts SET balance = balance - ? "
+ "WHERE account_id = ? AND balance >= ?";
try (PreparedStatement ps = primaryConn.prepareStatement(sql)) {
ps.setBigDecimal(1, amount);
ps.setString(2, accountId);
ps.setBigDecimal(3, amount);
if (ps.executeUpdate() == 0) {
throw new SQLException("Insufficient balance: " + accountId);
}
}
}
private void credit(String accountId, BigDecimal amount) throws SQLException {
String sql = "UPDATE accounts SET balance = balance + ? WHERE account_id = ?";
try (PreparedStatement ps = primaryConn.prepareStatement(sql)) {
ps.setBigDecimal(1, amount);
ps.setString(2, accountId);
ps.executeUpdate();
}
}
private void insertOutboxEvent(String source, String target, BigDecimal amount)
throws SQLException {
String sql = "INSERT INTO outbox (event_type, payload) VALUES (?, ?)";
try (PreparedStatement ps = primaryConn.prepareStatement(sql)) {
ps.setString(1, "TRANSFER_COMPLETED");
ps.setString(2, source + ":" + target + ":" + amount);
ps.executeUpdate();
}
}
}
What is the anti-pattern to avoid?
The anti-pattern is picking a trendy database first and contorting business rules to fit its consistency model.
Anti-pattern: 1. "Cassandra scales, let's use it for everything." 2. Try to enforce "balance >= 0" on eventual consistency. 3. Two concurrent withdrawals both succeed. 4. Balance goes negative: real money lost. Correct approach: 1. Ask: "What must never happen?" 2. Balance must never go negative ──► strong consistency. 3. Choose PostgreSQL with CHECK constraint for balances. 4. Choose Cassandra for the activity feed (no hard invariant).
What are common interview follow-up questions?
- Give an example of a business invariant that requires strong consistency.
- How does the Saga pattern handle eventual consistency across services?
- What is the difference between linearizability and serializability?
- How do you handle the case where a payment must be exactly-once but the messaging system is at-least-once?
What should a senior engineer know?
A senior engineer should be able to classify every data item in the system by its invariant, justify the consistency model per data boundary, and explain why the outbox pattern bridges the CP and AP worlds.
Senior Interview Answer Structure: 1. State the invariant: "balance must never go negative" 2. Choose consistency: strong (CP) for this data 3. Choose technology: PostgreSQL primary with ACID + CHECK 4. Contrast with cosmetic data: feed is AP, Cassandra 5. Bridge pattern: outbox + Kafka for CP → AP events 6. Anti-pattern: picking the DB before the invariant
Visual Overview
┌──────────────────────────────────────────────────────┐ │ Requirements-Driven Consistency Classification │ ├──────────────────────────────────────────────────────┤ │ │ │ Step 1: Identify the invariant │ │ ┌─────────────────────────────────────────┐ │ │ │ "What must NEVER happen?" │ │ │ └──────────────┬──────────────────────────┘ │ │ │ │ │ ┌──────────┴──────────┐ │ │ │ │ │ │ ▼ ▼ │ │ Hard Invariant No Hard Invariant │ │ "Balance < 0" "Feed lags 3s" │ │ │ │ │ │ ▼ ▼ │ │ Strong Consistency Eventual Consistency │ │ (CP) (AP) │ │ │ │ │ │ ▼ ▼ │ │ PostgreSQL Cassandra / Redis │ │ ACID + CHECK Multi-node, always-up │ │ Single primary Last-write-wins │ │ │ │ Step 2: Bridge with Outbox Pattern │ │ ┌──────────────┐ Kafka ┌──────────────┐ │ │ │ CP: Balance │ ────────► │ AP: Feed, │ │ │ │ (committed) │ event │ Notification │ │ │ └──────────────┘ └──────────────┘ │ └──────────────────────────────────────────────────────┘
Key Takeaway
Treating CAP as a single global switch is an oversimplification because consistency needs differ per operation and per piece of data. The disciplined method is to start from the business invariant: the thing that must never happen. Phrase each rule as a "must never" statement. "The available balance must never go negative." "A loan must not be disbursed twice." These are correctness constraints whose violation causes real financial harm, so they demand strong consistency with ACID transactions and database-level constraints.
In contrast, data like an activity feed, a profile picture, or a like count has no hard invariant and can lag a few seconds, so strong consistency buys nothing a user notices while costing latency and availability. There you choose eventual consistency and high availability.
The failure Pankaj always cites is a team that chose Cassandra for everything because it scales, then tried to enforce "balance must never go negative" on eventual consistency. During concurrent withdrawals an account went negative: real money lost. At the National Bank of India, the team fixed this by classifying each data item by its "must never happen" rule first. The balance ledger moved to a strongly-consistent PostgreSQL primary with a non-negative CHECK constraint. The feed and counters stayed on the eventually-consistent store. Even within one transaction, the money-moving core is strongly consistent and synchronous, while the downstream notification and analytics fire as asynchronous outbox events.
The anti-pattern is picking a trendy database first and contorting the business rules to fit. Requirements must drive the technology, not the other way around.
Remember one sentence: start from what must never happen; hard invariants get strong consistency, cosmetic data gets eventual consistency, and requirements choose the database rather than the reverse.
---