Back to Advanced: Replication, Consistency, CAP
Q5Question 5 of 5

Why Consistency Decisions Should Start From the Business Invariant

Key Concepts
  • Treating CAP as one global switch is wrong because consistency requirements vary per operation and per data item.
  • Start from the business invariant: phrase each rule as a "must never" statement.
  • Hard invariants (balance must never go negative, item must not be sold twice) demand strong consistency with ACID transactions.
  • Cosmetic data (activity feed, profile picture, like count) can safely lag and permits eventual consistency.
  • Only after classifying each data item by its invariant do you pick the technology: requirements drive the database choice.
  • Even within one transaction, the money-moving core runs strongly consistent while downstream events are eventually consistent.
  • The anti-pattern is picking a trendy database first and contorting business rules to fit its consistency model.

Definition

Start from "what must never happen": hard invariants demand strong consistency, cosmetic data permits eventual consistency; let requirements choose the technology.
  • Business invariant – a correctness constraint whose violation causes real financial or legal harm; phrased as "must never happen."
  • Hard invariant – demands strong consistency: linearizable writes, ACID transactions, row locking (e.g., balance must never go negative).
  • Cosmetic data – has no hard invariant and can lag; permits eventual consistency and high availability (e.g., activity feed, profile picture).
  • Requirements-driven – classify each data item by its invariant first, then pick the technology that enforces it.
  • Anti-pattern – picking a database (Cassandra, DynamoDB) first and hoping the business rules fit its consistency model.
  • Split-consistency – even within one transaction, the money core is strongly consistent while downstream notifications are eventually consistent.

At the National Bank of India, some rules can never be broken: Rajesh's available balance must never go negative, because that is real money that does not exist. Other things are cosmetic: his profile photo or his activity feed can be a few seconds out of date and nobody is harmed. You do not pick a database first and then hope it fits. You first ask "what must never go wrong here?" and then choose the technology that enforces exactly that. Money rules get the strict vault; cosmetic data gets the fast, relaxed store.

Understanding the Concepts

Why is "pick consistency or availability" an oversimplification?

Real systems are a mix. A single application has many data items and operations, each with a different tolerance for staleness and conflict. Labeling the whole system with one CAP letter forces the wrong trade-off somewhere.

DIAGRAM
Oversimplification:
  "Our system is AP."  ──► Every piece of data is eventually consistent
  Problem: balance goes negative during concurrent withdrawals

Correct approach:
  Per-data classification:
    Balance ledger  ──► Strong consistency (CP)
    Activity feed   ──► Eventual consistency (AP)
    Like counter    ──► Eventual consistency (AP)
    Profile photo   ──► Eventual consistency (AP)

How do you start from the business invariant?

Phrase each rule as a "must never" statement. If violation causes financial or legal harm, the data demands strong consistency. If violation is merely cosmetic, eventual consistency is safe.

JAVA
package com.bank.model;

/**
 * Business invariants for National Bank of India.
 * Each one determines the consistency model for its data.
 */
public class BusinessInvariants {

    // HARD INVARIANTS (strong consistency required)
    // "The available balance must never go negative."
    // "A loan must not be disbursed twice."
    // "The last item in stock must not be sold to two customers."

    // SOFT/COSMETIC (eventual consistency acceptable)
    // "The activity feed can lag a few seconds."
    // "The profile picture can be stale briefly after an update."
    // "The like count can be approximate."
    // "Recently viewed items can be eventually consistent."
}

What happens when you violate a hard invariant?

Violating a hard invariant causes real loss. If "balance must never go negative" is violated, the bank loses money.

JAVA
package com.bank.service;

import java.math.BigDecimal;
import java.sql.*;
import java.util.logging.Logger;

/**
 * Wrong: using eventual consistency for balance.
 * Two concurrent withdrawals can both succeed,
 * driving the balance negative.
 */
public class UnsafeWithdrawalService {

    private static final Logger LOG = Logger.getLogger(
        UnsafeWithdrawalService.class.getName());

    public void withdraw(Connection conn, String accountId, BigDecimal amount)
            throws SQLException {
        // Read from a (potentially stale) replica
        BigDecimal balance = readFromReplica(accountId);

        if (balance.compareTo(amount) >= 0) {
            // Both threads pass this check with stale balance
            String sql = "UPDATE accounts SET balance = balance - ? "
                       + "WHERE account_id = ?";
            try (PreparedStatement ps = conn.prepareStatement(sql)) {
                ps.setBigDecimal(1, amount);
                ps.setString(2, accountId);
                ps.executeUpdate();
            }
        }
        // BUG: balance can go negative under concurrent access
    }

    private BigDecimal readFromReplica(String accountId) {
        // Simulated stale read from an AP replica
        return new BigDecimal("50000.00");
    }
}
JAVA
package com.bank.service;

import java.math.BigDecimal;
import java.sql.*;
import java.util.logging.Logger;

/**
 * Correct: strong consistency for balance.
 * Uses the primary with a CHECK constraint.
 */
public class SafeWithdrawalService {

    private static final Logger LOG = Logger.getLogger(
        SafeWithdrawalService.class.getName());

    public boolean withdraw(Connection conn, String accountId, BigDecimal amount)
            throws SQLException {
        // Read-and-write on the primary with FOR UPDATE lock
        String sql = "UPDATE accounts SET balance = balance - ? "
                   + "WHERE account_id = ? AND balance >= ?";
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setBigDecimal(1, amount);
            ps.setString(2, accountId);
            ps.setBigDecimal(3, amount);
            int rows = ps.executeUpdate();
            if (rows == 0) {
                LOG.warning("Insufficient balance for " + accountId);
                return false;
            }
            return true;
        }
        // The CHECK (balance >= 0) constraint on the table
        // is the final safety net even if the WHERE is bypassed.
    }
}

How do you classify data and pick technology?

After classifying each data item by its invariant, the technology choice becomes obvious.

DIAGRAM
Classification Table:

  Data Item          Invariant           Consistency   Technology
  ─────────────────  ──────────────────  ────────────  ─────────────────
  Account balance    Must never go < 0   Strong (CP)   PostgreSQL primary
  Loan disbursement  Must not disburse   Strong (CP)   PostgreSQL + lock
                     twice
  Activity feed      Can lag seconds     Eventual (AP) Cassandra / Redis
  Profile picture    Can be stale        Eventual (AP) S3 + CDN
  Like counter       Can be approximate  Eventual (AP) Redis counter
  Notifications      Can delay           Eventual (AP) Kafka + consumer
SQL
-- Strong consistency: balance ledger on PostgreSQL
CREATE TABLE accounts (
    account_id   VARCHAR(20) PRIMARY KEY,
    holder_name  VARCHAR(100) NOT NULL,
    balance      NUMERIC(15,2) NOT NULL DEFAULT 0.00,
    CONSTRAINT chk_balance_non_negative CHECK (balance >= 0)
);

-- The CHECK constraint is the database-level invariant enforcement.
-- No amount of application logic can bypass it.

How do you split consistency within one transaction?

Even within one business transaction, you split the work. The money-moving core runs strongly consistent and synchronous. The downstream effects fire as asynchronous events that are allowed to be eventually consistent.

JAVA
package com.bank.service;

import java.math.BigDecimal;
import java.sql.*;
import java.util.logging.Logger;

public class TransferWithSplitConsistency {

    private static final Logger LOG = Logger.getLogger(
        TransferWithSplitConsistency.class.getName());

    private final Connection primaryConn;  // CP: strong consistency

    public TransferWithSplitConsistency(Connection primaryConn) {
        this.primaryConn = primaryConn;
    }

    public void transfer(String sourceId, String targetId, BigDecimal amount)
            throws SQLException {
        // STEP 1: Strongly consistent (CP)
        // Debit and credit in one ACID transaction on the primary
        primaryConn.setAutoCommit(false);
        try {
            debit(sourceId, amount);
            credit(targetId, amount);
            insertOutboxEvent(sourceId, targetId, amount);
            primaryConn.commit();
            LOG.info("Transfer committed: " + amount);
        } catch (SQLException e) {
            primaryConn.rollback();
            throw e;
        }

        // STEP 2: Eventually consistent (AP)
        // Outbox event is picked up by a Kafka publisher
        // Feed, notification, and analytics services consume it
        // A few seconds of lag is acceptable
    }

    private void debit(String accountId, BigDecimal amount) throws SQLException {
        String sql = "UPDATE accounts SET balance = balance - ? "
                   + "WHERE account_id = ? AND balance >= ?";
        try (PreparedStatement ps = primaryConn.prepareStatement(sql)) {
            ps.setBigDecimal(1, amount);
            ps.setString(2, accountId);
            ps.setBigDecimal(3, amount);
            if (ps.executeUpdate() == 0) {
                throw new SQLException("Insufficient balance: " + accountId);
            }
        }
    }

    private void credit(String accountId, BigDecimal amount) throws SQLException {
        String sql = "UPDATE accounts SET balance = balance + ? WHERE account_id = ?";
        try (PreparedStatement ps = primaryConn.prepareStatement(sql)) {
            ps.setBigDecimal(1, amount);
            ps.setString(2, accountId);
            ps.executeUpdate();
        }
    }

    private void insertOutboxEvent(String source, String target, BigDecimal amount)
            throws SQLException {
        String sql = "INSERT INTO outbox (event_type, payload) VALUES (?, ?)";
        try (PreparedStatement ps = primaryConn.prepareStatement(sql)) {
            ps.setString(1, "TRANSFER_COMPLETED");
            ps.setString(2, source + ":" + target + ":" + amount);
            ps.executeUpdate();
        }
    }
}

What is the anti-pattern to avoid?

The anti-pattern is picking a trendy database first and contorting business rules to fit its consistency model.

DIAGRAM
Anti-pattern:
  1. "Cassandra scales, let's use it for everything."
  2. Try to enforce "balance >= 0" on eventual consistency.
  3. Two concurrent withdrawals both succeed.
  4. Balance goes negative: real money lost.

Correct approach:
  1. Ask: "What must never happen?"
  2. Balance must never go negative ──► strong consistency.
  3. Choose PostgreSQL with CHECK constraint for balances.
  4. Choose Cassandra for the activity feed (no hard invariant).

What are common interview follow-up questions?

  • Give an example of a business invariant that requires strong consistency.
  • How does the Saga pattern handle eventual consistency across services?
  • What is the difference between linearizability and serializability?
  • How do you handle the case where a payment must be exactly-once but the messaging system is at-least-once?

What should a senior engineer know?

A senior engineer should be able to classify every data item in the system by its invariant, justify the consistency model per data boundary, and explain why the outbox pattern bridges the CP and AP worlds.

DIAGRAM
Senior Interview Answer Structure:
  1. State the invariant: "balance must never go negative"
  2. Choose consistency: strong (CP) for this data
  3. Choose technology: PostgreSQL primary with ACID + CHECK
  4. Contrast with cosmetic data: feed is AP, Cassandra
  5. Bridge pattern: outbox + Kafka for CP → AP events
  6. Anti-pattern: picking the DB before the invariant

Visual Overview

DIAGRAM
┌──────────────────────────────────────────────────────┐
│   Requirements-Driven Consistency Classification     │
├──────────────────────────────────────────────────────┤
│                                                      │
│  Step 1: Identify the invariant                      │
│  ┌─────────────────────────────────────────┐         │
│  │ "What must NEVER happen?"              │         │
│  └──────────────┬──────────────────────────┘         │
│                 │                                    │
│      ┌──────────┴──────────┐                         │
│      │                     │                         │
│      ▼                     ▼                         │
│  Hard Invariant        No Hard Invariant             │
│  "Balance < 0"         "Feed lags 3s"                │
│      │                     │                         │
│      ▼                     ▼                         │
│  Strong Consistency    Eventual Consistency           │
│  (CP)                  (AP)                           │
│      │                     │                         │
│      ▼                     ▼                         │
│  PostgreSQL            Cassandra / Redis             │
│  ACID + CHECK          Multi-node, always-up         │
│  Single primary        Last-write-wins               │
│                                                      │
│  Step 2: Bridge with Outbox Pattern                  │
│  ┌──────────────┐   Kafka   ┌──────────────┐         │
│  │ CP: Balance  │ ────────► │ AP: Feed,    │         │
│  │ (committed)  │  event    │ Notification │         │
│  └──────────────┘           └──────────────┘         │
└──────────────────────────────────────────────────────┘

Key Takeaway

Treating CAP as a single global switch is an oversimplification because consistency needs differ per operation and per piece of data. The disciplined method is to start from the business invariant: the thing that must never happen. Phrase each rule as a "must never" statement. "The available balance must never go negative." "A loan must not be disbursed twice." These are correctness constraints whose violation causes real financial harm, so they demand strong consistency with ACID transactions and database-level constraints.

In contrast, data like an activity feed, a profile picture, or a like count has no hard invariant and can lag a few seconds, so strong consistency buys nothing a user notices while costing latency and availability. There you choose eventual consistency and high availability.

The failure Pankaj always cites is a team that chose Cassandra for everything because it scales, then tried to enforce "balance must never go negative" on eventual consistency. During concurrent withdrawals an account went negative: real money lost. At the National Bank of India, the team fixed this by classifying each data item by its "must never happen" rule first. The balance ledger moved to a strongly-consistent PostgreSQL primary with a non-negative CHECK constraint. The feed and counters stayed on the eventually-consistent store. Even within one transaction, the money-moving core is strongly consistent and synchronous, while the downstream notification and analytics fire as asynchronous outbox events.

The anti-pattern is picking a trendy database first and contorting the business rules to fit. Requirements must drive the technology, not the other way around.

Remember one sentence: start from what must never happen; hard invariants get strong consistency, cosmetic data gets eventual consistency, and requirements choose the database rather than the reverse.

---

Share
Use ← → arrow keys to navigate