CAP Theorem in Practice

• CAP theorem : a distributed system cannot simultaneously guarantee consistency, availability, and partition tolerance; during a partition it must sacrifice one.
• Consistency (C) : every read sees the most recent write.
• Availability (A) : every request gets a non-error response.
• Partition tolerance (P) : the system keeps working despite dropped messages between nodes.
• CP : rejects requests during a partition to preserve correctness.
• AP : serves possibly stale data during a partition to preserve availability.
• Tunable consistency : choosing the CAP trade-off per operation rather than globally.

What is the CAP theorem?

The three properties are consistency (every read sees the most recent write), availability (every request gets a non-error response), and partition tolerance (the system keeps working despite dropped messages). The theorem says you cannot have all three during a network partition.

Why is partition tolerance not optional?

Partition tolerance is not optional in a distributed system because networks genuinely fail : switches go down, cables are cut, cloud availability zones lose connectivity. Since partitions happen, the real choice during a partition is only between consistency and availability.

What is a CP system?

A CP system refuses to serve rather than serve possibly wrong data during a partition. This preserves correctness at the cost of some unavailability.

package com.bank.cap;

public class CpSystem {
    public String readBalance(String accountId) {
        if (partitioned && !isLeaderReachable()) {
            throw new ServiceUnavailableException(
                "Rejecting read to preserve consistency");
        }
        return leader.query(accountId); // only serve when fresh
    }
}

What is an AP system?

An AP system keeps answering even when partitioned, returning whatever data the reachable node has, which may be stale, and reconciling once the partition heals through eventual consistency.

package com.bank.cap;

public class ApSystem {
    public long getLikeCount(String postId) {
        return nearestReplica.getOrDefault(postId, 0L); // never errors, may be stale
    }
}

How does the domain decide the choice?

A social media feed, product catalogue, or trending leaderboard can tolerate briefly stale data, so they choose AP. A bank ledger, inventory count that must not oversell, or uniqueness constraint cannot tolerate wrong data, so they choose CP.

| Data type | CAP choice | Reason |
|---|---|---|
| Bank balance | CP | Wrong balance = double-spend |
| Transaction ledger | CP | Must be append-only, correct |
| Social feed | AP | Stale like count is harmless |
| Product catalogue | AP | Brief staleness is acceptable |

Can you choose per-operation instead of per-system?

Yes. The same bank can serve a cached, possibly slightly stale balance display with AP semantics for responsiveness, while insisting on CP for the actual debit and credit that move money. This is tunable consistency.

package com.bank.cap;

import java.math.BigDecimal;

public class TunableConsistency {
    // AP: display balance (slightly stale is OK for display)
    public BigDecimal displayBalance(String accountId) {
        return cache.getOrDefault(accountId, BigDecimal.ZERO);
    }

    // CP: actual transfer (must be strongly consistent)
    public void transfer(String from, String to, BigDecimal amount) {
        if (!leaderReachable()) throw new ServiceUnavailableException("CP");
        leader.executeTransfer(from, to, amount);
    }
}

What is eventual consistency?

Eventual consistency guarantees that in the absence of new writes, all replicas will converge to the same value. The staleness window is usually milliseconds to seconds, which is harmless for social data but unacceptable for money.

What are common misconceptions about CAP?

The biggest misconception is that CAP is a permanent choice : "we are a CP system." In reality, CAP applies only during partitions, and when the network is healthy you can have both consistency and availability. Another misconception is that you must choose one globally; mature systems choose per-operation.

What should a senior engineer know?

A senior engineer knows that CAP is about network partitions, not about normal operation. They also know about PACELC, which extends CAP: during a Partition choose Availability or Consistency, Else (when no partition) choose Latency or Consistency. This captures the latency-consistency trade-off that exists even when the network is healthy.