Back to Observability: Logging, Tracing, Metrics
Q6Question 6 of 9

Why can't you simply deploy the old binary after writing data in a new format, and what is expand-and-contract?

Key Concepts
  • Code reverts instantly on redeploy; data does NOT -- rows written in the new format stay.
  • The old binary encounters data in a shape it was never programmed to read: crash, wrong results, or silent corruption.
  • Expand phase: add the new format additively, dual-write both old and new, so either binary can run.
  • Coexistence phase: run new code but keep producing old format; verify both formats agree; backfill history.
  • Contract phase (separate, later deploy): stop dual-writing, drop the old format -- the one irreversible step.
  • During the entire expand-coexist window, rollback is always safe because the old format is maintained.
  • Expand-and-contract applies to DB schemas, API shapes, event schemas, file formats, and cache layouts.

Definition

The old binary cannot read data written in a new format, so you keep both formats alive through expand (add new, keep old), coexist, and only contract (remove old) once rollback is no longer needed.
  • The trap -- code reverts instantly; data does not. Rows written in the new format stay after rollback.
  • Expand -- add the new format additively; dual-write both old and new in one deploy.
  • Coexist -- run new code but keep producing old format; verify; backfill history in throttled batches.
  • Contract -- stop dual-writing, drop old format; the one irreversible step, done as a separate, later deploy.
  • Rollback safety -- during expand and coexist, either binary can run because both formats are maintained.
  • Applies to -- DB schemas, API shapes, event schemas, file formats, cache layouts.

National Bank of India switches its ledger to a new format overnight. If a problem appears and they bring back yesterday's clerk, he cannot read a single entry. Expand-and-contract: write every entry in both old and new format for a long period. Either clerk can work. Only after the bank is completely sure do they stop writing the old format.

Understanding the Concepts

Why can't you just deploy the old binary?

Code reverts instantly. Data does not. Every row the new version wrote stays in the new format.

JAVA
// New version writes amount as integer paise in a new column
row.setAmountMinor(amount.movePointRight(2).longValueExact());
row.setAmount(null); // stops writing old column

// Old binary expects the old decimal column:
BigDecimal amount = row.getBigDecimal("amount"); // NULL → wrong result!

What is expand-and-contract?

Three phases that ensure rollback is always safe.

Phase 1: Expand (additive + dual-write)

SQL
-- Additive schema change: ADD, do not DROP
ALTER TABLE transfers ADD COLUMN amount_minor BIGINT;
-- Old column stays; new column added
JAVA
// Dual-write: BOTH formats maintained
void applyAmount(TransferRow row, BigDecimal amount) {
    row.setAmount(amount);                                   // OLD format
    row.setAmountMinor(amount.movePointRight(2).longValueExact()); // NEW format
}
// Now either binary can run: old reads amount, new reads amount_minor

Phase 2: Coexist

  • Run new code but keep producing old format.
  • Verify both formats agree continuously.
  • Backfill historical rows in throttled batches.
  • If a bug is found, rollback is SAFE because the old format was never abandoned.
JAVA
// Backfill old rows into new format
for (TransferRow row : repo.findWhereAmountMinorIsNull()) {
    row.setAmountMinor(row.getAmount().movePointRight(2).longValueExact());
    repo.save(row);
}

Phase 3: Contract (separate, much later deploy)

JAVA
// ONLY after you will NOT roll back
void applyAmount(TransferRow row, BigDecimal amount) {
    row.setAmountMinor(amount.movePointRight(2).longValueExact()); // NEW only
    // row.setAmount(...) removed
}
// ALTER TABLE transfers DROP COLUMN amount;

This is the one irreversible step. Done as a separate deploy, long after the rollout.

How does this look as a timeline?

DIAGRAM
  Deploy 1 (Expand): ADD new column, dual-write
       │
       ▼ (days/weeks of coexistence)
  Verify: both formats agree ✓
  Backfill: all old rows have new format ✓
  Confidence: high ✓
       │
       ▼
  Deploy 2 (Contract): DROP old column, stop dual-write
       ← rollback no longer safe after this point

Does this apply beyond DB schemas?

| Contract Type | Expand | Contract |
|---|---|---|
| DB column | Add new column, dual-write | Drop old column |
| API field | Add new field, keep old | Remove old field |
| Event schema | Produce both versions | Stop producing old version |
| Cache layout | Write both formats | Remove old format |

What are common mistakes?

JAVA
// WRONG: combining expand and contract in one deploy
// New version drops old column AND writes new → rollback impossible

// CORRECT: expand and contract are SEPARATE deploys
// Expand: add new, keep old (reversible)
// ... time passes, confidence builds ...
// Contract: remove old (irreversible, deliberate)

What should a senior engineer know?

  • The cost is temporary duplication (two formats, dual writes, verification).
  • That cost is the price of being able to undo a bad deploy without data loss.
  • For a bank, this is non-negotiable.
  • The quiet failure (old code reads new data wrong and produces wrong financial results) is far more dangerous than a loud crash.

Visual Overview

DIAGRAM
  Expand-and-Contract Timeline

  ┌──────────────────────────────────────────────┐
  │ Deploy 1: EXPAND                              │
  │ - ADD amount_minor column                     │
  │ - Dual-write: amount + amount_minor           │
  │ - Either old or new binary can run ✓          │
  │ - Rollback: SAFE                              │
  ├──────────────────────────────────────────────┤
  │ Coexistence (days/weeks)                      │
  │ - Verify both columns agree                   │
  │ - Backfill old rows                           │
  │ - Build confidence                            │
  │ - Rollback: STILL SAFE                        │
  ├──────────────────────────────────────────────┤
  │ Deploy 2: CONTRACT (separate, later)          │
  │ - Stop dual-writing                           │
  │ - DROP amount column                          │
  │ - Rollback: NO LONGER SAFE ← irreversible     │
  └──────────────────────────────────────────────┘

Key Takeaway

When a new version writes data in a new format and you need to roll back, deploying the old binary is not enough because code reverts instantly but data does not. Every row written in the new format stays, and the old binary encounters data it was never programmed to read: crash, wrong results, or silent corruption.

Expand-and-contract makes rollback safe by never having a moment where only one format exists during the risky window. In the expand phase, add the new format additively and dual-write both old and new. During coexistence, verify both formats agree and backfill history. Only in the contract phase, as a separate later deploy, stop dual-writing and drop the old format. Throughout expand and coexist, either binary can run, so rollback is always safe.

At National Bank of India, Pankaj used this to change amount storage from decimal rupees to integer paise. The team dual-wrote both fields, verified agreement, backfilled history, and only dropped the old column weeks later. The same pattern applies to API shapes, event schemas, and cache layouts.

Remember one sentence: the old binary cannot read data written in a new format, so keep both formats alive through expand (add new, keep old) and coexist, and only contract (remove old) once rollback is no longer needed.

---

Share
Use ← → arrow keys to navigate