Definition
- Cost comparison -- durability costs latency and throughput; accept loss when the cost of guaranteeing exceeds the cost of losing.
- Reconstructable -- derived analytics can be recomputed from the source of truth.
- Regenerable -- cache entries can be rebuilt from the primary store on the next miss.
- Low-stakes -- telemetry is high-volume and individually unimportant.
- Critical -- financial transactions and ownership records must NEVER be lost.
- RPO per class -- Recovery Point Objective: transfers = 0, telemetry = 60s, cache = total.
- Separation -- route each data class to a path matching its RPO; mixing overloads the critical path.
The ledger of who owns how much at National Bank of India must survive fire, flood, and power failure. But the hourly tally of customers is just a convenience; if it blows away, the bank shrugs. Treating the tally with the same fireproof-vault ceremony as the ledger wastes effort and can slow down the one thing that truly matters.
Understanding the Concepts
When do you accept data loss?
When the cost of guaranteeing durability exceeds the cost of losing the data.
| Data Class | Cost of Loss | RPO | Durability Strategy |
|---|---|---|---|
| Financial transactions | Catastrophic | 0 (zero) | Sync replication, acknowledged commits |
| Ownership records | Catastrophic | 0 | Same as above |
| Telemetry/metrics | Near zero | 60 seconds | Buffered, async flush |
| Derived analytics | Recomputation cost | Minutes | Recompute from source of truth |
| Cache entries | Cache-miss cost | Total loss OK | Regenerable on miss |
Why is mixing data classes on one path dangerous?
// WRONG: telemetry and transfers share the same strong-durability pipeline
// Telemetry volume is 100x transfers
// Telemetry flood saturates the replication link → transfers delayed!
// CORRECT: separate paths matching each class's RPO
transferRepo.save(transfer); // sync replication, acknowledged
telemetryBuffer.append(metric); // async flush, 60s buffer
cache.put(key, value); // in-memory, total loss OK
How do you define the loss window?
For each data class, state a Recovery Point Objective: the maximum data (measured in time) you are willing to lose.
package com.bank.config;
public class DurabilityPolicy {
// Financial: zero-loss → sync replication, acknowledged commit
public static final Duration TRANSFER_RPO = Duration.ZERO;
// Telemetry: 60-second window → async buffer, flush every 60s
public static final Duration TELEMETRY_RPO = Duration.ofSeconds(60);
// Cache: total loss acceptable → in-memory only
public static final Duration CACHE_RPO = Duration.ofDays(999); // effectively infinite
}
Why must the mechanism match the stated window?
// Claiming 60-second telemetry RPO but actually buffering 5 minutes:
// A crash loses 5 minutes, not 60 seconds → the stated RPO is a lie
// Size the buffer to the RPO: bufferSize = throughput × RPO
How do you verify the RPO?
@Test
void transferLossWindowIsZero() {
// Simulate crash immediately after acknowledged commit
BigDecimal before = totalMoney();
transferService.execute("SAV2001", "SAV3002", new BigDecimal("50000"));
simulateCrash();
restartAndRecover();
assertEquals(before, totalMoney()); // no money lost
}
@Test
void telemetryLossWithin60Seconds() {
telemetryService.record(metric);
simulateCrashBeforeFlush();
restartAndRecover();
// At most 60 seconds of telemetry lost → acceptable
}
What are common mistakes?
// WRONG: treating all data equally
// Forces cheap telemetry through expensive strong-durability path
// Saturates the critical pipeline → endangers financial transactions!
// CORRECT: classify, define RPO, route to matching path
What should a senior engineer know?
- The loss window must be stated explicitly, not left vague.
- Each RPO drives concrete engineering choices: replication mode, flush frequency, buffer size.
- In a regulated bank, these trade-offs must be auditable.
- Verify by crashing in the exact window between acknowledgement and durable commit.
What are the edge cases?
The trickiest edge case is data that looks low-value until it is needed for a regulatory audit. Telemetry that includes transaction correlation IDs might be needed months later for fraud investigation. Define the retention requirement with compliance before classifying data as losable.
package com.bank.data;
public class DataClassificationEdgeCases {
public static void main(String[] args) {
System.out.println("Edge case 1: telemetry with correlation IDs");
System.out.println(" Looks low-value, but needed for fraud investigation");
System.out.println(" Fix: keep correlation-bearing telemetry durable, discard raw metrics");
System.out.println();
System.out.println("Edge case 2: cache that holds the only copy");
System.out.println(" If the source DB is down, cache is the ONLY data");
System.out.println(" Fix: never treat cache as 'total loss OK' if it is the last copy");
System.out.println();
System.out.println("Edge case 3: derived data whose source is expensive to recompute");
System.out.println(" Recomputing 30 days of analytics takes 6 hours");
System.out.println(" Fix: RPO should reflect recomputation cost, not just data value");
}
}
Expected output:
Edge case 1: telemetry with correlation IDs
Looks low-value, but needed for fraud investigation
Fix: keep correlation-bearing telemetry durable, discard raw metrics
Edge case 2: cache that holds the only copy
If the source DB is down, cache is the ONLY data
Fix: never treat cache as 'total loss OK' if it is the last copy
Edge case 3: derived data whose source is expensive to recompute
Recomputing 30 days of analytics takes 6 hours
Fix: RPO should reflect recomputation cost, not just data value
Visual Overview
Data Classification and Durability Paths ┌────────────────┐ sync replication ┌──────────────┐ │ Transfers │ ────────────────► │ Durable DB │ RPO = 0 │ (critical) │ acknowledged │ (Postgres) │ └────────────────┘ └──────────────┘ ┌────────────────┐ async buffer ┌──────────────┐ │ Telemetry │ ────────────────► │ Time-series │ RPO = 60s │ (low-value) │ flush every 60s │ DB │ └────────────────┘ └──────────────┘ ┌────────────────┐ in-memory only ┌──────────────┐ │ Cache │ ────────────────► │ Redis │ RPO = total │ (regenerable) │ no persistence │ │ └────────────────┘ └──────────────┘ WRONG: all three on the same strong-durability path → telemetry flood saturates replication → transfers delayed!
Key Takeaway
Deliberately accepting data loss is the right engineering choice when the data is reconstructable or low-value enough that guaranteeing durability costs more than the loss. Financial transactions must never be lost (RPO = 0), but telemetry can tolerate a 60-second window, derived analytics can be recomputed, and cache entries can be regenerated on miss.
The danger of treating all data equally is not just waste but active harm: if a 100x volume of telemetry shares the critical strong-durability pipeline, it saturates the replication link and delays the transfers that actually need that path. Separating data classes onto paths matching their RPO protects the critical data.
At National Bank of India, transfers and telemetry shared one pipeline, and the telemetry flood delayed actual transfers. Pankaj classified the data, defined RPOs (transfers zero, telemetry 60s, cache total), and routed each class to a matching path. A crash test verified that transfers lost nothing and telemetry lost no more than its stated window.
Remember one sentence: accept data loss where the data is reconstructable or low-value, define the acceptable loss window explicitly per data class, route each class to a path matching its RPO, and verify by crash-testing, because each byte should be protected exactly as much as its value deserves.
---