Definition
- Durable log – Kafka as the partitioned, replayable ingestion layer for all transaction events.
- Feature lookup – enriching each event with the customer's recent spending velocity, typical locations, and average amount from keyed state or a feature store.
- Storage split – OLTP for low-latency serving of decisions and balances, OLAP/lake for analytics and model training.
- Dead-letter queue – quarantine for poison records that fail repeatedly, so one bad message never blocks a partition.
- Safer-but-slower – the bank's deliberate trade-off: exactly-once, block on uncertainty, quarantine the questionable.
- Scoring time budget – a strict latency bound for the fraud scorer with a safe default decision if exceeded.
Picture the National Bank of India wanting to stop fraud the instant a card is swiped. Every swipe drops onto a fast conveyor (Kafka). A sharp inspector (a Flink job) grabs each swipe, instantly looks up that customer's recent habits (feature lookups), and decides: allow, challenge, or block. Approved swipes update the live balance system (OLTP) so the customer sees it immediately, while a copy flows to the research archive (OLAP/lake) to study patterns. Guards at every door check the paperwork (validation), set aside jammed forms (dead-letter queue), and watch monitors (monitoring). The bank's rule is simple: when in doubt, take the safer, slightly slower action.
Understanding the Concepts
How does ingestion work in a fraud platform?
Ingestion starts with Kafka as a durable, partitioned log, partitioned by account or card so one entity's events stay ordered and consumers scale out. The semantics choice matters: at-least-once is simpler but can reprocess an event, while exactly-once guarantees each event is scored once.
package com.bank.fraud;
import java.util.Properties;
public class FraudIngestion {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "kafka:9092");
props.put("acks", "all"); // durability
props.put("enable.idempotence", "true"); // exactly-once
props.put("transactional.id", "fraud-producer-1");
System.out.println("Kafka producer config:");
System.out.println(" acks=all: wait for all replicas");
System.out.println(" idempotent: no duplicates");
System.out.println(" transactional: exactly-once");
System.out.println();
System.out.println("Partitioned by card_id:");
System.out.println(" All swipes for one card in order.");
}
}
Kafka producer config:
acks=all: wait for all replicas
idempotent: no duplicates
transactional: exactly-once
Partitioned by card_id:
All swipes for one card in order.
How does streaming fraud detection work?
The streaming detection engine is a stateful Flink or Spark job. For each event it performs feature lookups: recent spending velocity, typical locations, average amount, and device history, from keyed state and a fast feature store like Redis. It then applies rules and a model to produce a score and decision.
package com.bank.fraud;
import java.math.BigDecimal;
public class FraudScorer {
record Swipe(String cardId, BigDecimal amount,
String city, long timestamp) {}
record Features(BigDecimal avgAmount, String usualCity,
int txnsLastMinute) {}
public static void main(String[] args) {
Swipe swipe = new Swipe("CARD-2001",
new BigDecimal("95000"), "Mumbai", System.currentTimeMillis());
Features features = new Features(
new BigDecimal("15000"), "Delhi", 2);
// Rule 1: amount >> average
boolean highAmount = swipe.amount().compareTo(
features.avgAmount().multiply(new BigDecimal("5"))) > 0;
// Rule 2: unusual city
boolean unusualCity = !swipe.city().equals(
features.usualCity());
// Rule 3: velocity spike
boolean velocitySpike = features.txnsLastMinute() > 3;
String decision;
if (highAmount && unusualCity) {
decision = "BLOCK";
} else if (highAmount || velocitySpike) {
decision = "CHALLENGE";
} else {
decision = "ALLOW";
}
System.out.println("Card: " + swipe.cardId());
System.out.println("Decision: " + decision);
}
}
Card: CARD-2001
Decision: BLOCK
How does the storage split work?
An OLTP store serves the live decision path and current balances with low latency. An OLAP store or data lake receives a copy of all events and decisions for analytics, investigation, and model training, isolated so heavy workloads never slow the real-time path.
package com.bank.fraud;
public class StorageSplit {
public static void main(String[] args) {
System.out.println("OLTP (PostgreSQL):");
System.out.println(" Live card balances");
System.out.println(" Fraud decisions (allow/block)");
System.out.println(" Latency: < 10 ms");
System.out.println();
System.out.println("OLAP (Redshift / Lake):");
System.out.println(" All events + decisions (copy)");
System.out.println(" Model training data");
System.out.println(" Investigation queries");
System.out.println(" Isolated from real-time path");
}
}
OLTP (PostgreSQL):
Live card balances
Fraud decisions (allow/block)
Latency: < 10 ms
OLAP (Redshift / Lake):
All events + decisions (copy)
Model training data
Investigation queries
Isolated from real-time path
How do you ensure reliability?
Schema validation via a registry stops malformed producers. Per-event data validation checks nulls, ranges, and invariants. Poison records go to a dead-letter queue so one bad message never blocks a partition. Monitoring tracks lag, latency, decision rates, and drift.
package com.bank.fraud;
import java.math.BigDecimal;
public class ReliabilityControls {
public static void main(String[] args) {
// Schema validation
System.out.println("Schema registry: reject incompatible changes");
// Data validation
BigDecimal amount = null;
if (amount == null || amount.compareTo(BigDecimal.ZERO) <= 0) {
System.out.println("Validation: null amount -> DLQ");
}
// Monitoring
System.out.println("Monitor:");
System.out.println(" Consumer lag: < 1000 events");
System.out.println(" Scoring latency: p99 < 50 ms");
System.out.println(" Block rate: alert if > 5%");
System.out.println(" DLQ rate: alert if > 0.1%");
}
}
Schema registry: reject incompatible changes
Validation: null amount -> DLQ
Monitor:
Consumer lag: < 1000 events
Scoring latency: p99 < 50 ms
Block rate: alert if > 5%
DLQ rate: alert if > 0.1%
What is the safer-but-slower trade-off?
A bank deliberately prefers safer-but-slightly-slower choices: exactly-once over at-least-once, block or challenge on uncertainty rather than approve, quarantine questionable records rather than process blindly, and spend extra milliseconds on feature lookups for a better decision.
package com.bank.fraud;
public class SaferButSlower {
public static void main(String[] args) {
System.out.println("Bank trade-offs (safer-but-slower):");
System.out.println(" 1. Exactly-once > at-least-once");
System.out.println(" (more overhead, no duplicates)");
System.out.println(" 2. Block on uncertainty > approve");
System.out.println(" (false positive < false negative)");
System.out.println(" 3. Quarantine > process blindly");
System.out.println(" (DLQ over silent corruption)");
System.out.println(" 4. Feature lookup > skip for speed");
System.out.println(" (better decision > faster decision)");
System.out.println();
System.out.println("Constraint: scoring time budget.");
System.out.println(" If budget exceeded, safe default (CHALLENGE).");
}
}
Bank trade-offs (safer-but-slower):
1. Exactly-once > at-least-once
(more overhead, no duplicates)
2. Block on uncertainty > approve
(false positive < false negative)
3. Quarantine > process blindly
(DLQ over silent corruption)
4. Feature lookup > skip for speed
(better decision > faster decision)
Constraint: scoring time budget.
If budget exceeded, safe default (CHALLENGE).
How does the scoring time budget work?
Fraud systems must bound their own latency so a legitimate payment is not delayed too long. You set a strict scoring time budget with a safe default decision if the budget is exceeded, balancing safety with customer experience.
package com.bank.fraud;
public class ScoringTimeBudget {
public static void main(String[] args) {
long budgetMs = 100; // 100 ms max
long start = System.currentTimeMillis();
// Simulate scoring work
String decision = "ALLOW"; // default
long elapsed = System.currentTimeMillis() - start;
if (elapsed > budgetMs) {
decision = "CHALLENGE"; // safe default
System.out.println("Budget exceeded (" + elapsed + " ms)."
+ " Defaulting to CHALLENGE.");
} else {
System.out.println("Scored in " + elapsed + " ms."
+ " Decision: " + decision);
}
}
}
Scored in 0 ms. Decision: ALLOW
What are the end-to-end platform components?
The full platform has five layers: ingestion (Kafka), scoring (Flink), serving (OLTP), analytics (OLAP/lake), and reliability (validation, DLQ, monitoring). Every decision is idempotent so a replay never double-applies a block.
package com.bank.fraud;
public class PlatformComponents {
public static void main(String[] args) {
System.out.println("Fraud Platform Layers:");
System.out.println(" 1. Ingestion: Kafka (exactly-once)");
System.out.println(" 2. Scoring: Flink (stateful, features)");
System.out.println(" 3. Serving: PostgreSQL (live decisions)");
System.out.println(" 4. Analytics: Redshift (investigation)");
System.out.println(" 5. Reliability: registry, DLQ, monitoring");
}
}
Fraud Platform Layers:
1. Ingestion: Kafka (exactly-once)
2. Scoring: Flink (stateful, features)
3. Serving: PostgreSQL (live decisions)
4. Analytics: Redshift (investigation)
5. Reliability: registry, DLQ, monitoring
What are common interview follow-up questions?
- How do you handle model drift in a fraud scorer?
- What is the difference between at-least-once and exactly-once in Kafka?
- How do you A/B test a new fraud model in production?
- How do you handle feedback loops when blocked transactions affect training data?
- How do you size the scoring time budget?
Visual Overview
Real-Time Fraud Detection Platform
──────────────────────────────────
┌──────────┐ ┌──────────────┐ ┌──────────────┐
│ Card │───▶│ Kafka │───▶│ Flink │
│ Swipe │ │ (durable │ │ (stateful │
│ │ │ log) │ │ scorer) │
└──────────┘ └──────────────┘ └──────┬───────┘
│
┌────────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ OLTP │ │ OLAP │ │ DLQ │
│(decisions│ │(analytics│ │ (poison │
│ balances)│ │ training)│ │ records) │
└──────────┘ └──────────┘ └──────────┘
Monitoring: lag | latency | block rate | drift | DLQ rate
Key Takeaway
A real-time fraud detection platform has four layers. Ingestion uses Kafka as a durable partitioned log with exactly-once semantics for correctness. Streaming detection is a stateful Flink or Spark job that enriches each event with feature lookups from keyed state and a feature store like Redis, then applies rules and a model to decide allow, challenge, or block. The storage split separates an OLTP store for live decisions and balances from an OLAP store or lake for analytics and model training, isolated so heavy workloads never slow the real-time path. Reliability wraps everything: schema validation, per-event data validation, dead-letter queues, and comprehensive monitoring.
The trade-off a bank deliberately makes is safer-but-slightly-slower. It chooses exactly-once over at-least-once, blocks or challenges on uncertainty rather than approving, quarantines questionable records rather than processing blindly, and spends extra milliseconds on feature lookups for a better decision. The one balancing constraint is a strict scoring time budget with a safe default if exceeded, so legitimate payments are not delayed too long.
In a banking context at the National Bank of India, a normal swipe from Rajesh was allowed and posted immediately, a swipe far above his usual spend was blocked, a velocity spike for Priya was challenged, and a malformed event went to the dead-letter queue. Every decision was idempotent so a replay never double-applied a block.
Correctness and caution first, within a tight latency budget, is precisely what a bank expects from a fraud platform.
Remember one sentence: ingest via Kafka, score with stateful streaming, split OLTP from OLAP, wrap in reliability, and prefer safer-but-slower because a missed fraud costs far more than a few extra milliseconds.
---