Real-Time Fraud Detection

Fraud decisioning is a latency problem wearing a machine-learning costume. The model matters, but the engineering question is how to get a correct, enriched feature vector to that model within a few tens of milliseconds while the transaction is still waiting — and to do it under bursty load without dropping or double-counting events.

A decision made after the transaction settles is an incident report, not a control. The system has to consume a high-rate transaction stream, compute behavioural features over recent history, score, and return a decision inside the payment path — while remaining correct when a consumer restarts mid-window.

• Decision on the hot path, within the payment authorisation budget.
• Ordering guarantees per account so behavioural windows are correct.
• Graceful degradation: a slow model must not stall ingestion.
• Every event and every decision durably landed for audit and retraining.
• Horizontal scale by partition, with no single-node bottleneck.

A transaction is produced to Kafka keyed by account. A stream processor consumes the partition, updates windowed state for that account, and enriches the event with online profile features. The enriched vector is scored; the score and policy rules produce a decision that is emitted to a decisions topic within the authorisation window. In parallel, the raw event, the exact feature vector used, and the decision are appended to the lakehouse — so the offline training set is the online reality rather than a reconstruction of it.

Consumer state is checkpointed, so a restart resumes from the last committed offset and rebuilds window state rather than silently losing it. Model timeouts degrade to rules; a downstream outage backs pressure into Kafka retention instead of dropping events.

Throughput scales with partition count and consumer instances. The design target is 10M events/sec — this is a design target used to drive partitioning, state sizing and back-pressure decisions, not a measured production benchmark.

Correctness is checked at three points: schema contracts on ingestion, invariants on windowed state, and reconciliation between the decisions topic and the lakehouse to confirm no decision is lost between the online and offline views.

The result is an architecture where latency, ordering, correctness and failure behaviour are each an explicit decision with a stated trade-off. All figures are design targets from the system design; this is an engineering study, not a deployed production service.