{"title":"Design a Fraud Detection System","description":"","content":"Fraud detection decides, in real time, whether a payment or account action is legitimate or an attempt to steal money, identity, or account access. The problem is adversarial in a way most ML systems aren't, because the people generating the negative class are actively trying to evade the model. It's also extremely imbalanced (typically 0.05% to 0.3% of transactions are fraudulent), latency-bounded (the decision blocks a user checkout), and tied to real dollar losses on both sides of an error: approve a fraudster and the platform eats a chargeback, block a legitimate customer and the business loses the sale plus a unit of trust. A good fraud system combines unsupervised anomaly detection, supervised classification, graph features over an identity network, a rule engine for hard blocks, and a human review queue that doubles as a label-generation pipeline.\n\n---\n\n# 1. Problem Formulation\n\n### Clarifying Questions\n\nFraud looks different in every product. Payment fraud on a marketplace, account takeover on a bank, and synthetic signup fraud on a social network share machinery but differ in the features that matter, the action space, and who pays when the model is wrong. Pinning this down first saves a lot of wasted architecture.\n\n**Candidate:** \"What fraud types are in scope? Stolen-card payments, account takeover, synthetic identity, promo abuse, or all of them?\"\n\n**Interviewer:** \"The primary surface is card-not-present payment fraud on a marketplace that processes about 500 million transactions a day across 200 countries. Account takeover and signup fraud are secondary but related, because a compromised account or a fake account often drives the payment fraud that actually costs us money. Treat payment fraud as the main target and call out where account signals feed into it.\"\n\n**Candidate:** \"What's the action space? Are we just approving or declining, or is there something in the middle?\"\n\n**Interviewer:** \"Four actions: approve, challenge with a step-up like 3-D Secure or an OTP, send to a human review queue for manual decisioning, or hard-decline. The interesting design question is how to allocate risk across those four buckets, not just where to set a single threshold.\"\n\n**Candidate:** \"What's the latency budget? Does the scoring happen in-line with checkout, or asynchronously?\"\n\n**Interviewer:** \"In-line. The authorization call is blocking a checkout page somewhere, and our fraud service sits inside the payment gateway's decision path. You have a p99 budget of 100 milliseconds from when we receive the transaction to when we return a decision. That includes feature fetch, scoring, rules, and the action routing.\"\n\n**Candidate:** \"Where do labels come from, and how delayed are they?\"\n\n**Interviewer:** \"Three sources. Chargebacks are the ground truth but arrive 30 to 120 days after the transaction when the cardholder disputes the charge with their bank. Customer-reported fraud arrives within hours to days through a support ticket. Reviewer decisions from the manual queue arrive within a few hours and cover about 2% of transactions that we can't confidently approve or decline. The chargebacks are high quality but slow; the reviewer decisions are fast but cover a biased slice of borderline cases.\"\n\n**Candidate:** \"What are the cost asymmetries? Is a missed fraud worse than a false decline, or the other way around?\"\n\n**Interviewer:** \"Depends on segment. Average chargeback loss is around 120 dollars plus a fixed 15-dollar dispute fee and reputational damage with the card networks if our chargeback rate climbs past 0.9%. A false decline on an average transaction costs us the gross margin on that transaction plus the lifetime-value hit of annoying the customer, which internal modeling says is about 30 to 50 dollars per incident. So missed fraud is roughly three times more expensive per event, but fraud is 500 times rarer, and the auction-style rule is to optimize expected dollar loss, not raw event counts.\"\n\n**Candidate:** \"How adversarial is the environment? Do fraud patterns shift daily, weekly, or over months?\"\n\n**Interviewer:** \"Sophisticated fraud rings rotate tactics within days of a new detection method going live. Most fraud at volume uses techniques that are stable for weeks or months, but novel patterns appear every week or two, and you should plan for a sub-day response loop when a new pattern gets noticed.\"\n\n### Requirements\n\n**Functional Requirements:**\n\n- Return a risk score and a recommended action (approve, challenge, review, decline) for every transaction inside the 100ms budget.\n- Apply hard rules ahead of the model for known-bad signals (blacklisted cards, sanctioned accounts, known-compromised devices) so obvious fraud never reaches the scorer.\n- Feed low-confidence transactions to a human review queue ranked by expected dollar loss, and consume reviewer decisions back into the training pipeline as fast-arriving labels.\n- Produce explanations (top contributing features with direction and magnitude) for every transaction sent to review, so reviewers can validate or override the model.\n- Support an appeals path: a customer whose transaction was blocked can request a human re-review, and confirmed-good appeals are fed back as corrected labels.\n- Retrain on a daily cadence with a shadow evaluation pipeline, and support emergency re-deploys within hours when a novel fraud pattern is identified.\n\n**Non-Functional Requirements:**\n\n- P99 end-to-end latency under 100ms for the full decision path, including feature fetch, model inference, rules, and action routing.\n- Availability of 99.99% for the scoring service; a single failed decision is a dropped payment or a missed block.\n- Full audit trail for every decision, including feature values at scoring time, model version, rule evaluations, and the final action, retained for at least 13 months to support chargeback dispute investigations.\n- Graceful degradation: if the model is unavailable, the rules engine plus a cached risk table keeps the service decisioning at reduced accuracy rather than blocking all traffic.\n- Fairness guardrails: the model cannot use protected attributes directly, and subgroup decline rates (by country, card type, new versus returning customer) are monitored for drift.\n\n### ML Problem Framing\n\nThe core formulation is a binary classifier that predicts P(fraud | transaction, account, device, context). The output is a calibrated probability, because the downstream action router picks approve, challenge, review, or decline based on where the score falls on an expected-loss curve, and miscalibrated scores put transactions in the wrong bucket. The training label is 1 if the transaction became a confirmed chargeback or was confirmed as fraud by a reviewer, and 0 otherwise.\n\nTwo things make the framing more interesting than a generic classifier. First, the positive class is a moving target: fraudsters adapt, so the feature distribution of fraud shifts continuously while the feature distribution of legitimate traffic is relatively stable. A pure supervised model trained on historical fraud will miss tomorrow's fraud if tomorrow's tactics look different. This is where unsupervised anomaly detection earns its place in the stack: an isolation forest or autoencoder scoring deviation from the legitimate-traffic distribution surfaces suspicious transactions even when their specific pattern has never been labeled as fraud. The unsupervised score becomes a feature in the supervised model, not a replacement.\n\nSecond, fraud is often coordinated across accounts, cards, and devices that individually look innocuous. A single transaction from a single new account might score clean, but that account sharing a device fingerprint with fifty other new accounts should not. Graph features capturing identity-network structure surface collusive behavior that per-transaction features miss. These are computed asynchronously on a bipartite account-device-card graph and served to the model as precomputed node embeddings.\n\n### Metrics\n\n\n| Type | Metric | Why It Matters |\n|---|---|---|\n| Offline | PR-AUC | Area under the precision-recall curve; more informative than ROC on a 0.1% positive rate |\n| Offline | Recall at 1% false positive rate | Operating-point metric matching the target decline rate |\n| Offline | Expected dollar loss at threshold | Sum of missed-fraud dollars plus false-decline dollars, evaluated along the threshold curve |\n| Offline | Calibration ECE | Expected Calibration Error across deciles; wrong calibration sends transactions to the wrong action bucket |\n| Offline | Slice PR-AUC | Same metric broken out by country, card type, ticket size band, tenure bucket |\n| Online | Chargeback rate | Direct business outcome; target under 0.5% of approved GMV |\n| Online | False-decline rate | Fraction of legitimate customers blocked; target under 1.5% with tighter slices for high-tenure users |\n| Online | Dollar-weighted recall | Fraction of fraud dollars caught, weighted by transaction amount rather than count |\n| Online | Review queue SLA | P95 time from enqueue to reviewer decision; target under two hours |\n| Guardrail | Approval rate drift | Day-over-day shift in the approval rate; catches silent regressions |\n| Guardrail | Subgroup decline-rate ratio | Max ratio of decline rate across country or card-type subgroups; alarms on fairness drift |\n| Guardrail | Complaint rate | Support tickets per million approved transactions; catches false-decline waves before they show in dashboards |\n\n\nThe metric worth singling out is dollar-weighted recall. Fraud is asymmetric: a handful of large-ticket frauds can dwarf the dollar loss from hundreds of small ones. Counting events treats a 5-dollar gift card purchase and a 5,000-dollar electronics order as equal, which is wrong for anything downstream. Training and evaluation both use dollar-weighted loss so the model's optimization target matches what the business actually cares about.\n\n---\n\n# 2. System Architecture\n\nA single transaction flows through a decision path that starts when the payment gateway forwards an authorization request and ends with an approve, challenge, review, or decline verdict returned to the gateway. Five services sit inside the 100ms envelope, with async feedback loops on the side that keep features and models fresh without blocking decisions.\n\n\n```mermaid\nflowchart LR\n TXN[Transaction
Request]:::primary --> FS[Feature
Fetch]:::teal\n FS --> RULE[Rule
Engine]:::red\n RULE --> SCORE[Model
Scorer]:::primary\n SCORE --> CAL[Calibration
+ Action
Router]:::orange\n CAL --> DEC[Approve /
Challenge /
Review / Decline]:::green\n CAL -.log.-> LOG[Decision
+ Feature
Log]:::teal\n DEC -.review.-> Q[Review
Queue]:::orange\n Q -.labels.-> LOG\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nFeature fetch pulls three slices in parallel: account state (tenure, historical spend, recent activity velocity), device and network signals (device fingerprint, IP reputation, geolocation, prior sessions from this device), and transaction-level attributes (amount, merchant, card BIN, declared country). The rule engine runs next and applies hard blocks for known-bad entities: sanctioned merchants, blacklisted cards, accounts flagged by prior investigations. Any rule match short-circuits the decision before the model runs, both because it's faster and because hard rules are explainable in ways models aren't.\n\nThe model scorer receives features for any transaction that survives rules and produces a raw fraud probability. Calibration and action routing sit right after scoring: the raw score is mapped through a calibrator to correct training-sample bias, and the calibrated probability plus the transaction amount is used to pick an action on the expected-loss curve. The final decision, along with the full feature snapshot and the rule evaluation trace, is logged for audit and for training.\n\nThe review queue is a first-class component, not an afterthought. When the action router picks \"review,\" the transaction goes into a priority queue sorted by expected dollar loss (calibrated probability times transaction amount) and fed to human analysts. Their decisions flow back into the training pipeline within hours, which is much faster than waiting for chargebacks. The queue also produces explanation logs that get periodically sampled for model-behavior audits.\n\nThree asynchronous loops keep the stack adapting without blocking any single decision.\n\n\n```mermaid\nflowchart TD\n subgraph Daily[Daily Loop]\n LOG[Decision +
Feature Logs]:::primary --> LABEL[Label Join
CB + Reviewer +
Complaints]:::teal\n LABEL --> SAMP[Case-Control
Sampling]:::orange\n SAMP --> TRAIN[GBDT + DNN
Training]:::primary\n TRAIN --> CKPT[Model
Checkpoint]:::green\n end\n\n subgraph Stream[Streaming Loop]\n EVT[Transaction
Events]:::primary --> AGG[Velocity
Aggregator]:::teal\n AGG --> STORE[Online
Feature Store]:::orange\n DEV[Device +
Session
Events]:::primary --> STORE\n end\n\n subgraph Graph[Graph Loop - Hourly]\n IDEDGES[Account-Device-
Card Edges]:::primary --> GBUILD[Identity
Graph Builder]:::teal\n GBUILD --> GNN[GraphSAGE
Embedding Job]:::orange\n GNN --> GEMB[Graph
Embeddings]:::green\n GEMB --> STORE\n end\n\n CKPT -.served by.-> SRV[Scorer
Fleet]:::primary\n STORE -.read by.-> SRV\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe daily loop retrains the full model on the last 90 days of logged decisions joined against chargebacks, reviewer decisions, and customer complaints. The streaming loop keeps per-account and per-device velocity features current on a seconds-to-minutes timeline, because the 10 minutes between transactions 1 and 10 from a newly compromised card are exactly when the model has the most to gain from fresh features. The graph loop runs hourly, rebuilding the identity graph from fresh edge data and recomputing node embeddings; the embeddings land in the online store keyed by account ID, card fingerprint, and device hash.\n\nThe scale numbers that constrain the design look like this.\n\n\n| Dimension | Number | Implication |\n|---|---|---|\n| Transaction volume | 500M/day, peak 15K/s | Scorer fleet sized for 20K/s headroom |\n| Base fraud rate | 0.1 to 0.3% of transactions | Extreme imbalance; sampling and cost-weighted training essential |\n| Latency budget | 100ms p99 end-to-end | Feature fetch dominates; model runs on CPU with sub-30ms inference |\n| Feature store read fan-out | 40-60 features per request | Batched Redis reads; co-located replicas per region |\n| Identity graph size | ~500M accounts, ~200M devices | GraphSAGE with neighborhood sampling, not full-graph |\n| Chargeback label delay | 30-120 days median | Requires reviewer and complaint labels as faster signals |\n| Review queue throughput | ~2% of transactions, ~10M/day | Priority-sorted by expected loss, reviewer capacity ~5K/day/analyst |\n\n\n---\n\n# 3. Data\n\n### Sources\n\nThe signal for fraud comes from the transaction itself, the account history around it, the device and network it originates from, the merchant on the receiving side, and the identity graph connecting entities across the platform. No single source is sufficient: a fraudster can spoof any one of them, but faking all simultaneously is much harder.\n\n\n| Source | Content | Update Cadence |\n|---|---|---|\n| Transaction stream | Amount, currency, merchant, card BIN, billing address, declared country, entry mode | Real-time |\n| Account profile | Tenure, historical spend, payout history, linked cards, prior disputes | Batch daily + streaming writes |\n| Device and session | Device fingerprint, user-agent stability, session duration, behavior biometrics | Real-time |\n| Network signals | IP address, ASN, geo, VPN/proxy detection, TLS fingerprint | Real-time |\n| Chargeback feed | Dispute notifications from card networks with reason codes | Delayed 30-120 days |\n| Reviewer decisions | Analyst verdicts with feature attributions and notes | Hourly batch |\n| Customer reports | Fraud tickets, unauthorized-charge complaints | Within hours |\n| Identity graph | Shared-device, shared-card, shared-IP, shared-email edges | Hourly rebuild |\n| 3-D Secure feedback | Step-up challenge outcomes (pass/fail/abandon) | Near real-time |\n\n\nThe chargeback feed is the gold label, but its delay means the training pipeline always sees a 30-to-120-day-old view of fraud rates. Reviewer decisions fill the gap: they arrive within hours and cover roughly 2% of transactions, so they can't replace chargebacks but they dramatically shorten the feedback loop for the subset of transactions the model is most uncertain about. Customer complaints arrive even faster (often within a day) and flag fraud the reviewer queue missed or that slipped through at approve. In practice, training uses all three with different reliability weights: chargebacks at weight 1.0, reviewer confirmations at 0.9, customer reports at 0.7 after duplicate filtering.\n\n### Feature Engineering\n\nFeatures fall into six categories. The split matters because each category has a different freshness requirement and a different level of adversarial robustness.\n\n**Transaction attributes** are the raw fields of the authorization request: amount, currency, merchant ID, merchant category code, card BIN, entry mode, declared billing country, ship-to country. Freshness is trivially real-time because they arrive with the request. These features alone are weak signals (a 500-dollar electronics purchase isn't fraud on its own), but they're the foundation that every other feature conditions on.\n\n**Velocity features** count and sum activity over rolling windows: transactions in the last 1 minute, 1 hour, 24 hours, 7 days, 30 days, grouped by account, card, device, IP, and billing address. The classic fraud signature looks like \"five transactions in ten minutes, three different merchants, three different ship-to addresses,\" which only shows up when velocity is explicit. The streaming pipeline maintains these counters in a feature store; the p99 read has to stay under 20ms or the 100ms budget breaks.\n\n**Behavioral deviation features** measure how this transaction differs from the account's normal pattern. Z-scores on transaction amount relative to the account's historical distribution, flags for first-time-ever-seen merchant categories, deviations in time-of-day from the account's typical pattern. Behavioral deviation is the closest feature in a pure supervised model to what the unsupervised anomaly detector captures, which is why it carries a lot of weight.\n\n**Device and network features** cover the technical fingerprint of where the transaction came from. Device fingerprint stability (does this account always use the same fingerprint?), IP reputation scores from third-party feeds, VPN and proxy detection, TLS fingerprint consistency. These are especially valuable against account takeover, where a legitimate account is suddenly hit from a device it's never seen before.\n\n**Graph features** encode the structure of the identity network around this transaction. Degree counts (how many accounts share this device, how many cards share this billing address), cluster membership from a community-detection pass over the graph, and GNN node embeddings that summarize the full multi-hop neighborhood into a 64-dimensional vector. Graph features are where coordinated fraud rings finally become visible.\n\n**Merchant and counterparty features** describe the receiving side of the transaction. Merchant risk priors (historical chargeback rate at this merchant), recent authorization success rate, geographic velocity from this merchant, category-level fraud lift. A merchant that's been laundering stolen cards will show elevated chargebacks across many cards that individually look fine.\n\n\n| Feature | Type | Source | Freshness | Description |\n|---|---|---|---|---|\n| txn_amount | Float | Transaction stream | Real-time | Authorization amount in account currency |\n| txn_amount_zscore_30d | Float | Feature store | Hourly | Amount z-score vs the account's 30-day distribution |\n| account_tenure_days | Int | Profile store | Daily | Days since account creation |\n| account_txn_count_1h | Int | Streaming store | Seconds | Transactions from this account in the last hour |\n| card_new_merchant_flag | Bool | Feature store | Real-time | True if this card has never transacted at this merchant |\n| device_account_count_24h | Int | Feature store | Minutes | Distinct accounts seen on this device in 24 hours |\n| ip_country_mismatch | Bool | Derived | Real-time | True if IP country ≠ billing country ≠ card-issue country |\n| ip_is_vpn | Bool | Third-party feed | Hourly | VPN or proxy detection from the IP feed |\n| device_fingerprint_age_days | Int | Device store | Daily | Days since this fingerprint was first seen |\n| merchant_cb_rate_30d | Float | Merchant store | Daily | Chargeback rate at this merchant over 30 days |\n| graph_node_embedding | Float[64] | Graph job | Hourly | GraphSAGE embedding of the account node |\n| graph_shared_device_count | Int | Graph job | Hourly | Accounts sharing at least one device with this account |\n| anomaly_score | Float | Anomaly model | Real-time | Isolation-forest deviation score on this transaction |\n| step_up_pass_rate_30d | Float | 3DS feed | Daily | Fraction of step-ups this account has passed in 30 days |\n| behavior_biometric_deviation | Float | Session store | Real-time | Typing/swipe pattern divergence from account baseline |\n\n\nThe feature list is a representative slice, not the full set. A mature fraud system runs with 400 to 1,200 features depending on the business, and the long tail contains rare-but-high-signal flags that rarely fire but are decisive when they do.\n\n### Labels and Delayed Feedback\n\nChargebacks are the canonical label, but their 30-to-120-day delay creates a training dilemma: models trained only on fully-settled labels are three months behind the fraud landscape. Three label sources work together to shorten the loop.\n\nChargebacks arrive with a reason code (fraud, consumer dispute, processing error) and the training pipeline only treats \"fraud\" reason codes as positive labels; consumer disputes are relabeled as non-fraud since they reflect buyer remorse rather than stolen credentials. Reviewer decisions from the manual queue are available within hours and cover the borderline zone the model is most uncertain about, which is also the slice where they matter most. Customer complaints arrive through support tickets within hours to a few days and flag fraud that the model approved and reviewers never saw.\n\nTo handle the delay, the training pipeline uses a staged labeling strategy. A transaction from day T is initially labeled with any reviewer decision or customer complaint that's arrived by day T+3, and the model is retrained nightly using this provisional labeling. As chargebacks filter in over the following 90 days, the provisional labels are corrected: roughly 85% of reviewer-confirmed fraud eventually becomes a chargeback too, and the 15% that doesn't gets relabeled during the next full retrain. A weekly job rewrites the last 120 days of training data with the settled chargeback labels, and a monthly full rebuild joins everything from scratch to catch late arrivals.\n\nThis is an uncomfortable compromise: models trained on provisional labels are slightly noisier than models trained on settled labels, but models trained only on settled labels are useless against today's fraud. The industry consensus is that provisional labels with weekly correction is the right trade-off for real-time fraud systems.\n\n---\n\n# 4. Model\n\n### Baseline: Logistic Regression with Hand-Crafted Features\n\nThe baseline is a weighted logistic regression over a few hundred hand-engineered features: velocity counters, behavioral z-scores, device stability flags, merchant priors, and a handful of interaction terms that analysts have identified through years of fraud investigation. Training uses case-control sampling (keep all positives, sample 1 in 50 negatives) with inverse-probability weighting to recover calibrated probabilities. The model is cheap to train, easy to explain, and has been fraud detection's workhorse for three decades.\n\nThe appeal beyond simplicity is explainability. Regulators, chargeback investigators, and internal risk committees all ask \"why was this transaction flagged,\" and a linear model answers the question with a feature-weight table. For a regulated business, an interpretable baseline is not a training stage, it's a permanent parallel system that the more complex model is continuously compared against.\n\nThe limits show up when the feature engineering budget runs out. Every meaningful interaction has to be named, built, and monitored. The combinatorial space of \"velocity × merchant × device × geography\" is too large for hand-crafting to keep up with evolving fraud patterns, and the tenth thousandth hand-crafted cross feature adds less signal per hour of engineering time than an automated learner could produce for free.\n\n### Advanced: Three-Tower Hybrid\n\nThe production architecture in most mature fraud systems combines three complementary models whose outputs feed into a meta-learner that produces the final score.\n\n\n```mermaid\nflowchart LR\n FEAT[Transaction
+ Account
+ Context
Features]:::primary --> GBDT[GBDT
Risk Model]:::orange\n FEAT --> ANOM[Isolation
Forest /
Autoencoder]:::orange\n GRAPH[Identity
Graph]:::teal --> GNN[GraphSAGE
Embeddings]:::orange\n GBDT --> META[Meta-Learner
Calibrated
Logistic]:::primary\n ANOM --> META\n GNN --> META\n META --> P[Fraud
Probability]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe **GBDT supervised tower** is the primary predictor, typically LightGBM or XGBoost trained on the full labeled dataset with around 500 to 1,500 features and a few thousand trees. Gradient boosting dominates tabular fraud problems because it handles mixed-type features natively, learns nonlinear interactions without explicit crosses, and trains fast enough for a daily retrain on hundreds of millions of rows. Monotonicity constraints are added on a handful of features where the direction is known a priori: chargeback rate at the merchant should never decrease fraud probability, and account tenure should never increase it. Constraining monotonic features stops the model from picking up spurious inversions in low-data regions and makes explanations stable.\n\nThe **unsupervised anomaly tower** runs an isolation forest and a sparse autoencoder in parallel over the same feature vector. The isolation forest scores how easily a point is separated from the bulk of recent legitimate traffic; the autoencoder scores how poorly a point can be reconstructed from the legitimate-traffic manifold. Both scores are retrained weekly on a rolling 30-day window of approved transactions, which treats the approved-traffic distribution as \"normal\" and anything far from it as suspicious. The two anomaly scores feed the meta-learner as features, not as the final prediction. They matter because they carry signal about novel fraud patterns the supervised model has never seen labeled, filling in the distribution-shift blind spot.\n\nThe **graph tower** runs a GraphSAGE model on a bipartite identity graph whose nodes are accounts, cards, devices, and emails, and whose edges record shared ownership or usage. Node embeddings are recomputed hourly using neighborhood sampling (each node looks at a random sample of 20 one-hop and 10 two-hop neighbors rather than the full graph, which would be prohibitive at 500M nodes). The embedding for an account summarizes the structure of the network around it: densely connected to suspicious clusters, loosely connected to long-tenured accounts, sitting in a ring of recently-created accounts sharing a handful of devices. Those are all patterns that per-transaction features can't capture.\n\nThe **meta-learner** is a calibrated logistic regression or shallow neural network that takes the three tower outputs plus a small set of global context features and produces the final fraud probability. Keeping the meta-learner simple matters because it's the component whose calibration matters most for action routing, and simple calibrators are easier to keep well-calibrated under distribution shift.\n\n### Training\n\nTraining data is 90 days of logged transactions joined to chargeback, reviewer, and complaint labels. At a 0.2% positive rate, 99.8% of rows are negatives, and training on the full dataset wastes compute on essentially redundant examples. Case-control sampling keeps all positives and one in fifty negatives, reducing the training set by roughly 98% with minimal signal loss. Each surviving row carries a weight: positives have weight 1.0, negatives have weight 50.0 (the inverse of the sampling rate). Weighted log-loss recovers the full-data gradient in expectation.\n\nOn top of case-control sampling, the loss is weighted by transaction amount so the model optimizes dollar-weighted accuracy rather than event-weighted accuracy. A 5,000-dollar transaction's loss counts 1,000 times as much as a 5-dollar transaction's, which matches the business objective. The combined weight is `case_control_weight × amount_weight`, capped at a maximum to avoid a single extreme-amount transaction dominating a batch.\n\nTraining uses temporal splits rather than random splits. Random shuffling leaks future patterns into training and inflates offline metrics in a way that doesn't replicate online. A walk-forward scheme trains on days 1-83, validates on days 84-86, and tests on days 87-90, then rolls forward one day and repeats. The validation metrics drive hyperparameter tuning; the test metrics drive promotion decisions.\n\nThe loss function is dollar-weighted log-loss with an L2 regularizer on the GBDT's leaf outputs and a monotonicity constraint on the features listed above. The monotonicity constraint adds a pruning step inside the tree-building loop that rejects splits violating the declared direction. The regularizer damps leaf values so the model can't confidently predict high fraud on a single-sample leaf, which matters in the long tail where most features have only a handful of positive examples.\n\nThe daily retrain runs on a GPU cluster for the anomaly autoencoder and a CPU cluster for the GBDT and meta-learner, completing in roughly three hours end-to-end. The output is a model bundle containing all three towers plus the meta-learner plus the calibrator, which is versioned together and promoted through a shadow-evaluation gate before it ever scores live traffic. Emergency re-deploys in response to novel fraud patterns follow a shortened path: a reduced training window (the last 14 days with heavy weight on the newest patterns), a faster shadow window (4 hours instead of 24), and a more conservative canary rollout.\n\n---\n\n# 5. Serving\n\n### Inference Pipeline\n\nThe serving path for a single transaction looks like this.\n\n\n```mermaid\nflowchart LR\n TXN[Txn
t=0ms]:::primary --> FFETCH[Parallel Feature
Fetch
+15ms]:::teal\n FFETCH --> RULES[Rule
Engine
+3ms]:::red\n RULES --> SCORE[GBDT +
Anomaly +
Meta
+30ms]:::primary\n SCORE --> CAL[Calibrator
+2ms]:::orange\n CAL --> ROUTE[Action
Router
+3ms]:::orange\n ROUTE --> DEC[Decision
+2ms]:::green\n ROUTE -.log.-> LOG[Log +
Telemetry]:::teal\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nFeature fetch is the first stage and the biggest source of tail latency. The serving host issues parallel lookups to the online feature store (Redis for hot features, Aerospike for warm), the device and session service, the account profile cache, and the graph-embedding cache. Each lookup has a hard deadline; any lookup that misses the deadline returns a default value flagged as stale, and the feature vector assembled for the model includes a freshness bitmap so the model can learn to downweight stale features.\n\nThe rule engine runs next. Rules are compiled from a DSL into a decision graph that evaluates in under 3ms for a typical set of 200 to 500 rules. Rules cover three purposes: hard-block known-bad entities (sanctioned accounts, blacklisted cards), fast-approve low-risk flows (returning customer, small amount, established merchant), and route specific patterns to a specialized model (enterprise transactions routed to a separate enterprise-tuned scorer). A rule match can short-circuit the full model evaluation, which is why the rules sit before the scorer rather than after.\n\nThe model scorer runs the three towers either in parallel (on a host with enough cores) or sequentially with early-exit logic: if the GBDT score is decisively low (below the 5th percentile on recent traffic) and the anomaly tower isn't flagging, the graph tower is skipped because its embedding is already stale by up to an hour and won't change the decision on clearly-clean transactions. Early exit saves about 8ms on 85% of traffic and is worth the added complexity on high-volume platforms.\n\nCalibration converts the raw meta-learner score into a probability the action router can act on. The action router then compares the calibrated probability and the transaction amount against the expected-loss curve to pick approve, challenge, review, or decline. The curve is refreshed weekly from business cost parameters (current chargeback-loss average, current false-decline-cost estimate) and is the one place where business policy cleanly separates from ML model behavior.\n\n### Latency Budget\n\nThe 100ms p99 budget breaks down like this.\n\n\n| Stage | Budget | Notes |\n|---|---|---|\n| Network and parsing | 8ms | Gateway to fraud service RPC, request deserialization |\n| Feature fetch | 20ms | Parallel multi-get from feature store and ancillary services |\n| Rule engine | 3ms | Compiled decision graph, mostly hash lookups |\n| GBDT scoring | 12ms | 2,000 trees at depth 8, CPU-vectorized inference |\n| Anomaly scoring | 6ms | Isolation forest + autoencoder, parallel to GBDT |\n| Graph embedding lookup | 2ms | Pre-computed embedding read from feature store |\n| Meta-learner | 3ms | Logistic regression on a 20-dim input |\n| Calibration | 2ms | Isotonic regression lookup |\n| Action routing | 3ms | Threshold lookup against expected-loss curve |\n| Logging and response | 6ms | Async log emit, response serialization |\n| Buffer | 35ms | Reserved for tail variance in feature fetch |\n\n\nThe buffer is intentionally large because feature fetch is bimodal: the median is around 8ms but p99 can spike to 30-40ms when the feature store is under load or when a cold-cache lookup hits the warm tier. A smaller buffer would make the 100ms SLA brittle. In steady state, a typical transaction completes in 50-65ms, and the 100ms bucket exists for the bad cases.\n\n### Batch vs Real-Time\n\nReal-time paths are the scorer fleet itself and the streaming feature aggregators that keep velocity counters fresh. Everything else is batch or streaming-with-larger-windows. The graph embedding job runs hourly because recomputing the identity graph is expensive and the marginal accuracy gain from updating every 10 minutes versus every hour is small. The model retrain runs nightly because the incremental gain from continuous retraining doesn't pay for the operational complexity of atomic model swaps at fraud-scale traffic. The chargeback-feedback join runs nightly and rewrites labels as new chargebacks arrive.\n\nThe division of labor is set by what changes fastest. Transactions and user behavior change second by second, so they get real-time handling. Merchant risk and identity-graph structure shift over hours to days, so they get hourly updates. Model weights and calibration tables drift over days to weeks, so they get daily refreshes.\n\n### Degraded Serving\n\nWhen the model service fails or feature fetch starts timing out, the fallback path is rules plus a per-account risk table keyed on account tenure, historical chargeback history, and a bucketed amount. The table is refreshed every 15 minutes from the streaming pipeline and gives a reasonable, if coarse, approval decision in the absence of a live model. Revenue in degraded mode runs at roughly 85 to 92% of full-model mode, which is dramatically better than refusing to decision at all. Fallback is strict rather than graceful: the full scorer has a hard 80ms deadline, and any request past that is routed to the fallback rather than a mix of partial-model and fallback outputs. Mixed outputs would produce inconsistent rankings across transactions and complicate debugging.\n\nThe fallback path is exercised deliberately in a weekly chaos-engineering drill that shuts off the scorer for 5 minutes in a single region and measures the degraded-mode chargeback rate, false-decline rate, and reviewer-queue backlog. Fallback paths that aren't exercised regularly rot, and discovering a broken fallback during a real incident is how outages escalate from 5 minutes to 2 hours.\n\n---\n\n# 6. Deep Dives\n\n### 6.1 Extreme Class Imbalance and Cost-Sensitive Learning\n\nFraud at a 0.1 to 0.3% base rate is one of the most imbalanced supervised problems in production ML. Naive handling of imbalance produces two kinds of failure. A model trained on the raw distribution spends 99.7% of its gradient signal on non-fraud and barely learns the positive class at all. A model trained on a rebalanced distribution (say 10% positives) learns a useful decision boundary but produces probabilities biased high by roughly a factor of 30, which corrupts the threshold-based action router.\n\nThe production recipe is case-control sampling with inverse-probability weighting. All positives are kept, negatives are sampled at a rate between 1-in-20 and 1-in-100 depending on data volume, and each row's loss is weighted by the inverse of its inclusion probability. The effect on the learned function is that gradient descent proceeds as if it had the full dataset, but the computation cost is 1-to-5% of what full training would require. The correction for serving-time calibration uses the same closed-form transform that ad CTR systems use: if the model was trained on a population with negative-keep-rate `r` and outputs a probability `p'`, the calibrated probability is `p = p' / (p' + (1 - p') / r)`. Missing this correction is the most common cause of systematically over-declining transactions in a fraud system.\n\nOn top of class-frequency weighting, fraud systems add dollar weighting. A model that optimizes unweighted log-loss treats catching a 5-dollar fraud the same as catching a 5,000-dollar fraud, which is aligned with neither the business objective nor the chargeback cost curve. Weighting each training example by its transaction amount (capped at a reasonable ceiling to prevent single-example dominance) produces a model whose probability outputs reflect expected-loss ranking rather than event-count ranking. The cap matters: a 250,000-dollar transaction with a weight proportional to amount would single-handedly dominate a batch gradient, and the cap at around the 99th percentile of amounts keeps gradients stable.\n\nThe threshold where approve becomes challenge becomes review becomes decline is a business-policy choice, not a model choice. The model produces a calibrated probability; the action router computes expected loss at each candidate threshold and picks the operating point that minimizes the sum of missed-fraud losses and false-decline losses for each transaction amount band. The curve is refreshed weekly as cost parameters drift and reviewed by risk operations any time a meaningful shift in either the chargeback-cost average or the false-decline-cost estimate is observed.\n\n\n| Approach | How It Handles Imbalance | Strength | Weakness |\n|---|---|---|---|\n| Train on raw distribution | No adjustment | Simple, unbiased | 99%+ rows provide near-zero gradient; wastes compute |\n| Oversample positives | Replicate positives | Rebalances training | Overfits to specific positive examples; poor generalization |\n| Undersample negatives | Drop most negatives | Rebalances and speeds training | Loses information from hard negatives |\n| Case-control + IPW | Undersample negatives + weight | Recovers full-data gradient in expectation | Requires correct weight application at training and serving |\n| SMOTE | Synthesize positive interpolations | Smooths decision boundary | Synthetic fraud examples don't match real-fraud distribution |\n| Focal loss | Reweight by example difficulty | No sampling, all data used | Harder to calibrate; probability outputs need recalibration |\n\n\nThe industry pattern is case-control sampling with IPW for tabular fraud, because it's the only approach that cleanly recovers calibrated probabilities without a post-hoc correction step. Focal loss is popular in deep-learning-heavy shops, but it's more common in image or text fraud signals than in the tabular scoring path.\n\n\n```mermaid\nflowchart LR\n SCORE[Calibrated
Probability]:::primary --> EL[Expected Loss
= p*CB + (1-p)*FD]:::orange\n AMT[Transaction
Amount]:::primary --> EL\n EL --> LOW{p < 0.02}:::teal\n EL --> MID{0.02 <= p
< 0.15}:::teal\n EL --> HIGH{0.15 <= p
< 0.6}:::teal\n EL --> MAX{p >= 0.6}:::teal\n LOW --> A[Approve]:::green\n MID --> C[Challenge
3DS / OTP]:::orange\n HIGH --> R[Review
Queue]:::orange\n MAX --> D[Decline]:::red\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe thresholds in the diagram are illustrative, not universal. Each platform's expected-loss curve has its own shape based on chargeback costs, false-decline costs, and the amount distribution, and the boundaries between action buckets get retuned weekly as those inputs drift. The important structural point is that the mapping from probability to action isn't a single cutoff but a schedule that depends on transaction amount, because a 0.1 probability on a 10-dollar transaction has different expected loss than the same probability on a 5,000-dollar transaction.\n\n### 6.2 Graph-Based Fraud Ring Detection\n\nSophisticated fraud operates in rings. A single fraudster creates dozens or hundreds of accounts, rotates them across a pool of stolen cards, and uses a small set of devices and email providers to manage the whole operation. Each individual transaction from a ring can look innocuous: small amounts, plausible ship-to addresses, previously-seen merchants. The signature of the ring only appears when the structure of the network is visible.\n\nThe identity graph is the data representation that makes the structure visible. Nodes are entities (accounts, cards, devices, emails, shipping addresses, phone numbers). Edges are observed relationships: \"this account used this card,\" \"this account logged in from this device,\" \"this account has this shipping address.\" Edges are typed and weighted by count and recency. The graph is massive at scale (500M accounts, 200M devices, 150M cards, 100M emails, with billions of edges) and gets rebuilt hourly from the streaming transaction log.\n\nThree classes of graph-derived features feed the model.\n\n**Aggregate graph features** are simple counts computable by a graph query: how many accounts share this device, how many distinct cards has this account used, how many shipping addresses has this card been used with. These features are cheap, easy to monitor, and surprisingly powerful. A device serving 50 distinct accounts in 24 hours is prima facie suspicious regardless of what any single account does.\n\n**Community-detection features** run a clustering pass over the graph (label propagation or Louvain) and flag accounts sitting in tight clusters of recently-created accounts. Clusters with high chargeback rates become known risk clusters, and membership in such a cluster becomes a feature itself. Community detection runs daily on a compacted version of the graph, which is cheap enough that re-running it nightly is sustainable even at hundreds of millions of nodes.\n\n**GNN node embeddings** are the most expressive feature and also the most expensive. A GraphSAGE model aggregates features from each node's sampled neighborhood through two or three message-passing layers and produces a 64-dimensional embedding that summarizes the node's multi-hop context. The embedding for an account captures not just its immediate neighbors but the structure of the network a few hops out, which is where fraud rings become visible. Two accounts with similar embeddings are structurally similar in the graph, and rings of coordinated fraud accounts end up with clustered embeddings that the downstream model learns to flag.\n\n\n```mermaid\nflowchart LR\n RAW[Transaction
Log]:::primary --> EDGE[Edge
Extractor]:::teal\n EDGE --> GBUILD[Graph
Builder]:::orange\n GBUILD --> SAMPLE[Neighborhood
Sampler]:::teal\n SAMPLE --> MSG[Message
Passing]:::primary\n MSG --> MSG\n MSG --> EMB[Node
Embeddings]:::green\n EMB --> FS[Feature
Store]:::orange\n FS --> SCORE[Scorer]:::primary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe key engineering choice is neighborhood sampling rather than full-graph message passing. At 500M nodes, a full-graph GNN would be intractable: computing one layer of message passing requires touching every edge, and even with sparse primitives it's too slow to run hourly. Neighborhood sampling replaces the full neighborhood with a random sample at each layer (say 20 one-hop and 10 two-hop neighbors), which reduces the per-node compute to a fixed budget independent of graph size. The trade-off is that high-degree nodes (hub accounts, shared devices) have their neighborhoods undersampled, so the embedding for such a node is a noisier summary than it would be with full message passing. In practice the loss is small because the neighborhood's structure is similar across samples, and the hourly refresh averages out sampling noise.\n\nThree practical complications show up in production. First, the graph is heavily skewed: most nodes have degree under 5, but a tail of nodes (public WiFi IPs, rental devices, corporate email domains) has degree in the thousands. These high-degree nodes dominate message passing if not handled, so the typical fix is a degree cap that truncates contributions from any node with more than some threshold of neighbors. Second, the graph is sparse at creation: a brand-new account has no edges until it transacts, so cold-start-on-the-graph is a real problem. The fix is initial embeddings seeded from account metadata (country, signup method, device at signup) rather than from graph structure, with the graph-derived embedding fading in over the account's first dozen transactions. Third, adversaries adapt: once they know graph features are used, they stop sharing devices across accounts. The response is to add more edge types (IP ASN, TLS fingerprint, behavioral biometric clusters) and to treat the graph as an arms race rather than a solved problem.\n\nThe question that comes up in interviews is when a GNN is actually better than a GBDT with graph-aggregate features. The honest answer is that a GBDT with well-designed aggregate features (degree counts, cluster membership, neighbor-averaged risk) captures most of what a GNN learns, and the GBDT runs faster, trains faster, and is easier to monitor. The GNN earns its place when the ring structure is more than one hop away, because aggregate features degrade sharply past the immediate neighborhood, while a GNN with two message-passing layers sees two hops natively. Fraud rings that obscure themselves by routing through intermediate \"mule\" accounts are exactly the case where the GNN matters. For straightforward co-usage patterns, a GBDT is enough.\n\n### 6.3 Concept Drift and Adversarial Adaptation\n\nA fraud model is decaying from the moment it deploys. Fraudsters see the decline patterns, the step-up triggers, and the accounts that slip through, and they adapt. The adversarial feedback loop operates on a timescale of days to weeks for any given tactic, which means a model that doesn't retrain frequently falls behind. The challenge is telling adversarial drift (fraud tactics changing) from natural drift (user behavior changing because of a new feature launch, a holiday season, or a new geography coming online). The two require different responses.\n\nDrift detection runs on several signals in parallel. The first is the population stability index (PSI) on the distribution of input features: if the distribution of transaction amounts shifts by more than 0.2 on the PSI scale over a rolling 7-day window, an alarm fires and the feature is flagged for investigation. The second is the distribution of the model's predicted probability: if the histogram of predictions shifts such that the KL divergence against a rolling baseline exceeds a threshold, the model is producing systematically different scores than it did last week. The third is the actual chargeback rate on approved transactions, which is the business-outcome metric but arrives with a 30-to-120-day delay, so it's a confirmation signal rather than an early-warning signal. The fourth is the reviewer-agreement rate: if human reviewers start overriding the model's recommendations more often than usual, the model is likely picking up a pattern reviewers don't trust.\n\nThe response to drift depends on the source. Drift in an input feature (a new bank coming online produces a different BIN distribution) is handled by waiting for the daily retrain to absorb the new distribution. Drift in the prediction distribution with stable inputs points to a model bug or a calibration issue, and the response is to roll back to the previous model version and investigate. Drift in the chargeback rate without a corresponding drift in features or predictions is the scariest case, because it means fraud tactics have shifted in ways the model can't currently detect; this triggers an emergency feature investigation led by the risk analytics team, which often produces a new rule deployed within hours while the training pipeline catches up in days.\n\nChampion-challenger deployment is the structural defense against silent regressions. A candidate model is deployed alongside the production model and receives 100% of the same transactions in shadow mode: its predictions are logged but not acted on. After 48 to 72 hours of shadow data, the challenger's scores are compared to the champion's on the same transactions, and the challenger is promoted only if its performance on the overlap is at least as good across every major slice. Shadow mode catches two kinds of problem that offline evaluation misses: serving-time feature differences (the online feature store returns slightly different values than the training pipeline computed) and regressions on slices that didn't appear in the offline test set (a new geography, a new merchant category).\n\nRed-teaming is the other production practice worth noting. A small team inside the fraud organization is chartered with actively trying to evade the current model, using techniques ranging from adversarial feature perturbation (find the smallest change to a transaction that flips the score from decline to approve) to full synthetic-identity orchestration (create fake accounts, manufacture transaction history, then run an attack). The red team's findings feed directly into new features, new rules, and new training data. Red-teaming isn't optional at scale; without it, real adversaries do the testing for free and the first notification comes through the chargeback feed 90 days later.\n\n\n```mermaid\nflowchart LR\n FEAT[Feature
PSI]:::primary --> DET[Drift
Detector]:::orange\n PRED[Score
KL Divergence]:::primary --> DET\n REV[Reviewer
Agreement]:::primary --> DET\n CB[Chargeback
Rate]:::primary --> DET\n DET --> CAUSE{Root
Cause?}:::teal\n CAUSE --> NATURAL[Natural Drift
Wait for daily
retrain]:::green\n CAUSE --> MODEL[Model Bug
Rollback to
prior version]:::red\n CAUSE --> ADV[Adversarial
Emergency
retrain + rules]:::red\n ADV --> RULE[Deploy Rule
within hours]:::orange\n ADV --> RETRAIN[14-day retrain
within a day]:::orange\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe triage diagram is the mental model the on-call uses when a drift alarm fires. Matching the source of the drift to the right response matters because the three responses have very different costs: waiting for a daily retrain is cheap but slow, rolling back a model is fast but undoes any genuine wins in the rolled-back version, and an emergency retrain is expensive in engineering hours and carries rollout risk. Pattern-matching drift signals to the correct response is a skill fraud teams develop over years, and drills where the team walks through historical drift events help new on-calls build intuition before the real thing happens.\n\n### 6.4 Human Review Queue and the Feedback Loop\n\nThe review queue is more than a safety valve. It's a label-generation pipeline that produces high-quality, fast-arriving training signals for the slice of transactions the model is most uncertain about. Designing it well matters as much as designing the model.\n\nPriority scoring decides which transactions reach reviewers first. The naive ordering (highest-probability first) is wrong because reviewers waste time on obviously fraudulent transactions that the model was going to decline anyway. The better ordering is expected dollar loss times uncertainty: `amount × P(fraud) × (1 - |2 × P(fraud) - 1|)`. The last term peaks at probability 0.5 and falls to zero at 0 and 1, so transactions the model is genuinely unsure about float to the top of the queue. A reviewer spending an hour on a pool of 50% transactions where the model is uncertain produces much more useful training signal than spending the same hour confirming transactions the model already called 99% fraud.\n\nReviewer UX matters for both throughput and label quality. The reviewer interface shows the transaction details, the account and device history, a feature-attribution panel (SHAP values for the top 10 contributing features with direction), and related transactions from the same account or card. A good UI lets a reviewer hit a confident decision in under 60 seconds for clear cases and lets them escalate ambiguous cases to a senior analyst. Label quality is monitored through a continuous audit pass: 5% of reviewer decisions are double-reviewed, and disagreement rates are tracked per reviewer. Reviewers with elevated disagreement rates get retraining; ones with persistent issues get moved off the fraud queue.\n\nReviewer decisions feed back into training within hours. The fast feedback matters because it shortens the loop for novel fraud: if a reviewer identifies a new pattern on Monday morning, the feature is engineered Monday afternoon, the model is retrained with the new labels Monday night, and the retrained model ships Tuesday. Without reviewer labels, that loop would stretch to 90 days while the team waits for chargebacks to confirm the pattern.\n\n\n```mermaid\nflowchart LR\n TXN[Uncertain
Transaction]:::primary --> Q[Priority
Queue]:::orange\n Q --> REV[Reviewer
Interface]:::teal\n REV --> DEC[Reviewer
Decision]:::green\n DEC -.label.-> TRAIN[Training
Pipeline]:::primary\n DEC --> CUST[Customer
Notification]:::orange\n CUST --> APP[Appeals
Path]:::red\n APP --> REV\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nAppeals close a loop that's otherwise one-sided. A customer whose transaction was declined or account was suspended can request a re-review. Confirmed-good appeals are treated as corrected labels: the original decision was wrong, and the training pipeline inserts the corrected row with a higher weight than a typical negative. Appeals also surface false-decline patterns that are otherwise invisible: if appeals from a specific country or card type spike, the model or the rules are systematically over-declining on that segment, and the signal arrives long before the customer-complaint rate would. Most fraud teams run weekly appeal reviews and use them as a leading indicator of fairness problems.\n\nThe last operational detail is reviewer capacity planning. At 2% of 500M daily transactions, 10 million items a day flow into the review queue, and at a reviewer throughput of 5,000 decisions per analyst per day, the queue would need 2,000 analysts to clear in real time. No business runs that many, so the queue is capped by the action router: only the top X-percent of uncertain-and-high-value transactions actually go to review, and the rest are auto-actioned based on the model's recommendation. The cap is tuned to the available reviewer capacity, and when the available capacity shrinks, the model's direct-decision threshold tightens automatically so the queue doesn't back up past its SLA.\n\n---\n\n# 7. Evaluation and Iteration\n\n### Offline Evaluation\n\nOffline evaluation uses temporal holdout rather than random splits. Training on days 1-83, validating on days 84-86, testing on days 87-90 preserves the time ordering that real serving respects. Randomly shuffling leaks future patterns backward and overstates offline metrics by enough to change promotion decisions, which is why every mature fraud team runs temporal splits even when it costs them 5 to 10% on the raw PR-AUC number.\n\nThe headline offline metric is PR-AUC, because at 0.1% positive rate the ROC curve is dominated by the negative class and AUC-ROC above 0.99 is routine even for mediocre models. Operating-point metrics (recall at 1% FPR, precision at 95% recall) are reported alongside PR-AUC because they describe behavior at the specific point the action router actually uses. Expected dollar loss along the threshold curve is computed for each candidate model, and promotion requires the candidate's expected loss to be lower than the champion's at the chosen operating point.\n\nSlice evaluation is where regressions hide. A model that improves aggregate PR-AUC by 3% can still regress on a high-stakes slice (high-value merchants, international transactions, long-tenure customers). Dashboards break out metrics by country, card network, merchant category, transaction-amount band, and customer tenure, and the promotion bar is that no major slice regresses by more than 2% on PR-AUC or 5% on calibration error. The cost of being strict on slices is that some genuinely better models get blocked, but the cost of being lenient is a promotion that destroys trust in a specific customer segment in a way that takes months to rebuild.\n\nCalibration evaluation is its own pass. Reliability diagrams bucket predictions into deciles and plot predicted probability against observed rate. A well-calibrated model produces a diagonal; deviations flag subsampling-correction bugs, training-serving skew, or distribution shift that the model hasn't caught up to. The expected calibration error (ECE) across deciles is the single number that summarizes the plot; a new model ships only if ECE is below 0.02 across every major slice.\n\n### Online Evaluation\n\nOnline evaluation is an interleaved threshold experiment rather than a true A/B test. The reason is that a true A/B test would send some transactions to the new model and some to the old, which creates accounting complexity when the two models make different decisions on the same customer across multiple transactions. Interleaving instead runs both models on every transaction, picks the action from the treatment model for a random 5% slice, and compares outcomes over a two-to-four-week window.\n\nGuardrail metrics watched during the rollout include chargeback rate, false-decline rate, review queue backlog, customer complaint rate, and serving latency. The rollout ramps from 1% to 10% to 50% to 100% over 10 to 14 days, and any guardrail regression past a preset threshold halts the ramp and rolls back. The slowest ramp is the transition from 50% to 100%, because once the new model is serving majority traffic, any regression affects the bulk of GMV and takes longer to detect through chargebacks.\n\nThe measurement challenge with fraud is label delay. A two-week rollout completes before the chargebacks from its first day have arrived, so early signals lean on faster-arriving labels (reviewer agreement rate, customer complaint rate) plus proxy metrics (approval rate drift, score distribution stability). A secondary pass runs 90 days after full rollout to confirm the chargeback-rate impact with settled labels; this sometimes uncovers regressions that weren't visible in the first two weeks, and the response is a targeted investigation rather than a rollback, since rolling back after 90 days is usually worse than fixing forward.\n\n### Monitoring\n\nOnce a model is in production, monitoring spans four signal classes.\n\n\n| Signal | What It Catches | Alarm Threshold |\n|---|---|---|\n| Score distribution KL | Feature store outages, pipeline drift, model corruption | KL > 0.1 vs rolling 7-day baseline |\n| Calibration ECE per decile | Drift, sampling-correction bugs, miscalibrated retraining | ECE > 0.03 in any decile over a 6-hour window |\n| Feature freshness | Streaming pipeline lag, aggregator failure | P99 velocity feature older than 60 seconds |\n| Training-serving skew | Feature pipeline divergence, serving-side bugs | Top-50 features by SHAP importance, any >2% divergence |\n| Approval rate | False-decline waves, model promotion regressions | Day-over-day shift >0.5 percentage points |\n| Reviewer backlog | Queue overflow, reviewer capacity issues | P95 queue age > 4 hours |\n| Chargeback rate (delayed) | Business outcome; confirmation signal | Weekly rate > 0.5% of approved GMV |\n| Complaint rate | Fast proxy for false declines | >150 complaints per million approvals |\n| Subgroup decline ratio | Fairness drift | Max decline-rate ratio across country or card-type slices >1.5 |\n\n\nTraining-serving skew is the monitor that earns its keep most often. A feature that's mean-zero in the training pipeline but mean-0.08 at serving time silently corrupts every prediction, and the calibrator often partially masks the error by recalibrating against corrupted serving scores. The defense is a shadow job that recomputes features at serving time from the logged raw inputs and compares the computed values to what the live feature pipeline produced. Any divergence in the top-50 features by SHAP importance pages the on-call. This catches pipeline bugs within a day of them starting to bite, compared to the weeks it would take for the downstream chargeback signal to surface the same problem.\n\nRetraining cadence stays daily for the full model and hourly for the graph embedding job. Emergency retrains are triggered by drift alarms and go through a shortened path: reduced training window, faster shadow evaluation, more conservative canary ramp. The emergency path is practiced quarterly in a drill that pretends a novel fraud pattern has been detected and runs through the full response sequence end-to-end.\n\n---\n\n# Interview Questions\n\n**Q1: Why combine unsupervised anomaly detection with a supervised classifier instead of picking one?**\n\nThe supervised classifier is better on known fraud patterns because labels directly tell it what fraud looks like. The unsupervised anomaly detector is better on unknown patterns because it doesn't need labels; it just flags transactions that deviate from the legitimate-traffic distribution. Fraud is adversarial and novel patterns appear every week or two, so a pure supervised system is always a step behind whatever the adversaries do next. Using the anomaly score as a feature in the supervised model rather than a parallel decision preserves the calibrated probability output that the action router needs while still pulling in the novel-pattern signal. The combination catches both known fraud (where the supervised model is strong) and novel fraud (where the anomaly detector surfaces deviation before labels arrive to train on it).\n\n**Q2: How do you train a model when chargeback labels arrive 30 to 120 days after the transaction?**\n\nThe pipeline uses a staged labeling strategy with three label sources at different latencies. Chargebacks are the gold label but slow; reviewer decisions arrive within hours for the uncertain slice; customer complaints arrive within a day or two for fraud that approved through and got reported. Training starts with provisional labels from reviewers and complaints, retrains nightly, and then rewrites the last 120 days of training data with settled chargeback labels as they arrive on a weekly cycle. This gives the model both fast adaptation to new patterns (via reviewer labels) and eventual ground-truth correctness (via chargebacks). The provisional labels are noisier than chargebacks, but models trained only on settled chargebacks would be three months behind the fraud landscape, which is worse than the noise penalty.\n\n**Q3: Why does calibration matter so much for a fraud score, and how do you maintain it?**\n\nThe action router decides approve, challenge, review, or decline based on where the calibrated probability lands on an expected-loss curve, and miscalibrated scores route transactions to the wrong bucket. A model biased high by a factor of 2 pushes borderline transactions from challenge to decline and triples the false-decline rate; biased low, and real fraud gets auto-approved. Case-control sampling during training induces a known bias that's corrected at serving time with a closed-form probability transform. On top of that, an isotonic regression calibrator is fit daily on held-out data to correct residual distortions, and a lightweight Platt scaling layer on a rolling one-hour window handles fast drift from feature distribution shifts. Calibration error per decile is monitored continuously, and any decile exceeding an ECE of 0.03 pages the on-call.\n\n**Q4: When is a GNN actually better than a GBDT with graph-aggregate features for fraud detection?**\n\nA GBDT with well-designed aggregate features (degree counts, cluster membership, neighbor-averaged risk) captures most of the signal from the immediate one-hop neighborhood, trains faster, and is easier to monitor. The GNN earns its place when fraud rings obscure themselves by routing through intermediate accounts, because aggregate features degrade sharply past the immediate neighbors while a GNN with two message-passing layers sees two hops natively. For straightforward co-usage patterns (many accounts on one device, one card used across many shipping addresses), a GBDT is sufficient and much cheaper to run. For organized rings that use mule accounts to buffer between fraudsters and purchases, the multi-hop context is where the signal lives and the GNN is worth its complexity. In practice, production systems use both, with the GNN producing node embeddings that feed into a GBDT alongside the aggregate features.\n\n**Q5: How do you evaluate a fraud model offline when the fraud rate is under 0.3% and chargeback labels take months to arrive?**\n\nThe offline evaluation runs on temporal holdout with settled labels from transactions at least 90 days old, reporting PR-AUC, recall at the target false-positive rate, expected dollar loss along the threshold curve, and calibration ECE per decile. Slice evaluation across country, card type, merchant category, and customer tenure catches regressions that aggregate metrics hide. Because settled labels take months, a second evaluation pass uses provisional labels from reviewer decisions and customer complaints on more recent data, which is noisier but captures fraud patterns that emerged after the last settled batch. The two evaluations together create a trade-off: settled labels are reliable but describe an old fraud distribution; provisional labels describe current fraud but are noisier. Most teams promote models on settled-label metrics but monitor for regressions on provisional-label metrics, and any candidate that looks strong on settled labels but weak on provisional labels is a signal that the fraud distribution has shifted since the settled-label window closed.\n\n---\n\n# Summary\n\n- Fraud detection is a binary classification problem layered on top of an anomaly detector and a graph-based feature pipeline, because the adversarial environment makes any single modeling approach insufficient on its own.\n- The positive class is extremely rare (0.1 to 0.3%) and the training pipeline uses case-control sampling with inverse-probability weighting to recover calibrated probabilities efficiently, plus dollar weighting to align the loss with expected-loss business economics.\n- The action space has four buckets (approve, challenge, review, decline), and the calibrated probability is mapped to an action through an expected-loss curve derived from business cost parameters, which is the clean separation between model output and business policy.\n- Labels arrive on three timescales: chargebacks are the gold label but 30 to 120 days delayed, reviewer decisions arrive within hours for uncertain transactions, and customer complaints fill the gap for fraud that approved through; the pipeline trains on provisional labels nightly and rewrites with settled labels on a weekly cycle.\n- Graph features from an identity network capturing shared devices, cards, emails, and IPs surface coordinated fraud rings that per-transaction features miss; a GNN with neighborhood sampling produces node embeddings that are especially valuable for rings that route through intermediate accounts.\n- Serving has a 100ms p99 budget dominated by feature fetch; the scorer fleet uses early exit on obviously-clean transactions, and a rule engine sits ahead of the model to hard-block known-bad entities and route specific patterns to specialized scorers.\n- Concept drift and adversarial adaptation run on timescales of days to weeks, and the defensive stack is built around daily retraining, champion-challenger shadow deployment, population stability monitoring, and an in-house red team that actively tries to evade the current model.\n- The human review queue is a first-class component that doubles as a label-generation pipeline; priority scoring by expected dollar loss times uncertainty makes reviewer time produce the most useful training signal, and appeals close the loop on false declines.\n\nThe next chapter turns from transaction-level fraud to content moderation, which shares the adversarial dynamics and review-queue structure but moves into multi-modal classification on text, image, and video.","pageType":"ml-system-design"}

Design a Fraud Detection System

Ashish Pratap Singh

Get Premium