UPI Fraud Detection at a 0.01% Base Rate: Why Accuracy Is a Lie

I built UPI Fraud Guard, a fraud scoring service for UPI-style transaction streams, with Scikit-Learn, XGBoost, Pandas and a Flask serving layer (source on GitHub). At the operating point I measured 88% detection rate, under 0.1% false positive rate, roughly 85% precision, and under 100ms inference. Every one of those numbers exists because of a decision I had to make explicitly, because at a 0.01% fraud base rate the naive defaults quietly betray you.

This essay is about the 0.01% problem: about one fraudulent transaction in ten thousand. Almost everything about fraud modeling at that rate is counterintuitive, starting with the metric everyone quotes first.

Why headline accuracy lies at extreme imbalance

At 0.01% prevalence, a model that predicts "never fraud" for every transaction is 99.99% accurate. It also catches nothing, and nobody would call it a fraud model. Yet accuracy is the first number most dashboards show, because accuracy is the one metric that never needs the base rate in the room to look good.

The failure mode is structural, not statistical. With 9,999 legitimate transactions for every fraud, the majority class dominates every aggregate you can compute. Gradient boosting will happily grind loss down by learning the majority pattern and treating the minority as rounding error unless you force the trade-off into the open: class weights, scale_pos_weight, or an explicit threshold decision after training. I used class weighting during training and then settled the real decision at the threshold stage, because that is where the business actually lives.

The honest way to read any fraud metric is to ask three questions before looking at the number. What is the base rate. What is the false positive count at the chosen operating point. What does a missed fraud cost. If the person presenting the model cannot answer those three, the model is not understood, only measured.

The metrics that matter here

Precision and recall at the operating point, not in the abstract. In this build, 88% detection rate means the model catches 88 of every 100 frauds shown to it; roughly 85% precision means about 15% of the alerts it raises are false alarms. Those two numbers are the same trade seen from two sides, and they move together whenever you move the threshold. Neither is "the" score; together they describe the operating point.

PR-AUC over ROC-AUC. ROC-AUC measures ranking quality across all thresholds, and the enormous mass of true negatives flattens the false positive axis so thoroughly that a near-useless model can still post a flattering number. The precision-recall curve has no true-negative axis to hide behind. At extreme imbalance, average precision is the honest summary of ranking quality, and it is the curve I would show a stakeholder first.

False positives per day, at your actual volume. A sub-0.1% false positive rate sounds tiny until you multiply it by real transaction volume. The unit the business feels is customers wrongly blocked per day, not percent false positive. I keep this qualitative on purpose: the right number depends on your traffic, and inventing one would be theater. What you can say firmly is that at 0.01% prevalence the legitimate majority is so large that even a very small false positive rate produces a large absolute number of wrongly blocked customers, which is exactly why precision at the operating point, not accuracy, deserves the attention.

Detection rate inside a false-positive budget. Set the false positive ceiling first (the maximum blocks your review team and your customers can absorb), then maximize detection within it. This framing makes the trade-off legible to people who do not read confusion matrices: here is the budget of friction we accept; here is the fraud we catch inside it.

Feature engineering that carried the model

The features that did the work were behavioral, not demographic:

• Amount velocity. How much money moves, and how often, inside sliding windows. Fraudsters burn through freshly compromised instruments quickly; velocity windows catch the burst that any static amount threshold misses.
• Merchant diversity. How many distinct merchants a user pays in a window. A sudden explosion in distinct payees is a classic account-takeover signature, and it is cheap to compute.
• Geolocation entropy. How dispersed the locations behind the transactions are. Entropy turns where did this happen into how surprising is this scatter, which is the actual signal; a high-entropy burst from a normally tight pattern reads very differently from a routine travel week.
• Time-of-day. Transaction timing against the user's own rhythm. Fraud scripts do not keep your hours.

Why counts beat demographics for fraud: demographics are stable and shared by millions of people, so they separate almost nothing. Behavioral counts are individual and hard to fake at speed. A fraudster can buy a new identity; they cannot cheaply manufacture a three-month velocity history that looks ordinary. This is also why XGBoost was a comfortable fit: gradient-boosted trees thrive on exactly this kind of tabular, count-and-ratio feature space, need no heavy preprocessing, and expose feature importances you can actually discuss with the people who own the review queue.

Threshold-moving instead of retraining

The 0.5 threshold is a convention, not a business decision. XGBoost outputs a score; the threshold converts scores into actions: allow, step-up verification, block, or route to review. Choosing that cutoff is the highest-leverage decision in the system, and it is free. No retraining, no new features, just a different place to act on the same ranking.

Pick the operating point by pricing the two errors. A missed fraud costs the fraud loss plus the dispute and chargeback process. A false block costs a good customer friction, a possible support contact, and a small but real risk of churn. When the cost of a miss outweighs the cost of a false block, lower the threshold; when the review queue is drowning in bad alerts, raise it. The procedure is mechanical: take a labeled validation slice, sweep the threshold, plot precision and recall against it, and mark the point where cost-weighted error is minimized. Then review that point with the people who absorb the consequences, not just with the model's author.

Threshold-moving also handles drift that retraining cannot. Fraud comes in waves; the base rate is not stationary. When prevalence shifts, the optimal cutoff shifts with it, and you can recalibrate on recent labeled data in an afternoon. Retraining is for when the relationship between features and fraud changes (new attack vectors, new flows, new instruments); threshold-moving is for when the prior changes. Conflating the two means retraining quarterly to fix what a calibration review fixes weekly.

Plain version: the model ranks; the threshold decides what to do with the ranking. You can disagree with the ranking, which is retraining and features, or you can disagree with the decision, which is the threshold. Most teams retrain when they are really just holding the wrong threshold.

What still fails

Novel attack patterns. The model is a statistical summary of past fraud. A genuinely new playbook has no support in the training data and will pass at high confidence. This is not a tuning problem; it is the boundary of supervised learning applied to adversaries who adapt.

Collusion rings. Individually ordinary accounts behaving in a coordinated ring look unremarkable per account while being anomalous jointly. Single-transaction scoring is blind to this by construction. You need graph features and ring-level review, which is a different system, not a patch on this one.

Legitimate-but-unusual behavior. Travel, a new phone, a one-off large purchase at 2am. The model cannot know these are you. Step-up verification, not silent blocks, is the right action for the uncertain middle, and it preserves the customer relationship while still checking the risky moment.

Drift monitoring is mandatory, not optional. Watch the alert rate, the precision on sampled manual review, and the score distribution over time. A silent base-rate shift or a new attack vector shows up there first. If nobody owns the monitoring, the model is decorative, not defensive.

Honest limits

The measured 88% detection rate means 12% of fraud gets through. That is the honest state of supervised fraud modeling at this base rate, and anyone selling you solved is selling you a demo. The under 100ms inference budget is the easy part; scoring a feature vector is cheap. The hard part is labels: fraud labels arrive late, partial and noisy, and every number in this essay inherits that noise. Systems that tell you otherwise are hiding their label pipeline.

If your fraud project quotes accuracy, ask for the confusion matrix at the chosen operating point, the false positives per day at your volume, and the drift monitoring plan. If those three answers are not ready, the model is not ready. The full write-up and the source are linked above; I would rather you read the limitations section than any marketing paragraph, including this one.