FraudShield-EAC: What a Synthetic East African Mobile-Money Fraud Benchmark Can and Cannot Show
Year: 2026
Type: preprint
Abstract: A synthetic fraud benchmark is easy to build and hard to interpret: a high score may measure the generator rather than the model. We study this on FraudShield-EAC, a seeded, agent-based benchmark of East African digital payments (six channels, five countries, eight fraud scenarios) built to develop a fraud detection system. The benchmark is easy in a specific way: on the test period one engineered feature, a short-to-long-window velocity ratio, reaches an AUC of 0.879 on its own, so we report ranking metrics as margins over it. A calibrated XGBoost and LightGBM ensemble trained on all 792,162 training rows reaches an AUC of 0.975 (margin +0.096), but its recall at the specified flag threshold, 0.764, misses the target of 0.880, and more training rows did not close the gap in any run. At a fixed 30,000 rows, which rows were drawn moved recall at that threshold over a range of 0.107, against 0.032 for training on all rows; on AUC, volume mattered more. On a diagnostic model, ablations measure redundancy rather than importance (four disjoint feature groups each exceed AUC 0.80 alone), and leave-one-country-out and a novel variant held out in time are both null. The one variant specified in advance without burst structure, a reversal scam, is caught at 9.1% [4.2%, 18.4%] at a 1% false-positive budget, against 94.4% for the base scenarios. We distil the audit into a checklist for synthetic benchmarks. No fraud parameter is sourced, so every result describes the generator, not East African payments.
Author: Bayizere, Marius
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225913523
