Get started
Work

Dual-use ambiguity in static ransomware detection: false positive analysis and confidence-gated mitigation

Year: 2026
Type: article
Abstract: Static machine learning models are widely used for ransomware detection because Portable Executable (PE) features can be extracted before execution and often produce strong benchmark results. However, many PE-level indicators, including entropy, imports, API calls, resource metadata, and section-level attributes, are also present in legitimate encryption and compression utilities. This overlap creates a dual-use ambiguity that conventional benign–malicious benchmark testing may not reveal. This study evaluates five classifiers—Random Forest, XGBoost, LightGBM, Logistic Regression, and Support Vector Machine—using a 2,675-sample ransomware benchmark dataset and a separate 300-sample dual-use stress test containing benign encryption, compression, backup, password-management, cryptographic, secure-deletion, and file-protection tools. To eliminate data leakage, models are evaluated under a strict family-disjoint protocol across 40 ransomware family clusters. The results of the study suggest that under the family-disjoint evaluation, standalone XGBoost achieved 88.69% ransomware recall on previously unseen families but produced a 57.89% false-positive rate on the dual-use holdout, demonstrating that strong ransomware generalization does not necessarily translate into reliable discrimination of legitimate encryption-related software. The study then proceeds to develop a confidence-gated two-step approach, which employs an XGBoost first-stage classifier and an SVM second-stage classifier evaluated on a version-disjoint 209-sample dual-use holdout. The results demonstrate that the proposed approach decreases the dual-use false-positive rate to 35.41% on unseen ransomware families while maintaining 59.36% ransomware recall, outperforming the baseline of a standalone SVM with 37.32% false-positive rate. The results also illustrate the trade-off between the false-positive rates and ransomware recall of the confidence-thresholding approach compared to training a classifier on the test set directly and decision threshold optimization with score averaging. The score-averaging ensemble achieved a ransomware recall of 83.39% with a dual-use false-positive rate of 45.93%, while decision-threshold optimization for F1 score retained a higher ransomware recall of 86.22% but resulted in a 49.28% false-positive rate. These findings demonstrate that neither baseline adequately resolves the trade-off between ransomware detection and false-positive reduction, motivating the use of the proposed two-stage confidence-gated classification framework for more controlled handling of ambiguous dual-use samples.
DOI: 10.1038/s41598-026-67921-z
Language: English
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7204464606
FWCI:
Cites:
Cited by:
Related to: