Improving Cyber Threat Intelligence Analysis Using Supervised Machine Learning Models: A Comprehensive Statistical Evaluation
Year: 2026
Type: conference-paper
Abstract: Malware detection benchmarks have reported nearperfect accuracy on datasets like CIC-MalMem-2022, creating an illusion of a solved problem. This paper critically examines this narrative by introducing two novel quantitative metrics for dataset quality assessment: the Feature Redundancy Index (FRI) and the Learning Efficiency Score (LES). Through rigorous evaluation of four supervised learning models-XGBoost, Random Forest, Support Vector Machine, and Dense Neural Network-using stratified 10-fold cross-validation on the real MalMem2022 dataset, our analysis reveals that the benchmark's high performance is attributable to its inherent simplicity rather than algorithmic sophistication. Our variance analysis suggests that dataset selection may exert substantially greater influence on reported performance than model selection within the evaluated experimental setting. Furthermore, a LES of 1.001 indicates that 30% of training data is sufficient to achieve maximum accuracy, confirming the dataset lacks sufficient complexity. These findings fundamentally shift the research focus from incremental model improvements to the development of more challenging and representative benchmarks.
Authors: Noura Mohammed, Eid Emary, Mahmoud AttaAllah
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7214611348