How Much Have We Really Progressed? A Leakage-Free Re-Evaluation of API Call-Based Malware Family Classification on Mal-API-2019 and APIMDS
Year: 2026
Type: article
Abstract: Dynamic application programming interface (API) call sequences are widely used for malware family classification, yet the multiclass accuracies published on the Mal-API-2019 benchmark range from 46.6% to 99.6%, under protocols that are rarely comparable. We present a leakage-free re-evaluation on Mal-API-2019 (7107 sequences, eight families) and APIMDS (16,942 sequences, six classes), with shared partitions, five-fold cross-validation, macro-F1 and McNemar tests. An unweighted ensemble of four learners on call frequencies, TF-IDF n-grams and Skip-gram embeddings reaches 70.0% accuracy and 71.0% macro-F1 on Mal-API-2019, significantly outperforming a faithful re-implementation of the state-of-the-art EPCP ensemble (66.7% and 67.6%), and matches EPCP on APIMDS (91.6% and 79.9%). A single gradient-boosting model on n-grams is statistically indistinguishable from the ensemble, and with fixed configurations, deep sequential models remain behind on Mal-API-2019. The highest published results rely on one-vs-rest or binary metrics, or on protocols that can leak test information or are not fully specified. Duplicated sequences inflate all scores: on Mal-API-2019, about 12% of each test split is seen during training, adding 1 to 3 points without changing the ranking; on APIMDS, 45.4% of the samples are duplicates, and counting each distinct behaviour once lowers the scores to 87.8% and 74.0%. N-gram models are also robust to inserted blocks of junk calls. Our partitions and code will be released upon publication.
Source: Computers
Institution: Université Ibn-Tofail
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,172
OpenAlex ID: W7225663254
FWCI:
Cites:
Cited by:
Related to: