Impact of class balancing techniques or events per variable ratio on the conclusion of the software faults prediction models
Year: 2025
Type: article
Abstract: Software fault prediction (SFP) models are often trained on severely imbalanced data. Prior work also warns that low events‑per‑variable (EPV) ratios can yield unstable and biased estimates. To quantify the separate and joint impact of (i) class‑balancing techniques and (ii) EPV ratios on the accuracy and stability of SFP models trained on five real‑world, open‑source GitHub projects. We also examine whether classifier choice outweighs either factor. We extracted five projects from GitHub and built class‑level datasets. After feature selection (genetic algorithm), we evaluated five estimation techniques (PLS, CART, NB, RF, SVM) under a factorial design crossing EPV ∈ (1) with balancing ∈ (None, Up, Down, SMOTE, ROSE) [1]. AUC is used to measure performance. We report effect sizes and stability (across repeated runs) and profile computational cost. Baselines include (a) no balancing at each EPV level and (b) an additional “raw” baseline without EPV adjustment for the only dataset with non‑zero defects (Ceylon). (i) Balancing improves AUC in a minority of settings; SMOTE is the most reliable of the balancing methods, whereas ROSE is the least effective. (ii) EPV = 20 consistently yields higher accuracy and stability than EPV = 3 or 10. (iii) Classifier choice matters most: strong estimation models (RF/SVM) at low EPV often outperform weak estimation techniques (PLS/CART) at high EPV. For modern GitHub‑based SFP, increasing EPV and choosing an appropriate classifier have a greater impact than applying generic class‑balancing. We provide a reproducible protocol (data processing pipeline, seeds, and scripts) and outline an external validation plan on NASA, PROMISE, and Relink.
Authors: Asad Ali, Dost Muhammad, Mushtaq Ali, Fakhrud Din
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $3,190
OpenAlex ID: W4416564928