HEFS-EXR: A Robust Hybrid Ensemble Feature Selection Algorithm
Year: 2026
Type: preprint
Abstract: As data volumes grow, the performance of predictive models can be adversely affected by the presence of largely irrelevant and redundant features, especially in high-dimensional, small-sample data, where the risk of overfitting becomes more pronounced. Feature selection, therefore, emerges as a critical component of the machine learning process, enabling improved predictive performance, stability, and feature quality while supporting explainability, particularly in ``high-stakes'' domains. Existing feature selection approaches are numerous; however, ensemble learning-based feature selection methods remain comparatively scarce, despite their potential to improve predictive performance, stability, and generalizability. Because no single method consistently balances predictive performance, stability, and feature quality across diverse problem settings, we developed the HEFS-EXR method, a two-stage hybrid ensemble feature selection algorithm that leverages the strengths of multiple filter and wrapper feature selection methods and automatically determines the feature subset size at each stage. We conducted experiments on seven synthetic, one semi-synthetic, and 22 real-world data sets, and our findings indicate that HEFS-EXR consistently produces compact feature subsets, achieving dimensionality reductions ranging from 14% to 93% across both synthetic, semi-synthetic, and real-world data from diverse domains, without compromising predictive performance and, in many cases, improving it. The results also demonstrated improved stability, highlighting the method's robustness. Status Current Status: The full manuscript is currently under review at a peer-reviewed journal. Dataset Reference The synthetic and semi-synthetic data described in this study are archived in the associated Zenodo record for the data article: A. Bikaki and I. Kakadiaris, “A Benchmark Suite for Evaluating Feature Selection Methods on Synthetic and Semi-Synthetic Data”. Zenodo, Sep. 22, 2026. doi: 10.5281/zenodo.22743095.
Authors: Bikaki, Athina, Kakadiaris, Ioannis
Institution: University of Houston
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7148921904
Cites:
Cited by:
Related to: