Work

Information-theoretic quantification of high-order feature effects in classification problems

Year: 2025

Type: article

Abstract: Understanding the contribution of individual features in predictive models remains a central goal in interpretable machine learning, and while many model-agnostic methods exist to estimate feature importance, they often fall short in capturing high-order interactions and disentangling overlapping contributions. In this work, we present an information-theoretic extension of the High-order interactions for Feature importance (Hi-Fi) method, leveraging Conditional Mutual Information (CMI) estimated via a k-Nearest Neighbor (kNN) approach working on mixed discrete and continuous random variables. Our framework decomposes feature contributions into unique, synergistic, and redundant components, offering a richer, model-free understanding of their predictive roles. We validate the method using synthetic datasets with known Gaussian structures, where ground truth interaction patterns are numerically derived, and further test it on non-Gaussian and real-world gene expression data from TCGA-BRCA. Results indicate that the proposed estimator accurately recovers theoretical and expected findings, providing a potential use case for developing feature selection algorithms or model development based on the interaction analysis.

Source: Chaos Solitons & Fractals

DOI: 10.1016/j.chaos.2025.117724

Authors: Ivan Lazic, Chiara Barà, Marta Iovino, Sebastiano Stramaglia, Karolina Kasas-Lazetic, Nikša Jakovljević, Luca Faes

Institutions: University of Novi Sad, University of Palermo, Istituto Nazionale di Fisica Nucleare, Sezione di Bari, University of Bari Aldo Moro

Language: English

Topic: Explainable Artificial Intelligence (XAI)

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $3,970

Funders: Ministero dell'Università e della Ricerca, Ministero dell'Istruzione e del Merito