Hot off the Press: FEAT-KD: Learning Concise Representations for Single and Multi-Target Regression via TabNet Knowledge Distillation
Year: 2026
Type: conference-paper
Abstract: Interpretable regression methods often face a trade-off between predictive performance and model transparency. Feature Engineering Automation Tool (FEAT) is attractive because it produces intrinsically interpretable models in the form of a weighted linear combination of concise symbolic features. We propose FEAT-KD, a knowledge distillation approach that converts a trained TabNet regressor into the same white-box model structure used by FEAT. FEAT-KD distills individual TabNet components into short symbolic features by leveraging TabNet's sparse feature masks together with exhaustive symbolic search over restricted subproblems. The resulting model preserves the interpretability of FEAT while benefiting from the representational power and optimization behavior of deep tabular learning. Across regression benchmarks drawn from prior FEAT and TabNet studies, FEAT-KD consistently improves on FEAT variants in predictive performance while producing compact symbolic models, and remains competitive with the original TabNet teacher despite being much smaller and more interpretable. We further show that the framework naturally extends to multi-target regression, where shared symbolic features improve the interpretability of the full predictive system. FEAT-KD provides a practical route for turning accurate black-box tabular models into concise symbolic white-box regressors.
Authors: Kei Sen Fong, Mehul Motani
Institution: National University of Singapore
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7202350098