Work

Revisiting Weight Initialization for Transfer Learning on Tabular Data: The Feature-Adaptive Variance Initialization (FAVI) Approach

Year: 2026

Type: article

Abstract: Transfer learning has advanced significantly in domains like computer vision and natural language processing, yet its application to tabular data remains challenging, with traditional models like XGBoost often outperforming deep learning approaches due to issues like variance instability and slow convergence. This study investigates the impact of weight initialization techniques on transfer learning efficacy during partial fine-tuning, hypothesizing that optimized methods enhance (i) variance stability (consistent activation and gradient magnitudes), (ii) convergence speed (faster loss reduction), and (iii) generalization (improved out-of-distribution accuracy). Established techniques including Xavier/Glorot, He/Kaiming, Orthogonal, and fan-averaged initializations are evaluated on modern foundation models like TabuLa-8B, using benchmarks such as OpenML and CC-18 (72 datasets). Additionally, Feature-Adaptive Variance Initialization (FAVI) is proposed, which adapts variances based on per-feature statistics, which is mathematically proven to preserve unit output variance. Empirical results demonstrate 15–20% improvements in convergence speed and 1.5–2.5% in generalization. Research contributions include a theoretical formulation of FAVI and insights advancing tabular data modeling.

Source: Mathematics

DOI: 10.3390/math14071174

Authors: Miroslav Nikolić, Danilo Nikolić, Miroslav Stefanović, Tina Bigović, Sara Koprivica, Darko Stefanović

Institutions: Union University, University of Novi Sad, Microsoft (United States)

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $3,137

Funder: University of Novi Sad