Year: 2025
Type: article
Abstract: Deep neural networks are often susceptible to overfitting, necessitating effective regularization techniques. This paper introduces Spectral Adaptive Dropout, a novel frequency-based regularization technique that dynamically adjusts dropout rates based on the spectral characteristics of network gradients. The proposed approach addresses the limitations of traditional dropout methods by adaptively targeting high-frequency components that typically contribute to overfitting while preserving essential low-frequency information. Through extensive experimentation on character-level language modeling tasks, the study demonstrates that the method achieves a 1.10% improvement in validation loss while maintaining competitive inference speeds. Thise research explores several implementations including FFT-based analysis, wavelet decomposition, and per-attention-head adaptation, culminating in an optimized approach that balances computational efficiency with regularization effectiveness. Our results highlight the significant potential of incorporating frequency-domain information into regularization strategies for deep neural networks.
Source: Information
DOI: 10.3390/info16060475
Authors: Zhigao Huang, Musheng Chen, Shiyan Zheng
Institution: Quanzhou Normal University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,930
OpenAlex ID: W4411102447