On Dropout, Overfitting, and Interaction Effects in Deep Neural Networks
Year: 2021
Type: preprint
Abstract: We examine Dropout through the perspective of interactions. Given N variables, there are O(N2) possible pairwise interactions, O(N3) possible 3-way interactions, i.e. O(Nk) possible interactions of k variables. Conversely, the probability of an interaction of k variables surviving Dropout at rate p is O((1−p)k). In this paper, we show that these rates cancel, and as a result, Dropout selectively regularizes against learning higher-order interactions. We prove this new perspective analytically for Input Dropout and empirically for Activation Dropout. This perspective on Dropout has several practical implications: (1) higher Dropout rates should be used when we need stronger regularization against spurious high-order interactions, (2) caution must be used when interpreting Dropout-based feature saliency measures, and (3) networks trained with Input Dropout are biased estimators, even with infinite data. We also compare Dropout to regularization via weight decay and early stopping and find that it is difficult to obtain the same regularization against high-order interactions with these methods.
Source: arXiv (Cornell University)
Authors: Ben Lengerich, Eric P. Xing, Rich Caruana
Institution: Carnegie Mellon University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W3039560274