Out-of-Distribution Generalization with Maximal Invariant Predictor
Year: 2021
Type: preprint
Abstract: Out-of-Distribution (OOD) generalization is a problem of seeking the predictor function whose performance in the worst environment is optimal. This paper makes both theoretical and algorithmic contributions to the OOD problem. We consider a set of all invariant features conditioned to which the target variable and the environment variable becomes independent, and theoretically prove that one can seek an OOD optimal predictor by looking for the mutual-information maximizing feature amongst the invariant features. We establish this result as \textit{Maximal Invariant Predictor condition}. Our theoretical work is closely related to approaches like Invariant Risk Minimization and Invariant Rationalization. We also derive from our theory the \textit{Inter Gradient Alignment}(IGA) algorithm that uses a parametrization trick to conduct \textit{feature searching} and \textit{predictor training} at once. We develop an extension of the Colored-MNIST that can more accurately represent the pathological OOD situation than the original version, and demonstrate the superiority of IGA over previous methods on both the original and the extended version of Colored-MNIST.
Source: arXiv (Cornell University)
Authors: Masanori Koyama, Shoichiro Yamaguchi
Institution: Preferred Networks (Japan)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W3047419058
