Get started
Work

Applying Adaptive Sharpness-Aware Minimization to Improve Out-of-distribution Generalization

Year: 2023
Type: conference-paper
Abstract: Out-of-distribution (OoD) generalization in machine learning occurs when models trained on specific source domains struggle to generalize to unseen target domains due to variations in distributions, data collection conditions, and biases. These differences, known as domain shifts, can considerably impact model performance when facing domains not encountered during training. This circumstance arises partially due to the limitations of commonly used optimizers like SGD or ADAM, which lack the capability to preferentially converge towards optimal points characterized by high generalization capacity, commonly referred to as flat minima. Sharpness-Aware Minimization (SAM) emerges as a powerful tool for facilitating generalization in independent and identically distributed (i.i.d) data by aiming to find flat minima in the loss landscape that remain robust in the face of input sample perturbations. Among the variants of SAM, one notable variant is ASAM, an adaptive version designed to withstand parameter re-scaling and ensure robustness. In this study, our primary focus lies in the utilization of ASAM. Diverging from the conventional utilization of ASAM, which predominantly functions within the confines of i.i.d conditions, this paper applies ASAM in OoD data scenarios to evaluate its robustness. Our findings reveal that ASAM exhibits stable training behavior and superior generalization capabilities when compared to the Adam optimizer. Our experiments, conducted on the NICO dataset encompassing multiple domains, showcase ASAM’s remarkable performance, surpassing other state-of-the-art methods without the need for intricate data augmentation. ASAM achieves an accuracy rate of 87.25% (Animal) and 80.79% (Vehicle).
DOI: 10.1145/3628797.3629009
Language: English
Open Access status: bronze
OpenAlex ID: W4389366372
FWCI: 0.355
Cites: 15
Cited by: 2
Related to: 10