Year: 2023
Type: conference-paper
Abstract: Deep neural networks are susceptible to human imperceptible adversarial perturbations. One of the strongest defense mechanisms is Adversarial Training (AT). In this paper, we aim to address two predominant problems in AT. First, there is still little consensus on how to set hyperparameters with a performance guarantee for AT research, and customized settings impede a fair comparison between different model designs in AT research. Second, the robustly trained neural networks struggle to generalize well and suffer from tremendous overfitting. This paper focuses on the primary AT framework - Projected Gradient Descent Adversarial Training (PGD-AT). We approximate the dynamic of PGD-AT by a continuous-time Stochastic Differential Equation (SDE), and show that the diffusion term of this SDE determines the robust generalization. An immediate implication of this theoretical finding is that robust generalization is positively correlated with the ratio between learning rate and batch size. We further propose a novel approach, Diffusion Enhanced Adversarial Training (DEAT), to manipulate the diffusion term to improve robust generalization with virtually no extra computational burden. We theoretically show that DEAT obtains a tighter generalization bound than PGD-AT. Our empirical investigation is extensive and firmly attests that DEAT universally outperforms PGD-AT by a significant margin.
Authors: Jianhui Sun, Sanchit Sinha, Aidong Zhang
Institution: University of Virginia
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: National Science Foundation
Awards: III: Medium: Collaborative Research: Mining and Leveraging Knowledge Hypercubes for Complex Applications, Collaborative Research: CCRI: New: A Scalable Hardware and Software Environment Enabling Secure Multi-party Learning, Collaborative Research: PPoSS: LARGE: Co-designing Hardware, Software, and Algorithms to Enable Extreme-Scale Machine Learning Systems, III: Small: Multimodal Machine Learning for Data with Incomplete Modalities, III: Medium: Knowledge-Guided Meta Learning for Multi-Omics Survival Analysis
OpenAlex ID: W4379539742