Year: 2022
Type: preprint
Abstract: Certifying the robustness of model performance under bounded data distribution drifts has recently attracted intensive interest under the umbrella of distributional robustness. However, existing techniques either make strong assumptions on the model class and loss functions that can be certified, such as smoothness expressed via Lipschitz continuity of gradients, or require to solve complex optimization problems. As a result, the wider application of these techniques is currently limited by its scalability and flexibility -- these techniques often do not scale to large-scale datasets with modern deep neural networks or cannot handle loss functions which may be non-smooth such as the 0-1 loss. In this paper, we focus on the problem of certifying distributional robustness for blackbox models and bounded loss functions, and propose a novel certification framework based on the Hellinger distance. Our certification technique scales to ImageNet-scale datasets, complex models, and a diverse set of loss functions. We then focus on one specific application enabled by such scalability and flexibility, i.e., certifying out-of-domain generalization for large neural networks and loss functions such as accuracy and AUC. We experimentally validate our certification method on a number of datasets, ranging from ImageNet, where we provide the first non-vacuous certified out-of-domain generalization, to smaller classification tasks where we are able to compare with the state-of-the-art and show that our method performs considerably better.
Source: arXiv (Cornell University)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funders: National Science Foundation, Alfred P. Sloan Foundation, Cisco Systems, European Commission, Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, Eidgenössische Technische Hochschule Zürich, Innosuisse - Schweizerische Agentur für Innovationsförderung, Staatssekretariat für Bildung, Forschung und Innovation
Awards: AF: Small: Collaborative Research: Rigorous Approaches for Scalable Privacy-preserving Deep Learning, Governance and legal framework for managing artificial intelligence (AI), EASEML: Toward a More Accessible and Usable Machine Learning Platform for Non-expert Users, Machine-based Scoring of a Neuropsychological Test: The Rey-Osterrieth Complex Figure, Integrated Data Analysis Pipelines for Large-Scale Data Management, HPC, and Machine Learning, CAREER: DeepTrust: Enabling Robust Machine Learning with Exogenous Information, G993971353
OpenAlex ID: W4221143023