Robust Networks: Neural Networks Robust to Quantization Noise and Analog Computation Noise Based on Natural Gradient
Year: 2019
Type: dissertation
Abstract: : Deep neural networks (DNNs) have had tremendous success in a variety of\n\nstatistical learning applications due to their vast expressive power. Most\n\napplications run DNNs on the cloud on parallelized architectures. There is a need\n\nfor for efficient DNN inference on edge with low precision hardware and analog\n\naccelerators. To make trained models more robust for this setting, quantization and\n\nanalog compute noise are modeled as weight space perturbations to DNNs and an\n\ninformation theoretic regularization scheme is used to penalize the KL-divergence\n\nbetween perturbed and unperturbed models. This regularizer has similarities to\n\nboth natural gradient descent and knowledge distillation, but has the advantage of\n\nexplicitly promoting the network to and a broader minimum that is robust to\n\nweight space perturbations. In addition to the proposed regularization,\n\nKL-divergence is directly minimized using knowledge distillation. Initial validation\n\non FashionMNIST and CIFAR10 shows that the information theoretic regularizer\n\nand knowledge distillation outperform existing quantization schemes based on the\n\nstraight through estimator or L2 constrained quantization.
Author: Pradyumna Kadambi
Institution: Arizona State University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W3196071693
