Year: 2021
Type: preprint
Abstract: In computer vision and natural language processing, innovations in model architecture that increase model capacity have reliably translated into gains in performance. In stark contrast with this trend, state-of-the-art reinforcement learning (RL) algorithms often use small MLPs, and gains in performance typically originate from algorithmic innovations. It is natural to hypothesize that small datasets in RL necessitate simple models to avoid overfitting; however, this hypothesis is untested. In this paper we investigate how RL agents are affected by exchanging the small MLPs with larger modern networks with skip connections and normalization, focusing specifically on actor-critic algorithms. We empirically verify that naively adopting such architectures leads to instabilities and poor performance, likely contributing to the popularity of simple models in practice. However, we show that dataset size is not the limiting factor, and instead argue that instability from taking gradients through the critic is the culprit. We demonstrate that spectral normalization (SN) can mitigate this issue and enable stable training with large modern architectures. After smoothing with SN, larger models yield significant performance improvements -- suggesting that more "easy" gains may be had by focusing on model architectures in addition to algorithmic innovations.
Source: arXiv (Cornell University)
Institution: Cornell University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G1523888516, G2850601287, RI: AF: Small: Collaborative Research: Differentially Private Learning: From Theory to Applications, G335903494, G4504108201, G4805816520, Collaborative Research: CompSustNet: Expanding the Horizons of Computational Sustainability, G6098521345, III: Small: Collaborative Research: Towards Interpretable Machine Learning, TRIPODS: Data Science for Improved Decision-Making: Learning in the Context of Uncertainty, Causality, Privacy, and Network Structures, G709468757, S&AS: INT: Inference, Reasoning, and Learning for Robust Autonomous Driving, G8876996369, G8929337198
OpenAlex ID: W3208909168