Self-Tuning Deep Reinforcement Learning
Year: 2020
Type: preprint
Abstract: Reinforcement learning (RL) algorithms often require expensive manual orautomated hyperparameter searches in order to perform well on a new domain.
This need is particularly acute in modern deep RL architectures which often
incorporate many modules and multiple loss functions. In this paper, we take a
step towards addressing this issue by using metagradients (Xu et al., 2018) to
tune these hyperparameters via differentiable cross validation, whilst the
agent interacts with and learns from the environment. We present the
Self-Tuning Actor Critic (STAC) which uses this process to tune the
hyperparameters of the usual loss function of the IMPALA actor critic
agent(Espeholt et. al., 2018), to learn the hyperparameters that define
auxiliary loss functions, and to balance trade offs in off policy learning by
introducing and adapting the hyperparameters of a novel leaky V-trace operator.
The method is simple to use, sample efficient and does not require significant
increase in compute. Ablative studies show that the overall performance of STAC
improves as we adapt more hyperparameters. When applied to 57 games on the
Atari 2600 environment over 200 million frames our algorithm improves the
median human normalized score of the baseline from 243% to 364%.
Source: arXiv (Cornell University)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W3007031922