Year: 2020
Type: dissertation
Abstract: Reinforcement learning (RL) is a powerful tool for developing \nintelligent agents, and the use of neural networks makes RL techniques more \nscalable to challenging real-world applications, from task-oriented dialogue \nsystems to autonomous driving. However, one of the major bottlenecks to the \nadoption of RL is efficiency, as it often takes many time steps to learn an \nacceptable policy. To address this problem, we investigate the idea of \nallowing the agent to ask for advice from a teacher. We formalize this \nconcept in a framework called ask-for-help RL, which entails augmenting a \nMarkov decision process with a teacher-query action that can be taken at a \nfixed cost in any state. In this task, the agent faces a dilemma between \nexploration, exploitation, and teacher-querying. To make this trade-off, we \npropose an action selection strategy that is rooted in the classical notion \nof value-of-information, and suggest a practical implementation that is based \non deep Q-learning. This algorithm, called VOE/Q, can jointly decide between \ntaking a particular environment action or querying the teacher, and is \nsensitive to the query cost. We then perform experiments in two domains: a \nmaze navigation task and the Atari game Freeway. When the teacher is \nexcluded, the algorithm shows substantial gains over many other exploration \nstrategies from the literature. With the teacher included, we again find that \nthe algorithm outperforms baselines. By taking advantage of the teacher, \nhigher cumulative reward can be achieved than with standard RL alone. \nTogether, our results point to a promising approach to both RL and \nask-for-help RL.
Source: UWSpace (University of Waterloo)
Author: Colin Vandenhof
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W3038202767