The Architecture of a Multilayer Perceptron for Actor-Critic Algorithm with Energy-based Policy
Year: 2015
Type: preprint
Abstract: Learning and acting in a high dimensional state-action space are one of the central problems in the reinforcement learning (RL) communities. The recent development of model-free reinforcement learning algorithms based on an energy-based model have been shown to be effective in such domains. However, since the previous algorithms used neural networks with stochastic hidden units, these algorithms required Monte Carlo sampling methods for the action selections. In this paper, we investigate actor-critic algorithms based on the energy-based model. And we especially use neural networks with deterministic hidden units as an actor. Introducing the deterministic hidden units, we show that the gradient of the objective function is proportional to the gradient of the energy function. Then, we show the relationships between the energy-based approach and the conventional policy gradient algorithms by introducing the specific energy function. We reveal that the specificity of the RL paradigm severely disturbs learning of an actor when we use a multilayer perceptron as the representation of the policy. We therefore introduce " twin net " architecture. Finally, we empirically show the effectiveness of this architecture in several discrete domains.
Source: HAL
Author: Naoto Yoshida
Institution: Tohoku University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2901843362