Product Echo State Networks: Time-Series Computation with Multiplicative Neurons
Year: 2015
Type: preprint
Abstract: Echo state networks (ESN) are efficient and accurate artificial neural systems for time series processing and learning. An ESN consists of a core of recurrent neural networks, called a reservoir, with a small number of tunable parameters to generate a high-dimensional representation of an input, and a readout layer which is easily trained using regression to produce a desired output from the reservoir states. Certain computational tasks involve real-time calculation of high-order time correlations, which requires nonlinear transformation either in the reservoir or the readout layer. Traditional ESN employs a reservoir with sigmoid function neurons. In contrast, some types of biological neurons obey response curves that can be described as a product unit rather than a sum and threshold. Inspired by this class of neurons, we introduce an ESN with a reservoir of product nodes for time series computation. We find that the capacity of a product ESN for learning nonlinear functions surpasses that of a standard ESN. In particular, product ESN resolves the limitation of standard ESN to learn symmetric (even) nonlinear functions of the input. On standard benchmarks for chaotic prediction tasks, a product ESN network maintains the performance of a standard nonlinear ESN while being more amenable to mathematical analysis. Our study provides evidence that such networks are powerful in highly nonlinear tasks owing to high-order statistics generated by the recurrent product node reservoir.
Source: arXiv (Cornell University)
Institutions: University of New Mexico, Google (United States)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W1482744127
