Work

A hierarchy of recurrent networks for speech recognition

Year: 2009
Type: conference-paper
Abstract: Generative models for sequential data based on directed graphs of Restricted Boltzmann Machines (RBMs) are able to accurately model high dimensional se-quences as recently shown. In these models, temporal dependencies in the input are discovered by either buffering previous visible variables or by recurrent con-nections of the hidden variables. Here we propose a modification of these models, the Temporal Reservoir Machine (TRM). It utilizes a recurrent artificial neural network (ANN) for integrating information from the input over time. This infor-mation is then fed into a RBM at each time step. To avoid difficulties of recurrent network learning, the ANN remains untrained and hence can be thought of as a random feature extractor. Using the architecture of multi-layer RBMs (Deep Belief Networks), the TRMs can be used as a building block for complex hierar-chical models. This approach unifies RBM-based approaches for sequential data modeling and the Echo State Network, a powerful approach for black-box system identification. The TRM is tested on a spoken digits task under noisy conditions, and competitive performances compared to previous models are observed. 1
Language: English
Open Access status: green
OpenAlex ID: W13678974
FWCI: 1.376
Cites: 12
Cited by: 15
Related to: 10