Work

On the long-term learning ability of LSTM LMs

Year: 2021

Type: preprint

Abstract: We inspect the long-term learning ability of Long Short-Term Memory language models (LSTM LMs) by evaluating a contextual extension based on the Continuous Bag-of-Words (CBOW) model for both sentence- and discourse-level LSTM LMs and by analyzing its performance. We evaluate on text and speech. Sentence-level models using the long-term contextual module perform comparably to vanilla discourse-level LSTM LMs. On the other hand, the extension does not provide gains for discourse-level models. These findings indicate that discourse-level LSTM LMs already rely on contextual information to perform long-term learning.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2106.08927

Authors: Boes, Wim, Van Rompaey, Robbe, Verwimp, Lyan, Pelemans, Joris, Van hamme, Hugo, Wambacq, Patrick

Institutions: KU Leuven, Apple (Israel)

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Fonds Wetenschappelijk Onderzoek, Vlaamse regering

OpenAlex ID: W3127047550

Cites: 3

Cited by:

Related to: 10