Work

An Analysis of Neural Language Modeling at Multiple Scales

Year: 2018
Type: preprint
Abstract: Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-level (WikiText-103) datasets, respectively. Results are obtained in only 12 hours (WikiText-103) to 2 days (enwik8) using a single modern GPU.
Language: English
Open Access status: green
OpenAlex ID: W2792376130
Cites: 35
Cited by: 203
Related to: 10