Work

Memory-augmented neural networks for better discourse understanding

Year: 2019
Type: dissertation
Abstract: Natural language understanding is a long-standing goal of artificial intelligence and a vast number of approaches have been proposed.While much effort has been focused on word-or sentence-level phenomena, language often consists of collections of structured sentences, expressing a coherent message.Such groups of sentences are commonly known as discourse.The multi-sentence nature of discourse requires consideration of contexts beyond the sentence, making the application of conventional models difficult, especially those with limited memory capacity.Another demanding aspect of discourse understanding is the relationships between entities.Particularly, the prevalence of long range dependencies between distant entities in discourse necessitates the ability to not only store such entities stably over long timescales but also have access to and update the states of them regardless of the distance between each mention.Despite the successful application of recurrent neural network (RNNs) to a range of tasks, RNN-based modelsin particular long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) -suffer from the locality bias of their memory components, highly influenced by immediately neighbouring elements, rendering them unstable over long timescales.In this thesis, we focus on memory-augmented neural networks, a class of neural architectures with a global memory and easy access, both in terms of storing and updating knowledge, and show these models are well suited to a wide range of discourse tasks.We broadly classify memory-augmented models into two categories: static and dynamic memory networks and limit our focus to two models, one from each category: memory networks (MEMN2Ns) (Weston et al., 2015, Sukhbaatar et al., 2015) and recurrent entity networks (ENTNETs) (Henaff et al., 2017).The three main research questions we I hereby declare that except where specific reference is made to the work of others, the contents of this dissertation are original and have not been submitted in whole or in part for consideration for any other degree or qualification in this, or any other university.This dissertation is my own work and contains nothing which is the outcome of work done in collaboration with others, except as specified in the text and Acknowledgements.This dissertation contains fewer than 100,000 words including appendices, bibliography, footnotes, tables and equations and has fewer than 150 figures.
Author: Fei Liu
Language: English
Open Access status: green
OpenAlex ID: W3023675412
Cites: 194
Cited by:
Related to: 10