Year: 2022
Type: article
Abstract: We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve relevant relations for a given context to improve text generation. Experiments on WikiText-103, WMT19, and enwik8 English datasets demonstrate that our approach produces a better language model in terms of perplexity and bits per character. We also show that relational memory improves coherence, is complementary to token-based memory, and enables causal interventions. Our model provides a simple yet effective way to combine an autoregressive language model and a knowledge graph for more coherent and logical generation.
DOI: 10.1162/tacl_a_00476
Authors: Qi Liu, Dani Yogatama, Phil Blunsom
Institutions: University of Oxford, Google DeepMind (United Kingdom)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
Funder: DeepMind
OpenAlex ID: W4226365369