Work

The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models

Year: 2023

Type: conference-paper

Abstract: Large Language Models (LLMs) make natural interfaces to factual knowledge, but their usefulness is limited by their tendency to deliver inconsistent answers to semantically equivalent questions.For example, a model might predict both "Anne Redpath passed away in Edinburgh."and "Anne Redpath's life ended in London."In this work, we identify potential causes of inconsistency and evaluate the effectiveness of two mitigation strategies: up-scaling and augmenting the LM with a retrieval corpus.Our results on the LLaMA and Atlas models show that both strategies reduce inconsistency while retrieval augmentation is considerably more efficient.We further consider and disentangle the consistency contributions of different components of Atlas.For all LMs evaluated we find that syntactical form and other evaluation task artifacts impact consistency.Taken together, our results provide a better understanding of the factors affecting the factual consistency of language models.

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/v1/2023.emnlp-main.332

Authors: Lovisa Hagström, Denitsa Saynova, Tobias Norlund, Moa Johansson, Richard Johansson

Institutions: Chalmers University of Technology, University of Gothenburg

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Knut och Alice Wallenbergs Stiftelse, Vetenskapsrådet, Marcus och Amalia Wallenbergs minnesfond, National Supercomputer Centre, Linköpings Universitet

Awards: G7318745325, NAISS