Work

Inherent Disagreements in Human Textual Inferences

Year: 2019

Type: article

Abstract: We analyze human’s disagreements about the validity of natural language inferences. We show that, very often, disagreements are not dismissible as annotation “noise”, but rather persist as we collect more ratings and as we vary the amount of context provided to raters. We further show that the type of uncertainty captured by current state-of-the-art models for natural language inference is not reflective of the type of uncertainty present in human disagreements. We discuss implications of our results in relation to the recognizing textual entailment (RTE)/natural language inference (NLI) task. We argue for a refined evaluation objective that requires models to explicitly capture the full distribution of plausible human judgments.

Source: Transactions of the Association for Computational Linguistics

DOI: 10.1162/tacl_a_00293

Authors: Ellie Pavlick, Tom Kwiatkowski

Institutions: Brown University, Google (United States)

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

OpenAlex ID: W2985347336

FWCI: 11.69

Cites: 72

Cited by: 213

Related to: 10