Work

Meaning Beyond Truth Conditions: Evaluating Discourse Level Understanding via Anaphora Accessibility

Year: 2025

Type: conference-paper

Abstract: We present a hierarchy of natural language understanding abilities and argue for the importance of moving beyond assessments of understanding at the lexical and sentence levels to the discourse level.We propose the task of anaphora accessibility as a diagnostic for assessing discourse understanding, and to this end, present an evaluation dataset inspired by theoretical research in dynamic semantics.We evaluate human and LLM performance on our dataset and find that LLMs and humans align on some tasks and diverge on others.Such divergence can be explained by LLMs' reliance on specific lexical items during language comprehension, in contrast to human sensitivity to structural abstractions.

Source: Annual Meeting of the Association for Computational Linguistics (ACL)

DOI: 10.18653/v1/2025.acl-long.432

Authors: Xiaomeng Zhu, Zhenghao Zhou, Simon Charlow, Robert Frank

Institution: Yale University

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Yale University

OpenAlex ID: W4412945540