Work

Language models align with human judgments on key grammatical constructions

Year: 2024

Type: article

Abstract: Do large language models (LLMs) make human-like linguistic generalizations? Dentella et al. (2023) ("DGL") prompt several LLMs ("Is the following sentence grammatically correct in English?") to elicit grammaticality judgments of 80 English sentences, concluding that LLMs demonstrate a "yes-response bias" and a "failure to distinguish grammatical from ungrammatical sentences". We re-evaluate LLM performance using well-established practices and find that DGL's data in fact provide evidence for just how well LLMs capture human behaviors. Models not only achieve high accuracy overall, but also capture fine-grained variation in human linguistic judgments.

Source: Proceedings of the National Academy of Sciences

DOI: 10.1073/pnas.2400917121

Authors: Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, Roger Levy

Institutions: Harvard University, The University of Texas at Austin, University of Wisconsin–Madison, Georgia Institute of Technology, Massachusetts Institute of Technology

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

OpenAlex ID: W4391591605

FWCI: 7.084

Cites: 7

Cited by: 48

Related to: 10