Work

Linguistically-Informed Evaluation of LLMs on Acceptability Judgments in a Forced-Choice Paradigm

Year: 2026

Type: conference-paper

Abstract: Evaluating the grammatical abilities of large language models (LLMs) is important for both NLP and linguistic theory.We investigate the ability of large language models (LLMs) to perform acceptability judgments in a forcedchoice paradigm.We evaluate a subset of LLMs on 150 minimal sentence pairs sampled from Linguistic Inquiry and categorized using BLiMP linguistic phenomena.Our results show that while LLMs approximate human judgments, performance varies across models and phenomenon types, with stronger alignment on morphosyntactic phenomena than on linguistically and semantically demanding phenomena.Prompting strategies have minimal impact.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2026.acl-srw.103

Authors: Ziyue Liu, Nils Reiter

Institution: University of Cologne

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Deutsche Forschungsgemeinschaft, Universität zu Köln

Awards: G452762234, SFB 1252: Prominenz in Sprache