Work

Syntax in Large Language Models

Year: 2026

Type: article

Abstract: Presentation of the content of this paper at Session AI, Language, Data I of the Leibniz MMS Days 2026. Abstract: This is a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models, reporting on 1,015 model results from a range of syntactic phenomena and interpretability methods. Our analysis shows that the state of the art presents a healthy variety of methods and data, but an over-focus on a single language (English), a single model (BERT), and phenomena that are easy to get at (like part of speech and agreement). Results also suggest that TLMs capture these form-oriented phenomena well, but show more variable and weaker performance on phenomena at the syntax-semantics interface, like binding or filler-gap dependencies. We provide recommendations for future work, in particular reporting complete data, better aligning theoretical constructs and methods across studies, increasing the use of mechanistic methods, and broadening the empirical scope regarding languages and linguistic phenomena.

Source: Zenodo (CERN European Organization for Nuclear Research)

DOI: 10.5281/zenodo.18983126

Author: Boleda, Gemma

Institutions: Institució Catalana de Recerca i Estudis Avançats, Universitat Pompeu Fabra

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W7135076871

FWCI:

Cites:

Cited by:

Related to: