Assessing How Stimulus Quality Shapes Evaluations of LLM Syntactic Competence: A Focus on Parasitic Gaps
Year: 2025
Type: preprint
Abstract: The Argument from the Poverty of the Stimulus (APS) remains a key focus in linguistics and cognitive science, yet recent research using Large Language Models (LLMs) to investigate this argument has shown inconsistent outcomes when testing various syntactic phenomena. This paper explores the hypothesis that features of the stimuli used in such studies-including lexical ambiguities and structural complexities-may interfere with LLM performance, thereby skewing evaluation results. We developed an approach to re-evaluate LLM syntactic prediction ability, with a specific focus on GPT-2 and its handling of parasitic gaps (PGs). This approach involves two key steps: 1) establishing performance baselines using both unfiltered and filtered versions of previously employed stimuli, and 2) creating a new, refined dataset with the help of a stateof-the-art (SOTA) generative LLM (Gemini 2.5 Pro Preview). The refined dataset is guided by linguistically informed templates designed to eliminate the identified confounds. Our initial results reveal that GPT-2 exhibits significantly improved performance on this refined PG stimulus set compared to the baseline datasets. This finding suggests that stimulus quality plays a crucial role in surprisal-based assessments of LLM syntactic competence, with important implications for both the broader APS debate and the design of future LLM language performance evaluations.
Author: Naranhvwar Tvlg
Institution: Mongolian University of Science and Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4415123488
Cites:
Cited by:
Related to: