Work

What is an “Abstract Reasoner”? Revisiting Experiments and Arguments about Large Language Models

Year: 2025

Type: conference-paper

Abstract: Recent work has argued that large language models (LLMs) are not "abstract reasoners", citing their poor zero-shot performance on a variety of challenging tasks as evidence.We revisit these experiments in order to add nuance to the claim.First, we show that while LLMs indeed perform poorly in a zero-shot setting, even tuning a small subset of parameters for input encoding can enable near-perfect performance.However, we also show that this finetuning does not necessarily transfer across datasets.We take this collection of empirical results as an invitation to (re-)open the discussion of what it means to be an "abstract reasoner", and why it matters whether LLMs fit the bill. 1

Source: Conference on Computational Natural Language Learning

DOI: 10.18653/v1/2025.conll-1.11

Authors: Tian Yun, Chen Sun, Ellie Pavlick

Institution: Brown University

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Brown University, Advanced Research Projects Agency

OpenAlex ID: W4412888986