Year: 2025
Type: conference-paper
Abstract: While Large Language Models (LLMs) have demonstrated promising results on a range of reasoning benchmarks-particularly in formal logic, mathematical tasks, and Chain-of-Thought prompting-less is known about their capabilities in unconstrained natural language reasoning.Argumentative reasoning, a form of reasoning naturally expressed in language and central to everyday discourse, presents unique challenges for LLMs due to its reliance on context, implicit assumptions, and value judgments.This paper addresses a gap in the study of reasoning in LLMs by presenting the first largescale evaluation of their unconstrained natural language reasoning capabilities based on natural language argumentation.The paper offers three contributions: (i) the formalisation of a new strategy designed to evaluate argumentative reasoning in LLMs: argument-component selection; (ii) the creation of the Argument Reasoning Tasks (ART) dataset, a new benchmark for argument-component selection based on argument structures for natural language reasoning; and (iii) an extensive experimental analysis involving four different models, demonstrating the limitations of LLMs on natural language reasoning tasks.
Authors: Debela Gemechu, Ramon Ruiz-Dolz, Henrike Beyer, Chris Reed
Institution: University of Dundee
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412888691