Year: 2026
Type: conference-paper
Abstract: This paper introduces Vinclat, a novel evaluation dataset for Catalan carefully designed to assess the reasoning capabilities and cultural knowledge of LLMs.It comprises 1,000 highquality instances, meticulously crafted and reviewed by human annotators.Each instance presents a complex riddle that requires a twostep reasoning process involving inferential and abductive reasoning, along with other cognitive skills such as lexical retrieval, paraphrasing, flexibility in interpretation, pattern recognition, and associative thinking.Given four independent clues, models should infer intermediate concepts which, despite being seemingly unrelated, can be creatively connected to reach a final solution.The task targets a unique blend of capabilities, distinguishing it from existing NLP benchmarks.Our evaluation of state-ofthe-art models reveals that these still fall significantly short of human-level reasoning, although scaling trends suggest that the performance gap may narrow over time.This indicates that Vinclat provides a robust and long-term challenge, resisting the rapid saturation that is commonly observed in many existing evaluation datasets.
Institution: Barcelona Supercomputing Center
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Funder: Generalitat de Catalunya
OpenAlex ID: W7140079982