Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models
Year: 2025
Type: conference-paper
Abstract: Long-context language models (LCLMs) have exhibited impressive capabilities in longcontext understanding tasks.Among these, long-context referencing-a crucial task that requires LCLMs to attribute items of interest to specific parts of long-context data-remains underexplored.To bridge this gap, this paper proposes Referencing Evaluation for Longcontext Language Models (Ref-Long), a novel benchmark designed to assess the long-context referencing capability of LCLMs.Specifically, Ref-Long requires LCLMs to identify the indexes of documents that reference a specific key, emphasizing contextual relationships between the key and the documents over simple retrieval.Based on the task design, we construct three subsets ranging from synthetic to realistic scenarios to form the Ref-Long benchmark.Experimental results of 13 LCLMs reveal significant shortcomings in long-context referencing, even among advanced models like GPT-4o.To further investigate these challenges, we conduct comprehensive analyses, including human evaluations, task format adjustments, fine-tuning experiments, and error analyses, leading to several key insights.Our data and code can be found in https://github. com
Authors: Junjie Wu, Gefei Gu, Yanan Zheng, Dit-Yan Yeung, Arman Cohan
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funders: Impact Fund, Hong Kong Government
OpenAlex ID: W4412944807