Year: 2025
Type: conference-paper
Abstract: Large language models (LLMs) have been wellresearched in various long-context tasks.However, the scarcity of long-context summarization datasets hinders progress in this area.To address this, we introduce CNNSum, a multiscale long-context summarization benchmark based on Chinese novels, featuring humandriven annotations across four subsets totaling 695 samples, with lengths ranging from 16k to 128k.We benchmark numerous LLMs and conduct detailed human assessments to summarize abnormal output types.Furthermore, we extensively explore how to improve longcontext summarization.In our study: (1) Advanced LLMs may generate much subjective commentary, leading to vague summaries.(2) Currently, long-context summarization mainly relies on memory ability.The advantages of Large LLMs are hard to utilize, thus small LLMs are more cost-effective.(3) Different prompt types paired with various version models may cause large performance gaps.In further fine-tuning, these can be mitigated, and the Base version models perform better.(4) LLMs with RoPE-base scaled exhibit strong extrapolation potential; using short-context data can significantly improve long-context summarization performance.However, further applying other interpolation methods requires careful selection.(5) CNNSum provides more reliable evaluation results than other benchmarks.We release CNNSum to advance future research.
Authors: Lingxiao Wei, He Yan, Lu Xiangju, Junmin Zhu, Jun Wang, Wei Zhang
Institutions: East China Normal University, Shanghai Innovation Institute
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412888441