Work

LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization

Year: 2026

Type: conference-paper

Abstract: Although context windows have expanded significantly in recent years, hallucinations in long-context summarization remain a challenge.Long novels are better suited than news or papers for researching these hallucinations, due to their intrinsic information and detailed descriptions of events and dialogues.However, current research lacks a multi-scale benchmark for hallucination detection in longcontext novel summarization and does not fully explore how hallucinations change as the context grows longer.In this study, we propose LongNovel, a multi-scale long-context bilingual (Chinese and English) novel benchmark for hallucination detection.This benchmark is constructed from 29 Chinese novels (ranging from 16k to 100k tokens) and chapter-level data from the BookSum dataset.We design 8 hallucination types and employ a combination of Multi-Model Arbitration and Entity-Referenced Hallucination Generation to ensure both data authenticity and a balanced distribution of hallucination categories.Furthermore, we manually revise the content in the test set to guarantee data reliability.Extensive experimental results demonstrate that LongNovel is a challenging benchmark.We release Long-Novel for future research.1

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/2026.emnlp-main.1423