Year: 2026
Type: conference-paper
Abstract: Benchmarks for long-form summaries (four or more sentences) generated by language models increasingly serve as gold-standard references for developing, evaluating, and comparing faithfulness-checking systems.As their influence grows, understanding the challenges of annotating faithfulness errors within long, discourse-rich summaries becomes critical.We revisit three benchmarks spanning diverse text types and contrasting annotation designs.Using a discourse-aware evaluation framework together with human auditing, we identify cases where benchmark labels may be unreliable.Manual verification shows that 3.4%-5.4% of sentence-level labels warrant revision due to discourse-level inconsistencies that standard annotation procedures overlook.We introduce a taxonomy of five recurring annotation error types, propose revised labels, and show that correcting these cases leads to meaningful shifts in system rankings.We conclude with recommendations for future annotation practices.
Source: Linguistic Annotation Workshop
Authors: Yang Zhong, Yang Janet Liu, Diane Litman
Institution: University of Pittsburgh
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166837127