Year: 2026
Type: conference-paper
Abstract: The rise of large language models (LLMs) has enabled us to seek answers to inherently debatable questions on LLM chatbots, necessitating a reliable way to evaluate their ability.However, traditional QA benchmarks assume fixed answers are inadequate for this purpose.To address this, we introduce DEBATEQA, a dataset of 2,941 debatable questions, each accompanied by multiple human-annotated partial answers that capture a variety of perspectives.We develop two metrics: Perspective Diversity, which evaluates the comprehensiveness of perspectives, and Dispute Awareness, which assesses if the LLM acknowledges the question's debatable nature.Experiments demonstrate that both metrics are aligned with human preferences and stable across different underlying models.Using DEBATEQA with two metrics, we assess 12 prevalent LLMs and retrievalaugmented generation methods.Our findings reveal that while LLMs generally excel at recognizing debatable issues, their ability to provide comprehensive answers encompassing diverse perspectives varies considerably.
Authors: Rongwu Xu, Xuan Qi, Zehan Qi, Wei Xu, Zhijiang Guo
Institutions: Tsinghua University, University of Cambridge
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140110514