SycoQA: A Dataset for Evaluating Sycophantic Hallucinations in Large Language Models
Year: 2026
Type: conference-paper
Abstract: Sycophantic hallucinations refer to the tendency of large language models (LLMs) to generate hallucinated responses by excessively aligning with human preferences. Unlike conventional hallucinations caused by missing knowledge, sycophantic hallucinations are harder to prevent. Even when the model already contains the relevant knowledge and is given explicit context, it may still produce plausible but incorrect answers. Existing datasets for sycophantic hallucination typically lack diverse induction settings and broad evaluation across multiple domains. To address this gap, we introduce SycoQA to evaluate sycophantic hallucinations across a broader spectrum of capability dimensions. SycoQA consists of a core and an extension subset to evaluate sycophantic hallucinations under internal-knowledge-based and context-grounded settings, respectively. Specifically, the Core subset adopts multiple induction paradigms with progressively increasing intensity, while the Extension subset constructs data through contextual corruption to simulate the corresponding scenarios. We believe that SycoQA can facilitate further analysis of sycophantic behavior and support comprehensive evaluation of advanced downstream tasks, particularly multi-level induction-aware and context-robust sycophantic hallucination detection. The dataset is publicly available at https://github.com/hehebamei/SycoQA.
Institutions: Beijing Institute of Technology, Shenzhen MSU-BIT University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7225317014