HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering
Year: 2026
Type: conference-paper
Abstract: Large language models (LLMs) can generate fluent Arabic answers, yet factual errors remain difficult to detect, localize, explain, and verify.Existing hallucination benchmarks often provide only response-level labels, offering limited support for identifying the exact erroneous content, explaining why it is incorrect, or recovering the correct factual answer.We introduce HalluTruthQA, a fine-grained benchmark for hallucination evaluation in Arabic question answering.HalluTruthQA contains 2,400 expert-curated examples across four knowledge-intensive domains: Islamic knowledge, history, science, and geography.Each example includes an Arabic question, a model-generated answer, a verified reference answer, a binary hallucination label, and six candidate answers for factual verification.For hallucinated answers, the benchmark additionally provides character-level erroneous spans, human-written explanations, and macro-and micro-level hallucination types.We evaluate four open-source LLMs-ALLaM-7B, Falcon-H1R-7B, Qwen3-32B, and SILMA-in a zero-shot setting across four tasks: hallucination detection, spanlevel localization, factual verification, and explanation evaluation.The results show that these tasks measure complementary capabilities, with no single model achieving the best performance across all of them.The best observed scores are 0.880 Macro-F1 for hallucination detection, 0.516 F1-Sp for span localization, 0.852 LO-Score for factual verification, and 0.644 for explanation evaluation.Our findings highlight the importance of moving beyond binary hallucination detection toward fine-grained evaluation that also localizes, verifies, categorizes, and explains factual errors.The code, dataset, prompts, and evaluation scripts are available at https: //gitlab.com/nlpresearcher/HalluTruthQA.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7170180213