Year: 2026
Type: conference-paper
Abstract: Estimating question difficulty is a critical component in evaluating and improving large language models (LLMs) for question answering (QA).Existing approaches often rely on readability formulas, retrieval-based signals, or popularity statistics, which may not fully capture the reasoning challenges posed to modern LLMs.In this paper, we introduce Q-DAPS (Question Difficulty based on Answer Plausibility Scores) method, a novel approach that estimates question difficulty by computing the entropy of plausibility scores over candidate answers.We systematically evaluate Q-DAPS across four prominent QA datasets-TriviaQA, NQ, MuSiQue, and QASC-demonstrating that it consistently outperforms baselines.Moreover, Q-DAPS shows strong robustness across hyperparameter variations and question types.Extensive ablation studies further show that Q-DAPS remains robust across different plausibility estimation paradigms, model sizes, and realistic settings.Human evaluations further confirm strong alignment between Q-DAPS 's difficulty estimates and human judgments of question difficulty.Overall, Q-DAPS provides an interpretable, scalable, and bias-resilient approach to question difficulty estimation in modern QA systems.
Authors: Jamshid Mozafari, Bhawna Piryani, Adam Jatowt
Institution: Universität Innsbruck
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: Universität Innsbruck
OpenAlex ID: W7166818084