Large Language Models Systematically Favor Popular Options: Evidence and Mitigation Across MCQs
Year: 2026
Type: conference-paper
Abstract: Multiple-choice questions (MCQs) are a standard format for evaluating large language models (LLMs), yet the popularity of answer options can confound evaluation.Modern LLMs systematically prefer popular but incorrect options over less popular correct ones, a vulnerability we call popularity bias.This pattern aligns with confidence miscalibration: model confidence remains high even as accuracy collapses for popular options.To systematically isolate this phenomenon, we introduce PopMCQ, a benchmark with six controlled strategies that vary option popularity while keeping the correct answer fixed.In our most adversarial setting, where all distractors are more popular than the correct option, models choose popular but wrong answers 66% of the time.To mitigate this bias, we propose PopDebias, a lightweight inference-time correction that estimates and removes a popularity prior from model predictions.It requires no fine-tuning, is label-free at test time (using only a small calibration split for parameter fitting), and adds negligible computational cost.Experiments on 22 open-source LLMs (0.5B to 32B parameters) show consistent improvements, with accuracy gains up to 54.1 percentage points under strong popularity pressure 1 .-1.0 -0.50 0 0.25 Spearman = 0.5 (visual guide) DeepSeek-V2-Lite EntityQuestions -0.28 -0.90 -0.70 -0.52 0.23 -0.94 NQ -0.32 -0.90 -0.72 -0.55 0.15 -0.97 MuSiQue -0.23 -0.89 -0.66 -0.490.19 -0.95 Bias zone Alignment WebQuestions -0.34 -0.90 -0.68 -0.53 0.21 -0.98 S1 S2 S3 S4 S5 S6 -1.0 -0.50 0 0.25 SpearmanBase Pop.Trap Grad.
Institutions: Universität Innsbruck, Chungbuk National University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7204863744