Year: 2026
Type: conference-paper
Abstract: Large language models exhibit surprising sensitivity to the structure of the prompt, but the mechanisms underlying this sensitivity remain poorly understood.In this work, we conduct an in-depth investigation on a striking case: in multiple-choice question answering, placing context before the questions and options (CQO) outperforms the reverse order (QOC) by over 14%p, consistently over a wide range of models and datasets.Through systematic architectural analysis, we identify causal attention as the core mechanism: in QOC prompts, the causal mask prevents option tokens from attending to context, creating an information bottleneck where context becomes invisible to options.Some Cantonese don't like chili, so some southerners don't like chili.A.Some Cantonese love chili B.Some people who like peppers are (...) C.All Cantonese are southerners D.Some Cantonese like neither peppers (...) Which of the following can guarantee the above argument?Some Cantonese don't like chili, so some southerners don't like chili.A.Some Cantonese love chili B.Some people who like peppers are (...) C.All Cantonese are southerners D.Some Cantonese like neither peppers (...) Which of the following can guarantee the above argument?Context Question Option QOC 14.7 % Drop!
Authors: Hyunjong Ok, Jaeho Lee
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166801060