Year: 2026
Type: preprint
Abstract: Large language models (LLMs) serve as measurement instruments when researchers apply a fixed prompt to many texts or questions. A fixed prompt standardizes presentation, but incidental presentation features can push many responses in the same direction. We study one such feature, response-option order: when does its error survive aggregation, and which remedies remain effective when the instrument changes? Using responsibility and solution judgments on 150 European Union policy issues, we presented all 24 orders of four options twice, in four languages and 31 configurations of 21 open-weight models (1,771,200 responses). Comparing the two replications separates reproducible order effects from generation noise and isolates the part shared across items, which, like an interviewer effect in surveys, does not average out. In English, averaging 150 issues under one fixed order cancels order error about as well as averaging three unrelated issues, and conventional 95 percent intervals cover the order-averaged reference in only 49.0 and 27.1 percent of cases (medians). Drawing the order independently for each item requires no additional calls and reduces estimated mean squared error in all 246 replicated conditions. This design rule holds across languages, instructions, labels and temperatures. Fitted corrections depend on language, policy domain and task: out-of-domain correction often helps, but cross-task position corrections usually underperform local fits. Reasoning can increase residual error, and prespecified calibration gains do not repay pilot costs. We translate these results into a design and reporting workflow for LLM-based social measurement.
Source: Research Square
Institution: University of Debrecen
Language: English
Subfield: General Social Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7236712597
