Trait or Bias? Why item keying is indispensable when comparing language model responses across languages
Year: 2026
Type: preprint
Abstract: Cross-linguistic differences in large language model (LLM) questionnaire responses are increasingly interpreted as evidence of culturally patterned behaviour. Such inferences, however, assume that observed score differences reflect variation in the target construct rather than artefacts of measurement. The present study examines this assumption using three established Likert measures of social orientation administered in English and Chinese to four contemporary LLMs. Leveraging balanced-keying scales, we decompose observed cross-linguistic differences into a trait-related component and a response-bias component associated with item keying. We further test the role of item wording through keying manipulations that preserve substantive content while altering item direction. Across models and instruments, the observed language effect was found to comprise two separable components of comparable magnitude but opposite sign. The bias component was highly consistent, with English prompting producing greater agreement tendencies than Chinese prompting in every balanced-keying condition. Rewording experiments showed that the effect tracked item wording rather than substantive content, while analyses of forward-keyed instruments indicated that the two components were confounded in conventional scores. As a consequence, identical composite scores could arise from different combinations of trait-related variation and response bias. These findings do not challenge the existence of cross-linguistic variation in LLM responses. Rather, they challenge the interpretation of observed score differences as direct indicators of underlying cultural orientation. The results suggest that cross-linguistic differences in questionnaire responses reflect both construct-related variation and language-dependent response tendencies, and that balanced-keying designs are necessary to distinguish between them. More broadly, the study highlights the importance of measurement validity in cross-linguistic evaluations of LLMs and argues that prompt language should be treated as a substantive measurement variable rather than a neutral medium of expression.
Source: PsyArXiv (OSF Preprints)
Author: Qing Li
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7215848991