Does Reasoning Give a Language Model a Personality? Within-Model Effects of Thinking on Big Five Trait Scores and Construct Validity.
Year: 2026
Type: preprint
Abstract: Many current language models generate a chain of "reasoning" tokens before answering, and many expose this as a toggle. Does turning reasoning on change how a model responds to Big Five personality questionnaires? This study uses a within-model design: the same model answers the same items with thinking off and with thinking on. Four public-domain IPIP Big Five inventories (the 20-item Mini-IPIP, the 50-item IPIP Factor Markers, the 100-item Big Five Aspect Scales, and the 120-item IPIP-NEO, 290 items in total) were administered to ten open-weight hybrid models with a reasoning toggle, one reasoning-native open model (OLMo 3, which cannot disable thinking), and one commercial model (Claude Haiku 4.5, administered through its provider's Batch API). The think-off baseline for the ten hybrids reuses direct-answer data from a prior study; that reuse is validated directly, as a fresh think-off run reproduces the prior domain means to within 0.07 points on the five-point scale, against reasoning effects 5 to 28 times larger. Two findings emerge. (1) Reasoning shifts reported trait scores substantially and in a consistent direction: models present as more emotionally stable (about +0.9 on a 5-point scale) and less extraverted (about -0.6 to -0.8). The two largest effects hold across open-weight and commercial models; the smaller conscientiousness and agreeableness effects do not generalize and are not claimed. Convergent validity rises under reasoning (mean convergent r 0.65 to 0.84) while heterotrait correlation is flat, so the convergent-discriminant gap widens because same-trait agreement rises, not because traits separate. (2) Reasoning does not create within-model coherence: treating repeated generations as respondents, internal-consistency reliability stays near zero with thinking off and on (median alpha near zero; zero of 200 hybrid and zero of 20 commercial model-by-domain cells reach 0.70), and the same holds for the reasoning-native model that cannot stop thinking. Reasoning changes what a model says about its personality without making that personality cohere as an individual-level property. This deposit contains the full manuscript (PDF, LaTeX source, and a readable Markdown version), a plain-language summary, the raw response databases, all administration and analysis code, the four instrument files, and every derived table. The analysis scripts regenerate every result deterministically. The software in the bundle is released under the MIT License; the deposit license (CC-BY-4.0) applies to the manuscript and data.
Author: Johnson, Trevor
Language: English
Subfield: Clinical Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7166295057
Cites:
Cited by:
Related to: