Passing the Wrong Test: Model Size and Discriminant Validity in LLM-Based Psychological Assessment
Year: 2026
Type: preprint
Abstract: Validation studies of zero-shot language-model scoring of psychological interviews often rely primarily on a single convergent correlation with a questionnaire to establish validity. However, a bivariate correlation cannot reveal whether a model lacks discriminant validity, agrees poorly in absolute terms, produces non-random missing scores, or scores inconsistently. We evaluated eight open-weight families (1–70B active parameters) and eight commercial frontier models. The models scored the same sets of clinical interviews (Study 1: suicide recovery, n = 149; Study 2: future self-continuity, n = 164) across 52 configurations. Convergent validity improves with model size in both studies. However, the standard open-weight models tested achieve only about three-quarters of frontier convergent validity, with 11-fold greater mean absolute bias, more format failures, and lower repeatability in Study 1. In Study 2, standard open models (≤ 27B) are as strongly correlated with a closely related construct (suicide recovery) as with the target (mean difference −0.014), although they correlate less strongly with a distal neighbor (life satisfaction). All six frontier generations correlate more strongly with the target than with either neighbor (mean difference from recovery +0.117). On a local workstation, Gemma 4 31B is within the frontier range on all discriminant validity indices, outperforming its similar-sized predecessor. Llama 3.3 70B shows no greater discriminant validity than Gemma 4 31B, suggesting that discriminant validity is a function of release, not parameter count alone. Based on these findings, we suggest a minimum validation standard across five domains.
Language: English
Topic: Mental Health via Writing
Subfield: Social Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7223226170