Toward a Validity Argument for Automated Assessment from Transcript and Video Data
Year: 2026
Type: preprint
Abstract: Automated assessment of psychological constructs from interview transcripts and video data has seen rapid technical development, yet the field lacks a systematic psychometric validity framework for interpreting scores from such measures. This paper applies Kane's interpretive/use argument structure alongside the five sources of validity evidence specified in the \textit{Standards for Educational and Psychological Testing} to automated transcript- and video-based assessment, using clinical assessment of depression as the primary illustrative case. We distinguish two design approaches --- item-based scoring, in which large language models (LLMs) rate explicit, theoretically derived items, and end-to-end embedding or classification approaches --- and examine how one could accrue validity evidence for each across the five sources. We argue that the field has concentrated almost exclusively on evidence based on relations to other variables (convergent validity against self-report criteria) while leaving response process and consequential evidence largely unaddressed. Further, we argue that construct validity evidence is often dismissed too quickly, especially for end-to-end approaches. Both evidence sources require fundamental reconceptualization for automated behavioral measures, and we propose reinterpretations suited to this context. Although the analysis centers on depression, the framework applies broadly to any construct assessed from passively collected video or transcript data. Implications for the design of future validation studies are discussed.
Source: PsyArXiv (OSF Preprints)
Author: James Soland
Institution: University of Virginia
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7214622273