From traces to measures: Large language models as a tool for psychological measurement from text
Year: 2024
Type: preprint
Abstract: Large language models are increasingly being used to label or rate psychological features in text data. This approach helps address one of the limiting factors of digital trace data - their lack of an inherent target of measurement. However, this approach is also a form of psychological measurement (using observable variables to quantify a hypothetical latent construct). As such, these ratings are subject to the same psychometric considerations of reliability and validity as more standard psychological measures. Here we present a workflow for developing and evaluating large language model based measures of psychological features which incorporate these considerations. We also provide an example, attempting to measure the previously established constructs of attitude certainty, importance and moralization from text. Using a pool of prompts adapted from existing measurement instruments, we find they have good levels of internal consistency but only partially meet validity criteria.
Source: arXiv (Cornell University)
Language: English
Topic: Mental Health via Writing
Subfield: Social Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W4396913679