Work

Scaling up naturalistic narrative research: A reproducible large language model-based tool for evaluating contextual information gain

Year: 2026
Type: preprint
Abstract: Narratives are a universal part of natural human experience. As experimental stimuli, narratives allow scientists to study complex mental processes with direct relevance to everyday experience. These analyses typically require manual labeling of key event features. Such characterization of narrative stimuli are extremely labor-intensive and prone to subjectivity. Automated tools could reduce these difficulties and enable scaling to longer, varied stimuli by exposing the exact quantities computed. We address this by introducing a metric that uses open-weight LLMs to compute contextual information gain . Our metric correlates with human ratings of event importance and causal centrality, critically distinguishing the latter from semantic centrality, thus overcoming limitations of existing automated tools. The metric similarly captures shared coactivation in the default mode network, a network implicated in narrative comprehension. Taken together, these results validate an automated, empirically-grounded tool that researchers can use to identify behaviorally- and neurally-relevant computational quantities in naturalistic stimuli. Research Transparency Statement Conflicts of interest: None. Funding: This work was supported by NIH R01AG088306 and R01NS119468. Artificial intelligence: Artificial intelligence use was limited to the visual clarity of figures and supporting plots. No artificial intelligence was used for the generation of text in the paper. Ethics: All experimental procedures and analyses were in accordance with protocols approved by the Institutional Review Boards (IRB) of Johns Hopkins. Preregistration: None of the reported analyses were preregistered, all subject data was adapted from open source studies and cited accordingly.
Language: English
Open Access status: green
OpenAlex ID: W7234973848
Cites: 31
Cited by:
Related to: 10