Determining the effect of conversational depth on safety-relevant classification performance in large language models: a preregistered benchmark using real psychotherapy transcripts and a licensed-clinician comparison
Year: 2026
Type: preprint
Abstract: Large language models are increasingly proposed for mental health support and psychotherapeutic dialogue, yet most evaluations of their safety-relevant performance rely on single- or few-turn interactions. Accumulating evidence suggests that clinically relevant model behavior depends on the length of the conversation, the content of the preceding turns, and the broader sociotechnical context in which the exchange occurs, with some of the most serious harms occurring at extremely long conversation lengths. We address this gap by quantitatively assessing how LLMs detect target statements containing suicidal ideation at pre-specified points in real therapy transcripts, relative to licensed clinicians performing the same task. To do so, 400 clinician-validated suicidal ideation (SI) and non-SI statements are inserted at fixed depths, defined by the number of preceding patient–therapist speaker turns, within real, full-length psychotherapy transcripts. Both LLMs and therapists then classify each inserted statement as SI or non-SI, allowing us to quantify how speaker-turn depth, model family, and parameter scale shape classification accuracy and how the human–model gap evolves across extended therapeutic dialogue. Seven LLM families are evaluated: open-source Qwen, LLaMA, and Gemma across a range of parameter scales (~0.2B to 70B), and proprietary ChatGPT, Gemini, Qwen, and Anthropic. Three synthetic control contexts (semantically inert filler, an adversarial many-shot jailbreak sequence, and a clinically priming psychoeducation dialogue) are included alongside the five therapy transcripts to verify that models behave as predicted under contexts with known properties. Eight licensed clinicians perform the identical classification task on a balanced rotation across transcripts, controls, and depths, enabling direct comparison of LLM and human performance. We hypothesize that (i) LLM classification performance degrades with conversation depth; (ii) larger and newer LLMs perform better than smaller and older ones; (iii) clinician performance does not degrade with depth; (iv) LLM sensitivity shifts in predictable directions under control conditions while clinician sensitivity remains stable. The primary outcome is F1 at each (model × conversation × depth) cell, analyzed via mixed-effects regression with family-specific models. We expect the study to characterize the size of the human–model performance gap across conversational depth, identify whether degradation reflects sensitivity loss, specificity loss, or both, and provide a preregistered benchmark for evaluating the clinical safety of LLMs in extended therapeutic dialogue.
Authors: Mark Kalinich, John Torous
Language: English
Topic: Mental Health via Writing
Subfield: Social Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7167641247