Mostly Grounded, Occasionally Risky: Expert Evaluation of LLM-Generated Supervisory Feedback in a Psychotherapy Training Simulator
Year: 2026
Type: conference-paper
Abstract: Automated feedback is increasingly cited as a key advantage of AI-based psychotherapy training, yet the clinical groundedness of LLMgenerated supervisory feedback remains unevaluated.We present an expert evaluation of supervisory feedback generated by PRACTICE, an LLM-powered open-ended psychotherapy training simulator, across 21 feedback instances from four novice trainees.Two clinical psychology experts independently coded 167 feedback propositions as Justified, Unjustified, or Unsure.Inter-rater reliability was near-perfect (raw agreement = 98.2%; κ = 0.902).Of the 167 propositions, 149 (89.2%) were rated Justified; however, 52.4% of feedback instances contained at least one non-justified proposition, and qualitative analysis identified three recurring failure types: unsupported clinical inference, referential grounding error, and pragmatic failure.In clinical training contexts, even low error rates carry ethical weight: unjustified feedback risks reinforcing inappropriate clinical behaviors in trainees that can be transferred to real practice.These findings provide an initial empirical basis for the responsible deployment of LLM-generated feedback in clinical training and call for traceable, expert-auditable feedback architectures.
Institutions: Universitat Oberta de Catalunya, Universitat de Girona
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
OpenAlex ID: W7166804539