Work

PAIR: Prompt-Aware margIn Ranking for Counselor Reflection Scoring in Motivational Interviewing

Year: 2022

Type: conference-paper

Abstract: Reflections are a core verbal skill used by mental health counselors to express understanding and acknowledgement of the client's experience and concerns.In this paper, we propose a system for the automatic evaluation of counselor reflections.Specifically, our system takes as input one dialog turn containing a client prompt likely leading to a reflection and a counselor response to it, and outputs a numeric score indicating the quality of the reflection made by the counselor.We compile a dataset consisting of reflections portraying different levels of reflective listening skills, and propose Prompt-Aware margIn Ranking (PAIR), a novel framework for reflection scoring that contrasts positive and negative prompt and response pairs using adhoc multi-gap and prompt-aware margin ranking losses.Through empirical evaluations and deployment of our system in a real-life educational environment, we show that our scoring model outperforms several baselines on different metrics, and can be used to provide useful feedback to counseling trainees.

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/v1/2022.emnlp-main.11

Authors: Do June Min, Verónica Pérez-Rosas, Kenneth Resnicow, Rada Mihalcea

Institution: University of Michigan

Language: English

Topic: Speech and dialogue systems

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: John Templeton Foundation, University of Michigan, School of Public Health, University of Michigan

Award: Tracking and Understanding the Effects of Transformative Events in People’s Lives