Rehabilitation Quality Assessment Using Large Language Models: Algorithm Development and Validation Study (Preprint)
Year: 2026
Type: preprint
Abstract: BACKGROUND Strategy training is an evidence-based rehabilitation approach in which therapists guide stroke survivors to apply cognitive and metacognitive strategies to functional tasks with the goal of optimizing independence and recovery outcomes. Central to this intervention are therapist verbal cues, which are structured spoken prompts that support learning and strategy use. The type, frequency, and quality of these cues reflect how faithfully the intervention is being delivered. Systematic assessment of verbal cues is important for quality monitoring and therapist training; however, current assessment methods rely entirely on manual coding by trained human raters, making large-scale or real-time fidelity assessment impractical. OBJECTIVE This study evaluates large language models (LLMs) for automated analysis of therapist verbal cues in rehabilitation sessions and compares prompt-based and supervised fine-tuned LLM approaches against conventional natural language processing (NLP) approaches, such as Bidirectional Encoder Representations from Transformers (BERT), using both quantitative classification metrics and session-level qualitative assessment. METHODS Rehabilitation therapy session recordings from occupational therapy (OT), physical therapy (PT), and speech-language pathology (SLP) were transcribed and manually annotated to create a gold standard dataset of 3,833 therapist utterances labeled as directed, guided, or none. The dataset was split using a discipline-stratified design with a balanced training set and a naturally imbalanced test set. Models included a Bidirectional Encoder Representations from Transformers (BERT) baseline and LLM (GPT-5 mini, Qwen-2.5-7B, and Llama-3-8B. LLMs) evaluated using zero-shot, few-shot, and Low-Rank Adaptation (LoRA) fine-tuning. Performance was assessed using accuracy, precision, recall, and F1 score, with additional class-level evaluation and session-level fidelity assessment. RESULTS Fine-tuned LLMs achieved the highest performance, with Qwen-2.5-7B LoRA reaching 85.4% accuracy and a macro F1 score of 0.8638, outperforming BERT (77.1% accuracy; macro F1 0.7161) and prompt-based approaches. Class-level analysis showed that directed cues were most consistently identified, while guided cues and none categories were more challenging. Session-level quality assessment demonstrated that guided cue counts exceeded directed cue counts in all selected strategy training sessions, whereas pre-training sessions before therapist training showed higher or comparable directed cue counts. CONCLUSIONS Fine-tuned LLMs provide an effective approach for automated therapist cue classification and fidelity monitoring in rehabilitation, supporting future clinical implementation and evaluation across diverse settings.
Language: English
Subfield: Rehabilitation
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7169787366