Implementering av LLM-baserad rättning och feedback i sociala robotar för att förbättra det kliniska resonemanget hos läkarstudenter : En tvåstegsstudie som kombinerar kvalitativ och kvantitativ metod för att utvärdera LLM-baserad rättning och feedback i reumatologisk simulationsbaserad undervisning
Year: 2025
Type: dissertation
Abstract: Clinical reasoning (CR) is a cornerstone of medical competence, yet teaching and assessing it at scale remains challenging due to increasing student numbers and limited access to real-world patient interactions. While simulation-based learning tools, such as virtual patients (VPs), have improved exposure to clinical scenarios, they often lack timely, personalized feedback, an essential element for developing reflective and effective diagnostic skills. This thesis addresses this gap by exploring how Large Language Models (LLMs), embedded in social robots, can automate assessment and feedback in VP systems to support CR education. Existing VP platforms provide rich simulations but fail to deliver adaptive feedback at scale and human grading, while effective, is labour-intensive and inconsistent. Automated solutions using LLMs offer scalability but face scepticism regarding accuracy, pedagogical value, and trustworthiness. To solve this, a two-phase mixed-methods study was conducted in collaboration with the Karolinska Institute (KI). In Phase I, LLM-based assessment models (7 LLMs), feedback models (4 LLMs and 4 output formats), combined into a feedback system was developed to grade student VP CR performance (on 30 dialogues) and provide feedback. They were quantitatively evaluated against, and qualitatively evaluated by, expert rheumatologists. In Phase II, the feedback system was deployed and tested in-situ at a clinical course with third-year medical students (n=136). Its educational impact was evaluated through quantitative metrics including longitudinal VP CR performance (on 216 dialogues), end-of-course CR test performance (n=116), and a post-course questionnaire (n=67). Results show that LLMs can achieve grading accuracy comparable to rheumatology physicians and produce feedback that improves both VP dialogue performance and CR test outcomes. Students rated the feedback as credible and useful, highlighting its contribution to deeper learning. This work demonstrates that LLM-driven feedback systems can bridge the gap between simulation and real-world clinical practice, enabling more scalable, personalized, and pedagogically grounded CR education. These findings provide a benchmark for the performance of LLMs in assessing doctor–patient dialogues, and open avenues for broader integration of LLM tutors in medical education.
Authors: Ivegren, William, Schiött, Jonathan
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7038413603
