Can advanced large language models support radiology training? A performance assessment of DeepSeek R1
Year: 2025
Type: article
Abstract: Large language models (LLMs) are increasingly used in medical education, including radiology training. DeepSeek-R1, an open-access LLM, has gained attention for its performance and accessibility. This study assesses DeepSeek-R1's ability to answer radiology-related questions based on the European Training Curriculum for Radiology. Ninety questions were randomly selected from ten radiology subspecialties, covering Levels 1–3 of the curriculum. Three radiology residents (2nd-year, 4th-year, and subspecialty) reviewed responses for correctness, clarity, and safety using a 5-point scale. Additionally, 5 safety-related and 5 hallucination-based questions were included. Statistical analysis was conducted using RStudio, with the Kruskal-Wallis test assessing differences across groups. A weighted-Kappa test was used to assess inter- and intra-reader agreement. DeepSeek-R1 demonstrated high scores across correctness (4.1 ± 0.6), clarity (4.7 ± 0.6), and safety (4.8 ± 0.4). No significant differences were found across ESR levels or subspecialties. However, the 4th-year resident rated clarity significantly lower than the other residents (p = 0.0031). The model did not provide hallucinations and dangerous recommendations. DeepSeek-R1 shows promise as a supplementary educational tool in radiology training, offering accurate and clear responses while minimizing risks of misinformation. However, it remains essential to critically assess any answers from an LLM in case of inaccuracies. • This study evaluates DeepSeek-R1′s ability to answer radiology training questions based on the European Training Curriculum, rating the questions out of 5 in terms of correctness, clarity and safety. • The answers provided by DeepSeek-R1were rated as being highly correct (4.1/5), clear (4.7/5), and safe (4.8/5). • The model effectively identified misleading premises, avoided hallucinations and unsafe recommendations, whilst providing structured responses suitable for radiology trainees. • The model occasionally omitted information regarding the need to integrate clinical data into the diagnosis and sometimes failed to recognise the limits of radiology. • These findings suggest that LLMs have potential as educational tools in radiology, though further validation is needed for clinical applications.
Authors: Thomas Saliba, Jacopo Ferrari, Chiara Pozzessere, David Rotzinger, Guillaume Fahrni
Institution: Centre Hospitalier Universitaire Vaudois
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
SDG: Quality education
Open Access status: gold
Estimated APC paid: $2,900
OpenAlex ID: W4410487301