Quality assessment of DeepSeek-R1’s responses to guideline-related questions on hip osteoarthritis from AAOS
Year: 2025
Type: article
Abstract: BACKGROUND: This study aims to evaluate the quality of responses generated by DeepSeek to guideline-related questions on hip osteoarthritis from the American Academy of Orthopaedic Surgeons (AAOS). MATERIALS AND METHODS: Seventeen open-ended questions were derived from the 2023 AAOS Clinical Practice Guidelines for hip osteoarthritis. Four prompt types were applied to DeepSeek-V3 and R1. Responses were evaluated for accuracy, readability, and the mention of guidelines. Automatic evaluation and qualitative evaluation jointly evaluated the chain-of-thought of the R1 group. RESULTS: All the models showed superior response quality. Automatic and qualitative evaluations showed that the R1 group models performed better in the BERTScore and ROUGE-L metrics, indicating that their semantics and logic were closer to the reference text. This showed that the models in the R1 group could reflect chain-of-thought reasoning to a certain extent. The flesch-kincaid reading ease score of group R1 was significantly higher than that of group V3 (P<0.001). The flesch-kincaid grade level score also showed that the R1 group was significantly better than the V3 group (P<0.001). However, the R1 group still required reading levels at the level of a 12th-grade or a college freshman. In addition, the guideline mention rate was higher in group R1 than in group V3. CONCLUSION: DeepSeek-R1 demonstrated high accuracy in answering open-ended questions regarding hip osteoarthritis while effectively showcasing its chain-of-thought reasoning capabilities. Moreover, R1 based on background texts showed better readability, although its readability still required a 12th-grade or freshman reading level.
Source: International Journal of Surgery
Authors: Zhanpeng Guo, Yu Deng, Chunguang Liang, Haixiang Zhang, Daoyong Li, Mina Huang
Institutions: Jinzhou Medical University, Huzhou College
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
Estimated APC paid: $3,450
OpenAlex ID: W4416185929