Evaluation of the effectiveness of a medical decision support system for preliminary diagnosis in Moscow outpatient clinics
Year: 2025
Type: article
Study design: Observational Study
Abstract: The implementation of the medical decision support system (MDSS) in clinical practice requires careful monitoring to ensure patient safety and track the performance of artificial intelligence technologies.Goal: To assess the effectiveness of the TOP-3 MDSS in outpatient clinics of the Moscow Heath Care Department.Materials and methods: The Moscow Heath Care Department monitored the TOP-3 operation between October 1, 2020 and March 03, 2024 (n = 63,809,360 people). The Hit-3 metric was used to determine if the MDSS needs retraining. An additional study involving medical experts included a retrospective analysis of data from 3,000 patients and calculation of the diagnostic agreement rate among the MDSS, doctors and the experts.Results: The monitoring demonstrated mean Hit-3 of 63.5, 64.5 and 67.7 for MDSS ver. 1, 2, and 3, respectively. In cases where the clinicians disagreed with the MDSS (n = 2000), based on the patient complaints, experts agreed with the MDSS in 80.2% of cases, with clinicians in 11.5% of cases, and established a different diagnosis in 8.3% of cases. In cases where the clinicians’ and the MDSS’s conclusions matched (n = 1000), experts approved the diagnosis in 50.4% of cases, selected an alternative diagnosis suggested by the MDSS in 37.9% of cases, and established a different diagnosis for 11.7% of patients. Conclusion: The described monitoring methodology, supplemented by expert review, allowed for a comprehensive assessment of the MDSS to be implemented in the health care system. Based on the results of the TOP-3 effectiveness assessment, it was decided that the analyzed list of data from electronic health records should be expanded, which will be implemented in the upcoming version TOP-3+.
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
OpenAlex ID: W4415163485