Development of a machine learning ( ML ) model to predict 3‐month response to CGRP monoclonal antibodies in migraine: A pilot evaluation of physician‐ ML model interaction
Year: 2026
Type: article
Abstract: BACKGROUND: Predicting individual treatment responses to anti-calcitonin gene-related peptide monoclonal antibodies (CGRP-mAbs) in migraine remains challenging. OBJECTIVE: To develop a machine learning (ML) model for predicting 3-month treatment response to CGRP-mAbs and to evaluate its practical clinical utility in improving physicians' predictive performance. METHODS: This retrospective prognostic model development was combined with a prospective, multi-reader, multi-case crossover reader study. We used retrospective data from 228 patients with migraine treated with CGRP-mAbs at the headache outpatient clinic in Dokkyo Medical University from April 2022 to March 2023. The data were divided into training (n = 182) and test (n = 46) datasets. Phase 1: An ML prediction model was developed using the training dataset with Random Forest and five-fold nested cross-validation. The model predicted achievement of a ≥50% reduction in monthly migraine days at 3 months, based on 27 baseline clinical variables. We tested the model's performance on the test dataset. Phase 2: Using the test dataset, five headache specialists and three general neurologists predicted treatment response without the model and then with the model's assistance. Prediction performance was compared between conditions. RESULTS: Phase 1: The model's performance on the test dataset was as follows: accuracy of 76.1% (95% confidence interval [CI], 62.1%-86.1%), sensitivity (recall) of 87.0% (67.9%-95.5%), specificity of 65.2% (44.9%-81.2%), precision of 71.4% (52.9%-84.7%), F1-score of 78.4% (64.0%-89.4%), and kappa of 0.52 (0.27-0.74) on the test dataset. Phase 2: The eight participating physicians predicted the treatment effectiveness. Without the model, their performance metrics were median accuracy of 65.2% [interquartile range 61.4%-70.1%], mean sensitivity (recall) of 68.5% ± standard deviation 11.6%, specificity of 62.5% ± 18.4%, precision of 66.3% ± 10.3%, F1-score of 66.7% [64.5%-71.9%], and kappa of 0.30 [0.23-0.40]. With the model, these metrics were accuracy of 73.9% [72.8%-76.1%], sensitivity (recall) of 79.9% ± 13.1%, specificity of 64.7% ± 13.6%, precision of 70.2% ± 8.1%, F1-score of 76.6% [74.4%-76.9%], and kappa of 0.48 [0.46-0.52], respectively. The accuracy, sensitivity, F1-score, and kappa were statistically improved. In the switching analysis, which assessed how physicians' predictions changed after receiving ML model assistance, incorrect-to-correct changes occurred in 12.8% (47/368, 8 physicians × 46 cases) of the pooled evaluations, whereas correct-to-incorrect changes occurred in 6.0% (22/368). The overall net switching benefit was +6.8%, and incorrect-to-correct changes occurred significantly more often than correct-to-incorrect changes (p = 0.003). CONCLUSIONS: Although external validation is necessary, our findings provide preliminary evidence that ML-assisted prediction may support the prediction of CGRP-mAb response.
DOI: 10.1111/head.70233
Authors: Masahito Katsuki, Muneto Tatsumoto, Tomokazu Shimazu, Yasuhiko Matsumori, Marina Romozzi, Omid Kohandel Gargari, Alicia Gonzalez‐Martinez, Woo‐Seok Ha, Yutaro Fuse, Shiho Suzuki, Saro Kobayashi, Tomohiko Shiina, Mukuto Shioda, Hiroaki Fujita, Koichi Hirata, Yuya Kano, Ryosuke Fukazawa, Jackson T. S. Cheung, Siobhán O'Connor, Tomás Ward, Kieran Moran, Keisuke Suzuki
Institutions: Dublin City University, Insight SFI Research Centre for Data Analytics, Saitama Medical University, Nagaoka University of Technology, Canon (Japan), Dokkyo Medical University, Saitama Neuropsychiatric Institute, Università Cattolica del Sacro Cuore, Agostino Gemelli University Polyclinic, Istituti di Ricovero e Cura a Carattere Scientifico, Tehran University of Medical Sciences, Hospital Universitario de La Princesa, Instituto de Investigación Sanitaria del Hospital Universitario de La Princesa, Universidad Autónoma de Madrid, Yonsei University, Yonsei University Health System, Chiba University, Nagoya City University, University College London, National University of Ireland, Maynooth
Language: English
Subfield: Psychiatry and Mental health
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
Estimated APC paid: $4,150
OpenAlex ID: W7224102189