Work

Gradus et al. Respond to “Machine Learning and Suicide Prevention: New Directions”

Year: 2021

Type: article

Abstract: We appreciate the opportunity to provide further context for interpretation of our study of sex-specific predictors of nonfatal suicide attempts, which applied machine learning methods to data from Danish national registries (1). The main findings from our study are that psychiatric disorders and associated medications, certain social variables, and certain physical health diagnoses predict nonfatal suicide attempts. Bossarte et al. (2) review a list of methodologic considerations in studies such as ours, all of which we also carefully considered in designing and analyzing our study. Ultimately, they agree that these considerations do not meaningfully call into question our main findings. We respond below to three specific issues raised by Bossarte et al. that merit clarification or further discussion. First, the commentary asserts that we applied an uneven eligibility criterion to cases and the comparison subcohort regarding timing of Danish residence, and that this has implications for potential differences in the inclusion of immigrants in each group. The source population for the study is described in the first paragraph of the Methods section as persons who were “born or residing in Denmark on January 1, 1995” and thus, both cases and members of the comparison subcohort met this criterion for inclusion in the study. Further, balanced inclusion of immigrants was a specific design feature and strength of our study. In Table 1 (1), the proportion of immigrants among cases and the comparison subcohort is presented and shown to be essentially equivalent. The assertion in the commentary that different eligibility criteria were applied to cases and the comparison subcohort, or that the case group included immigrants while the comparison subcohort did not, is simply inaccurate and this is clear in the published paper. Second, the commentary writers compute the positive predictive value (PPV) of high-risk classification (top 5% of risk categorized by random forests) to be 0.4%. They rightly acknowledge that this seemingly disheartening PPV derives primarily from the low rate of suicidal behavior. As a thought experiment, imagine increasing the suicide attempt rate of 87/100,000 that Bossarte et al. (2) used in their computation to 10,000/100,000 (an epidemic of suicidal behavior—1 in 10 persons attempting suicide—never before witnessed). This dramatic increase would still only increase the PPV to 33%. Accordingly, we did not emphasize PPV in our manuscript for two major reasons. First, the low PPV of suicide prediction algorithms is well-established (3) but does not preclude clinical utility (4). Second, it is broadly understood by epidemiologists that even high-risk behaviors or circumstances are poorly predictive of rare outcomes. About 15% of current tobacco smokers will develop lung cancer over their lifetime (5) (equivalent to the PPV of lung cancer for high-risk categorization by smoking status), yet the public health community advocates for tobacco control in part on the basis of its lung cancer hazard despite this poor PPV. Additionally, the infection fatality ratio in the United States for persons infected with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is about 1.3% (6) (equivalent to the PPV of death from high-risk categorization by infection status, with important age and comorbidity patterns not shown), yet the country has spent more than a year and endless economic resources avoiding infections. It is well known to epidemiologists that many effective public health policies involve interventions to prevent risks that are low because, when these low risks act across a large population, the impact of the outcome is still quite important. Finally, Bossarte et al. (2) recommend that the field turn its attention away from using machine learning methods to predict who is at risk for suicidal behavior. It is time, they say, to use these methods toward targeting and evaluating interventions. This task is complex, and we are wary that the commentary is overly optimistic about the achievability of this goal, in part because the cited evidence base pertains to very different clinical settings from those typically found in suicide prevention. The commentary cites examples using trial emulation and machine learning methods to optimize treatment regimens and select patients for whom they are likely to be most effective. These cited examples refer to studies conducted in clinical settings in which data are routinely collected on clear indications for treatment, clear records of pharmaceutical treatments received, and for outcomes that are much more common and much more readily identified in administrative records than suicidal behavior. These methods might not be easily transportable to suicide or suicidal behavior. The population in need of care is not so easily identified, predictors of outcomes are not so strong (see above), interventions are not always recorded with detail that has exact correspondence to indications (e.g., different elements of various psychotherapies used in patients with high levels of psychiatric comorbidity), and the outcome is rare (see above). The commentary’s assertion that individual treatment rules can improve suicide prevention might eventually come to fruition, but it is unlikely to be realized soon because of these challenges. Despite this, we agree that research attempting to address these barriers related to improving targeted clinical interventions for suicidal behavior would be of value, but that does not diminish the value of the work that we have published and similar work. Our work is, to our knowledge, the first published study to demonstrate that nonfatal suicide attempts can be predicted in a full population (free from selection) in routinely collected real-world clinical data. This finding is an important advance over the preponderance of previous suicidal behavior prediction work that has relied on self-report data (likely inaccurately reported), highly selected (potentially biased) subsamples of populations, or both. Population-level prediction is one of many tasks machine learning will need to continue to accomplish to improve the evidence base to support prevention of suicide and suicidal behavior. For example, predictors of suicidal behavior are likely to change as the societal context in which they operate changes, leaving room for additional and ongoing population-level work like ours. The conclusion of the commentary by Bossarte et al. (2) implies that suicidal behavior risk prediction and improving targeted clinical intervention are mutually exclusive research enterprises in a strictly monotonic line of inquiry. We assert that the cumulative evidence nature of science requires continued improvement on previous work across domains of inquiry to ultimately make progress in preventing suicidal behavior. Author affiliations: Aarhus University Hospital, Aarhus, Denmark (Jaimie L. Gradus, Timothy L. Lash, Erzsébet Horváth-Puhó, Henrik T. Sørensen); Department of Psychiatry, Boston University School of Medicine, Boston, Massachusetts, United States (Jaimie L. Gradus, Amy E. Street); Department of Epidemiology, Boston University School of Public Health, Boston, Massachusetts, United States (Jaimie L. Gradus, Tammy Jiang); Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, Georgia, United States (Timothy L. Lash); Center for Anxiety and Related Disorders, Department of Psychological and Brain Sciences, Boston University, Boston, Massachusetts, United States (Anthony J. Rosellini); Department of Psychiatry, New York University School of Medicine, New York, New York, United States (Isaac Galatzer-Levy); and Department of Clinical Epidemiology, National Center for PTSD, VA Boston Healthcare System, Boston, Massachusetts, United States (Amy E. Street). This work was funded by the National Institute of Mental Health (grants R01MH109507 and 1R01MH110453-01A1). This work was also supported by the Lundbeck Foundation (grant R248-2017-521). Conflict of interest: none declared.

Source: American Journal of Epidemiology

DOI: 10.1093/aje/kwab113