Work

Natural Language Processing to Automate Cerebrovascular Event Identification in Stroke Alerts

Year: 2025

Type: article

Abstract: BACKGROUND: Frequent false-positive stroke alerts can strain health resources. Machine learning models can predict stroke alert accuracy and potentially reduce this strain, but these models require time-consuming labeling of large data sets. Weak labeling can accelerate machine learning model development by assigning annotations based on expert-defined heuristic rules rather than manual review. We sought to label a large, unlabeled sample of stroke alerts according to a binary outcome (presence/absence of acute cerebrovascular disease) using weak labeling. METHODS: We developed a weak labeling heuristic ensemble consisting of 4 hierarchical tiers, each of which generated a binary label using custom labeling algorithms. Tier 1 used rule-based named-entity recognition to generate binary labels from brain radiology reports. Tier 2 aggregated Tier 1 outputs over a 48-hour window. Tier 3 determined labels using diagnosis codes from stroke alert hospital encounters. Tier 4 generated a "final" encounter-level label based on label output combinations of Tiers 2 and 3. In 3 separate samples of stroke alerts, we determined sensitivity, specificity, and F1 scores of Tiers 1, 2, 3, and 4 by comparing Tier outputs to manual chart review. RESULTS: We identified 16 512 stroke alert activations between 2011 and 2021. For Tier 1, performance metrics were based on an initial manual review of 300 neuroimaging reports, achieving a sensitivity of 0.84, specificity of 0.96, and an F1 of 0.87. Tier 2 incorporated 716 neuroimaging reports with a sensitivity of 0.93, specificity of 0.90, and an F1 of 0.91. Tiers 3 and 4 were validated against 250 encounters. Tier 3 achieved a sensitivity of 0.77, specificity of 0.89, and an F1 of 0.80. Tier 4 achieved a sensitivity of 0.92, specificity of 0.86, and an F1 of 0.87. CONCLUSIONS: We successfully labeled a large registry of stroke alerts using weak labeling. This framework can potentially be extended to other clinical data sets.

Source: Stroke Vascular and Interventional Neurology

DOI: 10.1161/svin.125.001846