Work

Using Large Language Models to Identify Suicide Risk Factors in Pediatric Emergency Department Chart Notes

Year: 2026

Type: article

Abstract: Objective Electronic health records (EHR) have been used increasingly to understand mental health and suicide risk, yet work is relatively limited in young children. Structured EHR data (e.g., age, diagnosis) are easily accessible but may provide a limited depth of information. Unstructured clinical notes hold a wealth of information on clinical risk factors but require more intensive processing. Large language models (LLMs) are a rapidly advancing technology that may be a scalable and efficient way to analyze free-text clinical notes. We provide examples and sample code as a practical guide for using LLMs to identify the presence of known risk factors for suicide in EHR from pre-teen children in the emergency department (ED). Method EHR for 920 patients 8 to 12 years of age (N = 1,310 visits) with suicide thoughts and behaviors, behavioral health concerns, and/or a psychiatric consultation in the pediatric ED of a metropolitan hospital were reviewed from 2020 to 2023. A secure, HIPAA-compliant ChatGPT 4o instance was used to identify any descriptions in clinician free-text notes of three exemplar risk factors relevant in prior work: (1) prior hospitalization and ED visits, (2) sleep problems, and (3) sexual orientation and gender identity. This was compared to consensus human coding from prior work and ‘regular expression' keyword-matching using specificity, sensitivity/recall, precision, and F1 metrics. Results LLM prompts could efficiently and accurately identify the presence of each of three risk factors in ED EHR yielding high model performance metrics: hospitalization (F1 = 96%), sleep (F1 = 90%), and LGBTQ+ identity (F1 = 96%). The LLM approach was more effective than the keyword-matching, which was prone to false positives around hospitalization (specificity = 64%) and false negatives around sleep (sensitivity = 76%). LGBTQ+ identity may be best captured by the LLM or a combination of both approaches (F1 = 97%). Conclusion LLMs are being increasingly used to advance data analytics in research and health care. We provide an exemplar approach to prompt engineering to extract suicide risk factors from free-text clinical notes for pediatric ED patients that is scalable and efficient. This can help integrate data from free-text notes with more structured data, like symptom checklists. Models will change over time and prompts may need targeting with risk factor description or prompt-based (K-shot) training examples to match the nuances of the patient population and EHR system. We aim for this to guide future work and the development of clinical decision aids.

Source: JAACAP Open

DOI: 10.1016/j.jaacop.2026.09.002