Work

A fail-safe layered Guardrail for Emergency Detection in LLM Based Health Assistants

Year: 2026

Type: other

Abstract: Large language model (LLM) assistants that answer health questions face an asymmetric error cost: failing to recognise a medical emergency can cause serious harm, whereas an unnecessary escalation costs only inconvenience. We present the safety layer of HealthSense AI, a multi-agent health information assistant, and describe a three-layer guardrail in which a deterministic keyword router first classifies risk, an LLM classifier then makes every final decision, and the deterministic layer serves as a fail-safe floor whenever the LLM is unavailable. We evaluate the design on an 87-case test suite covering emergency must-block prompts, safe controls, output-side blocking, and adversarial paraphrases, including a 31-case independently written adversarial set. The final system achieves 100% emergency recall (25/25), 100% output-guardrail accuracy (13/13), and zero false alarms. We also report a failure found during development — an earlier deterministic whitelist that overfitted the initial test set — and the redesign it motivated. All code and test cases are public at github.com/smita-datascience/healthsense-ai.

Source: Zenodo (CERN European Organization for Nuclear Research)

DOI: 10.5281/zenodo.22296121

Author: Bagchi sanyal chaudhuri, smita

Institution: Indian Institute of Science Bangalore

Language: English

Topic: Adversarial Robustness in Machine Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

SDG: Good health and well-being

Open Access status: green

Funder: European Commission