Year: 2025
Type: conference-paper
Abstract: Prompt injection attacks pose a critical threat to large language models (LLMs), enabling goal hijacking and data leakage.Prompt guard models, though effective in defense, suffer from over-defense-falsely flagging benign inputs as malicious due to trigger word bias.To address this issue, we introduce NotInject, an evaluation dataset that systematically measures over-defense across various prompt guard models.NotInject contains 339 benign samples enriched with trigger words common in prompt injection attacks, enabling fine-grained evaluation.Our results show that state-of-the-art models suffer from over-defense issues, with accuracy dropping close to random guessing levels (60%).To mitigate this, we propose PIGuard, a novel prompt guard model that incorporates a new training strategy, Mitigating Over-defense for Free (MOF), which significantly reduces the bias on trigger words.PIGuard demonstrates state-of-the-art performance on diverse benchmarks including Not-Inject, surpassing the existing best model by 30.4%, offering a robust and open-source solution for detecting prompt injection attacks.The code and datasets are released at https: //github.com/leolee99/PIGuard.
Authors: Hao Li, Xiaogeng Liu, Ning Zhang, Chaowei Xiao
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412889730