Year: 2023
Type: preprint
Abstract: A prompt injection attack aims to inject malicious instruction/data into the input of an LLM-Integrated Application such that it produces results as an attacker desires. Existing works are limited to case studies. As a result, the literature lacks a systematic understanding of prompt injection attacks and their defenses. We aim to bridge the gap in this work. In particular, we propose a framework to formalize prompt injection attacks. Existing attacks are special cases in our framework. Moreover, based on our framework, we design a new attack by combining existing ones. Using our framework, we conduct a systematic evaluation on 5 prompt injection attacks and 10 defenses with 10 LLMs and 7 tasks. Our work provides a common benchmark for quantitatively evaluating future prompt injection attacks and defenses. To facilitate research on this topic, we make our platform public at https://github.com/liu00222/Open-Prompt-Injection.
Source: arXiv (Cornell University)
Authors: Liu, Yupei, Jia, Yuqi, Geng, Runpeng, Jia, Jinyuan, Gong, Neil Zhenqiang
Institutions: Pennsylvania State University, Duke University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G2996144953, CAREER: Graph-Based Security Analytics: New Algorithms, Robustness under Adversarial Settings, and Robustness Enhancements, SaTC: CORE: Medium: Collaborative: Towards Robust Machine Learning Systems, Collaborative Research: SaTC: CORE: Small: Securing Recommender Systems against Data Poisoning Attacks, Collaborative Research: SaTC: CORE: Medium: Towards Secure Federated Learning, AI Institute for Edge Computing Leveraging Next Generation Networks (Athena)
OpenAlex ID: W4387839052