Work

Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models via Diffusion Models

Year: 2024

Type: article

Abstract: Adversarial attacks, particularly targeted transfer-based attacks, can be used to assess the adversarial robustness of large visual-language models (VLMs), allowing for a more thorough examination of potential security flaws before deployment. However, previous transfer-based adversarial attacks incur high costs due to high iteration counts and complex method structure. Furthermore, due to the unnaturalness of adversarial semantics, the generated adversarial examples have low transferability. These issues limit the utility of existing methods for assessing robustness. To address these issues, we propose AdvDiffVLM, which uses diffusion models to generate natural, unrestricted and targeted adversarial examples via score matching. Specifically, AdvDiffVLM uses Adaptive Ensemble Gradient Estimation (AEGE) to modify the score during the diffusion model’s reverse generation process, ensuring that the produced adversarial examples have natural adversarial targeted semantics, which improves their transferability. Simultaneously, to improve the quality of adversarial examples, we use the GradCAM-guided Mask Generation (GCMG) to disperse adversarial semantics throughout the image rather than concentrating them in a single area. Finally, AdvDiffVLM embeds more target semantics into adversarial examples after multiple iterations. Experimental results show that our method generates adversarial examples 5x to 10x faster than state-of-the-art (SOTA) transfer-based adversarial attacks while maintaining higher quality adversarial examples. Furthermore, compared to previous transfer-based adversarial attacks, the adversarial examples generated by our method have better transferability. Notably, AdvDiffVLM can successfully attack a variety of commercial VLMs in a black-box environment, including GPT-4V. The code is available athttps://github.com/gq-max/AdvDiffVLM

Source: IEEE Transactions on Information Forensics and Security

DOI: 10.1109/tifs.2024.3518072

Authors: Qi Guo, Shanmin Pang, Xiaojun Jia, Yang Liu, Qing Guo

Institutions: Agency for Science, Technology and Research, Institute of High Performance Computing, Xi'an Jiaotong University, Nanyang Technological University

Language: English

Topic: Adversarial Robustness in Machine Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: National Natural Science Foundation of China

Awards: 面向语义关联的层次化视觉表示方法研究, G8252510314