Asymmetric Relational-Geometry Driven Universal Adversarial Perturbations for Vision-Language Models
Year: 2026
Type: conference-paper
Abstract: Although vision-language pre-trained (VLP) models have achieved remarkable success across multimodal tasks, they remain vulnerable to adversarial perturbations.Existing universal adversarial perturbation (UAP) methods in multimodal settings-whether generatorbased or optimization-based-often suffer from limited cross-model transferability, especially in black-box scenarios.We attribute this limitation to the prevalent use of symmetric or distribution-level objectives that overlook the asymmetric roles of image and text modalities and the relational nature of vision-language representations.To address this issue, we propose ARG-Attack, an optimization-based framework that learns universal perturbations under an asymmetric relational-geometry driven objective.Our method integrates three complementary components: a cosine-based loss that induces directional semantic drift in visual features, a center shift loss that geometrically regularizes adversarial embeddings toward a shared semantic center, and a relational polarity loss that explicitly disrupts image-text matching relationships.Together, these objectives enable effective cross-modal interaction without relying on model-specific training losses or probabilistic distribution matching.In addition, we adopt an adaptive gradient update strategy inspired by Adam optimization to stabilize training and accelerate convergence.Extensive experiments across multiple vision-language models and tasks demonstrate that ARG-Attack achieves competitive white-box performance and significantly outperforms state-of-the-art methods in black-box transfer settings.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166828153