Work

Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models

Year: 2026

Type: conference-paper

Abstract: The safety of large vision-language models is increasingly stress-tested by multimodal jailbreaks, yet existing attacks remain largely static at the meta level: template-based attacks freeze the image-text layout, while iterative attacks adapt only the image-text content with fixed attack strategies and frozen attacker parameters.We propose Meta-Adaptive Multimodal Jailbreaking (MAMJ), which instead optimizes the attacker itself along two axes: an attack strategy prompt (ASP) θ governing attack iteration and attacker weights ϕ determining attack effectiveness.Across groups of multimodal attack trajectories, an LLMbased critique first refines θ, after which groupaggregated attack-success-rate (ASR) rewards update ϕ.On MM-SafetyBench, MAMJ achieves 82.02%, 79.29%, and 83.04% ASR against GPT-4o, Gemini-3-Pro-Preview, and Seed 2.0, respectively, outperforming the strongest sample-level baseline by up to 24.1 percentage points.The learned attacker (θ ⋆ , ϕ ⋆ ) also transfers without retraining to unseen victims and remains effective under representative defenses.These results reveal a systemic vulnerability of frontier VLMs to meta-adaptive jailbreaks and motivate defenses against meta-level adversaries.

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/2026.emnlp-main.258