Evolutionary Prompt Optimization for Visual Language Models using Equivariant Semantics
Year: 2026
Type: conference-paper
Abstract: Large Language Models (LLMs) have recently enabled guided evolutionary optimization in complex, discrete search spaces, such as prompt engineering for Vision-Language Models (VLMs). However, existing evolutionary prompt optimization methods largely focus on unstructured classification prompts, such as those used in CLIP, and fail to generalize to modern generative VLMs, which require rigid instruction templates. Moreover, VLMs are highly sensitive to label vocabulary, yet existing methods typically treat changes in the target vocabulary as incidental by-products of prompt optimization rather than as explicit variables to be optimized. In this work, we propose an LLM-assisted evolutionary framework for open-vocabulary object detection that treats prompts and target label vocabularies as co-evolving, first-class representations within a discrete search space. Model-specific templates ensure syntactic validity, while the LLM performs guided mutations over prompts and labels, enforcing a form of semantic equivariance, where linguistically equivalent transformations induce improved alignment between labels, instructions, and images. Experiments on the COCO benchmark demonstrate that this evolutionary search substantially improves detection performance, highlighting a scalable, gradient-free approach for aligning VLMs with downstream tasks.
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7224005239
