Testing Large Vision-Language Models: A Systematic Review
Year: 2026
Type: preprint
Study design: Systematic Review
Abstract: Vision-Language Models (VLMs) have rapidly evolved into foundational technologies that power a broad spectrum of applications, including visual question answering, multimodal reasoning, autonomous driving, robotics, embodied agents, healthcare, and scientific discovery. Assuring their quality, including reliability, robustness, truthfulness, safety, fairness, efficiency, and security-is therefore an important software-engineering challenge. Although research on VLM benchmarks, evaluation frameworks, and quality assessment has grown rapidly, the landscape remains fragmented, with limited systematic understanding of what is tested, how testing is performed, and whether testing contributions are adopted in downstream software ecosystems. This paper presents a systematic mapping of VLM testing research and practice from a softwaretesting perspective. We analyze 307 primary studies published between 2020 and June 26, 2026, and develop a unified taxonomy characterizing the quality properties researchers seek to assure, the system components and representations targeted, and the methodologies employed. We complement the literature study with a large-scale analysis of downstream GitHub repositories referencing VLM testing artifacts, distinguishing operational reuse from evaluation/reporting references and documentary uptake. Our findings reveal a landscape dominated by functional reliability, reasoning, robustness, and hallucination assessment, with comparatively limited attention to privacy, uncertainty calibration, evaluation reliability, and embodied safety. Testing remains predominantly evaluation-centered, emphasizing observable model behavior and benchmarkdriven assessment, while component-level diagnosis, automated test generation, regression testing, and deployment-oriented assurance receive less attention. The downstream analysis further shows that research contributions diffuse selectively: reusable benchmarks, datasets, and evaluation infrastructure provide prominent pathways into software ecosystems, whereas direct transfer of complete testing methodologies is comparatively limited. These findings suggest that VLM testing has developed substantial capability-evaluation resources but has not yet matured into an equally reusable, lifecycle-oriented quality-assurance ecosystem. The resulting taxonomies and adoption evidence provide a foundation for identifying research gaps and guiding the development of more systematic, integrable, and deployment-aware VLM testing practices.
Source: HAL
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7213486361