Year: 2023
Type: conference-paper
Abstract: Vision-language models (VLMs), such as CLIP and ALIGN, are generally trained on datasets consisting of image-caption pairs obtained from the web. However, real-world multimodal datasets, such as healthcare data, are significantly more complex: each image (e.g. X-ray) is often paired with text (e.g. physician report) that describes many distinct attributes occurring in fine-grained regions of the image. We refer to these samples as exhibiting high pairwise complexity, since each image-text pair can be decomposed into a large number of region-attribute pairings. The extent to which VLMs can capture fine-grained relationships between image regions and textual attributes when trained on such data has not been previously evaluated. The first key contribution of this work is to demonstrate through systematic evaluations that as the pairwise complexity of the training dataset increases, standard VLMs struggle to learn region-attribute relationships, exhibiting performance degradations of up to 37% on retrieval tasks. In order to address this issue, we introduce ViLLA as our second key contribution. ViLLA, which is trained to capture fine-grained region-attribute relationships from complex datasets, involves two components: (a) a lightweight, self-supervised mapping model to decompose image-text samples into region-attribute pairs, and (b) a contrastive VLM to learn representations from generated region-attribute pairs. We demonstrate with experiments across four domains (synthetic, product, medical, and natural images) that ViLLA outperforms comparable VLMs on fine-grained reasoning tasks, such as zero-shot object detection (up to 3.6 AP50 points on COCO and 0.6 mAP points on LVIS) and retrieval (up to 14.2 R-Precision points1
Institution: Stanford University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funders: National Science Foundation, U.S. Department of Defense, Gordon and Betty Moore Foundation, Hertz Foundation, National Institutes of Health, Agency for Healthcare Research and Quality, National Defense Science and Engineering Graduate, National Institute of Arthritis and Musculoskeletal and Skin Diseases, National Institute of Biomedical Imaging and Bioengineering
Awards: G1764514577, G1884735594, G2039013894, G2272366923, G2797935254, G2911450482, G3018715022, G3322356154, G3709932587, Rapid Low-Cost Quantitative 3D MRI and Gait Assessment of the Knee, G4749485581, G4945478376, Graduate Research Fellowship Program (GRFP), G6141153606, G6162638667, G6466436341, G6661425536, G7729935405, G8842961228, G8877264399, Mobilize Center: Models for Mobile Sensing and Precision Rehabilitation, Imaging of Joint Response to Physiological Stress with Age, Sex and in Osteoarthritis, Designing Novel MR Imaging Tools to Quantify Lower-Limb Exercise Adaptations in Knee Osteoarthritis
OpenAlex ID: W4390873747