Work

PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting

Year: 2023

Type: conference-paper

Abstract: Vision-language (VL) Pre-training (VLP) has shown to well generalize VL models over a wide range of VL downstream tasks, especially for cross-modal retrieval. However, it hinges on a huge amount of image-text pairs, which requires tedious and costly curation. On the contrary,weakly-supervised VLP (W-VLP) explores means with object tags generated by a pre-trained object detector (OD) from images. Yet, they still require paired information, i.e. images and object-level annotations, as supervision to train an OD.

Source: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

DOI: 10.1145/3539618.3592038

Authors: Zixin Guo, Tzu-Jui Julius Wang, Selen Pehlivan, Abduljalil Radman, Jorma Laaksonen

Institution: Aalto University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: European High Performance Computing Joint Undertaking, Academy of Finland, China Scholarship Council

Award: Understanding speech and scene with ears and eyes