Year: 2024
Type: conference-paper
Abstract: Multimodal methods that align the semantics of two data modalities-usually text and image-have become increasingly widespread in recent years. While such bi-modal alignments have received much attention, less work has been done toward aligning more than two modalities. This paper explores several strategies for training late-fusion joint-embedding models with up to four simultaneous modalities. We show that, even on a comparatively small dataset, there are clear and disproportional advantages when considering all modalities concurrently rather than consecutively. The results indicate that, when aligning multiple modalities, a training set with even weak alignment between all modalities substantially outperforms the use of several pair-wise alignments and that the addition of a modality can disproportionally increase alignment quality across all modalities.
Institution: University of Zurich
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4407130665