Work

Modular and Parameter-Efficient Multimodal Fusion with Prompting

Year: 2022

Type: conference-paper

Abstract: Recent research has made impressive progress in large-scale multimodal pre-training.In the context of the rapid growth of model size, it is necessary to seek efficient and flexible methods other than finetuning.In this paper, we propose to use prompt vectors to align the modalities.Our method achieves comparable performance to several other multimodal fusion methods in low-resource settings.We further show that our method is modular and parameter-efficient for processing tasks involving two or more data modalities.

Source: Findings of the Association for Computational Linguistics: ACL (Findings of ACL)

DOI: 10.18653/v1/2022.findings-acl.234

Authors: Sheng Liang, Mengjie Zhao, Hinrich Schuetze

Institution: Ludwig-Maximilians-Universität München

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: H2020 European Research Council

Award: Non-sequence models for tokenization replacement