Work

Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification

Year: 2022

Type: conference-paper

Abstract: Social media is daily creating massive multimedia content with paired image and text, presenting the pressing need to automate the vision and language understanding for various multimodal classification tasks.Compared to the commonly researched visual-lingual data, social media posts tend to exhibit more implicit image-text relations.To better glue the crossmodal semantics therein, we capture hinting features from user comments, which are retrieved via jointly leveraging visual and lingual similarity.Afterwards, the classification tasks are explored via self-training in a teacherstudent framework, motivated by the usually limited labeled data scales in existing benchmarks.Substantial experiments are conducted on four multimodal social media benchmarks for image-text relation classification, sarcasm detection, sentiment classification, and hate speech detection.The results show that our method further advances the performance of previous state-of-the-art models, which do not employ comment modeling or self-training.

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/v1/2022.emnlp-main.381

Authors: Chunpu Xu, Jing Li

Institution: Hong Kong Polytechnic University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Baidu, National Natural Science Foundation of China, Hong Kong Polytechnic University, Young Scientists Fund

OpenAlex ID: W4385572977