Semantic-based temporal attention network for Arabic Video Captioning
Year: 2024
Type: article
Abstract: In recent years, there has been a surge in active research aiming to bridge the gap between computer vision and natural language. In a linguistically diverse region like the Arab world, it is essential to establish a mechanism that facilitates the understanding of visual aspects in native languages. In this paper, we present the first Arabic video captioning method in the literature, using an encoder-decoder paradigm based on CNN and LSTM. We employ a temporal attention mechanism along with semantic features to align keyframes with relevant semantic tags. Due to the lack of an Arabic captioning dataset, we use Google's machine translation system to generate Arabic captions for the MSVD and MSR-VTT datasets, which can be used to train end-to-end Arabic video captioning models. The semantic features are extracted from a neural semantic representation network, which has been specifically trained on Arabic tags for better understanding. Semitic languages like Arabic are heavily attributed to complex morphology, which poses challenges for video captioning. We alleviate these difficulties by employing the AraBERT model as a preprocessing tool. Experimental results on the translated MSVD and MSR-VTT benchmarks indicate that employing an end-to-end Arabic approach for video captioning significantly outperforms English-Arabic translated methods produced using state-of-the-art techniques.
Authors: Adel Jalal Yousif, Mohammed H. Al-Jammas
Institutions: University of Mosul, University of Diyala, Ninevah University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,700
OpenAlex ID: W4405555527