Image captioning generation based on grid-fusion features
Year: 2024
Type: conference-paper
Abstract: Image captioning is a challenging task that involves generating natural language descriptions for images. In this article the proposed novel approach for image captioning that leverages grid-based fusion features using convolutional networks (CNNs) and Contrastive Language-Image Pre-training (CLIP)-based models. Our method addresses the limitations of existing techniques by combining the strengths of CNNs for capturing spatial relationships in images and CLIP-based models for learning multimodal representations. First, we extract grid-based features from images using CNN, which allow us to capture fine-grained spatial information. These features are then fused with CLIP-based representations, which encode both image and text modalities in a shared embedding space. This fusion process enhances the semantic understanding of the images, leading to more descriptive captions. We evaluate our approach on the MS COCO dataset and achieve state-of-the-art performance in terms of BLEU scores and CIDEr scores. Our method outperforms existing techniques, demonstrating the effectiveness of combining CNNs and CLIP-based models for image captioning.
Authors: Prashant Kaushik, Prantik Biswas
Institution: Jaypee Institute of Information Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4403882638