Work

Hi-MetaCap: Configuring Object Relational Transformer in Meta-Learning Environment for Image Captioning in Hindi

Year: 2025

Type: article

Abstract: This article proposes a meta-learning-based, few-shot image captioning framework based on an ensemble of object-relational transformer models and a self-distillation strategy. Unlike traditional approaches, the proposed framework can train models using non-paired images and captions. In each iteration, multiple base models are trained on distinct data samples, forming an ensemble that generates pseudo-captions with confidence-based weighting. An efficient pseudo-feature generation method based on gradient descent enables learning from non-paired captions. These pseudo-features and pseudo-captions are then used to train the base models in subsequent iterations. The obtained results show that using just 1% of the paired image-caption training data significantly improves performance and generates meaningful captions. The proposed framework demonstrates the potential to reduce reliance on large, paired datasets while maintaining high-quality image captioning performance. 1

Source: ACM Transactions on Asian and Low-Resource Language Information Processing

DOI: 10.1145/3749642

Authors: Santosh Kumar Mishra, Soham Chakraborty, Sriparna Saha, Pushpak Bhattacharyya

Institutions: Rajiv Gandhi Institute of Petroleum Technology, KIIT University, Indian Institute of Technology Patna, Indian Institute of Technology Bombay

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

OpenAlex ID: W4412519249

FWCI:

Cites: 41

Cited by:

Related to: 10