Re3Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning
Year: 2026
Type: conference-paper
Abstract: Reinforcement Learning (RL) has demonstrated significant gains in image captioning, yet it is still limited in encouraging Large Vision-Language Models (LVLMs) to explore novel reasoning strategies.This limitation leads to a performance gap between RL and Supervised Fine-Tuning (SFT).In this paper, we argue that multi-modal retrieval can serve as an effective reasoning signal for caption refinement.Based on this insight, we present the Retrieval-Guided Refinement for Image Captioning (Re 3 Cap), a retrieval-guided reasoning strategy that enhances image captioning by performing retrieval on support sets.Instantiated by Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA), this strategy identifies hallucinations and omissions in image captions, leading to more accurate and detailed descriptions.Extensive experiments demonstrate the superiority of our method in image captioning, even compared with Supervised Fine-Tuning.Especially, Re 3 Cap outperforms GRPO with an average improvement of 8.64% in relation reasoning on the COCO-LN500 benchmark.𝑠 !"
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7204146434