Learning to Ask: Multi-Decoder Fine-Tuning for Multi-Hop Visual Question Generation with External Knowledge
Year: 2026
Type: conference-paper
Abstract: Multi-hop visual question generation (VQG) seeks to create coherent, fluent, and contextually rich questions by integrating knowledge from a structured knowledge graph (KG) with information inferred from image pairs, where at least one reasoning step involves a visual relationship between the images.Traditional supervised QG methods which rely on tokenlevel alignment with fixed gold labels struggle to capture diverse valid question formulations.We propose M3RQG (Multimodal Multi-hop Multi-decoder Retrieval-augmented Question Generation), a model-agnostic framework that integrates multimodal inputs (images, KG facts) with a multi-decoder architecture to optimize for multiple labels per sample to design multi-hop questions.M3RQG addresses these challenges: (1) generating meaningful visually-grounded questions given a pair of images, (2) generating rich questions that require multi-hop reasoning across images and KG facts, and (3) integrating diverse question labels during fine-tuning.We extend the We-bQA dataset with multi-hop questions generated by GPT-4V and Gemini, resulting in two complementary silver labels per sample.Our approach integrates retrieval-augmented generation (RAG) for accessing external knowledge, a PPO objective with ROUGE-based rewards to prioritize structural correctness, and a named entity overlap loss to improve factual accuracy.Experiments across BART, Phi-3.5, and LLaVA backbones demonstrate significant improvements in fluency, reasoning depth, and relevance.We release our code and dataset to facilitate future research 1 .π¬π¬ ππ =Taj Mahal π¬π¬ ππ =Bibi Ka Maqbara Q1: What is the height of the Taj Mahal?Q2: Which Mughal emperor commissioned the mausoleum built between 1631 and 1653 in Agra to honor his wife Mumtaz Mahal?Q3: Which Mughal prince commissioned the mausoleum built in Aurangabad in 1660 to honor his mother Dilras Banu Begum?Q4: What is the combined construction period of both monuments shown?Q5: Which image shows a monument with more visible minarets?Q6: Which monument is shown in the first image?Q7: Which mausoleum, commissioned in 1660 by Aurangzeb's son Azam Shah in memory of his mother, was designed by the son of the architect of the monument built between 1631 and 1653 by Shah Jahan for Mumtaz Mahal?Q8: Who commissioned this popular tomb in Maharashtra which has an architecture very similar to the mausoleum on the right bank of Yamuna commissioned in 1631 by Shah Jahan?
Authors: Arpan Phukan, Manish Gupta, Asif Ekbal
Institutions: Indian Institute of Technology Patna, Microsoft (India)
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140121792