Year: 2026
Type: conference-paper
Abstract: Visual Question Answering (VQA) models process all image patches uniformly despite questions typically requiring only a small subset of visual information.This inefficiency leads to unnecessary computation and can result in attention dilution across irrelevant image regions.We propose Question-Guided Sparse Attention (QGSA), a plug-and-play mechanism that dynamically selects relevant image patches conditioned on question semantics.Our approach introduces three components: (1) a differentiable patch selector based on Gumbel-Softmax reparameterisation that enables end-to-end training with hard patch selection at inference; (2) a self-supervised grounding loss that encourages spatial selectivity without bounding-box annotations, combining contrastive patch selection with patchword alignment via a frozen CLIP encoder; and (3) an adaptive sparsity mechanism that adjusts the number of selected patches according to estimated question complexity.Experiments on SmolVLM-256M-Instruct and SmolVLM-500M-Instruct across three VQA benchmarks (VQA-RAD, A-OKVQA, Ref-COCO) demonstrate that QGSA reduces crossattention FLOPs by 91-99% across input resolutions, achieving up to 76× theoretical speedup at 576px resolution, while maintaining exact accuracy parity with the dense baseline (∆ = 0.0 pp on all datasets).Wall-clock parity with the dense baseline is reached at 336px; realised end-to-end speedup requires larger models where cross-attention dominates total compute.QGSA consistently selects an average of k ≈ 17 patches out of 576 (256M model), up to k ≈ 18 (500M model), yielding up to a 34× reduction in the visual token sequence.These small-scale results validate the feasibility of question-conditioned sparse attention and provide a foundation for scaling to larger VLMs.
Author: Prasanth
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166810357