Year: 2026
Type: preprint
Abstract: Modern autoregressive Vision-Language Models (VLMs) can generate fluent answers while their visual-token representations become weakly tied to the image regions from which they originate. This limits patch-level explainability: a visual token should remain interpretable as the image patch it represents. We study this issue through Logit Lens maps, obtained by projecting each visual-token embedding through the LLM vocabulary head to measure how strongly each image patch is associated with a queried textual concept, such as "cat". We introduce Logit Lens Loss (LLL), a lightweight auxiliary objective that preserves localized visual semantics by directly aligning object-related visual tokens with the vocabulary concepts describing their image regions. LLL requires no architectural modification, mask decoder, or large-scale retraining. We evaluate LLL through both explanation quality and downstream performance. Across LLaVA-v1.5-7B and Qwen2.5-VL-7B, LLL yields sharper object confidence maps, improves grounding and hallucination metrics, transfers to zero-shot pointing, and preserves VQA performance.
Source: arXiv (Cornell University)
Authors: Esmaeilkhani, Parsa, Latecki, Longin Jan
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7127372134
Cites:
Cited by:
Related to: