Work

Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models

Year: 2026

Type: preprint

Abstract: Recent research has shown that aligning fine-grained text descriptions with localized image patches can significantly improve the zero-shot performance of pre-trained vision-language models (e.g., CLIP). However, we find that both fine-grained text descriptions and localized image patches often contain redundant information, making text-visual alignment less effective. In this paper, we tackle this issue from two perspectives: \emph{View Refinement} and \emph{Description refinement}, termed as \textit{\textbf{Bi}-refinement for \textbf{F}ine-grained \textbf{T}ext-visual \textbf{A}lignment} (BiFTA). \emph{View refinement} removes redundant image patches with high \emph{Intersection over Union} (IoU) ratios, resulting in more distinctive visual samples. \emph{Description refinement} removes redundant text descriptions with high pairwise cosine similarity, ensuring greater diversity in the remaining descriptions. BiFTA achieves superior zero-shot performance on 6 benchmark datasets for both ViT-based and ResNet-based CLIP, justifying the necessity to remove redundant information in visual-text alignment.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2601.20419

Authors: Sun, Yuhao, Cai, Chengyi, Zhang, Jiacheng, Ye, Zesheng, Yuan, Xingliang, Liu, Feng

Institution: The University of Melbourne

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Commonwealth Scientific and Industrial Research Organisation, University of Melbourne, Australian Research Council

Awards: Collaborative Research: NSF-CSIRO: RESILIENCE: Graph Representation Learning for Fair Teaming in Crisis Response, Trustworthy Hypothesis Transfer Learning

OpenAlex ID: W7126027879

Cites:

Cited by:

Related to: