Year: 2025
Type: conference-paper
Abstract: Recent advancements in vision-language models (VLMs) have improved performance by increasing the number of visual tokens, which are often significantly longer than text tokens.However, we observe that most real-world scenarios do not require such an extensive number of visual tokens.While the performance drops significantly in a small subset of OCR-related tasks, models still perform accurately in most other general VQA tasks with only 1/4 resolution.Therefore, we propose to dynamically process distinct samples with different resolutions, and present a new paradigm for visual token reduction, namely, VisionThink.It starts with a downsampled image and smartly decides whether it is sufficient for problem solving.Otherwise, the model could output a special token to request the higher-resolution image.Compared to existing Efficient VLM methods that reduce tokens using fixed pruning ratios or thresholds, VisionThink autonomously decides whether to reduce tokens case by case.As a result, it demonstrates strong fine-grained visual understanding capability on OCR-related tasks, and meanwhile saves substantial visual tokens on simpler tasks.We adopt reinforcement learning and propose the LLM-as-Judge strategy to successfully apply RL to general VQA tasks.Moreover, we carefully design a reward function and penalty mechanism to achieve a stable and reasonable image resize call ratio.Extensive experiments demonstrate the superiority, efficiency, and effectiveness of our method.
DOI: 10.52202/085713-3182
Authors: Senqiao Yang, Junyi Li, Xin Lai, Jinming Wu, Wei Li, Zejun MA, Bei Yu, Hengshuang Zhao, Jiaya Jia
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: G1018712551, G4599997883
OpenAlex ID: W4415310642
