RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs
Year: 2025
Type: conference-paper
Abstract: Current Multimodal Large Language Model (MLLM) architectures face a critical tradeoff between performance and efficiency: decoderonly architectures achieve higher performance but lower efficiency, while cross-attentionbased architectures offer greater efficiency but lower performance.The key distinction lies in how visual tokens are processed.Decoder-only architectures apply self-attention and FFN operations on visual tokens, while cross-attention architectures skip these computations.To investigate whether redundancy exists in this computationally expensive process, we propose a training-free framework for analyzing trained MLLMs.It consists of Probe-Activated Dynamic FFN and Hollow Attention, which enable adjustable reductions in computations for visual tokens, as well as a Layer Ranking Algorithm that prioritizes layers for these reductions.Extensive experiments demonstrate substantial, structured, and clustered redundancy unique to decoder-only MLLMs, offering valuable insights for future MLLM architecture design.Furthermore, by leveraging our reduction framework as a training-free inference acceleration approach, we achieve performance comparable to or better than stateof-the-art methods while remaining compatible with them.Code is available at https: //github.com/L-Hugh/RedundancyLens.
Authors: Hongliang Li, Jiaxin Zhang, Wenhui Liao, Dezhi Peng, Kai Ding, Lianwen Jin
Institution: South China University of Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Awards: G5270964542, G6561198578
OpenAlex ID: W4412944615