Work

Poster: Module Lightweighting and Path Transferring in Vision-Language Models for Efficient Edge Deployment

Year: 2024

Type: conference-paper

Abstract: We propose an efficient lightweight fine-tuning method that simplifies model design and reduces parameters, focusing on optimizing Visual-Language Models (VLMs) for edge deployment. As VLMs evolve, the parameter size becomes increasingly challenging for edge devices. To overcome this limitation, we combine lightweighting and fine-tuning into a single step. We decompose large linear layers in the vision encoder and introduce smaller matrices in parallel, creating a new path.During fine tuning, performance is improved by reducing the matrix size and increasing the depth, gradually phasing out the original path. We deploy the lightened and fine-tuned model on a Jetson TX2 and shows comparable performance compared to VLMs with larger parameters.

Source: ACM Conference on Embedded Networked Sensor Systems (SenSys)

DOI: 10.1145/3666025.3699390

Authors: Jiahua Bao, Zhipeng Wang, Ziqian Li, Jiaxing Du, Jialiang Yan, Jie Liu

Institution: Harbin Institute of Technology

Language: English

Topic: Advanced Neural Network Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funders: Key Research and Development Program of Heilongjiang, National Science and Technology Major Project

Award: G5315742300