Work

A novel multi-modal retrieval framework for tracking vehicles using natural language descriptions

Year: 2025

Type: article

Abstract: Recent advances in multimodal and contrastive learning have significantly enhanced image and video retrieval capabilities. This fusion provides numerous opportunities for multi-dimensional and multi-view retrieval, especially in multi-camera surveillance scenarios in traffic environments. This paper introduces a novel Multi-modal Vehicle Retrieval (MVR) system designed to retrieve the trajectories of tracked vehicles using natural language descriptions. The MVR system integrates an end-to-end text-video comparison learning model, utilizes CLIP for feature extraction, and uses a matching control system and multi-context-based attributes. Post-processing techniques are used to eliminate erroneous information. By comprehensively understanding vehicle characteristics, the MVR system can effectively identify trajectories based on natural language descriptions. Our method achieves a mean reciprocal ranking (MRR) score of 0.8966 on the test data set of the 7th AI City Challenge Track 2 for retrieving tracked vehicles through natural language descriptions, surpassing the previous top-ranked result on the public leaderboard.

Source: PLoS ONE

DOI: 10.1371/journal.pone.0327468

Authors: Changhao Zhang, Zhandong Liu, Ke Li, Yong Li, Xiangwei Qi, Nan Ding

Institution: Xinjiang Normal University

Language: English

Topic: Video Surveillance and Tracking Methods

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

SDG: Sustainable cities and communities

Open Access status: gold

Estimated APC paid: $2,382