Work

Multilingual Text-Based Person Search Based on Foundation Models

Year: 2025

Type: article

Abstract: In recent years, text-based person search (TBPS) has garnered significant attention, which involves tackling complex issues in both natural language processing and computer vision. Although significant progress has been made, most TBPS methods have focused mainly on English-language descriptions. As a result, person search performance in non-English languages remains significantly lower compared to English. Multilingual models, which can leverage high-resource languages to enhance understanding in low-resource ones, offer a promising solution for TBPS. In this paper, we propose a multilingual text-based person search method, named TBPS-mSigLIP, which utilizes a multilingual visual-language model as an encoder to extract both visual and textual features. Then, we incorporate various data augmentation techniques and loss functions to enhance model performance. To assess the effectiveness of multilingual models across both high- and low-resource languages, we have conducted extensive experiments on four benchmark datasets including CUHK-PEDES, 3000VnPersonSearch, 3000Vn-V2E, and PRW-TPS-CN with queries in three languages (i.e. Vietnamese, English, and Chinese). The proposed method achieves Rank@1 accuracies of 70.76%, 49.70%, 39.33%, and 46.78% on CUHK-PEDES, 3000VnPersonSearch, 3000Vn-V2E, and PRW-TPS-CN datasets, respectively. It outperforms existing methods on both the 3000VnPersonSearch and 3000Vn-V2E datasets, with Rank@1 improvements of 9.4% and 11.05% over CFFVL [1] and M-IRRA [2] on 3000VnPersonSearch, and 2.26% and 3.93% over TBPS-CLIP and M-IRRA on 3000Vn-V2E, respectively. On CUHK-PEDES and PRW-TPS-CN, the proposed framework demonstrates competitive performance compared to state-of-the-art methods. Furthermore, we introduce a two-stage transfer learning paradigm to reduce memory usage and training time while maintaining search performance. Our code is available at: https://github.com/sigm-seee/TBPS_mSigLIP.

Source: IEEE Access

DOI: 10.1109/access.2025.3643623