Top-K Collaborative Optimization for High-Dimensional Vector Deduplication
Year: 2026
Type: conference-paper
Abstract: With the widespread application of large pre-trained models in computer vision and natural language processing, massive high-dimensional feature vectors are generated in large-scale industrial scenarios, posing an urgent demand for efficient and accurate vector deduplication technology. To address the low retrieval efficiency, high hardware resource consumption, and the accuracy-speed trade-off faced by traditional methods in large-scale high-dimensional feature vector deduplication scenarios, this paper proposes a Top-K collaborative optimization scheme for high-dimensional vector deduplication. With Top-K similarity retrieval as the core hub, this scheme achieves end-to-end improvement of deduplication performance through the collaborative design of the full process of “pre-screening - retrieval - clustering - resource management”. Specifically, a model-aware dynamic hashing strategy is designed to customize hash functions according to the distribution differences of features from different pre-trained models, thereby accurately narrowing the scope of Top-K retrieval. A mixed-precision computing method combining “FP32 for high-frequency vectors + FP8 for low-frequency vectors” is adopted to adapt to GPU hardware characteristics and reduce video memory overhead. A dynamic K-value adjustment mechanism and a parallel disjoint-set algorithm are introduced to efficiently convert Top-K retrieval results into deduplication outputs, while maximizing hardware utilization through a dynamic resource management module. To verify the effectiveness of the proposed scheme, experiments were conducted on a private dataset comprising 50,000 1024-dimensional features from SSCD/ViT/CLIP, as well as public benchmarks (SIFT1M and DEEP10M). Experimental results show that the proposed scheme reduces the total deduplication time by 88.1%, 63.8%, and 42.0% respectively, decreases the GPU video memory usage by 41.7% compared with the GPU parallel benchmark scheme, and maintains a deduplication accuracy of 98.3% and a retention rate of 86.2%. It can effectively adapt to the large-scale deduplication requirements of high-dimensional features from multiple models, providing efficient technical support for scenarios such as content retrieval and dataset compression.
Author: Hu Ruoqing
Institution: Sun Yat-sen University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7204157465