Work

ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval

Year: 2025

Type: preprint

Abstract: Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval, whether in how they encode documents, define training objectives, or compute similarity scores. To address these limitations, we present ColMate, a document retrieval model that bridges the gap between multimodal representation learning and document retrieval. ColMate utilizes a novel OCR-based pretraining objective, a self-supervised masked contrastive learning objective, and a late interaction scoring mechanism more relevant to multimodal document structures and visual characteristics. ColMate obtains 3.61% improvements over existing retrieval models on the ViDoRe V2 benchmark, demonstrating stronger generalization to out-of-domain benchmarks.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2511.00903

Authors: Masry, Ahmed, Thakkar, Megh, Bechard, Patrice, Madhusudhan, Sathwik Tejaswi, Awal, Rabiul, Mishra, Shambhavi, Suresh, Akshay Kalkunte, Daruru, Srivatsava, Hoque, Enamul, Gella, Spandana, Scholak, Torsten, Rajeswar, Sai

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W4416433131

Cites:

Cited by:

Related to: