Work

TITE: Token-Independent Text Encoder for Information Retrieval

Year: 2025

Type: conference-paper

Abstract: Transformer-based retrieval approaches typically use the contextualized embedding of the first input token as a dense vector representation for queries and documents. The embeddings of all other tokens are also computed but then discarded, wasting resources. In this paper, we propose the Token-Independent Text Encoder (TITE) as a more efficient modification of the backbone encoder model. Using an attention-based pooling technique, TITE iteratively reduces the sequence length of hidden states layer by layer so that the final output is already a single sequence representation vector. Our empirical analyses on the TREC 2019 and 2020 Deep Learning tracks and the BEIR benchmark show that TITE is on par in terms of effectiveness compared to standard bi-encoder retrieval models while being up to 3.3 times faster at encoding queries and documents. Our code is available at: https://github.com/webis-de/SIGIR-25.

Source: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

DOI: 10.1145/3726302.3730094

Authors: Ferdinand Schlatt, Tim Hagen, Martin Potthast, Matthias Hagen

Institutions: Friedrich Schiller University Jena, University of Kassel, Center for Scalable Data Analytics and Artificial Intelligence

Language: English

Topic: Information Retrieval and Search Behavior

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Universitas Brawijaya

Award: G6072843279