An efficient late-interaction ModernBERT architecture for interpretable token-level semantic matching in Turkish
Year: 2026
Type: article
Abstract: Developing models that balance contextual understanding and computational efficiency remains a challenge in natural language processing, particularly for morphologically rich languages such as Turkish. This study proposes a hybrid Late-Interaction Fusion model based on ModernBERT to improve token-level semantic matching in Turkish texts. The model preserves the scalability of Bi-Encoders while incorporating contextual depth through a Multi-Head Attention–based late-interaction mechanism. The novelty of this work is that, unlike ColBERT-style late-interaction retrievers which combine pre-computed representations through a parameter-free similarity operator, the proposed fusion layer is a learned, query-conditioned cross-attention block that emits a calibrated relevance probability for every individual token, and that it is instantiated on a Turkish ModernBERT backbone trained on a two-million-pair corpus built through evidence-grounded LLM-assisted annotation. Experimental results show that the model attains a token-level F1-score of 0.557, closely approaching the span-based GLiNER baseline (0.580) while requiring substantially less computation, and clearly outperforming lexical matching (0.065), dense retrieval (0.129) and a standard Bi-Encoder (0.481). Additionally, the shared encoder enables fast convergence and the highest Macro-F1 (0.597) among the compared Turkish BERT variants in a downstream classification task, even with limited training data.
Source: Journal of Engineering Research
Institutions: Marmara University, Gedik University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,100
OpenAlex ID: W7202158728