Work

Can Calibration of Positional Encodings Enhance Long Context Utilization?

Year: 2026

Type: conference-paper

Abstract: Large language models suffer from positional biases like the "Lost in the Middle" (LiM) phenomenon and recency bias, which reduce the effective utilization of long contexts.In this work, we investigate the role of positional encodings in this context.Our empirical study confirms the persistence of these biases in modern large language models.Drawing on these findings, we introduce CALIOPE, a training-free framework for calibrating positional encodings at inference time.Our calibrators yield substantial improvements on needle-in-a-haystack and cross-chunk reasoning benchmarks, and offer a practical, lightweight method for improving long-context utilization.

Source: Findings of the Association for Computational Linguistics: EACL

DOI: 10.18653/v1/2026.findings-eacl.120

Authors: Tom Zehle, Matthias Aßenmacher

Institutions: Munich Center for Machine Learning, Ludwig-Maximilians-Universität München

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: European Commission, Deutsche Forschungsgemeinschaft, HORIZON EUROPE Framework Programme

Awards: BERD@NFDI NFDI für Wirtschaftsdaten und Verwandtes, European Large Open Multi-Modal Foundation Models For Robust Generalization On Arbitrary Data Streams