Year: 2025
Type: preprint
Abstract: Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces Numerical Token Integrity Loss (NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL.
Source: arXiv (Cornell University)
Authors: Fei, Xiang, Lu, Jinghui, Sun, Qi, Feng, Hao, Wang, Yanjie, Shi, Wei, Wang, An-Lan, Tang, Jingqun, Huang, Can
Institutions: ByteDance (China), City University of Hong Kong
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4415035317
Cites:
Cited by:
Related to: