Work

Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM Benchmarks

Year: 2026

Type: conference-paper

Abstract: Diacritics are orthographic marks added to letters to specify pronunciation, disambiguate lexical meanings, or indicate grammatical distinctions.Diacritics can significantly influence language processing tasks, especially in languages like Arabic, where diacritic usage varies widely across domains and contexts.While diacritics provide valuable linguistic information, their presence can increase subword fragmentation during tokenization, potentially degrading the performance of NLP models.In this paper, we systematically analyze the impact of diacritics on tokenization and benchmark task performance across major Large Language Models (LLMs).Our results demonstrate that while modern LLMs show robustness to the limited diacritics naturally found in texts, full diacritization leads to substantially increased token fragmentation and degraded performance, highlighting the need for careful handling of diacritics in the future development of Arabic LLMs. 1

Source: Findings of the Association for Computational Linguistics: EACL

DOI: 10.18653/v1/2026.findings-eacl.22

Authors: Go Inoue, Bashar Alhafni, Nizar Habash, Timothy Baldwin

Institution: New York University Abu Dhabi

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: New York University Abu Dhabi, York University

OpenAlex ID: W7140144669

FWCI:

Cites:

Cited by:

Related to: