Mapping Dialectal Landscape: A Sequence-to-Sequence Approach to Japanese Dialect-to-Standard Normalization
Year: 2026
Type: article
Abstract: Despite the progressing standardization of the Japanese language, regional dialects persist, particularly among older generations, causing communication gaps, which results in problems especially in healthcare and emergency contexts. This study proposes a text-to-text normalization method to convert eight Japanese dialects into standard Japanese using a fine-tuned mT5-small architecture. We evaluate the impact of learning rate schedulers, training duration, and data preprocessing on model performance. Our results demonstrate that the CharacTER (Character Translation Edit Rate) metric provides a more accurate evaluation than BLEU, which is practically ill-suited for the unsegmented nature of Japanese text. The optimal configuration minimizes character error rates by aligning input data with natural, unspaced Japanese orthography. Furthermore, we observe a statistically significant correlation between the model’s conversion error rate and the physical distance of the source dialect from Tokyo. This finding suggests that the model’s performance effectively serves as a proxy for measuring linguistic distance between dialectal variations and the standard language.
DOI: 10.3390/make8050115
Authors: Kinga Lasek, Michal Ptaszynski, Fumito Masui, Mujahid Khalifah, Juuso Eronen
Institution: Kitami Institute of Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,172
OpenAlex ID: W7155808691