Work

How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese

Year: 2023

Type: preprint

Abstract: This paper investigates the effect of tokenizers on the downstream performance of pretrained language models (PLMs) in scriptio continua languages where no explicit spaces exist between words, using Japanese as a case study. The tokenizer for such languages often consists of a morphological analyzer and a subword tokenizer, requiring us to conduct a comprehensive study of all possible pairs. However, previous studies lack this comprehensiveness. We therefore train extensive sets of tokenizers, build a PLM using each, and measure the downstream performance on a wide range of tasks. Our results demonstrate that each downstream task has a different optimal morphological analyzer, and that it is better to use Byte-Pair-Encoding or Unigram rather than WordPiece as a subword tokenizer, regardless of the type of task.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2306.09572

Authors: Fujii, Takuro, Shibata, Koki, Yamaguchi, Atsuki, Morishita, Terufumi, Sogawa, Yasuhiro

Institutions: Yokohama National University, University of Tsukuba, Hitachi (Japan)

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W4381247876

Cites:

Cited by: 2

Related to: 10