Year: 2024
Type: preprint
Abstract: Language models are typically trained on large corpora of text in their default orthographic form. However, this is not the only option; representing data as streams of phonemes can offer unique advantages, from deeper insights into phonological language acquisition to improved performance on sound-based tasks. The challenge lies in evaluating the impact of phoneme-based training, as most benchmarks are also orthographic. To address this, we develop a pipeline to convert text datasets into a continuous stream of phonemes. We apply this pipeline to the 100-million-word pre-training dataset from the BabyLM challenge, as well as to standard language and grammatical benchmarks, enabling us to pre-train and evaluate a model using phonemic input representations. Our results show that while phoneme-based training slightly reduces performance on traditional language understanding tasks, it offers valuable analytical and practical benefits.
Source: arXiv (Cornell University)
DOI: 10.17863/cam.125909
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G1690597291, G2264885075, Cambridge University Development Office in the United States Inc (CUDOUS) - to endow a global scholarship program at the post-baccalaureate level for academically gifted students, G581313191, Cambridge Service for Data Driven Discovery (CSD3) - A National Data Intensive Science Cloud for Converged Simulation, AI & Analytics, G7028966183
OpenAlex ID: W7130722367