Work

BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages

Year: 2018

Type: conference-paper

Abstract: We present BPEmb, a collection of pre-trained subword unit embeddings in 275 languages, based on Byte-Pair Encoding (BPE). In an evaluation using fine-grained entity typing as testbed, BPEmb performs competitively, and for some languages bet- ter than alternative subword approaches, while requiring vastly fewer resources and no tokenization. BPEmb is available at https://github.com/bheinzerling/bpemb

Source: Language Resources and Evaluation Conference (LREC)

DOI: 10.63317/3eqtu58tsqay

Authors: Benjamin Heinzerling, Michael Strube

Institutions: University of Cambridge, Heidelberg Institute for Theoretical Studies

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

OpenAlex ID: W2962685689

FWCI: 14.78

Cites: 13

Cited by: 181

Related to: 20