Work

Progress Report: Towards European LLMs

Year: 2024

Type: article

Abstract: We present preliminary results of the project OpenGPT-X. At present, the project has developed two multilingual large language models (LLMs) designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European Union. Trained on a dataset comprising around 60% non-English data and utilizing a custom multilingual tokenizer, our models address the limitations of existing LLMs that predominantly focus on English or a few high-resource languages. We detail the models' development principles, data processing techniques, tokenizer optimization, and training methodologies. The models demonstrate competitive performance across multilingual benchmarks, as evidenced by their performance on European versions of ARC, HellaSwag, MMLU, and TruthfulQA.

Source: Zenodo (CERN European Organization for Nuclear Research)

DOI: 10.5281/zenodo.13866364

Author: OpenGPT-X

Institutions: Forschungszentrum Jülich, German Research Centre for Artificial Intelligence, Fraunhofer Institute for Integrated Circuits, Fraunhofer Institute for Intelligent Analysis and Information Systems, Fraunhofer-Gesellschaft, Technische Universität Dresden

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W6911700267

FWCI:

Cites:

Cited by:

Related to: