Work

GPT-SW3: An Autoregressive Language Model for the Nordic Languages

Year: 2023

Type: preprint

Abstract: This paper details the process of developing the first native large generative language model for the Nordic languages, GPT-SW3. We cover all parts of the development process, from data collection and processing, training configuration and instruction finetuning, to evaluation and considerations for release strategies. We hope that this paper can serve as a guide and reference for other researchers that undertake the development of large generative models for smaller languages.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2305.12987

Authors: Ekgren, Ariel, Gyllensten, Amaru Cuba, Stollenwerk, Felix, Öhman, Joey, Isbister, Tim, Gogoulou, Evangelia, Carlsson, Fredrik, Heiman, Alice, Casademont, Judit, Sahlgren, Magnus

Institutions: AI Sweden, RISE Research Institutes of Sweden

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Nvidia, VINNOVA, Vetenskapsrådet

Awards: G3321450376, Examining the liberalizing effect of higher education: A longitudinal cohort study of a university student population, GPT-SW3: a foundational model for Swedish NLP, G712881263, G796431723, NAISS

OpenAlex ID: W4378464452

Cites:

Cited by: 3

Related to: 10