Work

Research on Distributed Training Architecture for Large Scale Models for Natural Language Processing

Year: 2025

Type: conference-paper

Abstract: Large-scale language models are critical for natural language processing, and efficient distributed learning is needed to train such models. In this paper, a distributed training scheme is developed to improve the efficiency and scalability of training large-scale language models. This training scheme is divided into a three-layer structure, and also incorporates an autotuned gradient compression method and a load balancing mechanism to improve training efficiency at any time. Experiments show that this scheme is highly efficient in terms of computation and resource utilization for large-scale training tasks, is much faster than current open systems, and offers new ideas for faster training of large-scale language models.

Source: International Conference on Generative Artificial Intelligence and Information Security (GAIIS)

DOI: 10.1145/3728725.3728812

Authors: Jiangchuan Gong, Yang Wang

Institution: Baidu (China)

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

OpenAlex ID: W4411015872

FWCI:

Cites: 10

Cited by:

Related to: 10