BCD-Megatron: A Cost-Effective Training System for Large Language Models
Year: 2026
Type: conference-paper
Abstract: Training large language models typically demands substantial GPU memory and significant financial investment, posing a major barrier for many small-to-medium research groups and companies. In this paper, we present a full-parameter pretraining and fine-tuning framework, BCD-Megatron, built on a block-wise update strategy, together with a set of engineering optimizations, enabling efficient model training on both lowcost consumer GPUs (e.g., RTX 4090) and high-performance computing clusters. By partitioning model parameters into blocks and updating only a subset of parameters in each training iteration, our approach markedly reduces the GPU memory requirement. Experimental results show that, under the same hardware configuration, our method reduces the training cost of a 7B-parameter model to 33% of that required by conventional fullparameter training on high-end clusters (A100/A800). Moreover, despite the substantial cost reduction, our approach achieves comparable wall-clock training time and accuracy to traditional methods. This progress significantly lowers the hardware barrier to large-scale model development and demonstrates the practicality and cost-effectiveness of full-parameter training using low-cost GPU clusters.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7172425507