Scalable Deep Neural Network Training: Overcoming Memory Constraints with Performance Preservation
Year: 2025
Type: article
Abstract: This paper introduces a novel method for training Deep Neural Networks (DNNs) with batch sizes that surpass memory constraints, without compromising model performance. Conventional methods struggle with memory limitations when using large batch sizes, often resulting in reduced efficiency and degraded training outcomes. Our proposed solution, Small-Batch Processing (SBP), overcomes these challenges by leveraging a batch streaming mechanism and a loss normalization strategy based on gradient accumulation. SBP processes smaller micro-batches sequentially, enabling large-batch training on a single device without requiring memory augmentation or additional GPUs. Experimental evaluations confirm that SBP matches the performance of traditional approaches while enabling the use of batch sizes that were previously infeasible due to memory constraints. This technique significantly enhances the potential for training DNNs in resource-limited environments, driving improvements in both scalability and training efficiency for deep learning applications.
Authors: Kaligoti Ravikumar, C. Sivakumar
Institution: Mohan Babu University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W4415227625
FWCI:
Cites:
Cited by:
Related to: