Year: 2025
Type: preprint
Abstract: Multimodal large language models (MLLMs) extend the capabilities of large language models (LLMs) by combining heterogeneous model architectures to handle diverse modalities like images and audio. However, this inherent heterogeneity in MLLM model structure and data types makes makeshift extensions to existing LLM training frameworks unsuitable for efficient MLLM training. While there are a few works that have attempted to address the heterogeneity in MLLM training, their approaches are limited to only superficially considering the characteristics of MLLMs. In this paper, we present Cornstarch, an efficient distributed MLLM training framework that contemplates MLLM's unique characteristics in both model and data parallelization. Cornstarch introduces frozen-aware pipeline parallelism and token workload-balanced context parallelism to improve MLLM training throughput. Our extensive evaluation shows that Cornstarch outperforms state-of-the-art solutions by $2.26\times$ on average in terms of MLLM training throughput. Cornstarch is an open-source project available at https://github.com/cornstarch-org/Cornstarch.
Source: arXiv (Cornell University)
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Collaborative Research: CNS Core: Large: Runtime Programmable Networks, Collaborative Research: CNS Core: Medium: Movement of Computation and Data in Splitkernel-disaggregated, Data-intensive Systems, Collaborative Research: AF: Medium: Fundamental Challenges in Discrete and Continuous Optimization, Collaborative Research: AF: Small: New Directions and Approaches in Discrepancy Theory, Collaborative Research: CNS Core: Medium: Systems Support for Federated Learning, Collaborative Research: DESC: Type II: Enhancing Data Center Efficiency through Intelligent Tiered Memory Management
OpenAlex ID: W4417145508