Year: 2026
Type: article
Abstract: The widespread adoption of the pre-training and fine-tuning paradigm has resulted in a large number of task-specific model checkpoints, motivating increasing interest in model merging as a training-free approach to endow a single model with multi-task capabilities. Existing model merging methods predominantly assume that all models are fine-tuned from the same pretrained checkpoint, an assumption that breaks down when models differ in initialization and exhibit structural heterogeneity. In such settings, direct parameter- or representation-based alignment becomes unreliable, leading to severe performance degradation. In this work, we propose A-EMM, an activation-driven, training-free model merging framework for simple heterogeneous models with different initializations. Instead of relying on weight-space alignment, A-EMM leverages neuron activations to guide the entire merging process through three stages: evaluation, matching, and merging. Specifically, neuron importance is estimated from activation statistics to enable layer-segment matching between heterogeneous models, after which neurons are aligned and merged based on activation similarity within each layer. This activation-centric design allows heterogeneous models to be merged without retraining while preserving their functional behaviors. Extensive experiments on multiple architectures and datasets demonstrate that A-EMM consistently achieves performance comparable to homogeneous model merging and slightly outperforms existing state-of-the-art heterogeneous merging methods, validating its effectiveness and robustness. The code is available at \href{https://github.com/uestcAs/A-EMM}{https://github.com/uestcAs/A-EMM}.
Authors: Shuo An, Ke Qin, Dongyang Zhang, Xin Wu
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Keywords: heterogeneous models, multi-task learning
Open Access status: green
Awards: G3601739905, G9001397922
OpenAlex ID: W7130946099
