Work

Training-free merging of depth-heterogeneous models via activation-driven neuron matching

Year: 2026
Type: article
Abstract: The widespread adoption of the pre-training and fine-tuning paradigm has resulted in a large number of task-specific model checkpoints, motivating increasing interest in model merging as a training-free approach to endow a single model with multi-task capabilities. Existing model merging methods predominantly assume that all models are fine-tuned from the same pretrained checkpoint, an assumption that breaks down when models differ in initialization and exhibit structural heterogeneity. In such settings, direct parameter- or representation-based alignment becomes unreliable, leading to severe performance degradation. In this work, we propose A-EMM, an activation-driven, training-free model merging framework for simple heterogeneous models with different initializations. Instead of relying on weight-space alignment, A-EMM leverages neuron activations to guide the entire merging process through three stages: evaluation, matching, and merging. Specifically, neuron importance is estimated from activation statistics to enable layer-segment matching between heterogeneous models, after which neurons are aligned and merged based on activation similarity within each layer. This activation-centric design allows heterogeneous models to be merged without retraining while preserving their functional behaviors. Extensive experiments on multiple architectures and datasets demonstrate that A-EMM consistently achieves performance comparable to homogeneous model merging and slightly outperforms existing state-of-the-art heterogeneous merging methods, validating its effectiveness and robustness. The code is available at \href{https://github.com/uestcAs/A-EMM}{https://github.com/uestcAs/A-EMM}.
Language: English
Open Access status: green
OpenAlex ID: W7130946099
FWCI:
Cites: 36
Cited by:
Related to: 10