Work

Will it Merge? On The Causes of Model Mergeability

Year: 2026

Type: conference-paper

Abstract: Model merging has emerged as a promising technique for combining multiple fine-tuned models into a single multitask model without retraining.However, the factors that determine whether merging will succeed or fail remain poorly understood.In this work, we investigate why specific models are merged better than others.To do so, we propose a concrete, measurable definition of mergeability.We investigate several potential causes for high or low mergeability, highlighting the base model knowledge as a dominant factor: Models finetuned on instances that the base model knows better are more mergeable than models finetuned on instances that the base model struggles with.Based on our mergeability definition, we explore a simple weighted merging technique that better preserves weak knowledge in the base model.

Source: Findings of the Association for Computational Linguistics: ACL (Findings of ACL)

DOI: 10.18653/v1/2026.findings-acl.1322

Authors: Adir Rahamim, Asaf Yehudai, Boaz Carmeli, Leshem Choshen, Yosi Mass, Yonatan Belinkov