KAFA-Merge: Model Merging Method With Layer-Based Bayes+Linear Search
Year: 2025
Type: article
Abstract: In this paper, we consider the model merging process for large language models (LLMs) under a two-stage optimization framework. Traditional merging methods usually apply fixed blending rates to all layers, which ignores the different levels of contribution of layers to the output. To overcome this limitation, we first group layers with similar functionality and use Bayesian optimization to determine the optimal blending rates for each group. Bayesian optimization offers a globally effective discovery strategy in high-dimensional and computationally expensive search spaces. However, an approach based solely on global search can lead to local artifacts being missed. Therefore, in the second stage, we perform Linear Search by applying controlled variations around the ratios obtained with Bayesian optimization and obtain more precise, locally optimized results. Extensive experiments on the Turkish versions of the Arc, Hellaswag, MMLU and GSM8K datasets show that our proposed Bayes+Linear strategy outperforms existing methods such as SLERP, Linear, Ties and Breadcrumbs. In addition to improving model accuracy and generalization capacity, the KAFA approach works without any additional fine-tuning, making it possible to effectively reuse large language models in different tasks.
Source: IEEE Access
Authors: Muzaffer Kaan Yuce, Mehmet Fatih Amasyali
Institutions: Yıldız Technical University, Istanbul Technical University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,075
OpenAlex ID: W4411550751