Work

Model Merging by Uncertainty-Based Gradient Matching

Year: 2023

Type: conference-paper

Abstract: Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averaging, task arithmetic, and Fisher-weighted averaging. Our new method gives consistent improvements for large language models and vision transformers, both in terms of performance and robustness to hyperparameters. Code available here.

Source: TUbilio (Technical University of Darmstadt)

DOI: 10.48550/arxiv.2310.12808

Authors: Daheim, Nico, Möllenhoff, Thomas, Ponti, Edoardo Maria, Gurevych, Iryna, Khan, Mohammad Emtiyaz

Institutions: Technische Universität Darmstadt, Hessian Center for Artificial Intelligence, University of Edinburgh, RIKEN Center for Advanced Intelligence Project

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Bundesministerium für Bildung und Forschung, Core Research for Evolutional Science and Technology

OpenAlex ID: W4387839046

FWCI: 0.396

Cites:

Cited by: 2

Related to: 10