Binary mask tuning on gradient: Towards multi-data question answering
Year: 2025
Type: article
Abstract: Pretrained language models have been widely applied in question-answering tasks. To achieve better generalization on unseen datasets, several previous studies often trained a single model on multiple datasets. However, owing to the noise and distribution differences among various datasets, the model tends to excessively adjust its weights during training on multiple datasets. This deviation from the initial pretraining state results in excellent performance on specific data but overfitting on other datasets. Ultimately, the model loses its ability to generalize to new data. In this paper, from the perspective of imposing constraints on model weights, we propose a novel fine-tuning method, binary mask tuning (BMT). We employ a carefully designed binary mask vector that is closely related to the data distribution to mask the gradient generated by backpropagation , achieving precise alignment of the subspace parameters required to fit the data from a huge parameter space. This approach aimed to enhance the adaptability of the model to the data distribution and improve parameter efficiency via more targeted fine-tuning. Our experiments demonstrate that BMT is not only effective in mitigating the tendency of the model to excessively adjust its weights but also in better capturing cross-dataset regularities and dataset-specific attributes in question-answering tasks across different datasets.
Source: Knowledge-Based Systems
Authors: Chuanyang Gong, Zhihua Wei, Ping Zhu, Duoqian Miao
Institution: Tongji University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $3,430
OpenAlex ID: W4409499183