Year: 2025
Type: article
Abstract: In this paper, we evaluated different approaches to freezing BERT-base layers and analyzed their impact on the quality and speed of training in the task of named entity recognition in two languages. Layer freezing is an optimization technique in deep neural network training in which specific layers of a model remain fixed. This means their weights do not change during the backpropagation process. By not updating these layers, the overall number of parameters requiring adjustment is reduced, which results in lower computational demands and faster training times. Partial freezing of layers proved to be an effective way to preserve key representations of the model and ensure its adaptation to new tasks. Experimental results showed that freezing from three to six layers allows to achieve stable model performance regardless of the training language. Unlike standard approaches, our method highlights cross-linguistic applicability and promotes energy-efficient training. We personally designed the experimental setup, implemented the freezing scenarios, and carried out all performance evaluations. This study aims to evaluate the effectiveness of layer freezing in a pre-trained BERT model when performing the named entity recognition task. Two variants of the freezing strategy are considered: in the first one the upper layers of the model are fixed, in the second one the lower layers remain unchanged. The analysis is based on two corpora, the English language CoNLL 2003 and the Kazakh language KazNERD. Our experiments showed that freezing three to six layers provides the best balance between training speed and model quality. On the CoNLL-2003 dataset, the training time decreased from 266 to 167 seconds and the Macro F1 score remained at 87%. On KazNERD, learning accelerated from 1609 to 958 seconds with an accuracy of 94-95 % and Macro F1 in the range of 71-72 %. Full freezing of all 12 layers caused a dramatic drop in quality, with Macro F1 dropping to 50 % on CoNLL and to 7 % on KazNERD. This emphasises the importance of limited freezing and fine-tuning of the model architecture. The study further examines how the choice of layers to freeze influences the model’s ability to adapt to new linguistic patterns and domain-specific terminology. These findings offer useful insights for researchers and practitioners aiming to enhance the efficiency of fine-tuning large language models while ensuring robust performance across different languages and datasets. The results also highlight the potential for optimizing resource usage in various NER applications without compromising critical language understanding.
DOI: 10.37943/22oxky5402
Authors: Bauyrzhan Kairatuly, Aday Shomanov
Institutions: Al-Farabi Kazakh National University, Nazarbayev University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W4412899405