Integrating Task Specific Information into Pretrained Language Models for Low Resource Fine Tuning
Year: 2020
Type: conference-paper
Abstract: Pretrained Language Models (PLMs) have improved the performance of natural language understanding in recent years.Such models are pretrained on large corpora, which encode the general prior knowledge of natural languages but are agnostic to information characteristic of downstream tasks.This often results in overfitting when fine-tuned with low resource datasets where task-specific information is limited.In this paper, we integrate label information as a task-specific prior into the self-attention component of pretrained BERT models.Experiments on several benchmarks and real-word datasets suggest that the proposed approach can largely improve the performance of pretrained models when finetuning with small datasets.The code repository is released in https://github.com/ RayWangWR/BERT_label_embedding.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W3103920202