Work

Task-specific pre-training improves models for paraphrase generation

Year: 2022
Type: conference-paper
Abstract: Paraphrase generation is a fundamental and longstanding problem in the Natural Language Processing field. With the huge success of transfer learning, the pre-train → fine-tune approach has become a standard choice. At the same time, popular task-agnostic pre-trainings usually require gigabyte datasets and hundreds of GPUs, while available pre-trained models are limited by fixed architecture and size (i.e. base, large). We propose a simple and efficient pre-training approach specifically for paraphrase generation, which noticeably boosts model quality and matches the performance of general-purpose pre-trained models. We also investigate how this procedure influences the scores across different architectures and show that it works for all of them.
Language: English
Open Access status: hybrid
OpenAlex ID: W4382201596
FWCI: 0.068
Cites: 25
Cited by: 1
Related to: 9