Year: 2022
Type: conference-paper
Abstract: Prefix-tuning is a parameter-efficient and powerful technique for adapting a pre-trained language model to a downstream application.However, it uses the same dataset-level tuned set of parameters for all examples in the dataset.We extend the framework with a dynamic method, CONTROL PREFIXES, which allows for the effective inclusion of input-dependent information, thereby demonstrating how prefixtuning can be used for controlled text generation tasks.The method incorporates attributelevel learnable representations into different layers of a pre-trained Transformer, enabling the generated text to be guided in a particular direction.We provide a systematic evaluation of the technique and apply it to five datasets from the GEM benchmark for natural language generation (NLG).Using only 0.1-2% additional trainable parameters, we show CON-TROL PREFIXES can even outperform full finetuning methods, and present state-of-the-art results on several data-to-text datasets, including WebNLG.We also examine the common case where input-dependent information is unavailable at test time and show CONTROL PREFIXES can excel in this setting also.
Authors: Jordan Clive, Kris Cao, Marek Rei
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4385574265