Year: 2022
Type: conference-paper
Abstract: Maximum likelihood estimation (MLE) is the predominant algorithm for training text generation models.This paradigm relies on direct supervision examples, which is not applicable to many emerging applications, such as generating adversarial attacks or generating prompts to control language models.Reinforcement learning (RL) on the other hand offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward.Yet previous RL algorithms for text generation, such as policy gradient (on-policy RL) and Q-learning (off-policy RL), are often notoriously inefficient or unstable to train due to the large sequence space and the sparse reward received only at the end of sequences.In this paper, we introduce a new RL formulation for text generation from the soft Q-learning (SQL) perspective.It enables us to draw from the latest RL advances, such as path consistency learning, to combine the best of on-/off-policy updates, and learn effectively from sparse reward.We apply the approach to a wide range of novel text generation tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.Experiments show our approach consistently outperforms both task-specialized algorithms and the previous RL methods. 1
Authors: Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, Zhiting Hu
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: National Science Foundation
Awards: Collaborative Research: SCH: Trustworthy and Explainable AI for Neurodegenerative Diseases, III: Small: A New Approach to Latent Space Learning with Diversity-Inducing Regularization and Applications to Healthcare Data Analytics, XPS: FULL: Broad-Purpose, Aggressively Asynchronous and Theoretically Sound Parallel Large-scale Machine Learning, CNS Core: Small: Toward Globally-Optimal Resource Distribution and Computation Acceleration in Multi-Tenant and Heterogeneous Machine Learning Systems, ML Basis for Intelligence Augmentation:Toward Personalized Modeling, Reasoning under Data-Knowledge Symbiosis, and Interpretable Interaction for AI-assisted Human Decision-making, RI: III: Medium: Scalable Machine Learning for Automating Scientific Discovery in Astrophysics, Collaborative Research: RI: Medium: A Rigorous, General Framework for Tractable Learning of Large-Scale DAGs from Data
OpenAlex ID: W4385574223