Work

Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning

Year: 2018

Type: conference-paper

Abstract: We present a study on reinforcement learning (RL) from human bandit feedback for sequence-to-sequence learning, exemplified by the task of bandit neural machine translation (NMT).We investigate the reliability of human bandit feedback, and analyze the influence of reliability on the learnability of a reward estimator, and the effect of the quality of reward estimates on the overall RL task.Our analysis of cardinal (5-point ratings) and ordinal (pairwise preferences) feedback shows that their intra-and inter-annotator αagreement is comparable.Best reliability is obtained for standardized cardinal feedback, and cardinal feedback is also easiest to learn and generalize from.Finally, improvements of over 1 BLEU can be obtained by integrating a regressionbased reward estimator trained on cardinal feedback for 800 translations into RL for NMT.This shows that RL is possible even from small amounts of fairly reliable human feedback, pointing to a great potential for applications at larger scale.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/p18-1165

Authors: Julia Kreutzer, Joshua Uyheng, Stefan Riezler

Institutions: Heidelberg University, Ateneo de Manila University

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Deutsche Forschungsgemeinschaft

OpenAlex ID: W2799082722