Work

Multi-Head Attention with Disagreement Regularization

Year: 2018
Type: conference-paper
Abstract: Multi-head attention is appealing for the ability to jointly attend to information from different representation subspaces at different positions.In this work, we introduce a disagreement regularization to explicitly encourage the diversity among multiple attention heads.Specifically, we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads.Experimental results on widely-used WMT14 English⇒German and WMT17 Chinese⇒English translation tasks demonstrate the effectiveness and universality of the proposed approach.* Zhaopeng Tu is the corresponding author of the paper.This work was mainly conducted when Jian Li and Baosong Yang were interning at Tencent AI Lab.
Language: English
Open Access status: hybrid
OpenAlex ID: W2962822108
FWCI: 8.791
Cites: 27
Cited by: 235
Related to: 10