Year: 2020
Type: conference-paper
Abstract: Low-resource language translation is a challenging but socially valuable NLP task.Building on recent work adapting the Transformer's normalization to this setting, we propose QKNORM, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity.Specifically, we apply ℓ 2 normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension.We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15. 1
Authors: Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, Yuxuan Chen
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W3099096615