Year: 2021
Type: conference-paper
Abstract: A supervised self-attention network is proposed to introduce the dependency structure into the Transformer model. This scheme is mainly proposed based on the characteristics of the self-attention mechanism in Transformer, which converts the dependency syntax tree into two equivalent adjacency matrices, and then uses the adjacency matrix to supervise the self-attention network in the Transformer's encoder. So that Transformer learns how to model the dependency structure of the source language. Although this scheme is simple, the effect is remarkable. In particular, there is no need to modify the network structure, as long as two additional loss functions are added in the process of training Transformer. In the decoding process, there is no need to use an external syntactic analyzer to perform syntactic analysis on the source language, but it can automatically construct the dependent syntactic structure of the source language. Experiments demonstrate that the quality of the constructed dependency syntactic structure is also good.
Authors: Ting Yang, Shinan Zhao, He Chen, Ting Liu
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W3206161508