Year: 2020
Type: conference-paper
Abstract: We focus on the recognition of Dyck-n (D n ) languages with self-attention (SA) networks, which has been deemed to be a difficult task for these networks.We compare the performance of two variants of SA, one with a starting symbol (SA + ) and one without (SA -).Our results show that SA + is able to generalize to longer sequences and deeper dependencies.For D 2 , we find that SA -completely breaks down on long sequences whereas the accuracy of SA + is 58.82%.We find attention maps learned by SA + to be amenable to interpretation and compatible with a stack-based language recognizer.Surprisingly, the performance of SA networks is at par with LSTMs, which provides evidence on the ability of SA to learn hierarchies without recursion.
Authors: Javid Ebrahimi, Dhruv Gelda, Wei Zhang
Institution: Visa (United States)
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W3099140684