Year: 2025
Type: preprint
Abstract: Transformers excel when dealing with sequential data. Generalizing transformer models to geometric domains, such as manifolds, we encounter the problem of not having a well-defined global order. We propose a solution with attention heads following a space-filling curve. As a first experimental example, we present the Spiroformer, a transformer that follows a polar spiral on the $2$-sphere.
Source: arXiv (Cornell University)
Authors: Maurin, M., Evangelista-Alvarado, M. Á., Suárez-Serrato, P.
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6979284435
Cites:
Cited by:
Related to: