Year: 2026
Type: conference-paper
Abstract: The Retentive Network (RetNet) has recently emerged as a formidable successor to the Transformer architecture.Although the self-attention mechanism excels at capturing global dependencies, its inherent quadratic complexity imposes significant memory constraints and inhibits scalability during long-sequence modeling.To overcome these challenges, RetNet introduces an innovative retention mechanism that integrates the inductive bias of recurrent neural networks with the parallelizable training advantages of attention-based models.This unified representation allows RetNet to achieve constant-time inference and linear-time training without sacrificing representational capacity.Despite the growing body of research demonstrating the efficacy of RetNet across diverse fields such as natural language processing, computer vision, and time-series analysis, a systematic synthesis of the current literature is currently unavailable.This paper presents the first comprehensive survey of Retentive Networks through a detailed examination of its architectural foundations, core innovations, and specialized variants.Furthermore, we provide a multidisciplinary analysis of its applications ranging from basic sequence tasks to complex crossmodal scenarios.Finally, we offer prospective insights and suggest strategic avenues for future inquiry to facilitate the continued evolution of RetNet in both academic research and largescale industrial applications.
Authors: Haiqi Yang, Zhiyuan Li, Yi Chang, Yuan Wu
Institution: Jilin University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4417119786