Condor: A Neural Connection Network for Enhanced Attention based on Generalized Local Analysis
Year: 2025
Type: article
Abstract: The attention mechanism in traditional neural networks relies on pairwise interactions between tokens, limiting its ability to capture complex, multi-token relationships. This study introduces Condor, a novel architecture that extends theattention mechanism through a neural connection network based on the GeneralizedLocal Analysis (GLA) framework. Our approach replaces static attention patternswith learnable basis functions that dynamically model relationships within a local window. The Condor architecture achieves linear computational complexity ofO(LWH) while maintaining the expressive power to capture sophisticated sequencepatterns. Experimental results on wikitext-2 demonstrate improved perplexity andfaster convergence compared to the standard Transformer, confirming that each attention head learns unique connection patterns specializing in different aspects of sequence modeling. Code is available at: https://github.com/Kim-Ai-gpu/Condor
Author: Kim, Youngseong
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7078727401
FWCI:
Cites:
Cited by:
Related to: