Year: 2025
Type: conference-paper
Abstract: Long-context understanding is crucial for many NLP applications, yet transformers struggle with efficiency due to the quadratic complexity of self-attention.Sparse attention methods alleviate this cost but often impose static, predefined masks, failing to capture heterogeneous attention patterns.This results in suboptimal token interactions, limiting adaptability and retrieval accuracy in long-sequence tasks.This work introduces a dynamic sparse attention mechanism that assigns adaptive masks at the attention-map level, preserving heterogeneous patterns across layers and heads.Unlike existing approaches, our method eliminates the need for fine-tuning and predefined mask structures while maintaining computational efficiency.By learning context-aware attention structures, it achieves high alignment with full-attention models, ensuring minimal performance degradation while reducing memory and compute overhead.This approach provides a scalable alternative to full attention, enabling the practical deployment of largescale Large Language Models (LLMs) without sacrificing retrieval performance.
Authors: Hanzhi Zhang, Heng Fan, Kewei Sha, Yan Huang, Yunhe Feng
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412888706