Year: 2026
Type: conference-paper
Abstract: This work explores efficient ultra-long context modeling.We posit that an effective solution requires three fundamental properties: sparsity, random-access flexibility, and length generalization.To achieve this, we leverage Hierarchical Sparse Attention (HSA), a novel attention mechanism that satisfies all three properties.We integrate HSA into the Transformer architecture to develop HSA-UltraLong, an 8B-parameter Mixture-of-Experts (MoE) model trained on over 8 trillion tokens.We rigorously evaluate the model across tasks with both in-domain and out-of-domain context lengths to validate its capabilities.Our model demonstrates comparable performance to full-attention baselines on in-domain sequence lengths.Crucially, it achieves over 90% accuracy on most in-context retrieval tasks with contexts up to 512 times the pre-training context length.This work reports our findings and remaining issues throughout the experiments, offering insights for future research in ultralong context modeling.
Institution: Ant Group (China)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166801505
