Product
Help
Pricing
About
Jobs
Log in
Get started
Keyword
Add to collection
attention collapse
Description:
A failure mode in attention mechanisms where attention weights become uninformative or overly concentrated, reducing model performance or useful information flow.
Related topics:
Advanced Neural Network Applications (13%),
Topic Modeling (13%),
Medical Image Segmentation Techniques (7%)
OpenAlex ID:
keywords/attention-collapse
DeepViT: Towards Deeper Vision Transformer
2021
·
Zhou, Daquan
, Kang, Bingyi
, et al.
·
arXiv (Cornell University)
·
861
PDF
Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice
2022
·
Wang, Peihao
, Zheng, Wenqing
, et al.
·
arXiv (Cornell University)
·
259
PDF
ConvFormer: Plug-and-Play CNN-Style Transformers for Improving Medical Image Segmentation
2023
·
Xian Lin
, Zengqiang Yan
, et al.
·
Lecture notes in computer science
·
135
PDF
What Do Self-Supervised Vision Transformers Learn?
2023
·
Park, Namuk
, Kim, Wonjae
, et al.
·
arXiv (Cornell University)
·
126
PDF
Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS
2019
·
Mutian He
, Yan Deng
, et al.
·
Interspeech (USB)
·
94
PDF
EISATC-Fusion: Inception Self-Attention Temporal Convolutional Network Fusion for Motor Imagery EEG Decoding
2024
·
Guangjin Liang
, Dianguo Cao
, et al.
·
IEEE Transactions on Neural Systems and Rehabilitation Engineering
·
56
PDF
Backdoor Cleaning without External Guidance in MLLM Fine-tuning
2025
·
Xuankun Rong
, Wenke Huang
, et al.
·
neural information processing systems
·
48
PDF
Distilled transformers with locally enhanced global representations for face forgery detection
2024
·
Yaning Zhang
, Qiufu Li
, et al.
·
Pattern Recognition
·
40
PDF
Transformer-based hierarchical dynamic decoders for salient object detection
2023
·
Qingping Zheng
, Ling Zheng
, et al.
·
Knowledge-Based Systems
·
28
DTMFormer: Dynamic Token Merging for Boosting Transformer-Based Medical Image Segmentation
2024
·
Zhehao Wang
, Xian Lin
, et al.
·
Proceedings of the AAAI Conference on Artificial Intelligence
·
25
PDF
Dual Channel‐Spatial Self‐Attention Transformer and CNN synergy network for 3D medical image segmentation
2024
·
Fan Yang
, Bo Wang
·
Applied Soft Computing
·
24
CRAT: Advanced transformer-based deep learning algorithms in OCT image classification
2025
·
Mingming Yang
, Junhui Du
, et al.
·
Biomedical Signal Processing and Control
·
20
When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models
2026
·
Sanyal, Sunny
, Shwartz-Ziv, Ravid
, et al.
·
Transactions on machine learning research
·
16
Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks
2026
·
Yuval Ran-Milo
·
Proceedings of the conference - Association for Computational Linguistics. Meeting
·
15
PDF
Background Suppression Network With Attention Collapse Inhibited Transformer for Optical Remote Sensing Object Detection
2024
·
Jiaojiao Li
, Haile Li
, et al.
·
IEEE Transactions on Geoscience and Remote Sensing
·
15
Prediction-Feedback DETR for Temporal Action Detection
2025
·
Jihwan Kim
, Miso Lee
, et al.
·
Proceedings of the AAAI Conference on Artificial Intelligence
·
12
PDF
A lightweight multi-scale multi-angle dynamic interactive transformer-CNN fusion model for 3D medical image segmentation
2024
·
Xin Hua
, Zhijiang Du
, et al.
·
Neurocomputing
·
12
Delta Attention Residuals
2026
·
Luo, Cheng
, Cai, Zefan
, et al.
·
arXiv (Cornell University)
·
10
PDF
FOCUS: Fused Observation of Channels for Unveiling Spectra
2025
·
Xiao, Xi
, Tsaris, Aristeidis
, et al.
·
arXiv (Cornell University)
·
10
PDF
Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
2026
·
Liu, Feilong
·
arXiv (Cornell University)
·
9
PDF
Long-term pre-training for temporal action detection with transformers
2025
·
Jihwan Kim
, Miso Lee
, et al.
·
Pattern Recognition
·
9
Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners
2023
·
Shi, Bowen
, Zhang, Xiaopeng
, et al.
·
International Conference on Learning Representations
·
9
Analyzing the Mechanism of Attention Collapse in VGGT from a Dynamics Perspective
2025
·
Li, Huan
, Luo, Longjun
, et al.
·
arXiv (Cornell University)
·
8
PDF
Multi-Scale Residual Spectral–Spatial Attention Combined with Improved Transformer for Hyperspectral Image Classification
2024
·
Aili Wang
, Kang Zhang
, et al.
·
Electronics
·
8
PDF
SpatialReward: Bridging the Perception Gap in Online RL for Image Editing via Explicit Spatial Reasoning
2026
·
Long, Yancheng
, Yang, Yankai
, et al.
·
arXiv (Cornell University)
·
7
PDF
Show more works
View as search
Works count:
105
Citation count:
1,924
Year
Topic
Type