Year: 2026
Type: conference-paper
Abstract: We introduce softpick, a rectified, not sum-toone, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations.Our experiments with 340M and 1.8B parameter models demonstrate that softpick achieves 0% sink rate consistently.The softpick transformers produce hidden states with significantly lower kurtosis and creates sparse attention maps.Quantized models using softpick outperform softmax on standard benchmarks, with a particularly pronounced advantage at lower bit precisions.Our analysis and discussion shows how softpick has the potential to open new possibilities for quantization, low-precision training, sparsity optimization, pruning, and interpretability.Our code: https://github.com/ zaydzuhri/softpick-attention. * Equal contributionSink Rate = 63.41%Sink Rate = 0.00%
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4417172974