Work

SSA: Improving Performance With a Better Scoring Function

Year: 2026

Type: conference-paper

Abstract: While transformer models exhibit strong incontext learning (ICL) abilities, they often fail to generalize under simple distribution shifts.We analyze these failures and identify Softmax, the scoring function in the attention mechanism, as a contributing factor.We propose Scaled Signed Averaging (SSA), a novel attention scoring function that mitigates these failures.SSA significantly improves performance on our ICL tasks and outperforms transformer models with Softmax on several NLP benchmarks and linguistic probing tasks, in both decoder-only and encoder-only architectures.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2026.acl-long.1385

Authors: Omar Naim, Swarnadeep Bhar, Jerome Bolte, Nicholas Asher

Institutions: Institut de Recherche en Informatique de Toulouse, Université de Toulouse, Toulouse School of Economics, Centre National de la Recherche Scientifique

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Agence Nationale de la Recherche

Awards: G3010527862, G3778207229, G3879958799, G602161992, Distant Supervision for Meeting Minutes with Rhetorical Relations