Get started
Work

Contrastive Attention Transformer for Classification

Year: 2026
Type: dissertation
Abstract: Most traditional classification methods process each data point in isolation, ignoring the structural information embedded in the relationships among data points. This dissertation develops the contrastive attention transformer, a novel classification architecture that attends a given point separately to each candidate class, producing one representation per class, and predicts the label by contrasting these representations through a shared classifier. We develop the framework for both binary and multiclass classification, proposing three loss functions under two complementary approaches. The comparative approach enforces the relative ordering of confidences across representations through the probability difference and proportional losses. The aggregation approach maximizes joint correctness through the conditional probability loss, which we show to coincide with a forecast aggregation model from decision analysis and to be equivalent to classification on the sum of the representations, motivating a flexible concatenation architecture. A key-value caching scheme further reduces the computational cost of attending to each class separately by 50 to 60 percent for large numbers of classes. We evaluate the framework on five binary classification tasks, four multiclass classification tasks, and a controlled CIFAR-100 experiment, spanning image, text, and tabular data. The best contrastive variant outperforms a feedforward network baseline and a standard transformer baseline with unrestricted attention on every binary and multiclass task. The CIFAR-100 experiment identifies the distinguishability of the classes, rather than the number of classes, as the primary driver of the contrastive method's advantage over unrestricted attention.
DOI: 10.7282/t3-ger4-v617
Author: Bai, Yunjiao
Language: English
Open Access status: green
OpenAlex ID: W7220982118
Cites:
Cited by:
Related to: 10