Year: 2025
Type: conference-paper
Abstract: Human annotations in natural language corpora vary due to differing human perspectives.This is especially prevalent in subjective tasks.In these datasets, certain data samples, i.e. annotatable instances, are more prone to label variation and can be indicated as ambiguous.This paper investigates methodologies for quantifying such label ambiguity by leveraging uncertainty estimation techniques when fine-tuning transformer-based models.We conducted experiments on three tasks characterized by subjective content and inherent label ambiguity: classifying sentiment, emotions and hate speech.The selected datasets include multiannotator labels, which we use to derive a label ambiguity score for each data sample.This score is the entropy of the empirical probability distribution of annotator labels.The results indicate that uncertainty estimation techniques can measure label ambiguity to some extent.Deep Ensembles consistently outperform other techniques, increasing the correlation coefficients between model uncertainty and annotator disagreement, but the observed correlations are low.When comparing the annotator label distributions with the predicted class distributions, we see that Label Smoothing is able to notably reduce this difference, however a discrepancy still exists.This suggests that uncertainty estimation techniques improve the quantification of label ambiguity, however their ability remains limited, highlighting the need for further research 1 .
Source: Linguistic Annotation Workshop
Authors: Richard Alies, Elena Merdjanovska, Alan Akbik
Institution: Humboldt-Universität zu Berlin
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: Deutsche Forschungsgemeinschaft
OpenAlex ID: W4412944259