Work

Annotator-Centric Active Learning for Subjective NLP Tasks

Year: 2024
Type: conference-paper
Abstract: Active Learning (AL) addresses the high costs of collecting human annotations by strategically annotating the most informative samples.However, for subjective NLP tasks, incorporating a wide range of perspectives in the annotation process is crucial to capture the variability in human judgments.We introduce Annotator-Centric Active Learning (ACAL), which incorporates an annotator selection strategy following data sampling.Our objective is two-fold:(1) to efficiently approximate the full diversity of human judgments, and (2) to assess model performance using annotator-centric metrics, which value minority and majority perspectives equally.We experiment with multiple annotator selection strategies across seven subjective NLP tasks, employing both traditional and novel, human-centered evaluation metrics.Our findings indicate that ACAL improves data efficiency and excels in annotator-centric performance evaluations.However, its success depends on the availability of a sufficiently large and diverse pool of annotators to sample from.
Language: English
Open Access status: hybrid
OpenAlex ID: W4404782575
FWCI: 3.292
Cites: 41
Cited by: 39
Related to: 10