Automated Topic Annotation in Brazilian Product Reviews: A Case Study of Adversarial Examples with Sabia-3
Year: 2024
Type: conference-paper
Abstract: High-quality annotated data is essential for many Natural Language Processing tasks, but traditional human annotation methods are often resource-intensive. Large Language Models (LLMs) offer potential solutions by generating labels for training datasets. This paper explores the effectiveness of using the Sabiá-3 LLM for automatically labeling data for a multi-label topic classification task in Brazilian Portuguese product reviews. We compare the performance of Sabia-3-generated labels against human annotations using the RePro dataset. The study evaluates Sabiá-3 on both random and adversarial datasets, highlighting its strengths in frequent topics, while identifying limitations in more nuanced categories. Models trained on Sabiá-3 annotations showed promising results in common categories but faced challenges with ambiguous cases. Our findings suggest that while LLMs can streamline parts of the annotation process, human oversight remains essential, particularly in complex or less frequent cases. This research contributes new insights into the use of LLMs for automated data annotation in Brazilian Portuguese.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4405168440