Melhorando a seleção de anotadores em Active Learning usando um Sistema de Recomendação Sensível à Disposição e à Fadiga
Year: 2025
Type: dissertation
Abstract: This study centers on overcoming the challenge of selecting the best annotators for each query in Active Learning (AL), with the objective of minimizing misclassifications. AL recognizes the challenges related to cost and time when acquiring labeled data, and decreases the number of labeled data needed. Nevertheless, there is still the necessity to reduce annotation errors, aiming to be as efficient as possible, to achieve the expected accuracy faster [Munro, 2021]. While multiple strategies for query-annotator pairs selection consider the imperfect nature of human annotators, and try to overcome this problem (such as proactive AL [Bahle et al., 2016; Calma et al., 2016] and collaborative interactive learning [Calma et al., 2016, 2018a; Donmez et al., 2008; Donmez and Carbonell, 2010]), most do not consider internal factors that affect productivity. Some of these factors are mood, attention, motivation, and fatigue levels, which have been shown to affect productivity [Caldwell et al., 2019; Calma et al., 2016; Fredrickson and Branigan, 2005; Tenney, Poole, and Diener, 2016].This work addresses this gap in the existing literature, by not only considering how the internal factors influence annotators (mood and fatigue levels) but also presenting a new query-annotator pair strategy, using a Knowledge-Based Recommendation System (RS). The RS ranks the available annotators, allowing to choose one or more to label the queried instance using their past accuracy values, and their mood and fatigue levels, as well as information about the instance queried. This work bases itself on existing literature on mood and fatigue influence on human performance, simulating annotators in a realistic manner, and predicting their performance with the RS.The results show that considering past accuracy values, as well as mood and fatigue levels reduces the number of annotation errors made by the annotators, and the uncertainty of the model through its training, when compared to not using internal factors. Accuracy and F1-score values were also better in the proposed approach, despite not being as substantial as the aforementioned. The methodologies and findings presented in this study begin to explore the open challenge of human cognitive factors affecting AL.
Author: Mortágua, Diana Maria Conceição
Institution: University of Coimbra
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7112784624
