Year: 2012
Type: conference-paper
Abstract: This article details work aiming at evaluating the quality of the manual annotation of gene renaming couples in scientific abstracts, which generates sparse annotations.To evaluate these annotations, we compare the results obtained using the commonly advocated inter-annotator agreement coefficients such as S, κ and π, the less known R, the weighted coefficients κω and α as well as the F-measure and the SER.We analyze to which extent they are relevant for our data.We then study the bias introduced by prevalence by changing the way the contingency table is built.We finally propose an original way to synthesize the results by computing distances between categories, based on the produced annotations.
Institutions: Centre National de la Recherche Scientifique, Institut de l'Information Scientifique et Technique, Université Sorbonne Paris Nord, Laboratoire d'Informatique de Paris-Nord, Conservatoire National des Arts et Métiers, Laboratoire National de Métrologie et d'Essais, Laboratoire commun de métrologie
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W646753399