Year: 2025
Type: conference-paper
Abstract: As precursor work in preparation for an international standard ISO/PWI 24617-16 Languageresource management – Semantic annotation – Part 16: Evaluative language, we aim to testand enhance the reliability of the annotationof subjective evaluation based on AppraisalTheory. We describe a comprehensive three- phase workflow tested on COVID-19 mediareports to achieve reliable agreement throughprogressive training and quality control. Ourmethodology addresses some of the key challenges through the refinement of targeted guideline refinements and the development of inter- active clarification tools, alongside a customplatform that enables the pre-classification ofsix evaluative categories, systematic annotationreview, and organized documentation. Wereport empirical results that demonstrate substantial improvements from the initial moderateagreement to a strong final consensus. Ourresearch offers both theoretical refinementsaddressing persistent classification challengesin evaluation and practical solutions for theimplementation of the annotation workflow, proposing a replicable methodology for theachievement of reliable annotation consistencyin the annotation of evaluative language. ©2025 the authors
Source: CityU Scholars
Institutions: City University of Hong Kong, China National Institute of Standardization, Tilburg University, Université Marie et Louis Pasteur, Beihang University, South China Normal University, Liaocheng University, Korea University, Fudan University, Brandeis University, Institut national de recherche en sciences et technologies du numérique
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G4343169309, G6997231746, G8956687605
OpenAlex ID: W4416029215
FWCI:
Cites:
Cited by:
Related to: