Work

Enhancing Peer Fairness via Data-Driven Analysis for Outlier Detection

Year: 2024

Type: article

Abstract: Fairness in peer review is of vital importance in academic activities. Current peer review systems focus on matching suitable experts with proposals but often ignore the existence of outliers. Previous research has shown that outlier scores in reviews could decrease the fairness of these systems. Therefore, outlier detection in peer review systems is essential for maintaining fairness. In this paper, we introduce a novel method that employs data-crossing analysis to detect outlier scores, aiming to improve the reliability of peer review processes. We utilize a confidential dataset from a review organization. Due to the inability to access ground truth scores, we systematically devise data-driven deviations from an estimated ground truth through data-crossing analysis. These deviations reveal inconsistencies and abnormal scoring behaviors of different reviewers. Subsequently, the review process is strengthened by providing a structured mechanism to identify and mitigate biases. Extensive experiments demonstrate its effectiveness in improving the accuracy and fairness of academic assessments, contributing to the broader application of AI-driven methodologies to achieve more reliable and equitable outcomes.

Source: Electronics

DOI: 10.3390/electronics13234735

Authors: Zhengkun Di, Jinqiannan Zhang, Weixing Tan, Xiaoqi Sun

Institutions: Shandong University, China Science and Technology Exchange Center

Language: English

Topic: Ethics and Social Impacts of AI

Subfield: Safety Research

Field: Social Sciences

Domain: Social Sciences

SDG: Reduced inequalities

Open Access status: gold

Estimated APC paid: $2,728