Year: 2021
Type: article
Abstract: It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper. More generally, the disparate mapping of criteria scores to final recommendations by different reviewers is a major source of inconsistency in peer review. In this paper we present a framework inspired by empirical risk minimization (ERM) for learning the community's aggregate mapping. The key challenge that arises is the specification of a loss function for ERM. We consider the class of L(p,q) loss functions, which is a matrix-extension of the standard class of Lp losses on vectors; here the choice of the loss function amounts to choosing the hyperparameters p and q. To deal with the absence of ground truth in our problem, we instead draw on computational social choice to identify desirable values of the hyperparameters p and q. Specifically, we characterize p=q=1 as the only choice of these hyperparameters that satisfies three natural axiomatic properties. Finally, we implement and apply our approach to reviews from IJCAI 2017.
DOI: 10.1613/jair.1.12554
Institutions: Carnegie Mellon University, Harvard University
Language: English
Subfield: Economics and Econometrics
Domain: Social Sciences
Open Access status: diamond
Awards: G1283689559, CIF: Medium: Foundations of Learning from Paired Comparisons and Direct Queries, AF: Small: Fair Division at Scale, CAREER: A Broad Synthesis of Artificial Intelligence and Social Choice, RI: Small: Computational Social Choice: For the People, G4504108201, AitF: Algorithms and Mechanisms for Kidney Exchange, G6945895031, CRII: CIF: Crowdsourcing-aware Learning, G8876996369, G8929337198
OpenAlex ID: W3008478051