Work

“Provably fair” algorithms may perpetuate racial and gender bias: a study of salary dispute resolution

Year: 2025

Type: book-chapter

Abstract: Prior work suggests automated dispute resolution tools using “provably fair” algorithms can address disparities between demographic groups. These methods use multi-criteria elicited preferences from all disputants and satisfy constraints to generate “fair” solutions. However, we analyze the potential for inequity to permeate proposals through the preference elicitation stage. This possibility arises if differences in dispositional attitudes differ between demographics, and those dispositions affect elicited preferences. Specifically, risk aversion plays a prominent role in predicting preferences. Risk aversion predicts a weaker relative preference for salary and a softer within-issue utility for each issue; this leads to worse compensation packages for risk-averse groups. These results raise important questions in AI-value alignment about whether an AI mediator should take explicit preferences at face value.

Source: Autonomous Agents and Multi-Agent Systems

DOI: 10.1007/s10458-025-09703-x

Authors: James Hale, Peter H. Kim, Jonathan Gratch

Institution: University of Southern California

Language: English

Topic: Ethics and Social Impacts of AI

Subfield: Safety Research

Field: Social Sciences

Domain: Social Sciences

SDGs: Reduced inequalities, Peace, justice, and strong institutions

Open Access status: hybrid

Estimated APC paid: $3,290