Work

The Pluralistic Moral Gap: Understanding Moral Judgment and Value Differences between Humans and Large Language Models

Year: 2026

Type: conference-paper

Abstract: People increasingly rely on Large Language Models (LLMs) for moral advice, but little is known about how closely LLMs align with human moral judgments.To address this, we introduce the Moral Dilemma Dataset, a benchmark of 1,618 real-world moral dilemmas paired with a distribution of human moral judgments consisting of a binary evaluation and a free-text rationale.We treat moral advice as a pluralistic distributional alignment task, comparing the distributions of LLM and human judgments across dilemmas.We find that models reproduce only high-consensus human judgments while alignment deteriorates sharply when human disagreement increases.Further, using a 60-value taxonomy built from 3,783 value expressions extracted from rationales, we show that LLMs rely on a narrower set of moral values than humans.These findings reveal a pluralistic moral gap-a mismatch in both the distribution and diversity of values expressed.To close this gap, we introduce Dynamic Moral Profiling (DMP), a Dirichletbased sampling method that conditions model outputs on human-derived value profiles.DMP improves alignment by 64.3% and enhances value diversity, offering a step toward more pluralistic and human-aligned moral advice.Judgment: You're in the wrong Rationale: If you can afford it without putting your kids at risk … Is it okay to not pay for my brother's graduate school?My brother got into grad school and asked me to help pay.I can afford it but I have two kids […] 󰫷 󰠢 right Moral judgment alignment.vs Human Evaluation LLM Evaluation wrong right wrong Judgment: You're in the right Rationale: You have a greater obligation towards your kids … Judgment: You're in the wrong Rationale: Family comes first.This happens once in a lifetime … Judgment: You're in the wrong Rationale: You should help him to be a role model for your kids … … … Real-world moral dilemmas Moral value alignment based on rationales.vs most prevalent values:

Source: Conference of the European Chapter of the Association for Computational Linguistics

DOI: 10.18653/v1/2026.eacl-long.305

Authors: Giuseppe Russo, Debora Nozza, Paul Röttger, Dirk Hovy

Institutions: École Polytechnique Fédérale de Lausanne, Bocconi University, University of Oxford

Language: English

Topic: Ethics and Social Impacts of AI

Subfield: Safety Research

Field: Social Sciences

Domain: Social Sciences

Open Access status: hybrid

Funders: European Commission, Università Bocconi

OpenAlex ID: W7140081780