Year: 2026
Type: conference-paper
Abstract: Recent work has shown that LLMs develop internally coherent utility functions that emerge with scale, yet whether these value systems encode systematic demographic hierarchies remains unexplored.We elicit pairwise preferences across 15 intersectional demographic groups (defined by race, gender, and their combinations) and 8 policy domains on three 7-8B instruction-tuned LLMs, fitting Thurstonian utility models to the resulting preference matrices.All three models converge on a compensatory hierarchy that inverts real-world structural advantage, consistently ranking marginalized groups, the highest and dominant groups are lowest.Intersectional utilities do not combine additively: single-axis audits that measure gender and race gaps independently overestimate the most extreme intersectional gap by 26-40% in our experiments.Geometrically, we identify a linear direction in the representation space that predicts the full utility hierarchy from neutral sentences alone, and show that this direction is substantially aligned with gender encoding but not with race encoding.Orthogonalization reveals that gender separation in representations is not fully explained by utility encoding.The hierarchy is already present in base (pre-alignment) models and is amplified several-fold by instruction tuning, suggesting it originates in pre-training data rather than alignment procedures.
Author: Pravish Sainath
Institution: Université de Montréal
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
SDG: Reduced inequalities
Open Access status: hybrid
OpenAlex ID: W7166848366