Year: 2024
Type: preprint
Abstract: Ensuring that generative AI systems align with human values is essential but challenging, especially when considering multiple human values and their potential trade-offs. Since human values can be personalized and dynamically change over time, the desirable levels of value alignment vary across different ethnic groups, industry sectors, and user cohorts. Within existing frameworks, it is hard to define human values and align AI systems accordingly across different directions simultaneously, such as harmlessness, helpfulness, and positiveness. To address this, we develop a novel, first-principle approach called Multi-Human-Value Alignment Palette (MAP), which navigates the alignment across multiple human values in a structured and reliable way. MAP formulates the alignment problem as an optimization task with user-defined constraints, which define human value targets. It can be efficiently solved via a primal-dual approach, which determines whether a user-defined alignment target is achievable and how to achieve it. We conduct a detailed theoretical analysis of MAP by quantifying the trade-offs between values, the sensitivity to constraints, the fundamental connection between multi-value alignment and sequential alignment, and proving that linear weighted rewards are sufficient for multi-value alignment. Extensive experiments demonstrate MAP's ability to align multiple values in a principled manner while delivering strong empirical performance across various tasks.
Source: arXiv (Cornell University)
Authors: Wang, Xinran, Le, Qi, Ahmed, Ammar, Diao, Enmao, Zhou, Yi, Baracaldo, Nathalie, Ding, Jie, Anwar, Ali
Institutions: University of Minnesota, IBM (United States)
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W4404311186