Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment
Year: 2026
Type: conference-paper
Abstract: Humans play a vital role at every stage of AI development, from data collection and curation to model development and evaluation.However, humans often disagree with each other and sometimes with themselves over time.It is essential to take disagreement into account when building human-centered AI systems, especially in domains where it is prevalent, such as AI safety, content moderation, or sentiment analysis.Disagreement often arises from subjective human opinion and can vary with one's identity, beliefs, and social environment.Despite this, current LLM evaluation approaches frequently rely on aggregating labels (often via plurality voting) to represent consensus, thereby obscuring minority perspectives.By failing to account for human disagreement, these evaluation methods contribute to the reproducibility crisis in AI.Human feedback is also crucial for ensuring that AI systems align with human values.For these systems to be trustworthy, it is critical to ensure that they reflect diverse human values and perspectives.In this thesis proposal, we present a humancentered and perspective-aware framework for reproducible ML evaluation and AI alignment.
Authors: Deepak Pandita, Christopher M. Homan
Institution: Rochester Institute of Technology
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W7166799601