Year: 2026
Type: dissertation
Abstract: When outcomes depend on human actions or interpretations, achieving effective and trustworthy AI performance requires systems that can reason about human behavior, communication, and social context. Such settings arise in applications including recommendation systems, public health monitoring, and policy communication, where understanding how people express, interact, and make decisions is essential. However, modeling human-centered data remains challenging due to its multimodal, relational, noisy, and often imbalanced nature. Recent advances in large language models (LLMs) and multimodal large language models (MLLMs) provide powerful foundations for reasoning over complex data. Yet, these models are not inherently aligned with the richness of human expression or the structure of social interactions. This gap motivates the need for methods that can better represent what humans express and how they are connected. This thesis develops computational frameworks for building human-aligned intelligent systems through representation augmentation and structuring. First, we propose modality-based augmentation methods that enrich representations across language, visual, and behavioral signals, enabling models to capture nuanced human expression. Second, we introduce graph-based augmentation techniques that model relationships among individuals, content, and communities, allowing AI systems to reason over social and semantic networks. Third, we integrate modalities and graphs to enable joint reasoning over expression and connections, improving performance in tasks involving complex human-centered data. Beyond modeling, this thesis presents alignment evaluation frameworks that assess how well AI systems reason across diverse contexts, including cultural and linguistic settings. These evaluations emphasize both aggregate performance and stratified analyses to uncover disparities in how models represent different groups. By addressing challenges in representation and evaluation, this thesis advances the development of human-aligned intelligent systems that are not only more accurate but also more robust, interpretable, and socially aware. Finally, we discuss promising directions for future research in aligning AI with human values and behaviors.
Authors: Lyu, Hanjia, Luo, Jiebo
Institution: University of Rochester
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7165316455
