Work

Heterogeneous machine learning with decentralized data

Year: 2025
Type: dissertation
Abstract: This thesis explores heterogeneous machine learning with decentralized data, where multiple clients with distinct data distributions jointly train or adapt machine learning models under the coordination of a central server.Throughout the process, clients' private data never leave their local devices.This paradigm underlies numerous real-world applications, such as collaborative training of financial fraud detection models among banks, or collective health monitoring enabled by massive wearable devices.We investigate three fundamental challenges in this setting.(P1) Effective model training: How can we train models from multiple source clients with heterogeneous labeled data so that models perform well across all clients?(P2) Adaptive model deployment: How can we adapt the trained model to each target client without labels, allowing it to adjust to its own data distribution for improved performance?(P3) Robust system design: How can we ensure robustness during both training and adaptation, preventing performance degradation caused by random failures or malicious attacks?To address (P1), we develop client clustering algorithms that enable knowledge transfer among clients with similar data distributions, allowing those with limited data to benefit from collaboration.For (P2), we first design task-specific adaptation algorithms that identify and mitigate distribution shifts across different modalities under one-to-one adaptation.We further extend to multi-client collaboration, where the model learns patterns of distribution shifts across clients to enable collaborative test-time adaptation.Finally, for (P3), we propose robust training algorithms resilient to abnormal or adversarial clients, and robust adaptation algorithms that mitigate model prediction bias and out-of-distribution data effects.Together, these contributions form an effective and robust framework for heterogeneous machine learning under decentralized data environments.First and foremost, I would like to express my deepest gratitude to my advisor, Dr. Jingrui He, for her unwavering guidance, support, and encouragement throughout my Ph.D. journey.The early years of my doctoral study unfolded under the shadow of the COVID-19 pandemic, when travel was restricted, communication was limited, and my first paper encountered multiple rejections.Facing difficulties in both life and research, Dr.He never gave up on me.Instead, she always believed in me, comforted me during setbacks, and encouraged me to move forward with optimism and persistence.Without her support, I would not have been able to complete my Ph.D. Dr.He has influenced me far beyond research itself.She is a true mentor in life, teaching me not only how to think critically and work rigorously, but also how to face challenges with courage and composure.Her famous advice, "Act like you want it!",has become a personal motto that continually reminds me to take ownership of my goals and strive for excellence.At the same time, she also taught us the importance of balance: to slow down, relax, and appreciate life beyond research.These lessons, both in spirit and in practice, have had a profound and lasting impact on me, shaping the way I approach my career and life ahead.I would also like to express my sincere appreciation to my thesis committee members, Dr. Pan Li, Dr. Tong Zhang, and Dr. Han Zhao, for their invaluable feedback and constructive suggestions, which significantly improved the quality and depth of this dissertation.Their insights helped me refine my research questions, sharpen my theoretical analysis, and broaden my perspective on the field of machine learning.I am equally grateful to Dr. Hanghang Tong, who has provided me with insightful advice and guidance in both research and academic growth.I would also like to thank Dr. Haohan Wang for his insightful research advice and valuable feedback, which have greatly influenced the direction and refinement of my work.My heartfelt appreciation goes to all my collaborators and every member of the iSAIL Lab and IDEA Lab at UIUC.A Ph.D. is never a solitary journey, and it is the people around me who have made this experience meaningful and fulfilling.I have been fortunate to collaborate with talented labmates and friends whose curiosity, dedication, and creativity constantly pushed me to improve.Our discussions-sometimes intense, sometimes playful-taught me how to exchange ideas openly and how to grow together as researchers.Beyond academics, their friendship made life at UIUC much more enjoyable.Whether it was last-minute coding for assignments, long nights of paper writing before deadlines, or weekend dinners that turned into long conversations, these moments have become some of my fondest memories.iii Because of all of you, the long winters in Urbana-Champaign felt much warmer and brighter.I would also like to express my deepest gratitude to my parents, Jun Cao and Juwu Bao, who have been the most consistent source of love and strength in my life.They have supported me unconditionally through every decision and every challenge, always reminding me that there is nothing to fear and that I will always have a home to return to.Their faith in me has given me the courage to explore the world and pursue what I truly love.Finally, to my partner, Rong Fu, thank you for your endless patience, understanding, and love.During most of my Ph.D. years, we lived apart, but you later chose to come to the United States and accompany me through this journey.Your presence has been my greatest comfort and my happiest memory.You stood by me through the toughest moments of research, celebrated every small victory, and reminded me to take breaks and enjoy life along the way.Sharing this chapter of my life with you has made all the difference.I am deeply grateful for your companionship and for the future we will continue to build together.regulations [23].At the personal level, similar challenges arise in large-scale networks of user devices.For instance, smartphones capture vast collections of user-generated photos that could be used to fine-tune or personalize vision models, but such images often contain private or locationsensitive content.Likewise, wearable devices continuously monitor health indicators such as heart rate, blood oxygen, and physical activity.Leveraging these decentralized data could enable early detection of health anomalies or personalized wellness recommendations, yet transmitting raw signals to a central server poses serious privacy and communication concerns.Together, these examples highlight the necessity of developing machine learning systems that can learn effectively and robustly under decentralized, heterogeneous, and privacy-constrained environments. KEY CHALLENGESWhile this paradigm offers significant potential, it also introduces several key challenges that fundamentally differentiate it from conventional centralized learning.C1: Heterogeneity A fundamental assumption of traditional ML is homogeneity, meaning that training and testing data come from the same underlying distribution.However, in decentralized settings, each client typically has a unique data distribution, leading to distribution shifts when models are trained and deployed across different clients.These shifts can take various forms [15,24], such as label shift, feature shift, and concept shift, each posing distinct challenges to model training and generalization.• Label shift occurs when the marginal label distribution P (y) differs across clients.For example, in a medical imaging federation, one hospital may have a higher proportion of positive disease cases than another, causing the global model to overfit to the majority label distribution.• Feature shift refers to differences in the marginal feature distribution P (x).For instance, in mobile-device learning, photos captured under different lighting conditions or geographical regions can exhibit substantially different visual statistics, reducing the effectiveness of a globally shared vision model.• Concept shift arises when the conditional distributions P (y|x) or P (x|y) vary across clients.For example, a resting heart rate of 40 beats per minute may indicate an abnormal condition for an average individual but is perfectly normal for a trained athlete.
Author: Bao, Wenxuan
Language: English
Open Access status: green
OpenAlex ID: W7220782764
Cites: 153
Cited by:
Related to: 10