Year: 2025
Type: dissertation
Abstract: Morpheus improves the performance and throughput of serving diffusion transformer models (DiTs) for high-resolution image generation by strategically distributing computation across GPUs and adaptively skipping less salient regions.Unlike Large Language Models (LLMs), which are often I/O bound, image generation models like DiTs are predominantly compute-bound, especially at high resolutions.Furthermore, different regions within an image generation task exhibit varying levels of computational difficulty and semantic importance.To address these challenges, we introduce several technical contributions in Morpheus.First, we leverage a cross-attention-based analysis to dynamically identify and partition the image into salient and less salient regions.Second, we develop an algorithm that partitions the total computation into sub-tasks, considering both regional saliency and available compute resources.Third, we propose an adaptive skip algorithm that dynamically decides, on-the-fly during the diffusion process, when to skip computation for less salient regions without significant quality loss.Fourth, a centralized task scheduler dynamically assigns sub-tasks to workers, intelligently filling potential idle time arising from skipping with sub-tasks from other queued requests to boost overall throughput.Finally, we employ an asynchronous communication pattern for KV cache management and migration, crucial for enabling efficient parallel execution and hiding communication latency behind computation.Our evaluation of Morpheus serving the Flux model on 4x A100 40GB GPUs demonstrates a 2.1x speed-up compared to single-GPU execution, while maintaining minimal image quality degradation.Completing this Master's thesis marks the culmination of a challenging yet rewarding journey through the MSCS program at UIUC.Over the past two years, I have been fortunate to receive invaluable guidance from my advisors and collaborators, and unwavering support from my family and friends.I wish to express my sincere gratitude to everyone who has contributed to my academic and personal development during this time.My deepest gratitude goes to my advisor, Professor Fan Lai.His invaluable guidance, consistent support, and encouragement have been instrumental throughout my research journey.Professor Lai's expertise and insights were crucial in shaping this research and navigating the complexities of graduate studies.I was fortunate to first work with him on the FedScale project during his PhD, an experience that ignited my passion for systems research and development.Upon his arrival at UIUC, he welcomed me into his group and entrusted me with leading a project aligned with my interests.I am also grateful for the opportunity to serve as a Teaching Assistant for his "Systems for Generative AI" course; this role provided departmental funding and allowed me the rewarding experience of teaching an introduction to diffusion models.Professor Lai consistently demonstrated respect, care, and support, both academically and personally.He fostered an environment where my interests were valued, granting me significant autonomy while always providing insightful advice when needed.Beyond our lab work, he treated me as a friend, showing genuine concern for my well-being and career path.He is truly an exceptional mentor, and I sincerely hope our collaboration can continue beyond my time at UIUC.I am also grateful to my previous advisor, Professor Daniel Kang.He played a crucial role in my admission to UIUC by providing a strong recommendation and generously funded my first year as promised.I appreciate his detailed and timely feedback and responsiveness while working together.The initial start task from him offered me the opportunity to gain foundational knowledge on Stable Diffusion, which proved beneficial for my subsequent open-source contributions.I thank him for his support during the initial phase of my studies.This thesis would not have been possible without the contributions of my collaborators.The primary research presented here benefited greatly from joint work with Songrun Xie, Ping He, Songhao Zhang, and Zhengzhong Tu, alongside Professor Lai.I am also grateful for the experience of collaborating on earlier projects: AIDB [1] with Akash Muttal and Tengjun Jin (advised by Professor Kang), and PilotDB [2] with Yuxuan Zhu and Tengjun Jin (also advised by Professor Kang).These collaborations were not only productive but
Author: Zhang, Chengsong
Institution: University of Illinois Urbana-Champaign
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7220756219