Work

Learning-based saliency-aware compression framework

Year: 2022
Type: dissertation
Abstract: Networked vision systems are prevalent nowadays due to the dominance of video traffic over the Internet.Videos in the networked vision system are transmitted over the Internet to a vision application, which processes videos by deep neural networks (DNN), i.e., computation offloading, or presents videos to human viewers, i.e., video streaming.The video codec is an essential component in the networked vision system, which compresses videos to a small size with minimal impact on the video quality.Traditional video codecs like MPEG, AVC, and HEVC have been widely adopted in the networked vision system in the past few decades.Recently, learned video codecs built on DNNs are gaining more attention due to their superior coding efficiency than traditional ones.Nonetheless, DNNs in networked vision systems introduce rate and quality heterogeneity, which lead to some issues in the networked vision system, e.g., low processing rate and poor coding efficiency.This dissertation aims at addressing the heterogeneity in networked vision systems.The thesis statement is that the rate and quality heterogeneity in networked vision systems should be addressed by leveraging the spatiotemporal saliency with learning-based adaptation in video compression.The spatiotemporal saliency characterizes how important pixels in the video are, which by our observation varies across different networked vision systems.Specifically, certain regions of interest (ROI) or keyframes are more significant in a given networked vision system.The key intuition is to adapt how information in a video is encoded spatially and temporally based on spatiotemporal saliency, i.e., saliency-aware adaptation to improve processing rate or coding efficiency with minimal impact on other metrics.The challenge lies in properly designing and training the adaptation module in a given networked vision system.To prove the thesis, this dissertation provides a suite of saliency-aware adaptation modules in the video codec, i.e., temporal adaptation (Learned Frame Sampling), spatial adaptation (Context-aware Compression and Spatial-adaptive Filter), and spatiotemporal adaptation (Space-time-aware Entropy Module), addressing the aforementioned challenge.Our evaluation covers various video categories, diverse vision applications, and state-of-the-art compression algorithms.The result demonstrates our approaches effectively improve the processing rate or the coding efficiency in networked vision systems compared to existing solutions without harming other performance metrics.I want to express my gratitude to many people who helped me through my Ph.D. First of all, I want to thank my advisor Klara Nahrstedt.I appreciated the academic freedom she gave me to pursue the topic that interests me.My research topic has changed multiple times in the first few years of my Ph.D. I'm glad that she always supported my idea.She is also tremendously helpful after knowing that I wanted to pursue an academic career, and she gave me many valuable pieces of advice.Undoubtedly, she is a considerate and warmhearted person who wants the best for her students.I cannot remember how often I heard compliments about her from her past students or even people who were not her students.Next, I'm grateful to my other Ph.D. committee members, Tarek Abdelzaher, Hari Sundaram, and Zhisheng Yan.They provided insightful comments on my preliminary works, which pushed me to look deeper into the problem of video compression and change it works.Zhisheng is also the collaborator of many of my works.He provided many constructive feedbacks on my work, and we had a lot of enlightening discussions.I learned a ton when working with him.I would like to thank my qualifying exam committee members, Indy Gupta, Brighten Godfrey, Matthew Caesar, and Sibin Mohan.They helped me to set a high standard for doing research and presentation.Moreover, one of my assigned papers, "Live Video Analytics at Scale with Approximation and Delay-Tolerance," on which I spent a lot of time, gave me significant insights and inspired many of my works later.Many students I met at University of Illinois Urbana-Champaign (UIUC) meant a lot to me.Senior students like
Author: Chen, Bo
Language: English
Open Access status: green
OpenAlex ID: W7220757803
Cites: 139
Cited by: 2
Related to: 10