Year: 2023
Type: article
Abstract: Machine Learning (ML) is rapidly revolutionizing the way modern-day systems are developed.However, testing ML-based systems is challenging due to 1) the presence of non-determinism, both internal (e.g., stochastic algorithms) and external (e.g., execution environment), and 2) the absence of well-defined accuracy specifications.Most traditional software testing techniques widely used today cannot tackle these challenges because they often assume determinism and require a precise test oracle.This dissertation presents work on automated testing and debugging of ML-based systems and on improving developer-written tests in such systems.To achieve these goals, this dissertation presents principled techniques that build on mathematical foundations from probability theory and statistics to reason about the underlying non-determinism and accuracy.The presented techniques help developers to detect more bugs and to efficiently navigate trade-offs between test quality and efficiency.This dissertation presents contributions along two key directions.First, a key challenge in testing ML-based systems is generating tests that can systematically and effectively explore the input space of the algorithms under test.However, because the inputs to an ML algorithm are complex objects, like a deep neural network model and data, generating inputs that are both syntactically and semantically correct is challenging.Additionally, ML algorithms do not come with well-defined test oracles, which makes it difficult to reason about correctness.This dissertation presents systematic test generation and debugging techniques that tackle these challenges by combining techniques from programming languages, differential testing, and probabilistic reasoning.These techniques have helped detect more than 50 previously unknown bugs in ML libraries and enabled faster debugging of failures.Second, this dissertation presents techniques that improve the quality of regression tests in ML libraries.When writing such tests, developers often fail to adequately account for the randomness of the algorithms under test and rely on guesswork in selecting various test configurations.Consequently, such regression tests often end up being either flaky, i.e., they pass or fail non-deterministically for same code, become expensive to run, or have lower fault-detection effectiveness.This dissertation presents novel test repair techniques that combine principled statistical methods, mathematical optimization, and domain knowledge to systematically tackle these challenges.These techniques have already improved the quality of over 200 tests in over 60 open-source ML libraries, many of which are used at companies like Google, Meta, Microsoft, and Uber as well as in many academic and scientific communities.working together, the hikes, the lunches and dinners, the quest for free food, and more.Over the years, I have collaborated and made friends with a lot of other fellow PhD students at UIUC, including Wing Lam, August Shi, Owolabi Legunsen, Angello Astorga, Chiao Hsieh, Wenyu Wang, Umang Mathur, Adithya Murali, and Liia Butler.I am grateful for all the collaborations, discussions, and the sports we played together.I had a lot of fun co-organizing the weekly sports activity with Wing and Liia.August, having dominated the flaky tests research area himself, was a great collaborator when I started my research on flaky tests in ML.Owolabi mentored and collaborated with me on my first project during my PhD.During this project, Owolabi showed me how to transform a working implementation of an idea to a full research paper, which I had no idea about as a first year student.I could not have asked for a better mentor in my junior years.Owolabi, having conquered the job market few years before me, helped me at each stage of my academic job search from preparing materials, to preparing my job talk, onsite interviews, and decision making.He was always happy to share his experience and wisdom with me, which I am grateful for.Parts of this dissertation were published at European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) 2018 (Chapter 2), 2019 (Chapter 3), and 2021 (Chapter 5); International Symposium on Software Testing and Analysis (ISSTA) 2021 (Chapter 6); and International Conference on Software Testing, Verification and Validation (ICST) 2022 (Chapter 4).I thank the anonymous reviewers and the program chairs of these venues for their feedback and valuable suggestions.During my PhD, I did two internships, one at Microsoft Research and another at Amazon Web Services.At MSR, I worked with Shuvendu Lahiri and Madan Musuvathi.I also collaborated with Max Schäfer (from GitHub) and Diego Garbervetsky.Through the internship, I learned a lot about leveraging machine learning for improving static analyses for JavaScript.At AWS, I worked with Daniel Kroening and Willem Visser, on testing deep learning compilers.Unfortunately, the pandemic forced the internships to be virtual.However, both companies ensured sure that I have a great experience and make connections.I am grateful to several institutions that have funded my research during my PhD.In particular,
Author: Dutta, Saikat
Institution: University of Illinois Urbana-Champaign
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
OpenAlex ID: W7220791325
