Year: 2024
Type: dissertation
Abstract: Detecting performance regressions early in the software development process is paramount since performance bugs can lead to severe issues when introduced into a productive system. However, it is impractical to run performance tests with every committed code change due to their resource-intense nature. This study investigates to what extent NLP methods specialized on source code can effectively predict software performance regressions by utilizing source code obtained through line coverage information from microbenchmark exe- cutions. Contributing to the overarching goal of supporting test case selection and thereby increasing efficiency of performance benchmarking, we evaluate several models at different levels of complexity ranging from a simple logistic regression classifier to Transformers. Our results show that all implemented models exhibit challenges in accurately predicting regression-introducing code changes and that simple ML classifiers employing a Bag-of-Words encoding reach similar predictive performance as a BERT-based Transformer model. We further employed a statistical n-gram model to examine if the 'natural- ness' of source code can serve as reliable indicator for software performance regressions and concluded that the approach is not applicable to the data set at hand. This further underlines the challenge of effectively predicting perfor- mance based on source code and puts into question whether the current quality and quantity of available data is sufficient in order to render an NLP-based machine learning approach on regression detection suitable.
Author: David, Lucie
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7067015922
