Work

Enhetstestning med hjälp av Generativ Artificiell Intelligens : En Utvärdering av Testsvitgenerering

Year: 2024
Type: dissertation
Abstract: Unit tests, essential in software development, serve to verify the correctness of individual code components, ensuring functionality and preventing regressions. However, creating them can be laborious and repetitive for developers. Large Language Model (LLM)s have demonstrated promise in automating code generation tasks. Despite this, concerns persist regarding the consistency and accuracy of the generated code. In this emerging field, research on LLM efficacy in generating unit tests remains limited. This thesis seeks to address this gap by evaluating the quality and reliability of JUnit and xUnit.net test suites generated by LLM. We aim to determine whether these suites provide meaningful assistance to developers or require extensive manual refinement. Tests are generated for the Java open-source project Commons Lang and one C\# project in the Securitas Digital Platform. Utilizing a developed pipeline, various LLM including GPT3.5-turbo-16k, GPT4, and GitHub CoPilot are employed with zero-shot prompting. GitHub CoPilot is exclusively used on the Securitas Digital code base, while Generative Pretrained Transformer (GPT) models are utilized solely for the Commons Lang project. A heuristic is developed for quality assessment and is employed when comparing the test suites generated by the LLM to those generated by automatic test-generation tools such as Randoop and human-written tests. Results reveal that the GPT3.5-turbo-16k generated test suite achieves 52% code coverage, with a more extensive check for program crashes and increased coverage of unique lines of code. However, compared to Randoop, it exhibits a higher proportion of non-compiling and failing tests, largely attributed to hallucinations and a lack of context. GPT4 outperforms GPT3.5-turbo-16k across all metrics with fewer tests. Neither Randoop nor the GPT generated tests match or surpass human-written tests in terms of quality. GitHub CoPilot-generated tests achieve only 43% coverage, significantly lower than human-written tests at 83.3%. In conclusion, while LLM-generated tests show promise in surpassing existing automated test tools, they require significant manual inspection and improvement. The models may function as a helping tool for developers in its current state. However, fully automating the testing process with LLM remains a challenge, requiring significant improvements in future research.
Author: Ali, Glacier
Language: English
Subfield: Software
Open Access status: green
OpenAlex ID: W6990871462
Cites:
Cited by:
Related to: 9