Year: 2024
Type: article
Abstract: In recent years, large language models (LLMs) have demonstrated remarkable capabilities in the field of code generation. However, existing research has primarily focused on algorithmic problem-solving code generation, with limited attention to the ability to generate framework code used in actual software development. Programming frameworks are vital tools in software development, effectively reducing development time and enhancing code compatibility. This paper takes the Qt framework in C++ as an example to systematically evaluate ChatGPT’s performance in code generation at different levels of granularity (project-level, class-level, and function-level). To this end, we designed a test dataset (comprising 10 code generation projects of varying complexity) to assess the model’s performance in terms of correctness, robustness, and user experience. In this process, we employed prompt engineering methods to ensure fair conversion. The experimental results show that while ChatGPT is capable of generating functional code in most cases, its performance in correctness, robustness, and user experience decreases as task complexity and code granularity increase. Nonetheless, with manual intervention or more detailed prompts, these issues can be largely resolved. Overall, ChatGPT shows potential in framework code generation, particularly for small to medium-sized tasks. This study reveals both the potential and limitations of LLMs in framework development, providing valuable insights for future improvements and applications.
Author: Haoming Zhu
Institution: Shandong University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4405592533