Year: 2025
Type: dissertation
Abstract: Developing and maintaining modern software is a collaborative effort that requires ongoing interaction among various stakeholders. An issue tracking system is essential for recording and organizing these interactions, playing a central role in software development workflows. Recently, large language models like Chat- GPT have become integral to these workflows, particularly in facilitating iterative issue resolution through natural language collaboration. Despite their widespread adoption, the role of these models in collaborative issue resolution remains largely underexplored. In this thesis, we aim to achieve three objectives. First, to understand the nuanced patterns of developers’ engagement with ChatGPT for issue resolution. Second, to conduct a user study with 73 developers to examine their perceptions and concerns, explaining why these engagement patterns occur. Third, to propose a method that could mitigate the concerns from our empirical and user studies. Our empirical studies reveal that developers primarily use ChatGPT for ideation, such as brainstorming alternative solutions. However, its usage for validation (e.g., checking factual information) remains notably low. The survey confirmed this pattern, indicating that developers appreciate ChatGPT for suggesting tools, APIs, and initial ideas, but find it the least reliable for validation due to issues like sycophancy and confident hallucinations. Furthermore, we observed that backend issues (e.g., API management) dominate discussions. Developers noted that while ChatGPT is useful for generating backend scaffolding, it is less effective when addressing issues that require local project-specific context. Additionally, our findings revealed that only 12.78% of code generated by ChatGPT is used to resolve issues, with just 5.83% incorporated without modification. Our user study attributed this limited reliance to concerns about hallucinated or unsafe APIs in the generated code. Lastly, our empirical study showed a positive sentiment towards using ChatGPT for routine tasks, such as initial scaffolding and data analytics, but consistent dissatisfaction when it comes to debugging. The survey reinforced this sentiment, showing that developers prefer Chat- GPT for routine work but favor peer or community-vetted solutions (such as QA forums) for debugging. Together, these findings highlight ChatGPT’s tendency to produce confident yet unreliable responses when lacking sufficient project-related contextual grounding. To address this, we propose GRIT to extract relational information from project repositories to enhance the contextual fidelity of AI-assisted issue resolution. GRIT outperforms standalone language models and retrieval-augmented generation baselines by 9.29% and 3.74%, respectively, when answering questions about GitHub issues. Additionally, it reduces hallucinations for fine-grained queries. In conclusion, our findings indicate that developers value ChatGPT for brainstorming solutions to issues but still depend on peer support for critical steps, such as validation. This thesis highlights both the collaborative advantages and limitations of these tools when used in issue trackers and their reliance on code. Our study recommends that tool developers consider our observations and proposed methods to enhance ChatGPT to better assist developers in the issue-resolution process.
Author: Das, Joy Krishan
Institution: University of Saskatchewan
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7120453540