Observing the Conduct of Systematic Reviews with Generative AI Support: An Experience Report from a Graduate Software Engineering Course
Year: 2026
Type: review
Abstract: Context: Secondary studies are fundamental practices in Evidence- Based Software Engineering, but teaching them requires activities that expose students to authentic methodological decisions. Objective: This paper reports an experience in a graduate course in which ten doctoral students in Software Engineering, organized into three groups, piloted secondary studies with and without support from generative AI. Method: A single-day classroom session was organized and observed, in which the groups conducted pilot systematic reviews with and without generative AI support. Classroom observations, produced artifacts, and interaction threads with assistants configured in ChatGPT were analyzed to reconstruct how each group appropriated the technology throughout the activity. Results: LLMs reduced initial barriers, accelerated the generation of alternatives, and made methodological problems more explicit, but they also favored excessive delegation, superficial validation, operational difficulties, and a shift in focus from conducting the SLR to using the tool. Conclusion: The experience offers a situated, observational account of how doctoral students engaged with generative AI during a systematic review activity, and the resulting insights also inform the design of a subsequent controlled study. The findings indicate that generative AI can support practical learning about SLRs, provided that its use is accompanied by human supervision, decision records, and critical reflection on its limitations.
Source: arXiv (Cornell University)
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W7214070567
FWCI:
Cites:
Cited by:
Related to: