Evaluating GenAI-mediated academic documentation: Accuracy of quotations and references
Year: 2026
Type: article
Abstract: Introduction. The emerging role of Generative AI (GenAI) tools in academic research and documentation raises questions about the accuracy and reliability of AI-generated quotations and bibliographic information. This study investigates the reliability of GenAI tools and their potential implications in an academic context. Objectives. The study aims to assess the reliability of GenAI in providing accurate content and verifiable bibliographic information by evaluating quotation, reference, and page-number accuracy across ChatGPT and Copilot, and examine implications for academic documentation practices. Method. A comparative evaluation was conducted on 24 research methods definitions, comparing ChatGPT and Copilot outputs against original source texts using a weighted rubric assessing quotation, reference, and page number accuracy. Data Analysis. The generated responses were evaluated against the original source texts to identify accurate, inaccurate, and missing quotations and bibliographic information. Performance was compared across the two platforms and the three accuracy criteria. Results. The results reveal substantial variability in performance across and within platforms. ChatGPT demonstrated comparatively higher mean weighted accuracy (60.42% versus 42.50% for Copilot), achieving 11 high scores. However, only 3 out of 24 ChatGPT responses reached full accuracy, compared with 5 for Copilot. Copilot showed lower overall weighted performance, with 19 out of 24 responses receiving low accuracy scores (below 60%), including inaccurate quotations and unsuccessful retrievals. Across both platforms, reference accuracy was consistently higher than quotation accuracy, with instances of paraphrased content presented as verbatim quotations, including quotation-level hallucination. Copilot produced more fully accurate references (19) than ChatGPT (15), while ChatGPT exhibited more partially accurate references (8 versus 4), involving bibliographic details such as publication years, page numbers, and editorial information. Conclusion. The findings raise significant concerns about the reliability of ChatGPT and Copilot for academic documentation, highlighting risks to accurate citation practices and the integrity of scholarly work. However, the findings are limited to the selected concepts and platforms and should not be generalised to GenAI tools more broadly. Originality. The findings contribute to the ongoing debate about the reliability of GenAI-generated academic documentation by providing a rubric-based empirical comparison of GenAI-mediated documentation against original source texts. By evaluating quotation, reference, and page-number accuracy separately, the study highlights important differences between bibliographic correctness and faithful reproduction of source content. This distinction demonstrates that accurate bibliographic information does not necessarily guarantee quotation accuracy. It offers evidence-based insights into quotation and citation accuracy, alongside recommendations for responsible integration of these tools into scholarly workflows.
Authors: Batool Hendal, Sakena A. Al-Alawi
Language: English
Subfield: Education
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W7225281930