Work

Accuracy of ChatGPT for literature citations in lower limb arthroplast

Year: 2025

Type: article

Abstract: Background Artificial Intelligence (AI) enables orthopaedic surgeons to analyze large datasets efficiently. Generative chatbots like ChatGPT utilize open-access sources to provide focused, modifiable results, potentially serving as valuable research aids. However, concerns remain regarding the legitimacy of generated citations, with prior studies reporting high “hallucination” rates. This study assessed the accuracy of citations generated for lower limb arthroplasty and evaluated the impact of prompt specificity on ChatGPT v3.5 and v4.0 performance. Methods In August 2024, ChatGPT v3.5 and v4.0 were queried using three levels of prompt specificity (simple, medium, and complex) to generate outlines with 10 citations for primary total knee arthroplasty (TKA), primary total hip arthroplasty (THA), revision TKA, and revision THA. Prompts included elements like citation style, source origin, and contextual indicators. Generated citations were verified against PubMed, Google Scholar, and Google, and classified as Nonexistent, Secondary Work, Improperly Cited, or Properly Cited. Chi-square and multivariate logistic regression identified differences and common errors. Results Among 240 citations (ChatGPT-3.5: n=120; ChatGPT-4.0: n=120), increasing prompt specificity significantly improved citation accuracy ( p < 0.001). Simple prompts produced no legitimate primary sources, while medium and complex prompts generated similar rates (66.3% vs. 68.8%) but reduced errors (39.6% vs. 21.8%). Recent publications increased errors in publication year, journal, and authorship (OR: 1.2–1.206), while high-impact journals reduced journal errors (OR: 0.581). Conclusion ChatGPT v3.5 and v4.0 show improved performance with tailored prompts, generating legitimate citations at twice the rate of hallucinations. While these models aid scientific writing, citation errors and hallucinations remain significant limitations.

Source: Journal of Orthopaedic Reports

DOI: 10.1016/j.jorep.2025.100756

Authors: Matthew C. Johnson, Matthew L. Magruder, Alexander K. Hahn, Ameer Tabbaa, Ezinma Dennis, Matthew J. Grosso