Work

Large language model use and hallucination awareness among medical students: a single-centre cross-sectional survey

Year: 2026
Type: preprint
Study design: Observational Study
Abstract: Background Large language models (LLMs) are increasingly used in medical learning, but their answers may contain plausible yet incorrect information. It remains unclear whether students’ awareness of these hallucinations is associated with how they judge the credibility of specialty-specific answers. Methods This single-centre cross-sectional survey was conducted at Sun Yat-sen University, China, from January to July 2026. Students completed a 17-item questionnaire on site and were classified into three groups by ophthalmic educational background. Students and five blinded senior ophthalmologists rated five examination-derived LLM answers on a five-point credibility scale. Analyses included descriptive statistics, Kendall’s concordance, Friedman and Kruskal–Wallis tests, and exploratory Spearman correlations with bootstrap confidence intervals and false-discovery-rate (FDR) correction. Results Of 145 invited students, 138 provided complete questionnaires (95.2%): 22 in the basic-knowledge group, 80 in the course-level group, and 36 in the specialist-level group. Overall, 127 (92.0%) had used an LLM and 121 (87.7%) perceived improved learning efficiency. Although 98 (71.0%) reported familiarity with hallucinations and 108 (78.3%) reported encountering them, only 35 (25.4%) felt able to identify them clearly. Student mean credibility ratings ranged from 3.51 to 3.67, compared with expert means of 3.0–5.0. Students rated the retinal-neovascularization answer 0.51 points above the expert mean and the traumatic-lens-dislocation answer 1.43 points below it. Hallucination familiarity was associated with LLM familiarity (ρ = 0.342), use frequency (ρ = 0.421), and self-rated detection ability (ρ = 0.274; all FDR-adjusted P < 0.01). Neither hallucination familiarity nor self-rated ability was significantly associated with closer agreement with experts. Conclusions Students commonly used LLMs and recognised hallucination risks, but few felt able to identify hallucinations clearly. Awareness was not significantly associated with closer agreement with expert credibility judgments. These findings support further evaluation of supervised, case-based teaching in source verification and appraisal of LLM-generated information. Objective hallucination detection requires separate assessment.
Language: English
Field: Medicine
Open Access status: green
OpenAlex ID: W7225863352
Cites: 14
Cited by:
Related to: 10