Clarity Without Credibility? Human Versus AI Abstracts in Otolaryngology
Year: 2026
Type: article
Abstract: Objective: This study evaluated whether otolaryngologists can distinguish between human- and machine-written abstracts. The primary question was whether large language models (LLMs) produce abstracts comparable in clarity and usefulness to human-authored work, and whether reviewers can identify authorship with accuracy. Methods: A blinded cross-sectional design was used. Forty-eight abstracts were evaluated, consisting of twenty-four human-authored abstracts and 24 generated by four LLMs. Human abstracts were drawn from articles published after July 2025 to minimize overlap with LLM training data. Twenty otolaryngologists independently reviewed all abstracts. Using a structured rubric, raters classified authorship, rated clarity, usefulness, and confidence on 5-point scales, and provided optional free-text explanations. Group comparisons were performed using chi-square and Mann-Whitney tests, with Kruskal-Wallis tests for model-level analyses. Results: Overall recognition accuracy was 44.7%. Human-written abstracts were more often misclassified as AI than AI-generated abstracts were mistaken for human. Human abstracts received significantly higher clarity and usefulness scores than LLM abstracts, though effect sizes were small. Confidence did not correlate with correctness, indicating miscalibration of rater judgments. Model-level performance varied. Grok-generated abstracts were most easily identified as AI, whereas GPT-5 and Claude 3.5 more frequently resembled human writing. Free-text rationales commonly referenced style, vagueness, or lack of detail when AI authorship was suspected. Conclusion: LLMs generate abstracts that increasingly resemble human scientific writing, yet still lag in perceived usefulness and credibility. Clinicians were only moderately successful at detecting authorship and were frequently confident in incorrect classifications. These findings highlight both the promise and risks of AI-assisted scientific communication.
DOI: 10.1002/wjo2.70104
Institutions: University of Nicosia, Sheba Medical Center, City St George's, University of London, Stanford Medicine, Stanford University, Medical University of Vienna, Ospedale Santa Maria della Misericordia di Udine, Azienda sanitaria universitaria Friuli Centrale, Lyon 1 Université, Centre National de la Recherche Scientifique, Université de Lyon, Institut de Biologie et de Chimie des Protéines, Hospices Civils de Lyon, Hôpital Lyon Sud, Laboratoire de Biologie Tissulaire et d'Ingénierie Thérapeutique, The University of Texas Health Science Center at Houston, University of Missouri–Kansas City, Tel Aviv University, Meir Medical Center
Language: English
Subfield: History and Philosophy of Science
Field: Arts and Humanities
Domain: Social Sciences
Open Access status: diamond
OpenAlex ID: W7140197622