Evaluating GPT-5-assisted structured interpretation of ultrasound-related Cochrane reviews: a two-level MedReview-based analysis, 1996–2023
Year: 2026
Type: article
Abstract: Ultrasound is widely used across clinical medicine, but its role within Cochrane reviews may reflect either broad clinical involvement or direct evaluation of ultrasound itself. At the same time, large language models are increasingly being used for evidence-synthesis tasks, yet their field-specific performance in ultrasound-related review interpretation remains insufficiently characterized. This study aimed to combine a two-level analysis of ultrasound-related Cochrane reviews with a practical evaluation of GPT-5 API–assisted structured interpretation against a dual-reviewer consensus reference standard. We conducted a MedReview-based descriptive meta-research study of Cochrane reviews published between 1996 and 2023. In stage 1, we identified a broad corpus of ultrasound-involved reviews. In stage 2, a focused subset was defined in which ultrasound itself was the primary subject of evaluation. GPT-5 API was used for structured extraction of review purpose, clinical domain, evidence certainty, and conclusion polarity. Two clinicians independently annotated the focused subset while blinded to GPT-5 outputs. Disagreements were resolved through consensus adjudication, and the final consensus labels were used as an operational reference standard. Agreement was assessed using exact agreement, Cohen’s kappa for nominal variables, and quadratic weighted kappa for evidence certainty. The broad ultrasound-involved corpus included 163 reviews. Manual review identified 54 ultrasound-focused reviews. In the consensus reference standard, evidence certainty was distributed as High 2 (3.7%), Moderate 17 (31.5%), Low 20 (37.0%), and Very low 15 (27.8%). Conclusion polarity was Strong support 5 (9.3%), Weak support 21 (38.9%), Insufficient evidence 23 (42.6%), and Against 5 (9.3%). Before consensus, inter-rater agreement between the two clinicians was moderate to substantial for review purpose, conclusion polarity, and clinical domain, but lower for evidence certainty. Against the final consensus reference standard, GPT-5 showed highest agreement for review purpose (48/54, 88.9%; κ = 0.793), followed by conclusion polarity (43/54, 79.6%; κ = 0.672), clinical domain (41/54, 75.9%; κ = 0.722), and evidence certainty (38/54, 70.4%; quadratic weighted κ = 0.782). A two-level analytical framework helps distinguish broad ultrasound involvement from direct ultrasound-focused evidence in Cochrane reviews. GPT-5 API showed useful but field-dependent performance in structured review interpretation. The addition of dual independent annotation and consensus adjudication indicated that review-level evidence certainty was difficult to label consistently between human reviewers, while GPT-5 achieved substantial weighted agreement with the consensus operational reference standard under the dominant-certainty rule. Within this ultrasound-related Cochrane review task, these findings support a human-in-the-loop approach to LLM-assisted evidence mapping rather than fully automated evidence adjudication. Not applicable.
Authors: Weizao Wu, Anchen Liu, Longyan Wu, Ran Huang, Dean Ta, Xin Ma
Institutions: Fudan University, Shanghai Sixth People's Hospital
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $2,990
OpenAlex ID: W7225593478