Comparative Benchmarking of ChatGPT, Claude, and Grok on Persian General Surgery Promotion Examinations With and Without an Uploaded Surgical Reference Package
Year: 2026
Type: preprint
Abstract: Background Large language models are increasingly used in medical education, yet their observed agreement with answer keys on locally developed, non-English surgical examinations remains incompletely characterized. Methods This comparative benchmark evaluated ChatGPT (Luna High), Claude Opus 5, and Grok 4/5 on 456 original Persian four-option questions from the 2024 (1403 Persian calendar), 2025 (1404 Persian calendar), and 2026 (1405 Persian calendar) Iranian General Surgery Residency Promotion Examinations. Each annual question file was submitted once to each model in two separate user-facing workflows: question file only and question file plus an uploaded surgical reference package. Responses were scored against the official Sanjesh answer keys using a strict single-option rule. Results With the uploaded reference package, aggregate answer-key agreement was 71.27% for ChatGPT, 75.44% for Grok, and 80.04% for Claude. In the question-file-only workflow, the corresponding values were 67.11%, 72.59%, and 79.82%. Claude had the highest observed aggregate agreement in both workflows. Conclusion In this single-run, web-enabled benchmark, Claude showed the highest observed aggregate agreement with the official answer keys. The uploaded reference package was associated with a positive change in ChatGPT agreement across all three years, whereas the direction of change varied for Grok and Claude. These findings are descriptive and do not establish a general model ranking or a causal effect of document upload.
Source: Research Square
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225274424