Work

AI through the looking glass: a controlled feasibility study of wearable AI assistance on plastic surgery in-service examination questions

Year: 2026
Type: article
Abstract: Purpose Large language models (LLMs) have demonstrated substantial accuracy on specialty in-service examinations. Commercially available smart glasses now make real-time AI assistance wearable, raising concerns about testing integrity. We developed and evaluated a smart glasses application that captures an examination question, submits it to an AI model, and delivers the answer directly to the wearer, to assess the feasibility and limitations of this workflow in a controlled simulation. Methods In Phase 1, three LLMs (Google Gemini 2.5 Flash, OpenAI GPT-4o, and Anthropic Claude Sonnet 4.6) and a majority-vote ensemble were benchmarked on 743 Plastic Surgery In-Service Training Examination (PSITE) question texts from the 2021–2023 examination cycles. Clinical images were not supplied. In Phase 2, the model with the highest observed accuracy was deployed through Rokid AI Glasses, a companion phone, and a cloud service. One study author tested the system on a 50-question subset selected from Phase 1. Outcomes were accuracy, first-attempt success, eventual completion, and successful-attempt latency. Results Gemini 2.5 Flash achieved the highest individual accuracy (82.8%), followed by Claude Sonnet 4.6 (80.9%) and GPT-4o (76.9%). The majority-vote ensemble achieved 81.6% accuracy and did not outperform Gemini. In Phase 2, the wearable system answered 76.0% of questions correctly (38/50; 95% CI, 62.6%–85.7%). Answers were returned on the first attempt for 45/50 questions (90.0%); all 50 were completed after 56 attempts. Six attempts failed because of application timeouts requiring restart. Mean successful-attempt latency was 8.9 s (SD 4.2), excluding failed attempts and restart time. Conclusions Commercially available smart glasses and AI services can be connected into a workflow that answers plastic surgery examination questions in real time, although errors and retries limit reliability. This controlled simulation illustrates a potential route to unauthorized assistance. The findings aim to encourage broader discussion of emerging risks to testing integrity and the future of education and assessment.
Language: English
Field: Medicine
Open Access status: hybrid
Estimated APC paid: $2,590
OpenAlex ID: W7236850640
FWCI:
Cites: 12
Cited by:
Related to: 10