A comparison of customized GPTs and GPT-4o for self-medication knowledge: A feasibility study using the Japanese registered salesperson examination
Year: 2026
Type: article
Abstract: Objective: This feasibility study aimed to assess whether minimally customized large language models (LLMs) can achieve knowledge levels relevant for self-medication support by evaluating their performance on the Japanese Registered Salesperson Examination. Registered salespersons provide consumers with guidance on appropriate selection and safe use of over-the-counter drugs in Japan. We explored whether LLMs customized through simple reference materials upload, without requiring programming skills or advanced prompt engineering, could achieve comparable knowledge levels. Methods: We used the 2024 Registered Salesperson Examination in Japan as a benchmark dataset, comprising 838 text-only multiple-choice questions across five domains. We created customized GPTs by uploading the official guide for examination question development as a PDF file to GPT-4o, without additional prompts or instructions. Both GPT-4o and the customized GPTs were evaluated using a zero-shot approach with web-browsing capabilities disabled. Results: < 0.05). Conclusion: This feasibility study demonstrates that minimal customization through reference materials upload can achieve performance improvements on knowledge-based assessments. However, examination performance represents only an indirect indicator of knowledge for self-medication support. Future research should evaluate LLMs performance in real-world case scenarios and assess practical utility and safety before implementation in consumer self-medication settings.
Source: Digital Health
Institutions: Sapporo City University, Hiroshima University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $2,000
Funder: Sapporo City University
OpenAlex ID: W7130737293