Year: 2025
Type: article
Abstract: Background Gender inference from names is widely used in bibliometric and epidemiologic research, including in general internal medicine. Traditional tools such as Gender API and NamSor are considered accurate but remain limited by misclassifications and unclassified cases. Recent studies suggest that ChatGPT may perform comparably to these tools. We aimed to test whether a two-step procedure could further improve ChatGPT’s performance. Methods We evaluated ChatGPT-5 against Gender API using a random sample of 1,000 Swiss physicians. A two-step one-shot prompt was applied: 1-assign gender directly from the name if reliable; 2-otherwise, verify using the internet. Gender API was applied to the same dataset with no threshold and at probability thresholds of ≥60%, ≥70%, ≥80%, and ≥90%. Confusion matrices, McNemar’s test, and accuracy metrics (errorCoded, errorCodedWithoutNA, naCoded) were computed. Results Of 1,000 physicians, 523 (52.3%) were women and 477 (47.7%) were men. ChatGPT-5 achieved 996 correct classifications (99.6%), with 4 errors and no unclassified cases, whereas Gender API (whole sample) achieved 977 correct classifications (97.7%), 18 errors, and 5 unclassified cases (p-value<0.001). At higher thresholds, Gender API reduced errors but produced up to 6.5% unclassified cases. Overall error rates (errorCoded) were 0.4% for ChatGPT-5 versus 2.3% for Gender API. ChatGPT-5 marked 10.1% of names as “checked” through internet verification, increasing to 69.6% among cases that Gender API misclassified or left unclassified. Conclusion ChatGPT-5 substantially outperformed Gender API in gender inference from physicians’ names, achieving near-perfect accuracy without unclassified cases. Its adaptive use of internet verification for difficult names may offer a robust and efficient approach for large-scale research.
Source: F1000Research
Author: Paul Sebo
Institution: University of Geneva
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $1,350
OpenAlex ID: W4415800109