Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not today
Year: 2023
Type: preprint
Abstract: Recent investigations show that large language models (LLMs), specifically GPT-4, not only have remarkable capabilities in common Natural Language Processing (NLP) tasks but also exhibit human-level performance on various professional and academic benchmarks. However, whether GPT-4 can be directly used in practical applications and replace traditional artificial intelligence (AI) tools in specialized domains requires further experimental validation. In this paper, we explore the potential of LLMs such as GPT-4 to outperform traditional AI tools in dementia diagnosis. Comprehensive comparisons between GPT-4 and traditional AI tools are conducted to examine their diagnostic accuracy in a clinical setting. Experimental results on two real clinical datasets show that, although LLMs like GPT-4 demonstrate potential for future advancements in dementia diagnosis, they currently do not surpass the performance of traditional AI tools. The interpretability and faithfulness of GPT-4 are also evaluated by comparison with real doctors. We discuss the limitations of GPT-4 in its current state and propose future research directions to enhance GPT-4 in dementia diagnosis.
Source: arXiv (Cornell University)
Authors: Wang, Zhuo, Li, Rongzhen, Dong, Bowen, Wang, Jie, Li, Xiuxing, Liu, Ning, Mao, Chenhui, Zhang, Wei, Dong, Liling, Gao, Jing, Wang, Jianyong
Institutions: Tsinghua University, Chinese Academy of Medical Sciences & Peking Union Medical College, Peking Union Medical College Hospital, First Affiliated Hospital of Nanchang University, Chinese Academy of Sciences, Institute of Computing Technology, University of Chinese Academy of Sciences, Shandong University, East China Normal University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
Funders: U.S. Department of Defense, Alzheimer's Association, Bristol-Myers Squibb, Eli Lilly and Company, Pfizer, Biogen, University of Southern California, Meso Scale Diagnostics, Alzheimer's Disease Neuroimaging Initiative, BioClinica, Novartis Pharmaceuticals Corporation, Northern California Institute for Research and Education, National Natural Science Foundation of China, Eisai, Servier, H. Lundbeck A/S, IXICO, National Institutes of Health, Genentech, Canadian Institutes of Health Research, National Key Research and Development Program of China
Awards: G1298807586, G1554142629, G1593117279, G1650819308, G1880935189, Alzheimer's Disease Neuroimaging Initiative, G2060226385, G2100670906, G2152643862, G2189010064, G2340207221, G2423387114, G2471646051, G2522063444, G2835601961, G29951154, G3131374986, G3216574877, G3553156159, G3710456697, G3752451908, G3870774570, G3908963977, G4119534169, G43180419, G4460497059, G4475667458, G4501223628, G470063782, G4800177743, G5088477901, G5207404921, G5209990037, G528601746, G5365730499, EFFECTS OF TRAUMATIC BRAIN INJURY (TBI) AND POST TRAUMATIC STRESS DISORDER (PTSD) ON ALZHEIMER'S DISEASE (AD) IN VETERANS USING IMAGING AND BIOMARKERS IN THE AD NEUROIMAGING INITIATIVE (ADNI)", G5654639750, G5740349702, G5850682332, G613619302, G6173816828, G6210868510, G6371410416, G6412110892, G6447063303, G6719828067, G6777344962, G6792506723, G6814835421, G6899163935, G72415811, G7298697282, G764699806, G7799614282, G781725486, G797295151, G8151151163, G8228018387, G8281378126, G839208290, G8852128998, G8895907669, G8897063270, G8957659042, G9000039018, G9000122335
OpenAlex ID: W4379474412