Work

From knowledge generation to knowledge verification: examining the biomedical generative capabilities of ChatGPT

Year: 2025

Type: article

Abstract: The generative capabilities of LLM models offer opportunities for accelerating tasks but raise concerns about the authenticity of the knowledge they produce. We present a computational approach that evaluates the factual accuracy of biomedical knowledge generated by an LLM. Our approach consists of generating disease-centric associations and verifying them using biomedical ontologies. Using ChatGPT, we designed prompt-engineering processes to establish linkages between diseases and related drugs, symptoms, and genes, and assessed consistency across multiple ChatGPT models (e.g., GPT-4, GPT-4o, and GPT-4o-mini). Results demonstrate high accuracy in identifying disease terms (88%-97%), drug names (90%-91%), and genetic information (88%-98%). Symptom term identification was lower (49%-61%) due to informal symptom descriptions. Verification reveals coverage of 89%-91% for disease-drug and disease-gene pairs; symptom-related associations show lower coverage (49%-62%). Despite high term accuracy, generated IDs were often invalid or redundant. GenAI tools can be reliable if used with care. Retrieval Augmented Generation (RAG) may enhance reliability.

Source: iScience

DOI: 10.1016/j.isci.2025.112492

Authors: Ahmed Abdeen Hamed, Alessandro Crimi, Magdalena M. Misiak, Byung Suk Lee

Institutions: Binghamton University, Northeastern University, AGH University of Krakow, Howard University, University of Vermont

Language: English

Topic: Biomedical Text Mining and Ontologies

Subfield: Molecular Biology

Field: Biochemistry, Genetics and Molecular Biology

Domain: Life Sciences

SDG: Good health and well-being

Open Access status: gold

Estimated APC paid: $3,240