Year: 2024
Type: conference-paper
Abstract: Ensuring the accuracy of responses provided by large language models (LLMs) is crucial, particularly in clinical settings where incorrect information may directly impact patient health.To address this challenge, we construct K-QA, a dataset containing 1,212 patient questions originating from real-world conversations held on K Health (an AI-driven clinical platform).We employ a panel of in-house physicians to answer and manually decompose a subset of K-QA into self-contained statements.Additionally, we formulate two NLI-based evaluation metrics approximating recall and precision: (1) comprehensiveness, measuring the percentage of essential clinical information in the generated answer and (2) hallucination rate, measuring the number of statements from the physician-curated response contradicted by the LLM answer.Finally, we use K-QA along with these metrics to evaluate several state-of-theart models, as well as the effect of in-context learning and medically-oriented augmented retrieval schemes developed by the authors.Our findings indicate that in-context learning improves the comprehensiveness of the models, and augmented retrieval is effective in reducing hallucinations.We will make K-QA available to to the community to spur research into medically accurate NLP applications.1
Institution: Hebrew University of Jerusalem
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
OpenAlex ID: W4402683847