Year: 2026
Type: conference-paper
Abstract: Clinical diagnosis is fundamentally interactive and incremental, yet the dominant paradigm for evaluating Large Language Models (LLMs) in medicine remains static QA benchmarks or template-based dialogues.These benchmarks say little about whether a model can serve as a diagnostic agent in a dynamic clinical encounter, with LLMs showing significant accuracy and reliability degradation in multi-turn settings.To address this issue, we present MTDiag, a large multiturn diagnostic dialogue dataset constructed from three heterogeneous sources: DDX-Plus, MIMIC-IV, and published case reports (AJCR), covering common ED presentations as well as long-tail rare and atypical conditions.All cases are normalized into a canonical PatientVector schema anchored in the most comprehensive and widely-adopted medical knowledge bases (UMLS concept identifiers, with ICD-10 diagnosis codes).We release the PatientVector schema, a UserLM-8Bbased utterance-generation pipeline, and the physician-validated dataset that converts structured clinical evidence into natural-language utterances.Importantly, we introduce and motivate clinical knowledge-grounded metrics for evaluating LLMs as diagnostic agents, beyond diagnostic accuracy, for the task of multi-turn differential diagnosis.cross-walk between any of the vocabularies via multilinguality ( resolves the same concept in multiple languages)
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: bronze
Awards: G2555749475, G3589708518
OpenAlex ID: W7204452336