Work

What Does Infect Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs

Year: 2026

Type: conference-paper

Abstract: In this paper, we introduce S-MedQA, an English medical question-answering (QA) dataset for benchmarking large language models (LLMs) in fine-grained clinical specialties.S-MedQA has over 24k examples, covers 15 medical specialties, and QA pairs can have multiple specialty annotations, e.g., when a question is cross-disciplinary, constructed with both machine and expert verification to maximize data availability and reliability.We use S-MedQA to investigate the role of clinical specialties in the knowledge-intensive scenario of medical QA.Our results show that 1) training on data from a clinical specialty does not necessarily lead to best performance on that specialty, and 2) regardless of the specialty the LLM was fine-tuned on, token probabilities of clinically relevant terms increase consistently across all specialties.Thus, we hypothesize improvement gains, at least in our settings, are derived mostly from domain shifting (e.g., general to medical) rather than specialty-specific knowledge injection, and suggest rethinking the role of fine-tuning data in the medical domain.To motivate further advancements in the clinical NLP field, we release S-MedQA and all code needed to reproduce all our experiments to the research community.1

Source: Conference of the European Chapter of the Association for Computational Linguistics

DOI: 10.18653/v1/2026.eacl-long.390

Authors: Xinlan Yan, Di Wu, Yibin Lei, Christof Monz, Iacer Calixto

Institutions: Amsterdam University Medical Centers, University of Amsterdam

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

SDG: Good health and well-being

Open Access status: hybrid

Funder: Nederlandse Organisatie voor Wetenschappelijk Onderzoek