Work

Identification of Interfering Factors and Risk Assessment of Misdiagnosis by LLMs in the Biochemical Diagnostic Pathway for Functional GEP/Pancreatic NET

Year: 2026
Type: preprint
Abstract: This project aims to evaluate the ability of large language models (LLMs) to recognize clinically relevant interference factors during the biochemical diagnostic workup of functional gastroenteropancreatic and pancreatic neuroendocrine tumors (GEP-NETs/pNETs). The study focuses on a clinically important source of diagnostic error: biochemical test results may have substantially different interpretations depending on concomitant medications, renal function, dietary factors, sampling conditions, or physiological states. Failure to recognize such interference may lead to inappropriate confirmation or exclusion of a functional neuroendocrine tumor, unnecessary diagnostic testing, delayed reassessment, or potentially harmful clinical decisions. We will conduct a cross-sectional, matched-pair, multi-model evaluation using expert-constructed clinical vignettes rather than real patient data. The planned test set will include functional neuroendocrine syndromes such as insulinoma, gastrinoma/Zollinger–Ellison syndrome, VIPoma, glucagonoma, somatostatinoma, and carcinoid syndrome. Each vignette pair will consist of two otherwise identical clinical scenarios containing the same symptoms and biochemical results. In the control arm, an inert comparator condition that does not interfere with biochemical interpretation will be included, whereas in the interference arm, the same position will contain a clinically relevant interference factor. This paired design is intended to isolate whether the model appropriately incorporates the interference factor into its reasoning rather than merely recognizing an abnormal laboratory result. The planned dataset consists of 72 matched vignette pairs. At least four LLMs, including both closed-source general-purpose models and open-source models that can be locally deployed, will be evaluated. Exact model versions, sampling parameters, system prompts, and the model evaluation window will be predefined and frozen before formal testing. Each vignette will be evaluated repeatedly to assess response stability. Model outputs will be collected in a predefined structured format to allow reproducible automated analysis, with blinded expert review used for validation, error classification, and assessment of potential clinical harm. The primary outcome will be the paired verdict flip rate (PVFR), defined as the proportion of vignette pairs in which an LLM correctly interprets both the non-interference and interference scenarios and appropriately changes its conclusion when the interference factor is introduced. Secondary outcomes will include interference-factor recognition rate, false diagnostic confirmation rate, false exclusion rate, unnecessary testing recommendations, frequency and severity of potentially harmful errors, response consistency across repeated generations, and calibration between model-reported confidence and actual correctness. Performance will also be described across different functional syndromes, categories of interference, and vignette difficulty levels. The expected outcome of this study is not to identify a universally “best” LLM, but to characterize the strengths, limitations, and clinically relevant failure modes of current LLMs when interpreting biochemical findings in functional neuroendocrine tumor diagnosis. We expect the study to clarify whether models can reliably distinguish between an abnormal result that supports a diagnosis and an abnormal result that should not be interpreted until a confounding condition has been corrected. The findings may help define situations in which LLM-generated clinical reasoning requires particular caution and may provide a reproducible framework for future evaluation of LLMs in laboratory-based diagnostic decision-making. In addition to the research manuscript, the project is intended to generate a reproducible research package including the vignette dataset, prompt template, structured output schema, biochemical interference codebook, model outputs, scoring framework, and analysis code. These materials will support transparency, independent verification, and future extension of the evaluation framework to other neuroendocrine or laboratory-based diagnostic scenarios.
Author: ShengYu Wei
Language: English
Field: Medicine
Open Access status: green
OpenAlex ID: W7204625184
Cites:
Cited by:
Related to: 10