Model Configuration and Task Design Associations With Hallucination Test Accuracy in Medical Language Model Benchmarks
Year: 2026
Type: article
Abstract: This analysis examines whether medical language model rankings remain consistent across configurations and evaluation tasks, informing international debates on reliable model selection. Published benchmark results from 2023 provide 70 observations covering ten configurations and seven tasks. Descriptive comparisons examine model families, parameter sizes within Llama 2, and 35 matched differences between base and tuned variants. Family rankings vary across tasks. Average base Llama 2 accuracy increases from 18.95 to 35.59 percent between seven and seventy billion parameters, whereas tuned accuracy peaks at thirteen billion. Across matched comparisons, tuning accompanies a mean difference of −9.30 percentage points, but the median difference is zero. Reasoning comparisons average −26.96 percentage points, while memory comparisons average 3.94 percentage points, exposing contrasting performance patterns concealed by aggregation. These descriptive results refine capability and task alignment by demonstrating that configuration advantages depend on evaluation demands rather than a uniform hierarchy. The contribution is a transparent comparison framework that distinguishes improvement frequency from magnitude. Institutions should require task specific evidence when selecting configurations and avoid interpreting aggregate benchmark superiority as demonstrated clinical reliability.
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225461255