Performance–efficiency trade-offs of large language models for structured extraction from Chinese clinical free text: a benchmark study of model scale, reasoning mode, and architecture
Year: 2026
Type: article
Abstract: Background Large language models (LLMs) may support structured extraction from clinical free text, but evaluation should consider accuracy, output validity, and inference efficiency. Objective To evaluate LLM performance–efficiency trade-offs in Chinese clinical text extraction across model scale, reasoning mode, temperature, architecture, and text-only versus multimodal configurations. Methods This single-center retrospective benchmark sampled 100 Chinese clinical reports per task: left ventricular ejection fraction (LVEF) extraction, carotid intima-media thickness (IMT) extraction, and coronary computed tomography angiography (CTA) classification. The same 300 reports were evaluated under 23 conditions, yielding 6900 outputs. Outcomes included output validity and all-report accuracy (exact-match for LVEF and IMT; classification for CTA). Class-specific precision, recall, and F1, macro-F1, and the Matthews correlation coefficient (MCC) were calculated among valid CTA responses. Local inference efficiency measures included tokens, time to first token, and end-to-end latency. Results Four designated default configurations achieved 99.7% overall accuracy and 100.0% validity. CTA accuracy was 99.0%, with valid-response macro-F1 of 0.984–0.985 and MCC of 0.969–0.970, although false-positive and false-negative errors differed. Reasoning increased Qwen3-0.6B overall accuracy from 31.0% to 88.7% and CTA MCC from 0.025 to 0.599. For Qwen3-32B, overall accuracy increased from 98.7% to 99.7%, CTA MCC from 0.890 to 0.970, and median end-to-end latency from 3.746 to 32.097 s. Temperature changes offered no consistent improvement across CTA metrics. Dense and mixture-of-experts configurations had closely spaced CTA metrics but different latency and token profiles. Text-only and multimodal Qwen configurations showed no consistent relative advantage across two temperatures. The primary dense-only parameter-scale association was not significant after multiplicity correction, and report-level estimates were sensitive to model composition. Conclusions Several evaluated configurations achieved high LVEF and IMT exact-match accuracy, CTA classification performance, and output validity on three predefined tasks. Class-specific metrics identified error differences that accuracy alone did not capture. Selection should consider all-report performance, output validity, class-specific errors, and inference requirements under intended deployment conditions.
Source: Frontiers in Medicine
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $3,801
OpenAlex ID: W7225828967