Schema Enforcement and Structured-Output Stability in Locally Deployed LLMs for Clinical Admission-Note Editing: A Proxy-Based Pre-Deployment Evaluation
Year: 2026
Type: article
Abstract: Background: Reliable structured-output generation is a prerequisite for using large language models (LLMs) in automated clinical documentation workflows, but many evaluations focus on clinical quality before testing whether outputs are parseable, schema-compliant, and stable. Methods: We evaluated three locally deployed open-weight LLMs in the 7- to 8-billion-parameter range (Llama3-Med42-8B, Meta-Llama-3-8B-Instruct, and Mistral-7B-Instruct-v0.3) for structured admission-note editing. Seventy de-identified English-language admission notes (35 internal medicine and 35 surgical) were processed by each model in three independent runs under two output-control conditions: a free-text JSON prompt and a schema-enforced structured-output condition. A total of 1260 local inferences were performed in LM Studio on consumer-grade hardware. Automated proxy metrics assessed JSON/schema validity, run-to-run stability, instruction compliance, verbosity, numeric-token preservation, and uncertainty-marker change without clinician adjudication of clinical correctness. Results: Under the free-text JSON prompt, the tested Mistral-7B-Instruct-v0.3/embedded-prompt configuration had the weakest structural reliability (74.3–78.6% first-pass validity per run; 18.6–21.4% persistent parse/schema failures after retry), with at least one final failure for 17 of 70 notes. In a message-format sensitivity analysis using Meta-Llama-3-8B-Instruct, embedding system instructions in the user message increased first-attempt invalid outputs compared with separate system/user roles (55/700, 7.9% vs. 12/700, 1.7%). Under schema enforcement, all models produced 70 of 70 first-pass valid, schema-compliant outputs in every run. Documentation behavior nevertheless differed by model, including differences in verbosity and numeric-token preservation. Conclusions: Schema enforcement removed parsing failures in this sample but did not eliminate model-specific editing behavior. Proxy-based screening can identify structurally unstable model-prompt or model-format configurations before clinician review.
Source: Healthcare
Authors: Ya-Lun Yang, Chia-Jung Chen, Tin-Kwang Lin, Shih-Chun Lin, Malcolm Koo
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $3,258
OpenAlex ID: W7168545375
FWCI:
Cites:
Cited by:
Related to: