Training the system around a frozen language model: a clinician's lessons and code checks on synthetic intensive care cases — a 2 × 2 factorial study with a measurement of expert reliability
Year: 2026
Type: preprint
Abstract: Aim. To test whether the decision of a local language model whose weights do not change becomes clinically more appropriate if it is given a memory of my lessons and code checks — and what each of them contributes; and, at the same time, to measure myself as the reader who judges these decisions. Methods. The system is built on the local model Gemma 4 26B A4B in the role of the intensivist on duty: the model reads a synthetic case, assesses the data, and writes one rescue decision with its rationale and an order sheet. Around the unchanged model the system learns in two ways: memory — 26 lessons written from my verdicts (I am an intensive care physician); code — checks that reject answers without a decision, carry a lesson through to action and put the order sheet in order. I compared four variants of one system — with nothing, with lessons, with checks, with both — on 10 new cases in three languages, three replicates each: 120 answers. I rated each answer twice — immediately, knowing the variant, and two weeks later blind, in random order — as appropriate, partially appropriate or inappropriate. The comparison is based on the blind reading; the unit of analysis is the case. Results. Between the two readings I agreed with myself in 54% of verdicts (κ = 0.24). In the blind reading the worst variant was lessons alone: not a single appropriate decision and 14 inappropriate out of 30; the full system — 6 appropriate and 6 inappropriate; the bare model and checks alone — in between. The largest difference was the system versus lessons alone (better in 8 cases of 10; nominal p = 0.02, 0.14 after correction for six comparisons); the system versus the bare model — not better. The logs revealed the mechanism: the model always cites a retrieved lesson but does not carry it out without checks (mandatory lesson components missing from the first answer appeared in 0 of 9 answers without checks versus 7 of 9 with checks); the checks also carry wrongly retrieved lessons through to action. The reader kept learning and grew tired — this is visible in his notes and in the length of his comments, but the data show no shift of ratings under fatigue; the least stable step of the scale was the top one. Conclusions. Without checks, memory is the worst variant; with checks it is no worse than the bare model and gives the fewest inappropriate decisions; the system did not become better than the bare model. A lesson transfers to its own error, and only when the system has recognised it. The reader is part of the system and must be measured just as the model is.
Author: Shlyakhta, Taras
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225820743
Cites:
Cited by:
Related to: