Agentic Adjudication of Clinical Reports: an evaluation design for second reads
Year: 2026
Type: preprint
Study design: Study Protocol
Abstract: A signed radiology report can carry an error that a second reader would catch: a side reversed, a measurement copied wrongly, a finding asserted and then denied, a dictation phrase that changes the meaning, an impression item with no supporting finding. Large language models now detect such text-internal errors at rates that match or exceed radiologists reading the same text in some published comparisons. This conceptual methods proposal asks the question that Study B of the companion paper Correction Across Clinical Handovers poses in general form: does an additional, separately obtained read add anything worthwhile to the strongest single-model workflow, and how would one know? The programme's panel procedure was built in the author's practice, CounterProof, for the review of financial-infrastructure software, where a finding names a line and a defect or vulnerability that a test can settle; the programme's aim is sound agentic institutions in finance and in healthcare alike. The companion paper was its first bounded step into clinical work; this proposal is the second, carrying the procedure to clinical text, where the settling read is a radiologist's. A screen of sixteen near-neighbour papers finds the question open for clinical reports: no published study adds one separately obtained read from a different model family to the strongest single-model workflow on reports and counts, at the level of individual findings, what it adds, what both miss and what it costs to adjudicate. What is published argues against a free gain: a majority of eight open-weight models scored below its best single member against a human reference; an identical evidence packet given to thirty-four models raised their agreement and the number of questions on which the majority was wrong; the most accurate models are the most error-correlated on fixed-option tasks; where both precisions are reported, synthetic error sets inflate the precision of Weng's GPT-4o methods and of ClinicalBERT. The gap is provisional: one near-neighbour paper was reached only as an abstract. The proposal specifies a study on de-identified, already-signed radiology report text with three error strata kept apart (script-planted with a hard sub-stratum of reasoning-dependent errors; model-planted, reported separately; untouched real reports for prevalence and precision, which needs radiologists and is not reachable in 2026), arms that include a rules checker, the strongest single-model workflow, the same model run repeatedly, a cross-vendor panel of application-programming-interface models on licence-cleared material, a panel of locally run models on the rest, and a base-model witness, the companion paper's seven measures, a decorrelation measure defined for free-text finding sets, and a paired kill rule whose test and item count are computed from development-set miss rates before pre-registration and are not computed in this draft. The 2026 form of the study is declared a ceiling check. The contribution is an evaluation design; no study has been conducted and no clinical benefit is established. Substantive AI assistance is disclosed: Claude drafted the proposal and conducted the literature screen; Kimi K2.7 and Grok reviewed its source notes in two rounds and the manuscript in three, DeepSeek read the manuscript three times and GLM once; Codex wrote the draft from which the companion paper was cut. These are not peer review or clinical validation. The human author is responsible for the argument, sources, contents, licence and release decision.
Author: Soons, Luuk Joseph
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225395656