Work

Building an Auditable Reference Standard for Clinical Information Extraction from Transesophageal Echocardiography Reports

Year: 2026
Type: preprint
Abstract: Objectives To develop an auditable, two-institution reference standard for intraoperative transesophageal echocardiography (TEE) reports and evaluate whether reviewer disagreement identifies limitations in the underlying clinical information model. Materials and Methods We selected 300 reports (150 per institution) from 925 at the University of Pennsylvania and University of Washington using random sampling and enrichment for challenging documentation. Four trained abstractors applied a shared 39-variable V1 codebook. Independent paired comparisons included 297 reports; three historical reports contributed only to the final reference standard. Categorical agreement used exact agreement and unweighted Cohen's κ; numeric left-ventricular ejection fraction (LVEF) used exact agreement and mean absolute differences. Disagreements underwent structured expert adjudication and reason classification. Additional analyses assessed reviewer-pair, sampling-stratified, and population-weighted agreement. Results Among 11,583 formal reviewer-comparison cells, 10,293 (88.9%) were resolved by reviewer agreement and 1,290 (11.1%) required targeted expert adjudication (1,289 tasks). Codebook/schema ambiguity (40.8%) and interpretation/classification differences (25.2%) together accounted for 66.0% of targeted tasks. Adjudication was concentrated in aortic dissection, valve type, and post-intervention findings. Categorical κ ranged from 0.364 to 0.960 and was inversely correlated with targeted-adjudication burden (Spearman ρ=−0.884; 95% CI, −0.955 to −0.734). Sampling-weighted κ changed by no more than 0.05 for 32 of 35 categorical variables. Discussion Structured adjudication identified recurring schema and interpretation challenges beyond individual abstraction errors, providing specific targets for information-model refinement. Conclusion As NLP and LLM-based abstraction tools move toward clinical use, the reference standards used to evaluate them require comparably rigorous development, adjudication, and independent validation.
Source: medRxiv
Language: English
Open Access status: green
OpenAlex ID: W7237237157
Cites: 25
Cited by:
Related to: 10