A Granularity Blind Spot in the Evaluation of SNOMED CT Concept Normalization
Year: 2026
Type: preprint
Abstract: Background. Clinical concept normalization maps free-text mentions to terminology codes but is usually evaluated with aggregate exact match and a binary seen/novel split. These summaries do not distinguish mentions recorded as finer-grained than their assigned code. Using native granularity labels in three Spanish SNOMED CT corpora, we tested whether exact-match error incidence differs between NARROW and EXACT reference strata across up to five lexical, neural and generative systems. Results. Systems failed to reproduce the corpus-assigned code under exact match more often for NARROW mentions (finer-grained than their assigned code) than for EXACT mentions (panel-weighted Mantel–Haenszel risk ratio 1.72, 95% CI 1.63–1.83). The Wu–Palmer-minus-exact credit gap was larger on NARROW mentions in all 14 system×benchmark cells (geometric mean ratio 1.87, 95% CI 1.71–2.05), driven by error incidence rather than a consistent difference in conditional error proximity. In the pooled Qwen analysis, sibling and cousin predictions contributed 56–67% of summed Wu–Palmer credit among non-exact predictions. Under a stricter same-lineage criterion, the uplift beyond exact match was larger for NARROW mentions in 9 of 14 cells. The larger graded gap retained its direction under alternative metrics, candidate spaces and xMEN. Graph-refined novelty classified 825 of 982 code-novel DisTEMIST mentions (84.0%) as ancestor-seen; the remaining 157 were fully isolated (6.3% of all 2,507 mentions). Majority-vote consensus in the blinded three-rater audit agreed with the corpus labels for 130 of 150 mentions (κ = 0.73). Conclusions. Aggregate exact match obscures a benchmark-relative difference between reference-granularity strata. For benchmarks with granularity labels, granularity-stratified exact-match error rates should be reported first; hierarchy-aware scores should be paired with relation types.
Source: Research Square
Authors: Yejin Jo, Yoojin Nam, YangWon Kim, Eunji Kim, Pa Hong
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225783878