Accuracy misleads under base-rate imbalance: a large-language-model rater recovers neurotype labels from semantic context, whereas a label-stripped keyword rule does not
Year: 2026
Type: preprint
Abstract: A label-blind test of two automated raters for subjective neuro-type coding. After stripping diagnostic words, labels, and fame cues, a single LLM rater on a 31-item gold-standard subset recovered the binary ND/non-ND umbrella at Cohen's kappa 0.793/0.788 (accuracy 0.93; fine-label weighted kappa 0.566; Spearman with continuous posteriors .544 overall, .399 ADHD, .256 ASD); its weakness was ADHD-vs-ASD subtype assignment (two high-P_ND .90 cases flipped between ADHD and ASD). A strict de-labeled mechanical rule bag on 200 subjects collapsed to binary kappa 0.029 despite a misleading raw accuracy of 0.80 (an NT-majority base-rate artifact), labeled 198/200 as NT, and missed 21 ADHD and 12 ASD key cases. The contrast shows the construct is a contextual semantic judgment rather than keyword matching; the LLM is a useful but same-source single-rater auxiliary that cannot replace the preregistered >=3-human-rater Fleiss kappa gate. Short report; labels are most-plausible hypotheses, not diagnoses.
Author: Xu, Xin
Institution: Zhejiang University
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7223463164
Cites:
Cited by:
Related to: