Work

Site confounding inflates reported accuracy in conjunctival anaemia screening: a reproduction study and confound audit on the EYES-DEFY-ANEMIA dataset

Year: 2026
Type: preprint
Abstract: Photographing the palpebral conjunctiva to screen for anaemia is an active research area, and published systems report high accuracy. Most of those results come from datasets assembled at more than one clinical site, where disease prevalence, patient age and skin tone differ between sites. This study asks how much of that accuracy survives when the classifier is graded inside a single site, and how predictable the site itself is from the same image features. Using the public EYES-DEFY-ANEMIA dataset (218 conjunctival photographs; 95 Indian and 123 Italian subjects; laboratory haemoglobin; CC BY-SA 4.0), I computed 21 colour statistics inside the supplied conjunctiva mask for each of the 200 subjects with a usable haemoglobin value and mask, and fitted an L2-regularised logistic regression implemented directly in numpy. Performance was measured by out-of-fold pair ranking (AUC) under five protocols: pooled across sites, site prediction, within each site, cross-site transfer, and an age-only control. Confidence intervals came from 2,000 bootstrap resamples. Fitted weights were tested for stability by deleting one subject at a time, and calibration was measured. Results: pooled anaemia classification reached AUC 0.881 (95% CI 0.833-0.927; 8,702 of 9,879 pairs correctly ordered). The same 21 features predicted which site the photograph came from at AUC 0.998 (9,948 of 9,964 pairs). Graded inside India alone, anaemia classification fell to 0.666 (0.550-0.778; 1,204 of 1,809). Inside Italy it was 0.952, but age alone reached 0.832 there, and Italian subjects were on average 16.5 years older. Transfer was asymmetric: India to Italy 0.784, Italy to India 0.638. Weight inspection showed the feature the hypothesis was built on carries almost no weight: conjunctival redness (a*) has a weight of -0.04 against +0.77 for the largest term, and it changes sign when any single subject is removed. Conclusion: in this dataset the headline accuracy is substantially a site signal. Reporting a pooled AUC without a site-prediction control and a within-site result overstates performance by roughly 0.2 AUC points. The deposit proposes a four-item reporting check and releases the feature table, out-of-fold predictions and fitted model so that every figure can be recomputed without the analysis code. The deposit includes a verification script (verify.py) that recomputes 46 of the reported numbers directly from the released data - cohort counts, every protocol with its pair counts, the abstention bands, calibration, the fitted weights and one subject scored end to end - using only the Python standard library. The paper can be checked without installing anything and without trusting the analysis code. This is a feasibility study on 200 public photographs from two sites. It is not a clinical validation and nothing in it supports medical use.
Language: English
Open Access status: green
OpenAlex ID: W7236801462
Cites:
Cited by:
Related to: 10