Do machine learning models really encode cross-cultural lifestyle-health differences? A leakage-free reanalysis of two university cohorts in India and Japan
Year: 2026
Type: article
Abstract: Introduction: Machine-learning (ML) studies increasingly compare feature-importance rankings between small, unevenly sized cohorts and interpret the differences as evidence of culturally specific "risk architectures." We show that such comparisons can be highly misleading without leakage-free evaluation, using a worked example from our own cross-sectional lifestyle and health data collected during June-August 2024 from two universities in India and Japan. Three problems were identified in our preliminary analysis: (1) an outcome (obesity, defined as BMI ≥ 25) that was tautologically predicted by including BMI itself as a model input; (2) model evaluation without a held-out test set; and (3) feature-importance comparisons that ignored sampling variability from an eightfold difference in cohort size. Methods: We addressed each problem using penalized logistic regression, Random Forest, and gradient-boosted trees with proper train/test splitting, held-out permutation importance, and bootstrap confidence intervals. We also ran a sensitivity analysis that repeatedly subsampled the Indian cohort down to the size of the Japanese cohort. Results: Removing body mass index (BMI) as a predictor collapsed obesity-model discrimination from perfect (AUC = 1.00) to chance level (AUC = 0.50-0.57), confirming that the BMI-obesity association was definitional rather than empirical. Held-out predictive performance for the remaining viable outcome, elevated self-rated stress, was modest at best (AUC 0.49-0.71). Critically, in the subsampling analysis, every feature-importance value observed in the actual Japanese data fell within the range produced by sample-size noise alone. Discussion: We present this analysis as a transparent, reproducible case study of specific, checkable pitfalls: label leakage, missing train/test evaluation, unadjusted small-sample comparison, and mislabeled psychometric instruments. These should be routinely screened for before ML feature importance is interpreted as evidence of population-level or cultural differences in digital health research.
Source: Frontiers in Digital Health
Language: English
Subfield: Health Information Management
Field: Health Professions
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $3,077
OpenAlex ID: W7222538825