Evaluating a candidate predictor beyond statistical significance: Atmospheric pressure and psychiatric day-care attendance as a worked example
Year: 2026
Type: article
Study design: Observational Study
Abstract: In observational research, statistical detectability, robustness to adjustment, and predictive utility are distinct properties of a candidate predictor, yet they are often conflated. A statistically significant association may weaken after accounting for co-varying factors and may add little to out-of-sample prediction. We illustrate this distinction using atmospheric pressure and daily attendance at a Japanese psychiatric day-care as a worked example. Using four consecutive fiscal years of facility-level daily attendance, we evaluated same-day atmospheric pressure as a pre-specified candidate predictor in three stages: statistical detectability, robustness to adjustment for co-varying meteorological factors and seasonality, and incremental out-of-sample predictive utility beyond institutional scheduling variables and recent attendance history. Same-day pressure was positively associated with attendance in all four years (incidence rate ratios 1.003-1.007 per hPa). However, effect-estimate stability varied across years, with marked attenuation in FY2022, moderate attenuation after precipitation adjustment in FY2023, and specification-sensitive precision in FY2024, whereas FY2025 was comparatively stable. In contrast, the structural baseline incorporating scheduling variables and recent attendance history provided strong out-of-sample prediction, explaining 55%-69% of the variance across years. Adding atmospheric pressure produced only small and unstable changes in the primary expanding-window cross-validated performance (mean ΔR2 ≈ +0.005), with increments varying in sign across years. Thus, a candidate predictor that was repeatedly statistically detectable showed limited stability of effect magnitude or precision under reasonable adjustment and little incremental predictive utility beyond a strong structural baseline. In this highly schedule-driven, single-facility setting, the weather signal examined contributed little incremental information for out-of-sample prediction of aggregate attendance. More broadly, this worked example illustrates why candidate predictors should be evaluated separately for detectability, robustness of effect estimates, and out-of-sample predictive utility before statistically significant associations are interpreted as meaningful or actionable.
Source: PLoS ONE
Authors: Takahiro Okuma, Minoru Ishige, Shuichi Kikuchi
Institution: Sodegaura Satsukidai Hospital
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,382
OpenAlex ID: W7225397675