What matters beyond model choice for wearable sleep staging? How personalization, evaluation choices, and easy-to-classify wake impact performance
Year: 2026
Type: article
Abstract: Study Objectives Methods to improve sleep classification of wearable data often emphasize model choice. However, without widely used benchmark datasets, it is difficult to determine factors driving performance gains (model architecture, dataset selection, or evaluation decisions). This study examines how dataset characteristics influence sleep classification performance beyond model choice. Methods We collected SleepAccel-Clinical, a dataset of Apple Watch acceleration and polysomnography from 28 individuals with sleep apnea. Neural network models were trained on this dataset alongside SleepAccel (31 healthy individuals) and DREAMT (100 individuals with suspected sleep disorders) to evaluate how training and testing set choices impact performance. All models were compared against an Easy To Classify (ETC) wake model, which estimates wake probability by smoothing and scaling activity in a brief time window. The ETC model’s average area under the receiver operating characteristic curve (AUROC) on each dataset was used to quantify intrinsic ease of classification. Results The ETC model accounted for substantial amount of the models’ performance across the architectures, datasets, and training conditions, with significant correlations (p≪ .01) between ETC performance and all other scenarios. Including individuals with obstructive sleep apnea (OSA) in the training data improved performance when testing on datasets in which sleep disorders were suspected. Conclusions Sleep classification performance depends heavily on training and testing dataset characteristics, not solely on model choice. The presence of ETC wake epochs can markedly inflate performance metrics. Researchers should benchmark models on widely available datasets and quantify intrinsic class separability when introducing new datasets. This article is part of the Consumer Sleep Technology Special Collection. Statement of Significance This work addresses a critical reproducibility challenge in sleep research by demonstrating that performance differences in wearable-based sleep classifiers are dominated by dataset characteristics rather than model architecture alone. By analyzing three datasets including a newly introduced cohort with sleep apnea, we reveal how the presence of ETC wake periods and training set choice substantially affect reported metrics. These findings challenge the field’s focus on architectural innovations while highlighting systematic biases in performance evaluation. We advocate for standardized benchmark datasets and transparent data sharing practices to enable meaningful comparisons across studies. This work provides essential guidance for researchers developing and evaluating wearable sleep classification systems, ultimately advancing toward more reliable and generalizable solutions.
Source: SLEEP Advances
Institution: Henry Ford Health
Language: English
Topic: Sleep and related disorders
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $2,054
OpenAlex ID: W7162425698
