Cross-cohort validation of EEG biomarkers using a pre-registered three-criterion framework: a demonstration in amyotrophic lateral sclerosis
Year: 2026
Type: preprint
Abstract: Background. Despite decades of EEG biomarker development for amyotrophic lateral sclerosis (ALS), no candidate has progressed to clinical deployment, partly because cross-cohort validation is rare. We aimed to develop and demonstrate a pre-registered three-criterion screening framework to identify which candidate EEG features show the combined properties required for biomarker translation: cross-sectional discrimination, within-subject reliability, and cross-cohort effect consistency. Methods. We pre-registered a validation framework adapted from FDA-NIH BEST principles and applied it to 18 EEG features across three independent cohorts: EEGET-ALS (six ALS patients, 170 healthy controls; home recordings over 17 days), UCL/Penn State (eight ALS patients; clinic recordings over 1-2 months), and a healthy adult test-retest reference (60 participants, three sessions over one month). Features were required to meet three pre-registered criteria: (1) cross-sectional discrimination (Cohen's d > 0.5), (2) within-subject reliability (ICC(3,1) > 0.6 in both ALS cohorts independently), and (3) cross-cohort effect direction consistency. Cross-cohort permutation testing (1000 iterations) and subject-level bootstrap resampling (500 iterations) were applied to assess robustness. Results. Three features met all three pre-registered criteria: gamma power, theta power, and spectral entropy. Cross-cohort permutation testing showed that no permutation, out of 1000, produced any feature in the three-cohort reliability intersection (Monte Carlo p = 0.001). Bootstrap resampling indicated that the composition of the selected set was sensitive to sample composition (median count 1, 95% CI [0, 4]), with gamma power (67% inclusion), spectral entropy (43%), and theta power (32%) the most stable features at a relaxed reliability threshold (ICC > 0.5); at the primary pre-registered threshold (ICC > 0.6), the corresponding inclusion probabilities are lower (57%, 27%, and 19%, respectively). A principal methodological finding was that cross-cohort differences in feature reliability were large relative to the apparent disease-associated signal, most clearly demonstrated by relative beta power (ICC = 0.09 in a home-recorded cohort versus 0.93 in a clinic-recorded cohort); because recording protocol, hardware, setting, population, and preprocessing all differ jointly between cohorts in this dataset, this is evidence of cohort-associated variability that plausibly includes a protocol contribution, not a separately identified, isolated protocol effect. Conclusions. This study provides a pre-registered, multi-cohort EEG biomarker screening framework for ALS with explicit bootstrap-quantified uncertainty. Gamma power, spectral entropy, and theta power are identified as priority candidates for prospective validation. A recurring methodological observation is that protocol-related cohort differences appear to contribute more to cross-cohort variability than disease-related effects within this dataset; because protocol is confounded with cohort, equipment, and preprocessing, this pattern is descriptive rather than causal. The observation nonetheless has direct implications for the design of multi-site EEG biomarker studies in neural engineering.
Source: medRxiv
Institutions: University of Toronto, Old Dominion University
Language: English
Subfield: Clinical Neurology
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225904430
