The Measurement Reliability Crisis in Event-Related Potential Research: Evidence from Bilingual Language Control
Year: 2026
Type: preprint
Abstract: Event-related potential (ERP) research exhibits a critical methodological blind spot: inter-condition correlations (ρXY), the parameter that mathematically determines difference wave reliability via the Lord–Novick formula, are systematically unmeasured. Classical Test Theory demonstrates that difference wave reliability is inversely related to ρXY, with high correlations suppressing reliability even when constituent waveforms are psychometrically sound. Systematic examination of four foundational ERP studies in bilingual language switching (2001–2016, Ncitations ≈ 1,575) confirms that none report difference wave reliability or inter-condition correlations, despite difference waves being the primary dependent variables isolating language control processes. Sensitivity analysis reveals that this vulnerability extends across the entire plausible parameter space: even moderate inter-condition correlations (ρXY = .50–.60) require constituent reliabilities approaching methodological ceilings (ρ = .85–.88) to achieve adequate difference wave reliability (ρDD′ ≥ .70). When experimental control produces high correlations (ρXY ≥ .70), degradation becomes catastrophic, with constituent reliabilities of .85 yielding difference wave reliabilities near zero. The absence of empirical ρXY measurements means the field cannot determine whether two decades of N2 debate reflect genuine neurocognitive complexity, measurement instability, or both. Three reforms can address this gap: mandatory reporting of difference wave reliability and inter-condition correlations, adoption of Generalizability Theory frameworks, and single-trial analytical methods that avoid categorical subtraction. These principles extend beyond bilingual research to all ERP domains using difference waves. Systematic psychometric verification should precede theoretical elaboration.
Source: PsyArXiv (OSF Preprints)
Authors: Projnya Mojumdar, Purba Mojumdar
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: green
OpenAlex ID: W7130712468