Work

Different but the same? Internal structure, measurement invariance, DIF, and reliability evidence across test modes

Year: 2026
Type: article
Study design: Observational Study
Abstract: This study examined whether the validity and reliability of scores differed between online and face-to-face administrations of instruments that varied in length and measured typical response and maximum performance. A nonrandomized comparative desing was used. Typical-response data (5-item Perseverance Scale; 30-item Perceived Academic Self-Efficacy Scale) were collected from 568 middle school students (335 paper-and-pencil, 233 online); achievement data (9-item reading and 25-item listening tests) were collected from 94 university preparatory students (50 paper-and-pencil, 44 online). Model fit and measurement invariance were examined using confirmatory factor analysis and multiple-group confirmatory factor analysis for ordinal data respectively; differential item functioning (DIF) using Mantel-Haenszel and logistic regression, requiring statistical significance and a non-negligible effect size, and Cronbach’s alpha coefficients using Feldt’s F-test. Both typical response scales showed acceptable fit in both modes. The Perseverance Scale reached metric invariance while the Self-Efficacy Scale reached only configural invariance. Alpha did not differ for the Perseverance Scale (0.78 in both modes) or the long achievement test (0.80 online vs. 0.67 face-to-face, p = 0.097), but was significantly lower online for self-efficacy (0.83 vs. 0.88, p = 0.011) and lower face-to-face for the short achievement test (0.79 vs. 0.59, p = 0.027). No item met the DIF criterion in the short form; in the long form, none showed uniform DIF, but 3 of 25 items (12%) showed non-uniform DIF. Psychometric equivalence across modes could not be assumed and should be evaluated empirically for each instrument and population.
Language: English
Open Access status: gold
Estimated APC paid: $3,801
OpenAlex ID: W7233767232
FWCI:
Cites: 61
Cited by:
Related to: 10
Parts: 1