Different but the same? Internal structure, measurement invariance, DIF, and reliability evidence across test modes
Year: 2026
Type: article
Study design: Observational Study
Abstract: This study examined whether the validity and reliability of scores differed between online and face-to-face administrations of instruments that varied in length and measured typical response and maximum performance. A nonrandomized comparative desing was used. Typical-response data (5-item Perseverance Scale; 30-item Perceived Academic Self-Efficacy Scale) were collected from 568 middle school students (335 paper-and-pencil, 233 online); achievement data (9-item reading and 25-item listening tests) were collected from 94 university preparatory students (50 paper-and-pencil, 44 online). Model fit and measurement invariance were examined using confirmatory factor analysis and multiple-group confirmatory factor analysis for ordinal data respectively; differential item functioning (DIF) using Mantel-Haenszel and logistic regression, requiring statistical significance and a non-negligible effect size, and Cronbach’s alpha coefficients using Feldt’s F-test. Both typical response scales showed acceptable fit in both modes. The Perseverance Scale reached metric invariance while the Self-Efficacy Scale reached only configural invariance. Alpha did not differ for the Perseverance Scale (0.78 in both modes) or the long achievement test (0.80 online vs. 0.67 face-to-face, p = 0.097), but was significantly lower online for self-efficacy (0.83 vs. 0.88, p = 0.011) and lower face-to-face for the short achievement test (0.79 vs. 0.59, p = 0.027). No item met the DIF criterion in the short form; in the long form, none showed uniform DIF, but 3 of 25 items (12%) showed non-uniform DIF. Psychometric equivalence across modes could not be assumed and should be evaluated empirically for each instrument and population.
Source: Frontiers in Psychology
Authors: Neslihan Tuğçe Özyeter(Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Writing – original draft, Writing – review & editing), Ömer Kamış(Data curation, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing), Hatice Özlem Anadol(Data curation, Investigation, Writing – original draft, Writing – review & editing)
Language: English
Field: Decision Sciences
Domain: Social Sciences
SDG: Quality education
Open Access status: gold
Estimated APC paid: $3,801
OpenAlex ID: W7233767232
