Supplemental Material for Applied Competence at the Bachelor’s Level: A Reliability Study of Undergraduate Trainees in Cognitive and Achievement Test Administration
Year: 2026
Type: supplementary-materials
Abstract: Reliability EstimatesThe primary analysis estimated interrater reliability following, where possible, the methods outlined by the test publishers.The WISC-V Technical and Interpretive Manual specifies the use of intraclass correlations (ICC), while the GORT-5 reports product-moment correlations between two raters and multiple administrations.The Woodcock Johnson-IV Tests of Achievement authors provide no mention or estimate of interrater reliability.We adopted ICCs as our primary index because they are the most widely recommended for continuous or ordinal scoring of standardized test items and can be applied consistently across measures.We sought additional information as needed within their cited sources (McGraw & Wong, 1996; Shrout & Fleiss, 1979) and more recent papers (Koo and Li, 2016).We used a "Case 3" design for total score reliability [ICC(3k)] and item-level reliability [ICC(3,1)], as "each target is rated by each of the same k judges, who are the only judges of interest" (Shrout & Fleiss, 1979, p. 421).This employs a two-way, random effects ANOVA to estimate intraclass correlations.We also used Krippendorff's alpha to confirm item-level ICCs (Hayes and Krippendorff, 2007; Marzi et al., 2024) because it is robust to non-interval scaling and rater imbalance.These values were found to be nearly identical to ICC in each case, though we report both to maintain thoroughness.
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: bronze
OpenAlex ID: W7129679728
Cites:
Cited by:
Related to: