The Examination of Reliability According to Classical Test and Generalizability on a Job Performance Scale
Year: 2010
Type: article
Abstract: In this research, the classical test theory and generalizability theory analyses were carried out with the data obtained by a job performance scale for the years 2005 and 2006. The reliability coefficients obtained (estimated) from the classical test theory and generalizability theory analyses were compared. In classical test theory, test retest and Cronbach's alpha reliability coefficient, Kendall's concordance coefficient for interrater reliability; in generalizability theory, G and Phi coefficients with multivariate model were computed and the consistency of these coefficients were examined. As the result of the research, it is seen that for the same measurement condition, the reliability coefficients obtained from the classical test theory and from the multivariate model of generalizability theory generated harmonious results. Key Words Reliability, Generalizability Theory, Classical Test Theory. The essence of Spearman's model was that any observed score could be envisioned as the composite of two hypothetical components (a true score component and a random error component) (Crocker & Algina, 1986). The true score formula can be written as Observed score= True score + Random error. The relationship between true score and observed score as expressed in a mathematical model was explained by Spearman in what has become known as the true score model or classical test theory. Based on the positive or negative influences of the random error component, it is possible for a respondent's observed score to be higher or lower than the same respondent's true score (Anastasi, 1988; Murphy & Davidshoper, 1998; Rust & Golombok, 1992; Tavsancil, 2002; Thorndike, 1988). The classical test theory partitions observed score variance into only two components, true score variance and random score variance (Guilford, 1954; Gulliksen, 1967; Lord & Novick, 1968). Consequently, it is possible to examine only a single source of measurement error at any given time. This poses a serious dilemma for the researcher, since in reality several types of measurement errors can exist concurrently (Baykul, 2000; Erku, 2003). In classical test theory, it possible to consider estimates of measurement error due to inconsistency in forms (equivalence), observers (interrater agreement), sampling the item domain (internal consistency or split-half), or time (test-retest or stability). Only one measurement error influence can be considered in a given analysis. Each type of reliability estimate can be used to determine the degree to which true scores deviate from observed scores. The problem, however, is that classical test theory is unable to examine inconsistencies in test forms, raters, items, or occasions simultaneously (Brennan, 2001; Shavelson & Webb, 1991). Generalizability theory, which subsumes classical test theory as a special case, also extends classical test theory by recognizing and estimating the magnitude of the multiple sources of error (Brennan, 2001; Crocker & Algina, 1986). Both sources of error variance and interactions among these sources can be considered simultaneously in a single generalizability analysis (Cronbach, Rajaratnam, & Gleser, 1963). Classical test theory admits consideration of only one type of measurement error at a time and does not consider the possible, completely independent, or separate interaction effects of the measurement error variance. In G theory, unlike classical test theory, a behavioral observation is considered only one sample of behavior from an infinite universe of admissible observations (Atilgan & Tezbasaran, 2005). The universe, therefore, is comprised of all the potential measurements that would be a direct substitute for the observation under investigation. The particular sample or observed score is not the focus of the generalizability analysis; rather, the analysis focuses on how well a particular sample of behavior generalizes to the larger universe of admissible observations. …
Authors: Atilla Yelboğa, Ezel Tavşancıl
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: diamond
OpenAlex ID: W2147832127