The use of expert judgement and pairwise comparison in the establishment of an assessment scale
Year: 2024
Type: article
Abstract: This paper explores the validation of an assessment by two complementary processes. Expert judgment assigns difficulty values to items on the basis of long-standing knowledge of as assessment scale; pairwise comparison establishes item difficulty values through statistical analysis. Given that expert judgement is accepted as the cornerstone of the accurate setting of test items at particular points on a proficiency scale, the accuracy of the expert judgement is crucial. In the study, the statistical pairwise comparison of test items provides a Rasch-calibrated, external perspective on item difficulty. Twelve experienced item writers rated a set of 23 multiple-choice items drawn from a large set of calibrated items via pairwise comparison principles. In the analysis, good fit statistics emerged, with difficulty values very comparable to the original expert-proposed values. The study validates how ‘expert’ item writers can be seen to produce accurately-located items, and that faith may be placed in their judgements.
Authors: Coniam, David, Lee, Tony, Lampropoulou, Leda
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W6965078164