The computer-based assessment of domain-specific problem-solving competence—A three-step scoring procedure
Year: 2020
Type: article
Abstract: Problem-solving competence is an important requirement in back offices across different industries. Thus, the assessment of problem-solving competence has become an important issue in learning and instruction in vocational and professional contexts. We developed a computer-based low-stakes assessment of problem-solving competence in the domain of controlling. The assessment required participants to process three complex and authentic business-related scenarios within an office simulation. The low number of items and the open-ended response threaten the reliability of the scoring. We collected data from 780 German students in Vocational Education and Training (VET) and applied the following procedure to score the participants’ solutions to the three scenarios: A fine-grained scoring resulted in a comprehensive set of response patterns for each scenario. Experts assigned partial credit scores for each of these response patterns and received feedback based on point-biserial correlations of WLE scores for the partial credit items with the total score of the respective subdimension. Finally, a multi-dimensional model was calibrated based on these partial credits and this model was further developed. This procedure leads to satisfying EAP/PV reliabilities between .78 and .84 for the subdimensions and .89 for the overarching construct.
Source: Cogent Education
Institutions: University of Mannheim, University of Hohenheim
Language: English
Field: Decision Sciences
Domain: Social Sciences
SDG: Quality education
Open Access status: gold
Estimated APC paid: $1,283
OpenAlex ID: W3005339283