Good Kitty, Bad Bank? Rescoring Miscalibrated CATs Improves Accuracy
Year: 2026
Type: preprint
Abstract: Computerized adaptive tests (CAT) depend on accurately calibrated item banks, but in practice, the item parameters governing deployment of the item banks are often based on small pilot samples, transported across populations, or assigned placeholder values pending proper calibration. We decompose the impact of miscalibration on subsequent ability estimates into two components: item selection error and scoring error. While selection error is not recoverable, rescoring using better calibrations, and without any additional testing burden, can potentially provide a post-hoc fix to scoring errors. In a simulation study, we find that rescoring substantially reduces root mean squared error (RMSE), with scoring error accounting for 86 to 96% of miscalibration-attributable error. In Study 2, we apply this framework to data from 2,657 children (ages 5 to 12) assessed across three pilot sites in the LEVANTE project. Using identical scoring methods with old versus new item parameters, we found that rescoring shifted scores substantially and reordered children's relative standing; external validation confirms the rescored estimates are better, not just different. Low-ability and younger children are disproportionately affected by rescoring, underscoring the equity implications. These results highlight the utility of rescoring for any CAT using early-stage, transported, or AI-generated item parameters, and how rescoring can enable rapid deployment of assessments to new populations.
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7163829031