Can ChatGPT and Bard generate aligned assessment items? A reliability analysis against human performance
Year: 2023
Type: article
Abstract: ChatGPT and Bard are AI chatbots based on Large Language Models (LLM) that are slated to promise different applications in diverse areas. In education, these AI technologies have been tested for applications in assessment and teaching. In assessment, AI has long been used in automated essay scoring and automated item generation. One psychometric property that these tools must have to assist or replace humans in assessment is high reliability in terms of agreement between AI scores and human raters. In this paper, the reliability of OpenAI’s ChatGPT and Google’s Bard LLMs tools against experienced and trained humans in perceiving and rating the complexity of writing prompts is measured. Intraclass correlation (ICC) as a performance metric showed that the reliability of both ChatGPT and Bard was low against the gold standard of human ratings.
Author: Abdolvahab Khademi
Institution: University of Maryland, Baltimore
Language: English
Subfield: Education
Field: Social Sciences
Domain: Social Sciences
SDG: Quality education
Open Access status: diamond
OpenAlex ID: W4376139550
