Year: 2024
Type: conference-paper
Abstract: Large language models (LLMs) bring significant opportunities in the field of education evaluation. Nowadays, most existing LLMs are introduced to generate reasonable scores for electronic and structurized documents. However, such systems require huge efforts for manual typewriting or advanced optical character recognition techniques. To this end, this work directly processes the scanning copy of English composition with multi-modal LLMs. Specifically, this research aims to utilize multi-modal LLMs for automated essay scoring (AES) and evaluate its reliability and accuracy. We take the English composition images of 1,511 tenth-grade examinees in a standardized test environment as the research objects for automated scoring, and explore the performance of the multi-modal LLM GPT-4o in evaluating composition image data. The study compares the consistency of the composition image scores of GPT-4o under zero-shot and two-shot prompts with the scores of the marking experts, and verifies them through multiple indicators such as the exact agreement coefficient, Pearson correlation coefficient, Coefficient of determination, Root Mean Square Error, and the probability that the GPT-4o score falls within a given confidence interval. Especially, comparing the scores given by experts, the R-square of the GPT-4o scoring reaches 0.66. The results show that GPT-4o has the ability to learn from two samples through prompt, can quickly adapt and provide high-quality and personalized evaluations for examinees, and can provide valuable support for human scoring.
Authors: Xiaohui Wang, Ruijie Yu, Yu Zhang, Yanyan Xu
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: gold
OpenAlex ID: W4406370936