Work

Multimodal Writing Evaluation in Digital Storytelling using Video-Based Output: Comparing performance of AI and Human Raters.

Year: 2024

Type: conference-paper

Abstract: With the rapid advancement of artificial intelligence (AI) models, it is now possible to perform content analysis on multimodal videos using this kind of technology. This study investigated the consistency between AI assessments and human evaluations of multimodal writing tasks, specifically comparing the scores assigned by AI models with those assigned by human raters. The research sample is composed of five multimodal writing assignments from a second language learning classroom. We compared the scores from AI and human raters using intraclass correlation coefficients (ICC) and within-group agreement reliability (rwg). The results showed minimal differences between the scores from the AI and human raters. Mean rwg values revealed a high level of agreement between human and AI assessments, whereas ICC values indicated slightly lower consistency compared to human raters. Additionally, human raters provided polarized responses to the feedback generated by AI. These findings provide evidence that AI could serve as a supplementary tool for assessing multimodal writing tasks.

Source: International Conference on Modern Educational Technology (ICMET)

DOI: 10.1145/3729434.3729446

Authors: Sichen Liu, Eunyoung Kim

Institutions: Japan Advanced Institute of Science and Technology, Guangdong Ocean University

Language: English

Topic: EFL/ESL Teaching and Learning

Subfield: Language and Linguistics

Field: Arts and Humanities

Domain: Social Sciences

SDG: Quality education

Open Access status: bronze

Funders: Guangdong Ocean University, Japan Society for the Promotion of Science