Get started
Work

Beyond Automated Scoring: An Evidence-Informed Human–GenAI Complementarity Framework for Formative Writing Assessment

Year: 2026
Type: preprint
Abstract: Generative artificial intelligence (GenAI) is increasingly used in formative writing assessment, yet research has focused largely on scoring accuracy and feedback quality rather than on how GenAI and teachers may contribute distinct assessment functions. This study examined teacher–GenAI rating consistency, agreement, and practical equivalence, as well as differences in their formative feedback characteristics. Using a convergent mixed-methods design, 100 ESL student essays were independently assessed by three teacher-raters and three GenAI systems using a common analytic writing rubric. Quantitative data were analyzed using descriptive statistics, intraclass correlation coefficients, Wilcoxon signed-rank tests, and Two One-Sided Tests (TOST) with a ± 0.30 equivalence margin. Formative feedback was examined through reflexive thematic analysis. Results showed good inter-rater reliability within teacher (ICC = .815) and GenAI (ICC = .879) groups but limited teacher–GenAI agreement (ICC = .353). No statistically significant differences were found between teacher- and GenAI-generated ratings across the assessed dimensions. Equivalence testing indicated practical equivalence for organization, language use, mechanics, and overall ratings, but not content. Qualitative findings showed that teachers emphasized language accuracy, organization, and specific corrective guidance, whereas GenAI provided broader, multidimensional feedback. Integrating these findings, the study develops an Evidence-Informed Human–GenAI Complementarity Framework for Formative Writing Assessment, positioning GenAI as a support for, rather than a replacement for, teacher judgment.
DOI: 10.21203/rs.3.rs-10784458/v1
Language: English
Subfield: Education
Open Access status: green
OpenAlex ID: W7204200228
Cites: 28
Cited by:
Related to: 10