Learning distribution-calibrated and prompt-robust quality representations for cross-prompt automated essay scoring inEnglish writing assessment
Year: 2026
Type: article
Abstract: In English language teaching and writing assessment, cross-prompt automated essay scoring (AES) requires models trained on existingsource prompts to generalize to unseen prompts without relying on target-prompt responses. A central challenge is that prompts differ not only in rubric ranges but also in score distributions and optimization characteristics, so numerical score normalization alone may not provide transferable supervision. This study proposes DCR-AES, a source-only framework that combines prompt-relative empirical-distribution calibration, separation-filtered cross-prompt relative-quality learning, and prompt-level distributionally robust optimization. Human scores are mapped to mid-rank empirical cumulative distribution function (eCDF) positions estimated exclusively from source-prompt training data. These positions are interpreted as prompt-relative ordinal coordinates rather than absolute cross-prompt quality equivalents and are used to construct ranking constraints only when the calibrated separation is sufficiently large. Prompt-level robust optimization further adapts source-domain contributions according to persistent training loss while retaining a uniform contribution from all source prompts. Under strict leave-one-prompt-out evaluation on ASAP++, the five-run analysis yields a prompt-level QWK of 0.593 ± 0.004 (95% CI [0.588, 0.598]) and an attribute-level QWK of 0.590 ± 0.004. DCR-AES improves over PLAES by 0.0179 QWK across the eight held-out prompts (Wilcoxon p = 0.0078; rank-biserial effect size = 1.00), does not differ significantly from ProTACT, and remains below SICTS and stronger resource-enhanced systems in absolute QWK. Additional analyses show that calibration gains are positively associated with source-target score-distribution shift, while learned DRO weights correlate strongly with persistent training loss but not with static sample count or score-distribution descriptors. The results support prompt-relative distribution information as useful auxiliary supervision while clarifying the limits of percentile alignment and loss-based prompt weighting.
Source: Scientific Reports
Author: Yibin Huan
Institution: Zhoukou Normal University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7225878185
