Year: 2026
Type: preprint
Abstract: AI-supported writing gains may combine durable independent proficiency, immediate assistance, and changes in measurement conditions. This fully synthetic Monte Carlo study compared six estimators under four primary scenarios (2,000 replications per scenario; n = 600 per replication), expanded sensitivity scenarios (1,200 replications each), and independent missing-data validation. A further 1,200-replication analysis per response mechanism evaluated fitted inverse-probability weighting (IPW) with stacked-sandwich uncertainty; paired-mode calibration was stress-tested under assessment-period effects, directional carryover, and task-by-mode interactions. Under zero true AI-related learning and a mean direct assistance effect of 0.50 standardised score units, assisted ANCOVA estimated the supported-outcome effect as 0.498 and yielded a statistically positive result in every primary replication when mistakenly interpreted as autonomous learning; it was not biased for its legitimate assisted-outcome target. Representative paired calibration reduced endpoint bias to −0.003 (94.7% coverage). In the new interval experiment, fitted-IPW stacked-sandwich coverage for the true unaided-transfer effect was 94.0% under MCAR, 93.8% under MAR, and 24.7% under MNAR; corresponding MNAR effect bias was −0.153. With balanced assessment order, a stipulated 0.20-unit carryover effect still produced +0.103 calibration bias. Conclusions: Supported performance, independent endpoint learning, and subsequent unaided transfer are distinct targets. Valid inference depends on construct alignment, measurement transportability, observability, selection assumptions, and appropriate uncertainty estimation. All effects and operating characteristics are conditional on synthetic data-generating mechanisms, not observed AI effects in classrooms.
Source: Research Square
Author: Tan Nguyen Huynh Si
Institution: University of Phan Thiet
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W7236208912
