Large Language Models on Wikipedia Editing: The Stylistic Mask Effect in a Randomized Controlled Intervention in Health Education
Year: 2026
Type: preprint
Study designs: Randomized Controlled Trial, Clinical Trial
Abstract: The effects of large language models (LLMs) on content, references, and language quality in Wikipedia assignments remain unclear. This randomized controlled study assessed how model assistance influences the quality of Wikipedia edits created by undergraduate audiology students. Thirty-six participants were assigned to two groups: Group 1 (G1) edited without model support, and Group 2 (G2) edited with ChatGPT support. Twenty blinded expert reviewers evaluated a sample of 30 texts (15 per group) using a six-item Likert-scale instrument covering Content, References, and Language. Inter-rater reliability was high (Gwet's AC2 = 0.80). The analysis suggests that LLM assistance may lead to a significant improvement in Language (β = 0.400; p = 0.023; Cliff's δ = 0.400). G2 improved by 0.68 ± 0.47, while G1 improved by 0.28 ± 0.57. The analysis found no significant differences for Content (β = 0.167; p = 0.214) or References (β = −0.267; p = 0.859), although G2 scores in the latter category trended lower. Output counts were not associated with quality improvement in either group. This asymmetry is proposed as a potential Stylistic Mask Effect. It describes a gap between surface language gains and limited substantive development. LLM assistance appears to risk substituting constructive contribution with linguistic polish. This record contains the preprint manuscript. The manuscript has been submitted for peer review and has not yet been published. Dataset and code are made available to support transparency of the reported analyses.
Language: English
Subfield: Communication
Field: Social Sciences
Domain: Social Sciences
SDG: Quality education
Open Access status: green
Awards: G1322486680, G3429860563, G7908040669
OpenAlex ID: W7162010701
Cites:
Cited by:
Related to: