Work

Whose Meaning Is It Anyway? Measuring Semantic Similarity in AI-Revised L2 Writing Across Genres and Models

Year: 2026
Type: preprint
Abstract: Large language models (LLMs) are widely used to revise second language (L2) writing, raising questions about whether revision preserves writers’ meaning, stance, and voice. This study examines segment-level semantic similarity between original and AI-revised texts across three genres and four LLMs. The corpus comprised 72 texts by undergraduate English majors: 24 argumentative essays, 24 personal statements, and 24 critiques. Each text was revised by ChatGPT, Gemini, DeepSeek, and Doubao using a standardised prompt. Original and revised texts were manually segmented and aligned into semantically corresponding units. Similarity was measured using all-MiniLM-L6-v2 sentence embeddings and cosine similarity. One-way and two-way ANOVAs and Tukey HSD tests examined genre and model differences; a stratified sample of high- and low-similarity pairs was qualitatively coded. Overall, similarity scores were relatively high, but significant genre and model effects and a genre-by-model interaction emerged. Mean similarity was highest for critiques, followed by personal statements and argumentative essays. Doubao yielded the highest mean similarity, followed by ChatGPT, DeepSeek, and Gemini. Qualitative analysis indicated that high-similarity revisions mainly involved verbatim retention, surface correction, and meaning-preserving paraphrase, whereas lower-similarity revisions featured rhetorical reframing, shifts in voice or register, expansion, condensation, and deletion. These findings suggest that semantic preservation depends on both genre and model, and that embedding-based metrics require qualitative interpretation. They support genre-sensitive, model-aware, and critically guided use of LLMs in L2 writing instruction.
Language: English
Open Access status: green
OpenAlex ID: W7235171666
Cites:
Cited by:
Related to: 10