Work

Prompt Sensitivity in AI Writing: A Methodological Caution for Corpus‐Based Analysis of AI‐Generated Language

Year: 2026

Type: article

Abstract: ABSTRACT Recent corpus‐based studies suggest generative AI systems such as ChatGPT can approximate human patterns of interpersonal metadiscourse in argumentative writing. These conclusions are typically based on AI‐generated texts produced using a single prompt, despite AI output being highly sensitive to even minor changes in prompt specification. This study examines prompt specificity as an elicitation variable in the production of metadiscourse. Using Hyland's interpersonal metadiscourse framework, we first compared university students’ argumentative essays from the LOCNESS corpus with GPT‐generated essays produced in response to a general essay (baseline) prompt specifying role, genre, topic, length and persuasive purpose, but providing no explicit guidance regarding metadiscourse use. We then generated a second GPT corpus using a metadiscourse‐inducing prompt that explicitly requested interactive, interactional, and engagement features that were found to be underrepresented in the initial GPT corpus relative to the LOCNESS corpus. The baseline prompt produced lower frequencies of transitions, hedges, self‐mention, attitude markers and engagement resources than human writing, while the metadiscourse‐inducing prompt substantially increased several target features over the baseline and human conditions. Some categories converged with baseline frequencies, but even where convergence occurred, these were often realised in more formulaic ways. The findings show prompt specificity significantly shapes both the distribution and rhetorical realisation of metadiscourse in AI‐generated writing. More broadly, the results highlight the importance of treating prompt specification as a key methodological variable when interpreting linguistic patterns in AI‐generated texts.

Source: Modern Language Journal

DOI: 10.1111/modl.70089

Authors: Peter Crosthwaite, Shuyi (Amelia) Sun, Feng (Kevin) Jiang

Institutions: The University of Queensland, Nankai University, Beihang University

Language: English

Topic: Discourse Analysis in Language Studies

Subfield: Literature and Literary Theory

Field: Arts and Humanities

Domain: Social Sciences

Open Access status: hybrid

Estimated APC paid: $3,800

Funder: Tianjin Municipal Education Commission