Work

People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text

Year: 2025

Type: conference-paper

Abstract: In this paper, we study how well humans can detect text generated by commercial LLMs (GPT-4O, CLAUDE-3.5-SONNET,O1-PRO).We hire annotators to read 300 non-fiction English articles, label them as either human-written or AI-generated, and provide paragraph-length explanations for their decisions.Our experiments show that annotators who frequently use LLMs for writing tasks excel at detecting AIgenerated text, even without any specialized training or feedback.In fact, the majority vote among five such "expert" annotators misclassifies only 1 of 300 articles, significantly outperforming most commercial and open-source detectors we evaluated even in the presence of evasion tactics like paraphrasing and humanization.Qualitative analysis of the experts' freeform explanations shows that while they rely heavily on specific lexical clues, they also pick up on more complex phenomena within the text that are challenging to assess for automatic detectors.We release our annotated dataset and code to spur future research into both human and automated detection of AI-generated text.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2025.acl-long.267

Authors: Jenna Russell, Marzena Karpinska, Mohit Iyyer

Institutions: University of Maryland, College Park, Microsoft (United States), University of Massachusetts Amherst

Language: English

Topic: Authorship Attribution and Profiling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funders: National Science Foundation, Open Philanthropy Project

Awards: Collaborative Research: STEM Learning Embedded in a Machine-in-the-Loop Collaborative Story Writing Game, Collaborative Research: RI: Medium: Multilingual Long-form QA with Retrieval-Augmented Language Models