Overview of the “Voight-Kampff” Generative AI Authorship Verification Task at PAN and ELOQUENT 2024
Year: 2024
Type: conference-paper
Abstract: The “Voight-Kampff” Generative AI Authorship Verification task aims to determine whether a text was generated by an AI or written by a human. The 2025 edition of the task explores two subtasks: Subtask 1 tests the detection of purely AI generated text with potentially unknown obfuscations, and as such continues our research from 2024. The task is again organized as a builder-breaker challenge together with the ELOQUENT lab. The PAN participants submitted 24 detectors. The best system archives a mean score of 0.99, the best baseline achieves a score of 0.92. ELOQUENT participants submitted 13 new test datasets with 22 obfuscated texts each. The most difficult dataset archives a mean C@1−1 score of 0.63. Subtask 2 investigates texts with six degrees of human-AI collaboration: (i) fully human-written, (ii) human-written, then machine-polished, (iii) machine-written, then machine-humanized (obfuscated), (iv) human-initiated, then machine-continued, (v) deeply mixed text,where some parts are written by a human and some are generated by a machine, and (vi) machine-written, then human-edited. The dataset contains over half a million examples in total and is composed from several relevant AI-detection datasets across multiple text genres. PAN participants submitted 21 detectors to subtask 2. The best system archives an F1 score of 0.65, the best baseline a score of 0.48. The data, baselines, and the code used for creating the datasets and evaluating the systems are available.
Authors: Bevendorff, Janek, Wang, Yuxia, Karlgren, Jussi, Wiegmann, Matti, Fröbe, Maik, Tsivgun, Akim, Su, Jinyan, Xie, Zhuohan, Abassy, Mervat, Mansurov, Jonibek, Xing, Rui, Ta, Minh Ngoc, Elozeiri, Kareem Ashraf, Gu, Tianle, Tomar, Raj Vardhan, Geng, Jiahui, Artemova, Ekaterina, Shelmanov, Artem, Habash, Nizar, Stamatatos, Efstathios, Gurevych, Iryna, Nakov, Preslav, Potthast, Martin, Stein, Benno
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: European Commission
OpenAlex ID: W7070370652