Work

A “Human-Like” Stroop Effect No Longer Implies a Human Participant

Year: 2026
Type: preprint
Abstract: Researchers have long assumed that online behavioral experiments are completed by humans, an assumption AI agents can now violate. We asked whether agent and human data can be distinguished. We compared Stroop data from 51 respondents each: humans verified in person, an artificial experiment agent, and an online sample. All agent-generated respondents showed the congruency effect and passed six quality checks. A one-class support vector machine (SVM) trained only on human data achieved a median accuracy of 63%. A supervised SVM trained on human and agent data achieved 94%. At an assumed 5% contamination rate, most of its flags would be false alarms. The supervised SVM did not generalize to an agent it was not trained on. Neither plausible task performance nor the tested SVMs reliably distinguished human from agent data. These findings suggest that human participation should be safeguarded through study design rather than through screening afterward.
Language: English
Open Access status: green
OpenAlex ID: W7226010504
Cites:
Cited by:
Related to: 10