A Brief AI Literacy Intervention Does Not Significantly Reduce Over-Reliance and Increases Under-Reliance on ChatGPT: A Randomized Study
Year: 2025
Type: preprint
Study designs: Randomized Controlled Trial, Clinical Trial
Abstract: In this study, we examined whether a brief AI literacy intervention influences high school students' reliance on recommendations from large language models (LLMs). In a randomized experiment, students were assigned to either a control group receiving a brief introduction to LLMs or an intervention group receiving additional information about how LLMs work, their limitations, and effective usage strategies. Participants then solved eight math puzzles with ChatGPT's advice, which was incorrect in half of the trials. Results indicated widespread over-reliance, with incorrect recommendations adopted in 52.1% of the trials. The intervention did not significantly reduce over-reliance. Instead, it led to an increase in under-reliance, as students were more likely to reject correct recommendations. These findings provide preliminary evidence that brief text-based interventions may be ineffective in fostering appropriate reliance. More comprehensive and interactive approaches may be required to meaningfully influence students' real-world reliance on LLMs.
Source: arXiv (Cornell University)
Authors: Puppart, Brett, Aru, Jaan
Institution: University of Tartu
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: green
Funder: Eesti Teadusagentuur
OpenAlex ID: W4416040326
Cites:
Cited by:
Related to: