AI chatbots in higher education: Comparing expectations to evidence
Year: 2026
Type: article
Study designs: Randomized Controlled Trial, Clinical Trial
Abstract: Given the rapid advancements, and notable failures, in large language model generative AI (genAI), there are elevated expectations that retrieval-augmented generation (RAG) AI chatbots will revolutionize higher education by offering individualized, always-available tutoring based on validated content. However, experimental evidence on their effectiveness remains scarce. Using a randomized controlled field experiment, this study examines the effects of a genAI chatbot on key precursors to learning success (i.e., interest, self-efficacy, and engagement) and academic achievement for ≈500 undergraduate students across two modalities (in-person and asynchronous online). We completed a semester-long controlled experiment with pre- and post-treatment surveys and tests. Despite expectations, we found the genAI chatbot had no statistically significant impact on any measured outcome. These early results challenge assumptions about AI’s instructional effectiveness and suggest universities should further investigate the pedagogical value of AI chatbots before making substantial investments or committing to long-term contracts. We recommend future research to increase the generalizability of the findings and to discover methods to improve efficacy of AI chatbots in higher education.
Authors: Andrew Thoeni, Luke K. Fryer
Institutions: University of North Florida, University of Hong Kong
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: gold
Estimated APC paid: $1,880
Funder: University of North Florida
OpenAlex ID: W7154691996