Work

Delaying access to a problem-skipping option increases effortful practice: Application of an A/B test in large-scale online learning

Year: 2017

Type: article

Study designs: Randomized Controlled Trial, Clinical Trial

Abstract: We report on an online double-blind randomized controlled field experiment (A/B test) in Math Garden, a computer adaptive practice system with over 150,000 active primary school children.The experiment was designed to eliminate an unforeseen opportunity to practice with minimal effort.Some children tend to skip problems that require deliberate effort, and only attempt problems that they can spontaneously answer.The intervention delayed the option to skip a problem, thereby promoting effortful practice.The results reveal an increase in the exerted effort, without being at the expense of engagement.Whether the additional effort positively affected the children's learning gains could not be concluded.Finally, in addition to these substantial results, the experiment demonstrates some of the advantages of A/B tests, such as the unique opportunity to apply truly blind randomized field experiments in educational science. 2004).In this paper, we show that large-scale experiments in online learning environments, also referred to as A/B tests, can be used to solve some of these issues.We report on a successful application of an A/B test in a large-scale online computer-adaptive practice system for Dutch primary schools (Math Garden, with over 150,000 active users).In the A/B test we delayed the option to skip a problem.This option was used by some children to skip difficult problems and practice with minimal effort, and the delay was thus aimed at promoting more effortful practice.Before describing the experimental details, we first aim to build a basic understanding of A/B tests in relation to traditional educational experiments, introduce the online practice system that is central to the experiment, and then discuss our motivation for this particular intervention.

Source: Computers & Education

DOI: 10.1016/j.compedu.2017.12.008

Authors: Alexander O. Savi, Nienke M. Ruijs, Gunter K.J. Maris, Han L.J. van der Maas

Institutions: University of Amsterdam, Cito (Netherlands)

Language: English

Topic: Intelligent Tutoring Systems and Adaptive Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

SDG: Quality education

Open Access status: bronze

Funder: Nederlandse Organisatie voor Wetenschappelijk Onderzoek