Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines
Year: 2022
Type: article
Abstract: Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (predictability, computability, stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning to the design of RL algorithms for the digital interventions setting. Furthermore, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We show how we used the PCS framework to design an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.
Source: Algorithms
DOI: 10.3390/a15080255
Authors: Anna L. Trella, Kelly W. Zhang, Inbal Nahum-Shani, Vivek Shetty, Finale Doshi-Velez, Susan A. Murphy
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,466
Funders: National Science Foundation, National Institutes of Health, National Institute of Mental Health, National Institute on Drug Abuse, National Cancer Institute, National Institute of Biomedical Imaging and Bioengineering, National Institute of Dental and Craniofacial Research, Division of Chemical, Bioengineering, Environmental, and Transport Systems
Awards: G2002134494, G2228155111, Center for Methodologies for Adapting and Personalizing Prevention, Treatment and Recovery Services for SUD and HIV (MAPS Center), G25851218, G2979645247, Data-driven subtyping in major depressive disorder, G4159675886, G4208625158, Graduate Research Fellowship Program (GRFP), G5556446122, G5867770422, G6068434618, AI Institute in Dynamic Systems, mHealth Center for Discovery, Optimization, and Translation of Temporally-Precise Interventions (mDOT), Novel use of mHealth data to identify states of vulnerability and receptivity to JITAIs, G7451301460, G7678611020, G773633582, Personalized Digital Behavior Change Interventions to Promote Oral Health, G9000057195, G9000121471, G958666079, G9641189
OpenAlex ID: W4286561193