Measuring cognitive, socio-affective, and behavioral engagement and evaluating their influence on the intervention outcomes in a provider-guided digital health intervention
Year: 2023
Type: article
Abstract: Background: Understanding intervention efficacy requires measuring multi-dimensional engagement, but this concept is rarely studied comprehensively. Purpose: This study aimed to enhance our understanding of engagement in socio-affective, cognitive, and behavioral dimensions, explore its impact on intervention outcome patient symptom controllability and assess the feasibility of predicting their socio-affective and cognitive engagement. Methods: Data from 68 patients randomized to Nurse WRITE, an effective nurse-guided digital health intervention (DHI) were analyzed. Content analysis of message boards summarized the total count of socio-affective and cognitive engagement classes. Additionally, socio-affective engagement was measured by total word count, while cognitive engagement was assessed through average response rate to nurses’ queries. The behavioral engagement was evaluated by counting symptom care plans and plan reviews. K-means clustering categorized patients based on six engagement measures. Regression analysis explored the relationship between patient factors, engagement, and symptom controllability. Machine learning models predicted engagement classes, and the LIME technique analyzed model challenges. Results: Details of socio-affective and cognitive engagement classes are in Appendix A Manuscript #1. Participants were clustered into high (n = 13), moderate (n = 17), and low engagers (n = 38). Details are in Appendix B Manuscript #2. Education was a significant predictor of higher engagement overall and in each dimension, which, in turn, was significantly associated with improved symptom controllability. More specifically, improved control was significantly correlated with engagement classes Intervention information, Acknowledgment, Cancer-related experience, Vocatives, and Appreciation. Our fine-tuned transformer models reliably predicted engagement, achieving mean F1 scores of 78.8 and 78.5 for socio-affective and cognitive classes, respectively. Both models had three main types of errors: context error, inaccurate weighting, and practical use of words. The socio-affective model tended to make mistakes related to practical word usage, while the cognitive model tended to make context errors. Conclusion: Engagement measures can provide insights for behavioral scientists to evaluate patient engagement in provider-guided DHIs. Higher education predicted increased communication, emotional connection, content/process coordination, and intervention activity completion, which in turn, enhanced symptom control. Future DHIs should consider patient factors and strategies to support multi-dimensional engagement. Reliable prediction of engagement can help interventionists adjust communication styles and tailor strategies to enhance engagement.
Author: Wang, Yan
Institutions: University of Pittsburgh, Capital Medical University
Language: English
Subfield: General Health Professions
Field: Health Professions
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7039461396