Addressing Practice Effects in Population-Based Studies of Trends in Late-Life Dementia and Cognitive Impairment
Year: 2024
Type: article
Abstract: Practice effects, defined by the American Psychological Association as “any change or improvement that results from practice or repetition of task items or activities,” have been a growing focus in longitudinal studies of the cognitive functioning of older adults. A PubMed search yields more than 1 500 results since the term first appeared in 1953. More than 200 articles use the term combined with “Dementia” or “Cognitive Impairment,” nearly two-thirds of which have appeared in the last 10 years. A key focus in recent years has been on accounting for biases related to practice effects in cohort studies to isolate the impact of aging or disease progression on cognitive function (1). Several articles have compared approaches for adjusting for practice effects in statistical models, again in the context of intraindividual, age-related change (2–4). An additional topic, relevant for this essay, highlights that practice effects are not simply a statistical nuisance to be avoided but may be a key marker of cognitive function (5–9). The existing literature is thin with respect to addressing practice effects when studying national trends in late-life dementia and cognitive function (10–12), and there is disagreement on how to address practice effects in this context (13,14). This essay offers insights into challenges in addressing practice effects when studying dementia and cognition trends with national panel surveys that periodically replenish their samples. We first summarize 2 articles appearing in this supplemental issue that address practice effects in the context of national dementia or cognition trends. We then provide a review of existing recommendations for addressing practice effects in studies modeling cognitive decline with age and outline challenges with these methods when studying trends with national panel surveys that periodically replenish their samples. We then describe the logic of approaching practice effects as a misclassification issue. Finally, we provide suggestions moving forward for better addressing cognition and dementia trends in a national survey context. In this supplemental issue, 2 new articles address time trends in cognition outcomes, using different data sources and measures of cognition—and different approaches to addressing practice effects. Myrskylä and colleagues (15) focus on trends in immediate and delayed word recall, a measure of memory, in the United States, England, and several European countries. In their main analysis, they estimate unweighted mixed effect models with a random intercept and random (age) slope. To address practice effects, they include a control for whether the individual had any prior exposure to the study’s cognitive tests or if the current round was the first test. They find improvement over time in both memory function and impairment in the United States, but at a slower pace than the improvements observed in England and other European countries, particularly for those younger than age 75. In another article in this issue, Freedman and Cornman (16) draw upon the National Health and Aging Trends Study (NHATS) to examine trends in probable dementia among the population ages 72 and older. As in prior research with NHATS (8), rather than attempting to model practice effects, the authors require individuals to meet probable dementia criteria in 2 consecutive rounds as a means of addressing potentially biasing effects of practice. They find the prevalence of probable dementia declined significantly from 11.9% in 2011 to 9.2% in 2019 and 8.2% in 2021. Notably, findings in both papers are sensitive to how practice effects are modeled. In their appendix, Myrskylä and colleagues (15) present unweighted mixed effect models with age, gender, and time trend with practice effects operationalized as any prior test or as the total number of prior tests. The trend parameter for memory score is 0.06 per year [95% CI = 0.04, 0.09] when practice effects are operationalized as any prior test, and −0.01 [−0.02, −0.00] when operationalized as the number of prior tests. During the review process for Freedman and Cornman (16), the authors also prepared sensitivity analyses demonstrating how different approaches to practice effects influenced findings, and while conclusions were robust, trend estimates varied depending on the approach (eg, single-round criteria with no practice effect, controlling for number of prior tests; controlling for first test; and 2-round criteria). Most of the advice to date about how to control for practice effects when studying cognition has been in the context of cohort studies where the focus is on cognitive decline with aging. For instance, Vivot and colleagues (3) compare methods for addressing practice effects using 3 cohort studies (Three-City Study–Dijon, Normative Aging Study, and Washington Heights-Inwood Community Aging Project) and Chen and colleagues (2) compare methods using several cohorts from Kaiser Permanente Northern California. Vivot and colleagues (3) estimate mixed models that control for age at study entry and include random intercepts and slopes for time since enrollment. They compare coefficients for time since enrollment across models using 4 different specifications for practice effects: no practice effect, first test, number of prior tests, and square root of number of prior tests. They find that in most instances including a practice effect does not change model fit, but in the few instances where it does, the models using the first test indicator typically fit best. They note that first visit is less collinear than number of visits with time since enrollment in studies with fixed test–retest intervals, but refrain from recommending a universal specification for practice effects. Chen and colleagues (2) demonstrate a 2-step method of incorporating a practice effect into a cohort study. They recommend first estimating an ancillary preliminary mixed-effects model with a subset of cases with complete data for a limited number of waves. The preliminary model produces an age-adjusted coefficient for wave as the practice effect. This practice effect from the first model is then used to correct time since enrollment coefficients in a second mixed-effects model that controls for age at entry and includes random intercepts. The practice effect from the first model is incorporated into the second model as an offset to (or an amount subtracted from) scores at waves 2 and beyond, thus creating practice-effect adjusted scores at follow-up waves. They conclude that the 2-step method is preferred over ignoring practice effects or including an indicator of first test because it better minimizes gaps between within-person (ie, same individuals at different ages) and between-person (ie, older vs younger individuals) cognition. Two key distinctions between the cohort analyses conducted to understand cognitive change with age and the type of trend analyses in this supplemental issue are relevant to a discussion of practice effects. First, the time scale of interest differs. Cohort studies are mainly interested in how cognition changes with age or with time since enrollment in the study controlling for age at study entry. In contrast, trend analyses that draw upon national panel studies focus on how the older population as a whole changes from year to year, typically controlling for shifts in the age distribution of the population. To study trends, all observations from a uniformly specified age group (eg, 65 and older) are included in each year and survey weights and adjustments for complex design are used so that inferences to the older population in each year can be made. Second, in cohort studies, individuals are often enrolled during an initial period of time and followed for the life of the study, but generally the cohort is not replaced as the cohort ages. In contrast, in national panel studies designed to track trends, the sample is typically replenished (new sample added) at regular intervals. The Health and Retirement Study (HRS), for example, adds birth cohorts who have aged into the 51–56-year age group during the most recent 6-year period and their partners (of any age). NHATS periodically replenishes all age groups according to the fraction who have aged in or out of the age group, died, or left the study since the last replenishment. Practice effects may bias cognition estimates in a given year—and hence the trend estimates over a period of years—if scores systematically differ in the first year that cognitive tests are administered or if they improve (or do not worsen) as a result of readministration. Following individuals over time results in the sample surviving to later years having more experience with the cognitive tests on average in later years than in earlier years. However, sample replenishment introduces a mixture of different times since enrollment (and number of tests) into the sample within a given year. In NHATS, for instance, all participants who were able to complete cognitive testing had a first administration in Round 1 (2011), but only about half the sample had a first administration in Round 5 (2015) when the study was first replenished. Since the distribution of the amount of practice varies from year to year, trend estimates may be biased if this variation is not controlled. Controlling for first (or number of prior) tests may be problematic in a national panel study with replenishment, depending on the replenishment design. In HRS, for example, the first test in a replenishment year reflects being age eligible for the replenishment sample (or being partnered with someone who is age eligible) and not having a proxy respond. In non-replenishment years, it reflects previous replenishment sample requirements being met and having a proxy respond in prior round(s) but not the current round. Thus, the first test can take on different meanings in different years, and its inclusion could potentially bias trend estimates. Moreover, in both HRS and NHATS, deviations from a perfect correlation between number of prior tests and years since added are typically the result of either missing a prior round or having a proxy respond to the survey in a prior or initial round. These sources of deviation may occur due to the individual having a cognitive or other health-related issue. Hence, as previously noted (14), indicators of any prior (vs first) test and number of prior tests not only reflect a practice mechanism driving improved performance but they are also in part determined by prior cognitive ability. Although Chen et al.’s 2-step approach was developed to address bias in time since enrollment, the approach could be adapted for application in the national trend context. For studying age-adjusted trends instead of age-at-entry-adjusted time since enrollment, a third step would be necessary: first estimate a preliminary model over a short time frame (to minimize year-related confounding) with age-adjusted wave as the practice effect; second, generate a set of modified scores at follow-up waves in the full sample using the practice effects as a wave-specific offset in a model estimated with all observations and waves, controlling for age at entry but ignoring year; third, estimate a model using the actual first visit and practice-effect-adjusted follow-up scores that includes year and controls for current age. Implicit in this approach are the assumptions that practice effects do not vary by level of cognition and are constant over the time frame of interest. The time frame for the preliminary model must also be short enough to minimize the influence of trends on the wave effect. An alternative approach, embedded in the analysis by Freedman and Cornman (12), builds upon the finding that practice effects are strongest in unimpaired individuals and much more limited for those with mild cognitive impairment or dementia (5–9). The approach recognizes that practice effects can result in misclassification errors that occur as the result of 2 distinct processes (Table 1). First, a lack of familiarity with test format at the initial test administration may result in a downward bias in initial test scores that are remedied at future visits (ie, a false positive at the initial test). This issue is relevant for those whose true status is unimpaired but who initially test as having mild cognitive impairment or dementia (group A) and for those with mild cognitive impairment who initially test as having dementia (group B). This type of misclassification would be missed by investigations into practice effects by level of impairment that rely on first test information for stratification purposes; that is, this type of misclassification results in the appearance that those with dementia at the first visit are learning. Second, repeated exposure to a test format due to a study’s panel design may inoculate an individual against scoring poorly despite some, typically mild, cognitive decline (ie, a false negative at a later test). In this framing, this second type of misclassification is limited to those with mild impairment (who might be misclassified upon repeat testing as unimpaired, group C). Those whose true status is unimpaired would not be misclassified and those with dementia would be unable to improve and hence also not be misclassified. Key Misclassification Errors Related to Initial and Repeated Cognitive Testing If the outcome of interest is dementia (rather than a cognitive score or milder forms of impairment) and the practice effect itself is not of substantive interest, reclassification may be used to address misclassification of Groups A and B. That is, individuals who have dementia in a given round but appear to recover at the next round can be reclassified at the initial round as not having dementia. This approach should mitigate misclassification bias without introducing issues of collinearity or reverse causality. The approach can be applied to all pairs of rounds to ensure that misclassification is treated uniformly across time (even if the levels fluctuate over time) and has the added benefit of recognizing that practice effects are negligible among those with dementia. However, the approach assumes no recovery from dementia, which may not be appropriate for the subset of dementia cases caused by treatable conditions (eg, depression, medication side effects, thyroid conditions, or vitamin deficiencies). If asked to design a national study to track trends in cognitive impairment and dementia prevalence that is resistant to practice effects, the solution is straightforward: select a fresh nationally representative cross-sectional sample each year, and test study participants once and only once. Such a design is expensive and does not offer the benefits of being able to study within-person changes in cognition and other dimensions of health with age or to track dementia incidence. We therefore offer researchers 2 approaches for addressing practice effects when using national panel data with replenishment to study cognition and dementia trends. Rather than controlling for practice effects with approaches that may introduce collinearity or reverse causality, researchers may want to consider modifying Chen et al.’s preliminary modeling approach or treating practice effects as a misclassification problem and correcting dementia classification accordingly. When combined with best practice techniques for analyzing national survey data—weighting and proper adjustment for standard errors to account for the complex design of the study, properly accounting for intraindividual correlations, including all segments of the older population, including those requiring a proxy respondent to complete their interview and those living in institutional settings—the use of strategies to minimize bias from practice effects can help researchers ensure they have unbiased estimates. Because these methods are still emerging in the trends context, sensitivity analyses that demonstrate how trends might change if practice effects were addressed in different ways are encouraged. Preparation of this commentary was supported by the National Institute on Aging of the National Institutes of Health (grant numbers U01AG032947 and P30AG012846). The views expressed are those of the authors alone and do not represent those of their employers or the funding agency. This commentary appears as part of a supplemental issue on Multidisciplinary Perspectives on Dementia and Related Population Health Trends, which was supported by the Michigan Center on the Demography of Aging at the University of Michigan, with funding from the National Institute on Aging of the National Institutes of Health (P30AG012846). None. We thank Lindsay Kobayashi, Joshua Ehrlich, Neil Mehta, and John Marcotte for helpful comments on earlier versions of this commentary.
Authors: Vicki A Freedman, Mengyao Hu
Language: English
Subfield: Psychiatry and Mental health
Field: Medicine
Domain: Health Sciences
Open Access status: bronze
OpenAlex ID: W4401500622