Retrospective evaluation of healthcare programs using electronic health record data: A scoping review of observational study methodology
Year: 2026
Type: preprint
Study design: Systematic Review
Abstract: Objective: Program evaluations are commonly used in healthcare to estimate eYects on patient outcomes, costs, and care delivery. When programs are implemented nonrandomly and evaluated retrospectively using electronic health record (EHR) data, important methodological challenges arise. We conducted a scoping review to characterize study designs and analytic approaches used in retrospective EHR-based healthcare program evaluations. Materials and Methods: We conducted a scoping review of peer-reviewed studies published between 2021 and 2025. Of 833 identified citations, 58 studies met eligibility criteria and underwent full-text review. Study designs, analytic methods, and approaches to addressing non-randomized implementation and EHR data limitations were extracted and summarized. Results: Descriptive designs were used in 26 evaluations (45%), while 32 (55%) employed analytic designs. Among analytic studies, cohort designs predominated (91%, n=29). Most evaluations (69%, n=40) relied on descriptive statistics alone or combined with p-valuebased hypothesis testing. Multivariable regression was used in 18 studies (31%), whereas methods to address non-random program deployment were infrequently applied. Although 57% of studies cited EHR data bias as a limitation, only four studies (7%) explicitly addressed it analytically. Discussion: Despite evaluation questions that focused on program impact, many studies relied on descriptive methods with limited ability to support causal inference. Approaches to address non-random program implementation were infrequently used, and EHR data limitations were often acknowledged but rarely addressed analytically. Conclusion: These findings suggest opportunities to better align retrospective EHR-based program evaluations with their intended inferential goals through more rigorous approaches to causal inference and greater attention to bias arising from incomplete or imperfect EHR data.
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7212011105
Cites:
Cited by:
Related to: