Get started
Work

Editors’ notes

Year: 2022
Type: article
Abstract: Our goal in publishing this volume is to describe and share some of the knowledge we have gained as a Coordinating and Evaluation Center for a 10-year educational program intervention initiative implemented at various institutions of higher education. Designed to understand which programs and activities best support the important long-term outcome of bringing greater diversity to who is leading and what is being taken up in biomedical research, our evaluation was logically designed, predictable, and unfussy – on paper. In practice, moving from “paper to people,” the study became challenging and complex. We put much thought and time into understanding how to best implement our evaluation study. We hope that the insights offered in this issue are transferable to others in terms of both the process and substance of our efforts. Our work is part of the National Institute of Health (NIH) Diversity Program Consortium (DPC), one of the most significant, comprehensive national efforts ever launched to diversify the biomedical research workforce, and address the paucity of diverse researchers with NIH R01 funded studies. The initiatives and programs established through the Consortium focus on changing who participates in and remains committed to conducting biomedical research. They are also intended to advance access and equity of diverse peoples and ideas in academia and the broader research enterprise by transforming institutional norms and practices. The DPC is comprised of three efforts that include the Coordination and Evaluation Center (CEC), BUilding Infrastructure Leading to Diversity (BUILD) Program, and the National Research Mentoring Network (NRMN). The DPC was first funded in 2014 for 5 years, with a renewal in 2019 for an additional 5 years. A primary role and function of the CEC at UCLA has been to design, collect, and report evidence regarding the interventions across the 10 BUILD sites. Expected outcomes are articulated in a consortium-wide agreed upon set of “Hallmarks of Success” (https://diversityprogramconsortium.org/pages/hallmarks_yr6-10). This contrasts with other NIH training grants, for example, where Principal Investigators implement programs according to general Request for Applications (RFA) guidelines and reporting requirements, often working independently within an institution or with limited collaboration outside of the grantee institution and with no external evaluation. Yet, typical of NIH training grants, each BUILD site has implemented consortium-wide efforts differently. The CEC is responsible for evaluating the outcomes of the Diversity Program Consortium efforts. The original details of the evaluation design are explained in Davidson et al. and McCreath et al. in BMC Proceedings, Volume 11 (2017). The articles in this New Directions for Evaluation issue reflect new and updated thinking about our evaluation design and implementation details. Notable for NDE readers is the tension for the CEC between meeting the stipulated expectations of the original RFA, which includes a quasi-experimental design across multiple program sites, and providing a more nuanced understanding of each site's program processes and outcomes using culturally responsive and equity-focused theories of evaluation. From the outset, we intended to ground our work in evaluation theory, specifically, utilization-focused evaluation and theory-driven evaluation, with a stance or perspective, informed by equitable evaluation practices – that is, our work was intended to be both scientific and in service of equity. We struggled to achieve a user-oriented approach with the program sites because of the concurrent conduct of local level evaluation studies (stipulated as part of the BUILD RFA) alongside the national CEC evaluation study and other relational and procedural issues. Over time, we enhanced our study (e.g., adding a case study component) and adjusted its conduct (e.g., evolving our engagement with local level evaluators; becoming more explicit about our equity stance). Even with 8 years behind us, we continue to learn how to meet the evaluation information needs of the NIH and the sites and how best to respond and adapt to shifts in the education and biomedical fields more broadly. For example, throughout what will be a 10-year evaluation, we have discussed at length issues related to the terminology used to describe the participants in the BUILD programs. When the study was initially solicited, we used URG (underrepresented groups) and commonly accepted sex and racial/ethnic categories for data collection. Recently, BIPOC (Black, Indigenous, Peoples of Color) has emerged as a preferred term among some groups, and there has been an interest in understanding and including sexual gender minorities (SGM) in research. We know that definitions and recommended terms for diverse groups can vary by discipline, sector, generation, and groups themselves who are at varying levels of identity development. We strived to be very sensitive to the issues of terminology and identities, not just out of respect to our diverse participants, stakeholders and our own members of the evaluation and DPC community, but for the field. Unfortunately, we were unable to be as responsive as we should have to these shifts. But, as any thoughtful group should, we continue to discuss how our practice may or may not be reinforcing structural and other biases as our work progresses and becomes more complex in terms of our questions. We are still collecting primary and accessing secondary data, conducting analyses, and reporting results while this issue was in production. That is to say, this evaluation is still a work in progress. We remain focused on providing the best information possible about the work of the DPC so that others may learn from our evaluation findings – the program's processes and outcomes. The papers in this volume are not intended to report on the outcomes of the DPC. Instead, the subject of focus is the evaluation itself, as we consider what we have done well, what we did not do so well and should have anticipated, and some surprises that we could not have imagined, such as a once in a generation pandemic. Described are our efforts to conduct the best evaluation possible given the requirements of the RFA, and that aligned with the principles and objectives of the programs that we are studying. The authors represent all aspects of the CEC and local level BUILD evaluators, NIH affiliated program officers, and analysis from a well-established external professional and academic evaluation scholar. We hope that you locate a helpful lesson, a cautionary note, and perhaps even a few nuggets of insight within these pages. The volume opens with a chapter that familiarizes readers with the context in which the consortium-wide evaluation study has taken place. It provides an understanding of the complexity of our evaluation effort as the first large-scale, national, systematic, longitudinal evaluation of harmonized interventions focused on undergraduate biomedical research training programs in the history of the NIH and the National Institute of General Medical Sciences (Guerrero, Seeman, McCreath, Norris, and Maccalla). Chapter 2 (Cobian, White-Lewis, Hurtado, and Ramos) describes the case study aspect of our work. Ramirez, Joseph, and Oh (Chapter 3) explain the engagement strategies used in the study through the lens of Estrada et al.’s (2014) tailored panel management. In Chapter 4, Crepsi and Cobian use the BUILD Scholar program—one component of the broader initiative—to demonstrate the application of meta-analysis to combine effect estimates from different sites of a multisite initiative. Maccalla, Purnell, McCreath, Dennis and Seeman describe our efforts to capture a nuanced understanding of dosage of the BUILD program across the ten sites in Chapter 5. In Chapter 6, Christie and Wright add to our understandings about the theories guiding the BUILD site-level evaluations to make explicit the purpose and goals of their complementary evaluation work. In Chapter 7, well-experienced local BUILD evaluators, Hwalek, Honoré, and Brown, offer counsel on best establishing, facilitating, and conducting collaborative and coordinated studies to meet the demands of both the local and national-level evaluation efforts. In Chapter 8, Azzam uses Cronbach's generalizability framework to discuss some of the critical insights gained from the chapters included in this volume about multi-site evaluation studies that advance our knowledge of best to conduct these kinds of complex evaluations. In the last chapter, our NIH colleagues Gibbs, Reynolds, Epou, and Gammie describe learnings from the CEC evaluation to strengthen current and future NIH research training and capacity-building activities and evaluation work. Finally, we dedicate this volume to our close colleague and dear friend, Steven P. Wallace, PhD. Steve was a Multiple Principal Investigator on the CEC and co-lead of the evaluation core. Steve passed away suddenly on March 30, 2021. He helped to conceptualize and implement the DPC evaluation and this issue. His vision, brilliance, pragmatic sensibility, and compassion is evident throughout. Lourdes Guerrero and Christina Christie, April 2022 Dr. Lourdes Guerrero was associate adjunct professor at the David Geffen School of Medicine at UCLA and served as an associate lead for the Evaluation Core of the Coordination and Evaluation Center for the Diversity Program Consortium. She is currently a researcher a UC San Diego. Dr. Christina Christie is professor and Wasserman Dean of the School of Education and Information Studies at UCLA and serves as multiple principal investigator for the Coordination and Evaluation Center and director of the Evaluation Core for the Diversity Program Consortium.
DOI: 10.1002/ev.20501
Language: English
Subfield: Safety Research
Open Access status: hybrid
Estimated APC paid: $3,050
OpenAlex ID: W4298063510
FWCI:
Cites:
Cited by:
Related to: 10