Work

The correlation between performance on the American Board of Emergency Medicine (ABEM) qualifying and oral certifying examinations

Year: 2023

Type: article

Abstract: Initial certification by the American Board of Emergency Medicine (ABEM) includes passing both a written, multiple-choice question qualifying examination (QE) and an oral certifying examination (OCE). The ABEM QE and OCE are intended to test different, yet complimentary knowledge, skills, and abilities (i.e., competencies) outlined in the Model of the Clinical Practice of Emergency Medicine.1 Prior research provides validity evidence for the OCE.2-4 Physicians taking the OCE report that cases used for the examination are like the patients they provide care for in clinical practice.4, 5 Prior studies have shown substantial reliability for the OCE, including high inter-rater agreement on examination scoring.5, 6 However, the correlation between scores on the QE and OCE has not been reported, which is important to understand the utility of the two examinations. A very low correlation would challenge the construct validity of a two-examination certification process. It would suggest that the cognitive skills and medical knowledge assessed on the QE have limited applicability to performance in the OCE. A very high correlation coefficient would suggest that either the QE or the OCE is redundant and unnecessary—a single examination would create a similar assessment outcome as a two-examination process. A moderate correlation would suggest that the OCE measures some competencies that are related to the QE and some competencies that are unique to the OCE. The objective of this study was to measure the association between physician performance on the ABEM QE and the OCE. We conducted an institutional review board–approved, cross-sectional study of physicians taking the ABEM QE and OCE between the years 1980 and 2022 to determine the relationship between the examination scores. All physicians who had a first attempt on both the QE and the OCE were included. We used first-time test attempts to avoid biases related to prior test performance and repeated test taking. The QE is a 305-item multiple-choice assessment of emergency medicine knowledge and cognitive skills. The QE scores range from 0 to 100, are an equated scaled score, and have a high degree of reliability (Cronbach's alpha of 0.84–0.90). The standard error of measurement for the QE has for the past 5 years (2018–2022) has ranged from 1.76 to 1.86. The OCE has scores that range from 1.00 to 8.00, are a scaled score, and have a high degree of reliability (Rasch and Cronbach's reliability coefficients from 0.90 to 0.99). The standard error of measurement for the OCE ranges from 0.02 to 0.06. Pairwise complete observations were used for the study, meaning that both the QE and the OCE had to be attempted. Data were collected during test administrations by the Board from 1980 to 2022. ABEM's scores were verified through internal replication prior to reporting. All data were deidentified and are reported in aggregate. The Pearson product-moment correlation (r) was chosen to describe the bivariate correlation between two normally distributed continuous variables, the scores on both the QE and the OCE. Prior analysis has shown that these examination scores were normally distributed. A priori categories of relationship strength for a given correlation coefficient would be 0.5 to 0.8 strong positive, 0.2 to 0.5 moderate positive, and 0.0 to 0.2 weak positive.7 In addition to the correlation, it was assumed that there was a linear relationship between QE and OCE scores. For this reason, a simple regression model was fit. A total of 55,108 physicians were included. This cohort included physicians who completed Accreditation Council for Graduate Medical Education (ACGME)-accredited emergency medicine residency and physicians who, prior to 1988, met practice-based criteria. All test takers also had an unencumbered medical license. The main results were the descriptive performance statistics for the QE and the VOE. The mean (±SD) score for the QE was 82.35 (±6.30; median 83, min 23, max 100, range 77). The mean (±SD) score for the OCE was 5.84 (±0.39; median 5.84, min 2.82, max 7.8, range 4.98). The QE and OCE scores were normally distributed, scaled scores. The Pearson's product-moment correlation calculated between the QE and OCE scores was r = 0.33 (95% CI 0.32–0.34). The alternative hypothesis that the true correlation is not equal to 0 is supported by the data: t = 81.545, df = 55,106, and p < 0.001. There is a statistically significant association between performance on the QE and performance on the OCE (Figure 1). This suggests that better performance on the QE is associated with a higher score on the OCE. The predictive capacity of the QE was similarly demonstrated by the relationship between the QE and OCE. A linear regression model between these two variables indicated that for every point of QE score, a predicted OCE score goes up by 0.02 points (p < 0.001, 95% CI 0.02–0.02). The R2 of 0.11 or proportion of variance explained is expected (squaring the Pearson's r of 0.33 yielded 0.11). There are several limitations to this study. First, the relationship between the ABEM QE and OCE is likely more complex than measuring a single global performance event of physicians taking an examination. The results of our analysis do not allow for a causal relationship to be established between QE and OCE performance, although a moderate association is suggested. Additionally, this analysis is an aggregate analysis of the relationship between the QE and OCE. It does not describe whether or how the performance relationship has changed over time. However, it is unlikely that the correlation between the QE and OCE would have varied significantly given the consistency with which ABEM has approached test development and administration. Finally, the ABEM QE and OCE use different psychometric scales and have different passing standards that have evolved over time. These differences could affect the magnitude of the correlation coefficient, but how these variances would alter the correlation between QE and OCE is uncertain. This study is the first, to our knowledge, to explore how performance on the QE is associated with performance on OCE. The association is sufficient enough to demonstrate that the two examinations are related (i.e., convergent validity) but the association is not so high that it obviates the suggestion that the OCE measures substantially different content or competencies. As seen in the regression analysis, the association is not tightly clustered, suggesting that other factors are contributing to the OCE score. The ABEM QE and OCE are moderately correlated (above 0.2 and below 0.5), suggesting convergent validity evidence. Convergent evidence suggests that assessments that are built on similar constructs should have related results.8 The QE and OCE have related results, but the association is not so strong that the QE could be used alone to measure the same medical knowledge, cognitive skills, and other abilities necessary to practice emergency medicine that are measured by the OCE. The moderate correlation concurrently provides divergent evidence, meaning that the OCE is not a redundant reformulation of the QE construct (i.e., medical knowledge and cognitive skills).8 ABEM constructs the QE and OCE using competencies outlined in the Model of the Clinical Practice of Emergency Medicine.1 Many competencies that are necessary to be a practicing emergency physician cannot be assessed on a multiple-choice question examination (i.e., the QE) and are therefore assessed on the OCE. Because the OCE assesses different competencies than the QE, the combination of the two provide a complementary, comprehensive summative evaluation for initial certification. In summary, these results suggest that the QE and OCE examinations used for initial certification by the ABEM are moderately correlated. In other words, the QE and OCE are related but different examinations. These results add validity support to the use of the OCE for initial certification decisions because the test measures additional competencies that are necessary for the competent practice of emergency medicine. These additional competencies assessed on the OCE are supplementary to the QE, not duplicative. Future research is needed to continue to provide additional validity evidence for the OCE such as predictive validity and consequential validity evidence. Diane L. Gorgas: study design, manuscript development, and review. Yvette Calderon: manuscript review. Wallace A. Carter: manuscript review. Suzanne R. White: manuscript review. Kim M. Feldhaus: manuscript review. Earl J. Reisdorff: manuscript review. Kevin B. Joldersma: study design, data analysis, and manuscript review. Chadd K. Kraus: manuscript development and manuscript review. The authors thank Ms. Lyndsay Tyler for her valued work coordinating the development of the manuscript. Drs. Gorgas, Calderon, Carter, White, and Feldhaus serve on the American Board of Emergency Medicine Board of Directors. Drs. Reisdorff, Joldersma, and Kraus are employed by the American Board of Emergency Medicine. The American Board of Emergency Medicine receives revenue from administering both examinations used in this study.

Source: Academic Emergency Medicine

DOI: 10.1111/acem.14780