Work

2016 R evision of the SCAI position statement on public reporting

Year: 2016

Type: article

Abstract: The public reporting of institutional and individual operator results of percutaneous coronary interventions (PCIs) is intended to provide meaningful information to the public and enhance the delivery of superlative health care. By giving consumers specific outcome data 1, patients will be empowered to participate more fully in decisions concerning their medical care. The influence that public reports wield could increase if publicly reported information proves to be an accurate representation of "value" in health care delivery, and if third-party payers use this information to allocate reimbursement in a value-based system 2. Despite these well-intended goals, there is uncertainty whether existing programs correctly identify high- and low-performing PCI centers and operators. Moreover, there is emerging evidence that public reporting can deleteriously influence case selection by encouraging risk avoidance behaviors. Thus, potentially beneficial procedures might be withheld from high-risk patients who can derive the greatest benefit, because operators and facilities fear being labeled as outliers 3-8. This position statement updates the prior Society for Cardiac Angiography and Interventions (SCAI) Policy on Public Reporting 1. SCAI continues to endorse public reporting, provided the reports are not misleading, deliver meaningful information to consumers to help inform their choices, and facilitate quality improvement. Offering the public accurate and understandable metrics, including measures to assess the appropriateness of case selection, are essential to achieve this aim. Mandatory public reporting highlights the dilemma of balancing the provision of necessary revascularization procedures to high-risk patients with the consequences of negative outcomes. Table 1 summarizes the data presented to the public in 4 states with active programs 5-9. Other states (e.g., Pennsylvania [PA], California [CA]) have rudimentary programs. A summary of the experience of the two states with the most established programs follows. New York State's (NY) annual reports contain every hospital's and operator's PCI volume, unadjusted (observed) in-hospital/30-day mortality rate (OMR), expected mortality rate (EMR), risk-adjusted mortality rate (RAMR), and outlier status (significantly higher, lower, or not different from the statewide mortality rate), calculated on a rolling 3-year basis. The OMR is the observed number of deaths divided by the total number of cases, whereas the EMR is the sum of the predicted probabilities of death (as a function of patients' own risk factors) for all patients divided by the total number of patients. The RAMR is then obtained as the OMR/EMR ratio of the provider multiplied by the registry's overall mortality rate. Non-federal hospitals in Massachusetts (MA) performing PCIs annually report Standardized Mortality Incidence Rates (SMIR), using a Bayesian hierarchical modeling approach (see below) analogous to using OMR/EMR. However, in lieu of OMR, the predicted mortality rate is used instead. By employing hierarchical modeling, fewer hospitals are identified as outliers than in NY. SMIR is also reported separately for patients presenting with ST-elevation myocardial infarction (STEMI) or cardiogenic shock. In the future, the state will shift to reporting risk-adjusted 30-day mortality. For public reporting to be effective, the information reported should (1) address the consumer's important questions and priorities; (2) present credible information that is interpretable by the consumer; (3) stimulate the consumer to act on the information; and (4) avoid generating false perceptions regarding individual and/or institutional quality of care. Although public reporting is intended to assist patients in making better choices and support the identification of outliers with poor performance 10, there are few data to confirm this actually occurs 11. The reasons include an apparent lack of consumer interest, difficulty interpreting the presented information, perceived inaccuracies, and a lack of opportunity for patients to act on public reports (e.g., in emergent clinical situations or when care is directed by payers). RAMR after 30 days has been the principal metric presented in public reports. Although patients are understandably concerned about procedural mortality, SCAI believes that the RAMR is insufficient as a summary measure of quality. The primary reason is that mortality after PCI depends more on the clinical acuteness at presentation and comorbidities than on technical proficiency or operator judgment; that is, the variation between providers is much smaller than the variation between patients. Patient severity of illness and comorbidities are highly predictive of mortality after PCI, especially acute myocardial infarction (AMI), cardiogenic shock, and other high-risk settings. The principle underlying risk adjustment is that correcting for confounding factors can mathematically equalize these factors. In theory, RAMR should provide complete correction and be used to identify "outliers" above an arbitrary (e.g., >95th) percentile in order to identify variations in interventional program quality 12. Many interventionists lack confidence that risk adjustment is precise at the high end of risk 5, 7. They have doubts about whether risk adjustment models include some factors frequently used to make clinical decisions (unmeasured confounders). Clinicians are concerned that misinterpretation by the public and regulators will lead to unwarranted damage to professional and institutional reputations. This could result in some excellent operators and hospitals with apparently higher mortality rates being perceived as having poor quality and outcomes because of a high-risk case mix or referrals to their facility 11, 13. Notably, when a complex PCI concludes without mortality, no additional "credit" for high technical competence is assigned to the operator; yet, when a death occurs related to the seriousness of the underlying condition, it is imputed to be a flaw of the operator 14. Each dataset identifies predictive variables based on its own modeling and weights them according to their population, which might not be applicable to all datasets. Their predictive accuracy when applied to a validation subset is frequently omitted 20. Although 1.3% is the median RAMR for PCI nationally, individual facility values above or below the median might not be significantly different. Notably, it has never been demonstrated that programs identified in this manner have demonstrably lower quality compared with other hospitals. Additionally, because RAMR is low for most institutions, it is a weak discriminator of programs and operators. This occurs because the low mortality achieved with PCI requires large numbers of patients to reduce variance and narrow the confidence intervals (CIs). Even when a hospital's OMR greatly exceeds its EMR, it is unclear if it can be appropriately labeled an outlier hospital that delivers poor quality care 12. Using National Cardiovascular Data Registry (NCDR) data, Sherwood et al. 21 found that hospitals performing PCIs in a greater number of high-risk cases did not have a poorer RAMR than facilities treating lower-risk patients. The risk adjustment model used was well calibrated even among very high-risk patients. In fact, hospitals treating the highest risk patients actually had better RAMR than facilities treating patients with a lower severity of illness. However, it is critical to recognize that this study assessed risk through measured variables; unmeasured factors are not assessed by published risk scores and might result in penalization by RAMR statistics. Risk adjustment is performed with logistic regression models or hierarchical multilevel models. The advantages and disadvantages of these methods have been debated 22, 23, but both types of models might be suboptimal discriminators between good and poor performers. Hierarchical modeling is currently the preferred method for both hospital and physician quality comparisons because it accounts for the "clustered" nature of observations. When comparing hospitals, individual procedures within a particular hospital are not truly independent from one another. Rather, they are related because of the characteristics associated with the particular facility. Hierarchical models are given this nomenclature because they account for the fact that the data are organized in tiered fashion: procedures are clustered within patients, patients are clustered around physicians, and physicians are clustered around institutions. Another advantage is that these models can limit widely varying estimates that can occur due to small sample size. Hierarchical models that use "Bayesian" or "empirical Bayesian" methods employ "shrinkage," in which point estimates for RAMR are moved closer toward the mean. The rationale is to reduce wide fluctuations in RAMR that might occur due to chance, and to mitigate potential type I error due to multiple hypothesis testing. Consequently, Bayesian hierarchical modeling approaches tend to identify fewer statistically significant differences than classical modeling, but it is uncertain whether this correction is an accurate reflection of risk. If an outlier is identified, however, it is more likely that the observation is real; conversely, this type of model makes it very unlikely for a low-volume hospital or provider to be identified as an outlier. Because RAMR using hierarchical models might preferentially identify larger volume facilities as outliers, a "level playing field" might not exist if such statistics are used to compare hospitals with one another. Some models report only point estimates because the 95% CIs are wide (due to low death rates). Point estimates can mislead the public by appearing to delineate differences in RAMR, when in reality; showing statistical significance might be nearly impossible when sample sizes are small. Rank ordering or categorization of provider quality is misleading when based on uncertainly associated RAMR point estimates and should be discouraged because it falsely implies a high level of precision. An important reason why very high-risk patients are not removed from most risk adjustment models is that, if allowed, there would be considerably fewer deaths remaining in the dataset to evaluate. Censoring deaths at the high end of risk could render the models nonpredictive, causing the regression equations to lose stability. It is possible that there might be no remaining statistical differences among programs or operators. For example, the mortality in AMI is nearly 10 times higher than all other patient subgroups, largely driven by poor outcomes in cardiogenic shock. If these cases are excluded, residual mortality becomes very low, and the number of lower-risk patients needed is too large to allow meaningful discrimination among institutions. It is unusual for the public to possess the statistical background necessary to accurately interpret RAMR and to use the data to choose the best provider. One reason is that they are unable to identify optimal outcomes for a particular dataset (Table 2). Once a death occurs, regardless of the clinical circumstances, the correction factor does not adjust the RAMR to zero. It is unlikely that the public understands the proper context of nonzero values, and might misperceive that >0 mortality necessarily signifies poor quality. It is crucial to recognize that patient mortality might occur despite the best clinical decision-making and technical skill. In current public reporting programs, the guidelines that inform consumers how to interpret the data are difficult to comprehend due to the complexity of statistical methodology; the fact that crude mortality rate depends strongly on case mix is not clarified. The public does not appreciate the limitations of risk adjustment to completely correct for differences in patient populations 5, 7, 24. Experience in states that have implemented reporting programs confirms that facilities and operators develop risk-averse behaviors to avoid being identified as outliers. This reaction can lead to poorer patient outcomes when patients who have the most to gain from a high-risk procedure do not receive treatment. Reluctance to perform high-risk procedures that might save lives, based on apprehension of reported outcomes, is an unintended detrimental consequence of public reporting. In comparing PCI outcomes in the states of Michigan (a state with collaborative quality improvement [CQI]) and NY (which mandates public reporting) using NCDR registry data, Boyden et al 25 found that patients undergoing PCI in NY had a lower-risk profile, with a lower proportion of patients with AMI and cardiogenic shock, compared with Michigan. In a propensity-matched analysis, patients in NY who had PCI were less likely to be referred for emergent, urgent, or salvage procedures (odds ratio [OR] 0.67; 95% CI 0.51–0.88; P < 0.0001), and had lower in-hospital mortality (OR 0.72; 95% CI 0.63–0.83, P < 0.0001). Thus, mandated public reporting in NY was associated with fewer high-risk patients undergoing PCI compared with CQI in Michigan. Moscucci et al. 26 compared the pre-procedural severity of illness of PCI patients in NY with eight Michigan hospitals. These investigators found that NY patients were less likely to undergo PCI for AMI, cardiogenic shock, and cardiac arrest. The unadjusted in-hospital mortality rate was significantly lower in NY than in Michigan (0.8% vs. 1.5%; P < 0.0001), whereas there was no significant difference in RAMR between the two regions (adjusted OR 1.05; P = 0.70). These results could be suggestive of risk avoidance of high-risk patients driven by the fear of public reporting of procedural mortality rates in NY. In contrast, Cavendar et al. 27 used the NCDR to evaluate over 1.3 million PCIs in a 2-year period, comparing outcomes from three states that had mandated public reporting (MA, NY, PA) with those that did not. They found that, relative to patients treated in non-public reporting states, those who underwent PCI in public reporting states had similar predicted in-hospital mortality (1.39% vs. 1.37%; P = 0.17) but lower observed in-hospital mortality (1.19% vs. 1.41%; adjusted OR 0.80; 95% CI 0.74–0.88; P < 0.001). Fewer patients in states with mandated public reporting underwent PCI for cardiogenic shock or salvage procedures, suggesting risk avoidance in PCI case selection. However, 47% of included PCIs were elective; because risk is low in such patients, risk avoidance behavior is less. Only 2.2% had shock in reporting states, and few were high-risk. None of these studies 25-27 were able to assess risk avoidance directly, since the study populations were limited to only those patients undergoing PCI; patients not undergoing PCI were not assessed. Additional evidence of risk avoidance comes from physician surveys. Narins et al. 28 reported that a survey of interventional cardiologists showed that 79% "agreed" or "strongly agreed" that publication of statewide PCI report cards has, in certain instances, influenced their decision to perform PCI. Furthermore, 83% of interventionists "agreed" or "strongly agreed" that patients who might benefit from PCI might not receive the required procedure as a result of public reporting of physician-specific mortality rates. Risk avoidance is also suggested through an analysis of patients with AMI that includes both those undergoing and not undergoing PCI. Joynt et al. 6 evaluated almost 100,000 Medicare patients with AMI in 10 Northeastern states and found that in three states with public reporting of outcomes, 12% fewer PCI procedures were performed, with the greatest difference in STEMI and cardiogenic shock/arrest patients, compared with seven nonreporting states. In MA, the odds of receiving PCI for AMI were comparable to the odds in non-reporting states prior to public reporting (40.6% vs. 41.8%; OR 1.00; 95% CI 0.71–1.41). However, after implementation of public reporting, odds of undergoing PCI in MA decreased compared with nonreporting states (41.1% vs. 45.6%; OR 0.81; 95% CI 0.47–1.38; P = 0.03 for difference in differences). There was a significant decline in the proportion of patients presenting with AMI who were treated with PCI in MA, which began concurrently with the onset of public reporting (Fig. 1), with the differences most pronounced in those with cardiogenic shock or cardiac arrest. However, 30-day mortality in PCI patients did not differ between public reporting and non-reporting states; the unadjusted rates were 12.8% vs. 12.1%, respectively. This study shows that public reporting does not result in improved outcomes or quality-of-care, but may lead to avoidance of high-risk cases. Trends in treatment of AMI with PCI in MA and in neighboring non-reporting states. (From Ref. 6 with permission). [Color figure can be viewed at wileyonlinelibrary.com] Additional evidence of risk-avoidance in the setting of public reporting comes from analyses of patients presenting with emergent and high-risk conditions such as AMI, shock, and cardiac arrest. Apolito et al. 4 analyzed the prospectively collected SHOCK (Should We Emergently Revascularize Occluded Coronaries for Cardiogenic Shock) Trial registry dataset. After propensity score adjustment, NY patients were less likely than non-NY patients to undergo coronary angiography (OR 0.46; 95% CI 0.31–0.68; P < 0.001) and PCI (OR 0.51; 95% CI 0.33–0.77; P = 0.002) for shock. Although post-PCI and post-CABG mortality was numerically lower for centers in NY (the only public reporting state), compared with centers in other states, RAMR was significantly higher in NY (Fig. 2). The RAMR of non-revascularized patients was >2-fold greater in NY patients versus non–NY patients. Moreover, NY patients waited longer to receive CABG: only 32.3% of NY patients vs. 75.5% of non-NY patients (P < 0.001) had CABG within 3 days of shock onset. Mortality in the SHOCK Trial registry for NY State versus the rest of U.S. (From Ref. 4 with permission). [Color figure can be viewed at wileyonlinelibrary.com] Most recently, Waldo et al. (8) analyzed data from the Nationwide Inpatient Sample in patients presenting with AMI and found that states with public reporting (NY and MA) tended to have lower adjusted rates of PCI, particularly for STEMI and cardiac arrest/shock. Although adjusted mortality was lower in patients with AMI undergoing PCI in reporting states (confirming Cavender et al), the overall adjusted mortality for patients with MI was significantly higher in reporting states (adjusted OR 1.21; 95% CI 1.06–1.37). Consistent with the concern for risk avoidance, any mortality advantage gained by PCI with public reporting appeared to be by the mortality in those patients who did not undergo PCI for AMI in public reporting states (Fig. mortality for patients presenting with AMI by Public Reporting and PCI (From Ref. with permission). [Color figure can be viewed at wileyonlinelibrary.com] approaches have been suggested to mitigate risk avoidance very high-risk patients from the reported or one for cases and one for high-risk cases to be the of the model depends on the numbers of cases in risk the of the variables the and whether the models account for case there is insufficient discrimination in the and high-risk of most especially when high-risk patients are in the sample used to the model This is when high-risk patients are from reporting. Many contain patients at the low and the high of risk but are not well in the because such patients are referred for CABG or are treated Because the discrimination in this is low, the model would be for these patients. Although of patients can be NY patients with cardiogenic shock and those with cardiac and from RAMR This was operators could use their clinical and such patients without fear of being labeled an outlier in a public In MA, two additional variables were using clinical information that was not collected in the data and that are used in the of The the use is based on presentation with at the of of the or high-risk including in with AMI or PCI of the remaining coronary A of high-risk cases, is as (1) having high-risk not by current risk adjustment or (2) procedures for which PCI was the only or best for of risk cases are from public reporting. who underwent PCI for use were found to have a increase in the odds of in-hospital death after adjustment for other risk factors and conditions Although these cases only of all the of a to their risk the predictive accuracy of the risk model from to P < et al. used the Nationwide Inpatient Sample data to compare data from NY with other states, including MA, and Michigan 4 and The results demonstrated that, since the of shock in in the NY PCI for patients presenting with AMI and shock by compared with only in other states (P < 0.001). adjusted overall in-hospital mortality in patients with MI presenting with shock decreased by in NY compared to in states. et al the rates of PCI, or CABG in NY vs. the states of New and and after the Once a significantly higher proportion of the patients underwent PCI, or or revascularization or after the of cardiogenic shock from public reporting in NY. risk of patients with cardiogenic shock and myocardial infarction receiving PCI compared to York states in NY and states 95% confidence on data from Ref. [Color figure can be viewed at wileyonlinelibrary.com] risk of in-hospital death compared for patients with cardiogenic shock and myocardial infarction compared to York states in NY and states 95% confidence on data from Ref. [Color figure can be viewed at wileyonlinelibrary.com] Another are those revascularization who were for CABG There are multiple reasons for including and other clinical and status is associated with greater in-hospital mortality and greater mortality These high-risk patients are not for by current methods because the do not the necessary an unmeasured status may be significant that it might the apparently higher mortality observed with PCI compared with CABG in outcome studies whether high-risk cases or the of RAMR models to assess quality. Patient characteristics that are not for in risk models have the potential to results that falsely or lower measured especially if such characteristics are both important and of the patients from RAMR would increase the that such patients receive needed care without fear on the of providers that an outcome would their own report A concern is whether this could a operators can the by as patients as possible as high-risk This might independent and of correct Public reporting at the operator level implies that poor outcomes are related to operator or clinical However, most poor outcomes are due to acuteness of and than factors An approach to an annual is to the NY reports the individual operator risk-adjusted mortality calculated from the data over the three however, the of this approach is individual physician data over to increase sample is not necessarily that occur in medical and over a (e.g., 3 make individual physician data and In an analysis of hospitals, et al questions about the of statistical to provider performance and the high variance of for low-volume data are unlikely to identify outliers or in a SCAI of public reporting as a to stimulate public in health care. the and of clinical data to the public will and providers to for in outcomes. A summary of to public reporting methods is found in Table In to the RAMR and its EMR, and their SCAI strongly that public reporting programs a of outcome (Table a complete of measures that assess case and such as and would be Although the public reporting of RAMR to be a necessary of reporting outcomes, SCAI does not support its use as the or summary measure of quality. RAMR PCI is a but and metric for the of operator and hospital quality. It is strongly that RAMR be because its in performance quality is uncertain and to risk avoidance in case selection. comparisons among providers and hospitals accurate risk adjustment to account for case selection and a presentation of its An or of risk can the quality of care being Risk adjustment the of mortality, but for a complete of all of care in complex This can only be by a and of procedural mortality at the level of the facility. The unintended consequence of reporting RAMR is the risk-averse behavior it Risk avoidance behavior is to public and it can no longer be that public reporting to this Public reporting of outcomes in high-risk patients can lead to behaviors at the It might risk-averse who are as being operators. For these SCAI reporting RAMR both and of high-risk cases. SCAI that patients with do not and those who have cardiac or cardiogenic shock should be from public reporting after confirms the accuracy of this high-risk of patients. SCAI of data within The reason for a to might be highly predictive and should be evaluated in SCAI studies in this to allow the of cases to be in and to identify that should be in the It is strongly that public reporting from to institutional risk-adjusted outcomes reporting for states. mortality measures are and are to a of SCAI concern about the of reporting the RAMR for individual operators. Public reporting of outcomes in high-risk patients, if at should accurately the performance of those operators and institutions. in the there will be to mortality statistics that are for individual operator quality. the present however, such methods do not especially for the highest risk patients.

Source: Catheterization and Cardiovascular Interventions

DOI: 10.1002/ccd.26818