Author + information
- Received December 7, 2012
- Revision received January 18, 2013
- Accepted February 14, 2013
- Published online June 1, 2013.
- Edward L. Hannan, PhD∗∗ (, )
- Louise Szypulski Farrell, MS∗,
- Gary Walford, MD†,
- Alice K. Jacobs, MD‡,
- Peter B. Berger, MD§,
- David R. Holmes Jr., MD⋮,
- Nicholas J. Stamato, MD¶,
- Samin Sharma, MD# and
- Spencer B. King III, MD∗∗
- ∗University at Albany, State University of New York, Albany, New York
- †Johns Hopkins University, Baltimore, Maryland
- ‡Boston Medical Center, Boston, Massachusetts
- §Geisinger Medical Center, Danville, Pennsylvania
- ⋮Mayo Clinic, Rochester, Minnesota
- ¶United Health Services, Binghamton, New York
- #Mount Sinai Hospital, New York, New York
- ∗∗St. Joseph's Health System, Atlanta, Georgia
- ↵∗Reprint requests and correspondence:
Dr. Edward L. Hannan, School of Public Health, State University of New York, University at Albany, One University Place, Rensselaer, New York 12144-3456.
Objectives This study sought to develop a percutaneous coronary intervention (PCI) risk score for in-hospital/30-day mortality.
Background Risk scores are simplified linear scores that provide clinicians with quick estimates of patients' short-term mortality rates for informed consent and to determine the appropriate intervention. Earlier PCI risk scores were based on in-hospital mortality. However, for PCI, a substantial percentage of patients die within 30 days of the procedure after discharge.
Methods New York's Percutaneous Coronary Interventions Reporting System was used to develop an in-hospital/30-day logistic regression model for patients undergoing PCI in 2010, and this model was converted into a simple linear risk score that estimates mortality rates. The score was validated by applying it to 2009 New York PCI data. Subsequent analyses evaluated the ability of the score to predict complications and length of stay.
Results A total of 54,223 patients were used to develop the risk score. There are 11 risk factors that make up the score, with risk factor scores ranging from 1 to 9, and the highest total score is 34. The score was validated based on patients undergoing PCI in the previous year, and accurately predicted mortality for all patients as well as patients who recently suffered a myocardial infarction (MI).
Conclusions The PCI risk score developed here enables clinicians to estimate in-hospital/30-day mortality very quickly and quite accurately. It accurately predicts mortality for patients undergoing PCI in the previous year and for MI patients, and is also moderately related to perioperative complications and length of stay.
Several groups of investigators have developed statistical models to predict short-term mortality for patients with percutaneous coronary interventions (PCIs) (1–11). Many of these studies provide logistic regression models that could be used with online risk calculators or personal digital assistants (1,3,4,7,8). However, many clinicians still prefer risk scores (risk indices) that enable referring and treating clinicians to quickly sum up points assigned to each risk score, and use a table that assigns a predicted probability of short-term mortality associated with each score (2,5,6,9). All of these risk scores for PCI have been limited to prediction of in-hospital mortality (2,5,6,9). This is problematic because a recent study has demonstrated that nearly two-thirds as many PCI patients die after discharge but within 30 days after the index procedure as the patients who die in the index admission (7).
The purpose of this study is to develop a PCI risk score for in-hospital/30-day mortality. The score is developed using a large population-based registry in New York, the Percutaneous Coronary Interventions Reporting System (PCIRS), which has been in existence since 1992. Although an earlier PCI risk score was created using PCIRS, 30-day mortality after discharge was not available at that time. This is a critical omission in that and other risk scores that are limited to in-hospital mortality, given the very short length of stay for most PCI patients and the relatively high mortality rate for them within the first month after the procedure (9).
The data in the study were obtained from the PCIRS in 2009 and 2010. The PCIRS is a population-based registry founded in 1992 by the New York State Department of Health that contains detailed information on all PCIs performed in nonfederal hospitals in New York. Information in the registry includes lesion and vessel-specific data, demographics, risk factors, complications, discharge destination, and types of devices used, as well as hospital and physician identifiers. In-hospital mortality is one of the options for discharge status. The completeness of PCIRS data is assured by matching it to New York's administrative database, the Statewide Planning and Research Cooperative System (SPARCS), which contains information on both inpatient and outpatient PCIs. Accuracy of in-hospital mortality is confirmed by matching with SPARCS, and the accuracy of risk-factor reporting is assured by auditing samples of records from hospitals.
In order to obtain deaths that occur with 30 days of the index procedure after discharge, the PCIRS was linked to New York vital statistics data and to the Social Security Administration Master File.
Patients in the system consisted of all patients undergoing PCI in nonfederal hospitals in New York in 2010. The total number of patients in the study was 54,223. To examine the validity of the risk model developed using 2010 data, the resulting statistical model was used to predict outcomes for the 54,041 patients undergoing PCI in New York in 2009.
First, a logistic regression model was used to predict in-hospital/30-day mortality (the dependent variable) based on all of the available risk factors in the PCIRS. The first part of this process consisted of identifying which of these risk factors had a significant (p < 0.10) bivariate relationship with the dependent variable. Risk factors examined included age, sex, body mass index, body surface area, pre-procedural myocardial infarction (MI), hemodynamic state, ventricular function, number of vessels diseased, left main disease, and numerous comorbidities. Continuous variables such as age, body mass index, and body surface area were tested using Student t tests and Wilcoxon rank-sum tests; all other variables were tested using chi-square tests.
Risk factors that were significantly related to the mortality measure were then used as candidate independent variables in subsequent logistic regression analyses. The database was split into 2 halves so that each half had roughly the same prevalence for various risk factors and for the outcome rate. A stepwise logistic regression model with in-hospital/30-day mortality as the dependent variable was fit on the first half of the data, and risk factors with a p value <0.10 were used as candidates in another stepwise logistic regression model applied to the second half of the data. Variables that emerged as significant in this model (p < 0.10) were then used as candidate independent variables for the entire 2010 PCI database. The variables that emerged as significant (p < 0.05) in this model were included in the final model. In all models, continuous variables were tested to determine which form (linear, quadratic, linear spline) was most strongly associated with mortality. The fit of the final model was evaluated on the basis of discrimination using the c-statistic (10) and calibration using the Hosmer-Lemeshow statistic (11).
We then converted the logistic regression model into a PCI risk score using the methods applied in our earlier studies (9,12) that were first described by Sullivan et al. (13). First, the continuous variable (age), which was expressed in terms of a linear spline function in the logistic regression model, was split into ranges so that the risk score methodology could be applied. Groups were chosen so that the mortality rates would be as similar as possible within groups and as dissimilar as possible between groups. The ranges chosen for age were <66, 66 to 75, 76 to 85, and 86+ years. Reference values for ranges except the most extreme one were chosen as the midpoints of the ranges (e.g., the reference group for the 66 to 75 age range was 70.5). For the most extreme range at the other end of the spectrum from the reference category (86+), the reference value for the group was chosen as the midpoint between the beginning of the range and the 99th percentile of the range. The constant corresponding to 1 point in the risk score was obtained by multiplying one-half the length of each age range (10 years) by the age coefficient (5 × 0.0668 = 0.334). For all other risk factors, each of which was represented by 1 or more categories in the logistic regression model, the coefficient of each category was divided by 0.334 and then rounded off to the nearest integer. For example, the risk factor “MI within 6 to 11 h prior to procedure” had a coefficient of 1.786, and 1.786/0.334 = 5.35, which rounds to 5.
The total risk score for each patient was obtained by summing the scores of all of the patient's risk factors. For each risk score, the probability of in-hospital/30-day mortality was obtained by developing a new logistic regression model using the risk score as the single independent variable and mortality as the binary dependent variable. The predicted value for each risk score was then obtained by plugging the score into the model. The accuracy of the risk scores in predicting mortality was examined by comparing, for every risk score, the predicted mortality rate for all 2010 PCI patients with that score with the observed mortality rate for that risk score. Confidence intervals were calculated for the observed rates for the risk scores and the predicted values for the scores were inspected to determine if they fell inside the confidence intervals. The same process was then used to see how well the 2010 risk scores predicted mortality for patients undergoing PCI in New York in 2009. Recalibration was required before testing the 2010 risk scores on 2009 data because the underlying in-hospital/30-day mortality rates were different. This was done by multiplying the predicted risk for each 2009 patient by the ratio of the 2009 observed mortality rate and the rate predicted for 2009 using the 2010 logistic regression model. The resulting probabilities are the recalibrated predicted risks for 2009 patients.
The correspondences between the PCI mortality risk score and other adverse outcome measures (complications, length of stay) were also examined for each measure by plotting adverse outcome rates for each value of risk score to determine whether there were higher values of the adverse outcome measure as the risk score increased. Complications analyzed included stroke, vessel injury at the catheter entry site requiring intervention, and stent thrombosis.
All statistical analyses were conducted in SAS version 9.1 (SAS Institute, Cary, North Carolina).
A total of 54,223 PCI patients who underwent PCI in a total of 58 hospitals were used to develop the risk index. The in-hospital/30-day mortality rate for these patients was 1.03%. Table 1 presents bivariate relationships between risk factors in the registry and the presence of in-hospital/30-day mortality for PCI. The significant independent risk factors in the logistic regression model for mortality included age, hemodynamic instability, ejection fraction, pre-procedural MI (with and without ST-segment elevation), peripheral vascular disease, congestive heart failure, malignant ventricular arrhythmia, chronic obstructive pulmonary disease, renal failure, 2 or 3 vessels diseased, and left main disease (Table 2). All variables except age, ejection fraction, MI, and renal failure are binary variables, and ejection fraction, MI, and renal failure are categorical variables with more than 2 categories. Age was represented as linear spline functions, with the risk for age flat until age 66 and rising thereafter. The c-statistic for the logistic regression model was 0.89 and the Hosmer-Lemeshow statistic was 16.11, indicating excellent discrimination and fair calibration.
Scores for the various risk factors are presented in Table 3. As indicated, the highest scores were associated with shock (9 points) and ST-segment elevation MI 12 to 23 h before admission (8 points). The minimum possible risk score is 0 points for a patient without any of the risk factors listed in Table 3, and the maximum possible risk score is 43. The highest score for any patient in 2010 was 34. As noted in Table 4, the predicted probabilities of in-hospital/30-day mortality ranged from 0.09% for a risk score of 0, to 98.74% for a risk score of 34. As an example of how the risk score can be used, a 70 year old who is hemodynamically stable, has an ejection fraction of 40%, had an MI without ST-segment elevation 5 days before admission, and has peripheral vascular disease, no congestive heart failure, no malignant ventricular arrhythmia, no renal failure, and 2-vessel disease would have a total risk score of 1+ 1 + 3 +1 + 1 = 7. From Table 4, the predicted probability of in-hospital/30-day mortality for this patient would be 0.94%.
Figure 1 demonstrates the correspondence between observed and predicted rates for each risk score where observed and predicted rates were obtained from 2010 data. As the figure demonstrates, the observed and predicted values are quite close together, and the predicted rates fall within the 95% confidence interval (CI) for the observed rates for each of the risk score ranges in the figure except for a risk score of 13 to 14, for which the predicted rate (7.89%) was slightly below the lower bound of the observed rate (7.95%).
Figure 2 examines the same correspondence, except that it is restricted to patients with an acute MI (onset within 24 h of the PCI). For each of the ranges of risk scores, the predicted mortality rates were within the 95% CIs for the observed mortality rates except for scores of 13 to 14, where the predicted mortality rate was 7.93% and the lower bound for the CI for the observed mortality rates was 8.94%.
Figure 3 contrasts the observed rates for each risk score in the year 2009 with the predicted values based on the 2010 risk model after recalibrating the 2009 risk score probabilities to reflect the differences in performance between 2009 and 2010. The predicted values and observed values again demonstrate a very good correspondence, with the predicted values for all risk score ranges falling within the corresponding 95% CI for the observed value for all ranges, except score = 1, where the predicted rate (0.13%) was slightly above the upper bound for the observed rate (0.12%). Figure 4 examines the ability of the 2010 statistical model to predict mortality for the 2009 acute MI patients. As indicated, for all ranges of risk scores used, the predicted mortality rates fell within the 95% CI for the observed mortality rates.
Figure 5 examines the correspondence between the 2010 risk score and the probability of 1 or more post-procedural complications (from stroke, stent thrombosis, and vessel injury at the catheterization site) in the index admission. As noted, the complication rate rises or remains the same with each increase in risk score, with the lowest complication rate (0.3%) corresponding to a risk score of 0, and the highest complication rate (2.2%) associated with patients with a risk score of 15 or higher. The c-statistic for complications was a modest 0.67.
Figure 6 shows that there is a reasonably good correspondence between the 2010 risk score and the 2010 post-procedural length of stay. The length of stay rises from a mean of 1.1 days for patients with a risk score of 0 or 1, to a mean of 8.7 days for patients with a risk score of 15 or higher.
Risk scores are simplified, usually additive, scores that assign a specified number of points to important risk factors so that the sum of the scores can be assigned to predicted (usually short-term) mortality rates following a given procedure. The primary purpose of a PCI risk score of this nature is to enable referring cardiologists and patients to assess the patients' chances of short-term survival after PCI for the purpose of informed consent and as an aid to determining whether PCI is the best intervention for each patient. It is also valuable in assessing the additional risk associated with individual risk factors for short-term survival.
This study was conducted to update a PCI risk score developed in New York a few years ago in order to provide more contemporary data for prediction purposes. More importantly, the previous New York risk score was developed using in-hospital mortality as the outcome measure (9), and to the best of our knowledge, this is also true of most other PCI short-term risk scores (2,5,6). However, much of the early mortality following PCI occurs after discharge following the index procedure because of the short length of stay that is typical of PCI. For example, in 2009 in New York, the percentage of patients undergoing PCI who died during the index hospitalization was 0.55%, and another 0.36% died after discharge, but within 30 days of the index procedure. Thus, 0.36/(0.36 + 0.55) = 40% of all short-term deaths occurred after discharge within 30 days of the index procedure (7). Consequently, it is important to be able to predict short-term out-of-hospital deaths for the purposes of informed consent and procedure choice. The purpose of this study was to develop a risk score that accurately predicts all short-term deaths, in and outside of the hospital.
Our previous PCI risk score that used in-hospital mortality as the outcome rather than in-hospital/30-day mortality was based on a mortality rate of 0.70%, whereas the risk score presented here, based on in-hospital/30-day mortality, had an adverse outcome rate of 1.03%. Nevertheless, many of the risk factors represented in the score were identical: age, female sex, hemodynamic instability, shock, ejection fraction, pre-procedural MI, peripheral vascular disease, renal failure, and left main disease were all represented in both risk scores. All risk factors represented in the old score were represented in the new score. However, there were 3 risk factors in the new risk score that were not contained in the previous score: malignant ventricular arrhythmia, chronic obstructive pulmonary disease, and 2/3 vessel disease. The risk factors with the highest score in the original risk score based on in-hospital mortality were shock and MI <24 h with stent thrombosis (each with 9 points). In the new risk score, the risk factors with the highest scores were shock (9 points) and ST-segment elevation MI 12 to 23 h before admission (8 points).
We found that the risk score developed in this study, a very simplified estimate of the logistic regression prediction, does predict in-hospital/30-day mortality quite accurately for the database (2010 in New York) on which the estimate was developed, and also predicts well for another year (2009) of PCI data in New York. For example, the c-statistic of 0.89 of the logistic regression model was very similar to the 0.90 c-statistic of Hamburger et al. (1), and comparisons of observed and predicted mortality rates for various ranges of risk scores confirmed that they were generally not statistically different. Our c-statistic also compares favorably to the 0.762 value obtained by Shahian et al. (14) for 30-day mortality, although it should be noted that their study was limited to patients aged 65 and older, and this may have limited the discriminatory power of their model. It is notable that our risk score did not fit as accurately as our earlier risk score that was developed for just in-hospital mortality, and we postulate that this is in part due to the difficulty of fitting a statistical model to 2 types of patient deaths (in-hospital and after discharge within 30 days of the procedure). Nonetheless, we feel that this risk score is more valuable in that it includes all deaths within 30 days, a duration of time that is aligned with patients' and providers' understanding of “early outcome” as a basis upon which to make informed decisions.
As noted in Table 3, more than 90% of the patients undergoing PCI had a risk score of 9 or less, and more than 70% had a score of 5 or less, with a predicted short-term mortality of <0.5%. Thus, for a large proportion of patients, the short-term mortality risk is not a deterrent in choosing PCI in lieu of a competing intervention.
To determine whether the risk score we developed is generalizable, it should be tested against other populations. The means by which it should be tested depends on the purpose of the risk score. The 2 main purposes for which risk scores have been used are: 1) to compare the performance (risk-adjusted outcome) in another population with the performance in the population in which the risk score was developed; and 2) to predict the bedside risk of a specific patient in the new population for purposes of informed consent and as an aid in treatment determination. For the first purpose, it is probably better to use a logistic regression model like those published in New York's annual reports because although not as easy to use, these models provide more exact estimates of predicted mortality (7).
The second purpose should be the primary use of a risk score, but to use the score appropriately for this purpose, the score should be recalibrated in order to make it more relevant in the other setting, which may have a significantly different risk-adjusted mortality rate. When used as an aid to treatment determination, it would ideally be used in conjunction with other considerations (e.g., longer-term mortality, symptom relief).
The recalibration process consists of calculating a new mortality rate for each risk score in the other population by multiplying the observed mortality rate for the New York (2010) population by the ratio of the observed mortality rate in the other population divided by the rate of the other population predicted by the New York risk score (the mean of the predicted probabilities of mortality for all patients in the other population). This recalibrated score can then be used to predict the probability of short-term mortality for each patient in the other population.
An advantage of the New York risk score is that it is based on data from all 58 New York State nonfederal hospitals in which PCI was performed in 2010. We used population-based data that included virtually all PCIs in a large region (New York State) in the analyses. Use of such a database contributes to the accuracy and generalizability of our findings. Furthermore, the completeness of the database has been verified by matching it to New York's administrative database, and the accuracy of the data has been enhanced by matching to this administrative data and to vital statistics data, and by extensive auditing by New York's utilization review agent. The use of vital statistics data to confirm 30-day deaths outside of the hospital is particularly important because we found that hospitals grossly underreport 30-day deaths. We also found that, as did our previous risk score limited to in-hospital mortality, this risk score performed reasonably well as a predictor of complication rate and of length of stay. However, any attempts to predict complications and length of stay with strong accuracy should involve developing separate risk scores for these adverse outcomes.
Funded in part by New York State Department of Health. Dr. Berger is a consultant for Medicure and Janssen; and receives research funding (to the Geisinger Clinic) from Helena, Novartis, Tethys, Thrombovision, Accumetrics, AstraZeneca, Haemoscope, The Medicines Co., Bristol-Meyers Squibb/Sanofi, and Eli Lilly/Daiichi Sankyo. Dr. Sharma is on the Speakers' Bureau for Boston Scientific Inc, Abbott Vascular, Eli Lilly, and The Medicines Co. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose. Eric Bates, MD, served as Guest Editor for this paper.
- Abbreviations and Acronyms
- confidence interval
- myocardial infarction
- percutaneous coronary intervention
- Received December 7, 2012.
- Revision received January 18, 2013.
- Accepted February 14, 2013.
- American College of Cardiology Foundation
- Chowdhary S.,
- Ivanov J.,
- Mackie K.,
- et al.
- O'Connor G.T.,
- Malenka D.J.,
- Quinton H.,
- et al.,
- Northern New England Cardiovascular Disease study group
- Shaw R.E.,
- Anderson H.V.,
- Brindis R.G.,
- et al.
- Moscucci M.,
- Kline-Rogers E.,
- Share D.,
- et al.
- Qureshi M.A.,
- Safian R.D.,
- Grines C.L.,
- et al.
- New York State Department of Health
- Holmes D.R.,
- Selzer F.,
- Johnston J.M.,
- et al.
- Wu C.,
- Hannan E.L.,
- Walford G.,
- et al.
- Hosmer D.W.,
- Lemeshow S.
- Hannan E.L.,
- Wu C.,
- Bennett E.V.,
- et al.
- Shahian D.M.,
- O'Brien S.M.,
- Sheng S.,
- et al.