Can valid and practical risk-prediction or casemix adjustment models, including adjustment for comorbidity, be generated from English hospital administrative data (Hospital Episode Statistics)? A national observational study

Article history

The research reported in this issue of the journal was funded by the HS&DR programme or one of its proceeding programmes as project number 09/2001/32. The contractual start date was in September 2010. The final report began editorial review in March 2014 and was accepted for publication in August 2014. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

The Dr Foster Unit at Imperial College London is principally funded by Dr Foster Intelligence, an independent health-care information company. The Dr Foster Unit is affiliated with the National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre. Dr Alex Bottle and Professor Paul Aylin are part funded by a grant to the unit from Dr Foster Intelligence outside the submitted work.

Copyright statement

© Queen’s Printer and Controller of HMSO 2014. This work was produced by Bottle et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Hospital Episode Statistics database

Our unit holds inpatient and day case HES and Secondary Uses Service (SUS) data from 1996/7 to the present with monthly SUS feeds. We apply published HES cleaning rules to the SUS feed. Details of this and our other processing, such as episode and spell linkage, can be found on documents on our website (www1.imperial.ac.uk/publichealth/departments/pcph/research/drfosters/reports/ under ‘HSMR methodology’). HES data cleaning rules may be found on the NHS Information Centre website under HES. Throughout this report, we refer to all these records as ‘HES data’. Briefly, each record in the inpatient part of the database is a finished consultant episode, representing the continuous period of time during which the patient is under the care of a consultant or allied health professional. We link episodes into ‘spells’ (admissions to one provider) and link spells into ‘superspells’, the latter combining any interhospital transfers. We will refer to superspells as ‘admissions’ and in general use them as the unit of analysis throughout.

We hold outpatient HES and SUS data since they became part of HES in 2003/4 and hold A&E records since they became part of HES in 2007/8. The most recent A&E records we had for the project were for 2011/12. The Information Centre’s contractors provide a file to enable us to attach the date and cause of death to HES records, with the latest date of death being August 2011.

We derive a number of fields, in particular the main diagnosis group, the main procedure group and the Carstairs deprivation quintile. We use the Agency for Healthcare Research and Quality (AHRQ) Clinical Classification System (CCS) to turn the International Classification of Diseases, Tenth Edition (ICD-10) codes of the primary diagnosis field into one of 259 diagnosis groups designed for health services research (see www.ahrq.org for details). For procedures, no such system exists for the UK’s Office for Population, Censuses and Surveys (OPCS) procedure codes. Over several years, in conjunction with clinicians, we have created a number of procedure groups. 2 The Carstairs quintile3 is assigned at the super output area geographical level using information from the 2001 census; the necessary 2011 census information was not available in time for this project. Although Carstairs is therefore based on older information than the Index of Multiple Deprivation, its resolution is greater.

We now briefly consider the quality of HES data.

Definition of outcome measures and predictors

The two most commonly used mortality measures are in-hospital mortality (any time during the index stay) and 30-day total mortality (in or out of hospital within 30 days of the index admission or procedure date). For readmission, the most commonly used internationally is the 30-day all-cause version; in the UK, 28 days is typically used. Unless otherwise stated, ‘readmission’ as an outcome means within 28 days; for the HF analyses we used 7-, 30-, 90-, 182- and 365-day follow-up periods rather than 28 days.

For HF, the number of admissions and inpatient bed-days during the 365 days following discharge were also counted; these and also the first readmission were split by primary diagnosis into those for HF and those for any other cause (non-HF). OPD non-attendance was simply defined using the ATENTYPE (attendance type) field. The first appointment following inpatient discharge for selected acute conditions was readily identified using dates; of particular interest here was whether or not non-attendance rate varied by the time interval between discharge and the appointment, as if so this would represent an opportunity for the hospitals to reduce missed appointments. The patient’s first new appointment for the specialties of general medicine and general surgery were identified using the field FIRSTATT and also by tracking back 3 years in the database to ensure that this was each patient’s first for that specialty (or at least their first for 3 years).

The construction of the RTT measures followed the same outline for each specialty as for cystectomy in urology. 7 All procedure fields with dates between 1 and 29 days inclusive of the index procedure date, even if performed after transfer or discharge from the index stay, were extracted from the database and inspected by a senior surgeon and clinical coder to determine which were likely to be reinterventions. These can potentially encompass surgical operations, both surgical and non-surgical procedures, including radiological interventions. For orthopaedics, we defined RTT between 1 and 89 days after the index procedure in order to capture infections, which may take a few weeks to develop. The index procedures of hip and knee replacements were each divided into subgroups with advice from a senior surgeon. Hip operations were divided into total with cement, total without cement, hybrid and hip resurfacing. Knee operations were divided into total, unicondylar (or unicompartmental) and patellofemoral replacement, taking into account the national changes in coding practice in 2009/10 regarding Z845 (tibiofemoral joint) and W581 (resurfacing). The small number of otherwise unclassified procedures were excluded. Using laterality Z codes, we matched the RTT to the joint where possible – around 95% of joint procedures had laterality codes.

Potential covariates for the risk-adjustment models were taken from our national monitoring system that uses HES:2 age, sex, deprivation quintile, year, month of admission for respiratory conditions, admission method (elective or emergency if the patient group included both), emergency admissions in the year before the index stay, comorbidities and potentially an interaction between age and comorbidity. For the HF readmission models, we also tried incorporating previous admissions using more sophisticated methods, and we used the number of prior OPD appointments attended and the number missed. HF models also included some procedures. Further details are given in the relevant later sections.

Table 1 summarises which outcomes will be presented for which patient groups.

Review of comorbidity indices

The two most commonly used indices are the Charlson1 and Elixhauser,8 originally described using International Classification of Diseases, Ninth Edition (ICD-9) with US data. In each index, points are given for the presence of a set of codes representing diseases associated with higher or sometimes lower risk than if the disease is not present. The points are then summed to give a score for the admission. 9 The Charlson index is now over 20 years old and both indices need calibrating on the data set of interest; to our knowledge, very little has been published from the UK on this. The weights (or scores) for these two indices may be inappropriate for the UK because of differing populations and/or coding practices. We have to date been using an Australian version of Charlson10 in our risk-adjustment models (there are a variety of others for ICD-9), but discussions with clinical coders raised questions over the suitability of some codes when used in the UK. According to their advice, we modified the codes for acute myocardial infarction (AMI) and dementia in particular. 11

A decade or so later, Elixhauser et al. 8 constructed a more extensive index covering 30 conditions. This index was designed for administrative databases such as HES that lack present on admission information, which means that they cannot distinguish between comorbidities and complications that arise during the stay. This is why Elixhauser does not include AMI or stroke, which are both part of Charlson. Some comorbidities were associated with lower risk of mortality and were given negative scores, confirmed in a Canadian analysis. 12 One reason for this was given by Elixhauser: low-risk patients may be given more codes for less acute problems than acutely ill people, for whom coders will focus on problems relevant to the acute situation. The presence of codes for non-life-threatening disease may therefore be a marker for relatively healthy patients.

Including Charlson and Elixhauser, there have been different approaches of selecting sets of comorbidities. The most common typically involved considering the prevalence of each condition and medical expert opinion. 13–17 Gagne et al. 18 and Thombs et al. 19 simply combined all the conditions listed in Elixhauser and the Deyo adaptation of the Charlson index except those conditions thought to be related to the main diagnosis. The simplest approach is to count the number of conditions, thereby giving them equal weight.

To find the best-performing one(s), we conducted a systematic review of multiple comparison studies on comorbidity measures/indices in use with administrative data. The review used a new meta-analytical approach, hypergeometric tests, to identify the best-performing indices. We now briefly outline the search strategy, results and conclusions. Full details may be found elsewhere. 20

The literature search was conducted first using three electronic databases, MEDLINE, EMBASE and PubMed, up to 18 March 2011. Search terms included Charlson, Elixhauser, comorbidity, casemix, case-mix, mortality and morbidity. In addition, the bibliography of the chosen articles and the relevant reviews were searched. 21–25 Two authors independently reviewed the titles and abstracts and the methodology section of the articles and selected the relevant potential articles. All disagreements were resolved by consensus.

Articles were included if administrative data were used and comparisons between predictive performances of at least two indices (perhaps also including common covariates) were made. Articles were excluded if clinical databases without ICD codes were used or if indices were used for the purpose of adjustment only, without any comparison of indices.

The c-statistics and the confidence intervals, or Akaike information criterion (AIC) or other appropriate statistics, were looked at if they were available, to compare the predictive performances of the measures/indices. If c-statistics alone without confidence intervals (or p-values) were provided, an arbitrary ≥ 0.02 cut-off point was chosen to define the difference between two measures/indices as significant.

All mortality outcomes were divided into two major groups: (1) ‘short term’ – all inpatient mortality and any mortality within 30 days of admission; and (2) ‘long term’ – outpatient mortality or that 30 or more days from admission.

After removing duplicates from different databases, we retrieved 1312 studies to review, subsequently reduced to 54 articles eligible for data abstraction. Of these, 90% were carried out in North America.

In summary, for short-term mortality (up to 30 days), the use of empirical weights (of whichever index) or the small group of various other comorbidity measures performed best. For long-term mortality (30+ days), these ‘other measures’, the Romano version of Charlson and the Elixhauser measure performed significantly better. As others have suggested, there may be useful gains if the two indices are combined. In practice, this would mean taking the Elixhauser set and adding dementia if we excluded AMI and stroke (Charlson components), which could occur after admission as complications but whose timing cannot be determined using HES. Following a review of the Sundararajan et al. comorbidity ICD definitions10 by external clinical coders, we amended the Charlson algorithm so that it uses extra codes for AMI and dementia in particular.

Statistical methods

Recalibration

Being built around a sigmoid output node, ANNs produce output that can be interpreted as a probability estimate, as does LR. The greater variational freedom inherent in the algorithm then means that the result is potentially better calibrated. Therefore we did not recalibrate the ANN output.

Methods that are known to have calibration issues, such as random forests,48 or are not even designed to produce meaningful probability estimates, such as SVMs, have to be recalibrated. We also found a number of the LR models to show overprediction of low risk in particular. As LR may be viewed as an ANN with no hidden nodes, back-propagation is applicable and can be adapted easily to optimise the internal shape parameters of the output node. Adjusting the shape parameters changes the calibration. Standard LR is based on optimising a log-likelihood-based measure and results in perfect overall calibration (i.e. total observed counts equal total predicted counts). Optimising the mean square error does not result in perfect overall calibration. However, the difference in probability estimates using each optimisation scheme encodes the deviance from an optimal model for which the difference would be zero. (This follows from the fact that both measures are so-called ‘strictly proper scoring rules’.) We found that feeding the probability estimates back into a further LR iteration in conjunction with the other covariates improved both discrimination and calibration. This will be labelled ‘LR+’ in the results. Only feeding back in the standard LR probability estimates also improved the final result but to a much lesser degree.

The standard method for recalibration is isotonic regression pool-adjacent-violator algorithm (IR-PAV),49 which we have used throughout. However, this results in a series of steps in the predicted probability curve, which is biologically implausible. Later in the project, we developed a smoothing technique using splines – full details are found elsewhere. 50 To illustrate, the original LR probability estimates for 30-day total mortality following AMI are mapped onto recalibrated estimates for the two methods, compared in Figure 1. The blue ‘diagonal’ line represents no recalibration. The effect of the recalibration is to lower very low probabilities and high probabilities and to push up those in the middle. In other words, LR first over-, then under- and then overestimates probabilities as one moves from low to high values. Depending on the exact use, correct calibration in addition to discrimination may be of importance. Note that the bulk of the patients in this example are to be found between 0.05 and 0.4 so the large discrepancy for very large values affects only few patients.

FIGURE 1.

Comparison of standard (PAV) and alternative (SEP) recalibration methods. Graph (a) shows probabilities up to 0.5; and graph (b) shows the whole data set. SEPs, splined empirical probabilities.

Methods for including information from previous admissions into the prediction models

Many patients, particularly the elderly and those with chronic conditions, have a number of health service contacts. As mentioned before, data quality considerations currently limit the use of A&E attendances (we decided not to use these) and OPD appointments to counting OPD appointments attended and missed; see the last section of Chapter 3. With previous admissions, one may use the procedure and diagnosis information (the latter can be used to augment comorbidity codes as mentioned earlier) and also the timing of them relative to the index contact. A simple but crude approach is to count the number of emergency admissions for any reason within the previous year. 2 To use the primary diagnosis of these previous admissions, one needs to use some grouping algorithm such as the AHRQ’s CCS groups and then combine the infrequent ones. The relative timing, which is recorded in number of days between admission dates, may also be incorporated, as in principle may the duration of each stay.

Intuitively, one may think that a recent readmission better predicts the outcome of the index admission than one that lies further back in time. If we ignore the durations and consider just the fact of admission, then a 3-year lookback period generates roughly 1000 binary variables – one for each day. Determining the effect of each of these variables cannot be done by simply adding them to a LR model. Some aggregation is needed. This also smooths out small time differences (e.g. whether a previous admission is 673 days in the past or 674 is unlikely to be of major significance). We used a two-step process as follows.

First step:

• We generated the 60 lowest-order orthogonal polynomials of our 1000-dimensional vector space of admission dates. The base polynomial (a horizontal line) then is equivalent to just summing the number of admissions; the next polynomial (a diagonal) translates into a sum weighted by the time difference to the present admission, etc.

• The value of the sums weighted by these polynomials results in 60 new variables. These were used in a LR model.

• Using the LR coefficients yields a weighting of polynomials, which results in a single weighted sum that in a LR model would give the same result.

• As expected, this weighted sum puts about five times greater emphasis on very recent previous admissions than those further in the distant past. The decline in weights occurs over about the first half-year or so.

Second step:

• The weighted sum above may be used to generate new weighted polynomials. These will give greater prominence in magnitude and resolution to where the weighting is largest, that is to recent admissions. The base polynomial stays the same.

• One or more such polynomials may now be used as weighted sums to generate variables that can be added to a LR model.

• We also included a flag for no previous admission within the last 3 years.

To test this, we considered two years of index HF admissions as used in the readmission analyses. Our outcome was unplanned readmission within 28 days. All of these models contained the set of casemix variables used in the comparison of methods above plus the flag for no previous admission. The orthogonal polynomials were generated in SAS using PROC IML, and the weighted sums were implemented in SAS using PROC FCMP.

The effect of the semi-competing risk of death on readmission-type measures

As death precludes subsequent readmission, using LR – which ignores any effect of death either during or following the index discharge – may be potentially misleading. We therefore also applied cause-specific proportional hazards modelling and subdistribution proportional hazards modelling. These two survival analysis methods make different assumptions regarding post-discharge deaths;51,52 other methods exist, but these are the two most widely used. The PSHREG macro in SAS was run for subdistribution hazards. 53 If the odds ratios (ORs) and two sets of hazard ratios all agree, then we can be fairly confident that the effect of post-discharge deaths is minor. For the proportional hazards models we inspected the standard errors and deviance and Schoenfeld residuals; as our focus here was on the validity of using LR as the standard method rather than on the precise specification of a survival analysis model, we did not perform the usual formal tests of proportionality.

As is standard, our analyses of readmissions were restricted to patients discharged alive from their index admission. We did not attempt to take account of deaths during the index admission, and we acknowledge that this is a potential limitation.

Methods for predicting future bed-days in heart failure patients

As well as modelling the first unplanned readmission, as is common practice, we also wanted to consider broader measures of future hospital use. Internationally, the literature on HF readmissions is dominated by studies predicting a single readmission, usually 30-day readmissions,54,55 though some used longer follow-up periods of 90 days56 and 12 months. 57 Braunstein et al. 58 modelled the number of all-cause and HF ambulatory care-sensitive conditions (ACSCs) and all-cause ACSC hospitalisations during 12 months. The ACSC literature tends to focus on trying to predict the high-risk patients. 59,60 Few studies have modelled the number of further admissions. Johnson et al. 61 considered the number of HF-related hospitalisations during 3 years of follow-up using a negative binomial model. Although Chun et al. 62 modelled cardiac and non-cardiac hospitalisations from discharge until death, they took a survival analysis approach that included repeated-events time-to-event analysis. We found no study that considered total future bed-days in HF patients.

Resource-use measures are often characterised by distributions displaying overdispersion and an excess number of zero counts, so a range of count data models including Poisson, negative binomial with quadratic variance function (NB2), zero-inflated Poisson, zero-inflated negative binomial (ZINB), Poisson hurdle and negative binomial hurdle (NBH) models were fitted. The NB2 model allows for overdispersion by relaxing the Poisson model assumption of equal mean and variance. The zero-inflated and hurdle models provide alternative ways of capturing the excess zero counts. The former treats zeroes as arising from one of two sources – always zero and possibly zero – and the latter treats zeroes as arising from only one source. For a review of such models, see Hu et al. 63

Mortality rates in the year after discharge from the index HF admission are high. Accordingly, the number of days the patient survived following discharge from their index HF admission, capped at 365 days, was used as on offset term in the regression models to control for differing opportunity to accrue readmissions and bed-days. 61

‘Buckets’ approach

An alternative approach to modelling the number of bed-days directly is to categorise the count. We assigned patients with a positive number of bed-days to one of five resource-use buckets. The thresholds for each bucket were allocated such that the total resource use (number of patients × number of bed-days) in each bucket was approximately equal. The factors influencing membership of the highest (and two highest) resource-use buckets were examined using LR.

Other modelling details

The models described above assume that observations are independent, so the conditional ICC was calculated to assess the need for models taking account of the clustering of patients in hospitals. We used the same set of covariates as for the first readmission. This time, we also examined the additional value of including specific ICD-10 Z-codes as proxies for care in the community, patient support networks and likely adherence to post-discharge advice. One candidate was the code for living alone, whose recording in HES has been mandatory since 2009.

Nested models were compared using likelihood ratio tests, log-likelihood, AIC and Bayesian information criterion. For pairs of generalised linear models we additionally compared the deviance. Non-nested models were compared using the tests of Vuong64 and Clarke. 65

The mean predicted probability for each count value and observed probabilities were compared graphically. Pearson’s chi-square test and the chi-square goodness of fit test proposed by Cameron and Trivedi (p. 195)66 were not used, as they are not informative with large sample sizes such as ours.

The discrimination and calibration of the logistic models were assessed using the c-statistic and HL statistics respectively.

Adjusting for comorbidity: results from updating the Charlson index

Amending the ICD codes following the expert clinical coder review and literature search led to increases in prevalence for dementia and AMI in particular. Comparison with the original 1987 published weights showed notable declines in relative importance for human immunodeficiency virus (HIV) and increases for dementia, a condition that was not even included in the original 1998 formulation of the Elixhauser index. For illustration, we present the weights derived from 2008/9 inpatients (Table 2).

As we found in our systematic review, Elixhauser generally outperformed Charlson for both mortality and readmissions. To illustrate, Table 3 gives the c-statistic and adjusted R² values for both indices for AMI and COPD for mortality. More details are given elsewhere. 67

Incorporating information from prior admissions is often done using a 1-year lookback period but sometimes using a longer one. Table 4 compares the ORs for a set of comorbidities relevant to ACSs when using a 1- and a 5-year lookback and when using none, i.e. when using only the secondary diagnosis codes from the index admission. Some effects were modified when using prior admissions. Overall, a 5-year period offered little benefit over a 1-year period, and a 1-year period offered little over no lookback. Further details are found elsewhere. 67

Such lookback approaches ignore the potential impact of the timing in relation to the index admission. More sophisticated approaches using polynomials as described earlier were applied to our HF cohort, and we now give the results.

Mortality and readmission from logistic regression

Following on from the previous section, we used the Elixhauser set plus dementia to adjust for comorbidity in this section. Model performance in terms of discrimination varied considerably by patient group and outcome (Table 5), with readmissions of any follow-up length being harder to predict than death. Discrimination was generally higher for 28- than for 7-day readmission but similar for the two death outcomes. Because of the large sample sizes, the confidence intervals for the c-statistics were very narrow and are not shown in this report.

Calibration was much better for the procedures than for the diagnosis groups. It is common to find that LR models are not always well calibrated, with overprediction of low risk and underprediction of high risk common problems. Where calibration was found to be unacceptable as judged by the HL p-value (though see earlier commentary on this measure), we found overprediction of low risk to be a common explanation. A typical example is given below for 30-day total mortality following AMI. Figure 2 plots the ratio of the observed to predicted deaths.

FIGURE 2.

Ratio of observed to predicted deaths for each risk decile for 30-day mortality following AMI.

Comparison of methods: logistic regression and machine learning

For the LR models in this section of the analysis, we therefore took output from both the usual LR models and those with the extra recalibration step as described in the earlier recalibration subsection. We compared these with several machine learning approaches. Table 6 gives the resulting discrimination for AMI and elective colorectal surgery for mortality and readmission (results were similar for pneumonia and are not shown). In general, ANNs sometimes gave slightly better c-statistics, but LR performed comparatively well. Overfitting was a problem for the random forests in particular: for example, for AMI mortality, the training set c-statistic reached 0.92, which fell to 0.74 for the test set. For SVMs, only the best results are shown; performance varied by the SVMs’ parameters. In general, we found that overfitting was severe on the raw data for any kernel. However, transforming the data to principal components analysis space, as described earlier, massively reduced overfitting and also stabilised the algorithm. The total predicted outcomes differed from the total observed outcomes because the results are for the test set, whereas any calibration is learnt/trained on the training data set. Even the best method would suffer from at least some such statistical fluctuations.

It is noteworthy that random forests suffered more from overfitting than the other methods. This also had repercussions for recalibration: the raw probability estimates were generally too low. In particular we observed that random forests picked up a sizeable number of positive outcomes and grouped them together. Pool adjacent violators recalibration then maps these onto high-probability estimates. Unfortunately, these high probabilities were accurate only on the training data set (overfitting), and hence the corresponding HL test resulted in extremely large chi-squared values, which we have not given in the table.

When these models are used in risk adjustment, the resulting RRs at unit level are also of interest. These were derived by summing the observed and summing the predicted for each hospital and dividing the former sum by the latter sum. Table 7 compares the RRs derived from LR (both standard and recalibrated) with those derived from the other methods for AMI in terms of their funnel plot outlier status. Hospital trusts with at least 50 AMIs in two years were included. The patterns for colorectal surgery were similar and are not shown.

There were more low than high outliers. In general, there was a lot of similarity between the sets of SMRs themselves and between the numbers of outliers across six of the seven methods. The exception was the random forests, for which recalibration consistently gave the highest total expected count for the four sets (AMI and colorectal, mortality and readmission) and resulted in by far the highest number of hospitals flagged as significantly low on the funnel plots. This was due to overfitting and the assigning of a very high predicted risk to too many patients.

Return-to-theatre metrics for orthopaedics

Following on from our earlier work in urology for cystectomy,7 we give results for total or partial hip replacement (HR) and knee replacement (KR); colorectal excision results may be found elsewhere. 68

There were 260,206 index HR procedures and 286,590 index KR procedures during the 6 years 2007/8 to 2012/13 combined. Because of invalid or unknown consultant team codes, 2153 HRs and 2874 KRs were excluded. A further 112 HRs and 29 KRs were removed at hospitals performing fewer than 30 procedures over the 6 years. This left 260,370 HRs among 2029 surgeons and 315,454 KRs among 2061 surgeons for analysis. Of these, there were 5353 RTTs within 90 days for a rate of 2.1% for HR and 5508 RTTs within 90 days for a rate of 1.8% for KR.

The most common reinterventions were closed reduction of dislocated total prosthetic replacement for HR (34% of the total) and attention to total prosthetic replacement of knee joint not elsewhere classified for KR (28% of the total), the latter mostly representing manipulations under anaesthesia for stiffness. Age, sex and many comorbidities were significant predictors for both index procedures. Age had different relations for hips (ages over 75 years had higher odds) from those for knees (ages under 60 years had higher odds, particularly those under 45 years). Some comorbidities, such as arrhythmias, dementia, obesity, fluid disorders and Parkinson’s disease, were associated with higher odds of RTT for both index procedures, but others, such as liver disease, depression and alcohol abuse, raised the odds only for HR. There was no relation with area deprivation. Hip resurfacing had around half the odds of other hip subgroups; partial knee replacements had around half the odds of other knee subgroups.

Figure 3 shows the variation for RTT following HR by surgeon (the plot for KRs is very similar and is not shown). Because of the known unreliability of consultant codes in HES for unplanned activity, we analysed and give the figures for elective procedures only.

FIGURE 3.

Funnel plot for RTT following HR by surgeon.

The discrimination of the two models was low (c = 0.61 for HR and 0.60 for KR) but the calibration was acceptable using the HL test (chi-squared 9.2, p = 0.324 for HR, and chi-squared 13.3, p = 0.102 for KR).

Variation by surgeon for HR was similar to that for KR (standard deviation 0.22, Table 8). Patient factors explained little of this for either procedure. Accounting for the hospital explained around a quarter of the variation between surgeons for HR and nearly half (44%) for KR. This suggests that three-quarters of a surgeon’s RTT rate for HR and half their rate for KR are explained by factors other than patient factors or the hospital at which they operate.

Outpatients department non-attendance

Many reasons for patients missing their OPD appointment have been identified. We now give a brief summary of the literature that is not meant to be exhaustive. Reasons for missing include simple forgetting or confusion over the appointment’s time or location,69 illness severity (either mild, including recovery from symptoms, or severe),70,71 administrative and/or communication errors,69,72–74 being busy or unable to take time off work or arrange child care. 75,76 Previous unsatisfactory experience of health care, such as long waiting times and unhelpful staff with poor communication skills, also has an impact,77,78 as do not being involved in the referral decision-making process69,71 and not understanding the reasons for the appointment. 79 Previous non-attendance is a strong predictor for future behaviour related to attendance,78 with the probability of missing a second appointment being much higher than the probability of not turning up to the first. 75 There is an association between the number of appointments and rates of non-attendance, with high users of outpatient services being more likely to default on an appointment. 78 These patients are likely to have long-term chronic diseases, which may have not changed since their last appointment. 73 Because of this factor, there is often a difference in attendance rates between those attending their first appointment and those attending subsequent follow-up appointments. King et al. 80 found that patients with diabetes and glaucoma felt that, as their condition was stable, missing an appointment was unlikely to harm them.

Several studies have looked at the relation between non-attendance and factors such as demographic information and personal circumstances, for example family size or socioeconomic deprivation. With age, it is likely that non-attendance rates have a bimodal distribution, with young adults and teenagers as well as the elderly being those most likely to miss their appointments. 71 It has been reported that alcoholics, intravenous drug users and those who are pregnant have higher non-attendance rates,75 as well as past and current smokers. 81,82 Patients who are single parents or have children living at home are more likely to be non-attenders. 81 Those with young, large families are also more likely to miss appointments,76,83 as well as those with a poor family support system. 69 Area-level deprivation scores are often strong predictors of non-attendance. 70,84 Gatrad85 found higher rates of non-attendance in Asian populations for both new and follow-up appointments, though these attendance rates improved when culturally specific interventions were introduced.

As well as analysing first-time appointments, we also considered patients recently discharged from inpatient treatment. Just as the time interval between referral and appointment has been found to be a strong predictor of non-attendance,86 so is the time interval between discharge from inpatient admission and the first subsequent outpatient appointment. 70 If there is a lengthy delay then patients are less likely to attend the appointment for a number of reasons, including the appointment being unnecessary as their condition has improved, or simply the patient forgetting the appointment. 78 Two studies found that those waiting more than 2 months from referral to appointment were least likely to attend. 73,87 On the other hand, too short an interval, such as less than 3 days or a week, between referral and appointment had a higher rate of non-attendance, as this short notice does not give patients enough time to reschedule their commitments. 73,79 Hamilton et al. 70 found that, for every 1-week increase in interval, patients were 5–9% more likely to miss their appointment.

While many of the foregoing factors such as personal circumstances are not available in HES, several key ones are, such as age, area-level deprivation, prior non-attendance and time interval between inpatient discharge and the first subsequent appointment. We also used HES to track later hospital activity in our patient cohorts.

Table 9 gives the results for the first appointment in general medical and general surgical OPD; Table 10 gives the same but for the first appointment following inpatient discharge. In both tables, to reduce size, only a few age groups and none of the many diagnosis group and subgroup combinations have been shown.

Odds ratios for each patient factor were very similar whether the index appointment was for general medicine or general surgery. Non-attendance was most likely in the under-30s. Many of the factors found to be important in Table 8 were also important here, with the exception of gender. In addition, index length of stay (LOS), comorbidity (Charlson index) and time interval between discharge and subsequent appointment were also included in the models and were all significant. Following discharge from an inpatient admission, factors were similar whatever acute condition the patient had been admitted for, and therefore all seven conditions have been combined in Table 10.

Key predictors were young and very old age, male gender, deprivation, previous non-attendance and previous emergency admissions. The time interval between discharge and the first appointment thereafter was also a predictor for three of the seven CCS groups: ischaemic heart disease, stroke and HF. Intervals of less than a week had the highest odds of non-attendance. The ‘optimal’ interval for scheduling appointments appeared to be 3–10 weeks for ischaemic heart disease, but no such window was apparent for the other diagnoses considered separately. Combining all seven patient groups suggested that a 6-week interval was optimal, with intervals of 3 and 4 weeks also having significantly lower odds of non-attendance than 12 or more weeks.

The performance of these models was only moderate. Discrimination was 0.67, and there was significant overprediction of low risk.

Patients who did not attend their first post-discharge appointment had more emergency admissions, total inpatient bed-days and further non-attendances in the subsequent year than those who did; however, they had fewer elective admissions and total OPD appointments (Table 11). The means in the two groups varied by index diagnosis group, but the relative differences were consistent. We therefore show the combined figures for all diagnosis groups considered. Although the mean contacts and death rates were lower following a first general medical or general surgical appointment after GP referral, the patterns were the same and are not shown.

Table 11 shows the crude differences between attenders and non-attenders, but the earlier tables showed that the two groups differ in several characteristics that may at least partly confound such unadjusted comparisons. As an example, we ran a LR with death in the year after the index appointment as the outcome, adjusting for the factors in the tables plus the fact of non-attendance as an extra predictor. These models suggested that non-attendance was associated with about a 50% higher odds of death, only slightly reduced from the unadjusted figure.

Other readmission measures and future bed-days: the example of heart failure

During 2008/9 and 2009/10 there were 74,419 live discharges with their first admission with a primary diagnosis of HF. Within 7 days of discharge, 4.7% had an emergency readmission; within a year, around half had been readmitted. A third of these readmissions within 7 days were for HF, dropping to 21% for those within a year. This was matched for patients with an index admission of COPD. Many of the findings for COPD were similar to those for HF. We therefore focus on the HF cohort in this section.

Discrimination was low for all periods of follow-up between 7 and 365 days for either all causes combined or cause-specific measures (c = 0.58 to 0.60). All models were well calibrated (p > 0.10 using HL).

Predictors were similar across all follow-up periods for all-cause readmissions (Table 12) but sometimes differed by cause of readmission (Table 13). Hazard ratios from the two types of survival analysis considered were generally very similar and were also similar to the ORs from hierarchical LR; we therefore present only the latter. Results for 90 and 182 days matched those for 30 and 365 days and are not shown.

Length of stay was seen to have a bigger association within 7 days than within 30 or more days and was more important for HF. LOS was truncated at 3+ days for presentation, as analysis using a highest band of 14+ days did not alter the conclusions.

Readmission rates varied widely by hospital for each cause. For this analysis, we combined 4 years of index HF admissions (2009/10 to 2012/13) and included only emergency admissions in order to be more contemporaneous with the National HF Audit, which published figures by hospital for the first time for 2011. Thirty-day rates for HF ranged from 1.5% to 9.0% (median 5.4%), while 30-day rates for non-HF ranged from 7.6% to 17.6% (median 13.6%). Rates showed statistically significant but modest negative correlations (rho from –0.18 to –0.24, p < 0.05) with all six publicly available quality of care measures from the National HF Audit88 for readmissions for HF within 7 days but only three of them for readmissions for HF within 30 days; there were no significant correlations between these measures and non-HF readmission rates.

Finally, we applied a range of models to try to predict not just the next unplanned readmission but future unplanned inpatient activity within the year after discharge: readmissions (all-cause and HF-specific) and bed-days (all-cause and HF-specific). Activity was also divided into ‘buckets’ and models fitted to predict which patient would go on to be in the highest resource-use categories. Around half the patients had no further emergency admissions in the year after discharge; the Poisson model therefore fitted the data poorly, particularly for bed-days. When they converged, all other model specifications performed well for readmissions and bed-days. The NBH model performed best, though convergence was hard to achieve for readmissions for HF only, so the ZINB was used for that outcome. The best-performing model for the number of readmissions was a NBH model. Convergence problems prevented the ZINB model from being fitted.

To simplify, we present the results for all-cause readmissions only. Figures 4 and 5 compare the actual with the predicted number of bed-days and readmissions under various model assumptions.

FIGURE 4.

Actual versus predicted number of all-cause emergency readmissions within a year of discharge using different models (ZINB did not converge). NB, negative binomial; P, Poisson; PH, Poisson hurdle; ZIP, zero-inflated Poisson.

FIGURE 5.

Actual versus predicted all-cause emergency bed-days (capped at 12 for presentation) within a year of discharge using different models. NB, negative binomial; P, Poisson; PH, Poisson hurdle; ZIP, zero-inflated Poisson.

Defining buckets using bed-days, the significant predictors for membership of the highest resource-use bucket (fifth) were almost all the same for those for the two highest resource-use buckets. Predictors for the highest resource use were similar to those for the first readmission within 30 days given above, with the exceptions of gender (females had raised odds only for the bucket), very old age (raised odds only for readmission), defibrillator implantation (raised odds only for readmission) and LOS (raised odds for any stays of a week or more compared with 1–6 days for the bucket, whereas only stays of a month or more had raised odds for readmission).

Comparing results for the count models is more complicated because the best-fitting models yielded a separate set of coefficients for zero from that for non-zero bed-days or readmissions. Table 14 gives the incident rate ratios from the hurdle models alongside the ORs from the 365-day readmission model, with all-cause hospitalisations in each case. Many results were consistent. Deprivation, previous OPD non-attendance and living alone were all associated with higher odds of one or more readmissions, a greater number of readmissions and more bed-days; previous selection for CABG showed reduced risk of all three. Older age was associated with higher odds of readmission and lower odds of no readmissions or bed-days. In the hurdle models for patients with at least some future inpatient activity, however, older age did not predict extra admissions but did predict more bed-days. This may be interpreted as older patients having longer stays when they are admitted but not being admitted more often than those aged in their 60s.

Several factors showed a stronger association with the readmissions outcomes than with future bed-days. These included previous defibrillator implantation, ischaemic heart disease, mental health conditions, diabetes, peripheral vascular disease and chronic lung diseases. Long LOS was associated with more bed-days but not readmission; indeed, a very long index LOS was associated with lower odds of readmission.

How well do such models perform for various outcomes?

The best-performing models were for mortality for the procedure groups considered. In general, discrimination (c-statistic) was high for mortality, low for first readmission and RTT and intermediate for other readmission measures and missed OPD appointments. Poor discrimination can be caused by risk adjustment that lacks key variables and by data noise, but it can also be due to variations in quality of care. Calibration was often a problem, particularly for the diagnosis groups, mainly because of overprediction for low-risk patients. Recalibration is advisable for a number of patient groups (see the section on machine learning methods in Chapter 4). HES lacks some important physiological variables to give as accurate prediction for individual patients as more detailed clinical databases, though previous work has found that augmentation with just a few items, such as creatinine, can be sufficient. 89 In risk-adjustment modelling for benchmarking units, the impact of miscalibration on standardised mortality ratios is modest for a typical hospital, as we showed for ACS admissions,67 though it will vary by hospital.

A key difference between risk prediction for an individual patient and risk adjustment for comparing units is that much greater accuracy in the prediction is in general required for the former than for the latter. Some of the differences in casemix can be expected to ‘average out’ when aggregating to unit level. Also, if the casemix does not vary by unit, then risk adjustment will not be necessary. For prediction when used for bedside clinical management, discrimination is generally more important than calibration, whereas the reverse is true for risk adjustment. One reason for this is that a low c-statistic has three principal causes: poor data quality, the lack of key patient factors in the database and variations in the quality and delivery of care. Whereas data quality can be assessed, there is no easy way of distinguishing between the other two. Readmission models, as we and many others have found, often have c-statistics in the region of just 0.60, whereas for mortality they can exceed 0.8 or even 0.9. Patient factors that explain a fair proportion of the variation in mortality can be less important or even insignificant in readmission models, where factors such as health-seeking behaviour that are not captured routinely in databases contribute more. We suspect that a fair amount of the unexplained variation for readmission is nonetheless due to differences in the delivery of care both before and after discharge, but it is not possible to quantify these using HES.

How should one adjust for comorbidity?

Our systematic review revealed the importance of calibrating the comorbidity weights or scores to the database used rather than relying on published weights from another country. Our analysis using HES suggested that the Elixhauser set plus dementia with our amended codes performed best. Consideration should be given to including interaction terms with age because this improves the fit. We also tried using polynomials to incorporate the time between previous admissions and the index one for HF patients, a group with high use of NHS services. This approach worked better than simply counting the number of previous unplanned admissions, though the impact was far from spectacular.

When predicting future inpatient activity, is the 30-day all-cause readmission measure sufficient?

As others have found, the predictive power of readmission models is generally low with administrative data, within both 7 and 30 days of discharge. Our analysis of the HF cohort using follow-up periods of between 7 and 365 days suggests that 30 days is as good a time point as any in terms of model performance and which patient factors are important predictors of the outcome (with the exception of index LOS). Hospital-level variation as estimated from the hierarchical models was also similar for 7, 30 and 365 days. Comorbidity is common in patients admitted for HF or COPD. This is reflected in the primary diagnosis of their subsequent readmissions; for HF patients, for example, only a third of their readmissions within a year of index discharge were for HF. We found that a number of factors predicted HF but non-HF readmissions and vice versa. More importantly, hospital-level readmission rates for HF did not correlate with rates for non-HF, and national audit process measures correlated with only the former. This suggests that the use of all-cause readmissions is of limited use in quality improvement, though it remains the more relevant measure for patients and for benchmarking.

Our count and bucket models of subsequent inpatient activity for the HF cohort were harder and took longer to fit and, as many predictors were common across outcomes, only added some extra useful information. For instance, it was notable that a number of the predictors of the fact of readmission, including a number of comorbidities, showed weaker associations with the number of future bed-days.

What do machine learning methods offer in this context that logistic regression does not?

The machine learning methods that we tried here are popular but did not offer noticeably better prediction or adjustment with these data and outcomes than LR as measured by discrimination (c-statistics). They were more difficult to implement and took longer to run, particularly the SVMs. The advantage of ANNs lay in their superior calibration, with the standard logistic link function used in regression often found to be a fairly poor fit. However, we also improved the calibration of LR using an extra step, which would be useful for risk adjustment at the cost of interpretable ORs for each factor included in the model.

Study strengths and limitations

This project has benefited from the use of rich, national data by a team experienced in using them, a selection of advanced statistical methods as well as the standard LR, and a variety of patient outcomes including, but also going beyond the common ones of, death and readmission. We focused on common patient groups and outcomes that are of interest to patients, the NHS and policy-makers.

We considered a large number of patient groups as defined by their primary diagnosis or procedure, but for reasons of brevity and interpretation we have limited the results in Chapter 3 to some illustrative examples. Our analyses could be extended in various ways given in the next section, particularly by repeating our HF and RTT analyses on other patient groups. RTT needs to be derived separately for each index procedure, in consultation with surgical and coding experts following empirical analysis, and is therefore quite a time-consuming process.

We considered only a few of the growing number of machine learning methods, restricting ourselves to those that were likely to be the most appropriate for the administrative data set we were using. Methods such as boosted trees may be worth trying on richer data. For comorbidity adjustment, we focused on the two most widely used and validated indices, Charlson and Elixhauser. The newer but so far much less adopted Holman index is promising, but we were unable to test it to the same degree as the two better-established measures. This is still a developing field, and we have recently heard of a superior German index that has not yet been published.

For prediction of future hospital use in HF patients, we considered a range of count models, but the list was not exhaustive. We did not use the Healthcare Resource Group tariff information, which would give an indication of the financial burden, though some imputation is needed for missing values. These tariffs do not in any case represent actual costs.

With a large number of models to fit, we have focused on the c-statistic and HL test to describe the model fit, together with calibration plots. We have generally split the data into a training and a testing set, though our experience in this project and previous work has found this to be unnecessary with LR. 90 However, we have not assessed such goodness of fit measures such as influential points and outliers. In general, we did not test for or include covariate interactions except between age and comorbidity index in some cases. Such interactions are unlikely to add anything of real value to the models but could be tried.

We have found that discrimination is higher if coding levels are higher,11 but for this project we used all hospitals combined rather than split by coding levels. It would be worth repeating our modelling on US data, for example, as their recording of secondary diagnoses is notably higher, and CMS data have present on admission information in order to exclude complication codes. Likewise, it would be possible to include those hospitals with high levels of coverage and completeness for their A&E records as we did in a separate project for birth records,91 though the effect on the results in terms of reduced sample size and potential bias would need to be assessed.

Finally, in this project we have focused on the quality of HES data, the performance of the risk models and the definition of the outcome measures. There are several other criteria that characterise a good indicator beyond model performance. These include high frequency; variation across providers; consistency over time unless changes in apparent performance are due to changes in quality of care; correct attribution of outcome to provider (a particular issue when patients are transferred between hospitals or, especially for readmission, discharged to community services); and actionability. If the information on performance is not presented in a way that users understand, for example, then the users are unlikely to pay much attention.

Recommendations for future research

There is growing interest in reinventions, and our approach to defining RTT could be applied to other index procedures and specialties. Similarly, it would be interesting to know whether the count models are as useful for other chronic conditions such as diabetes as we found them to be for HF. For readmissions, we considered the first and also the total number, but more sophisticated approaches such as multistate analysis or cluster analysis to look for patterns of activity could be usefully employed.

We have used time-to-event methods only for modelling readmission and investigating patient-level predictors. The methods could be also applied to determining to what extent variation in hospital-level performance is explained by differences in the timing of outcomes. For example, when do the ‘excess’ deaths or readmissions occur at hospitals with high rates compared with those with low rates? Some recent evidence for HF patients suggests that the timing of readmissions within the standard 30-day follow-up period does not vary in Medicare patients,92 but this is not well established with other populations, patient groups or outcomes.

If the A&E records improve in quality, then they could potentially be used both in risk-adjustment models and also for outcome measures. Post-discharge attendances not resulting in admission should be considered alongside those that do. As well as inpatient activity, attendances (and OPD appointments) could be used much more in economic modelling. Reattendance rates could be calculated and explored as an indicator rather than relying on costly casenote reviews.

Dissemination activity

Up to September 2014, we have disseminated various findings from this project as:

1. Systematic review of studies comparing two or more comorbidity indices: paper published by Medical Care in 2012. 20

2. Provider profiling models for ACS mortality using administrative data: paper published by the International Journal of Cardiology online first in 2012 and in print in 2013. 67

3. Project overview and early results of the comparison of methods: invited oral presentation at the International Society of Clinical Biostatistics conference in Bergen, August 2012.

4. Comparison of methods and calibration issues: two oral presentations at the IMA (The Institute of Mathematics and its Applications) Modelling in Healthcare conference in London, March 2013.

5. Oral presentation of results of HF readmission modelling to the Institute of Electrical and Electronics Engineers (IEEE) meeting in Philadelphia, September 2013, with publication in conference proceedings.

6. Poster and short oral presentation of HF readmissions modelling and hospital-level rates at the International Society for Quality in Health Care (ISQua) conference in Edinburgh, October 2013.

7. Classifier calibration using splined empirical probabilities in clinical risk prediction: paper published online first by Health Care Management Science in its special conference issue in February 2014. 50

8. Poster on count models for HF accepted for the Cambridge biostatistics meeting, March 2014.

9. Effect of the readmission primary diagnosis and time interval in HF patients: analysis of English administrative data: paper published by the European Journal of Heart Failure online in July 2014. 93

10. Return to theatre for elective hip and knee replacements: what is the relative importance of patient factors, surgeon and hospital? Paper in press at the Bone and Joint Journal. 94

One more paper has been submitted to journals and is under review.

We are working with relevant charities to decide the best way to publicise our work relating to HF.

Implications for practice and translation of findings

The use of risk prediction and risk stratification models is well established in medical decision-making. Some risk scores are simple enough to be calculated by hand, whereas others need a calculator, smartphone app or other electronic device. Risk-adjusted outcomes at the unit level can be displayed graphically or in tabular form using similar technology, and there are a growing number of websites that allow a prospective patient to choose a hospital this way. Many private companies offer interactive information tools that allow drill-down and stratified analyses, for example to see if a hospital has a higher mortality at the weekend than during the week or if the uptake of services by demographic subgroup matches the demographic distribution of the surrounding population that it serves.

As well as some methodological lessons regarding risk adjustment and prediction, this project has produced some findings that relate to clinical practice. Our analyses of OPD non-attendance and readmission showed clear variations by factors such as age, sex and deprivation that other researchers had previously identified but also the strong association with the number of previously missed OPD appointments for both outcomes. We found that patients who miss appointments go on to have poorer outcomes a year later that are unlikely to be explained wholly by disease status or comorbidity. This suggests that repeat non-attendance be considered a warning sign that closer monitoring is needed.

The second finding relating to non-attendance is that we found that the lowest non-attendance rate at the first appointment following discharge from an emergency admission for one of several acute conditions occurred 6 weeks after discharge. The highest rate occurred within a week or more than 12 weeks after. While we were unable from the data to identify reliably whether or not this first post-discharge appointment had been planned before the admission, the finding does suggest scheduling more follow-up slots for around 6 weeks after discharge in order to reduce non-attendance.

Finally, the fact that we found no significant correlation between HF and non-HF readmission rates and that performance on the national audit process measures correlated mostly with just the HF readmission rates suggests that more focus should be put on these patients’ comorbidities. Good performance and/or low rates of readmission for HF often did not translate into low rates of readmission for other conditions. For quality improvement, we recommend splitting the readmission outcome and monitoring its diagnosis-specific components.

Contributions of authors

Alex Bottle (Senior Lecturer in Medical Statistics) oversaw the project, prepared the HES data extracts, oversaw and contributed to the analysis, and contributed to writing the report.

Rene Gaudoin (Research Associate) performed the ANN, SVM and polynomial analysis and contributed to writing the report.

Rosalind Goudie (Research Assistant) performed much of the HF analysis and contributed to writing the report.

Simon Jones (Professor of Healthcare Management and Policy) performed the random forests analysis and contributed to writing the report.

Paul Aylin (Professor of Epidemiology and Consultant in Public Health) contributed to the management of the project and to the writing of the report.

Publications

Sharabiani MTA, Aylin P, Bottle A. Systematic review of comorbidity indices for administrative data. Med Care 2012;50:1109–18.

Bottle A, Sanders RD, Mozid A, Aylin P. Provider profiling models for acute coronary syndrome mortality using administrative data. Int J Cardiol 2013;168:338–43.

Gaudoin R, Montana G, Jones S, Aylin P, Bottle A. Classifier calibration using splined empirical probabilities in clinical risk prediction. [epub ahead of print 21 February 2014]. Health Care Manag Sci 2014.

Bottle A, Aylin P, Bell D. Effect of the readmission primary diagnosis and time interval in heart failure patients: analysis of English administrative data. Eur J Heart Fail 2014;16:846–53.

Bottle A, Aylin P, Loeffler M. Return to theatre for elective hip and knee replacements: what is the relative importance of patient factors, surgeon and hospital? Bone Joint J 2014; in press.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health.