Improving risk adjustment in the PRAiS (Partial Risk Adjustment in Surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomes

Why is mortality monitored and why should we try to adjust for risk?

Approximately 3500 children aged < 16 years have heart surgery each year in the UK. 1 Although overall 30-day survival is over 97%, congenital heart disease (CHD) is a spectrum of disorders, and the more serious and complex abnormalities are an important cause of childhood mortality, morbidity and disability.

Since 2000, all UK specialist centres have contributed procedure data to the National Congenital Heart Disease Audit (NCHDA), one of seven national audits within the National Institute of Cardiovascular Outcomes Research (NICOR). Life status is independently obtained from the Office for National Statistics (ONS); when no ONS tracking is available, discharge status is used instead and checked with individual units. Centre-specific mortality outcomes for individual procedure categories have been published online by NICOR since 2007. 2

Outcomes following children’s heart surgery have long been the subject of clinical, regulatory, media and public scrutiny in the UK. Complex surgical procedures on extremely small hearts are among the most technically challenging and resource intensive in the field. In the UK, past events, public inquiries and plans to reduce the number of centres performing such surgery all provide a rich source of backstories and a level of public awareness that make this specialty ripe for comment and journalism. 3–6 Mortality has, understandably, always been the dominant reported outcome and is often perceived as a straightforward measure of performance by the media and the public.

There is a reasonable expectation from the paediatric cardiac profession that, in publishing their outcomes, audit will be ‘fair’ to clinical teams. That is, the reporting of outcomes should take into account the hugely diverse set of diagnoses and comorbid conditions that patients present with, the wide range of surgical procedures performed, the differences in case mix between centres and the impact of relatively small numbers of patients on what can reliably be inferred from data. These characteristics of the specialty make risk adjustment in the presentation of outcomes analysis essential, but they also make it very difficult to achieve.

Until recently, the only risk adjustment method easily available to NICOR was to report mortality within procedure categories, thus partially accounting for variations in case mix. One example is the arterial switch operation, a definitive surgical procedure to repair the heart when the main artery and vein connections to the heart are the opposite of what they should be. The structure of such hearts can vary drastically, from a ‘simple’ isolated inversion of the great arteries to a complex physiology with several accompanying abnormalities. A child may also have additional non-cardiac problems and/or an underlying chromosomal abnormality. These sorts of issues highlight the importance of incorporating case mix into interpreting observed short-term outcomes following heart surgery in children, beyond consideration of the procedure performed. 6

Why did we develop the PRAiS risk model?

There have been several models aiming to incorporate risk assessment into outcome measures. In the early 2000s, the Risk Adjustment for Congenital Heart Surgery (RACHS-1) score was introduced. 7 This method involves gathering a panel of experts who assign patients to one of six predefined risk categories on the basis of the presence or absence of individual diagnosis and procedure codes. Cases with combinations of cardiac surgical procedures are placed in the category of the highest risk procedure. Around the same time, the Aristotle tool emerged to evaluate quality of care based on the complexity of the operation and on specific patient characteristics. 8 Also based on a review by a panel of experts, Aristotle gives a precise score for the complexity of 145 specific paediatric cardiac operations. Both RACHS-1 and Aristotle are examples of ‘consensus-based’, subjective, risk stratification tools, essentially meaning that experts have sat down and decided how a particular operation compares with others in terms of risk, which for these systems is usually considered synonymous with complexity. Although certainly valuable and useful, these methods have started to give way to recent empirical approaches, based on the emerging availability of databases incorporating the outcomes of tens of thousands of patients. The Society of Thoracic Surgeons-European Association of Cardiothoracic Surgery (STS-EACTS) score, or STAT-score, introduced in 2009, was based on data from > 75,000 paediatric cardiac surgery procedures performed between 2002 and 2007 in Europe and North America and was an important step towards monitoring mortality, as clinical teams could benchmark current outcomes against achieved outcomes in the recent past. 9

None of the above risk models is easy to use in the routine monitoring of outcomes using UK national audit data, none was calibrated on UK data and all mainly used procedure information. In our previous National Institute for Health Research (NIHR) Health Services and Delivery Research-funded project (09/2001/13), which ended in 2011, we (VT, KB, MU, CP and SC) developed a new risk model for 30-day outcome after paediatric cardiac surgery, referred to as PRAiS (Partial Risk Adjustment in Surgery). 10,11 This empirical model was developed for the purpose of routine local in-house monitoring of risk-adjusted outcomes within UK paediatric cardiac surgical units, and incorporated not only the procedure but also cardiac diagnosis, the number of functioning ventricles and age category (neonate, infant or child), as well as continuous age, continuous weight, presence of a non-Down syndrome comorbidity and whether or not surgery was performed on cardiopulmonary bypass. The model was developed using data from all paediatric cardiac surgery procedures performed in the UK between 2000 and 2010, and in validation it compared well with RACHS-1, Aristotle and the STS-EACTS score. The intention was that, by facilitating local routine monitoring, units could regularly examine their recent outcomes (with a time lag of 30 days), compared with the annual results published by NICOR, which could represent a time lag of up to 18 months.

What happened with PRAiS after the end of the first National Institute for Health Research grant?

Following the development of the PRAiS risk model, several units expressed an interest in piloting the model for the in-house routine monitoring of outcomes. In 2012, the analytical team at the University College London (UCL) Clinical Operational Research Unit (CORU) (CP, MU and SC) worked with Great Ormond Street Hospital, Evelina London Children’s Hospital and Glasgow Royal Hospital for Sick Children in a pilot study to implement the new risk model. We developed prototype software to allow units to use their own routinely collected data to produce variable life-adjusted display (VLAD) charts for 30-day mortality after children’s heart surgery after partial risk adjustment with PRAiS. 12,13 The software was codesigned with the clinical teams to be robust and easy to use and to produce output that was helpful for team discussion of outcomes. The advantage of using VLAD charts is that they show the accumulation of outcomes over time, allowing trends in outcomes (both negative and positive) to be spotted quickly and discussed.

The pilot study was successful, with the pilot units keen to continue using the software and other units also showing an interest. 14 As a result, the team at UCL CORU decided to further develop the software that implements PRAiS into a package that could be rolled out across the UK. This involved including comprehensive error checking of any entered data, further work on the software in response to feedback from pilot units, the development of user manuals and the recalibration of the PRAiS risk model on all national data from 2007 to 2010, in part to address an observed imbalance in neonatal outcomes in the original development and validation data sets. 11 The software to implement PRAiS was licenced by UCL Business in April 2013 and licences were purchased by NICOR with funding from NHS England for all English hospitals that perform children’s heart surgery and for the NICOR congenital heart audit itself. All units that contribute data to NICOR’s congenital heart audit now possess licensed copies of the PRAiS software.

In the following 6 months, all English hospitals downloaded the software to implement PRAiS, and it was also used by NICOR to include risk adjustment in the reporting of national outcomes for the first time. 1 An important part of the software licence was the inclusion of a half-day consultancy visit to units, to discuss not only the practical use of the software, but also, and more importantly in the analytical team’s view, the caveats of using risk adjustment to monitor outcomes. As the PRAiS model began to be used to monitor outcomes both locally within units and nationally as part of NICOR’s audit, data quality for information previously collected but not actively used (such as comorbidity and diagnosis codes) improved rapidly (and retrospectively, as recent data were revisited by hospitals back to 2009). In particular, the proportion of surgical episodes with a recorded comorbidity (excluding Down syndrome) doubled from 15% (2000–10 original data set) to 30% of cases. It also appeared that national outcomes between 2009 and 2012 had improved since the time period that the PRAiS risk model was calibrated on (2007–10). 1 In July 2013, at NICOR’s request, CORU analysts (CP and SC) recalibrated the risk model on the 2009–12 data set (but left the risk factors identified in the original development process unchanged) and updated the software to implement the model. 15 These changes are documented on UCL CORU’s website (www.ucl.ac.uk/operational-research/AnalysisTools/PRAiS).

PRAiS is now being used in three main ways.

1. Local quality assurance and improvement, for which it is used every month to monitor survival trends, and any up or down ticks can be discussed as a team. It is also used to audit particular population cohorts or operations types (e.g. all shunt operations or all neonates).

2. National quality assurance to ensure that no hospital has a survival rate that is potentially concerning because it is much lower than expected using PRAiS.

3. National audit annual benchmarking between hospitals and also against historic national performance (e.g. is one hospital doing markedly better and can the clinical community learn anything from those processes? Are national outcomes overall improving compared with historical data and, if so, do we understand why?).

What prompted this project to update the model?

Data completeness and quality of comorbidity information was poor in the original 2000–10 data set that was used to develop PRAiS. Although we explored different methods for incorporating information about different types of comorbidity and multiple comorbidity as part of our original project, none of the models using such methods proved to be robust. 10 Faced with the choice of excluding comorbidity entirely as a risk factor or using a very crude measure of comorbidity as a ‘yes/no’ variable, we chose the latter. This was because the definite presence of at least one non-Down syndrome comorbidity was significantly associated with mortality in multivariable analysis, because comorbidity was considered extremely important in risk adjustment by clinical collaborators (VT and KB) and because it was hoped that inclusion of the crude risk factor would drive future improvement in data quality concerning comorbidities. 10,11 In addition, other international risk adjustment systems have started to use comorbidity information as a risk factor, in recognition of its importance. 16,17

Our consultancy visits to English hospitals with the PRAiS software are now complete. Feedback on the software to implement PRAiS and the usefulness of the VLAD charts was very positive; however, a consistent concern, expressed during almost all visits, was the treatment of comorbidity within the PRAiS risk model, highlighting its importance as a risk factor. On the one hand, this justified the inclusion of comorbidity within PRAiS, but, on the other hand, it emphasised the need to revisit how comorbidity is incorporated within the model. The improvement in national audit data quality since 2009 (noted above) means that by 2015 there were enough data with better-quality comorbidity information for us to explore a more sophisticated inclusion of comorbidity within the PRAiS risk model.

We proposed to return to PRAiS model development to improve comorbidity information and to improve the risk adjustment achieved through the use of PRAiS in local and national audit.

Public understanding of mortality outcomes following children’s heart surgery

The public and media response relating to the cessation of children’s heart surgery in one unit in 2010, the brief suspension of heart surgery in another in 2013 and other recent coverage all prove the immense public interest in understanding what happens to children after heart surgery and, in particular, fears about what deaths after heart surgery mean about the care provided within units. 18–27 The UK is one of the few countries that publish mortality outcomes after children’s heart surgery, and NICOR’s results are, understandably, used by journalists, politicians and the public to make judgements about whether or not heart surgery is ‘safe’. 28 Such judgements are fraught with difficulties and are very stressful both for the families of children who have heart disease and for the clinical teams treating these children.

Although the PRAiS risk model was originally developed for the local in-house routine monitoring of outcomes, it has also been adopted by the NICOR’s congenital heart audit for reporting annual outcomes for each UK centre. 1 Using risk models for comparative audit is fairer than using raw mortality, but risk adjustment does not in and of itself make comparisons ‘fair’. 14,15 Although comparing the number of deaths seen in different units seems straightforward, risk adjustment or not, unfortunately it is not that simple (whether in congenital audit or elsewhere). 15,29,30 We have written on the difficulties of interpreting comparative mortality data using PRAiS, and the NICOR Congenital Audit has also written resources for the public on its public portal; however, these sources are not easily found without prior knowledge of their existence and are not necessarily easily digested by the non-expert. 14,15,31,32

We believed that there was a real need to, first, develop better resources for the public about how to interpret evidence on mortality following children’s heart surgery and, second, to disseminate these resources widely. We had the enthusiastic support of charities the Children’s Heart Federation (CHF) (a user group for families of children with heart disease) and Sense about Science (a charity dedicated to the public understanding of science and evidence). The second strand of our project was (originally) to develop a video animation on the interpretation of mortality outcomes for the public, with help and user input from both charities.

Aim 1: updating the PRAiS risk model for 30-day mortality following paediatric cardiac surgery

Aim 1 was to improve the PRAiS risk model for 30-day mortality following paediatric cardiac surgery by incorporating more detailed information about comorbid conditions.

The objectives to achieve aim 1 were to:

• explore the relationship between the existing six comorbidity groupings (defined as part of the original risk model development process but not included in final model) and mortality, both in the presence and in the absence of other risk factors, and the consequent potential impact on the robustness of the PRAiS risk model (1.1)

• decide on the suitability of existing comorbidity groups in the light of initial exploratory analysis, devise any necessary modifications and consider options for changing current groupings of specific procedure and diagnosis categories, with expert input from clinicians and data managers from multiple centres (1.2)

• modify the existing mapping of individual comorbidity codes to broader comorbidity categories and to a single 30-day patient episode with expert clinical input (1.3)

• explore trade-offs in reducing detail in existing risk factors (e.g. specific procedure categories) to incorporate new comorbidity categories within the PRAiS risk model while maintaining a robust calibration (1.4)

• calibrate a new PRAiS risk model, after deciding on the final risk factors by consideration of statistical goodness of fit and clinical face validity (1.5)

• update the software that implements PRAiS with the new parameterisation (1.6).

Aim 2: developing a public website to communicate how PRAiS is used to monitor children’s heart surgery

Aim 2 was to develop, test and disseminate online resources for families affected by CHD in children, the public and the media to facilitate the appropriate interpretation of published mortality data following paediatric cardiac surgery.

The objectives to achieve aim 2 were to:

• confirm our understanding of the current and planned presentations of mortality outcome data by the NICOR congenital audit (2.1)

• coproduce a web tool with patient groups and interested users that includes an explanatory website, an interactive animation and a short video to facilitate the interpretation of mortality outcomes (2.2)

• undertake a formative mixed-methods evaluation of the web tool to strengthen the final outputs (2.3)

• disseminate the developed material via the CHF and Sense about Science and to evaluate the usability and efficacy of the final web tool as an aid to the public understanding of outcome data; in addition, to share the material with other charities such as the British Heart Foundation and also with the NICOR Congenital Audit (2.4).

Defining episodes of surgical management for analysis

Using the same approach as for PRAiS 1, a ‘30-day episode of care’ was the unit of analysis. A 30-day episode starts with the first surgical procedure on a patient. 10 This episode is then assigned an outcome of alive or dead, according to the vital status of the patient 30 days after this first surgical procedure. Any further surgical procedures that the same patient underwent within 30 days of this first procedure were not included in model development. The next procedure recorded for the same patient > 30 days after the first surgical procedure was treated as the start of a new 30-day episode.

Examples of how the 30-day episode was allocated are shown in Figure 1.

FIGURE 1.

Illustration of how the ‘30-day episode’ was allocated: example patient histories.

Initial removal of records outside the scope of the project

From the main data set, we initially removed all records in which the patient was > 16 years old at the time of the procedure; any procedure performed before April 2009; any non-cardiac procedures; records relating to catheter and ‘other’ type (i.e. non-surgical) procedures; and non-cardiac or trivial/minor procedures.

The PRAiS 1 risk model had also excluded ‘hybrid’-type procedures, as these were not considered surgical cardiac operations. Since then, a new hybrid procedure has evolved to treat the most serious congenital heart condition, hypoplastic left heart syndrome (HLHS). On the advice of the expert panel and the NCHDA at the first expert panel meeting, HLHS hybrid procedures were also included in our model development. All other hybrid-type procedures were excluded.

Oxford’s John Radcliffe Hospital stopped performing paediatric cardiac surgery in 2010. Following discussion with the expert panel, it was decided that procedures performed at this unit should be excluded, as they would not be indicative of future outcomes.

To construct 30-day episodes, it was essential to identify records within the data set relating to the same individual patient. To do this, we initially linked all procedure records that had the same pseudonymised NHS and/or the same pseudonymised hospital number.

We then checked for inconsistencies between records now identified as being for the same patient. We first identified all records for an apparently single patient for whom sequential ages at operation were inconsistent with procedure dates. The manual inspection of procedural, diagnostic and other demographic factors for these patient records determined whether the patients were, in fact, different, or whether a mistake had been made in the recording of date of birth or procedure dates. However, neither episode allocation nor 30-day life status was affected in those cases in which it was determined that records apparently relating to the same patient were, in fact, for different patients. For patients for whom there appeared to be an error in the date of birth or date of procedure, we used indicators such as weight and sternotomy sequence instead of the procedure date to infer the correct order and the timing of procedures for determining episode allocation.

This reduced data set contained 22,917 uncleaned procedure records, corresponding to 18,836 unique patients.

Removal of duplicate records

Some records appeared to be duplicates based on systematic comparison of pseudonymised patient NHS number or hospital number, year and month of operation, age-specific procedure, procedure type and procedure and diagnosis codes included. There is a plausible mechanism for record duplication: if an institution tries to update an individual record within NCHDA by changing either the patient’s hospital identification number or the procedure date (both of which are key fields), a second, duplicate, record will be created instead of the original record being overwritten. We used a formal protocol (see Appendix 1) to determine possible duplicates, and to determine which record of each pair to retain, based on the completeness and plausibility of the data contained in the records. Our identification of duplicates was checked with the clinical co-applicants and shared with the NCHDA. A total of 59 duplicate records in the main data set were removed from the analysis.

We then proceeded to allocate 30-day surgical episodes of care as described in Figure 1, resulting in 21,840 episodes.

Allocation of 30-day life status

The NCHDA receives independent life tracking from the ONS. However, a patient is not recorded as dead in ONS until any ongoing inquest is complete, which can lead to delays in updating the life status of a minority of children who have died. The ONS life status in the data set was the most recent life status at data extraction (2015), and if a hospital discharge life status was ‘dead’ and the ONS life status was ‘alive’, we used the hospital discharge status, consistent with current practice at NCHDA. We note that the only records likely to be affected by the ONS inquest issue would be children operated on in 2014–15 with a discharge status of alive who then died following discharge, which is unlikely to affect more than a very small number of episodes, especially as we received the 2014–15 data in March 2016, allowing almost an extra year for any inquest to be completed.

Note that ONS tracking is not available for children from overseas or for those who do not have a NHS number for any other reason. The NCHDA includes these children in its public reports and so these children must also be included in the development of the PRAiS risk model, using the same method for allocating life status.

The overall algorithm we use for assigning the episode life status as shown in Figure 3 and is consistent with the method used in the NCHDA.

FIGURE 3.

Allocation of 30-day status.

In addition, subsequent records for the same patient were used to check the 30-day life status. For example, if a patient had a further operation at least 30 days after their previous procedure, they were considered alive at 30 days for the initial procedure, even if this was not supported by discharge or life status.

When the life status was not known at 30 days post procedure, the discharge status prior to 30 days was used as a proxy for 30-day life status, which is consistent with NCHDA practice and the PRAiS 1 process. This affected 3165 episodes (14% of records) in the main data set (99% of which correspond to patients from outside England and Wales for whom no ONS tracking is available and who were discharged alive before 30 days).

We were able to allocate the 30-day life status for all but two surgical episodes. These two records were excluded from the analysis (see Figure 2), leaving us with a final data set comprising 21,838 30-day surgical episodes.

Data cleaning of key variables

The NCHDA data set is widely recognised as being of extremely high quality (arguably the best of its type in the world) and the data quality has increased steadily over time; however, given the sheer size of the data set and the complexity of paediatric cardiac diagnoses and surgery, it is inevitable that there were some errors and anomalies in the data. The analytical team spent considerable amounts of time understanding the data set and, when necessary, cleaning it.

Age and weight fields

Some of the ages, weights and combinations of age/weight recorded for an episode were biologically implausible. When a weight was recorded as identically zero, it was treated as missing data. There were five patients in the main data set whose weight was determined as having been recorded in grams instead of kilograms (all children in the first month of life whose weights were recorded as > 600 kg). These weights were converted manually into kilograms.

The data set was then subdivided into the same 23 age bands that were used for the development of PRAiS 1 (narrower at younger ages), and the means and standard deviations (SDs) of the recorded weights within each band were calculated. We then calculated the weight-for-age z-scores for each episode within the data set. Episodes in the main data set, with missing weights (n = 3) or a z-score of either < –5 or > 5 (considered infeasible) (n = 23, shown in blue in Figure 4), were assigned the mean weight corresponding to their age band. The z-score thresholds were chosen after consultation with the expert panel and an independent statistician, and were chosen to ensure that patient weights outside these boundaries were definitely biologically implausible.

FIGURE 4.

Adjustments made to weight.

Finally, a further 10 episodes had their weight manually assigned to the mean weight for age following inspection by a clinician (KB) on the grounds of biological implausibility (all considered too low to be feasible). These episodes are shown in light green in Figure 4.

Figure 4 shows the distribution of age and weight in the main data set prior to adjustment with the identified anomalous weights highlighted.

Specific procedure information

The specific procedure algorithm was first developed by the NCHDA steering committee over 10 years ago to combine the individual EPCC codes submitted to NCHDA as part of a patient’s procedure record to define a standard procedure category (‘a specific procedure’). The algorithm defines the series of EPCC codes that may be included to identify an individual operation, and in a proportion of operations in which the definition may be more complicated it also defines the list of EPCC codes that must be excluded. The algorithm contains a hierarchy that ranks the recognisable procedures in order, with the most complex at the top (grouped first) and the least complex at the bottom (grouped last). The algorithm has been refined and improved by the NCHDA steering committee year on year to match evolving practice, such that the definitions of each operation remain tight and consistent. There are currently 49 surgical specific procedure categories.

We used the most recent specific procedure algorithm available to determine specific procedures for all surgical episodes in the main data set. By the end of the model development, this was the Specific Procedure Algorithm v5.05 from May 2016 (https://nicor4.nicor.org.uk/chd/an_paeds.nsf/vwcontent/technical%20information?opendocument). In recent years a new procedure to treat HLHS, called the HLHS hybrid approach procedure, has become increasingly common and is expected to become more widespread in the future. Although this procedure is not currently included in the specific procedure algorithm, the expert panel felt that, owing to the expected increase in its use and the high risk of the procedure, it should be included in PRAiS 2. This inclusion of HLHS hybrids was then discussed and affirmed by the NCHDA steering committee. The expert panel developed a coding addition to the specific procedure algorithm to include this procedure, taking into account how the procedure had been coded in the past and how it should be coded prospectively.

One of the main aims of the update for the PRAiS model was to balance an increase in diagnosis and comorbidity information included in the model with a reduction in the number of degrees of freedom used by procedural information. The increase in the number of specific procedures included in the specific procedure algorithm, from 41 to 49, since PRAiS 1 was developed made this problem more acute. This balance was discussed at the expert panel meeting in July 2015. The analysts (CP and LR) demonstrated how risk model performance could be maintained with some illustrative broader specific groupings offset by increased comorbidity and diagnostic information. The expert panel agreed that the analytical team would investigate different methods of grouping specific procedures into broader procedural risk groups and share the most statistically promising groupings at the next expert panel meeting.

The analytical team initially grouped specific procedures by the ages at which they occurred and the risk associated with them using classification and regression tree (CART) analysis. 34 These groups were then adjusted iteratively by the analysts (LR and CP) after consultation with the clinical expert panel and testing of the stability and performance of the resultant risk models under cross-validation, to ensure that the groups were clinically valid and that they demonstrated consistently good performance. The expert panel provided advice on procedures for which the observed mortality was not considered representative of the risk or about instances when procedures that were qualitatively different should not be combined in a single risk group.

Procedure type

In PRAiS 1, only bypass and non-bypass surgical procedures were included in the model. The relatively new HLHS hybrid approach procedure involves both surgical and catheter techniques, so it did not strictly fit into a bypass/non-bypass procedure type category. 35 How to allocate a procedure type for HLHS hybrid procedures was discussed with the expert panel.

Diagnostic information

The diagnosis allocated to episodes from the individual recorded EPCC codes was a slightly modified version of the scheme used by coauthors KB and SC in their work on longer-term infant heart surgery outcomes, which itself was adapted from the scheme originally developed by coauthors for the PRAiS 1 risk model. 36,37 The diagnosis allocation algorithm results in 29 hierarchical diagnosis groups, and full details are provided in Report Supplementary Material 2. In short, for each episode, each recorded diagnosis code is mapped uniquely to one of the 29 primary diagnosis groups, and then the highest ranked (most serious) primary diagnosis is allocated as the overall primary diagnosis group for that episode. In PRAiS 1, these primary diagnosis groups were then further allocated to one of three broad diagnosis groups used in the final model. 10

The reduction in the number of degrees of freedom used by procedure information meant that we could explore increasing the number of diagnosis risk groups used for PRAiS 2. New diagnosis risk groups were developed in a similar way to, and alongside, the broader procedure risk groups, using CART analysis to initially group diagnoses by the age at which patients with that diagnosis had procedures, and the risk associated with that diagnosis. These groups were then adjusted iteratively by the analysts after consultation with the expert panel and testing of the stability and performance under cross-validation, to ensure that the groups were clinically valid and were not unduly affected by the apparent risk shown in the data set if the expert panel did not feel that this reflected the true risk of the diagnosis. We also took into consideration when there was significant correlation between procedures and diagnosis (e.g. some procedures are only performed for a single diagnosis) and strived to develop parsimonious groupings.

Definite indication of univentricular heart allocation

An episode was determined to relate to a patient with a UVH structure if they had a diagnosis code implying a UVH or if they had a procedure that would be performed only on a patient with a UVH. A patient was thus assigned to the UVH category if they met at least one of the following criteria.

• Had one of the following overall diagnosis groups:

  • HLHS

  • functionally UVH

  • pulmonary atresia and intact ventricular septum (IVS).

• Had one of the following EPCC diagnosis codes:

  • 010119 – double outlet right ventricle: with non-committed ventricular septal defect (VSD)

  • 159060 – ‘failed’ Fontan-type circulation

  • 010309 – atrioventricular and/or ventriculoarterial connections abnormal.

• Had one of the following specific procedures:

  • Norwood procedure

  • HLHS hybrid approach procedure

  • Fontan procedure

  • bidirectional cavopulmonary shunt.

• Had one of the following EPCC procedure codes:

  • 121000 – Norwood-type procedure

  • 123001 – Fontan-type procedure

  • 123005 – total cavopulmonary connection (TCPC) using extracardiac inferior caval vein/pulmonary artery conduit with fenestration

  • 123006 – TCPC with fenestrated lateral atrial tunnel.

  • 123013 – Fontan procedure with atrioventricular connection

  • 123027 – Fenestration of Fontan-type connection

  • 123028 – Fontan-type connection without fenestration

  • 123031 – takedown of Fontan-type procedure

  • 123032 – Fontan procedure with direct atriopulmonary anastomosis

  • 123034 – conversion of Fontan repair to TCPC

  • 123037 – Fontan-type procedure revision or conversion

  • 123050 – TCPC

  • 123051 – TCPC with lateral atrial tunnel

  • 123054 – TCPC using extracardiac inferior caval vein/pulmonary artery conduit

  • 123056 – takedown of TCPC

  • 123111 – bidirectional superior cavopulmonary (Glenn) anastomosis

  • 123115 – hemi-Fontan procedure

  • 123144 – bilateral bidirectional superior cavopulmonary (Glenn) anastomoses

  • 123172 – superior caval vein to pulmonary artery anastomosis.

Comorbidity information

During the development of PRAiS 1, coapplicant KB devised a scheme whereby EPCC comorbidity codes were placed into one of the following groups:

• acquired comorbidity

• congenital non-Down syndrome comorbidity

• prematurity

• Down syndrome.

The binary group of ‘any indication of a non-Down syndrome comorbidity’ was included in the PRAiS 1 model. The original four groupings developed by KB formed the basis of the expert panel’s discussion on how to add more detailed comorbidity information to the PRAiS risk model. Other particular additional risk factors considered by the panel were the most common EPCC codes in the data set and those highlighted by Jacobs et al. 16,38 in the US context, not all of which had been included in the above categories.

Other categories of additional risk factor that were considered were acquired heart conditions not directly related to a patient’s primary congenital cardiac diagnosis and indicators that a patient was severely ill (e.g. shock).

Possible ways to include more comorbidity information that were discussed included:

• a simple count of the number of non-duplicate EPCC comorbidity codes recorded

• definite indication of different comorbidity/additional risk factor groups

• a count of the number of instances in each of these groups

• common combinations of types of additional risk factor.

We also explicitly considered the potential for ‘gaming’ by including risk factors. A candidate risk factor could be open to gaming if additional risk could be counted in the calculation based on factors included in the data that did not necessarily reflect an actual increase in risk. This would lead to an artificially high predicted risk, leading to a more favourable conclusion of predicted versus actual 30-day mortality and potentially allowing a poorly performing hospital to avoid investigation as a result of national audit reporting. A possible way that a candidate risk factor could be ‘gamed’ would be if several different diagnostic or comorbidity codes relating to the same condition were counted as making individual contributions to the risk for that episode, or if risk factors that are present in many patients with CHD without necessarily leading to an increased risk of 30-day mortality, such as cyanosis, were included in the model.

Age and weight

The association of both age and weight with mortality is non-linear. We investigated several methods of accounting for this non-linearity. In PRAiS 1, a continuous linear age and a categorical age band variable (neonate, infant and child) were included as risk factors in the model alongside continuous weight. 10 However, this inevitably resulted in relatively large, unrealistic changes in predicted risk at age band boundaries.

For PRAiS 2, with encouragement from the expert panel, we explored using cubic splines and fractional polynomials to model the non-linear association of age and weight with mortality. Cubic splines are piecewise third-order polynomials that pass through a set of ‘knots’. 39 Fractional polynomials are continuous functions that can contain repeated, integer, non-integer and logarithmic terms. 40

During discussion about comorbidities and additional risk factors, a low weight (< 2.5 kg) risk factor was also discussed as a possible proxy for prematurity.

Field completeness

Since PRAiS 1 was developed, the quality of the data in the NCHDA data set has improved greatly. Missing data were not a significant problem in model development. Data completeness is summarised in Table 1.

The increase in the amount of comorbidity information included in the NCHDA data is shown in Figure 5, while the proportion of procedures in the data set that have the explicit ‘no pre-procedural risk factors’ comorbidity code has stayed reasonably stable over the years.

FIGURE 5.

The completeness of the comorbidity field over time for surgical procedures.

Specific procedures

We show the mortality rate by specific procedure category in Figure 6.

FIGURE 6.

Mortality rates (95% CIs) based on 30-day status for specific procedures within the main data set (n = 21,838). The dotted green line indicates the overall average mortality rate (2.5%). ASD, atrial septal defect; CI, confidence interval; DORV, double outlet right ventricle; PAPVC, partial anomalous pulmonary venous connection; REV, Réparation à l’Etage Ventriculaire; RV–PA, right ventricle to pulmonary artery; TAPVC, total anomalous pulmonary venous connection.

Diagnosis

We show the mortality rate by diagnosis group in Figure 7.

FIGURE 7.

Mortality rates (95% CIs) based on 30-day status for diagnosis within the main data set (n = 21,838). The dotted green line indicates the overall average mortality rate (2.5%). ASD, atrial septal defect; AVSD, atrioventricular septal defect; CI, confidence interval; DORV, double outlet right ventricle; TAPVC, total anomalous pulmonary venous connection; TGA, transposition of the great arteries.

Age and weight

Figure 8 shows the higher incidence of death within 30-day surgical episodes for younger patients and for patients with lower weight.

FIGURE 8.

Weight vs. age scatterplot for episodes in the main data set (n = 21,838).

Procedure type and univentricular heart status

The frequency of episodes by UVH status and procedure type is given in Table 2, along with observed mortality rates.

Additional risk factors

The frequency of episodes by additional risk factor and number of additional risk factors are given in Tables 3 and 4, respectively, along with the observed mortality rate.

Benefit of the expert panel

The clinical validity of the risk factors to be included in PRAiS 2 was discussed at length and in considerable detail by the expert panel, both during and between expert panel meetings. In particular, the clinical knowledge of the panel was combined with the evidence from the data to determine the groups of EPCC codes to be included as comorbidities and additional risk factors in the model, how specific procedures and diagnosis should be allocated to broad risk groups and how developments in treatment and outcomes should be incorporated into the updated model. This discussion continued throughout the modelling process.

The process adopted for constructing the final risk model thus consisted of the iterative use of the steps described in the rest of this section.

Multivariable logistic regression

The final set of risk factors included in the model was determined based on a combination of their ability to discriminate between high- and low-risk patients consistently over subsets of the data, their clinical face validity and their ease of use prospectively. All multivariable logistic regression analysis was conducted using the statistical software package Stata, release version 13.0 (2013; StataCorp LP, College Station, TX, USA). The regression output included the risk factor coefficients, the area under the receiving operator characteristic (AUROC) curve, Akaike information criterion (AIC) and the predicted risk for each episode. The AUROC curve measures the discriminative power of a model (i.e. its ability to distinguish between high- and low-risk episodes). 39 A model with an AUROC curve of 1 would predict every outcome perfectly, and an AUROC curve of 0.5 is no better than chance. The AIC is a measure of the goodness of fit of a model that penalises a higher number of model parameters. When comparing models, a lower AIC is preferable. 41

As discussed in Defining episodes of surgical management for analysis, the data set contained more than one episode for some patients. Our aim was to develop a risk model for use at episode rather than patient level in monitoring short-term risk-adjusted outcomes for an entire paediatric cardiac surgical programme, not just for those undergoing their first operation; hence, we treat each surgical episode as independent in running the logistic regressions.

For this reason, the logistic regression output should not be used to infer risk at a patient level, as not all observations used in the regression were independent (with respect to patients). When running our cross-validation procedure (see next section), the data split was adjusted so that each record pertaining to the same patient was in the same cross-validation fold, so that none of the models produced by the cross-validation was tested on patients who had also featured in the development of that model.

25 × fivefold cross-validation

The original PRAiS 1 model developed in 2011 was validated by splitting the data into a development set (70%) and a test set (30%). All analyses were carried out on the development set, and, after calibration within the development set, the model was validated on the test set. 10,33 This was also the original protocol for PRAiS 2. However, there are inherent problems with this method of model validation. As the data are split only once, natural variation in the episodes included in the development and test sets can lead to an imprecise estimate of predictive accuracy. It is also not an efficient method as 30% of the data are never used as part of model development. After discussion with GA from UCL Medical Statistics and our independent statistical expert, DS, we decided to use fivefold cross-validation repeated 25 times (25 × fivefold cross-validation) to develop and test the PRAiS 2 risk model. 42–44 We preferred cross-validation to bootstrapping to ensure that all of the data were used in developing and testing the model, particularly as there exist small but high-risk subpopulations that are important to include.

For a fivefold cross-validation, the process is as follows.

1. The data are randomly split into five equally sized subsets; the model is developed on four subsets (80% of the data) and tested on the subset that was excluded (20% of the data). This is repeated so that all five subsets are treated as the test set once.

2. Step 2 can be repeated on different partitions of the data to increase the reliability performance estimates. We did this 25 times (25 × fivefold cross-validation).

3. The results for the predictive accuracy are averaged over all of the cross-validations to obtain a measure of the predictive accuracy of the final model.

4. The chosen model is then calibrated on the entire data set.

The advantages of 25 × fivefold cross-validation over maintaining a single quarantined test set are as follows. 43

• As each entry in the data is used an equal amount to test the model, there is less variability in the performance estimate.

• The model development process and the chosen risk factors are validated and use all of the available data.

• Repeating the fivefold partition leads to a more accurate performance estimate.

The data splits were stratified by year and unit to ensure a representative case mix and, as discussed above, all episodes for a patient were constrained to lie in the same data split.

To mimic prospective use, no adjustment of weights of this nature was made when using an episode in the test set for each cross-validation run.

Assessment of model performance for each considered model

To assess model performance we used the following measures.

In the main data set:

• the AUROC curve and AIC of the model developed on the whole main data set.

Under cross-validation:

• The median and range of the AUROC curve over the cross-validated test sets. We used the somersd command in Stata to calculate an overall AUROC curve for each of the 25 repeats of the cross-validation. 45 The median and full range of these 25 values were then calculated.

• For each cross-validation fold, a calibration slope (the regression coefficient, β) and the intercept (the constant term, α) were calculated by regressing the outcome on the linear prediction in the test set for that cross-validation. If a model was perfectly calibrated, we would expect β = 1 and α = 0. If β < 1, this can be evidence of overfitting, whereas β > 1 can be evidence of underfitting. If α < 0 the model has overpredicted observed deaths, and if α > 0 the model has underpredicted observed deaths. 46 The coefficient β can also be interpreted as the ‘shrinkage factor’, with a value of much less than 1 being a possible indication of overfitting and of the possible need to apply shrinkage to the model. The calibration slope and intercept were used to test for calibration rather than the more traditional Hosmer–Lemeshow test, as the former method is now considered superior. 41 The median and range of the calibration slope and intercept across the 125 cross-validation test sets.

In the external 2014–15 test data set following final model selection:

• the AUROC curve of the model developed on the main data set and tested on the 2014–15 data, including confidence intervals (CIs)

• the calibration slope and intercept of the linear predictor developed on the main data set and regressed on the 2014–15 data set, including CIs.

The regression output was also used to construct charts, known as MADCAP (Mean Adjusted Deaths Compared Against Predictions) charts,47 of cumulative predicted and observed deaths against episode number, with episodes ordered by increasing predicted risk. These give a graphical means of summarising the performance of the risk model with respect to discrimination and calibration. The end points of the two lines indicate the overall numbers of deaths predicted and observed over a series of cases. The extent to which the slope of the cumulative observed deaths increases with episode number gives information about the discrimination of the model: the greater the ‘bowing’ towards the bottom right-hand corner of the MADCAP chart, the better the discrimination of the model. Any major deviations of the ‘observed’ curve from the ‘predicted’ curve provide information on where the model may work less well across the spectrum of predicted risk.

Comparison of MADCAP charts between different models was used as a way of informing decisions about the value added or lost in adopting different approaches to how specific procedures, diagnoses and comorbidities were included in the model.

Finally, we investigated any episodes that had particularly large leverage or influence39 to see whether they were genuine outliers or data errors.

Clinical considerations

Age and weight

The relationships between age and weight and 30-day mortality are shown in Figures 9 and 10, respectively, which show the weighted-moving average of mortality over the range of ages and weight in the data set. These figures show that the low ages and weights have a very strong relationship with higher mortality, which drops quickly and levels off as patients in this data set get older and heavier.

FIGURE 9.

The weighted moving average 30-day mortality by age.

FIGURE 10.

The weighted moving average 30-day mortality by weight.

As the non-linear nature of the relationship between age and weight was now going to be accounted for more accurately, we did not include a specific low weight (< 2.5 kg) risk factor in the model.

We found that a cubic spline could not adequately model the relationship at all ages, but that fractional polynomials were able to capture the non-linear association for both age and weight.

After considering various two-dimensional fractional polynomials, with a selection of terms with powers of ±2, ±1, ±0.5, 3 and logarithms, that were the best fit in different cross-validation runs, we used ax+bx for both age and weight, as it was simple function that performed very well. Moving to these non-linear functions of age allowed us to discard the categorical age bands used in PRAiS 1.

Comorbidities and additional risk factors

Several months of iterative refinement of comorbidity and additional risk factor categorisation resulted in six final groupings for consideration, shown in Table 5. The details of how these factors are allocated are given in Report Supplementary Material 2.

In almost all cross-validation runs using all six risk factors, neither prematurity nor Down syndrome was significant at the 5% level in multivariable regression.

Down syndrome had been excluded from the comorbidity risk factor in PRAiS 1 as its presence was not associated with an increased risk of mortality and a majority of Down syndrome patients have a single specific procedure. When this was discussed again by the expert panel, they believed that patients with Down syndrome did not have a risk from this procedure that was substantially different from that of non-Down syndrome patients.

Although prematurity was significantly associated with mortality in univariate regression, this association disappeared once other risk factors were included. It appeared that the better treatment of age and weight (using fractional polynomials) was adequately capturing the additional risk due to prematurity (as these patients are almost all of low weight and young).

Given these considerations, and on the advice of the analysts (LR and CP), the expert panel agreed to exclude Down syndrome and prematurity from further consideration as risk factors for PRAiS 2.

The other four comorbidity and additional risk factor groups devised by the expert panel (see Table 5) remained significantly associated with higher mortality in all cross-validation multivariable logistics regressions and were included in the final model.

Year of surgery

While exploring early versions of the PRAiS 2 risk model by calendar year from 2009 to 2014, we found a clear trend of overestimating risk in the later years, particularly after 2012. This was not unexpected, given the already observed decrease over time in mortality up to 2010. 48 Raw mortality has continued to fall (albeit not evenly), with particular reduction post 2012 (Figure 11). This trend suggested that any model calibrated on data from 2009 to 2014, without taking an epoch effect into account, would already be out of date when used prospectively by hospitals for local quality improvement and assurance and by national audit. To include year of surgery as a continuous or ordinal variable would, however, be to assume that improvement over time is inevitable, and to ignore ceiling and threshold effects. The epoch effect is likely to be caused by a combination of changes in outcomes and coding; however, the expert panel felt that there had been a real improvement in outcomes in recent years, which was reflected in an overall drop in the observed 30-day mortality rate. There was a step change in completeness of the comorbidity field in 2009, and, although there has been improvement since then, it has been more gradual. The need for accurate and currently applicable data needed to be balanced with using the most data possible to get an accurate model, and using data from 2009 onwards with an epoch effect applied from 2013 provided an appropriate compromise. We therefore decided to explore including a binary epoch variable in the model indicating whether an episode occurred pre 2013 or from 2013 onwards.

FIGURE 11.

Observed vs. predicted risk for the final model with and without the post 2012 flag.

After finalising the model, we reran it in the development set with and without the epoch variable. Figure 11 shows the difference between observed and predicted deaths from each model evaluated in the main data set, for each year, weighted by the number of episodes in that year. The model that includes the epoch effect provides a better fit to the mortality over time and the binary epoch flag is highly significant (p < 0.005; AIC with the flag of 4222 vs. AIC without the flag of 4229). We discussed the inclusion of this parameter with the expert panel, asking whether or not they thought that such an epoch effect was clinically plausible, that is, was it plausible that outcomes had improved over the past few years instead of being due to natural variation. The expert panel considered that there had been a real improvement in outcomes and recommended the inclusion of the epoch flag in the final model.

Specific procedure and diagnosis risk groups

As procedure and diagnosis are intrinsically linked, the expert panel considered options for adjusting the CART-generated diagnosis and specific procedure groups together.

Following the second expert panel meeting, the team of analysts considered a shortlist of candidate models and compared their performance using MADCAP charts and consideration of individual specific procedure and broader group frequency/mortality in the national data set.

The initial risk groups presented at the meeting are shown in Tables 6 and 7.

The final candidate models for consideration then consisted of combinations of the model shown to the expert panel (model A), plus suggested changes to some of the broader specific procedure and diagnosis groupings that the expert panel made at the meeting.

This final candidate list consisted of the following.

• Model 1: the model presented at the meeting, which contained 10 specific procedure risk groups and nine diagnosis risk groups (the two lowest risk diagnosis groups shown at the meeting were combined) and was based on CART analysis using age at procedure and mortality of the specific procedures and diagnosis (see Tables 6 and 7).

• Model 2: the same specific procedure risk groups as model 1, with two additional diagnosis risk groups added and the following changes to the diagnosis groups –

  • three new separate groups consisting of transposition of the great arteries (TGA) plus IVS, patent ductus arteriosus (PDA) and acquired diagnosis, respectively

  • miscellaneous primary congenital diagnosis and total anomalous pulmonary venous connection (TAPVC) moved to diagnosis risk group E.

• Model 3: the same diagnosis risk groups as model 1, with six additional specific procedure risk groups added and the following changes to the diagnosis groups –

  • TAPVC repair plus arterial shunt, truncus and interruption repair and arterial switch plus aortic arch obstruction repair (with/without VSD closure) moved from specific procedure risk group A to specific procedure risk group B

  • mitral valve replacement moving from specific procedure risk group E to specific procedure risk group D

  • and the following new groups, when formed:

    • arterial shunt

    • repair of TAPVC, arterial switch plus VSD closure and isolated pulmonary artery band

    • PDA ligation (surgical)

    • bidirectional cavopulmonary shunt

    • Fontan procedure

    • atrioventricular septal defect (complete) repair.

• Model 4: the altered specific procedure groupings from model 3 and the altered diagnosis risk groups from model 2.

• Model 5: the same as model 3 with ‘repair of TAPVC’ moved to specific procedure risk group C.

• Model 6: the same as model 4 with ‘repair of TAPVC’ moved to specific procedure risk group C.

The model performance of the candidate models is shown in Table 8.

All of these models performed very well, with high median AUROC curve values, β values close to 1 and α values close to 0. The new models 1–6 outperformed PRAiS 1. The changes suggested by the expert panel for clinical reasons generally improved the model performance, with the AIC of model 1 being the highest of the candidate models (excluding PRAiS 1) and model 4 being the lowest. As the other indicators of model performance were so similar between the candidate models, we chose to progress with the model with the minimum AIC, model 4.

Once we had selected model 4, we used plots of Pregibon leverage and influence statistics against regression residuals and predicted risk to identify episodes with particularly high leverage or high influence. 49 On inspection, these episodes would be expected to have either high influence or high leverage as they involved unusual patients; for example, patients undergoing a low-risk procedure who died within 30 days had high influence, and patients who had unusual, but not unreasonable, features, such as having an older child with a weight of > 100 kg, had high leverage. We were thus satisfied that these episodes represent genuinely unusual cases and not outliers resulting from data errors, so we did not exclude them from the analysis.

The specific procedure and diagnosis risk groups that were used in the final model are detailed in Tables 9 and 10.

Final parameter selection

Once our final model parameters had been chosen, we considered whether or not we needed to apply shrinkage to our calculated model coefficients. 41 Although, as seen from Table 8, for model 4 the median shrinkage parameter β (equivalent to the calibration slope) was 0.92 (range 0.64–1.25), close to 1, we explored whether or not shrinking our coefficients improved our calibration under cross-validation. However, shrinkage did not improve the performance of the model in the cross-validation test data sets, so we did not consider it further.

Based on the analysis and considerations set out above, the final model, decided on jointly by the clinical and analytical teams, was a logistic regression model with the following variables:

• age (included as a×age+bage)

• weight (included as c×weight+dweight)

• broader specific procedure groupings (15 risk groups plus a ‘no specific procedure’ group)

• procedure type (bypass or non-bypass/hybrid)

• broader diagnosis groupings (11 risk groups)

• UVH attribute (indicator variable)

• presence/absence of a recorded congenital comorbidity

• presence/absence of a recorded acquired comorbidity

• presence/absence of a recorded severity of illness indicator

• presence/absence of a recorded additional cardiac risk factor

• indicator variable for whether an episode occurred pre 2013 or from 2013 onwards.

Probability of death within 30 days following paediatric cardiac surgery =11+e−z, where:

Parameters i = 1 to 34 are tabulated in Table 11, along with their corresponding regression coefficients, B_i, and the condition that must be satisfied for X_i = 1 (X_i = 0 otherwise).

Model performance in the main data set

When calibrated over the whole of the main data set, the AUROC curve for the model was 0.84 (95% CI 0.83 to 0.86).

Figure 12 shows the MADCAP chart in the main data set for the final model calibrated on the main data set. It necessarily has to be perfectly calibrated (predicted deaths = observed deaths), but discrimination is excellent and there are no major deviations in calibration across the spectrum of predicted risk.

FIGURE 12.

Cumulative deaths among the main data set plotted against episode number with episodes ordered by increasing risk as predicted using the final model calibrated on the main data set (n = 21,838).

Figures 13–21 show the performance of the final model calibrated on the main data set for key population subgroups defined by risk factor. Ideally, the model would perform well within such subgroups to provide confidence that the model will perform well prospectively for variable case mix (e.g. distribution of risk factors can differ between units).

FIGURE 13.

The MADCAP chart showing the performance of the final model for episodes relating to non-bypass (including hybrid procedure; n = 4856) and bypass procedures (n = 16,443). (a) Non-bypass; and (b) bypass.

FIGURE 14.

The MADCAP chart showing the performance of the model in episodes with an indicator of UVH (n = 3737).

FIGURE 15.

The MADCAP chart showing the performance of the model for episodes with an additional cardiac risk factor present (n = 1053).

FIGURE 16.

The MADCAP chart showing the performance of the model for episodes with an acquired comorbidity present (n = 1254).

FIGURE 17.

The MADCAP chart showing the performance of the model for episodes with a congenital comorbidity present (n = 2445).

FIGURE 18.

The MADCAP chart showing the performance of the model for episodes with a severity of illness indicator present (n = 2260).

FIGURE 19.

The MADCAP chart showing the performance of the model for episodes with Down syndrome (n = 1690).

FIGURE 20.

The MADCAP chart showing the performance of the model for episodes with prematurity (n = 2664).

FIGURE 21.

The MADCAP chart showing the performance of the model for episodes relating to (a) neonates (< 30 days; n = 4709); (b) infants (31 days–1 year; n = 8685); and (c) children (1–16 years; n = 8444).

As a final check on model performance without Down syndrome, prematurity or age bands, we examined MADCAP plots within these subgroups for the final model, which did not include any of these factors as explicit risk factors. These MADCAP charts are shown in Figure 19 for Down syndrome, Figure 20 for prematurity and Figure 21 for age bands. We can see that the model performs well in both discrimination and calibration for these subgroups.

None of the MADCAP charts provided any cause for concern.

Performance under cross-validation

Under cross-validation, the model described in Final parameter selection had a median AUROC curve of 0.83 (total range 0.82 to 0.83), a median calibration slope of 0.92 (total range 0.64 to 1.25) and a median intercept of –0.23 (range –1.08 to 0.85) (see also Table 8). The model had good discrimination in the cross-validation test sets, there was little evidence of overfitting and, in general, it tended to slightly overpredict the risk for patients (intercept < 0).

As an example of the performance of the model under cross-validation, Figures 22–26 show the MADCAP charts for the five test sets used in the 15th repeat of the cross-validation. Figure 27 shows the MADCAP chart for these five sets of predicted risk in the each of the five test data sets combined. Although some of the test sets show some under-/overprediction, when combined to cover the whole of the main data set, there is little indication of a systematic bias.

FIGURE 22.

The MADCAP chart showing the performance of the model tested on the first fold of the 15th cross-validation repeat (n = 4367).

FIGURE 23.

The MADCAP chart showing the performance of the model tested on the second fold of the 15th cross-validation repeat (n = 4368).

FIGURE 24.

The MADCAP chart showing the performance of the model tested on the third fold of the 15th cross-validation repeat (n = 4366).

FIGURE 25.

The MADCAP chart showing the performance of the model tested on the fourth fold of the 15th cross-validation repeat (n = 4369).

FIGURE 26.

The MADCAP chart showing the performance of the model tested on the fifth fold of the 15th cross-validation repeat (n = 4368).

FIGURE 27.

The MADCAP chart showing the performance of the model in the five test sets used in the 15th cross-validation repeat combined (n = 21,838).

Comparison with performance of PRAiS 1

In terms of the added value of the updates to PRAiS 1, the performance of a model using the risk factors from PRAiS 1 recalibrated under cross-validation had a median AUROC curve of 0.80 (range 0.80 to 0.81), a median calibration slope of 0.90 (range 0.63 to 1.31) and an intercept of –0.29 (range –1.15 to 1.06). The performance using the updated risk factors was, therefore, markedly better (unsurprising given the AIC calculated for Table 8).

PRAiS 2 outperformed PRAiS 1 when picking out especially high-risk patients. For example, the MADCAP charts for episodes that included an acquired comorbidity for PRAiS 1 are shown in Figure 28 (compare the equivalent performance in the PRAiS 2 model shown in Figure 27). On the other hand, PRAiS 2 still performed well within the subgroup defined by having a PRAiS 1 non-Down syndrome comorbidity (shown in Figure 29).

FIGURE 28.

The MADCAP chart showing the performance of the PRAiS 1 risk model recalibrated on the main data set for the episodes with an acquired comorbidity present in the main data set (n = 1254).

FIGURE 29.

MADCAP chart showing the performance of the PRAiS 2 risk model calibrated on the main data set for the episodes with a PRAiS 1 non-Down syndrome comorbidity present in the main data set (n = 7091).

Validation in the 2014–15 data set: descriptive analysis

The 2014–15 data set contained 7278 records. The data set was cleaned in the same way as the main data set, except that no adjustments were made to outlying patients’ weights, to reflect future use. The records included and excluded by the data cleaning are detailed in Figure 30.

FIGURE 30.

Inclusions in and exclusions from the 2014–15 data set.

The 2014–15 data set used for testing comprised 4207 30-day surgical episodes, corresponding to 3989 unique patients. The overall 30-day mortality in the 2014–15 data set was 2.3%, compared with 2.5% in the main data set.

Figures 31–38 show a comparison between the main data set and the 2014–15 data sets in terms of the episode-level mortality rates associated with the factors included in the risk model. These show broad similarity in the univariate associations between individual factors and 30-day mortality in the two data sets. That said, there are some features worth noting:

• The reduction in mortality for episodes with an acquired comorbidity shown in Figure 35 was coupled with an increase in the proportion of episodes with this additional risk factor from 5.7% in the main data set to 13.7% in the 2014–15 data set. This may be an early indication that the additional risk for this group is being diluted as reporting increases. This would need to be explored further in any future recalibration. The reduction in mortality for episodes with a congenital comorbidity shown in Figure 37 was not linked to as much of an increase in reporting of congenital comorbidities, and was more in line with the mortality trends of the data set as a whole.

• The large reductions in mortality in specific procedure risk group 10, shown in Figure 31, and in diagnosis risk group 2, shown in Figure 32, were investigated by the analysts, and, although the reduction in mortality was quite large, there was no reason to think that these reductions were not genuine for these groups (whether by change in practice, chance variation or both).

• The increase in mortality for episodes in specific procedure risk group 1 and diagnosis risk group 1 mainly relate to patients with HLHS undergoing their primary procedure. On investigation, a higher proportion of these patients underwent a hybrid procedure than underwent the traditional Norwood procedure, the latter of which generally is carried out on sicker patients and has a higher 30-day mortality rate.

FIGURE 31.

Observed mortality for specific procedure risk groups in the main data set (black with CIs, n = 21,838) and the 2014–15 data set (green, n = 4207).

FIGURE 32.

Observed mortality for diagnosis risk groups in the main data set (black with CIs, n = 21,838) and the 2014–15 data set (green, n = 4207).

FIGURE 33.

Observed mortality for bypass and non-bypass procedure types in the main data set (black with CIs, n = 21,838) and the 2014–15 data set (green, n = 4207).

FIGURE 34.

Observed mortality for episodes with and without a UVH indicator in the main data set (black with CIs, n = 21,838) and the 2014–15 data set (green, n = 4207).

FIGURE 35.

Observed mortality for episodes with and without an acquired comorbidity in the main data set (black with CIs, n = 21,838) and the 2014–15 data set (green, n = 4207).

FIGURE 36.

Observed mortality for episodes with and without an additional cardiac risk factor in the main data set (black with CIs, n = 21,838) and the 2014–15 data set (green, n = 4207).

FIGURE 37.

Observed mortality for episodes with and without a congenital comorbidity in the main data set (black with CIs, n = 21,838) and the 2014–15 data set (green, n = 4207).

FIGURE 38.

Observed mortality for episodes with and without a severity of illness indicator in the main data set (black with CIs, n = 21,838) and the 2014–15 data set (green, n = 4207).

Validation in the 2014–15 data set: performance

The receiving operating characteristic curve in the 2014–15 data set is shown in Figure 39. The AUROC curve in the 2014–15 set was 0.86 (95% CI 0.82 to 0.89), compared with 0.84 (95% CI 0.83 to 0.86) in the main data set on which it was calibrated. The performance of the model described in Final parameter selection across the spectrum of predicted risk is shown in Figure 40.

FIGURE 39.

The receiving operator characteristic curve of the model calibrated in the main data set and tested on the 2014–15 data set.

FIGURE 40.

Cumulative deaths in the 2014–15 data set plotted against episode number with episodes ordered by increasing risk, as predicted using the model set out in Table 11 (n = 4207).

The calibration slope in the 2014–15 data set was 1.01 (95% CI 0.83 to 1.18) and the intercept was 0.11 (95% CI –0.45 to 0.67). This shows that, although there was a slight (non-significant) underprediction of risk in the validation data set, the PRAiS 2 model showed excellent discrimination and no evidence of overfitting (calibration slope of 1.01). Overall, there were 97 observed deaths in the test set, compared with 89 predicted.

Recalibration across all of the data

Following the successful internal and external validation of the model, the coefficients of the PRAiS 2 model were recalibrated over the whole data set from April 2009 to March 2015. Any adjustments required to recorded weights (see Age and weight) were recalculated using the whole data set.

Comparing the predicted risk using the PRAiS 2 model calibrated on April 2009–March 2014 with that calibrated on April 2014–March 2015, we can see excellent agreement (Figure 41), providing a further check that our chosen risk factors are stable across the main and the 2014–15 data sets.

FIGURE 41.

Comparison of predicted risk for the final model calibrated on the main data set and on the whole data set for episodes in the main data set (n = 21,838) and in the 2014–15 data set (n = 4207).

The final model is as follows:

Probability of death within 30 days following paediatric cardiac surgery =11+e−z where:

Parameters i = 1 to 34 are tabulated in Table 12 along with their corresponding regression coefficients, B_i, and the condition that must be satisfied for X_i (X_i = 0 otherwise).

Note that, for prospective use, the ‘post 2013’ coefficient can be absorbed into the constant term.

Example episodes

Table 13 shows the risk calculated by PRAiS 1 and PRAiS 2 for some example episodes. For the groups of examples shown, the episodes have some very similar attributes, which are treated very differently by the PRAiS 1 and PRAiS 2 risk models.

Patients A and B have the same diagnosis and weight and have the same procedure performed only 1 day apart in terms of age. However, patient B has a severity of illness indicator and patient A does not have any additional risk factors. In PRAiS 1, although patient B has a non-Down syndrome comorbidity, the resulting risk is very similar to that for patient A, as patient A is a neonate and patient B is an infant, and so the increase in risk due to the non-Down syndrome comorbidity is balanced with an artefactual reduction in risk from the neonate to infant age band. The PRAiS 2 risk model captures the fact that patient B has a much higher risk of mortality within 30 days.

Patients C and D have the same procedure performed within 2 months of age, and have the same diagnosis and additional risk factors. However, patient D has a much lower weight than patient C. As is the case for patients A and B, in PRAiS 1 this increased risk is outweighed by the shift up an age band, meaning that patient C has a higher predicted risk than patient D, which is reversed in PRAiS 2.

Patients E, F and G all have the same procedure performed at the same age and have the same weight and diagnosis. They all have a PRAiS 1 non-Down syndrome comorbidity, and so are predicted the same risk by PRAiS 1. However, in PRAiS 2, the type and number of additional risk factors leads to better discrimination between these patients.

Institutional case mix and model performance

In addition to understanding the performance of the model across the spectrum of predicted risk and within key subgroups of patients, to assess the fitness for purpose of the model it is important to understand the distribution of case mix and, in particular, differences in case mix between institutions. Essentially, it is important to understand whether or not and under what circumstances differences in case mix and the differential performance of the risk model in different subgroups would combine to give an artefactual impression of better or worse risk-adjusted outcomes at one unit compared with another. We note that, throughout model development, we consulted with the expert panel about known differences in case mix between units and these discussions informed some of the decisions on which specific procedures and diagnoses to group. Such institutional differences in case mix are less important for local quality improvement as there the comparison is mostly like with like (current vs. historical outcomes for the same hospital) and when there have been changes in case mix these will be well understood by the local team.

The differences in case mix between centres, illustrated by predicted 30-day mortality risk, using the model across the model recalibrated on all of data, are shown in Figure 42. Each bar shows the distribution of predicted risk for a single institution.

FIGURE 42.

Comparison of the case mix between different institutions.

These differences in case mix were not considered problematic, given the excellent performance of PRAiS 2, but we would like to note one case-mix problem that is present within the model. For one specific procedure, the HLHS hybrid approach, we are aware that the model significantly underpredicts risk. Out of the 60 HLHS hybrid approach procedures in the whole 2009–15 data, there are 10 observed deaths, but there are only six predicted deaths based on the model recalibrated on 2009–15 data. This is because, in the UK, patients who have this procedure performed are generally sicker than patients with HLHS, who go down the traditional surgical treatment pathway, in ways that are not well captured by the NCHDA data set. The procedure is relatively rare, and so we could not use it as its own risk factor. Additionally, this procedure is predominantly performed at just one unit. The underprediction of risk for this procedure, therefore, may unfairly affect this unit in prospective national audit.

The expert panel discussed this issue and recommended including HLHS hybrid procedures within PRAiS 2 despite these issues, as they believed that the use of this procedure would increase over time and that it is a high-risk procedure that should be included in audit. It has been proposed to the NCHDA steering committee that the PRAIS 2 model for future national audit is run twice on data including and excluding hybrid HLHS procedures so that this effect can be accounted for. It is hoped that as this procedure becomes more common, PRAiS 2 could be recalibrated to better reflect the additional risk in a few years’ time.

Update to the PRAiS software

Once the risk factors had been finalised, we started work on updating the PRAiS Microsoft Excel software in May 2016. Many small details needed to be updated, but the major changes were:

• update of the specific procedure algorithm

• update of episode allocation to allow HLHS hybrids

• update of diagnosis group allocation

• introduction of broad specific procedure groupings

• introduction of new comorbidity and additional risk factor allocations

• update with the final risk model equation.

In addition, we added a new column for ‘NCHDA unique procedure identifier’ and information on predicted deaths at the request of some of the hospitals’ data managers.

There is a great of deal of code behind the PRAiS software and so we needed to spend a reasonable amount of time testing it to make sure that it was working as intended. It was tested independently by CP, LR, MU and SC. In particular, CP and LR used different subsets of between 1000 and 2000 actual raw records from the NCHDA data set, ran them through the PRAiS software (to generate specific procedures, capture errors, allocate 30-day episodes, allocate risk factors, calculate risks and assign deaths) and then compared the PRAiS output with that previously generated in the statistical software package Stata (which was used for model development). We required that the number of episodes, the allocation of risk factors, the number of calculated deaths and the estimates of predicted risk matched exactly between the PRAiS software and the Stata data.

We released the updated PRAiS software (PRAiS v3.0) along with updated documentation to UCL Business on 20 June 2016, and UCL Business released it to all UK and Ireland hospitals that perform heart surgery, as well as to NICOR, on 23 June 2016. It is to be noted that all of these centres and NICOR already held licences for the PRAiS software and received this update free of charge.

Thanks to close collaboration with members of the expert panel, JS and TW, a rarely used procedural code was found to cause a bug when it was entered as diagnosis. A little later, the national audit body spotted a misspelling in one of the specific procedures. We fixed these issues immediately and the latest version (3.0.2) is currently available on the UCL Business website.

We know that the national audit body and several centres are using the new software and risk model already, and, as far as we know, with no problems.

Who

This was a multidisciplinary effort with interwoven strands led by the principal investigator CP, DS (Winton Professor for the Public Understanding of Risk), TR (an experimental psychologist) and EJ of the charity Sense about Science. An explanation of how the roles fitted together is given below.

Overall strategy

We knew that the presentation of the complex ideas behind using risk adjustment for audit would not be straightforward. We also knew that DS’s expertise in communicating complex ideas about risk, and his track record of working with MP to produce useful tools and animations around interpreting risk, would be invaluable. 54–60 However, Sense about Science emphasised the importance of involving potential users as early as possible and so, with the welcome support of NIHR, we planned to coproduce the material using a series of workshops with potential users, starting from the very beginning of the project.

The output was aimed at two reasonably distinct audiences:

1. Older patients, parents and families of children who have had/will have heart surgery. This group is likely to have a more emotional involvement with published results and more likely to be focused on specific questions concerning individual hospitals.

2. Other interested users, including press officers and policy advisors for medical charities or professional bodies, members of the media, NHS England, family liaison services of paediatric hospitals, patient advocates and interested members of the public.

We thus decided to convene two sets of groups in parallel throughout the process: one comprising parents and families and one comprising other interested users. We recruited parents/carers and did not recruit children, as the large majority of children who undergo heart surgery are aged < 5 years. Family members were recruited through the CHF and other charities affiliated with the CHF. The CHF initially added a news item to its website and Facebook page (www.facebook.com), and forwarded the responses to Sense about Science. Later, CHF also passed on details of other charities that could help [e.g. Little Hearts Matter (www.lhm.org.uk) and Tiny Tickers (www.tinytickers.org)], and these charities also added news items to their Facebook pages and newsletters. For workshops 2–4, we offered parents a range of dates and times, including weekends and evenings. Parents who expressed an interest in earlier workshops but could not attend were also individually invited to attend later workshops. Parents were offered compensation for their time at the Involve rates of pay and travel expenses. 61

Sense about Science recruited the non-parent participants for the second strand of workshops. Details of each participant’s role are given in Appendix 3.

We held four workshops for each group of users in June 2015, October 2015, February 2016 and April 2016; these were organised and facilitated by Sense about Science and held at their offices in London (i.e. there were 4 × 2 focus groups in total). Each workshop lasted 1.5 hours and refreshments were provided. To ensure a fresh perspective at each time point, no workshop participant attended more than one workshop, but they were all invited to provide feedback on new versions of the website and animations via e-mail/online survey. We aimed to have about six people per group at each meeting (i.e. a total of up to 24 family members and 24 interested users).

For all workshops, participants received only minimal details about the project and were not required to prepare or read anything prior to the workshop. Each workshop began with refreshments, at which point participants were introduced to the facilitators (EJ and JT), to project team observers and to each other, before EJ gave an introduction to Sense about Science and JT gave a brief presentation on the background of the project, including the cost of poor risk communications in this area, and explained how the workshop would run. EJ and JT also asked for (and received) permission to record each meeting to facilitate the write-up of the discussion. The recordings were not shared outside the Sense about Science team. The participants were informed that the project team observers (always including at least one of CP, MP and DS) would remain silent unless there were specific questions about the technical content of the website. From the second set of workshops onwards, all material was viewed via the web pages, with each participant having access to their own laptop; participants were given approximately 15 minutes at the start of the workshop, after the introductions, to explore the material on their own with no further guidance from Sense about Science. It was made clear to parents that if at any point they did not feel comfortable to continue, they were free to leave. At the end of each workshop, the participants were asked if they wanted to be kept up to date with the project progress and to continue to provide feedback and comments via e-mail. All participants chose to stay involved. After each workshop, Sense about Science wrote a report on the discussion, including recommendations of changes, which was then shared with the rest of the aim 2 project team (CP, MP, DS, TR and EB).

Starting in late autumn 2015, once the web material was in a more advanced state, the King’s College London (KCL) psychology team began formal experiments to explore further how people understood the presented statistics and some of the key sections of the new drafted material. For this component of aim 2, we adopted a mixed-methods approach to evaluating candidate components for the website, involving experiments, think-aloud studies and qualitative research. The final component was an experimental comparison between the website and the NICOR report. 62 Our research focused on (1) comprehension and interpretation of prediction interval plots, (2) evaluation of hospital survival data when presented using prediction-interval data, and (3) user understanding of issues relating to the monitoring of paediatric cardiac surgery. This work was undertaken at KCL by TR (project coinvestigator) and EB. For the design and content of our studies, we drew on the content of the workshops run by Sense about Science (see Website development: April 2015–April 2016) (most of which EB attended as an observer), discussions with DS and CP (particularly, but not exclusively, about the mathematical and statistical concepts under investigation) and with MP (particularly in relation to data presentation via the website). These experiments provided additional input into the content and design of the website, including the animations.

Between workshops, CP and MP worked closely together to update the website content, with sharing and commentary by other team members. Content was updated in response to the Sense about Science reports from the workshops and, later, the insights arising from the psychology experiments.

Periodically, we also sent existing material to the CHF and the expert panel from aim 1 (see Chapter 2), in November 2015, February 2016 and April/May 2016, to invite feedback and to ensure that we had accurately described the role of the audit body and the process by which they monitored survival statistics for children’s heart surgery. In April/May 2016 we also sent the close-to-final version of the web material to all previous workshop participants, both for further feedback and to show how their previous participation had influenced the site. A summary of the strands of the project and how they interacted is shown in Figure 43.

FIGURE 43.

Summary of the development plan for the web material. SaS, Sense about Science.

We knew that the multidisciplinary, multi-institutional and coproduction approach to building the website would provide a challenge, as the extent and scope of the interaction of the different strands was new to all partners. To support our work in this new approach, we worked with LM, an experienced facilitator of multidimensional initiatives. She facilitated three aim 2 project team meetings in March 2015, January 2016 and June 2016, in which she took on the role of a ‘critical friend’. She asked questions, stimulated discussion and reflection and served as a sounding board for the team to air thoughts, learning and challenges from the project at key stages of its development.

We discuss each of the strands in more detail in subsequent sections. Website development: April 2015–April 2016 discusses the workshop and psychology input. The run-up to launch and the website’s reception and future dissemination is discussed in the subsequent sections (see from Feedback from previous workshop participants, expert panel, the National Institute for Cardiovascular Outcomes Research and others to Aim 2 dissemination). Finally, the last facilitated team meeting generated a lot of discussion about the learning from this project and how it could be used in the future. These discussions, the reflections from individual team members and the reflections from LM are all included in Appendices 5 and 6.

We note that we provide considerable detail in our description of the workshop feedback and how the feedback impacted on the site because we want to demonstrate how the final website was truly coproduced with workshop participants and its scope gradually considerably broadened in response to user requirements and suggestions.

Starting the web material development

Before planning the workshops and drafting any material at all, CP met with coapplicants RF and KB as representatives of NICOR in April 2015. This meeting confirmed understanding of NICOR’s current and planned output for its annual reports. It should be noted that, prior to the project starting, NICOR had planned a patient day in May 2015, from which we were hoping to gather some insights via RF and KB, but this event did not, in the end, take place.

The initial focus of the envisaged web material was on explaining the key graphic in the NICOR annual report, an example of which from the 2010–13 report is shown in Figure 44 alongside its accompanying table (Figure 45). 62

FIGURE 44.

Example of the key NICOR output, taken from the 2010–13 annual report (© 2014 UCL; may be reused by NHS and government organisations without permission). 62

FIGURE 45.

Table of outcomes taken from the NCHDA 2010–13 report (© 2014 UCL; may be reused by NHS and government organisations without permission). 62 This accompanies the key graphic in Figure 44.

This graphic shows how each hospital’s actual survival compares with its expected survival as calculated by the PRAiS risk model (in this case PRAiS 1). In Figure 44 the vertical axis is the ratio of actual survival to predicted survival (1 representing an exact match). If this value is > 1 then survival at that hospital was better than predicted and if the value is < 1 then survival was lower than predicted. The white interval shows the 95% prediction interval for this ratio: essentially, the range within which we expect hospital outcomes to lie. The width of the range depends on the number of operations a hospital has performed.

In April 2015, Sense about Science and CP met with DS and MP to discuss the initial content of the web material and plans for the first workshops.

During this discussion, MP suggested that by rotating the chart (see Figure 44) by 90^o, it could become another column on the table (see Figure 45), which would perhaps make the relationship between the table and the graphic easier to understand.

We thus decided to prepare the following in advance of the first sets of workshops:

• introductory text

• NICOR table of results

• NICOR results graphic (how table translates to plot)

• a horizontal version of the table and graphic

• explanations of the terms used in the graphic

• ‘frequently asked questions’ (FAQs) for broader questions around monitoring children’s heart surgery.

For the FAQs, we initially came up with the following.

• Which hospital should I go to?

• Is hospital X safe?

• What about my child’s operation?

• What happens if survival for hospital X is low?

• What does it mean if survival at hospital X is low?

• How good/accurate are the data?

• Limitations.

• Difficulties of risk adjustment.

• Benefits of risk adjustment.

After some discussion, and on the advice of Sense about Science, we decided to present each of these potential questions on paper, rather than on a website, alongside paper outputs of the NICOR graphic and table. This served two purposes. First, we could listen and respond to the order in which people asked questions and, second, it reinforced the fact that users were involved from the very beginning and encouraged workshop participants to ask for extra material and be open about suggesting changes to both the content and the navigation of the proposed website.

After this meeting, DS drafted the graphic and table explanations, while CP drafted the background and FAQ text. These initial drafts were then shared with Sense about Science, MP and TR, all of whom commented extensively and suggested important changes to the text. Both DS and CP found the perspective of experts from different backgrounds (TR and Sense about Science) extremely valuable.

Workshop round 1

The first two workshops were held on 29 June 2015 (non-parent) and 30 June 2015 (parent). Five non-parent participants and four parents, respectively, attended each workshop. CP, DS and MP attended both meetings as observers.

At each meeting we presented the NICOR table (see Figure 45) and graphic (see Figure 44), as well as the new horizontal display shown in Figure 46.

FIGURE 46.

New horizontal display of the graphic alongside the table content.

In addition, we provided paper copies of the background material and initial FAQs, with each FAQ on a separate piece of paper so that we could show the questions in any order. These drafts are provided in full in Appendix 4. The main feedback points are given below.

Workshop round 2

The second non-parent workshop was held on 6 October. Owing to difficulty recruiting parents during the working day, the second parent workshop was delayed until 10 December. The difficulty in finding several parents who could all attend at the same time prompted us to change our approach to the recruitment of parents. Instead of fixing a date and then asking the CHF to recruit parents for that date, we switched to offering parents a range of dates and times, including evenings and weekends for more flexibility. Via the CHF, we also asked other charities such as Little Hearts Matter and Heartline Families (www.heartline.org.uk) to post notices on their social media sites asking for parents who would be interested in being involved. Switching to this approach worked much better and we had few problems recruiting parents for workshops 3 and 4.

After workshop 1, we tried to address all of the feedback and started building a prototype website. By the second non-parent workshop, we had developed three draft web pages: an introduction page (Figures 47 and 48), a data page (Figure 49) and a FAQ page (see Figure 52).

FIGURE 47.

Our first attempt at the introduction page.

FIGURE 48.

The rest of the our first introduction page.

FIGURE 49.

Our first attempt at the data page.

The introduction page now started with a section called ‘What is this site for?’, followed by a list of hospitals that perform heart surgery in children, alongside links to each hospital and also to any hospital-specific charities or support groups.

We also had sections describing why survival is difficult to interpret and introducing the concept of risk adjustment (see Figure 48). In these sections we tried to be clear that the risk adjustment method was objective (based on patient characteristics rather than on any judgements about that hospital’s quality). We also added content to make it explicit that hospitals should not be compared with each other using their raw survival rates.

For the data page, we moved to using:

• a horizontal display as a final column on the data table

• using survival rates instead of the ratio of actual to predicted survival

• colouring the central predicted range.

We added hover-overs to the each bar along with explanations of what they meant (Figure 50). Clicking on a hospital name brought up that hospital’s results in detail, along with an interpretation of what the results meant (Figure 51). From the bottom of the page, the user could select different colour themes for the graphics, which we wanted to test on the workshop participants. It should be noted that from workshop 2 onwards the displayed data were from the 2011–14 NICOR annual report (which was published in the summer of 2015).

FIGURE 50.

Example hover-over in the data table explaining the predicted range.

FIGURE 51.

Example of what was shown when clicking on a single hospital.

The FAQ section at this stage was simply a collation of the existing text, revised after the first set of workshops (Figure 52).

FIGURE 52.

First attempt at a FAQ page. We were clear that this page was very messy to look at but we wanted to get its content right before concentrating on the layout.

The second parent workshop

Owing to the difficulty in recruiting parents, as discussed above, the second parent workshop did not take place until 10 December, almost 2 months after the non-parent workshop. Only two parents could attend on this date because a third parent was ill and it had become difficult to arrange anything in the weeks leading up to Christmas. The project team members present at this workshop were CP and EB.

Before this workshop, we incorporated much of the feedback from the second non-parent workshop. The updated introduction, data and FAQ pages are shown in Figures 53–57.

In the updated introduction page (Figure 53), we explored using expandable headings to reduce the amount of visible text. Particularly important sections were left expanded and the text was shortened and made more straightforward.

FIGURE 53.

The updated introduction page for the second parent workshop.

We created a new ‘mapped data’ page (Figure 54), designed by MP, to allow users to explore individual hospitals. The default page showed a ‘sample hospital’ that we used to explain how we presented the data graphically. For this workshop, we explored using a different colour theme.

FIGURE 54.

A new ‘mapped data’ page for the second parent workshop.

The mapped data page (Figure 55) gave the relevant raw numbers for individual hospitals (number of surgical episodes, deaths and survivors, observed survival rate and an interpretation of the survival rate in relation to its predicted range). We also provided links to the hospital’s website and any associated charity.

FIGURE 55.

The mapped data page showing the display for a specific hospital.

The tabled data page (Figure 56) did not have many changes, other than the use of a different colour scheme, the collation of each hospital and its code into a single column, and the addition of an information button for the observed survival rate to explain why it was not possible to order the table by survival.

FIGURE 56.

The updated table data page for the second parent workshop.

We renamed the FAQ page ‘understanding the data’ and were now exploring different options for making the layout more intuitive and less off-putting (Figure 57).

FIGURE 57.

The updated FAQ page for the second parent workshop.

First set of psychology experiments (December 2015–January 2016)

Below we discuss each of the three experiments (1a–c) in turn. A summary of how they fed into the website development is provided at the end of this subsection.

Experiment 1a

Rationale

This experiment was designed to assess people’s understanding and evaluation of prediction interval plots, comparing the type of plot used in the NICOR report (outcomes plotted using the survival ratio; see Figure 44) with the type of plot that we were considering using for the explanatory website (outcomes plotted using percentage survival rate; see Figure 51).

Methods

Participants and design

Seventy-seven university students were randomly assigned to one of two conditions. The participants were shown either plots with survival data displayed as ratios (observed/expected survival), as used in the annual audit report for these data (ratio-plot condition; n = 39), or plots displaying percentage survival rates that had been designed for the explanatory website (percentage-plot condition; n = 38). Figure 58 provides an illustration of the kinds of plots that were shown to participants.

FIGURE 58.

Graphs shown to participants in study 1b: ratio-scale plot (a) and percentage-scale plot (b).

Materials and procedure

An online survey was constructed using Qualtrics software (www.qualtrics.com; 2016, Qualtrics, Provo, UT, USA). The first part of the survey comprised demographic questions. The next part presented information about survival rate data for paediatric heart surgery, which was adapted from the national audit report for these data. The main part of the survey comprised three categories of questions about these data: comprehension, interpretation and evaluation. All questions were accompanied by graphical displays of hypothetical (albeit plausible) survival rate data for 14 hospitals.

Question set 1 consisted of a graphical display of hypothetical data, about which participants were asked comprehension questions (e.g. ‘what does the light grey shaded area mean?’). Question set 2 consisted of a graphical display of a different set of hypothetical data, about which participants were asked for their interpretation [e.g. ‘please select which hospital(s) has a survival rate that is higher than predicted’]. Question set 3 consisted of graphical displays of six hypothetical data sets (with the order of the data sets randomised). The questions asked about participants’ evaluations of the hospitals whose survival-rate data were displayed in each graph. There were four questions for each hospital, with ratings on a scale of 1 (‘strongly agree’) to 7 (‘strongly disagree’):

• I am concerned about hospital X.

• I feel confident about hospital X.

• I would recommend using hospital X.

• I would discourage people from using hospital X.

The six data sets used for these questions were carefully constructed to investigate a variety of possible interpretations of the data, allowing us to explore how key features of the graphical displays (e.g. control limits and sample size) influenced participants’ evaluations.

Results

Comprehension and interpretation questions

Participants in either condition selected the correct response almost all of the time for these questions (e.g. > 90% accuracy for questions about the ‘labelling’ or ‘meaning’ of regions of the prediction interval plots). However, the question about the meaning of the point denoting the survival ratio or percentage survival (‘what does the black dot mean?’) provoked many errors, with significantly lower accuracy for the ratio-scale plot (41.0%) than for the percentage-scale plot (71.1%) [χ²(1, N = 77) = 7.04; p = 0.008]. The most common answer to this question for participants in the ratio-plot condition was ‘observed survival rate’ (n = 19, 48.7%), which was incorrect. Participants expressed more doubt for this than for other comprehension questions, with three participants in the ratio-plot condition and four participants in the percentage-plot condition selecting ‘I do not know’.

Evaluation data

Responses to the four evaluation items (concerned, confident, recommend and discourage) were combined into an overall evaluation score for each hospital within each data set (‘7’ = highly positive evaluation; ‘1’ = highly negative evaluation).

An items analysis was conducted, treating each mean evaluation for a given hospital in a given data set as a data point, with separate data points for the ratio-scale and percentage-scale plots. This allowed detailed comparison between conditions and showed that the pattern of means was very similar between conditions, with the item means correlating (r = 0.97) between the two conditions. There was, however, a small but reliable tendency for more positive evaluations in the percentage-plot condition, with the mean (SD) evaluation score being 4.95 (1.02) in the percentage-plot condition and 4.73 (1.02) in the ratio-plot condition. These means differed significantly (p < 0.001; difference, d = 0.22).

Figure 59 shows mean evaluation (y-axis) as a function of each hospital’s outcome plotted as a survival ratio (x-axis). Three regression lines are shown on each figure: Loess regression lines (solid black) fitted to the full data set; and two linear regression lines fitted separately for those hospitals with action/predicted survival ratios above 1 (solid blue line) and those with survival ratios below 1 (broken green line). Note that a survival ratio above/below 1 also corresponds to a black dot shown above/below the mid-point of the predicted range; consequently, some information about survival ratio can (in principle) be inferred from a percentage-scale plot.

FIGURE 59.

Mean valuation per hospital as a function of actual outcome observed in that hospital represented as a survival ratio. (a) R² = 0.919 (linear relationship for all data); and (b) R² = 0.826 (linear relationship for all data).

Figure 59a shows that the ratio-scale plot yields evaluations that are very closely tied to the survival ratio, but, notably, the steepness of this relationship changes at a point close to a survival ratio of 1 (with the Loess curve switching from following one linear regression line to following the other). One could regard this as a ‘penalty’ for hospitals with survival ratios below 1, or as more discriminating evaluations when the survival ratio is below 1 than when it is above 1. The mapping between hospital outcomes and evaluations follows a similar pattern for the percentage-scale plot, although evaluations are not as closely tied to the plotted outcome measure (percentage survival in this case) as was the case for the ratio-scale plot. It should be noted also that the ‘penalty for lower outcomes’ does not ‘kick in’ quite as early (see Figure 59b), with the break point being around a survival ratio of 0.995 rather than around 1.0. Seemingly, some tolerance, or ‘grace’, is shown when evaluating points that sit a little below the middle of the predicted range on the percentage-scale plot (relative to the ratio-scale plot).

This tolerance, or grace, associated with the percentage-scale plot is also implied in Figure 60. This shows evaluations from both types of plot (ratio-scale plot represented by diamond icons; percentage-scale plot represented by open triangle icons) as a function of the position of the back dot relative to the predicted range (0 = lower 97.5% limit; 0.5 = mid-point; 1.0 = upper 97.5% limit). Black dots towards the end of or outside the predicted range yield similar evaluations for both kinds of plot, while black dots around the middle or slightly below the middle of the predicted range yield more positive evaluations when presented via the percentage-scale plot.

FIGURE 60.

Mean evaluation as a function of the position of actual outcomes relative to the prediction interval (prediction interval scaled from 0 to 1; hence x < 0.0 denotes ‘survival worse than predicted’ and x > 1.0 denotes ‘survival better than predicted’).

Conclusions

Overall scores or comprehension and interpretation did not differ significantly between conditions, suggesting that for many purposes there is no clear advantage to using ratio-scale plots over percentage-scale plots, or vice versa. Importantly, however, participants in the percentage-plot condition were significantly better at identifying the ‘meaning’ of the black dot that signifies each hospitals level of performance. We regard this as a strong argument for using the percentage-plot display because it suggests that even people (such as our participants) who are used to interpreting quantitative data have difficulty comprehending the ratio outcome measure.

We might characterise the evaluation data in the following way. Hospitals with ‘good’ outcomes reap a progressive ‘reward’ to their evaluations as outcome rates increase, while hospitals with ‘poor’ outcomes reap a progressive ‘punishment’ to their evaluations as outcome rates increase. However, consistent with the greater sensitivity to negative outcomes than to positive outcomes that is often reported (i.e. loss aversion, whereby ‘losses loom larger than gains’), the progressive punishment for poor outcomes seems to be greater than the progressive reward for good outcomes. 63 Importantly, on this reading of the data, the ratio-scale plot and percentage-scale plot seem to have a different reference point on the outcome scale that subjectively distinguishes ‘good’ from ‘poor’. The rate change in evaluation occurs close to the mid-point of the prediction interval (i.e. around survival ratio = 1.0) when outcomes are plotted as a survival ratio, while the point of rate change in evaluation is about one-quarter of the distance into the prediction interval (from the lower limit) when outcomes are plotted as a percentage. Thus, potentially, the ratio-scale plot could promote undue concern about hospitals whose outcomes are within the lower half of the predicted range (where there is no strong evidence that the hospital is performing poorly).

Experiment 1c

Rationale

This mixed-methods study was run as an auxiliary study to experiment 1a, and was designed to further explore people’s understanding of prediction interval plots, and the effect of the type of plot used (survival ratio vs. percentage survival rate).

Methods

Forty-nine participants who had participated in experiment 1a took part and were assigned to the same condition (ratio-scale plot or percentage-scale plot) as those in study 1a (ratio-scale plot, n = 24; percentage-scale plot, n = 25). Paper-based questionnaires were constructed, which included background information about predicted ranges (adapted from an explanatory website under development), for participants to read and refer to, as well as a graph appropriate to their assigned condition (Figure 61). The participants answered two comprehension questions, two inference questions, one question regarding the judgement of ‘acceptable’ outcomes and a sixth question asking participants to explain their answer to the fifth. For questions 1–5, participants marked a dot on the graph to indicate their answer for hospitals for which survival rates/ratios had been omitted (e.g. ‘mark a dot for hospital F, which would indicate that the hospital’s survival rate is much higher than predicted’).

FIGURE 61.

Stimuli for experiment 1c: ratio-scale plot (a) and percentage-scale plot (b).

Comprehension and inference

In keeping with experiment 1a, the responses to these (four) questions implied a good understanding of some of the key features of prediction-interval plots, with no significant differences between the two types of plot.

Judgement of minimum acceptable survival rates

The majority of participants placed their threshold (for the ‘lowest acceptable survival rate’) in the survival-as-predicted area (ratio-scale plot, 65%; percentage-scale plot, 52%). This implies a more stringent standard than that applied by NICOR, which the participants had been informed of and seemingly understood (given their answers to the two preceding questions). This aligns with our interpretation of experiment 1a, whereby survival rates are regarded as ‘poor’ when they are ‘as predicted’ but are below the predicted (point-estimate) survival rate – especially for the ratio-scale plot. The mean threshold value was close to the lower bound of the survival-as-predicted range (in either condition), and the location of those thresholds did not differ significantly between conditions.

Content analysis of threshold explanations

Participants’ explanations for where they placed their threshold were analysed for content. Eleven themes were identified; three of these in an explanation were each, independently, significantly associated with condition and/or the location of the participant’s threshold: (1) takes into account the number of procedures, (2) comparison with data of other hospitals and (3) emphasises the importance of being within the ‘as-predicted’ range. These associations are described below.

Some participants indicated that they compared an observed survival rate not only with the predicted ranges for the identified hospital, but also with the predicted ranges for other hospitals (n = 15). One participant with a threshold in the ‘survival much lower than predicted’ range explained their choice: ‘this is because, while it is out of the predicted range for the hospital itself, it is within the ranges for the other hospitals’. Another participant explained ‘97.5%, because it would still lie within the “survival as predicted” range but would not fall below a rate lower than “average” of the other hospitals’. Other examples of this type of explanation include ‘anything lower than the lowest range out of all the hospitals (hospital B) would to me be considered unacceptable’ and ‘this is because, while it is out of thepredicted range for the hospital itself, it is within the ranges for the other hospitals’. There was a striking difference between conditions for this type of explanation: of the 15 participants who provided an explanation of this type, significantly more (n = 12) were from the percentage-plot condition [χ²(1, N = 44) = 7.86; p = 0.005]. This suggests that percentage-scale plots may encourage interinstitutional comparisons, which – in this context – are inappropriate owing to the variation in case mix between hospitals. This could be due to a number of factors, including incidental features such as the horizontal orientation of the graph that we used. One possibility that having a better understanding of percentage survival than the survival ratio measure – which is an important difference that we identified in experiment 1a – actually encourages people to ‘do more’ with the measure. Thus, if percentages are straightforward to understand, they may facilitate comparisons based on that measure. Further research is required to determine how to discourage inappropriate comparison between hospitals – but without harming people’s understanding of the data.

In addition, emphasising the importance of being within the ‘as predicted’ survival range (theme 10) was significantly more common in the ratio-plot condition (n = 15) than in the percentage-plot condition (n = 7) [χ²(1, N = 44) = 5.37; p = 0.02]. This may partly reflect the fact that more participants in the ratio-plot condition (n = 15) placed their threshold in the ‘survival as predicted’ range than in the percentage-scale plot condition (n = 11) but it is too large an effect to be solely explained by this. Thus, in tandem with percentage-scale plots seemingly encouraging inappropriate comparisons between hospitals, ratio-scale plots may encourage the appropriate consideration of survival rates relative to the hospital’s own prediction interval.

One perhaps unsurprising finding was that threshold location was significantly associated with emphasising the importance of the ‘as predicted’ survival range, this theme being most prevalent when thresholds were within the survival-as-predicted area [χ²(2, N = 44) = 12.81; p = 0.002]. There was an association between taking the total number of procedures into account and the location of lowest acceptable threshold [χ²(2, N = 44) = 8.16; p = 0.017]. To provide an interpretation of this association, we note that none of the five participants who mentioned the number of procedures put their threshold in the survival-as-predicted range (whereas 26/39 of the other participants did so). There was also an association between comparison with data of other hospitals and the location of lowest acceptable threshold [χ²(2, N = 44) = 6.40; p = 0.041], which reflects a tendency to mark a lower threshold when explicit comparisons with other hospitals are made.

Although not associated with condition or threshold, another interesting feature of these explanations was that participants sometimes acknowledged that a hospital could fall outside the predicted range by chance (n = 10, 20.8%). Some participants acknowledged chance, but chose to discount this information [‘if a score were to be below its prediction, it could be a sign that something is wrong within the hospital’ (ignoring the possibility that a low score could be due to chance)] or explained that they would not take the risk of using such a hospital, ‘although being outside of the predicted range could be a result of chance; it seems safer to trust the hospitals who have matched their predicted survival rate’ and ‘even if it is due to chance they should aim to improve survival rate’. This represents a potentially important feature of people’s reasoning about uncertainty: they may acknowledge chance or sampling variability as a potential cause and yet be unwilling to accept this as the best explanation of the data.

Conclusion

Participants working with percentage survival rate plots were more inclined to explain their judgments on the basis of comparisons between hospitals and (accordingly) less inclined to mention comparing a hospital’s outcomes against its prediction interval. However, comparing a hospital’s percentage survival against its prediction interval (and not against other hospitals) is exactly what should be done (for the very reasons that call for risk-adjusted data and the creation of prediction intervals). Therefore, in a subsequent study we sought to test whether or not the tendency for inappropriate comparisons between institutions could be reduced.

Workshop round 3

We note that our original aim of producing an animation and some accompanying text that simply explained the published NICOR graphic had now been, of necessity, considerably expanded. Workshop participants had made it clear that the website needed a lot of background material that was carefully written and carefully presented. Because users can navigate a website in any order they like, we needed to be careful that key messages were repeated in different parts on the website and that there was sufficient detail to avoid misinterpretation of information on the data pages.

Reflecting on the feedback of workshop 2, CP and MP decided that adding a new page explaining how to navigate the website was not a good solution and that, instead, we should develop a site that was intuitive to navigate and needed as little work as possible on the part of users to work out where key information was and how to navigate between sections. MP suggested a new, much simpler homepage that gave an overview of the site content (Figure 62). By this stage, we also had a new logo, designed by CORU’s research manager Andrew Wilshere.

FIGURE 62.

The new, simpler home page developed for the third set of workshops.

The introduction content had been moved to a new page called ‘What, why, how?’, which now had a ‘Key points’ section at the top (Figure 63). These points were decided based on previous feedback from the workshops and also from expert panel members who, by this stage, had seen and commented on the site.

FIGURE 63.

The new ‘What, why, how’ page developed for the third set of workshops.

CP extensively reworked the text from the previous introduction page to make it shorter and easier to read. At the bottom of this page were explicit sections entitled ‘what this site cannot do’ and ‘what this site can do’. On the suggestion of previous workshop participants, we added direct links to charities in the ‘what this site cannot do’ section.

The mapped data page had been updated to incorporate the sample hospital information as a legend within individual hospital displays (Figure 64).

FIGURE 64.

The new individual hospital display with added legend.

We had also finalised the colour scheme, settling on a dark blue predicted range and a light blue extended predicted range. The tabled data page now included some summary detail about the data at the top, as requested by previous participants (Figure 65). Clicking on either the hospital name or the hospital code from the data table gave the same pop-up as in Figure 64.

FIGURE 65.

The data table page updated for the third workshop.

After exploring several options for the FAQs layout, we settled on the version shown in Figure 66 (after the third non-parent workshop and before the third parent workshop). We had changed the name to ‘Everything else’ and added an information icon. After discussion with Sense about Science, CP also went through the FAQs and provided a short ‘headline’ answer alongside each existing long answer (Figure 67).

FIGURE 66.

The updated FAQ section for the third workshop.

FIGURE 67.

Example FAQ showing the new ‘headline’ answer format.

Second set of psychology experiments (March 2016)

Below we discuss each of the three experiments (2a–c) in turn. A summary of how they fed into the website development is provided at the end of this subsection.

Experiment 2a

Rationale

Drawing on previous research from experimental psychology on ‘evaluability’ (i.e. what makes it easier or harder for people to evaluate the dimensions of an option), this experiment tested whether or not we could alter participants’ evaluation of survival data simply by having them evaluate one hospital at a time, thereby removing the possibility of an ‘easy’ comparison between hospitals.

Methods

Sixty-five participants were recruited from the KCL research volunteer list. An online survey was constructed using Qualtrics software, which had a similar structure to the survey in experiment 1a (demographics, audit report excerpt, comprehension questions and then evaluation questions) but with the comprehension questions divided into two sets and interpolated by the provision of additional guidance on prediction interval plots (taken from the draft version of our website). The presentation order of these two sets of three questions was counterbalanced. The stimuli for the evaluation questions were four of the six hypothetical data sets from experiment 1a – always presented with percentage survival rate as the outcome measure – with order of the data sets randomised. Participants answered two of the four evaluation questions for each hospital (I feel confident about hospital X; I would recommend using hospital X) with ratings on a scale of 1 (‘strongly disagree’) to 7 (‘strongly agree’). These responses were given individually for a given hospital (i.e. only one question on the screen at a time) but with the (sequential) presentation of these questions blocked by question (order of questions was randomised) within data set (order of hospitals within data set was randomised).

For the main manipulation in this experiment, participants were randomly assigned to one of two conditions. The participants saw plots showing data for all 14 hospitals in a data set (combined condition; n = 32) or plots showing only the data for one hospital at a time (individual condition; n = 33). Thus, when answering an evaluation question, participants in the individual condition saw only the data pertaining to the hospital identified in the current evaluation question, while participants in the combined condition saw plots similar to those in the percentage-plot condition of experiment 1a.

Results

Comprehension questions

Accuracy scores for the number of correct responses (out of three) were computed for each set and subjected to a 2 (set) by 2 (order group) analysis of variance. There was no significant effect of set or group order group; however, there was a substantial and statistically significant interaction between question set and order group [F(1,63) = 21.71; p < 0.001; η² = 0.256]. Partial eta-squared, η², is a standardised measure of effect size that provides an estimate of the proportion of variance in the dependent variable that is accounted for by a factor in an analysis of variance model. Elsewhere, for differences between two means, we use the standardised mean difference, d, to report effect size (in terms of the number of SDs that separate to means). Thus, performance on a given set was better after reading the guidance than before [t(63) = 2.39; p = 0.020 for set 1; and t(63) = 4.42; p < 0.001 for set 2]. In addition, participants were more accurate on whichever set they completed second (after they had read the guidance) [t(63) = 2.71; p = 0.011; and t(63) = 3.82; p = 0.001] (Figure 68).

FIGURE 68.

Mean percentage accuracy (with 95% CIs for the mean) before and after reading the guidance, shown separately for each question set.

Evaluation scores

Responses to the two evaluation questions (‘feel confident’ and ‘recommend’) were averaged together to create an overall evaluation score for each hospital within each data set. These mean evaluations are summarised in Figure 69. As per experiment 1a, we used an items analysis to perform a ‘global’ comparison between the combined condition and the individual condition, which revealed that the mean evaluation score was higher in the individual condition (mean = 5.24, SD = 0.87) than in the combined condition (mean = 5.02, SD = 0.97). This difference was small (d = 0.24) but statistically significant [t(55) = 7.95; p < 0.001]. Figure 69 illustrates that the discrepancy is most prominent for hospitals with lower survival rates: a reading of these data that is confirmed by more detailed analyses of them.

FIGURE 69.

Mean evaluation as a function of whether evaluations were made with data for all hospitals showing (combined condition) or for only the hospital being evaluated (individual condition). Loess regression lines are plotted separately for each condition.

Conclusions

Presenting survival rate data with accompanying prediction intervals for one hospital at a time results in more positive evaluations (relative to presenting survival-rate data and prediction intervals for several hospitals), particularly for hospitals that have survival rates that fall towards the lower end of the distribution. Our favoured interpretation of this is that individual (separate) presentation of the data makes comparison with other hospitals difficult and forces participants to compare a hospital’s survival rate with its prediction interval. This results in more positive evaluations for hospitals that would otherwise ‘suffer’ in a comparison against other hospitals, that is, those with lower survival rates. However, perhaps these hospitals should not be evaluated positively. The rationale for using prediction intervals speaks to that question as follows. If a hospital’s survival rate is within its prediction interval, we have no evidence that the hospital’s survival rate is lower than predicted; arguably, such a hospital should not receive negative evaluations purely on the basis of its survival rate. By contrast, if a hospital’s survival rate falls below its predicted range, there is some evidence that survival rates are lower than predicted (given case mix). This is a cause for concern and would trigger an investigation of a hospital’s performance, and, therefore, a negative evaluation is not inappropriate. Further analysis suggests that participants’ judgments in the individual condition were broadly consistent with this nuanced view of how hospitals should be evaluated: the more ‘generous’ evaluation of hospitals in the individual condition is largely restricted to hospitals that have survival rates that are within their predicted range for this measure.

Workshop round 4

Developing animations

The feedback from the third set of workshops was, in general, very positive; by this time the basic content, look and navigation had been finalised, albeit with a few tweaks still to be done. The project team now shifted to concentrating on animation content. Three main themes had arisen from the workshops that we believed were key:

1. an explanation of how the PRAiS formula is used to calculate the predicted range and how its width is determined by pre-setting the probability of a hospital‘s surviving rate falling outside the range (in this case 5%, as the central range is a 95% prediction interval)

2. key points about hospital survival data (comparing raw survival rates between hospitals is not appropriate; different hospitals have different predicted ranges; hospitals that do fewer operations have wider predicted ranges)

3. an easy guide to how to interpret the data as we were presenting them.

DS and MP wrote the first drafts of the two animation scripts. The first animation covered:

• why you would not expect hospitals to all have the same survival rate

• why it does not make sense to compare raw survival rates

• that you can only compare a hospital’s survival with its own predicted range of survival

• the PRAiS risk model factors and how they are used to calculate the predicted chance of survival for each child a hospital treated

• how these individual chances are combined to calculate a predicted and an extended predicted range of survival

• explanation of different widths and different relative widths of the predicted range in different hospitals.

The second animation covered:

• explanation of how we show the data

• what it means if the survival dot is in the extended predicted range or outside the extended the predicted range

• what happens if a hospital’s survival rate is below its predicted range

• a guide to further sections of the website.

By this point, the website was already much larger in scope than we had expected and looking considerably more ‘polished’ that we had first intended. MP suggested that we contract external help for designing the animations, especially for iconography. In particular, MP pointed out that in animation 1, we would need to have graphics to represent children undergoing heart surgery and that some would need to represent children dying after surgery. Finding a way to do this that was easy to follow and visually intuitive, but nonetheless sensitive to the subject matter, was not a trivial task.

After obtaining agreement from NIHR, CP approached Qudos Animations. From the middle of March to the end of May 2016, CP and MP worked closely with the company to refine the script and storyboard for the proposed animations. This involved a new way of working for Qudos Animations, as we insisted that users were given the opportunity to feedback on imagery, script and storyboard, which meant building more time into the process and ensuring that Qudos Animations was prepared to change the storyboard and imagery quite late into the process. We were also fortunate that a professional actor, Leighton Pugh, who had a wealth of experience recording commercial audio books, offered his voiceover services free of charge to the project.

The final set of workshops thus focused very much on the animation storyboards that had been developed over the previous 6 weeks. We combined a rough draft voiceover of the existing script by Leighton Pugh with a Microsoft PowerPoint^® (2013, Microsoft Corporation, Redmond, WA, USA) slide show of each animation storyboard to give workshop participants a flavour of the content and feel of the animations. We also provided print-outs of the storyboard to facilitate a more detailed discussion. Storyboards from animation 1 are shown in Figures 70 and 71.

FIGURE 70.

Example storyboard from animation 1 shown in the fourth set of workshops, frames 1–6.

FIGURE 71.

Example storyboard from animation 1, frames 7–12.

We started animation 1 (frames 1–4; see Figure 70) with what had become the key message for the website: that because different hospitals treat different children, they would not be expected to have the same survival rate; that it would, thus, not make sense to compare survival rates directly; and that instead a hospital’s survival rate should be compared with its predicted range only. In frame 6 (see Figure 70), we see the draft icons for children of different ages who have been treated by a hospital. Frames 7–8 (see Figure 71) show the move from data to a predicted chance of survival for each child treated and the introduction of three simple colours to show different levels of risk.

The key section explaining how the predicted range is calculated then starts in frame 10 (see Figure 71), in which we consider a hypothetical hospital that treated exactly 100 children. The core idea, designed by DS, CP and MP, was to show ‘possible futures’ for those 100 children: out of 100 children, each with their own predicted chance of survival, some would probably not survive. The animation would take the viewer through 20 possible futures, showing that each future could have both different children and a different number of children surviving, resulting in 20 possible overall survival rates for that hospital. In our example simulation, 19 out of 20 of the possible overall survival rates would lie inside the predicted range. This sequence involves difficult mathematical concepts that are of a standard typically found in the last year of Advanced level at secondary school or in the first year of an undergraduate degree, but they lie at the heart of outcomes monitoring across the NHS (and beyond). DS and CP were keen to try to explain this using the animation as far as possible; certainly we believed that showing it through a simulation was more likely to be successful than describing it in plain text on the website. In frame 12 (see Figure 71), we had an example of how a death of a child was represented (in this case two darker ‘child’ icons out of the 100). We knew that this was a key aspect to get right and asked participants for feedback on this specifically.

The website itself had evolved since the February workshops, taking into account the feedback arising from these. This related mainly to wording changes, but we also changed the layout of the ‘map’ tab on the ‘explore the data’ page to make it easier for users to select hospitals (Figure 72).

FIGURE 72.

The map data page for the fourth workshop.

Website feedback

All participants were very positive about the website and found it easy to navigate. Parents in particular commented that the text was easy to read and ‘nicely categorised’. A new observation was that the website looked professional and serious, which participants found reassuring. The sense of quality was reinforced by use of the institutional logos and the funding statement at the footer of each page, the links to charities and support groups and the clear statements about what the website could and could not do. These aspects had not been considered previously by the project team, but once they had been pointed out it was clear that they were an important contributor to the overall look and feel of the website.

Many participants commented that it looked as if the brief descriptions of each page on the home page (Figure 73) should have corresponding hyperlinks, and it was initially confusing that they did not. Detailed feedback was also provided, such as increasing the size of the logo, making all boxes on the home page the same size and making the header a uniform background colour.

FIGURE 73.

The version of the home page used for the fourth set of workshops.

With regard to ‘Explore the data’, participants said that this page was very easy to navigate and understand, and they liked that it was ‘straightforward’ and ‘to the point’. Regarding the table (now list) view, parents said that we needed to add links to the relevant FAQ if a hospital’s survival rate was outside its predicted range. No participant wanted to be able to sort by raw survival rate. Non-parent participants suggested having some of the key points from ‘What/why/how’ repeated on the ‘Explore the data’ tab. All participants liked the updated legend in the hospital detail section but asked if this could be made hospital specific to match the width of the predicted range and extended range for that hospital’s data.

Non-parent participants felt that there was too much information on the ‘Everything else’ tab (i.e. the FAQs), but parents liked this section and found it useful. The project team decided not to change the content of this section. As part of the changes since February, the phrase ‘chance factors’ had been replaced with ‘unforeseeable factors’. All participants much preferred this revised phrase. Non-parent participants emphasised the need for consistency between the language used in the animations and the language used on the website.

Finalising the animations

After the final set of workshops, Qudos Animations worked with MP and CP to finalise the animations. Particularly for the animation ‘How is the predicted range calculated?’, this involved MP generating simulations of possible futures for the examples used in the animation. CP and MP worked together to make sure that the age and predicted risk distribution of the children was consistent with the national data set and provided Qudos Animations with the exact content of the simulations to be used in the animation.

Once Qudos Animations had provided the final versions of the animations, we explored how best to provide a transcript. Guided by MP’s expertise, the team decided that providing optional closed captions within the videos would be the best solution. Unfortunately, not all browsers support closed captions in videos directly, and we needed help from a JavaScript (Code School, Orlando, FL, USA; www.javascript.com) library to smooth over these cracks and provide as consistent an interface as possible. We eventually selected the open-source video.js player (Brightcove Inc., Boston, MA, USA), which is well supported and often used for this purpose. 64

Finalising the content

With the content and look of the website finalised, CP, EJ and JT independently checked the content of every page, concentrating in particular on the content of the questions and answers on the ‘Everything else’ page. CP updated the language in several of the answers to match the final language used in the animations and in the ‘What/why/how’ tab, and checked all of the hyperlinks in the website. It was at this stage that we added in the internal links between pages, in particular to the two animations and from the data to particularly relevant questions in the ‘Everything else’ section.

The team discussed whether or not to add a comment or forum section to the website, as had originally been suggested in the protocol. However, MP pointed out that any such section would need to be actively moderated and we did not have resource left after the project ended to enable us to commit to this. We thus agreed on adding a ‘Contact us’ section, which listed three e-mail addresses, each for a different kind of query: general queries about the website (e-mail address for Sense about Science), queries about the risk model (e-mail address for CP) and reports of technical problems (e-mail address for MP). We also directed users to the national audit body website for any questions about either individual hospital data or the national audit process.

CP also wrote the ‘About us’ and ‘How we developed this site’ sections for the ‘Everything else’ page, and the team provided their photos for the ‘About us’ section. The CHF preferred not to have a photograph but to be referenced only as an organisation.

Learning from the workshops showed that the key points at the top of the ‘What/why/how’ section were extremely valuable and should be made as attractive and accessible as possible. We wanted to adopt the same style as had been used in the two site animations, so CP and MP chose representative images from the Qudos animations. MP then extracted the artwork from the After Effects animation sources, kindly provided by Qudos, to generate the images used in the key points section of ‘What/why/how’ (Figure 74).

FIGURE 74.

Final set of ‘key points’ at the top of the ‘What/why/how’ web page.

Final check of the national audit data with the National Institute of Cardiovascular Outcomes Research

The NICOR NCHDA annual reports provide the overall number of operations (surgical episodes) and the actual survival rate to one decimal place. Using this information, we calculated the numbers of survivors and deaths for each hospital, but these were subject to potential rounding errors. RF and CP worked with the national audit data lead to double check that the numbers provided in the ‘Explore the data’ sections were accurate. We also agreed that NCHDA would share the data for future reports with CP once the annual report was public. This would ensure that the website was kept up to date with the latest report in close to real time. It should be noted that the site has been designed by MP to ensure that updating it once a year with the latest report data is relatively easy.

Mobile phone and tablet compatibility

MP adopted Twitter’s (Twitter, Inc., San Francisco, CA, USA) Bootstrap Cascading Style Sheets (CSS) library (http://getbootstrap.com) early in the project’s development, as it is well tested and very capable. It adopts a ’mobile-first’ approach, meaning that when a page is being designed it is easiest to first mark up the HTML (HyperText Markup Language) for mobile phones, and adjust the code for wider (e.g. desktop) screens later.

In the workshops, we used a mix of laptops and tablets, which had the great advantage of enabling us to identify formatting issues early in the process. With hindsight, MP believes that we could have usefully used the ’mobile-first’ strategy to design content as well as layout. That would have forced us to adopt a layered approach to content at an earlier stage and would have highlighted the importance of microcopy (small bits of crucial text).

Moving to final domain and ensuring browser compatibility

CP purchased the final domain name of ‘childrensheartsurgery.info’ in the spring of 2016 (the site content is hosted on a University of Cambridge server). At the start of the project we were unsure in what final environment this site would reside, so we could make very few assumptions about web server capabilities: only that the server would need to be capable of serving static files. MP thus opted to develop the site as a single-page application in ClojureScript (https://clojurescript.org) instead of Facebook’s (Facebook, Inc., Menlo Park, CA, USA) React JavaScript library using the tonsky/rum wrapper (GitHub, San Francisco, CA, USA; https://github.com/tonsky/rum). MP used Figwheel (GitHub) to provide immediate run-time loading of changes as they were edited. This approach is inherently simple and allowed MP to make changes quickly as the site evolved and avoid feeling trapped by early architectural decisions.

However, this did mean that MP had to do some extra work when moving to production to ensure that the browser’s ‘back’ button would work properly and that we were using easy-to-type, search engine-compatible, clean URLs (Uniform Resource Locators). This was perhaps the trickiest part of the development effort, as HTML5 browser history support is not easy to understand. Clean URLs also meant that if we wanted to move from a single-page application to a server-hosted application in the future, it would be possible to do so without any user bookmarks being affected.

Internet Explorer (Microsoft Corporation, Redmond, WA, USA) was late in supporting the HTML5 features we have come to depend on. We have not attempted to support Internet Explorer 8 and Internet Explorer 9, which are in any case slowly falling out of use, with Internet Explorer 11 the final release of Internet Explorer (replaced by Microsoft Edge). During production tests, MP added advice to the home page for Internet Explorer 8 and Internet Explorer 9 users to help them to gain access to the site. It should be noted that in cases when institutions have not yet upgraded their machines to later versions of Internet Explorer, other browsers such as Google Chrome (Google, Inc., Mountain View, CA, USA; www.google.com/chrome/) or Mozilla Firefox (www.mozilla.org/en-GB/firefox/new/) must be used.

Rationale

Building on the studies of First set of psychology experiments (December 2015–January 2016) and Second set of experiments run in March 2016, which evaluated individual elements of the website, this study provided a ‘summary evaluation’ of the final website that had been developed by the aim 2 team. The objective of the study was to determine whether or not the website provided ‘added value’ to the NCHDA report of the national audit data. Thus, we conducted a small-scale randomised controlled trial comparing people’s ability to answer questions about the audit data and their interpretation when they used the NICOR report (only), the website (only) or both the report and the website (together).

Methods

Results

Questions that had a single correct answer were scored as correct (1 point) or incorrect (0 points). Other questions were scored according to a mark scheme that reflected the quality of answers, assigning ‘marks out of N’ according to the number of valid points included in the answer, which were then scaled on a 0–1 scale to give all questions similar weight. These were then combined (by addition) into three summary scores for each participant: comprehension (out of 11), understanding (out of 15) and an overall score (out of 26).

Table 14 summarises the accuracy of answers by condition, for individual questions and for the three summary scores. Mean comprehension accuracy was significantly greater for participants who had access to the website (mean 90.5%, SD 16.1%) than for those who were given access the report only (mean 78.8%, SD 13.0%), t(33) = 2.17; p = 0.037; 95% CI mean difference 0.8% to 22.7%; d = 0.77 (large effect). The mean understanding score was significantly higher among participants who had access to the website (mean 53.8%, SD 17.7%) than among those who were given access the report only [mean 31.8%, SD 17.2%; t(33) = 3.52; p = 0.001; 95% CI mean difference 9.3% to 34.7%; d = 1.25 (large effect)]. The mean overall survey score was significantly higher among participants who had access to the website (mean 69.3%, SD 13.7%) than among those who were given access the report only [mean 51.7%, SD 11.3%; t(33) = 3.83; p = 0.001; 95% CI mean difference 8.3% to 27.0%; d = 1.36 (large effect)].

An inspection of Table 14 indicates that the differences between the conditions are greater for some types of question than for others. Notably, for the comprehension questions, determining the ‘meaning’ of the black dot (i.e. identifying the outcome measure) was much harder from the report (alone) than from the explanatory website (question c.1s). This is consistent with what we found in experiment 1a. The other large differences were for questions about hospitals with survival rates outside the predicted range, in particular the actions that the audit body takes when this occurs (questions o.3 and o.4) and how often this occurs by chance when a hospital is operating as predicted (questions o.5 and o.6).

Conclusions

The results of this study suggest that the explanatory website does improve people’s understanding of the complex survival data generated by the PRAiS model and reported in the NCHDA report. It is worth noting the areas in which the website seems to be particularly helpful. First, comprehension for the outcome measure used to evaluate performance is substantially better for the website than for the report, which – when taken together with other questions – results in better overall comprehension for the plots when people have access to the website than when they have access to the report alone. Second, the website seems to provide people with clearer information about the audit process triggered when a hospital’s survival rate falls outside its predicted range than can be found in the NICOR report. Third, the website also provides information about how often the survival rate will fall outside the predicted range by chance, information that participants found very hard to glean from the report. These second and third strengths of the website may be particularly important for ameliorating ‘alarmist’ reactions to the data, because the site seems to help people to appreciate that not all unexpected differences are meaningful, and that there is a process in place through which the health service follows up on the data when there is potential cause for concern.

The embedded nature of the experiments meant that we were forced to trade off between maximising statistical power and having results available in sufficient time to inform the next stage of website development. If time-constraints had permitted, we would have wished to exceed 50 participants per cell in order to exceed 70% power to detect a mean difference of d = 0.5 with a between-subjects comparison (which we would regard as an ‘important’ size of difference to detect).

Importantly (and acknowledging that we would have wanted a larger study for firm conclusions in this regard), although there were several types of question for which accuracy in the report condition was similar to that for the website or combined conditions, there were no questions for which access to the website led to substantially lower accuracy. In addition, we had wondered if having a website with different data formats from those of the NCHDA report might cause confusion that would not arise when using only one of these information sources. However (and, again, a caveat regarding sample size), we found no evidence of this, suggesting that the website has value both as a standalone tool and as a tool to be used in conjunction with the NCHDA report. Finally, although the website improves people’s understanding of these data, we can see from Table 14 that it does not lead to a complete understanding. Therefore, it might be valuable to explore whether or not the website has further value when used as a teaching tool, for instance through training members of cardiac teams or family support groups to use it to guide members of the public towards a better understanding of these data.

Launch strategy

Sense about Science (EJ/JT) led the website launch strategy and implementation. The audiences identified for user testing were a reflection of the target audience for launch:

1. parents of children with heart conditions

2. professionals:

   1. health professional (including surgeons, family liaison staff at hospitals, cardiac nurses)

   2. NHS commissioners

   3. policy professionals (e.g. at learned and professional societies)

   4. communications professionals, journalists, science writers

   5. opinion leaders (e.g. celebrity GPs)

   6. medical charities

   7. organisations interested in good risk communication

3. members of the public interested in how the NHS monitors heart surgery.

Professionals who make decisions, write reports and answer journalists’ queries about these data constituted a key audience through which future miscommunication could be reduced. One way of reaching the policy and health-care professional audience, at the same time as reaching a public audience, was through national press outlets. The second way to reach them was via their membership bodies and academic publications.

Summary of coverage received, interest and dissemination

Media response to the launch was very good considering the limited space available as a result of coverage of the lead-up to the UK’s European Union referendum (which was held on 23 June 2016). This problem would have been even worse had the launch taken place after the referendum. Coverage of the launch is summarised as follows.

• Press coverage (dissemination to professional, parents and interested public):

  • CP and DS wrote a guest blog in The Guardian. 65

  • BBC Radio 4’s World at One programme on 21 June 2016 featured DS and Alex Smith, a parent of a child with a heart condition.

  • BBC Radio Cambridge interviewed DS on 22 June 2016.

  • EurekAlert! (press wire). 66

• Science weeklies (dissemination to research community and clinicians):

  • An article in British Medical Journal. 67

  • An editorial in The Lancet. 68

• Medical charity dissemination (parents of children with heart conditions):

  • CHF explained on their website that ‘parents’ views have been at the heart of developments that today launch a parent-led tool opening up NHS children’s heart surgery data to families’. 69

  • Children’s Heart Surgery Fund: ‘Understanding Children’s Heart Surgery Outcomes’. 69

• Civil society organisations (interested members of the public and parents of children with heart conditions):

  • A parent participant wrote a blog on Mumsnet. 70

• Medical charity membership organisation (to reach medical charities to encourage others to take on our approach of involving audience early):

  • Sense about Science wrote about a blog for the Association of Medical Research Charities about why it was important to involve patients and families, and to involve them early, in the website’s creation. 71

• Funders/academic institutions/statistical organisations (to reach researchers with tips for public engagement and how to communicate risk well):

  • CP wrote a blog for NIHR. 72

  • The Royal Statistical Society wrote a blog about the project. 73

  • In a KCL news piece, TR shared his reflections on the project. 74

  • DS wrote a blog on the language of chance. 75

• Learned and professional societies (to reach surgeons with a useful tool to improve their understanding and to signpost to patients/colleagues):

  • The Royal College of Surgeons of Edinburgh added a news item to their website. 76

  • Royal College of Surgeons included information on the project in their newsletter to members and fellows.

Post launch

We track usage of the site using Google Analytics (Google, Inc., Mountain View, CA, USA). 77 After an initial surge in hits in the first week after launch, the website received an average of about 70 visits per day in July, 67% of which were from first users, and a bounce rate of 37%, which is considered very good. 78 Most visitors (70%) have been from the UK, but the site has also received international interest, particularly from the USA and Australia. Although most views have been via desktop, > 30% of people viewing the site have been using smartphones or tablets (as of December 2016), confirming the importance of designing the site with usage from non-desktop platforms in mind.

As of December 2016, and not including the substantial activity after its launch (over 5000 visitors between mid-June and August), the website has been receiving 400–600 visitors a month (3 months’ data: September–December 2016), with 67% new visitors, a bounce rate of 35%, and 54% of visitors from the UK. The proportion of new visitors has been steady, with 20% of new visitors spending > 3 minutes and 12% of visitors spending > 10 minutes on the site. If the family of every child undergoing heart surgery each year looked at the website, we would expect approximately 375 new visitors per month (there are approximately 4500 surgical episodes a year in the UK). Obviously we do not know if all of our UK site visitors are such families, but our new UK visitor rate of around 145–220 per month suggests that at least some are finding and viewing the website.

Importantly, in addition to the endorsement and promotion listed in Summary of coverage received, interest and dissemination, we have made progress in achieving long-term visibility through key endorsements and permanent links.

In particular, NICOR have discussed the website and now link to it from the NICOR NCHDA congenital page. 32 NICOR also said that they would include the website as a resource in press releases about future annual reports, which will provide an important signpost to journalists seeking to understand the results.

The NICOR congenital steering committee (NCHDA) also sent an e-mail to each hospital that carries out heart surgery on children to suggest that they add a link to the website on their web pages. Almost all of the charity and user groups listed on the website were enthusiastic and agreed to include permanent links to the website (at least five charities have already done so). 79 Another important source is NHS Choices, which has added the link to its pages on CHD. 80

Details of the website, as a trusted independent resource, were included in the letter sent to parents of children treated at Bristol Royal Infirmary, in advance of the Care Quality Commission inquiry report at the end of June 2016. We also shared the website with the Health Quality Improvement Partnership, which provided positive feedback and in turn shared the website with Professor Sir Bruce Keogh, NHS England’s medical director. 81–83

Key learning points

In our final facilitated meeting in June 2016, we distilled some key learning points for different groups of stakeholders. Our thoughts are summarised in the following sections.

Capturing impacts going forward

Although this project is over, the website will be updated annually with the latest national audit report data. In terms of utilisation, we (MP/CP/Sense about Science) will monitor for:

• an uptick in traffic after the publication of a NCHDA report

• an uptick in traffic after the publication of news stories about children’s heart surgery in the UK and Ireland

• an increase in the number of permanent links to the website.

More hard-to-measure impacts that we hope to see over the coming years are a change in the language used about survival after children’s heart surgery (e.g. in the NCHDA report and/or in media stories), extending perhaps to the reporting of other published hospital outcomes.

Another hard-to-measure but interesting potential impact would be to track other efforts utilising the approach, ideas, presentation and/or layout that we used for the website. For instance, the approach may be used to communicate information about other clinical procedures. Sense about Science has offered to keep track of examples of such potential diffuse impacts.

Generalisable learning

The key generalisable aspects from aim 1 are:

• Close collaborative working between clinical experts and analysts improves both the performance and the acceptability of risk adjustment methods. The latter in particular is crucial for translating research into practice.

• Comorbidity and diagnosis information are increasingly recognised as important determinants of risk. Our careful and thorough approach to categorising these features for complex paediatric patients provides a template for other clinical specialties. In particular, although we have considered the paediatric cardiac population, many of the considerations around defining comorbidity will be directly relevant to other paediatric specialties.

• As 30-day survival continues to improve, focus is shifting onto other outcomes for children with CHD, such as complications after surgery and longer-term survival. Comorbidity and diagnosis will be important modifiers for these outcomes and the categorisations developed for this project will be directly transferable to consideration of these other outcomes.

The key generalisable aspects for aim 2 are:

• the coproduction workshop methodology used to build the website

• the learning from independent psychology experiments that fed into the key messages of the web material

• the need to build in sufficient time to respond and reflect at each iteration

• the language we developed to describe sophisticated mathematical concepts around prediction, uncertainty and risk.

Given that the current trend is towards greater transparency and the increased publication of outcomes across many more clinical specialties, this project’s learning could be very important to others in the coming years. Indeed, in the short time the website has been live, we have been approached by other organisations (one surgical, one intensive care) for advice on how they can learn from and replicate our experience.

Future recommendations

This project focused on a single surgical outcome – mortality – and on building both a better statistical model with which to monitor surgical mortality in children with heart disease and a website to try to ensure that the way in which surgical mortality is monitored in this specialty is transparent for both families and the public. As discussed previously, survival rates for children with CHD have increased dramatically (30-day survival is now > 97.5%), but CHD remains a lifelong condition requiring lifelong engagement with clinical services managed by various health-care providers. Patients often need a complex combination of medicines, hospital visits and treatments, leading to high levels of health-care utilisation. 85–89 CHD services are expensive and high profile and have an enormous impact on patients’ lives, but there is currently no cohesive strategy to measure outcomes across settings and providers. Thus, 30-day survival in children after cardiac intervention is no longer an adequate proxy on its own for quality in CHD services. 85,90–92

National reviews have identified areas of CHD services in which improvements could be made but for which there is no adequate system for objectively defining, measuring, acting on and reporting on these areas. 6,92 Examples are reducing post-discharge deaths (3.2% for infants); addressing barriers to accessing care in the community; reducing complications following surgery; reducing loss to follow-up after hospital discharge, particularly when moving from paediatric to adult services; and improving reporting of outcomes for adult patients. 36,41,42,93–97 The 2015 NHS England review of CHD services called for better reporting of quality using a broader range of outcomes relevant to a range of stakeholders, including parent and adult patient groups. 92

The facets of quality given above are merely examples, and different stakeholders will have different perspectives on the importance of different possible quality criteria. Even given a set of criteria that has broad support, identifying or developing appropriate metrics for monitoring it is far from straightforward. In order to prevent gaming, irrelevance, anger or measurement overload, metrics would need to be carefully developed with their purpose and audience in mind.

At the same time, how the metrics relate to quality criteria, how they are used and what they mean needs to be communicated to patients, the media, policy-makers and the public in a transparent, accessible and informative way. 91,98–102 The website we developed in aim 2 addresses some of these reporting and communication challenges, but there is little doubt that it could be improved, and any future attempts should incorporate more extensive post-launch follow-up and evaluation.

Thus, we see that the major future recommendations for research are addressing the following questions:

• What are the important facets of quality in CHD services and how do perspectives on these differ between stakeholder groups?

• Can identified quality criteria be measured routinely and meaningfully, and what are the sets of metrics that can be used for national quality assurance, local quality improvement and transparent reporting to patients, commissioners and the public?

• Does codeveloping metrics with a broad stakeholder group significantly change the type of metrics developed and improve their acceptability within the CHD community?

• What visual representations of quality measures are effective in facilitating decision-making regarding CHD services and what are the implications for other contexts?

• What are the most useful data representations and communication tools for sharing outcomes with patients and the public?

• What facilitates the routine use of monitoring software tools (such as the PRAiS tool) in practice, and what are the implications for national roll-out?

• How can we better evaluate how successful we are at communicating quality to patients, the media and the public?

• Can and should public reporting of quality in NHS services be multiplatform (e.g. website, paper, applications)?

• Can we create a commonality of quality reporting across different subspecialties in the NHS?

Non-procedural risk factor groups

Based on consensus, and in consideration of the grouping options, relevant EPCC codes and review of relevant literature, the expert panel suggested categories of:10,16

Down syndrome, congenital comorbidities excluding Down syndrome, acquired comorbidities, prematurity and conditions indicating increased severity of illness in surgical candidates. The conditions included within the category of increased severity of illness incorporated pre-operative mechanical ventilation, which was included as an acquired comorbidity in PRAiS 1 as the expert panel considered that severity of illness group was a better fit; and other measures of illness severity that feature in the updated version of the STS STAT score, such as pre-operative mechanical circulatory support. 16 These clinical groups of pre-operative risk factors contain conditions that are separated based on distinct aetiologies, and as such have the potential to enable discrimination of risk across the diverse case mix of paediatric cardiac surgery. Figure 76 shows the EPCC categories where the individual conditions in each category originate; for example, most conditions in congenital non-Down syndrome comorbidity are ‘Diagnostic comorbidities: hereditary, neonatal and non-cardiac codes’ and most conditions in acquired comorbidity are from ‘Diagnostic codes for Preoperative risk factors’ and ‘Diagnostic codes for lung and thoracic disease’.

FIGURE 76.

How categories of EPCC codes map to the new comorbidity and additional risk factor categories.

The panel noted evidence that additional or acquired cardiac specific conditions, in particular elevated pulmonary vascular resistance and impaired ventricular function, may place surgical candidates at increased risk, and therefore it proposed the formation of a further category of additional cardiac risk factors. 105–107 As is shown in Figure 76, conditions within the category ‘Diagnostic codes for acquired and post-procedural abnormalities or disorders’ (acquired cardiac conditions) were predominant in this group. 37 Codes were put forward for inclusion as additional cardiac risk factors only when they were considered to have an unambiguously negative impact in any surgical candidate regardless of the primary cardiac diagnosis (e.g. cardiomyopathy, supra-systemic pulmonary artery pressure), and codes were ruled out when it was considered that, clinically, the impact of the condition under consideration might be variable from one patient to another and might even be the primary indication for surgery (e.g. pulmonary regurgitation) or when a code highly specific to a given primary diagnosis that was already a factor in the risk model (intramural coronary in transposition). 108–110

Both PRAiS 1 and PRAiS 2 include the risk factors of primary diagnosis and univentricular status, and hence the expert panel elected not to consider prior cardiac surgery as a risk factor given its overlap with these features.

A total of 23 codes included in PRAiS 1 were dropped on the basis that they were non-clinically relevant or were extremely rare (e.g. pes carinatum, abdominal aortic aneurysm). The code for gastero-oesophageal reflux, which was included in PRAiS 1, was dropped because it was considered to be too challenging to define consistently.

The number of individual EPCC codes (some of which denote the same condition with slight variations depicted by different individual codes) within each risk factor group was as follows: Down syndrome (one relevant EPCC code), congenital comorbidities excluding Down syndrome (51 relevant EPCC codes), acquired comorbidities (37 relevant EPCC codes), prematurity (three relevant EPCC codes) and conditions indicating increased severity of illness (nine relevant EPCC codes) and additional cardiac risk factors (40 relevant EPCC codes). The expert panel felt that a simple count of comorbidities could lead to inflated predicted risk if several comorbidity codes for similar conditions were used, when the actual additional risk from the extra codes would be minimal. For this reason it was decided not to include an overall count of comorbidities or a count within different categories. A yes/no indicator for each different category of additional risk meant that predicted risk would be increased if a surgical procedure episode contained different types of comorbidities or additional risk factors, while meaning that records with several similar additional risk factors from the same group did not have any additional predicted risk.

Risk factor frequencies and outomes

The frequency with which the selected risk factors were found within the study data set is shown in Table 15 alongside the mortality rate for surgical episodes with the stated risk factor, whether or not other risk factors were also present. The unadjusted odds ratios indicated that the univariate risk of post-operative death was significantly higher for congenital and acquired comorbidity, severity of illness indicator, additional cardiac risk factors and prematurity. Conversely, univariate risk of post-operative death was signficantly lower in surgical episodes where Down syndrome was recorded. When multivariable risk of post-operative mortality was considered, based on PRAiS 2 (adjusted for specific procedure, primary cardiac diagnosis, univentricular status, weight, age, congenital and acquired comorbidity, severity of illness indicator and additional cardiac risk factors), there was no longer evidence for increased risk of post-operative death with prematurity, but the other four factors remained statistically significant.

The broad underpinning definitions for the significant risk factors of congenital comorbidity, acquired comorbidity, severity of illness indicator, additional cardiac risk factors, including the time lines for their identification, are presented in Table 16. These individual risk factors are identified by the presence of specific EPCC codes in the NCHDA data set that were clinically judged as meeting the specified definition, the most common of which are listed in Table 16. The full list of individual EPCC codes selected for inclusion within these additional risk factor groups is presented in Report Supplementary Material 2.

As expected, additional risk factors arose in various combinations within individual surgical episodes, and the proportion of surgical episodes in the data set across the range of possible combinations is shown in Table 17. There were 16,093 (73.7%) surgical episodes with no additional risk factors of any kind, and only 10 surgical episodes (> 0.1%) with all four risk factors present. The most common risk factor as a single condition was congenital comorbidity in 1881 (8.6%), followed by severity of illness indicator in 1538 (7.0%), additional cardiac risk factors in 656 (3%) and, last, acquired comorbidity in 575 (2.6%) episodes. The most common combination of two risk factor groups was acquired comorbidity and severity of illness indicator, in 274 (1.3%) episodes.

Within the study data set, the proportion of surgical episodes in the neonatal age group was 21.6%, in infancy was 39.8% and in childhood was 38.7%; and the proportion of surgical episodes involving a functionally UVH was 17.1% versus 82.9% involving a biventricular heart (Table 18). Notable differences across the age groups were the greatly increased proportion of surgical episodes with a severity of illness risk factor pre-operatively in the first year of life versus the much lower proportion in older children. Furthermore, congenital comorbidity was more commonly noted with increasing age, most likely because certain conditions may take time to identify, and congenital comorbidity was more common in biventricular children than in patients with functionally UVHs, probably reflecting the distribution of particular specific known combinations of cardiac and non-cardiac congenital defects.