Development, implementation and evaluation of an early warning system improvement programme for children in hospital: the PUMA mixed-methods study

Search strategy

A comprehensive search was conducted across a range of databases to identify relevant studies in the English language. Published and unpublished literature was considered if publicly available, as were studies in press. The following databases were searched from inception to May 2018: British Nursing Index, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Cochrane Central Register of Controlled Trials, Database of Abstracts of Reviews of Effects, EMBASE, Health Management Information Consortium (HMIC), MEDLINE, MEDLINE In-Process, Scopus and Web of Knowledge (Science Citation Index). To identify additional papers, either published or unpublished, or research reported in the grey literature, a range of relevant websites and trial registers were searched, including ClinicalTrials.gov. To identify published papers that had not yet been catalogued in the electronic databases, recent editions of key journals were hand-searched. The search terms included ‘early warning scores’, ‘alert criteria’, ‘rapid response’, ‘track and trigger’ and ‘early medical intervention’. See Appendix 2, Tables 20 and 21, for the search strategies and results.

Eligibility screening and study selection

Population, intervention, control/comparison, outcomes (PICO) parameters guided inclusion criteria for the validation and effectiveness studies (see Appendix 3, Tables 22 and 23). Papers reporting development of validation of a PTTT were included for review 1, whereas papers reporting the implementation of any broader ‘paediatric early warning system’ (with or without a PTTT) were eligible for review 2. Both reviews were limited to studies that involved inpatients aged 0–18 years. Outcome measures considered were mortality and critical events, including: unplanned admission to a higher level of care, cardiac arrest, respiratory arrest, medical emergencies requiring immediate assistance, children reviewed by PICU staff on the ward (in specialist centres) or reviewed by external PICU staff (for non-specialist centres), acuity at PICU admission and PICU outcomes. A range of study designs was considered for both reviews.

Two of the review authors independently screened the titles and abstracts yielded in the search. Full texts were reviewed independently by six reviewers against the above eligibility criteria and were assigned to the relevant review question if included. Reasons for exclusion were recorded. Separate data extraction forms were developed for validation and effectiveness studies. The forms had common elements (study design, country, setting, study population, description of the PTTT or early warning system, statistical techniques used, outcomes assessed). Additional data items for validation studies included the items in the PTTT, modifications to the PTTT from previous versions, predictive ability of individual items and the overall tool, sensitivity and specificity and inter-rater and intrarater reliability. Effectiveness studies included an assessment of outcomes in terms of mortality and various morbidity variables. Two reviewers carried out data extraction; discrepancies were resolved by discussion. For effectiveness studies, effect sizes and 95% confidence intervals (CIs) were calculated or reported as risk ratios (RRs) or odds ratios (ORs) as appropriate, with p-values reported to assess statistical significance. Data analysis was conducted using an online medical statistics tool.

Quality appraisal

The methodological quality and risk of bias were assessed for each included study using a modified version of the Downs and Black rating scale29 (see Appendix 4, Tables 24 and 25).

Study design

We performed a hermeneutic systematic review of the relevant literature. A hermeneutic systematic review is an iterative process, integrating analysis and interpretation of evidence with literature searching and is designed to develop a better understanding of the field. 30 The popularity of the method is growing in health services research, for which it has value in generating insights from heterogeneous literatures that cannot be synthesised through standard review methodology, and which would otherwise produce inconclusive findings. 21,31 The purpose of the review was not exhaustive aggregation of evidence, but to develop an understanding of the social, material and contextual factors associated with successful or unsuccessful paediatric early warning systems.

Theoretical framework

Translational mobilisation theory and NPT informed the data extraction strategy, interpretation of the evidence and the development of a propositional model of the minimal conceptual requirements of a paediatric early warning system. 25,27,28,32

Focus of the review

The literature in this field identifies four integrated components that work together to provide a safety system for at-risk patients: (1) the afferent component, which detects deterioration and triggers a response; (2) the efferent component, which consists of the people and resources providing a response; (3) a process improvement component, which includes system auditing and monitoring; and (4) an administrative component focusing on organisational leadership and education required to implement and sustain the system. 33 Our focus was limited to the afferent components of the system.

Stages of the review

Data collection

A customised online PUMA database was created for site staff to upload monthly data forms. Staff were able to log in to the database via a password-protected home screen and were only ever able to access and submit forms for their own site. Data were uploaded by either principal investigators (PIs) or research nurses, and all staff responsible for entering data at each site were trained in using the database prior to entering data. Each monthly submission was quality assessed in real time by a member of the PUMA study team to allow timely resolution of any data queries or missing values.

Outcome measures

A provisional set of outcome measures was drawn up based on preliminary findings from systematic reviews 1 and 2. As part of the systematic reviews, an evaluation was conducted of the most commonly used outcome metrics reported in the literature for assessment of the validity and effectiveness of PTTTs and paediatric early warning systems. The feasibility of collecting these outcomes at each site was explored through preliminary piloting work, prior to commencement of data collection in August 2015.

Appendix 8, Table 26, shows the final outcomes that were selected as the most suitable proxies for inpatient deterioration, and the definition of each outcome that was agreed with the four sites. It is important to note that these cannot be used, in any way, to infer the processes leading to that outcome; for example, it is impossible to determine if the deaths were the result of missed deterioration or an unavoidable consequence of a disease process.

Timing of outcome assessments

Primary and secondary outcomes, and the patient bed-day denominator, were entered by site staff into the PUMA database on a monthly basis at each of the four sites. Table 3 summarises the various data points for each of the hospitals, for the pre-implementation, implementation and post-implementation stages.

Sample size calculation

We used a simulation-based approach to calculate power, as it was challenging to calculate an accurate sample size. 36,37 Our initial calculation was based on the original aim of the study to implement a PTTT in each site.

We obtained historical data for one tertiary centre (Alder Hey) and one DGH (Morriston Hospital). These data showed a 1% prevalence (i.e. 10 events per 1000 patients) for unplanned transfers to a PICU. Tibballs et al. 38 have previously shown that the implementation of paediatric calling criteria with a rapid response team resulted in a RR of 0.65 in terms of total avoidable hospital mortality. We assumed that the PUMA intervention might result in a similar RR of 0.65. 38 The monthly recorded data for unplanned admission in both hospitals were used to estimate monthly death rate pre and post intervention. From obtained data, the estimated effect size, mean difference and common standard deviation were 2.8, 2.0 and 0.7, respectively. We estimated that we would have 90% power with a total of 24 months of observations (12 pre and 12 post intervention) for an effect size of at least 2. 37

The initial aims of the study were changed from the implementation of PTTT to a complex intervention: the PUMA programme. We retained the focus on collecting data 12 months pre and 12 months post intervention, but allowed 12 months for phase-in of the intervention, to give a total of 36 months. During the lifetime of the project, we were able to collect up to 6 additional months of data retrospectively for the pre-intervention period. This gave us 42 months of data and increased the sample size.

Data analysis

An ITS analysis was used to assess the effectiveness of the PUMA programme at reducing rates of inpatient deterioration over time. An ITS approach allows exploration of the longitudinal effect of the intervention through regression modelling. This approach controls for pre-intervention trends and assesses the extent to which an intervention ‘interrupted’ the trajectory of this trend. 39 See Appendix 9, Figure 27, for further details about an ITS approach.

The most common approach to an ITS analysis is to compare trends across two separate time periods: a pre- and a post-intervention phase. Typically, the intervention being studied is relatively clear-cut, for example a change in national policy, which would be expected to have an immediate effect (i.e. level change) on the outcome being studied. In this study, however, we expected that the complete implementation of the PUMA programme was likely to take longer, but that we might be able to observe gradual changes in measures of inpatient deterioration. Therefore, we decided to investigate both the short-term effect of the PUMA programme (by comparing pre-intervention levels in outcomes with implementation and post-intervention period levels in outcomes) and the longer-term effect (by exploring trends in outcomes during the implementation and after the intervention, in the post-intervention period, when teams would have had some time to embed local initiatives). Data from each hospital were analysed separately as independent case studies.

Overview

Embedded ethnographic case studies were undertaken on one ward in each of the four hospitals. Qualitative methods (observation, interviews and documentary analysis) were deployed to undertake a pre- and post-implementation review of the local paediatric early warning system in everyday clinical practice.

The aim of the pre-implementation stage was to understand current practice at baseline: evaluating the paediatric early warning system in practice and observing how the system was shaped by local context. The aim of the post-implementation stage was to evaluate any changes to the paediatric early warning system after implementation of the PUMA programme, in order to understand the impact of the intervention.

Case study wards

In contrast to the quantitative analysis, which summarised aggregate outcome measures at a hospital-wide level, the ethnographic case studies were conducted on a single ward. Table 4 summarises the wards selected at each site.

Data collection

Table 5 shows the qualitative data collected at each case study site. In both the pre- and post-implementation phase, data were generated through:

• ethnographic observation of everyday practice (by shadowing individuals – nurses, doctors, support staff – and discussing their practice, and attending key meetings and events)

• interviews with clinical team members, service managers, PIs and family members or carers

• analysis of relevant documents and artefacts.

Observations were conducted at different times of day/night and on different days of the week, including weekends, to ensure that a range of time periods were covered. Our concern was with understanding the network of actors (people, processes, technologies and artefacts) and their inter-relationships in each paediatric early warning system. Drawing on the theoretical framework and the systematic review findings, we developed a template to guide the observations and interviews as shown in Figure 1 (see Report Supplementary Material 2). However, data generation was not absolutely constrained by this; rather, in each case, the strategy was to ‘follow the actors’ (human and non-human). This ensured that there was a consistent approach across case studies to facilitate comparative analyses, but flexibility to modify data generation in response to the singular features of each site. We focused on what participants did, the tools they used, the concepts they deployed and the factors that facilitated and constrained action. Adopting a TMT lens directed attention to the sociomaterial relationships within each paediatric early warning system and how the local institutional context conditioned the possibilities for action. 25,26

FIGURE 1.

Observations template.

In addition, we also undertook a series of semistructured interviews with parents/carers to explore their views and experiences, and semistructured interviews with a sample of clinical staff and relevant service managers.

Ethnographic observations and embedded interviews were recorded contemporaneously as low inference-style field notes and expanded on as soon as possible after the data were collected. Staff interviews were digitally recorded with consent and organised to take place either in private offices or by telephone. Interviews with an opportunistic sample of parents who had a physiologically unstable child were undertaken when the child was still an inpatient, but at a time when their condition was considered by clinical staff to be stable. For the purposes of this study, we did not include parents whose child had died, but we interviewed parents whose child had (1) been monitored only, (2) received intervention to prevent deterioration or (3) experienced a critical event. Documents/records were treated as both a resource and a topic. Their content was analysed to inform our understanding of organisational processes and practices. Their form was analysed to develop a better understanding of their design, affordances and inter-relationships.

We replicated this ethnographic process (both non-participant observations and interviews) in the post-implementation period, modifying the interview style and content, as well as the primary focus of the observations, to explore, in detail, staff experiences of the PUMA programme, changes to the system, factors consequential for impact and any unintended consequences. We also reassessed the paediatric early warning system using a structured template based on the PUMA Standard as a guide to observation, in order to analyse changes in these relationships brought about by the improvement programme, and the implications this had for normalisation.

Analysis

At all stages, data collection and analysis were undertaken concurrently, facilitating a progressive narrowing of focus designed to develop an in-depth understanding of the paediatric early warning systems and the implications of the improvement programme for practice. The various materials collected (field notes, interview transcripts and documents) were coded using a common framework and used to develop concrete descriptions of relevant aspects of paediatric early warning systems, which were mapped onto the PUMA Standard. Local PIs also contributed to this sense-making process.

Analysis was undertaken in two main stages:

• Stage 1 involved developing a description and analysis of the pre- and post-implementation paediatric early warning system in each hospital and the implementation process. This entailed the development of richly descriptive accounts, extending to up to 25,000 words, which were then subject to further analysis, refined and condensed into summaries for the purposes of the report.

• Stage 2 involved cross-case analysis to understand the relationship between the intervention, context, mechanisms and outcomes to inform the extension and development of the PUMA programme.

Overview

The process evaluation focused on the implementation of the PUMA programme in order to understand participants’ experiences, but also to identify where and how the programme might be strengthened.

Data

Analysis

Analysis was thematic, focusing on delivery and response to the core components of the PUMA programme, communication of PUMA, understanding of PUMA, barriers to change and implementation, facilitators of change and implementation, and sustainability.

Outcome measures

Paediatric track-and-trigger tools were evaluated for their ability to predict a wide range of clinical outcomes. Composite measures were used in eight studies,17,54,61,63,65,77–79 cardiac/respiratory arrest or a ‘code call’ was used (singularly or as part of a composite outcome) in six studies,18,54,60,61,77,78 and 22 studies used transfer to a PICU or a paediatric HDU as the main outcome. 17,19,50,54,59–62,64–66,68,71,73–77,79–81

Predictive ability of individual paediatric track-and-trigger tool components

Three validation papers reported on the performance characteristics of individual components of the tool for predicting adverse outcomes. 19,65,73 Parshuram et al. ,19 for instance, reported area under receiver operating characteristic (AUROC) curve values for individual PTTT items of a pilot version of the Bedside PEWS: ranging from 0.54 (bolus fluid) to 0.81 (heart rate), compared with 0.91 for the overall PTTT. 19 All other studies reported outcomes for the PTTT as a whole.

Paediatric early warning system score

The predictive ability of the 16-item PEWS was assessed by one internal (AUROC = 0.90)18 and two external case–control studies (AUROC range 0.82–0.88)60,61 with a range of outcome measures and scoring thresholds. One case–control study18 used an observed prevalence rate to calculate a positive predictive value (PPV) of 4.2% for the tool in predicting code calls (i.e. for every 1000 patients triggering the PTTT, 42 would be expected to deteriorate).

Bedside Paediatric Early Warning Score and derivatives

The Bedside PEWS was evaluated in one internal (AUROC = 0.91)19 and five external case–control studies (AUROC range 0.73–0.90)50,60,61,76,79 for a range of different outcome measures and at different scoring thresholds. One case–control study50 calculated a PPV of 2.1% for identifying children requiring urgent PICU transfer within 24 hours of admission, based on locally observed prevalence rates. A modified version of the Bedside PEWS (with temperature added) demonstrated an AUROC of 0.86 in an external case–control study with a composite outcome of death, arrest or unplanned PICU transfer. 61

Brighton Paediatric Early Warning Score and derivatives

Six different PTTTs based on the original Brighton PEWS were evaluated across 11 studies,50,61,64,69,71–74,77,78,81 The Modified Brighton PEWS (a) was evaluated for its ability to predict PICU transfers in one large prospective cohort study (AUROC = 0.92, PPV = 5.8%),64 and an external case–control study tested the same score for predicting urgent PICU transfers within 24 hours of admission (AUROC = 0.74, PPV = 2.1%). 50

An external case–control study used a composite measure of death, arrest or PICU transfer to evaluate the Modified Brighton PEWS (b) (AUROC = 0.79) and the Modified Brighton PEWS (d) (AUROC = 0.74). 61 The latter tool was evaluated in a further internal case–control study for predicting PICU transfer (AUROC = 0.82). 80

The Children’s Hospital Early Warning Score had a reported AUROC of 0.90 for predicting PICU transfers or arrests in a large internal case–control study. 69 A modification for cardiac patients, the Cardiac Children’s Hospital Early Warning Score (C-CHEWS), was evaluated by one internal study on a cardiac unit (AUROC = 0.90)77 looking at arrests or unplanned PICU transfers, and by two external studies of oncology/haematology units for the same outcome (AUROC = 0.95). 73,74 Finally, the Children’s Hospital Los Angeles PEWS was evaluated in a small internal case–control study for prediction of re-admission to PICU after initial PICU discharge (AUROC = 0.71). 72

Melbourne Activation Criteria and derivatives

The MAC was assessed by one external case–control study with an outcome of death, arrest or unplanned PICU transfer (AUROC = 0.71)61 and by a large external cohort study with an outcome of death or unplanned PICU or HDU transfer (AUROC = 0.79, PPV = 3.6%). 65 A derivative of the MAC, using an aggregate score, the Cardiff & Vale Paediatric Early Warning Score (C&VPEWS), was tested using the same cohort and outcome measures as used in a previously mentioned external study (AUROC = 0.86, PPV = 5.9%)17 and was the best-performing PTTT in an external case–control study evaluating multiple PTTTs (AUROC = 0.89). 61

Bristol Paediatric Early Warning Tool

The Bristol Paediatric Early Warning Tool was evaluated by five external validation studies: two chart review studies (no AUROC);59,67 one small cohort study of PICU transfers (AUROC = 0.91, PPV = 11%);66 and two case–control studies looking at code calls (AUROC = 0.75)60 and a composite of death, arrests and PICU transfers (AUROC = 0.62). 61

Other paediatric track-and-trigger tools

The NHS III PEWS was tested by one external cohort study looking at a composite of death or unplanned transfers to PICU or HDU (AUROC = 0.88, PPV = 4.3%)63 and one external case–control study looking at a composite of death, arrests and PICU transfers (AUROC = 0.82). 61 Zhai et al. 50 developed and retrospectively evaluated a logistic regression algorithm in an internal case–control study looking at urgent PICU transfers in the first 24 hours after admission (AUROC = 0.91, PPV = 4.8%).

Across PTTTs, studies reporting performance characteristics of a tool at a range of different scoring thresholds demonstrate the expected interaction and trade-off between sensitivity and specificity: at lower triggering thresholds, sensitivity is high but specificity is low; at higher thresholds, the opposite is true.

Inter-rater reliability and completeness of data

Accurate assessment of the ability of a PTTT to predict clinical deterioration is contingent on accuracy and reliability of tool scoring (whether by bedside nurses in practice or by researchers abstracting data) and the availability of underpinning observations. Only five papers made reference to accuracy or reliability of scoring,60,64,73,77,78 with mixed results; for example, two nurses separately scoring a subset of patients on the Modified Brighton PEWS (a) achieved an intra-class coefficient of 0.92,64 but a study nurse and bedside nurse achieved only 67% agreement in scoring the C-CHEWS. 77 Completeness of data was reported in 11 studies. 17–19,50,61–63,65,73,76,78 An evaluation of the Modified Bedside PEWS (a) reported that ‘the PEWS was correctly performed and could be used for inclusion in the study’ in 59% of cases,62 a prospective study bench-testing the C&VPEWS found an average completeness rate of 44% for the seven different parameters in daily practice,17 and a multicentre study of the Bedside PEWS reported that ‘only 5.1% [of observation sets] had measurements on all 7 items’. 76

Box 1 presents a summary of the review 1 findings.

Type of early warning system interventions

Seventeen interventions involved the introduction of a new PTTT,38,46–49,84,96–107 one intervention introduced a mandatory triggering element to an existing PTTT106 and one study reported a large, multicentre analysis of MET introduction with no details on PTTT use. 108 Twelve interventions included the introduction of a new MET or RRT,38,46–49,84,96–100,104 and four further interventions87,89,100,106 introduced a new PTTT in a hospital with an existing MET or RRT. Therefore, only three studies evaluated a PTTT in the absence of a dedicated response team. 102,103,105 A staff education programme was explicitly described in 10 interventions. 38,46,48,84,97,98,102,103,105,107

Of the 18 studies that used a PTTT, only seven used a tool that had been formally evaluated for validity: three used the Bedside PEWS,84,100,105 two used the MAC,38,98 one used the Modified Brighton PEWS (b)107 and one used the C-CHEWS. 102 One study did not report the PTTT used,97 and 10 studies used a variety of calling criteria and local modifications to validated tools that had not been evaluated for validity. 46–49,96,99,101,103,104,106

Mortality (ward or hospital wide)

Two uncontrolled before-and-after studies (both with a MET/RRT) reported significant mortality rate reductions post intervention: one in hospital-wide deaths per 100 discharges (RR 0.82, 95% CI 0.70 to 0.95)48 and one in total hospital deaths per 1000 admissions (RR 0.65, 95% CI 0.57 to 0.75) and deaths on the ward (‘unexpected deaths’) per 1000 admissions (RR 0.35, 95% CI 0.13 to 0.92). 98 Seven studies found no reductions in mortality, including two high-quality multicentre studies. 38,46,84,96,99,100,108 Parshuram et al. 84 conducted a cluster randomised trial and found no difference in all-cause hospital mortality rates between 10 hospitals randomly selected to receive an intervention centred around use of the Bedside PEWS and 11 usual care hospitals 1 year post intervention (OR 1.01, 95% CI 0.61 to 1.69). Kutty et al. 108 assessed the impact of MET implementation in 38 US paediatric hospitals with an ITS study and reported no difference in the slope of hospital mortality rates 5 years post intervention and the expected slope based on pre-implementation trends (OR 0.94, 95% CI 0.93 to 0.95).

Paediatric intensive care unit mortality

Two uncontrolled before-and-after studies (both with a MET/RRT) reported a significant post-intervention reduction in rates of PICU mortality among ward transfers (RR 0.31, 95% CI 0.13 to 0.72),49 and PICU mortality rates among patients re-admitted within 48 hours (RR 0.43, 95% CI 0.17 to 0.99). 99 Six studies (including a high-quality cluster randomised trial and an ITS study) reported no post-intervention change in PICU mortality rates using a variety of metrics. 84,100–104

Cardiac and respiratory arrests

Two uncontrolled before-and-after studies (both with a RRT/MET) reported significant post-intervention rate reductions in subcategories of cardiac arrests: one in ‘near cardiopulmonary arrests’ (RR 0.54, 95% CI 0.52 to 0.57),99 but not in ‘actual cardiopulmonary arrests’, and one in ‘preventable cardiac arrests’ (RR 0.45, 95% CI 0.20 to 0.97),98 but not ‘unexpected cardiac arrests’. One uncontrolled before-and-after study (with a RRT/MET) reported a significant post-intervention reduction in rates of ward respiratory arrests per 1000 patient-days (RR 0.27, 95% CI 0.07 to 0.95). 47 Seven studies (including one high-quality cluster randomised trial and one high-quality ITS study) found no change in cardiac arrest rates using a variety of metrics,38,46,47,84,97,100 or cardiac and respiratory arrests combined. 96

Calls for urgent review/assistance

Two uncontrolled before-and-after studies (all with RRTs/METs) reported significant post-intervention reductions in the rates of code calls (RR 0.29, 95% CI 0.10 to 0.65;48 RR 0.71, 95% CI 0.61 to 0.8399), whereas three studies found no change in the rate of code calls. 46,49,107 One uncontrolled before-and-after study in a community hospital (without a RRT/MET) found significant post-intervention reductions in rates of urgent calls to the in-house paediatrician (RR 0.23, 95% CI 0.11 to 0.46) and respiratory therapist (RR 0.36, 95% CI 0.13 to 0.95). 105 Two uncontrolled before-and-after studies (with RRTs/METs) found increases in the rates of RRT calls (RR 1.59, 95% CI 1.33 to 1.90)107 and outreach team calls (RR 1.92, 95% CI 1.79 to 2.07). 101 One study found no change in the rate of RRT calls. 106

Paediatric intensive care unit transfers

One uncontrolled before-and-after study (without a RRT/MET) found a significant post-intervention decrease in the rate of unplanned PICU transfers per 1000 patient-days (RR 0.70, 95% CI 0.56 to 0.88). 102 Four studies (including one high-quality cluster randomised trial and one high-quality ITS study) found no change in the rates of PICU admissions post intervention. 84,100,101,105

Paediatric intensive care unit outcomes

Two studies, one ITS study and one multicentre cluster randomised trial (both with RRTs/METs), found significant reductions in rates of ‘critical deterioration events’ (life-sustaining interventions administered within 12 hours of PICU admission) relative to pre-implementation trends (incidence rate ratio 0.38, 95% CI 0.20 to 0.75)100 and relative to control hospitals (OR 0.77, 95% CI 0.61 to 0.97). 84 One controlled before-and-after study (without a RRT/MET) reported a significant reduction in rates of invasive ventilation given to emergency PICU admissions post intervention (RR 0.83, 95% CI 0.72 to 0.97), with no significant change observed in a control group of patients admitted to PICU from outside the hospital. 103 One uncontrolled before-and-after study reported a significant post-intervention decrease in the rate of PICU admissions receiving mechanical ventilation (RR 0.85, 95% CI 0.73 to 0.99), but an increase in the rate of early intubation (RR 1.87, 95% CI 1.33 to 2.62). 104

Implementation outcomes

Only three studies reported outcomes relating to the quality of implementation of the intervention. One study reported that 99% of audited observation sets of the Bedside PEWS had at least five vital signs present post intervention, up from 76% pre intervention (no change in control hospitals). 84 A previous study of the same PTTT reported that 3% of audited cases had used the incorrect age chart, but reported an intraclass coefficient of 0.90 for agreement between bedside nurses scoring the PTTT in practice and research nurses retrospectively assigning scores. 105 Finally, error rates in C-CHEWS scoring were reported to have reduced from an initial 47% to < 10% by the end of the study. 102

Box 2 presents a summary of the review 2 findings.

Search results

Eighty-two papers were included in the review (Figure 3). Forty-six papers focused on TTT implementation and use in paediatric and adult contexts (24 from the paediatric search and the remaining 22 from the adult-focused search); the remaining 36 papers contributed supplementary data on factors related to the wider early warning system. No studies were located that adopted a whole-systems approach to detecting and responding to deterioration. See Appendix 17, Table 41, for a detailed breakdown of the search process.

FIGURE 3.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram for study screening and selection: review 3.

Translational mobilisation theory was used to analyse the evidence to identify the sociomaterial and contextual factors associated with successful and unsuccessful paediatric early warning systems. In TMT, the primary unit of analysis is the ‘project’, which defines the social and material actors (people, materials, technologies) and their relationships involved in achieving a goal. The goals of the afferent paediatric warning system are as follows: first, the child is identified as at risk and a vital signs monitoring regime is instigated; second, evidence of deterioration is identified through monitoring and categorised as such; and, third, timely and appropriate action is initiated in response to deterioration.

The analysis of the literature suggested that three subsystems within the afferent component of early warning systems support the following:

• the detection of signs of deterioration

• the planning needed to ensure that teams are ready to act when deterioration is detected

• the initiation of timely action.

Principle 1: outcomes directed

The first principle of OUTCOME is that improvement is driven by an agreed outcome, rather than by predefined interventions. This reflects a growing concern that health-care improvement is often solution driven, rather than focused on improving practice. The emphasis on outcomes in the framework is informed by the concept of ‘projects’, which, in TMT, refers to a ‘goal-oriented enterprise, constructed by the interests that gather around it, and which has an associated division of labour, tools, technologies, practices, norms, rules and conventions’. 32 Thinking about improvement in terms of the associated project helps to define the boundaries of the initiative. In the PUMA study, this focused improvement efforts on the afferent component of a paediatric early warning system, which detects deterioration and triggers timely and appropriate action, and excluded the efferent component, which consists of the people and resources providing a response. 24

Principle 2: functions oriented

The second principle of OUTCOME is that improvement is oriented towards the functions necessary to achieve the goal. This requires specification of the primary mechanisms of action that are necessary in an overall process for the goal to be achieved. In the PUMA study, the core functions of an afferent early warning system were identified through the application of TMT to the systematic review and refined through discussions with clinicians to produce seven functions in total: monitor, record, interpret, review, prepare, escalate and evaluate. 13

Principle 3: systems focused

The third principle of OUTCOME is that improvement is focused on the sociomaterial resources, processes and mechanisms needed to enact the essential functions for achieving the goal. This requires specification of the minimum system requirements and draws on the concept of the strategic action field in TMT. Strategic action fields provide the structures, organising logics, technologies and materials, and interpretative repertoires that condition projects of collective action. 32

In the PUMA study, the system standard was specified in a propositional model of minimal conceptual requirements organised around the seven functions of an afferent paediatric early warning system (PUMA Standard). The model drew together two kinds of evidence from the systematic review: evidence of the challenges that must be overcome in detecting and acting on deterioration and evidence on proposed and/or evaluated solutions to challenges. The propositional model was reviewed and refined by parents with experience of a child’s deterioration and by clinical experts on the PUMA study team. The model was refined during the implementation phase of the PUMA programme to provide a more easily accessible version of the original (Figure 4) and summarised in the PUMA Wheel (Figure 5). Please note that the original, branded colour scheme differs from the colours present in the tables and figures of this report; the original colours can be seen in Report Supplementary Material 3.

FIGURE 4.

The core components of a paediatric early warning system: the PUMA Standard. Reproduced from Allen et al. 43 This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reproduced from Allen et al. 43 This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

FIGURE 5.

The core components of a paediatric early warning system: the PUMA Wheel. Reproduced from Allen et al. 43 This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reproduced from Allen et al. 43 This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Principle 4: context specific

The fourth principle of OUTCOME is that improvement is focused on the development of context-specific initiatives to achieve the goal. Proponents of change often favour top-down approaches to bring about improvements; the list of interventions and improvement efforts that flounder when spread or scaled up continues to grow,197–199 however, in part because of failures to normalise and embed interventions into local contexts. As Braithwaite201 has argued, all meaningful improvement is local, and yet QI has been criticised for applying insufficient attention to rigorous evaluation and sharing the lessons of successes and failures to facilitate sustainability and spread. Avoiding these pitfalls requires structures to support systematic and rigorous local improvement efforts in relation to a service standard. In addition to specification of the minimum system requirements to support an improvement project, OUTCOME also involves the development of associated assessment tools that can be deployed to improve understanding of the local system and identify areas for improvement.

In the PUMA study, two complementary assessment tools were developed from the PUMA Standard: the Staff System Assessment Tool (SSAT) and the Family Feedback Tool (FFT). These were refined during the course of the study, along with the PUMA Standard. The tools were designed to prompt wider discussion among the improvement team, in order to reach a shared understanding of the local afferent paediatric warning system and areas that might be targeted for improvement.

Principle 5: locally led

The fifth principle of OUTCOME is that improvement capitalises on the expertise and knowledge of those delivering services. This is intended to encourage local ownership of the improvement initiative. An amended version of the Model for Improvement supports teams in driving their own improvement processes and is designed to operationalise the core constructs of NPT. It is based on a five-step process:

1. form an improvement team

2. assess the system

3. select and plan improvement initiatives

4. implement and review initiatives

5. sustain progress.

In the PUMA study, local leadership of the improvement process was supported through workshops, facilitation and written guidance.

Principle 6: learning systems

The final principle of OUTCOME is to create a learning system around the improvement project, with participants attuned to system features with strong feedback loops. 201 Health-care systems are dynamic, and wider changes to the system may be consequential for an area of practice, resulting in ‘drift’,210 or the need for further adjustments to the system. OUTCOME deploys the use of assessment tools to keep systems under review, and structures for supporting local leadership.

In the PUMA study, PIs were provided with written guidance on how to ‘sustain progress’ and were encouraged to repeat the system assessment every 12–24 months to reflexively monitor performance, select and plan initiatives, and implement and review initiatives. Table 7 summarises the principles, structures and theory informing the OUTCOME framework and its application to the PUMA study.

The hospital

Case study 1 was undertaken at Alder Hey Children’s Hospital, a large (260-bed) tertiary paediatric hospital in the north-east of England. The hospital has a national profile as a centre of excellence, and had recently moved into a new bespoke building, with dedicated family accommodation.

The ward

The ward case study focused on a specialist cardiac medical/surgical ward, which cared for cardiac and cardiac–surgical patients. The patient population was diverse and included children with established diagnoses and/or relatively stable conditions, others with more recent, unexpected or uncertain diagnoses and/or acute critical care needs, and day-surgery cases.

The ward had 24 beds divided into three ‘pods’, each with a nurses’ station. The orange pod comprised two four-bedded bays and was used for day-surgery patients and low-acuity children. The green pod was used for non-critical cardiac inpatients, with the sickest children allocated to beds visible from the nurses’ station. The blue pod was used for the most unwell and highest-dependency patients, and was classified as having ‘HDU’ beds (see Report Supplementary Material 4 for details).

Staffing

Ward staff comprised a ward manager, 38 band 5 nurses, six band 6 nurses, a nurse educator, four health-care assistants (HCAs) and a play specialist. Student nurses had placements on the ward. Many of the nurses were very experienced and had specialist skills, although none was Advanced Paediatric Life Support (APLS) trained. All qualified staff worked across regular and HDU patient beds. Staff turnover was low and agency staff usage was rare, with gaps in the rota filled by ward staff working ‘bank’ shifts.

Nurses and HCAs worked 12-hour shifts (07.00–19.00). The ward manager and nurse educator worked 08.00–16.00 weekdays, but frequently exceeded these hours. Nurses worked in one of the pods. A band 6 nurse was designated shift co-ordinator, but also carried a case load.

The cardiology medical team managed all patients; cardiology and surgical teams jointly managed surgical patients. The medical team comprised cardiology consultants, registrars and other junior doctors. Junior doctors rotated every 6 months. The cardiology team provided weekday and weekend cover, with day shifts starting around 08.00 and ending around 20.00. An on-site registrar and on-call cardiology consultant provided night cover. Medical staff from the PICU, which was adjacent to the ward, provided occasional ad hoc support to the cardiac registrar during night shifts. There was no RRT.

Routines

Nursing handover was at 07.00 and 19.00.

A shift co-ordinator’s handover was at 07.00 and 19.00.

There were two daily bed management meetings, attended by shift co-ordinators.

The medical team’s daily routine included (1) handover between the night and day shift registrars, (2) a daily ward round attended by the medical team and shift co-ordinator and (3) an evening ‘board round’ between the consultant, registrar and shift co-ordinator.

Family involvement

The ward environment was designed to accommodate parents, and staff encouraged them to be present on the ward. Sixteen single-occupancy rooms on the ward could accommodate parents overnight. Relationships between the ward team and families were highly valued and very positive, but, although the new infrastructure provided high levels of privacy, it prevented easy nurse–family communication.

Detect

A detailed trust policy specified that observations were to be conducted on admission and during each clinical assessment. Nurses were required to conduct observations at least every 4 hours, but most children needed more frequent monitoring. Awareness of the detail of the formal policy was uneven, but appeared to be embedded in practice. There was a high level of awareness of the need for children with complex diagnoses and/or medical histories to have extra/additional vital signs monitored, or for certain key vital signs to be monitored more frequently. An electronic recording and scoring system was in use (MEDITECH-6; Medical Information Technology, Inc., Westwood, MA, USA), which included a ‘red clock’ that indicated when observations were due and acted as a prompt to ensure that all relevant vital signs were recorded, although nurses sometimes struggled to complete observations within schedule. Trust policy specified that the minimum frequency of observations for each patient was set by nursing staff on admission and reviewed at each nursing handover. Observation frequency for each patient was entered into MEDITECH-6.

Children admitted to the green and blue pods were monitored continuously. Real-time vital signs were displayed on bedside screens and terminals at the nurses’ stations. When vital signs fell outside predefined parameters, an alarm sounded at both the patient bedside and the nurses’ station. Elective day-surgery patients in the orange pod were monitored intermittently using portable equipment, which triggered alarms and could be connected to the nurses’ station as required.

Although continuous monitoring was widely used, scheduled observations were conducted manually and nurses were encouraged to look beyond the vital signs readings in assessing a child’s status, with nursing work allocated to ensure continuity of care and facilitate pattern recognition:

[I]t’s no good just looking at a monitor, because you need to be looking at their chest expansion and if they’re using extra muscles and whether its equal and whether they’ve got any other signs like a nasal flow, or other signs that they’re struggling to breathe.

Senior nurse

Relocation to the new hospital had created major challenges; the old accommodation was open plan and afforded high levels of surveillance over children, and colleagues and families were visible. With the new ward layout, this was lost, and affected how the unit worked. During the pre-implementation phase, nurses were still adjusting to these disruptive effects.

Although monitoring equipment was functional and available at all bedsides, nurses put effort into generating ‘good’ observational data; some of the ‘probes’ used to gather information on oxygen saturations or respiratory rate, for instance, did not fit small babies, and nurses had to actively work with the monitors, adjusting and calibrating different pieces of equipment, to ensure that vital signs readings were accurate. MEDITECH-6 used vital signs data and other variables to automatically calculate and communicate a ‘score’. This included ‘nurse concern of deterioration’, which scored 1, and ‘parental concern of deterioration’, which scored 2.

MEDITECH-6 was a relatively new intervention, and nurses were adjusting to its use in practice. Although MEDITECH-6 replicated the paper-based PTTT that it replaced, it had different affordances and its integration into practice had implications for nurses’ workflow. First, unlike the paper system, it did not immediately provide an overview of vital signs trends. Accessing this information required navigating through several screens (see Report Supplementary Material 5). Second, compared with recording observations on a paper chart, data entry in the electronic system was more time-consuming. Third, the requirement to access mobile computers resulted in nurses recording observations on scrap paper before later entering data into MEDITECH-6. Such practices carried the risk that vital information could be lost or misplaced, as illustrated by the following extract from the field notes:

[entering observation data into computer] I’m sure I did [9]’s obs[ervation].

I’m sure you did, you wrote it on a paper towel.

I can’t find it; I’m sure I did all three.

There are some pieces of paper towel here, are any of these yours? [3 × paper towel with numbers written on, no patient names or room numbers].

No, those are all [other nurses’].

The ward staff had good relationships with families, and often had cared for children previously. Many families had experience with, and understanding of, their child’s condition, but others were orienting themselves to an unexpected and/or new diagnosis. Although parents were encouraged to monitor their child and develop an understanding of bedside monitoring equipment, staff understood and responded to differences in capacity, and tended to give very knowledgeable parents more involvement in patient care. During the ward round, the consultant asked parents (when present) about their own understanding of their child’s health. Parents of children with long-term conditions were regarded as an important source of knowledge on baseline vital signs parameters.

Plan

There were a number of processes for reviewing individual patients and unit capacity, and these produced different levels of situation awareness across the nursing and medical teams.

Nursing handover was the key mechanism by which nurses formally collated and communicated information on patients’ statuses. Separate handovers were conducted in each pod; thus, nurses had an awareness of the patients in their area only. There were no formal face-to-face meetings between day and night staff; handovers were audio-recordings created by the nurse caring for the patient. All staff from the incoming shift reviewed this information by listening to the recording and annotating electronic pre-populated printed handover sheets. The quality of recordings was variable, however, and there was no formal opportunity for discussion between nurses from each shift.

In the nursing team, it was only the shift co-ordinator who had overall situation awareness of the ward and related units. A co-ordinator’s handover took place at 07.00 and 19.00. This was a face-to-face meeting in which co-ordinators shared detailed patient information, expected admissions, discharges and ‘step-downs’ from PICU, highlighted the ward’s most unwell patients and discussed concerns, and reviewed nurse staffing. The shift co-ordinators liaised with the ward nurses for updates on their patients before the meeting.

After the co-ordinator’s meeting, there was a bed management huddle attended by senior nurses from all critical care units. There were updates on the current status of HDU and PICU patients and information on planned and potential ‘step-down’ admissions to the ward. The shift co-ordinator communicated the ward’s current staffing levels and capacity.

Medical handover was an important mechanism for doctors to review children’s status. This took place at 08.00 and 20.00, and involved the night and day shift registrars only. All children were discussed consecutively and in detail, focusing on the patients of most concern. Outstanding work tasks were also identified. The registrar also attended PICU handover for an update on cardiac patients and expected discharges to the ward. Junior doctors received updated information on patients through an electronic handover sheet, which would have been updated by the outgoing medical team.

There was a 09.00 daily ward round led by the cardiac consultant and attended by the registrar and junior doctors. This was the primary mechanism by which the additional needs of ‘at-risk’ patients were identified and shared between nursing and medical staff. Ward rounds lacked a consistent format, however. In one model the medical team was accompanied by the shift co-ordinator and saw all their patients in turn. The nurse looking after each patient either remained in the patient’s room and communicated with the medical team directly, or, if possible, they liaised with the co-ordinator, who consulted the medical team on their behalf. There were other instances when the ward round was shared and conducted in two pods simultaneously, so the co-ordinator could attend only part of the round. In a third model, the ward round was conducted from an office; nurses were unable to attend because they could not leave the patients, and this created communication difficulties. Bedside ward rounds were markedly more formal events when compared with interprofessional communication at all other times. Communication between the senior doctors and parents was prioritised, with the nurse often left outside the room. Shift co-ordinators carried a case load and would frequently be called away to address other issues.

There was an evening ‘board round’ at 18.00 attended by the registrar, consultant, junior doctors and shift co-ordinator. The consultant led a team review of each patient to identify priority tasks and concerns. The co-ordinator found it more challenging to attend this handover, because of other competing priorities. It was also difficult to predict who would be present, and whether or not the results of the board round would be fed into nurse handover information for the next shift.

The ward had an electronic board linked to MEDITECH-6, which was intended to display up-to-date information on patients’ statuses across the ward. This was unused because it was inaccurate owing to delays in entering vital signs data, and was infrequently updated when patients were moved/discharged.

Act

There was a hospital-wide PTTT score built in to MEDITECH-6. Pop-up boxes prompted staff to identify and record a course of action when a score of > 3 was generated. Nurses were also required to indicate whether there had been a senior nurse or medical review during each observation, or if it was needed. The formal policy provided general guidance on escalation, but made reference to following instructions on the observation chart, which specified how to make use of the score, in what circumstances escalation should be triggered and the actions indicated. Nurses were well informed of the escalation policy and the respective responsibilities of nursing and medical staff.

When vital signs fell outside predefined parameters, an alarm sounded at the patient bedside and at the nurses’ station. However, many patients had vital signs readings (e.g. oxygen saturation levels) outside acceptable parameters for the general paediatric population, but normal for them because of their condition. MEDITECH-6 allowed vital signs thresholds to be adjusted, but this was the responsibility of the consultant and was rarely done. Nurses checked if parameters had been adjusted in order to assess whether or not to worry about alarms. Alarms activated frequently, and in the vast majority of circumstances were quickly silenced by either nursing staff or a parent/family member. Staff frequently discussed limitations of the MEDITECH-6/PTTT system, and repeatedly emphasised the importance of using their own professional judgement, alongside scores, when making patient assessments.

The ward had a supportive culture; junior nurses raised concerns first with the shift co-ordinator and were confident that these would be attended to. Interprofessional relationships were more variable. During the day, doctors had a high presence on the ward and, although nurses were mainly positive about the quality of relationships with medical staff, they also made reference to, and we observed, instances when they struggled to get doctors to act on their concerns:

I think the doctors and the nurse relationship is really, really good here [. . .] it depends which doctor [. . .] sometimes doctors don’t realise how serious you mean [. . .] like this little boy, [. . .] I knew him very well, so I knew that he really wasn’t right, whereas the doctor that, who happened to be on that day didn’t really know and he said ‘oh for that condition it can be that, that can be normal for that condition’ and I was thinking ‘yeah, but for him it’s not normal’ [. . .] so it’s often a problem there when the doctors don’t really understand what you’re trying to tell them, because they’ve got other ideas or something like that, you know.

Staff nurse

There was evidence that nurses could not always persuade doctors to respect their subjective assessments, and often they would make use of the more heavily weighted ‘parental concern’ variable in the PTTT to generate a score that better reflected their own clinical judgement. There were particular challenges at night when there were insufficient band 6 nurses to ensure consistent cover on all shifts, and junior nurses reportedly had more challenges in communicating with doctors.

There was a high level of contact between nursing staff (particularly) and parents, and this required the filtering of expressions of parental concern, to ensure that only relevant concerns were factored into clinical decision-making and the management of patient care. Report Supplementary Material 6, Table 1, shows a summary of the pre-implementation system strengths and weaknesses identified by the PUMA team.

Monthly critical deterioration review

Led by a consultant and senior nurse, these meetings were intended to address weaknesses in the organisation’s approach to reviewing and learning from critical deterioration events. It related to the evaluate function in the propositional model.

The initiative was to comprise ‘critical deterioration review meetings’, held between a senior nurse and consultant paediatrician, and an online, anonymous survey of staff members involved in cases of critical deterioration. In a series of (three or four) meetings, multiple cases of critical deterioration were reviewed in detail. Eleven survey questions were designed and piloted with nursing and medical teams to generate ‘direct feedback’ from the teams involved through an anonymous staff survey, but the project then stalled and the initiative did not become embedded in practice. The work of collating information in preparation for each meeting was significant, as was the time required to review each case in the necessary detail, and the initiative was not sustained.

Out-of-hours standard operating procedure for on-call doctors

The cardiac ward had its own out-of-hours cover, whereas a medical and surgical team (one consultant and one registrar) covered the rest of the hospital. The out-of-hours SOP was developed in response to weaknesses in identifying and planning for risk. Normal practice was for doctors to review every patient currently admitted to the hospital, regardless of clinical acuity. The initiative was led by a consultant, building on previous activity around the organisation of medical work, who began by observing current junior doctors’ working practices and established that the handover was particularly poor at sharing key information on the ‘sickest kids’ across the hospital between doctors, and between medical and nursing teams. The new SOP introduced a system of working in which on-call medical teams were able to prioritise workload according to clinical need. It also introduced a third member of on-call medical staff (an additional registrar) and required nursing teams to follow a SBAR technique model of communication when requesting an out-of-hours medical review. The initiative was successfully introduced and embedded across the organisation.

Family engagement tool

This initiative aimed to ‘improve family satisfaction’ and ‘increase family involvement in care planning’ through the design and implementation of a communication tool to be implemented on the cardiac ward. A senior nurse led the initiative in partnership with a cardiology consultant, with additional input from a ward manager, a senior nurse, a junior doctor and a play specialist. Possible approaches were discussed in a series of meetings, but progress stalled because the technological infrastructure was not available. The team decided to implement ideas and approaches that had been successfully embedded in practice elsewhere and selected MyPad, a communication board used in the general HDU. However, although some progress was made in tailoring the tool for use on the cardiac ward, the initiative ceased as a result of significant changes in the structure of the senior nursing team, and the discovery of plans to implement communication technologies, namely iPads (Apple Inc., Cupertino, CA, USA), across the organisation.

Training clinical staff

This initiative resulted from organisational-level mandated changes in response to the serious critical incident. As outlined in one of the Improvement Team planning documents, the CQC identified a number of areas of system weakness, which aligned with the priority areas identified through the system assessment, namely ‘Staff were not compliant with Trust policy in obtaining and recording observations as per PEW tool guidance’ and ‘Lack of a standardised tool to recognise sepsis across the Trust’. The selected initiative consisted of retraining clinical staff on the Alder Hey PEWS, recognition and response to patient deterioration, and the NICE-endorsed sepsis screening pathway (Sepsis 6). 212 A ‘staff competency’ document was used alongside training sessions to assess and record each staff member’s level of understanding, and data on staff receiving and completing training were collected across the trust. Initiative four was successfully introduced across the whole organisation. Because this initiative was a mandated change in response to the CQC report, organisational-level support was guaranteed from the outset; therefore, a high level of practical, financial and regulatory assistance was provided throughout the development and implementation process. However, site leads indicated that the initiative ‘absorbed a high percentage of time’ and required ‘very tight project management’, with negative consequences for other PUMA initiatives.

Standard operating procedure for improving ward rounds

This initiative was designed to improve processes for reviewing children and planning for action on the cardiac ward and was led by a senior nurse working with a cardiology consultant. In the pre-implementation period, sometimes the ward round was divided into two medical teams undertaken simultaneously or conducted away from the ward. This meant that the shift co-ordinator could not attend both rounds, and therefore lacked situational awareness; the medical team also had a fragmented understanding of all patients. There was also significant variation in the way evening ‘board rounds’ were conducted: it was difficult to predict, on a regular basis, who would be present, and whether or not the results of the board round would be fed into nurse handover information for the next shift.

Some progress was made in getting key staff members to discuss the issue, but the momentum was not sustained and no ward round SOP was implemented. For the initiative to be successful, both nursing and medical teams needed to be involved; securing the direct involvement of senior medical staff was a particular challenge:

[T]he consultant team couldn’t agree what they wanted, how they wanted it to look. So we’ve spent months trying to get some sort of consensus.

Staff nurse

Lower trigger threshold

The threshold on which a score triggered senior (medical) review was lowered after the critical incident took place (November 2016).

Sepsis 6 pathway

The Sepsis 6 pathway212 was implemented as a direct result of the CQC visit and the recognition that there was no standardised tool to recognise sepsis across the trust. Training for the pathway was implemented as part of the PUMA training initiative. In addition, it was introduced to the MEDITECH-6 system. This required nurses to answer an additional range of questions while inputting patient data. This was extra work, but the disruptive effects were mitigated by the fact that, at the time of this change, the electronic system had become normalised.

Appointment of additional staff: specialist sepsis nurses

A number of specialist sepsis nurses were employed from June 2017. This specialist team covered the whole hospital. They worked in advisory capacity if there were incidents involving sepsis and were also involved in training.

Development of a standard operating procedure to define and clarify the shift co-ordinator role

In a further development, the trust implemented a SOP for the shift co-ordinator role, to address a perceived lack of consistency. The aims were to generate role clarity, to ensure that the co-ordinator was able to support and direct the team to ensure that child safety was maintained, and to make ward staff aware of the roles and responsibilities of the co-ordinator.

Introduction of safety huddle

The cardiac ward implemented a formal safety huddle, which was led by the ward manager, who had been involved in the system assessment on the cardiac ward. The huddle was a 5-minute meeting immediately prior to both the morning and evening nursing handover meetings. The outgoing shift co-ordinator presented information relating to both ward and patient concerns to all incoming nursing staff. A handover sheet was introduced to help support this process that contained sections on ‘key messages’, ‘watchers/PEW triggers’, ‘safeguarding concerns’ and ‘recent incidents’.

Ward-specific escalation plan

A ward-specific ‘Escalation Plan for a Deteriorating Cardiac Patient’ was introduced, led by the ward manager and nurse educator, working in partnership with senior medical (cardiology) colleagues. The policy was developed in response to the fact that the particular clinical characteristics of many cardiac patients did not ‘fit’ the trust-level PTTT system and required vital signs parameters to be adjusted. This long-recognised issue was identified in the PUMA system assessment, although not prioritised in the action plan.

The final policy specified cardiac-specific guidance on the electronic PTTT system and provided updated detail on the role of the shift co-ordinator within the escalation pathway. Specific time frames were provided (‘medical review within 30 minutes’) and instances when consultant review must be sought were highlighted (‘consultant must be alerted to a new PEWS of 4 or nursing or parental concern or increased lactate above 3’). Printed copies of the new plan were placed at each nurses’ station, and, echoing the changes brought about by the out-of-hours SOP, were bundled with the SBAR technique sheets (with the aim of aiding clear and direct communication).

Communication checklist for high-dependency unit and paediatric intensive care unit transfers

A communication checklist for improving communication between staff and the family members of patients transferred from the HDU or PICU was developed and introduced to the cardiac ward in June 2018. This detailed information to be discussed with parents on admission, including information on the electronic monitoring system and a reminder to inform families that they can ask for help at any time. Nurses were required to sign off each completed task before returning the completed sheet to the ward manager. However, although the checklist was in regular use for a short period of time, it did not become embedded as a result of significant changes in the senior nursing team. Progress on alternative approaches was halted because it was anticipated that the trust would be introducing a hospital-wide system for communicating with parents. Report Supplementary Material 9, Table 1, summarises the implementation of action plan initiatives for Alder Hey and Appendix 21, Table 45, summarises all the embedded changes over the course of the study.

Detect

In the post-implementation phase, it was evident that the MEDITECH-6 had become normalised, and the addition of more mobile computers helped to obviate some of the challenges nurses had experienced with timely data entry. In addition, the ‘Sepsis 6 pathway’212 was in routine use. Although there was a loss of some senior experienced members of the team, all nursing staff had received training on PEWS/Sepsis 6, and all senior nursing staff had received APLS training.

More recently qualified nurses who had joined the ward appeared to value the PTTT score.

The new cardiac-specific escalation policy was in use. This document provided staff with clear details on when to escalate (according to PTTT score and key vital signs). The policy formalised the co-ordinator’s level of seniority and responsibility within the escalation pathway, and the specific care requirements and typical vital signs observations of cardiac patients. Laminated, printed copies were positioned at each of the nurses’ stations, and electronic copies were available on the trust’s intranet.

Plan

The main mechanisms for reviewing and planning for action remained the same, but situational awareness across the nursing team had been improved through the introduction of the safety huddle.

Some of the challenges with the ward round remained; some ward rounds continued to be conducted away from the patient bedside, and thus excluded nursing staff and families. Co-ordinators were still allocated their own patient case load, and frequently ‘pulled away’ from the morning ward round because of multiple competing demands.

The out-of-hours working SOP was designed to address issues with the organisation of doctors’ work, but had minimal impact on the ward system because the cardiac ward had historically received a high level of out-of-hours medical cover. It was reported that cardiology team members could be more difficult to reach during the night if rostered to provide hospital-wide cover.

Although the timing and frequency of the evening board round remained variable, senior nurses reported that they are able to attend in the majority of instances, and the electronic board used during the meeting was more frequently updated, but rarely used at any other time.

Act

Consistent band 6 cover, coupled with the new ward-specific escalation policy – which clearly states that the nurse in charge may escalate directly to registrar or consultant without having to ‘go through’ the junior medical team – provides a clear, formalised mechanism for effectively communicating concerns across professional boundaries: roles and responsibilities are better understood.

All senior nursing staff had received APLS training, which included responding to deterioration.

The hospital

Case study 2 was undertaken at Arrowe Park, a medium to large district general teaching hospital in the north-west of England. Children’s services comprised one ward and a paediatric assessment unit (PAU). There is no PICU, but there are links with a tertiary children’s hospital to refer children with high acuity and/or complex needs.

The ward

The ward case study focused on the 32-bed children’s ward, comprising 16 individual rooms and three four-bed bays, two HDU beds and an emergency room. See Report Supplementary Material 11 for details.

The ward cared for children with a wide range of conditions: children with breathing difficulties, those living with long-term complex conditions, children with feeding problems associated with being a premature neonate, elective/day surgery cases, and children and young people experiencing a mental health crisis.

Staffing

The ward team included a ward manager and deputy, qualified nurses (27 band 5 and five band 6), three HCAs and a play specialist. Student nurses had placements on the ward. Nurses and HCAs worked 12-hour shifts (07.30–19.30). A small number of staff worked 07.30–14.30. The ward manager and deputy worked office hours. External agency staff were used on occasion.

Nurses worked in one of three ward sections, with patients allocated to ensure continuity of care. There was a designated nurse in charge for the shift, who carried a case load. HCAs were allocated patients, but worked under the supervision of a registered nurse. Children admitted to the HDU had one-to-one nursing care. Not all nurses were HDU trained, so HDU admissions affected nursing work on the ward.

The medical team comprised consultants, registrars, senior house officers (SHOs) and ANPs. Junior doctors rotated between specialties every 6 months. The medical team provided weekday and weekend cover, spanning a variety of shifts. Day cover, from 09.00 to 17.00, included two to four SHOs, two registrars and one consultant. From 05.00 to 21.00, one SHO and one registrar would provide cover, with one consultant working until 19.00 in the summer and until 20.30 in the winter. A SHO, a registrar and an on-call consultant provided night-shift cover, with a second SHO working until 23.00. On the weekend, during the day, there were two SHOs and one registrar, with the consultant staying until 15.00. Weekend night-shift cover mirrored that during the week, except that there was no SHO overlap and the consultant was on call from 15.00. Some children were cared for by adult specialist and surgical teams located in the main hospital.

Routines

Nursing handover was at 07.30 and 19.30.

The medical team worked on both the ward and the PAU. Morning handover took place on the ward at 09.00. The evening handover took place in the PAU at 21.00.

The ward round followed the morning handover at 09.30.

Family involvement

Involving families was central to patient care. Parents were encouraged to stay with their child and the single-occupancy rooms were designed to accommodate a parent or carer overnight.

Detect

A PTTT was used on the ward and PAU (see Report Supplementary Material 12). This was an adaption of the Brighton PEWS. In the pre-implementation phase, this was a paper-based system laid out in a four-page A4 booklet, which included instructions on escalation, parameters to be monitored (see Report Supplementary Material 13) and space for medical staff to determine the frequency of observations. Vital signs were recorded on the chart, which showed trends and included a traffic-light system indicating the significance of observations.

There were five versions of the chart, reflecting aged-based parameters. An observations policy was in place, which required nurses to generate a score at every observation, adhering to the following monitoring guidelines.

Doctors were responsible for setting observation frequency on admission. This rarely happened, however, and was usually determined by nurses based on their assessment of the child and family concern. HCAs and junior nurses sought direction from more experienced staff in determining observation frequency. Within the ward team, roles and responsibilities in relation to monitoring were well understood and work was allocated to ensure that the nurse had the appropriate monitoring skills.

Monitoring was conducted manually and equipment was available when required. A couple of rooms had continuous monitoring facilities, and were reserved for the sickest children. Readings could not be reviewed remotely, but baby monitors linked to the nurses’ station were routinely used. The large number of individual cubicles made routine surveillance of children difficult, and there was a degree of dependence on parents to summon help.

Monitoring and recording largely took place concurrently, with vital signs documented on the chart at the bedside. The ward was preparing to implement a new electronic monitoring system and staff were required to record observations on the paper chart and the electronic system to familiarise themselves with the system. A lack of available computers made this difficult, however, and, although some HCAs were observed adding information, nursing staff considered this to be unnecessary duplication.

Nursing and medical handovers were the main mechanisms for sharing information about prior risk factors that might increase a child’s risk of deterioration. Knowing an individual patient was highly valued by nurses, and pattern recognition was often used by more experienced staff to question an elevated score. Nurses were extremely supportive of one another, with frequent sense-checking about normal parameters, providing learning opportunities for more junior staff. In addition, staff engaged with parents to establish their child’s normal baseline on admission, but also subsequently during patient examinations and ward rounds. Indeed, nurses relied on parents staying overnight to monitor the patient. Although parents were not provided with formal instructions on how to raise concerns, nurses involved them in monitoring activity and encouraged them to alert staff if worried. Parents reported that they felt able to alert staff by asking them directly when they had minor concerns about their child, which was facilitated by the high visibility of staff on the ward and understanding that buzzers could be used.

Plan

The principal mechanisms for reviewing children’s statuses and identifying children of concern were the nursing and medical handovers and the ward round.

Nursing handover followed a narrative format to provide a comprehensive picture of the patient; salient details were recorded on a blank handover sheet. The nursing handover was divided into the two sections of the ward and the nurses had an overview of all patients in their section, but not the ward as a whole. As both handovers took place at the same time, the nurse in charge attended handover for one section and then received a second ‘mini-handover’ from the senior nurse at the other end of the ward. The purpose was to share information to develop a ward overview.

The doctors’ handover was attended by all doctors on that shift, ANPs and, when possible, the nurse in charge. The registrar from the previous shift handed over patients to the oncoming team, focusing on patients of most concern, and highlighting to-do lists for the upcoming shift. A printed handover sheet was distributed that contained information on the patient’s diagnosis, treatment plans and outstanding jobs (see Report Supplementary Material 14). The doctors’ handover covered all general medical patients on the ward, but excluded children under the care of a specialty or surgical team, who were managed by the relevant adult team.

The ward round followed handover; the registrar and the consultant divided patients between them, and were accompanied by the junior doctors and ANPs. Sickest children were seen first, although sometimes discharges could take priority. The consultant and registrar led discussions with parents/family members (and children with capacity). The nurse looking after the patient was usually present during the ward round and communicated with the medical team directly, often acting as an advocate for the parent or reminding them of conversations that they might have had earlier. This was not always possible, however, particularly if the nurse was busy with another patient. In these instances, it was usual for doctors to find the relevant nurse to update them on any decisions made.

Beyond formal handovers and the daily ward round, information on patients was routinely relayed throughout the day. Doctors were involved in these conversations when on ward during the morning; the majority of doctors moved to PAU during the afternoon and evening, and these conversations were less frequent.

The nurse in charge and the ward manager maintained an overview of ward activity (new admissions, changing patient acuity) and considered the implications for staff organisation. These were ongoing conversations throughout the shift, but were focused on their capacity to cope with the workload after a patient had deteriorated, rather than proactively planning for the management of at-risk patients.

There was also a mechanism called ‘the block’: a folder containing information on staffing levels and patient acuity. The block was supposed to be completed by the senior nursing staff to proactively manage capacity. However, it was not used systematically and was primarily completed at the end of the shift to document, rather than manage, activity.

Act

There was a graded escalation policy, linked to the severity of the score, and outlined on the PTTT chart. Staff had a good understanding of who to contact without having to refer to the chart itself. Although the score was the formal trigger for action, a decision about whether or not to escalate depended on who was calculating the score and how they defined the level of risk. Elevated scores were routinely not acted on by more experienced nurses, who justified their decision-making in the light of professional judgement. HCAs, by contrast, were far more likely to adhere to the escalation policy. In addition, although staff concern was not included in the score, the chart encouraged staff to raise concerns and all expressed their confidence in doing this:

This tool is intended to augment clinical judgement and NOT replace it. If you are worried about any child please escalate your concerns.

All ward staff appeared comfortable communicating with doctors whenever they had a concern about a patient, with or without an elevated score. Nursing staff clearly asserted that professional hierarchies did not prevent them from expressing concerns to the relevant staff, but they did encounter challenges in getting medical staff to act on their concerns. There was also difficulty reported around escalating care for non-general paediatric patients, again particularly for junior staff unfamiliar with procedures.

During the admission process, parents were encouraged to raise concerns, and were instructed to use the bedside buzzers or come to the nurses’ station if they required anything. Parents reported that they felt able to alert staff by asking them directly when they had concerns about their child, which was facilitated by the high visibility of staff on the ward. No written instructions on raising concerns or escalating care were provided, however, and some parents reported having been too worried on admission to process the guidance given by nursing staff.

Usual practice following a critical event or death would be to debrief and then discuss it during the weekly care improvement meeting, which focused on potential learning from key events of the week, for instance incorrect medication given. See Report Supplementary Material 6, Table 2, for a summary of the pre-implementation system strengths and weaknesses identified by the PUMA study team.

The PUMA initiatives

Non-PUMA study initiatives

Two changes were introduced during the implementation phase, the planning for which pre-dated PUMA. This included the introduction of a safety huddle and the roll-out of an electronic recording and scoring system.

Detect

There were major changes to the detection components of the paediatric early warning system in the post-implementation phase as a result of the introduction of the electronic recording and scoring system. When staff worked with the paper-based chart, it was straightforward to identify the vital signs that needed to be monitored for each patient. With the introduction of the electronic system, this was more difficult, and some more junior staff struggled to remember precisely what needed to be observed. The new electronic system required staff to manually select a patient’s chart, which often meant that the nurse would be aware of only the most recent vital sign, rather than the overall pattern, with the potential to negatively affect nurses’ use of professional judgement.

The electronic system had no impact on ensuring that doctors specified the frequency of observations, as per the hospital policy. In practice, this was still undertaken by nurses.

The electronic system also required manual input of vital signs data. The requirement for nurses to manually input data, and a general lack of availability of computers, particularly during busy times such as ward round, resulted in batched data entry. This would mean that the input of patient data into the system would be delayed, although everyone emphasised that, if a patient was scoring, they would escalate if needed. Delays to data entry also created a risk of errors being introduced into the process:

So often your first set of, um, of obs[ervations] that you do when you come out of handover, um, sits in your pocket, um, and sometimes a child could be seen by the doctor, not knowing what the current PEWS are, because they’re sat in your pocket.

Senior nurse

The loss of experienced nursing staff increased the need to support others:

Our problem is our skills set. We have been losing seniors who have been replaced by juniors.

Senior nurse, field notes

Plan

Nursing and medical handover and the ward round continued to be important mechanisms for identifying patients at risk and for forward planning. Changes to this process included the inclusion of all patients on the medical handover sheet. Although these patients were not formally handed over, it gave doctors an awareness of all children on the ward. In addition, when concern had been expressed about a patient during the safety huddle, whether they were medical or under the care of an adult specialist, the paediatric doctors would see the patient during their ward round.

The nursing handover sheet changed to follow a SBAR format. Although largely accepted, this was dependent on the individual nurse handing over, and there was some drift towards a narrative format. However, nursing handovers were reported as being quicker and more succinct than in the pre-implementation phase.

An important change in the post-implementation paediatric early warning system was the use of a morning safety huddle. Attended by all staff groups, it allowed quick identification of children at risk of deterioration, alongside bed management, safeguarding concerns and key messages of the day. For support staff, the huddle provided them the opportunity to obtain information that was not previously passed on to them. It also gave nurses a snapshot of acuity levels at the ‘other end of the ward’, whereas previously they would have had an awareness of the status of children in the area that they were working in only. One effect of the introduction of the safety huddle was to change the language used to identify potentially at-risk patients, with use of the term ‘watcher’ and ‘one to watch’ used more uniformly across all staff groups.

The safety huddle, in conjunction with the changes to the handover sheet, also enabled doctors to be aware of all patients on the ward, whereas previously they would have handed over only the general medical patients.

The introduction of the electronic scoring and recording system increased doctors’ ability to review children’s vital signs at a distance from the ward. There was also evidence of closer working between the ward and PAU:

It is getting better and the nurse in charge will go if they are not busy. But even if they, they will phone down to the Assessment Unit to let them know what is going on. So I think that the safety huddle has had an impact on how the two teams communicate – they are getting better at letting each other know what their situation is like.

Ward manager, field notes

The block was still used to review staffing, but this continued to be more to do with reporting what has happened, rather than a mechanism for actively managing the ward.

The ward manager or deputy now attended the post-ward round doctors’ meeting. This was largely in relation to bed management and patient flows.

Although the safety huddle had been embedded in practice, the nurse in charge was attending the medical handover less regularly than in the pre-implementation phase. Thus, at the end of the study, although the wider team situational awareness of the ward as a whole was more comprehensive, the exchange of information between the nurse in charge and doctors was less systematic than it was during the pre-implementation phase. Doctors considered that nursing attendance at medical handover was important, indeed more important than the safety huddle, whereas the nurse in charge had to balance this with their other commitments on the ward.

Act

Since the introduction of the electronic monitoring system, the escalation pathway was less accessible than it had been with the paper-based system. The significance of this change was compounded by the increased use of agency and new staff, who were unclear of the escalation procedure. In such cases, the first point of call was invariably the nurse in charge, regardless of the concern. Although the majority of nurses and HCAs made an effort to state that they would have no reservations in activating an alert to medical staff should a patient need it, there was some indication that the newer staff would speak to a senior nurse in the first instance.

Difficulties in escalating care for non-general paediatric patients, particularly for junior staff unfamiliar with procedures, remain unchanged in the post-implementation phase. There was also the suggestion that, with doctors able to access children’s monitoring information remotely from the ward, doctors were less inclined to act on nurses’ concerns.

The hospital

Case study 3 was undertaken at the Noah’s Ark Children’s Hospital for Wales, a large tertiary (120 beds) service located in Cardiff. The purpose-built facility, which opened in 2015, has five wards [general medical (× 2), specialist cardiac/renal medical, surgical and oncology], a HDU and a PICU.

The ward

The ward case study focused on a medical ward, where children were cared for by general medical and specialist paediatric and adult consultants. The ward comprised 18 beds divided into two zones, ‘Turtle’ and ‘Lemur’, with a central nurses’ station. The ward was a mix of one single-occupancy room, four double bays, one four-bedded bay and a family room. See Report Supplementary Material 17 for the ward layout.

Staffing

Ward staff comprised a ward manager, two deputies shared across the two medical wards, band 6 nurses (n = 4), band 5 nurses (n = 29), HCAs (n = 5) and play specialists. The ward also had student nurses on placement. Staff turnover was high; many experienced staff had left for specialist roles, diluting the ward’s skill mix. There was widespread use of external agency staff.

Nurses and HCAs worked from 07.00 to 19.30. Typically, five registered nurses and a HCA were on shift, but staff could be moved in response to changing demand in the hospital. There was a designated nurse in charge of the shift (typically a senior band 6, although, at night, this could also be a band 5), who also carried a case load. Nurses were allocated patients from both sections of the ward. More experienced HCAs were allocated patients, but all observations had to be supervised by a qualified nurse.

The medical team covered the two general medical wards and worked 08.00–20.00. The eight-person rota included two consultants, two registrars and junior doctors (who rotated every 6 months), supported by ANPs who worked variable shift patterns. Generally, doctors divided themselves between the two general medical wards, the PAU and accident and emergency (A&E). General medical doctors were present on the ward immediately following ward round, with one SHO often based on the ward throughout the day. The hospital worked with the Hospital at Night model:215 two registrars and a SHO covered the general, specialty and surgical wards, and PAU and A&E, supported by an on-call consultant.

The specialist teams included paediatric specialists (oncology/renal/cardiac) and adult specialist with paediatric training (e.g. orthopaedics; ear, nose and throat).

Routines

Nursing handovers were at 07.00 and 19.00. Handover commenced with a safety briefing led by the nurse in charge of the previous shift, followed by a walking handover of individual patients.

The medical handovers were at 08.30 and 20.30 and followed the same format. They were led by the outgoing registrar and attended by all doctors on the incoming shift. Information was shared on all medical patients in the hospital. ANPs were present as part of the medical team. No senior nurses attended the doctors’ handover. Patients of concern were not systematically identified; handover would proceed in order of beds, with any concerns being highlighted as the patient was discussed.

The medical ward round was at approximately 09.00.

Many ward patients were under the care of specialist consultants; these were seen on the specialist ward rounds.

Detect

Monitoring core vital signs was integral to nurses’ work, although not all nurses were equally skilled in monitoring techniques. The hospital’s observation policy specified which observations should be conducted routinely and set a minimum frequency of every 4 hours, unless otherwise indicated (see Report Supplementary Material 18). The policy was well understood by nurses, who undertook a range of ‘additional assessments’: ‘eyeballing’ patients, touching patients to assess temperature and assessing tracheal tug (a proxy for increased effort of breathing). Nurses often increased the frequency of observations based on their assessments. They also made professional judgements in deciding which vital signs to monitor, which may have been non-compliant with observation policy, but took into account wider considerations about a child’s care:

He’s not too hot, so I won’t do his temperature now. The fact his heart rate is low and I’ve touched him and he’s not hot, we’ll just let him go into a deeper sleep and then do his temperature, because they don’t want to wake him up just for that.

Nurse, field notes

Nurses had a high level of awareness of conditions that required additional observations. Patient categorisations such as ‘bronch baby’, ‘premie’ and ‘ex-25 weeker’ oriented staff to risk factors and additional observations required. Beyond formal observations, nurses valued having oversight of patients, although the physical layout of the ward made this challenging:

It is wonderful for the parents, the parents love it, but it is really tricky for the staff. We can’t see everything that is going on.

Nurse, field notes

The four-bedded bay and two single cubicles had fixed continuous monitoring. Higher-acuity patients were allocated to these beds or rooms immediately adjacent to the nurses’ station, which allowed greater visibility. Mobile continuous monitoring units could be provided if required.

The sound of alarms was commonplace, triggered by a drop in oxygen saturation levels or if a monitor had been disconnected. Nurses always responded, but often to check that the equipment was working correctly. Staff frequently readjusted probes to encourage oxygen saturation level readings to improve. There were no monitors at the nurses’ station, however, and nurses were required to determine the alarm’s location. Depending on the patient, nurses also permitted parents to turn off monitors:

[T]hey’ve said ‘you can switch it off if you want to because, as long as we check it every now and again’, I just turn it on and have a look [laughter] and check it and turn it off again.

Parent

A monitoring equipment trolley was located in the middle of the ward, which was inconvenient for staff working in more distant rooms. In addition, field observations revealed that nurses were often unable to locate a specific item, and/or the equipment was broken. Thermometers were a particular issue and, once located, would be held by staff rather than being returned to the equipment trolley.

You don’t know what I’ve done with my thermometer do you?

I don’t but I can tell you where I’ve stashed mine.

Field notes

There were no formal routines for involving families, although staff considered themselves to regularly involve families in defining baselines, as well as encouraging them to talk about any concerns. Nurses also actively engaged parents in monitoring their child’s condition. During interviews, two parents reported that nursing staff had explained to them acceptable ranges in observation numbers and that they then played an active role in monitoring their child. One parent noted that staff had spent time explaining what the numbers on continuous monitoring equipment related to, and the parent reported that she was not concerned as long as the patient’s heart rate stayed at < 200 beats per minute.

Observations were recorded on paper charts. There were no colour-coded vital signs thresholds and no PTTT score. Although charts allowed nurses to clearly see patterns and trends, staff with previous experience of working with PTTTs favoured the colour-coded vital signs thresholds, as these were useful triggers for action:

I really don’t like the documentation here so much. [. . .] Because it’s just black and white, [. . .] the obs[ervation] chart is not so nice. [. . .] When you see a heart rate up to a hundred per cent, in the dark red, so you see, everybody would see, OK, err and a student would see that is not right.

Nurse

There were no formal resources on the ward that allowed staff to check a child’s observations against normal thresholds, with an assumption that these should be known by all staff.

Observation charts were located in a folder stored outside a patient’s room. Nurses were often unable to find a folder immediately, typically because medical staff were using it. During these times, nursing staff still performed a full round of observations on their patients, noting the results on informal pieces of paper and later transferring these to observation charts.

Continuity of care was ensured whenever possible, and was prioritised during patient allocation. Nurses had high levels of engagement with parents and involved them in interpretations of a child’s vital signs:

He is very clammy. I asked mum if he gets clammy at night and she said he does but not like this, this is more than normal.

Nurse, field notes

Family concern was also regularly highlighted during nursing handover. Sometimes this was described as something relevant, and other times it was described as parental anxiety or a lack of understanding of the child’s condition. Not all parents were treated as equally legitimate in raising concerns.

Plan

There were a number of mechanisms for reviewing individual patients and unit capacity, which produced different levels of situational awareness across the nursing and medical teams.

Ward staff received a safety briefing at 07.00. This was used to highlight patients at risk of deterioration, or those with particularly challenging pre-existing conditions, as well as general information. The shift’s handover sheet was also distributed during the safety briefing. This was electronically generated, containing information relating to diagnosis, treatment, medication and tests needed or awaiting results (see Report Supplementary Material 19).

More detailed information on all patients was shared during nursing handover, which was conducted outside the patients’ rooms, to enable a visual review of the children. Nurses asked questions to make sense of the case, for example when a point was not clear, or if they felt that the account being provided contradicted their experience of looking after that child during a previous shift. Nurses received handover for all patients and so had an overview of the whole ward. Both the morning and evening handovers followed the same format. During the week, the ward manager, who was a highly experienced clinician, reviewed all patients to identify individuals at risk.

A whiteboard indicated the location of patients on the ward, nurse–patient allocation and the on-shift consultant. A yellow star denoted the sickest patients, although was not always used. The board was regularly updated at nursing handover.

A bed management meeting took place at 14.00 and 20.00 on the PAU. This was held with the patient flow co-ordinators and ward managers, to review organisational capacity. The medical handover took place twice a day at 08.30 and 20.30, and was an opportunity to review all patients in the medical service, with on-coming doctors and ANPs making notes on an electronic pre-populated handover sheet. However, there was no systematic mechanism for identifying children of concern and no ward nurse in attendance.

The general medical team ward round was an opportunity to review all patients and talk to parents. The format varied. Sometimes, the team saw all patients; on other occasions, the team divided into two, with the consultant focused on the sickest patients and the registrar leading the other team. The whole team came together after the ward round to review decisions and plan. In the week, the ward manager always attended the ward round. Children under the care of specialist teams would be seen during the morning ward round for that team, but the timing of this was less predictable and nurse involvement was inconsistent. Nurses would often have to check the medical notes to establish the plans for a child’s care.

Although there were mechanisms in place to ensure that nursing and medical teams had an overview of the patients in their care, there was no opportunity for a shared nursing and medical understanding, and information flows were fragmented. Doctors were not always aware of the most at-risk children on the ward, and did not always prioritise activity appropriately. In addition, the general medical team’s purview extended to all the medical patients in the hospital across two wards, the PAU and A&E, whereas the purview of nurses included children under the care of specialists and the medical patients on a single ward.

Act

There was no hospital-wide PTTT score and no formal escalation policy. Junior nurses typically escalated concerns to the nurse in charge, who then made a decision about escalating to medical colleagues, working up through the medical hierarchy if they were not satisfied with the response. One doctor, usually a SHO, was ward based during the day and was visible at the nurses’ station, which meant that concerns were often addressed informally. Although this seemed to work satisfactorily in practice, there was sometimes a lack of clarity about the process when escalating; this could be particularly challenging at night, when there was more limited medical cover and/or when the nurse raising concerns was more junior. There were particular challenges contacting the on-call consultant at night and some of the specialist consultants, especially if they were from the adult service:

The [name of specialist consultant] was phoned, and said he would come in in 2 hours to review [the patient], but I said, ‘sorry that’s not good enough’. He then told me off for forcing him to make decisions under pressure. But I wasn’t happy at all, I didn’t want to wait, so I sent her to HDU.

Consultant, field notes

At interview, many parents described feeling able to raise concerns about their child with nurses, although this was observed to be more difficult for families for whom English was not their first language. However, involvement was uneven, with not all parents instructed on the use of buzzers to alert the nurse. The physical layout of the ward could make it difficult for families to locate the nurses. See Report Supplementary Material 6, Table 3, for a summary of the pre-implementation system strengths and weaknesses identified by the PUMA team.

The PUMA programme initiatives

The team proposed four initiatives (see Report Supplementary Material 8, Table 3):

1. modified SHINE posters in clinical areas

2. electronic site board at nurses’ stations

3. reviewing and adjusting existing communication mediums

4. escalation plan.

Detect

There were no discernible changes to monitoring, recording or interpreting activities.

We observed a few SHINE posters on the ward, but did not see that nurses oriented parents to them on admission and we observed no examples of their use in practice. Parent interviews revealed no awareness of the tool, although parents did show confidence in alerting nurses to any concerns regarding deterioration.

There had been some improvements to equipment with the introduction of central monitoring, and an increased number of saturation machines and breathing machines (Optiflow Nasal High Flow therapy devices) provided since 2015, although it was not possible to discern their impact.

Plan

The nurses’ morning safety briefing had been getting ‘longer and longer’, with a growing list of additional items of information, with the same announcements appearing over the week. This carried the risk of core information on at-risk children being crowded out. The nurses had started putting some of the information on an information board to keep the briefing shorter (see Report Supplementary Material 23).

To work more efficiently, nurses were now allocated to work in one area of the ward, and received handover on these patients only, rather than on the whole ward. At one level, this reduced the scope of nurses’ situational awareness. At another level, the shortened handover improved nurses’ concentration.

There has been an improvement in the shared situational awareness of children at risk as a result of the implementation of the ‘4Ss’ whiteboard. The senior nurse telephones through to the doctors’ handover to alert them about bed status and the sickest patients, which allowed the medical team to plan ahead before the ward round.

Act

As part of the PUMA programme, the hospital had implemented a formal escalation policy. Although there were low levels of awareness that a formal policy existed, staff had high levels of awareness of the escalation process.

The appointment of new PICU consultants had generated variation in thresholds for admission to PICU, which increased the likelihood of acceptance.

The hospital

Case study 4 was undertaken at Morriston Hospital, a 700-bedded DGH, and part of the newly formed Swansea Bay University Health Board (previously Abertawe Bro Morgannwg University Health Board). The paediatric service included a medical ward and a surgical ward, a PAU, a four-bedded medical HDU and a three-bedded surgical HDU. There was no PICU; very sick children were transferred to the tertiary centre approximately 40 miles away.

The ward

The ward case study focused on a 16-bedded medical ward that provided care for children with long-term conditions; acute episodes of illness, including breathing difficulties; or physical, developmental or feeding problems. Some children regularly attended the ward for treatments. HDU beds were in a separate area in the ward and staffed on a 1 : 2 nurse-to-patient ratio. See Report Supplementary Material 24 for details of the layout.

Staffing

The ward team comprised a ward manager, band 6 (× 2) and band 5 (× 16) nurses, three HCAs, four nursery nurses and two play specialists. Student nurses undertook educational placements on the ward. A complex shift system operated, including long days (07.00–19.30), early shift (07.00–15.00), normal days (07.00–16.30), middle shift (11.30–19.30) and night shifts (19.00–07.30). HCAs worked 07.00–14.30. Staffing was highest between 11.30 and 14.30. The nurse in charge carried a case load. All nurses who worked in the HDU area were HDU trained, but not all ward nurses were HDU trained. The nursing team included both experienced (60%) and recently qualified staff (40%). Nurses worked with the named-nursing model; patients were allocated according to their clinical needs. HCAs had their own patients, but these were typically non-complex cases, and they worked under the supervision of a registered nurse. New admissions to the ward were allocated to a suitably qualified nurse, and the allocation of work revised as necessary to optimise skill mix.

The medical team provided 24-hour cover. There were a variety of shift patterns that were different for the SHOs and the middle grades. Six doctors provided weekday cover: two consultants and four junior doctors (registrar and SHOs). The junior doctors rotated every 4–6 months. Throughout the day, at least two junior doctors were based on the case study ward, dealing with medications, blood tests and reviewing patients as requested by nursing staff. At night and at weekends, a registrar, a SHO and an on-call consultant covered the inpatient paediatric service.

Routines

The main nursing handover at 07.00 was an oral handover from one of the night staff to the day staff. Patients were allocated after handover; this was recorded on a whiteboard in the treatment room. Additional handovers took place at shift changes, typically a one-to-one handover from the outgoing to the remaining or incoming nurses. Handover to the night staff at 19.00 was by the nurse in charge, who handed over all patients for the ward.

The morning doctors’ handover took place at 08.30, and was a full department handover from two doctors who had been on night duty across the paediatric whole department. There was an afternoon handover at 16.30 between the consultant of the week, the on-call consultant and junior doctors, and a 20.00 handover from the middle-grade doctors, who handed over to the doctors working the night shift. A consultant would usually telephone or attend the 20.00 handover in person.

The medical ward round took place at approximately 09.30. This was followed by the nursing–medical board round, attended by the doctors and the nurse in charge.

Family involvement

Involving families was central to patient care, but there were limited facilities for parents to stay with their child over night. Parents slept in a chair next to the child and the ward policy restricted this to one parent only.

Detect

Observation policy was based on RCN guidelines,213 although staff did not explicitly orient to this in practice. All children were required to have vital signs monitored at least every 4 hours, unless otherwise directed by the doctor. The normal set of vital signs observations consisted of heart rate, oxygen levels, temperature and respiratory rate (see Report Supplementary Material 25). In the exceptional cases when a child required additional clinical observations, this was communicated at handover and recorded on a whiteboard, which functioned as a common information space and was located in the treatment room. Roles and responsibilities in relation to detection appeared to be reasonably well understood.

Equipment was problematic: ward staff often had to search for an equipment trolley to perform observations. In addition, there was a shortage of paediatric probes for measuring oxygen saturation and pulse, and a lack of the correct size of cuffs for monitoring blood pressure; equipment was often found not to be working. There was no facility to perform continuous monitoring, which nurses perceived would be of value in cases when obtaining accurate observations could be difficult, such as in young children for whom the process could be distressing.

Formal observation policy required that heart/pulse rate should be checked manually for 1 minute. Throughout the period of observation, the use of a manual pulse was not observed at all, and heart rate was routinely taken from automated readings. Nurses’ reliance on machine readings reflected the additional time it took to manually obtain observations, and a lack of skills in manual approaches in the case of more junior staff. It was also difficult to conduct observations as required at night and during weekends, when there were only two nurses working.

There was no PTTT in use on the ward. There were local guidelines for observation frequency for some conditions, but nurses were encouraged to observe the child as a whole, to understand the normal parameters for each individual patient and to share any concerns. Increasing the frequency of observations was largely left to nurses’ discretion. There were also posters specifying normal physiological thresholds.

On admission, the named nurse encouraged parents to ask for help if the child’s status changed, and established baseline details of the child’s condition, which initiated parental involvement in detection. Nurses took pride in ‘knowing their patients’, and much of this knowledge of ‘what is normal’ for each child was derived from the parents in these first interactions. Nurses also had established relationships with families whose child regularly attended the ward. Nurses at the start of each shift also reinforced parental involvement. During medical review and ward rounds, doctors sought parents’ views on their child’s status and checked if they had specific concerns.

Vital signs were recorded on a paper-based observation chart, which displayed trends. A different observation chart was used in the PAU, which undermined monitoring continuity, as it was necessary to start a new chart on admission to the ward. There was no visual reminder or guidance for staff on the observation charts to indicate whether the plotted observation readings were within normal ranges or a cause for concern. There was no system in place for storing charts, and so these were sometimes difficult to locate, which could result in observations being recorded on pieces of paper before formal charting. Separating the time of collecting observations from the time in which they were recorded on the chart delayed the opportunity to interpret the vital signs alongside previous observations.

Both doctors and nurses asked parents regularly for their perspectives on their child, and this was combined with observations, knowledge of baseline parameters and test results, to provide up-to-date reviews. Parental engagement was uneven. Although there were many examples of constructive and supportive relationships, it could be challenging for nurses to manage parental concerns not considered to have a clinical foundation. Conversely, nurses reported that parents did not always raise concerns with doctors, even though they had raised their concerns with nurses beforehand. This offered an interesting insight into interprofessional relationships on the ward: nurses expressed frustration with this, because they considered that parents had more authority to question the doctors than they themselves, and this was a missed opportunity.

Nurses and support workers continuously reviewed the statuses of children in their care, undertaking additional observations as required and discussing the case with colleagues. From this, a plan for increasing the observation of children or referral to a doctor was made, collaboratively. Nurses emphasised that assessing a child went far beyond physiological indicators:

Paediatrics is very much observation of the whole patient, not just the figures. [. . .] sometimes you just look at the child, as a nurse, and you know this child is seriously sick. And it goes by nasal flaring, tracheal tugs, you don’t even need to do the respiratory rate, you can actually see the child is seriously ill. [. . .] You take in the whole picture of the patient [. . .] not just a parameter.

Senior Nurse

Plan

There were a number of mechanisms for reviewing individual patients: the separate nursing and medical handovers, the ward round and the post-ward round nursing–medical board round. The 07.00 nursing handover was the primary means for nurses to review all patients and ward status. Nurses received handover on all patients and made notes on a printed bed plan of the ward. There was no pre-populated electronic handover sheet. Each child was considered in bed order. Patients at risk of deterioration were not explicitly highlighted in one single section of nursing handover, but nurses generally noted if they were concerned about a patient. At the evening nursing handover, individual nurses handed over their patients to the night staff. Smaller handovers occurred at shift changes, but related only to the patients being handed over, not the whole ward.

The medical staff handover took place at 08.30. This was a full department handover attended by the doctors working the day shift on the PAU and medical and surgical wards. Doctors worked with pre-populated electronic handover sheets, which were updated, at a minimum, at the end of each shift. It included information on concerns and treatment plans on all paediatric patients. Additional handovers took place at 16.30 between the consultant and consultant on call and at 20.00 between the middle-grade doctors and the night-shift team. Consultants often called in for the 20.00 handover.

An explicit purpose of the medical handover was to identify the children of most concern who should be reviewed first, including children in the HDU:

[O]ne of the first questions they’ll say is ‘which ones are the children that you’re a bit concerned about or that we need to be more aware of, or that needs to be seen sooner rather than later?’, so we’ve already got an idea, often, from leaving handover which ones to keep an eye out for.

Registrar

Children were reviewed and treatment plans agreed on the ward rounds. On the medical ward round, the team divided into two groups, one led by the consultant and the other led by the registrar. The consultant saw the most ill patients, starting in the HDU. Although senior medical staff expressed a preference for a nurse to attend the ward round, this was considered impractical by the nursing team because the ward round took place at the same time as nurses were taking breaks and the nurse in charge carried a case load.

The main opportunity for a joint review by doctors and nurses was the board round that immediately followed the ward round. The nurse in charge, or their nominee, attended the board round and communicated information to colleagues. The board round took place in the treatment room and information was reviewed and updated on the whiteboard. This was consulted frequently by both medical and nursing staff, and was an important central resource for sharing information. Nurses took responsibility for maintaining the board, but did not always have time to update it.

Beyond these formal mechanisms, reviews of patient status and ward activity were woven through daily activities. Doctors and nurses discussed patients and planned for action; a new admission could prompt a review of overall acuity across the ward and the identification of patients considered to be at risk:

Because, with our assessment of when people come in, kids come into the ward, we will often rearrange, if we think somebody is a potential to get unwell, we will try to get them closest to this end, to the desk and the HDU area of the ward, rather than being, like, 30 feet down the corridor. So we’ll have them up here anyway.

Nurse

Act

This site did not use a PTTT; thus, there was no standardised score or trigger that prompted action. The escalation policy was based on the RCN guidelines,213 which recommend additional checks of vital signs, but no other forms of action. There were some general indicators in place for actions related to vital signs, but there was low awareness of a formal graded escalation policy. Decisions about when and how to escalate care was at the discretion of clinical staff.

Nurses reported that they would have no hesitation in seeking a senior medical review if this was indicated. Nurses typically first discussed the situation with each other; there was a strong informal network of peer support on the ward, and official support from more senior experienced nurses. Furthermore, during the day, doctors were often present on the ward or in the attached HDU, and so, in practice, nurses were pragmatic and, rather than working their way through the medical hierarchy, would speak to doctors who were on the ward or nearby. The doctors also preferred that junior nurses consulted with more senior nurse colleagues initially, with one doctor noting the tension between nurses reporting concerns and doctors’ ability to manage requests:

And so I do think it’s a good idea for junior nurses to speak to somebody senior on their ward, but if they can’t, if they’re off the ward, then they shouldn’t feel that they have to wait . . . And that’s, and that’s a challenge, I think, in trying to balance those two things, so encouraging people to not always ring, but not wanting them to feel like they can’t ring.

Consultant

Escalation could be a delicate balance, and we observed that tensions could sometimes surface, both within the nursing team and between nurses and doctors. HDU staff routinely reviewed cases of mortality or critical incidents. See Report Supplementary Material 6, Table 4, for a summary of the Morriston pre-implementation system strengths and weaknesses identified by the PUMA study team.

The PUMA programme initiatives

Fifteen initiatives were proposed at this site (see Report Supplementary Material 8, Table 4):

1. Update and disseminate the observation policy.

2. Create posters and cards for staff to signpost abnormal thresholds for vital signs.

3. Update observation charts to include normal range thresholds for vital signs.

4. Conduct inventory of equipment.

5. Formally establish Deteriorating Child Study Day across the health board.

6. Roll out an in-house electronic learning (e-learning) package for nursing and medical staff.

7. Ward nursing staff to spend more time observing HDU staff.

8. Move to adopt three daily huddles/board rounds.

9. Introduce process for identifying ‘watchers’ at each huddle, for example with markers on a whiteboard.

10. Review handover content. Possibility of including nursing staff in medical handover.

11. Re-establish a nursing supernumerary role.

12. Establish a staff training course on situational awareness.

13. Review and disseminate existing escalation policy.

14. Review communication tools to aid escalation of patient care.

15. Explore tools for family/parental involvement.

Although work in some areas had already started before the PUMA programme, these were all included in the site action plan. Together, these were designed to improve the following: detection, by making changes to monitoring, recording and interpreting activity; planning for escalation, by prioritising/formalising reviewing of patients and using advance planning for escalation; and action, by timely escalation and confident planned response to deterioration.

Some initiatives were designed to come into effect as soon as possible. This was driven strongly by the leads of each initiative and enabled, in part, by earlier development work.

Detect

As a result of the equipment inventory, access to monitoring equipment on the ward had improved. Each cubicle/room had its own monitoring equipment, so that searching for the right tools was no longer necessary. New mobile observation trolleys with appropriate equipment (paediatric probes) and multiple operational capacity to record temperature, oxygen saturation, respiratory rate and blood pressure were held in the central corridor of the ward and used by staff in the patient bay areas. Observation trolleys were cleaned daily, and breakages or stock issues were reported at the same time. Ward staff unanimously considered the new equipment to have made their work much easier, with equipment much more accessible.

A laminated copy of the RCN observation and escalation guidelines213 was included in every patient file on admission, as a reminder to staff. However, despite the initiative to raise awareness of the new policy, this was not evident in the post-implementation fieldwork. This was one of the earliest initiatives implemented as part of the PUMA programme; therefore, it is possible that the policy had been normalised by the post-implementation period.

The detection of vital signs outside normal physiological parameters had been addressed in several ways. First, by early implementation of easily accessible information for staff through updated posters and cards showing normal/abnormal thresholds. Although the posters were not in evidence in the post-implementation period, the majority of staff were aware of the cards, and card use continued, albeit infrequently. Second, the accessible information campaign was subsequently followed by the introduction of the observation chart, which was colour-coded to indicate normal vital signs thresholds. The same chart was implemented across the medical ward, the PAU and the surgical ward, which improved monitoring continuity and recognition of vital signs trends.

The challenges nurses had in locating observation charts in the pre-implementation period, which created delays in formally recording observations, had been mitigated by ward-level changes to the storage and management of patient information. Observation notes were kept in the patient notes in individual red ring binders and stored in a trolley in the treatment room. There was a notice to the doctors to remind them to replace the red binders in the trolley as soon as possible, and not to separate the contents. Nursing staff observed that one or two regular offenders needed to be reminded, but, for the most part, staff appeared to take binders and return them appropriately, and there was no confusion over the location of the observation charts.

Plan

Although the improvement team decided that there was no need to increase the frequency of board rounds/safety huddles, the post-implementation fieldwork indicated that there was increased attention to situation awareness. The team had implemented the 5Ss at every handover. At-risk children were consistently designated a watcher status at board round and the contents of the whiteboard appeared to be more regularly updated and more complete. Nurses appeared to update the board during their work, while they performed the required tasks shown and those decided during handover. Doctors looked at the whiteboard frequently in between seeing patients in the ward round. We were not able to assess the impact of changes to the nurses’ shifts on communication.

Act

There was little evidence of increased awareness about the escalation policy, as in the pre-implementation phase, and no discernible differences in staff accounts of summoning medical teams. However, staff interviews in the post-implementation period suggested that, when there was a risk of deterioration, patients were moved into the HDU more quickly than previously:

The children who are sick in our cubes are escalated quickly to HDU [. . .] You know all the doctors seem to be very quick [. . .] quick and prompt, like, on moving them before, you know, things deteriorate.

Nurse

From paediatric track-and-trigger tools to the formalisation of a whole-systems approach

In the paediatric community, there has been considerable interest and debate about the potential of PTTTs to improve processes for detecting and acting on deterioration among hospitalised children. Although there is little evidence for the effectiveness of any specific tool in reducing mortality or critical events, PTTTs do have value as mechanisms for co-ordinating action across clinical teams, but they depend on certain preconditions for their use. In addition, paper-based and electronic PTTTs function differently in the overall system. At the time of writing, there is a policy impetus for the implementation of national PTTTs; on the basis of these findings, there is no good reason to question such initiatives. It is increasingly clear, however, that PTTTs are not the sole solution to improving processes for detecting and acting on deterioration; they should be implemented as part of a wider systems approach. Indeed, over the life of the study, there has been a growing professional awareness of the need for a whole-systems approach to improving processes for detecting and acting on deterioration, but hitherto no formal framework has existed for improvement purposes. The PUMA Standard addresses this need.

Localisation and standardisation

A central tenet of the PUMA programme is that paediatric early warning systems are shaped by the local organisational context; the overall philosophy of OUTCOME is to enable context-appropriate approaches to improvement. These findings highlight how factors such as technology, architecture, shift patterns and the social organisation of nursing and medical work all affect the functioning of the system and condition the options for intervening to bring about improvement. We have shown how the PUMA programme facilitated locally tailored approaches oriented to a common standard across the varied service contexts in four study and three pilot sites; therefore, it has value as a framework for continuous improvement across diverse national and international contexts. Implemented at scale, there is also the potential for shared learning across health-care systems, whereby organisations with similar contextual conditions share successful approaches to the same problem.

Although the overall philosophy of the PUMA programme highlights the importance of local context-appropriate approaches to improving paediatric early warning systems, the study findings point to a number of areas where common standards may have value.

First, clinical expertise is an essential component of any paediatric early warning system, and staff turnover has potentially disruptive effects. Professional development is thus a critical component of all systems. Two sites struggled to embed educational programmes, yet, where such programmes were a mandated requirement, they were successfully implemented. There are compelling examples elsewhere, whereby mandated multidisciplinary training has brought about improvements in practice,221,222 an approach that merits further consideration in the context of paediatric early warning systems.

Second, lack of access to appropriate monitoring equipment affects the system negatively. A process to ensure that the correct equipment is available and functioning is a prerequisite of any paediatric early warning system, irrespective of the singular features of local context.

Third, all sites recognised the importance of involving parents in detecting and acting on deterioration, but had limited success in implementing changes to the system. Parental involvement in the detection of deterioration is difficult to address outside wider strategies to facilitate parental involvement in their child’s care.

Learning systems

The PUMA programme provides structures to support a learning system and our insights on the dynamic qualities of paediatric early warning systems indicate that regular assessment of system functioning has value, in order to intervene to ensure alignment with the PUMA Standard. At the end of the study, some sites proposed to use the PUMA Standard as a structure for systematically assessing critical events. Beyond critical incident analysis, however, the PUMA Standard and assessment tools offer resources for systematically appraising paediatric early warning system functioning as part of a continuous improvement culture. In other areas of health-care practice, checklists have value in ensuring that all the elements necessary to an activity are lined up in the right place and at the right time. However, even when their use is mandated, checklists all too often lose their effectiveness as staff bypass processes that they feel are redundant. PUMA offers a self-evaluation approach that engages staff in understanding local challenges that they need to address and offers structures to support systematic and rigorous local improvement efforts in relation to a service standard. Embedded in PUMA is the understanding that the context of care is continuously changing; therefore, it encourages a framework for deterioration review and the ability to set local stands for audit.

Detecting and acting on deterioration beyond paediatric hospital wards

The PUMA Standard is functions oriented, so it has applicability beyond paediatric inpatient wards. Work is already under way to implement the PUMA programme in Qatar, where it has been deployed to bring about improvements in a paediatric emergency department, which has now been open for 2 years, with 100,000 patients presenting per year. There is also scope for extending the approach to address global health challenges in developing health-care systems, where rescue trajectories must be implemented in conditions of significant sociomaterial constraint. Beyond paediatrics, TTTs have been deployed widely in adult care contexts to identify signs of deterioration. Although there is stronger quantitative evidence for TTT use in terms of clinical outcomes, here too more work is needed to understand the core components of early warning systems and the mechanisms of action of a TTT within an overall system. 223 Because it is underpinned by a functions-based approach, with minor adjustments, the PUMA programme has applicability for use across other patient populations beyond paediatrics.

Wider application of OUTCOME to health-care improvement

In the face of disappointing results in more than a decade of activity to bring about improvements in health-care quality and safety, there is a growing recognition of the need to move away from top-down solutions informed by a cause-and-effect logic, and to embrace more locally tailored approaches founded on an understanding of health care as a complex adaptive system. 201 In the context of this paradigm shift, the OUTCOME approach could be extended to other areas of health-care organisation and delivery, where there is a desire to adopt new approaches to service improvement.

In PUMA, the PUMA Standard and associated assessment tools were central to the improvement programme and a distinctive feature of the OUTCOME approach. The development of the PUMA Standard, through three systematic reviews and consultation with clinicians and parents, required significant expertise and resources. Alternative less resource-intensive approaches might include the use of existing clinical guidelines or service-level specifications. Standards could also be agreed through conducting a Delphi study or through professional consensus. In the longer term, a logical corollary of an OUTCOME approach is the generation of outcomes-focused systems standards through the use of new methods for systematic reviews and guideline development.

OUTCOME also provides a mechanism for PPI in QI processes. In the PUMA study, there was PPI in the initial stages of the project in agreeing the goals to be achieved (principle 1) and local system assessment (principle 4), but, going forward, there is scope for this to be extended to include defining local initiatives (principle 5) and learning systems (principle 6).

Quantitative outcomes for evaluating deterioration in paediatric contexts

Determining the impact and effectiveness of the PUMA programme using quantitative measures of inpatient deterioration was challenging. The original commissioning brief related to interventions to reduce mortality, and so our primary outcome (‘adverse events’) was a composite measure that included this measure and other related clinical metrics such as respiratory arrests and unplanned intensive care presentations.

The decision to use a composite metric for the primary outcome mirrors what has been done in many other single-site effectiveness studies of paediatric early warning system interventions. 199 As with other studies, it was largely a pragmatic decision, reflecting the low event rates of individual clinical outcomes such as mortality and arrests in hospitalised children. Even using this composite outcome, incorporating unplanned HDU and PICU transfers, we observed several zero months in the smallest DGH. Low event rates for key outcome metrics in DGHs point to the difficulty in assessing changes over time in smaller hospitals, and is a key reason that the literature on paediatric early warning systems is currently dominated by studies conducted in large specialist centres.

Mortality is significantly lower in children than in adult inpatient settings;1,224 there is an ongoing decline in child mortality over time,2 and even inpatient deterioration is a relatively infrequent occurrence in the context of large numbers of children moving through a hospital in a short space of time. 3,4 Our review of the literature also indicated huge variations in the definition of outcome measures, which makes synthesis and comparison difficult (see Chapter 3). Analytic approaches to rare event modelling, such as Bayesian belief networks, could be adapted from other fields to support the focus on preventing these events; however, a clear assessment of potential is required. Much of the safety literature on rare events requires clear causal pathways to be identifiable and measurable; the complexity of child deterioration and death may not be amenable to such approaches.

Including HDU and PICU transfers as markers of inpatient deterioration is common in the literature, but not without its problems. As we were able to demonstrate in the qualitative work, use varies in response to other system pressures or changes in clinical practices of senior staff.

The findings lend weight to debates about the appropriateness of downstream individual-level outcome measures in this field, and point to the need to reach agreement on upstream indicators of paediatric early warning system performance that are aligned with the PUMA Standard, for example monitoring compliance, situational awareness, parental involvement, staff knowledge and skills, or organisational culture in relation to escalation. The PUMA Standard offers a valuable framework for progressing this thinking, through consensus methods, such as a Delphi study.

Relational co-ordination in paediatric early warning systems

All of the study sites identified the need to address communication between nursing and medical teams to bring about improvements in situational awareness. Sites identified a mixture of structural and relational initiatives to improve interprofessional co-ordination. All sites experienced challenges in implementing structural approaches to co-ordination by closer alignment of nursing and medical organisational arrangements; in two sites, efforts to augment relational co-ordination through the introduction of supernumerary status of the nurse in charge were not successful. The findings suggest that senior nurses have a central role in ensuring situational awareness in paediatric early warning systems, but this is constrained by the requirement to carry a clinical case load. Research in other areas of health care has shown the importance of nurses in co-ordinating care across the interdisciplinary team. 214 Given the challenges of structural approaches to co-ordination, new research is necessary to explore the costs and consequences of models of nursing that facilitate implementation of a supernumerary nurse co-ordinator role.

Extension of the approach

There is potential for further research to examine the extension of the approach to other paediatric contexts and other areas of health-care practice. Of particular interest is whether or not the PUMA programme has value in improving paediatric early warning systems in developing health-care systems, where the sociomaterial contexts for practice differ widely from those of the UK NHS, and how far the PUMA Standard may have applicability in the adult care context. In addition, there would be value in further research that deploys the OUTCOME framework to address other areas of health-care practice, beyond paediatric early warning systems, where system complexity continues to confound service quality and patient safety, for example the organisation of hospital discharges or the management of complex continuing care arrangements in the community.

Evaluation of impacts of mandated and voluntary system improvement programmes

The PUMA programme was explicitly designed to harness local systems expertise to bring about contextually appropriate improvements. The approach was informed by insights from a critical body of work in improvement science, which has highlighted the limitations of the top-down, solutions-driven models typical of modern health-care systems. These findings highlight the challenges of locally led improvement in the absence of organisational sponsorship (in three of the study sites), as well as the potential impacts on clinical outcomes on goal-oriented mandated system-level change, as illustrated by the example of Alder Hey. Further research is needed to explore how approaches to improvement that are goal-oriented and locally owned could be strengthened through a mandatory framework. In particular, research to examine the probable impacts of mandating a system-wide improvement programme in the context of paediatric early warning systems merits serious consideration.

The value of translational mobilisation theory as a theoretical framework for complex systems research

Translational mobilisation theory is a new theory, which was used for the first time, to our knowledge, in this study to inform the hermeneutic systematic review, the development of the PUMA Standard and qualitative data generation processes. As Davidoff et al. 225 argue, there is an urgent need for the use of more formal theory in improvement research, not least because it facilitates learning, accumulative understanding and knowledge transfer. The PUMA study indicates that TMT offers a useful framework for understanding complex organisational systems that is grounded in the material and cognitive processes through which health-care activities are accomplished and the relational mechanisms that support or inhibit concerted action, and thus has value for future research.

Description

#1 “early warning score*”

#2 “early warning system*”

#3 “early warning tool*”

#4 “VitalPAC Early Warning Score”

#5 “activation criteria”

#6 “Rapid Response Team”

#7 “Rapid Response system*”

#8 “Track and trigger”

#9 “trigger tools”

#10 “calling criteria”

#11 “Alert criteria”

#12 “Rapid Response”

#13 #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9 or #10 or #11 or #12

#14 pediatric* or paediatric* or infant* or child* or baby or toddler or babies or teen* or adolescent*

#15 #13 and #14

#16 “Pediatric Early Warning”

#17 “Paediatric Early Warning”

#18 “p?ediatric alert”

#19 “Pediatric Rapid Response”

#20 “Pediatric Advanced Warning Score*”

#21 “Paediatric Advanced Warning Score*”

#22 “infant early warning”

#23 “Bedside PEWS”

#24 “Bedside paediatric early warning”

#25 #16 or #17 or #18 or #19 or #20 or #21 or #22 or #23 or #24

#26 #15 or #25 Publication Year from 2016 to 2018

Search strategy

1. (“early warning” adj5 scor*).ab,ti. (568)

2. (“early warning” adj5 system* adj5 (deteriorat* or mortality or death or outcome* or harm* or safety)).ab,ti. (51)

3. “acute illness severity”.mp. (38)

4. early intervention/and ((prevent* or reduc* or improv*) adj5 (deteriorat* or mortality or death or outcome* or harm* or safety)).ab,ti. (1185)

5. (“early medical intervention” adj5 (tool* or scor* or index* or indicator* or indice* or assessment* or guide* or instrument* or criteria or parameter* or deteriorat* or mortality or death or monitor* or outcome* or harm* or safety)).ab,ti. (10)

6. *”severity of illness index”/and ((tool* or scor* or index* or indicator* or indice* or assessment* or instrument* or criteria or parameter*) adj5 ((prevent* or reduc* or improv*) adj5 (deteriorat* or mortality or death or outcome* or harm* or safety))).ab,ti. (3)

7. exp Health Status Indicators/and ((tool* or scor* or index* or indicator* or indice* or assessment* or instrument* or criteria or parameter*) adj3 ((prevent* or reduc* or improv*) adj3 (deteriorat* or mortality or death or outcome* or harm* or safety))).ab,ti. (7)

8. rapid response team/(849)

9. “alarm monitor”/and (prevent* or reduc* or improv*).mp. (245)

10. (“clinical alarm” adj5 (prevent* or reduc* or improv*)).mp. (2)

11. (outreach adj3 emergency).tw. (46)

12. VitalPAC Early Warning Score.tw. (15)

13. medical emergency team.tw. (395)

14. Rapid Response Systems.mp. (140)

15. (“rapid response” adj5 (prevent* or reduc* or improv*)).tw. (191)

16. (“medical device” adj3 (prevent* or reduc* or improv*)).mp. (187)

17. (((Detecting or managing) adj3 deterioration) and warning).tw. (11)

18. track-and-trigger system.tw. (24)

19. (Track adj trigger).tw. (4)

20. (Track and trigger).tw. (241)

21. trigger tools.tw. (47)

22. (“alert criteria” or “activation criteria” or “calling criteria”).tw. (209)

23. SBAR technique*.mp. (5)

24. (score adj3 severity of illness).tw. (393)

25. or/1-24 (4295)

26. limit 25 to (infant <to one year> or child <unspecified age> or preschool child <1 to 6 years> or school child <7 to 12 years> or adolescent <13 to 17 years>) (533)

27. P?ediatric Early Warning.mp. (120)

28. p?ediatric alert.tw. (7)

29. p?ediatric early warning systems.mp. (4)

30. p?ediatric risk of mortality.tw. (527)

31. P?ediatric Rapid Response Team.tw. (14)

32. Point-of-Care Systems/and ((paediatric or pediatric) adj3 (improve or identify or detect* or outcome or early or critical or emergency)).tw. (23)

33. P?ediatric Advanced Warning Score.tw. (3)

34. neonatal early warning.tw. (1)

35. infant early warning.tw. (0)

36. p?ediatric rapid response.tw. (31)

37. Bedside paediatric early warning.tw. (5)

38. Bedside PEWS.tw. (7)

39. or/27-38 (707)

40. 26 or 39 (1155)

41. limit 40 to human (1065)

Search strategy

1. (“early warning” adj5 scor*).ab,ti. (23)

2. (“early warning” adj5 system* adj5 (deteriorat* or mortality or death or outcome* or harm* or safety)).ab,ti. (6)

3. “acute illness severity”.mp. (3)

4. “early medical intervention”/and ((prevent* or reduc* or improv*) adj5 (deteriorat* or mortality or death or outcome* or harm* or safety)).ab,ti. (0)

5. (“early medical intervention” adj5 (tool* or scor* or index* or indicator* or indice* or assessment* or guide* or instrument* or criteria or parameter* or deteriorat* or mortality or death or monitor* or outcome* or harm* or safety)).ab,ti. (0)

6. Health Status Indicators.mp. and ((tool* or scor* or index* or indicator* or indice* or assessment* or instrument* or criteria or parameter*) adj3 ((prevent* or reduc* or improv*) adj3 (deteriorat* or mortality or death or outcome* or harm* or safety))).ab,ti. (0)

7. exp “Severity of illness index”/and ((tool* or scor* or index* or indicator* or indice* or assessment* or instrument* or criteria or parameter*) adj5 ((prevent* or reduc* or improv*) adj5 (deteriorat* or mortality or death or outcome* or harm* or safety))).ab,ti. (0)

8. “activation criteria”.ab,ti. (2)

9. exp Rapid response teams/(39)

10. Clinical Alarms.mp. (0)

11. (outreach adj3 emergency).tw. (2)

12. VitalPAC Early Warning Score.tw. (0)

13. medical emergency team.tw. (15)

14. Rapid Response Systems.mp. (8)

15. Rapid Response Team.tw. (27)

16. ((Detecting or managing) adj3 deterioration).tw. (1)

17. track-and-trigger system.tw. (2)

18. (Track adj trigger).tw. (1)

19. (Track and trigger).tw. (8)

20. trigger tools.tw. (4)

21. Calling criteria.tw. (1)

22. Alert criteria.mp. (1)

23. Rapid response.tw. (111)

24. (score adj3 severity of illness).tw. (3)

25. or/1-24 (171)

26. (pediatric* or paediatric* or infant*1 or child* or baby or toddler or babies or teen* or adolescent*).mp. (40,161)

27. 25 and 26 (14)

28. p?ediatric alert.tw. (0)

29. p?ediatric early warning systems.mp. (1)

30. p?ediatric risk of mortality.tw. (4)

31. Pediatric Rapid Response Team.tw. (0)

32. Point-of-Care.mp. and ((paediatric or pediatric) adj3 (improve or identify or detect* or outcome or early or critical or emergency)).tw. (0)

33. Pediatric Advanced Warning Score.tw. (0)

34. neonatal early warning.tw. (0)

35. infant early warning.tw. (0)

36. paediatric rapid response.tw. (1)

37. pediatric rapid response.tw. (0)

38. Bedside paediatric early warning.tw. (0)

39. Bedside PEWS.tw. (0)

40. p?ediatric early warning.mp. (2)

41. care.mp. and ((paediatric or pediatric) adj3 (improve or identify or detect* or outcome or early or critical or emergency)).tw. [mp = title, other title, abstract, heading words] (57)

42. or/28-41 (59)

43. 27 or 42 (70)

Search strategy

1. (“early warning” adj5 scor*).ab,ti. (260)

2. (“early warning” adj5 system* adj5 (deteriorat* or mortality or death or outcome* or harm* or safety)).ab,ti. (24)

3. “acute illness severity”.mp. (21)

4. “early medical intervention”/and ((prevent* or reduc* or improv*) adj5 (deteriorat* or mortality or death or outcome* or harm* or safety)).ab,ti. (99)

5. (“early medical intervention” adj5 (tool* or scor* or index* or indicator* or indice* or assessment* or guide* or instrument* or criteria or parameter* or deteriorat* or mortality or death or monitor* or outcome* or harm* or safety)).ab,ti. (7)

6. exp Health Status Indicators/and ((tool* or scor* or index* or indicator* or indice* or assessment* or instrument* or criteria or parameter*) adj3 ((prevent* or reduc* or improv*) adj3 (deteriorat* or mortality or death or outcome* or harm* or safety))).ab,ti. (166)

7. “Severity of Illness Index”/and ((tool* or scor* or index* or indicator* or indice* or assessment* or instrument* or criteria or parameter*) adj5 ((prevent* or reduc* or improv*) adj5 (deteriorat* or mortality or death or outcome* or harm* or safety))).ab,ti. (274)

8. exp Hospitals/and ((Detecting or managing) adj3 deterioration).tw. (2)

9. (“medical device” adj3 (prevent* or reduc* or improv*)).mp. (58)

10. (“alert criteria” or “activation criteria” or “calling criteria”).tw. (121)

11. Hospital Rapid Response Team/(334)

12. Clinical Alarms/(332)

13. (outreach adj3 emergency).tw. (32)

14. VitalPAC Early Warning Score.tw. (10)

15. medical emergency team.tw. (247)

16. Rapid Response Systems.mp. (87)

17. Rapid Response Team.tw. (185)

18. (((Detecting or managing) adj3 deterioration) and warning).tw. (8)

19. track-and-trigger system.tw. (14)

20. (Track adj trigger).tw. (2)

21. (Track and trigger).tw. (137)

22. trigger tools.tw. (22)

23. SBAR technique*.mp. (3)

24. (“rapid response” adj5 (prevent* or reduc* or improv*)).tw. (117)

25. (score adj3 severity of illness).tw. (243)

26. or/1-25 (2286)

27. limit 26 to (humans and “all child (0 to 18 years)”) (453)

28. P?ediatric Early Warning.mp. (38)

29. p?ediatric alert.tw. (5)

30. p?ediatric early warning systems.mp. (3)

31. p?ediatric risk of mortality.tw. (400)

32. P?ediatric Rapid Response Team.tw. (6)

33. Point-of-Care Systems/and ((paediatric or pediatric) adj3 (improve or identify or detect* or outcome or early or critical or emergency)).tw. (79)

34. P?ediatric Advanced Warning Score.tw. (2)

35. neonatal early warning.tw. (0)

36. infant early warning.tw. (0)

37. p?ediatric rapid response.tw. (20)

38. Bedside paediatric early warning.tw. (2)

39. Bedside PEWS.tw. (2)

40. or/28-39 (542)

41. 27 or 40 (943)

Trial registers

• ClinicalTrials.gov: https://clinicaltrials.gov/.

• UK Clinical Trials Gateway: www.ukctg.nihr.ac.uk/default.aspx.

• The World Health Organization trial search portal for studies worldwide: https://apps.who.int/trialsearch.

Journal websites

• British Medical Journal: www.bmj.com/theBMJ.

• BMJ Quality and Safety: http://qualitysafety.bmj.com/.

Websites and organisations

• The Health Foundation.

Exploratory analysis 1

This analysis showed a pre-intervention upwards trajectory in adverse events (β = 0.02, 95% CI 0.00 to 0.03), compared with a downwards trend in adverse events in the post-intervention period, which was significantly different from the pre-intervention trajectory (β = –0.09, 95% CI –0.16 to –0.01; p = 0.03).

FIGURE 28.

Alder Hey exploratory analysis 1.

Exploratory analysis 2

In this analysis, we explored the effect of choosing different cut-off points for the pre- and post-intervention periods, across each of the 12 months of the implementation period. This created 13 separate models, with different pre- and post-intervention inflection points. Each of these cut-off points showed a significant downwards trajectory for post-intervention trends of adverse events, relative to a slight upwards trend in the pre-intervention period.