Prehospital early warning scores for adults with suspected sepsis: the PHEWS observational cohort and decision-analytic modelling study

The problem

Sepsis is a life-threatening reaction to an infection in which the immune system overreacts to infection and causes organ damage. The Third International Consensus Definitions for Sepsis and Septic Shock (sepsis-3) defines sepsis as life-threatening organ dysfunction caused by a dysregulated host response to infection, in which organ dysfunction can be represented by an increase in the Sequential (Sepsis-related) Organ Failure Assessment (SOFA) score of two points or more, and is associated with an in-hospital mortality > 10%. 1

This definition has recognised limitations. Sepsis is a broad term applied to an incompletely understood process with, as yet, no criteria that uniquely identify a septic patient. In adults, the clinical presentation of infection and new organ damage is often seen in people with long-term conditions where a self-limiting or easily treatable infection is exacerbating underlying comorbidity, rather than causing organ damage through a dysregulated response. 2 In such circumstances, comorbidities or frailty may be the main determinants of mortality. 3,4

Early recognition and treatment of sepsis has the potential to reduce mortality, depending upon the extent to which mortality risk is due to a dysregulated response to infection. Guidelines for sepsis highlight the importance of early recognition and treatment, with treatment recommended within 1 hour of presentation for those at highest risk. 5–9 This can only be achieved if sepsis is recognised and prioritised in the emergency care system.

Sepsis can be recognised by identifying clinical features of the host response or organ dysfunction, such as altered mental state, low blood pressure or rapid respiratory rate. Early warning scores use simple measurements to calculate a score, with a higher score indicating a higher risk of serious illness and adverse outcomes. They can be used by prehospital providers, such as ambulance paramedics, to identify people with suspected sepsis who need to be prioritised for treatment. 10 Prioritisation can involve a range of options based around pre-alerting the emergency department (ED), allowing the patient with sepsis to be seen before other patients. This could lead to the patient with sepsis being seen immediately on arrival by a clinician who is able to provide time-critical treatment for sepsis and refer for urgent specialist care. Early warning scores could also be used to select people for prehospital treatment for sepsis, such as intravenous fluids or antibiotic therapy, to further reduce treatment delays.

The recognised limitations of the definition of sepsis and uncertainty around the benefits of early treatment can lead to lack of clarity in determining what an early warning score for sepsis is intended to diagnose or predict. An early warning score could be used to identify patients with a diagnosis of sepsis according to the sepsis-3 definition, to predict patients with the highest risk of adverse outcome from sepsis, or to identify patients most likely to benefit from early treatment or a specific intervention for sepsis.

An effective early warning score needs to be accurate and implemented with an appropriate balance between sensitivity and specificity. High sensitivity is needed to ensure that people with the potential to benefit from urgent treatment are not missed, with consequent delayed treatment and avoidable mortality and morbidity. However, sacrificing specificity to maximise sensitivity can result in overtriage, in which people without sepsis or the potential to benefit from urgent treatment are inappropriately prioritised. This increases the pressure on EDs and impairs their ability to provide rapid treatment to those who require it. It may also result in inappropriate prehospital treatment with associated consequences, especially if prehospital scope of practice for sepsis includes antibiotic therapy.

The problem of overtriage is compounded if early warning scores are applied to an unselected population with a low prevalence of sepsis or a high prevalence of conditions that increase early warning scores in the absence of sepsis. National Institute for Health and Care Excellence (NICE) guidance advises thinking ‘could this be sepsis?’5 if a person presents with signs or symptoms that indicate possible infection and highlights that people with sepsis may have non-specific, non-localised presentations, and may not have a high temperature. This could result in sepsis being suspected in any patient attended by emergency ambulance for a medical complaint that is not attributable to a clear alternative cause. An early warning score with poor specificity could therefore result in overtriage of a huge number of patients, placing severe pressure on the emergency care system.

Uncertainty and evidence deficit

The NICE Guideline Development Group (GDG) identified a number of early warning scores that are easy to use, only require simple measurements and could therefore be used in the prehospital setting. 5 These are the Simple Triage Scoring System (STSS), Rapid Emergency Medicine Score (REMS) or modified-REMS, the Modified Early Warning score (MEWS) and National Early Warning score (NEWS). 11–14 They were developed through expert consensus or analysis of routine data from hospitalised patients and contain similar measures (heart rate, respiratory rate, oxygen saturation, blood pressure and conscious level) but differ in their calculation. REMS, MEWS and NEWS have been shown to predict adverse outcome in acute medical admissions, while STSS has been shown to predict mortality in inpatients with suspected infection.

The NICE GDG identified other early warning scores for use in hospital, such as Mortality in the Emergency Department, Sepsis and Predisposition, Infection, Response and Organ dysfunction, but did not recommend them for prehospital use on the basis that they include blood tests that are not currently available in the prehospital setting. 5

A meta-analysis and systematic review of in-hospital studies suggested that early warning scores predicted mortality in sepsis with limited accuracy, based on poor-quality data. 15 A systematic review of early warning scores undertaken for NICE guidance identified 47 relevant studies (including studies of in-hospital scores). All were judged as being of very low quality. There was significant variability in population, outcomes and analysis, so meta-analysis was not possible. No relevant economic evaluations were identified. The guideline recommended that clinicians consider using an early warning score to assess people with suspected sepsis in acute hospital settings and recommended research to determine whether early warning scores can be used to improve the detection of sepsis.

Two systematic reviews of prehospital identification of sepsis also reported limited existing evidence and a need for further research. Smyth et al. reported three studies developing prehospital sepsis screening tools for adults and six studies of paramedic diagnosis of sepsis. 16 Lane et al. reported nine studies of prehospital identification of sepsis. 17 Three of the studies from these systematic reviews evaluated sepsis-specific prehospital scores [Prehospital Early Sepsis Detection (PRESEP), Prehospital Severe Sepsis (PRESS) and the CIS], and the other studies evaluated MEWS, the Systemic Inflammatory Response Syndrome (SIRS) criteria and the Robson tool. 18 Study quality was poor, sensitivity (0.43–0.86) and specificity (0.47–0.87) varied markedly, and none of the screening tools had been validated.

More recently, the qSOFA score has been derived and validated to predict death in hospitalised patients with suspected sepsis and NEWS has been updated to become the National Early Warning Score, version 2 (NEWS2). 19–21 A meta-analysis of 38 recent studies of qSOFA reported pooled sensitivity of 0.61 and pooled specificity of 0.72 for mortality. 22 A systematic review comparing qSOFA and hospital early warning scores for prognosis in suspected sepsis in ED patients suggested that qSOFA has better specificity for predicting adverse outcome at its recommended threshold but NEWS has better sensitivity. 23

Other recent studies have developed scores in the prehospital setting. Smyth et al. derived and validated the Screening to Enhance PrehoSpital Identification of Sepsis (SEPSIS) tool to identify patients with high risk of sepsis in medical cases attended by emergency ambulance with an area under the receiver operating characteristic (ROC) curve of 0.86, and sensitivity 0.80 and specificity 0.78 at the recommended threshold. 24 Bayer et al. developed the PRESEP score to identify sepsis in prehospital patients with suspected sepsis with area under the ROC curve 0.93, sensitivity 0.85 and specificity 0.86. 25 Polito et al. developed the PRESS score to identify severe sepsis in physiologically abnormal prehospital patients with suspected sepsis with sensitivity 0.86 and specificity 0.47. 26

In developing our research proposal, we searched for studies evaluating the sensitivity and specificity or the effect of implementation of early warning scores for suspected sepsis in a prehospital population. 10 We identified 13 studies evaluating 20 scores. All but one were retrospective studies. The study populations varied from including all medical cases to including only those with presumed or diagnosed sepsis. Definitions of the reference standard included diagnosis (sepsis), prognosis (mortality) or health service use (admission to intensive care). Sensitivity and specificity varied substantially across the scores and studies. The most extensively studied score, qSOFA (studied in 9 of the 13 studies), had sensitivity ranging from 0.16 to 0.86 and specificity ranging from 0.16 to 0.97.

Two single-centre uncontrolled before versus after implementation studies have evaluated the potential effect of a score on practice. Polito et al. showed that implementation of the PRESS score was associated with improved sepsis recognition by prehospital personnel. 26 Borelli et al. showed that implementation of a prehospital sepsis screening tool was associated with improved clinician compliance with Surviving Sepsis Campaign 3-hour sepsis bundle recommendations. 27

Previous studies have important limitations other than the low quality identified in the NICE review:

1. Early warning scores should ideally identify patients who have the greatest potential to benefit from prioritisation and urgent treatment. Studies using mortality as the outcome or reference standard may not achieve this aim. 28 Firstly, patients whose lives are saved by urgent treatment will be categorised as reference standard negative despite having clearly benefited. Secondly, patients with severe pre-existing life-limiting conditions that make life-saving treatment futile or inappropriate will be classified as reference standard positive despite having little potential to benefit. Early warning scores developed to predict adverse outcomes such as mortality may therefore identify those with irreversible mortality while missing those with greatest potential to benefit from urgent treatment.

2. Early warning scores need to be operationalised by using a threshold for decision-making that optimises the trade-off between sensitivity and specificity in terms of the benefits, harms and costs of prioritisation. Existing studies have not explicitly examined the trade-off in these terms and the NICE review identified no relevant economic evaluations. Although the cost of applying an early warning score is small, the potential knock-on costs of overtriage are substantial.

3. Early warning scores should be evaluated in the population in whom the score will be used. REMS, MEWS and NEWS were developed and validated in acute medical inpatients with a range of medical complaints, while STSS and qSOFA were developed in inpatients with suspected sepsis. Inpatient populations, especially those identified as having suspected sepsis by hospital clinicians, are likely to have a higher prevalence of severe sepsis than prehospital populations. Using an early warning score developed for an inpatient population in the prehospital setting could lead to substantial overtriage.

Research therefore needs to use a reference standard or outcome that reflects potential to benefit from urgent treatment, examine the trade-off between sensitivity and specificity in terms of the benefit, harms and costs of using an early warning score to prioritise patients, and evaluate early warning scores in the prehospital population.

Study population

We used routine ambulance service data to identify all episodes in which Yorkshire or West Midlands Ambulance Service transported adults as an emergency to the participating hospitals over the course of 2019 and a diagnosis of sepsis could have been suspected. We excluded episodes with injury, mental health problems, cardiac arrest or direct transfer to specialist services (including maternity, cardiac or stroke services). We also excluded episodes with no vital signs recorded since vital signs are essential to calculating early warning scores.

The participating ambulance services used an item on the ePRF to record the ambulance clinician diagnostic impression from a range of standardised options. These options differed between the two ambulance services, so clinical experts in the research team grouped the diagnostic impressions into five categories:

1. sepsis

2. infection (excluding sepsis)

3. non-specific diagnostic impression in which sepsis could be suspected

4. other diagnostic impression in which sepsis would not usually be suspected

5. diagnostic impression meets exclusion criteria.

Appendix 1 shows the categorisation of diagnostic impressions in the two ambulance services. We excluded episodes with a diagnostic impression in category 5 or no diagnostic impression recorded. Categories 1–4 were used to evaluate how the accuracy of early warning scores varied according to their application on the basis of diagnostic impression. NICE guidance suggests that sepsis could be suspected in any non-specific presentation but only applying early warning scores to those with suspected sepsis or suspected infection could provide a better balance of sensitivity and specificity.

The ambulance services differed in the way that diagnostic impression was recorded, with Yorkshire Ambulance Service only allowing one diagnosis to be recorded, while West Midlands Ambulance Service allowed multiple diagnoses, with no ranking in order of likelihood or importance. We categorised the West Midlands episodes according to whichever impression was nearest the top of our ranked order. For example, a patient with diagnostic impressions of atrial fibrillation, urinary tract infection and sepsis would be categorised in group 1. The exception was category 5, where any diagnosis meeting the exclusion criteria resulted in exclusion.

This represented a fundamental difference in the way that diagnostic category would be used in principle and in practice alongside an early warning scores. In Yorkshire, patients in category 1 would be those in whom the ambulance clinician considered sepsis to be the most likely diagnosis. In the West Midlands, patients in category 1 would be those in whom sepsis was one of the main differential diagnoses.

We identified patients with multiple episodes recorded so that we could use episode or patient as the unit of analysis. We only used data from the first episodes in analyses involving patient as the unit of analysis.

Prehospital early warning scores for sepsis

We evaluated any early warning scores that could be used by prehospital professionals and calculated from routinely available prehospital data. We included dichotomous scores (i.e. rules) that simply categorise into high- and low-risk groups, but refer collectively to the index test as constituting early warning scores for simplicity.

We searched EMBASE, CINHAL, PubMed, Clinicaltrials.gov, the ISRCTN registry and Research Registry for completed and ongoing studies of early warning scores for suspected sepsis in a prehospital population using the search strategy outlined in Appendix 2.

We planned to convene a group of 10–15 experts from prehospital care, emergency medicine and critical care to review the early warning scores identified by the literature search and select those that could be used in prehospital care and calculated from routinely available prehospital data. However, the COVID-19 pandemic substantially reduced the ability of relevant clinicians to participate in this activity, so we amended the study protocol to allow clinical experts from the research team to review early warning scores for evaluation and reduce the need for the project to draw upon overstretched clinical expertise.

The clinical experts convened over two online meetings and selected early warning scores that fulfilled the following criteria: (1) the variables constituting the score are likely to be measured and recorded in a standard manner; (2) categorisation and allocation of points to form a score follows a logical and intuitive process; and (3) the process of calculating a score is simple and reproducible, with a low risk of error. They also considered whether thresholds used to categorise a continuous measure were based on accepted cut-points, such as accepted normal ranges, but did not exclude scores that failed to meet these criteria.

The clinical experts selected 21 early warning scores that could potentially be used by prehospital professionals and calculated from routine ambulance service data. They determined whether any modification to the score was required to allow it to be calculated from routine data and then specified the modification required. They also specified how the score should be calculated if any elements were missing in routine data. This usually involved assuming that any missing variables were normal or negative. Finally, they identified a decision-making threshold for each score using existing literature or their knowledge of use of the scores in practice.

Public representatives reviewed the early warning scores to determine whether their use was likely to be acceptable to the patient and the public, taking into account whether measuring or recording variables for the score could be intrusive for the patient, and whether the score raises concerns about equity, such as in relation to age, gender, ethnic group or socioeconomic status. Several scores used age as a variable, which was felt to be acceptable given the association between age and risk of adverse outcome, provided use did not result in any age discrimination. None of the scores used gender, ethnic groups or socioeconomic variables. Some concerns were raised about use of residence in a care home as a variable in the PRESS score. This was dropped from the score on the basis of these concerns and lack of ambulance service data to confirm residence.

Table 1 outlines the scores and compares their constituent variables. Appendix 3 provides details of how each score is calculated, any modifications or assumptions in calculating the score from routine data, and the threshold for decision-making. The scores used combinations of age and six physiological measurements (temperature, heart rate, respiratory rate, peripheral oxygen saturation, conscious level, and blood pressure), along with a small number of other variables. Conscious level was assessed using either the Glasgow Coma Scale (GCS) or the alert, confusion, voice, pain, unresponsive (ACVPU) scale. Blood pressure was assessed using either the systolic blood pressure or the mean arterial pressure. Modification of the UK Sepsis Trust (UKST) red-flag criteria involved removing lactate, oliguria and recent chemotherapy from the criteria because these were not routinely recorded and lactate is not routinely measured in the prehospital setting.

We planned, as part of the expert group work, to use consensus methods to create additional scores for evaluation. However, having reviewed the existing scores, the clinical experts determined that there was little potential to develop an additional score from routinely recorded data that would provide a clinically important improvement on the extensive range of existing scores.

Calculation of the early warning score from electronic patient-report form data

Age, heart rate, respiratory rate, systolic blood pressure, peripheral oxygen saturation, temperature and GCS were recorded on the ePRF. Mean arterial blood pressure was calculated using a formula in which the diastolic blood pressure is doubled and added to systolic blood pressure, and then divided by three. Conscious level and ACVPU were inferred from GCS. We used the first recorded measurement for each variable, if it was recorded multiple times. If the variable was not recorded in the first set of observations, then the first recorded measurement was used from a subsequent set of observations.

Appendix 3 outlines how other variables used in the early warning scores were calculated from the ePRF. Reference standard results were unknown at the time of early warning score calculation.

Use of the early warning score alongside diagnostic impression

The index tests for our evaluation were early warning scores and paramedic diagnostic impression, either used alone or in combination. We evaluated how early warning scores could be used alongside the prehospital diagnostic impression by applying each early warning scores in the following ways:

1. Score applied only to cases in category 1 (sepsis), with cases in categories 2–4 considered index test negative.

2. Score applied only to cases in categories 1 and 2 (sepsis or infection), with cases in categories 3 and 4 considered index test negative.

3. Score applied only to cases in categories 1–3 (sepsis, infection or non-specific), with cases in category 4 considered index test negative.

4. Score applied to all eligible cases, regardless of diagnostic category (i.e. evaluation of the early warning scores alone).

We also evaluated diagnostic category alone as an index test (with no early warning scores) using three different thresholds: (1) category 1 positive, categories 2 to 4 negative; (2) categories 1 and 2 positive, categories 3 and 4 negative; and (3) categories 1 to 3 positive, category 4 negative.

The reference standard: sepsis requiring urgent treatment

The purpose of prehospital early warning scores is to prioritise patients who have potential to benefit from urgent treatment for sepsis. We therefore defined the reference standard as being positive if the patient met the sepsis-3 definition of sepsis and received treatment for sepsis within 4 hours of initial assessment at hospital. This definition recognised that some patients with sepsis have life-limiting conditions (such as metastatic malignancy or end-stage chronic disease) that would make urgent treatment for sepsis futile or run counter their wishes. The reference standard for the primary analysis (treated sepsis) required both diagnosis and treatment. We undertook a secondary analysis using a reference standard that only required the sepsis-3 definition to be met (diagnosed sepsis). The rationale for this approach was that the primary analysis would determine the accuracy for identifying cases most likely to benefit from urgent treatment. The secondary analysis would determine whether accuracy differed when the reference standard included cases that could have benefited from early diagnosis (e.g. to allow prognostication or consideration of treatment options), even though they did not receive urgent treatment for sepsis.

Determining the reference standard involved hospital record review and expert judgement, but we needed a large sample size to detect a sufficient number of cases with a positive reference standard. We therefore used a screening process to select cases for reference standard review that were likely to meet the reference standard definition and avoid review for those unlikely to meet the definition. We used routine hospital data to select those with a primary or secondary International Classification of Disease (ICD) 10 admission code or cause of death compatible with sepsis, or an ED code for sepsis. Research nurses briefly reviewed the ED records of these cases and selected patients for expert review if they had a diagnosis of sepsis or any treatment for sepsis recorded in the ED notes or if sepsis was recorded as an admission diagnosis on the hospital discharge summary. Cases were excluded if there was no evidence of sepsis diagnosis or treatment, or if sepsis developed after hospital admission.

Two experts independently reviewed hospital records for all patients selected by the research nurses and determined whether the following criteria were met: (1) evidence of infection and life-threatening organ dysfunction (as defined by sepsis-3, the Third International Consensus Definitions for Sepsis and Septic Shock) within 4 hours of initial assessment; (2) treatment for sepsis was given within 4 hours. Evidence of infection could include microbiology reports identifying organisms, radiology reports identifying infective changes, or consistent markers of infection (elevated white cell count, C-reactive protein and temperature). Organ dysfunction, in accordance with the sepsis-3 definition, was defined as a SOFA score of 2 or more points worse than normal. The SOFA score was estimated using the ED observations chart and first blood results after admission. If arterial blood gas sampling had not been recorded, the arterial partial pressure of oxygen was estimated from the peripheral oxygen saturation. The normal SOFA score was estimated using previous hospital records and blood tests. Elements of the normal SOFA score were assumed to be zero unless there was clear evidence that they would not be normal, such as evidence of pre-existing confusion in patients with dementia or evidence of hypoxia or long-term oxygen therapy in patients with chronic lung disease. Treatment for sepsis usually involved evidence of administration of any antimicrobial that could be used to treat a sepsis-causing organism. Oral antimicrobial therapy was not considered to be treatment for sepsis unless there was a specific reason why intravenous antibiotics were not used (such as delays in gaining intravenous access). The expert reviewers were encouraged to make judgements in all cases, even when information was incomplete, since the potential bias from excluding cases with incomplete information was considered to be greater than the potential bias from misclassification. Index test results were not available to assessors during reference standard review, although assessors were aware of constituent elements of early warning scores recorded in ED notes.

If the two reviewers disagreed on the overall sepsis-3 judgement or whether treatment for sepsis was given, then a consensus decision was reached through discussion, with a third expert available to resolve any cases where consensus could not be reached. Disagreements over an element of the sepsis-3 definition (evidence of infection or change in SOFA score) were left unresolved if they did not affect the overall judgement, that is, if the reviewers agreed that the sepsis-3 definition was not met but disagreed on which criterion was not met.

Data linkage and management

Each ambulance service provided data from all eligible cases transported to one of the participating hospitals over 2019. The ambulance service created a unique study identification number for each case. Two linked databases were created: (1) containing the unique identification, time and date of call, and personal details, which was sent to NHS Digital; (2) containing the unique identification, time and date of call, and all non-personal details, which was sent to the Sheffield Clinical Trials Research Unit (CTRU).

We planned that NHS Digital would use the personal details from the ambulance data to link each case to the Emergency Care Data Set data for the related ED attendance and any related Hospital Episodes Statistics or Office for National Statistics data from subsequent hospital admissions. They would then send selected de-identified data from the Emergency Care Data Set, Hospital Episodes Statistics and Office for National Statistics data alongside the unique study identification number to the Sheffield CTRU, and would send a data set to a research nurse at each participating hospital consisting of the unique study identification number, personal data and the selected Emergency Care Data Set, Hospital Episodes Statistics and Office for National Statistics data. The research nurses would then undertake the process outlined above to determine the reference standard.

Our initial application to the NHS Digital DARS was rejected on the basis that the data flow process, involving NHS Digital sending personal data to multiple recipients (Sheffield CTRU and the four hospitals), incurred an unacceptable risk to NHS Digital. We therefore amended the data flow process so that NHS Digital would only send data to Sheffield CTRU, who would then send NHS numbers for patients requiring reference standard adjudication to the four hospitals. The regulatory authorities and NHS Digital approved the amended process.

After NHS Digital approved the application and signed a Data-Sharing Agreement with the University of Sheffield there was a prolonged delay in receiving data during which a scheduled date for data release (19 May 2022) passed without receiving any data. In September 2022, after 9 months of waiting for NHS Digital to provide data and no indication of when it would be provided, and with 4 months until the funder had stated that the project had to be completed, we still had not received linked data. At this point, after consultation with the study steering committee and public representatives, and approval from regulatory authorities, we implemented a rescue plan to allow us to collect reference standard data at one of the participating sites. Details of the timeline for seeking linked data from NHS Digital are outlined in Appendix 4.

The rescue plan involved Yorkshire Ambulance Service identifying all eligible cases transported to the Sheffield Northern General Hospital that had an NHS number recorded, excluding cases where the patient had opted out from access to their personal details for research purposes, and then adding NHS numbers to the data sent to Sheffield CTRU. The data were then sent to the Chief Investigator, who works as a clinician at the Sheffield Northern General Hospital. The Chief Investigator shared the data with the Scientific Computing team at Sheffield Teaching Hospitals NHS Trust, who searched the hospital information system for any ED attendances or hospital admissions with the same NHS number, occurring within 48 hours of ambulance transport to hospital. The Scientific Computing team informed Sheffield CTRU of any cases that could not be linked or had no coded attendance or admission within 48 hours. These were excluded from the analysis. The Scientific Computing team then provided the Chief Investigator with ED codes and admission ICD10 codes or causes of death for all cases identified as having a coded attendance or admission within 48 hours. The Chief Investigator then identified all cases with an ED code of septicaemia or neutropenic sepsis, or a primary or secondary ICD10 code for sepsis.

As outlined above, the research nurses then screened the ED records and hospital discharge summaries of cases with a code for sepsis and selected for expert review those with a diagnosis of sepsis or treatment for sepsis recorded. Four consultants in emergency medicine who work in the Northern General Hospital ED (SG, KI, SC and GF) undertook expert review, with two independently reviewing each case.

Data analysis

We originally planned to undertake analysis across both ambulance services and all four hospitals, but revised this plan to undertake a separate analysis for each hospital because: (1) inability to use NHS Digital linkage meant that we could only measure the reference standard at Sheffield Northern General Hospital; (2) the two ambulance services used diagnostic impressions in fundamentally different ways, and evaluation of early warning scores alongside diagnostic impression was considered to be crucial.

Descriptive analysis of the ambulance service data for each participating hospital used episode as the unit of analysis to compare the index tests (early warning scores and paramedic diagnostic impressions) in terms of the mean [standard deviation (SD)] number of episodes per day that would be positive (i.e. prioritised or selected for treatment). We also reported the mean total eligible attendances per day and the mean number per day that received a pre-alert from the ambulance service. We then calculated the proportion of eligible attendances that would be prioritised using each index test strategy. This indicated whether each strategy is likely to be manageable and result in meaningful prioritisation, on the basis that prioritisation requires sufficient staff resources, sepsis is one of a number of conditions that might benefit from prioritisation, and prioritisation will not be meaningful if too many attendances are prioritised. We used three categories to indicate whether each strategy was likely to be manageable, based on the opinions of clinicians in the research team:

• Green: < 5% of the mean daily attendances would be prioritised – likely to be manageable.

• Amber: 5–10% would be prioritised – uncertain.

• Red: more than 10% would be prioritised – unlikely to be manageable.

We evaluated each index test based upon using the recommended threshold (see Appendix 3) with the exception of NEWS2 and qSOFA, where we evaluated all potential thresholds. NEWS2 is routinely used in NHS ambulance services and can be implemented at any threshold with minimal additional training, while qSOFA is simple to calculate and recommended in international guidelines. 1 Implementing other scores would require additional training and possibly decision-support tools. We therefore wanted to compare these scores to all NEWS2 thresholds to ensure that superior performance did not just reflect the chosen NEWS2 threshold.

When calculating early warning scores, missing parameters were generally assumed normal (scored zero): details for each score are outlined in Appendix 3. Cases were excluded from analysis of a specific early warning scores if more than half of the variables used to calculate the score were missing.

We used data from a subset of the cohort (patients with NHS numbers who were transported to the Sheffield Northern General Hospital) to estimate the accuracy of each index test compared to the reference standard. We used the patient as the unit of analysis and only included the first eligible episode per patient. Primary analysis used treated sepsis as the reference standard and secondary analysis used diagnosed sepsis (see The reference standard: sepsis requiring urgent treatment). Kappa scores were calculated to determine the agreement between adjudicators for each element of the reference standard, the overall reference standard judgement and the judgement of whether treatment for sepsis was provided.

We constructed ROC curves to evaluate sensitivity and specificity over the range of each score. We calculated the area under the ROC curve and sensitivities and specificities at key cut-points, each with a 95% confidence interval (CI).

In our proposal, we planned to explore whether it is possible to statistically derive a clinically credible new score using multivariable logistic regression. We ultimately did not include this in our statistical analysis plan for the following reasons: (1) the large number of existing scores using the same limited number of available variables suggested that there was little potential to derive a new score with superior accuracy (other than due to overfitting); (2) the limited time available after delays incurred waiting for NHS Digital compelled us to prioritise the core analysis; (3) limitations in the population sample with reference standard adjudication (single site, only those with NHS numbers) undermined the potential applicability of a derived model to other settings.

Selection of strategies for the decision-analytic modelling

We used the results of the primary (treatment) analysis to compare the sensitivity and specificity of each strategy (combination of early warning score and diagnostic impression) to all other strategies and exclude any strategy that had inferior sensitivity and specificity to another strategy (i.e. both parameters inferior or one parameter equivalent and the other inferior). We used the point estimate to three decimal places for this assessment. Clinical experts in the group then reviewed these strategies for inclusion in the decision-analytic modelling. They excluded the strategies that were considered unmanageable due to prioritising > 10% of eligible cases in the analysis of mean daily rates of attendances prioritised. They then excluded strategies that would require additional training for ambulance and ED staff if they did not offer clearly superior accuracy to points on the NEWS2 ROC curve or an alternative rationale for use instead of NEWS2. The rationale for this decision was that NEWS2 is widely used by NHS ambulance services and hospitals, so any alternative strategy requiring additional training would need to offer clearly superior accuracy. We included the accuracy of pre-alert practice in 2019 alongside diagnostic impression as a strategy in this assessment in case it could represent a worthwhile comparator strategy in the modelling.

Sample size

We based our planned sample size estimate on data from Smyth et al. reporting 3.7% prevalence of high-risk sepsis in adults transported to hospital with a non-specific emergency presentation,24 and data from reviews of sepsis by the National Confidential Enquiry into Patient Outcome and Death (NCEPOD) and the Intensive Care National Audit and Research Centre (ICNARC). 40,41 NCEPOD identified 3363 adults (≥ 16 years) seen by the Critical Care Outreach Team or admitted directly to critical care with a diagnosis of sepsis across 305 hospitals over a 2-week period. ICNARC reported 22,081 admissions to critical care with septic shock across 205 hospitals over 2012. These studies suggest that the key determinant of sample size is the number of reference standard positive cases (to estimate sensitivity with acceptable precision) and an incidence of around two reference standard positive cases per week at a typical hospital.

Our plan to collect data across four hospitals over 1 year was based on attempting to ensure generalisability across multiple sites and avoid seasonality. Data from Smyth suggested around 23,000 eligible episodes over 1 year at each participating hospital, including around 847 with the reference standard, providing a total sample size of 92,000 across four hospitals with 3388 reference standard positive cases. This would result in an overpowered study with unnecessary and excessive work delivering reference standard validation. We therefore planned to randomly sample cases at each site to achieve a sufficient number of reference standard positive cases.

We estimated that around 2000 episodes would be selected for research nurse review (500 at each site), with around 1000 for expert hospital record review, to identify 200 reference standard positive episodes. This sample size would allow us to estimate the sensitivity of an early warning score with a standard error of 2.1% assuming sensitivity of 90%, and the area under the ROC curve with a standard error of 2% assuming an area under the ROC curve of at least 0.75. 42 The sample size also satisfies the recommendation by Collins et al. that external validation studies be based on a minimum of 100–200 events. 43

Following the failure of NHS Digital to provide linked data and the activation of our rescue plan, we attempted to pragmatically deliver an appropriate sample size within the time and resources available. We decided that random sampling of cases would not be practical and instead used consecutive sampling of cases in time order. In the event we were able to review all available cases across 2019 at the Sheffield site and deliver a sample of reference standard positive cases that exceeded our planned sample size.

Ethical approvals

The study gained initial approval from the London – Stanmore Research Ethics Committee (REC) (reference number 19/LO/1443) on 23 September 2019 and the Health Research Authority (HRA) on 30 January 2020. Due to the use of personal data for record linkage Confidentiality Advisory Group (CAG) approval was sought and granted on 13 January 2020. Over the course of the study, three substantial amendments were submitted to the REC, HRA and CAG. Dates of approval are provided in Table 2 and the resulting changes are outlined in the text below.

Overview of the ambulance service data

There were 178,333 ambulance episodes provided from 2019: 27,564 from West Midlands Ambulance Service and 150,769 from Yorkshire Ambulance Service. The Yorkshire Ambulance Service data set (Doncaster, Sheffield and Rotherham sites) was missing a period of 9 days in February. The West Midlands Ambulance Service data set (Coventry site) was missing data for the second half of each of the last 6 months of the year.

We excluded 83,311 episodes that met study exclusion criteria, had no unique patient identifier or had no data to calculate early warning scores. The analysis set included 95,022 episodes from 71,204 unique patients across the four sites. Table 3 compares the characteristics of the 71,204 first episodes to those of the 23,818 repeat episodes. The repeat episodes involved patients who tended to be older (median age 75 vs. 66 years) and had slightly more abnormal physiological measurements. Diagnostic impressions of sepsis (4.4% vs. 3.8%), infection (9.7% vs. 8.8%) and non-specific presentations (41.8% vs. 37.1%) were slightly more common in the repeat attendances.

Table 4 shows the characteristics of the 71,204 patients by hospital site. Patients at the larger hospitals (Sheffield and Coventry) tended to be older and more ethnically diverse. Patients transported to Coventry by West Midlands Ambulance Service were more likely to have a diagnostic impression of sepsis (6.5% vs. 2.5–3.6%), infection (21.4% vs. 3.7–5.5%) or non-specific presentations (40.8% vs. 33.9–38.3%), reflecting the potential for paramedics in West Midlands Ambulance Service to record multiple diagnostic impressions.

Daily rates of prioritised attendances

Tables 5–8 show, for each hospital, the mean (SD) number of attendances per day that would be prioritised according to each strategy using the threshold in the second column and when applied to the paramedic diagnostic impression(s) in the remaining columns. The first row shows the number of attendances prioritised using paramedic diagnostic impression alone. The second row shows the number of attendances that were recorded as being pre-alerted to the ED in 2019.

The colours in Tables 5–8 indicate the proportion of eligible attendances that would be prioritised and thus whether the number of prioritised cases would be manageable. Green indicates that < 5% of the mean daily attendances would be prioritised (likely to be manageable), amber indicates that 5–10% would be prioritised (uncertain) and red indicates that more than 10% would be prioritised (unlikely to be manageable).

The mean (SD) number of daily attendances meeting the study inclusion criteria were 93.5 (14.7) at Sheffield Northern General Hospital, 59.5 (10.8) at Doncaster Royal Infirmary, 51.3 (8.9) at Rotherham General Hospital, and 74 (11) at University Hospitals Coventry and Warwickshire. The proportion of attendances that the ambulance service pre-alerted to the hospital varied from 5.1% (4.8/93.5) at Sheffield Northern General Hospital to 10.3% (7.6/74) at University Hospitals Coventry and Warwickshire, with around 1 in 5 of the pre-alerted attendances at the Yorkshire hospitals having a diagnostic impression of sepsis, compared to 1 in 3 in the West Midlands.

The proportions of cases in categories 1 and 2 were higher at University Hospitals Coventry and Warwickshire because paramedics at West Midlands Ambulance Service can select multiple diagnostic impressions whereas paramedics at Yorkshire Ambulance Service can select only one. As a consequence, the index tests would generally result in more attendances being prioritised at University Hospitals Coventry and Warwickshire than at the Yorkshire hospitals. The combination of an early warning score with a diagnostic impression of sepsis in the West Midlands would result in a similar proportion of attendances being prioritised as the combination of an early warning score with a diagnostic impression of sepsis or infection in Yorkshire.

Most early warning scores operating at their pre-specified threshold would prioritise a substantial proportion of attendances if they were applied to attendances other than just those with a diagnostic impression of sepsis or infection. The exceptions are qSOFA (threshold > 1), the SEPSIS score, the CIS (threshold > 4), the Paramedic Initiated Treatment of Sepsis Targeting Out-of-hospital Patients clinical trial (PITSTOP) rule, the PRESS score and the sepsis alert criteria. It should be noted that the PITSTOP rule and sepsis alert criteria, along with the Prehospital Sepsis Assessment Tool (PreSAT), Robson and Suffoletto criteria, all specify paramedic suspicion of sepsis or infection as a qualifying criterion, so should only be applied to diagnostic categories 1 and 2.

In summary, to avoid prioritising a substantial proportion of cases, most early warning scores would need to be applied only to attendances with a primary diagnostic impression of sepsis or infection, or with sepsis as one of a number of diagnostic impressions, depending upon how the ambulance service records diagnostic impression.

Data linkage to Sheffield Teaching Hospitals National Health Service Trust

National Health Service numbers were available to link ambulance service to hospital data for 14,050/24,955 (56.3%) patients transported to Sheffield Teaching Hospitals. Table 9 compares the characteristics of the patients with NHS numbers available to those without. The patients with NHS numbers were markedly older (median age 71 vs. 55 years). They were more likely to be female (54.7% vs. 53.0%) and white ethnicity (95.7% vs. 91.8%), although these characteristics may reflect their older age. They were more likely to have a diagnostic impression of sepsis (2.9% vs. 2.0%), infection (6.5% vs. 4.3%) or non-specific presentation (35.3% vs. 32.0%). The difference in median age between the linked and unlinked populations was greatest for those in the ‘other’ diagnostic impression category and was not seen in those with a diagnostic impression of sepsis.

The Scientific Computing department of Sheffield Teaching Hospitals NHS Trust identified a hospital admission or ED attendance within 48 hours for 12,870 of the 14,050 first episodes with an NHS number transported to the Northern General Hospital (91.6%). The remaining episodes comprised 431 (3.1%) with no data returned for the NHS number and 749 (5.3%) with no admission or attendance within 48 hours. Some 8260 (64.1%) had a hospital admission, of whom 275 (3.3%) had no primary diagnosis code and 347 (4.2%) had no secondary code. Some 10,964 (85.2%) had an ED attendance, of whom 14 (0.1%) had no ED code.

Overall, 684 episodes had a primary or secondary admission diagnosis of sepsis or an ED diagnosis of septicaemia or neutropenic sepsis, and were screened by the research nurses. Of these, 594 had a primary or secondary admission diagnosis of sepsis (315 primary, 322 secondary, 43 both) and 160 had an ED diagnosis of septicaemia or neutropenic sepsis (70 also has an admission primary or secondary diagnosis of sepsis, 90 did not).

The research nurses referred 655/684 (95.8%) for expert review. The experts judged that 368/655 (56.2%) episodes met the sepsis-3 definition and 348/368 (94.6%) of these received treatment for sepsis. Therefore, out of the 12,870 episodes that were linked with hospital attendance or admission, 684 (5.3%) had a diagnostic code for sepsis, 655 (5.1%) had evidence of sepsis diagnosis or treatment in their ED or admission records, 368 (2.9%) met the secondary (diagnosis) reference standard and 348 (2.7%) met the primary (treatment) reference standard. Figure 1 summarises the flow of cases to create the diagnostic accuracy cohort.

FIGURE 1.

Participant flow to create the Sheffield diagnostic accuracy cohort. NGH, Northern General Hospital.

Adjudication of the reference standard

Table 10 shows the agreement between the two doctors adjudicating the reference standard. Agreement was very good on the two key judgements of whether the case met the sepsis-3 criteria and whether treatment for sepsis was given. Agreement was not as good on whether there was evidence of infection, but the cases in which there was disagreement tended to be those in which the doctors agreed that the SOFA score criterion was not met, so they did not affect overall judgement on the sepsis-3 definition.

There was radiological evidence of infection in 175/348 (50.1%) cases with the primary reference standard, microbiological evidence of infection 171 (49.0%) and other evidence of infection in 328 (94.0%). The site of suspected infection was the chest in 155 (44.4%) urine in 78 (22.3%), biliary in 43 (12.3%), abdominal in 16 (4.6%), skin in 25 (7.2%), other in 6 (1.7%) and unknown in 26 (7.4%). They had a mean clinical frailty score of 5.6 (median 6.0, range 2.0–9.0; 3 missing) and a mean SOFA score of 3.9 (median 3.0, range 2.0 to 14.0; 13 missing). They included 138 (39.5%) who had a Do Not Attempt Cardiopulmonary Resuscitation decision recorded, of whom 102 (29.2%) were pre-existing and 36 (10.3%) were made on attendance. Some 28 (8.0%) were admitted to critical care and 261 (74.8%) survived to hospital discharge or 30 days after attendance, whichever was sooner.

Accuracy for the primary (treatment) reference standard

Table 11 shows the area under the ROC curve for each early warning scores for the primary reference standard, when used alongside diagnostic impression. The first column indicates the early warning score and the remaining columns report the area under the ROC curve when the score is only applied to patients with a diagnostic impression of sepsis, only to patients with a diagnostic impression of sepsis or infection, only to patients with a diagnostic impression of sepsis, infection or a non-specific diagnosis, or when applied to all patients.

The area under the ROC curve for diagnostic impression alone was 0.822 (95% CI 0.799 to 0.845). The area under the ROC curve decreases as each early warning score is used more selectively on the basis of paramedic diagnostic impression. This could be interpreted as suggesting that early warning scores should not be used selectively. However, Figure 2 shows the ROC curves for NEWS2 alongside the categorisations of diagnostic impression. The area under the ROC curve is clearly smaller when NEWS2 is used selectively but the low prevalence of sepsis in the cohort and need to achieve high specificity means that the optimal threshold on the ROC curve is likely to lie where specificity exceeds 0.95. At this point, the ROC curves are very close together and the more selective strategies operating at lower NEWS2 thresholds offer slightly better accuracy.

FIGURE 2.

ROC curves for NEWS2 used alongside paramedic diagnostic impression.

Table 11 also shows that the area under the ROC curve for NEWS2 is generally equivalent to or greater than the other early warning scores. The exceptions are the SEPSIS score when applied to those with non-specific presentations (including sepsis and infection) or all presentations. However, the SEPSIS score does not provide better accuracy than NEWS2 at thresholds providing a positive predictive value exceeding 0.2.

Figures 3–6 show the ROC curves for each score, or a plot of sensitivity and 1−specificity for dichotomised scores, when applied only to attendances with a diagnostic impression of sepsis (see Figure 3), only to attendances with a diagnostic impression of sepsis or infection (Figure 4), only to attendances with a diagnostic impression of sepsis, infection or a non-specific impression (Figure 5), and applied to all attendances (Figure 6). Each figure has two plots to avoid presenting too many overlapping curves on the same figure. The curves in Figures 3–5 have a straight line from the point on the curve at which the sensitivity of the combination of the score and paramedic diagnostic impression reaches the sensitivity of the paramedic diagnostic impression alone. This is because these strategies all involve applying the scores selectively on the basis of paramedic diagnostic impression. The sensitivity of the combination of the score and diagnostic impression cannot therefore exceed the sensitivity of the diagnostic impression alone.

FIGURE 3.

ROC curves for early warning scores applied to diagnostic impression of sepsis.

FIGURE 4.

ROC curves for early warning scores applied to diagnostic impression of sepsis or infection.

FIGURE 5.

ROC curves for early warning scores applied to diagnostic impression of sepsis, infection or non-specific presentation.

FIGURE 6.

ROC curves for early warning scores applied to all diagnostic impressions.

The figures show that NEWS2 has superior or similar accuracy to the other scores, with the possible exception of the SEPSIS score when applied to non-specific presentations or all cases at a threshold that gives sensitivity and specificity both around 0.8–0.9.

Table 12 shows the accuracy of the categorised paramedic diagnostic impression for the primary reference standard. Paramedic diagnostic impression of sepsis identified around 33% of treated sepsis cases, diagnostic impression of infection (including sepsis) identified around 57%, non-specific presentation (including sepsis and infection) identified around 90%, leaving around 10% with diagnostic impressions that would not suggest sepsis. Along with the limited positive predictive value (28.5% of those with a paramedic diagnostic impression of sepsis had the primary reference standard), this illustrates the diagnostic challenge of identifying sepsis in the prehospital setting.

Tables 13–16 show the diagnostic accuracy parameters for each threshold of the NEWS2 score when applied only to attendances with a diagnostic impression of sepsis (see Table 13), only to attendances with a diagnostic impression of sepsis or infection (see Table 14), only to attendances with a diagnostic impression of sepsis, infection or a non-specific impression (see Table 15), and applied to all attendances (see Table 16). The low prevalence of the reference standard (347/12,859, 2.7%) means that the PPV is relatively low even when specificity is high, and the NPV is relatively high even when sensitivity is low. Tables 17–20 report the same parameters for all thresholds of qSOFA and the pre-specified thresholds of the other early warning scores. The first row, marked ‘pre-alert’, shows the sensitivity and specificity of pre-alert practice in 2019, as recorded on the PRF.

The low prevalence of the primary reference standard (2.7%) means that high specificity is required to achieve acceptable positive predictive value. The judgements made in the previous section around the feasibility of strategies that prioritised a large number of presentations, and the opinion of clinical experts in the research team, suggest that strategies with a positive predictive value below 0.2 are unlikely to be considered feasible (i.e. strategies resulting in fewer than 1 in 5 prioritised presentations actually having sepsis requiring urgent treatment). We would obviously like to prioritise as many cases of sepsis as possible (maximise sensitivity). The results are probably best interpreted by comparing the trade-off between sensitivity and positive predictive value, rather than sensitivity and specificity, because positive predictive value is more meaningful to clinicians and patients, and high specificity could be misinterpreted as being acceptable even though it is associated with poor positive predictive value.

Tables 13–16 show how sensitivity decreases and positive predictive value increases as the NEWS2 thresholds increases, and how selective application of NEWS2 on the basis of paramedic diagnostic impression improves positive predictive value at the expense of sensitivity. Determination of an optimal threshold depends upon a number of factors that may vary over time and between settings, but the results suggest that the best sensitivity that can be achieved with positive predictive value exceeding 0.2 is the sensitivity of 0.522 achieved by using NEWS2 with a threshold of > 4 in presentations with a diagnostic impression of sepsis or infection. The Academy of Medical Royal Colleges clinical decision support framework recommends that this threshold is used to identify presentations needing treatment within 3 hours of arrival. A threshold of > 6 is recommended to identify presentations needing treatment within 1 hour of arrival, which our data suggest has sensitivity of 0.447 and positive predictive value of 0.274.

Tables 17–20 show substantial variation in sensitivity and positive predictive value between the early warning scores at their recommended thresholds. However, none of the early warning scores appear to be clearly more accurate than NEWS2 if an appropriate threshold for NEWS2 is used. The widespread use of NEWS2 in the NHS suggests that this should be the default option unless an alternative score offers superior accuracy. One possible exception to this judgement is the use of qSOFA at a threshold > 1 in presentations with an impression of sepsis or infection. The sepsis-3 consensus group recommended using qSOFA > 1 in presentations with evidence of infection to allow early identification of possible sepsis and could offer an internationally recognised alternative to NEWS2. Our results suggest that this strategy would have sensitivity of 0.305 and positive predictive value of 0.356, which is similar to the sensitivity (0.314) and positive predictive value (0.333) of using NEWS2 > 8 in presentations with an impression of sepsis or infection. These tables also show that the sensitivity and positive predictive value of pre-alert practice in 2019, as recorded on the PRF, was relatively poor, with sensitivity and positive predictive value ranging from 0.129 (95% CI 0.098 to 0.169) and 0.29 (95% CI 0.225 to 0.366) when applied only to those with an impression of sepsis, to 0.23 (95% CI 0.189 to 0.277) and 0.131 (95% CI 0.106 to 0.16) when applied to all eligible cases. It was not mandatory to record pre-alerts in Yorkshire Ambulance Service in 2019, so these results may be affected by under-reporting.

Accuracy for the secondary (diagnostic) reference standard

We repeated the accuracy analysis using the secondary (diagnostic) reference standard instead of the primary (treatment) reference standard, but there was no meaningful difference in the results. This is unsurprising, given that there were only 20 additional reference standard positive cases in the secondary analysis. The results are available in Report Supplementary Material 1.

Selection of strategies for the decision-analytic modelling

None of the alternative strategies were judged to offer clearly superior accuracy to NEWS2, so only NEWS2 strategies were included in the modelling. We also limited the NEWS2 strategies to those applied alongside a diagnostic impression of sepsis or a diagnostic impression of infection or sepsis. Although NEWS2 could prioritise a manageable number of cases when applied to non-specific presentations or all cases if a high threshold were used, we noted that equivalent or better accuracy could be achieved at a lower threshold when applied to patients with a diagnostic impression of infection or sepsis. The clinical experts decided to include one additional strategy in the modelling – a strategy of using a qSOFA score > 1 in patients with a diagnostic impression of infection or sepsis. This strategy is recommended in the sepsis-3 guidance and represents a relatively simple strategy with equivalent accuracy to a point on the NEWS2 ROC curve.

Population

Members of the study team recently undertook a retrospective single-centre descriptive study based in Sheffield Northern General Hospital on the characteristics and outcomes of suspected sepsis patients in the ED. 2 The median age of the subset of patients with diagnosed sepsis meeting the sepsis-3 definition was 77 years [interquartile range (IQR) 65–85] and 39.8% (100/251) of the population were female. These figures are used in the model with the median assumed equal to the mean. The sepsis prevalence used in the model is 2.70% based on data in the retrospective cohort study presented in Chapter 2.

Two ambulance populations are modelled to represent large and small service sizes, with the difference being the number of daily ambulances arrivals (excluding children, trauma, maternity and mental health). Data on the average number of daily ambulance arrivals for a small hospital were taken from Rotherham General Hospital data (average of 51.3) and from Sheffield Teaching Hospital data for a large hospital (average of 93.5), both part of the retrospective cohort study.

Strategies implemented – early warning score sensitivity and specificity

Based on analyses of sensitivity and specificity in conjunction with clinical advice, 23 strategies were compared with no prioritisation (see Selection of strategies for the decision-analytic modelling) These were 10 strategies for people classified as category 1 (sepsis) and 13 strategies were analysed for patients classified as either category 1 or category 2 (sepsis or infection). These strategies are shown in Table 21. All early warning scores were compared to a strategy of no prioritisation of patients. Due to insufficient data, no formal comparisons of early warning scores were performed; decision-makers would need to assess this individually based on the total number of pre-alerts per day, and assessing the likelihood that there would be reductions in mortality, morbidity and LoS generated from using each early warning score.

Sensitivity and specificity data for each combination of early warning score and the diagnostic impression category to which it is applied were sourced from the retrospective cohort study. As NEWS2 was recently endorsed by NHS England for use in ambulance settings49,50 and the Academy of Medical Royal Colleges has recently produced guidance for prioritisation of sepsis based on varying thresholds of NEWS2,51 scenarios were included which evaluate the NEWS2 score across its varying thresholds when applied to patients in category 1 and in categories 1 or 2. As discussed in Methods, data on the reference standard needed to estimate the sensitivity and specificity could only be measured with data from Sheffield Northern General Hospital rather than the four ambulance services as initially planned. The specificity and sensitivity of each strategy (combination of early warning score and diagnostic impression to which it is applied) are shown in Accuracy for the primary (treatment) reference standard.

Baseline mortality risk due to sepsis

In order to estimate the benefit of prehospital treatment or prehospital early warning score leading to immediate treatment, an estimate of baseline sepsis mortality rate was required.

Many studies found in Search 1 used the control arms of studies or country-specific cohort data for baseline mortality; however. the majority of these were based in North America or Europe (France, Spain, Italy) and therefore not necessarily generalisable to our research. Nine studies used mortality data from the PROWESS trial, a worldwide randomised controlled trial (RCT) of Drotrecogin alfa (activated) for severe sepsis. A UK health technology assessment conducted by Westwood et al. 52 used a range of probabilities for baseline mortality taken from previous literature (Christ-Crain et al.;53,54 Bouadma et al. ;55 Qu et al. 56 Roh et al. 57). The probability of mortality for adults ranged from 6.2% to 7.2% in the ED and from 16.9% to 18.2% for adults in the ICU. Mouncey et al. 46 reported a 24% 28-day mortality and 29% 90-day mortality rate for the usual-care arm of their UK trial.

Stevenson et al. 58 used a baseline 30-day mortality rate of 13% and a hospital mortality rate of 21%, both taken from a previous health technology assessment report59 reporting on data from four NHS hospitals in the North-West of England. The 29% 90-day mortality rate reported by Mouncey et al. 46 was then used in sensitivity analyses. Both Soares et al. 60 and Green et al. 61,62 reported a baseline mortality rate relevant to the UK. They used figures from the ICNARC case-mix programme which identified admissions of severe sepsis in England, Wales and Northern Ireland. Soares et al. 60 used the time period of 2007–9, which was the most recent available at the time of their study. Overall hospital mortality was 40.6% (95% CI 40.0% to 41.2%) and baseline mortality for critical care units was 29.1% (95% CI 28.6% to 29.7%). Green et al. 18,19 did not report the time period used but reported a 28-day mortality rate of 41.5% (95% CI 40.8% to 42.3%) for patients with severe sepsis and 46.2% (95% CI 45.3% to 47.1%) for patients with severe sepsis and multiple organ dysfunctions.

Members of the team were aware from personal communication with Dr Lisa Sabir (Academic Clinical Fellow in Emergency Medicine) of a single-centre retrospective study of adult patients with suspected sepsis in ED. Other studies usually have a selected cohort from which sepsis mortality is estimated due to the primary aim being to study a treatment or diagnostic test. Dr Sabir’s study was specifically designed to describe the characteristics and outcomes of a randomly selected cohort of patients attending an ED with suspected sepsis and diagnosed with sepsis according to the sepsis-3 definition. 1 Due to this, this study was deemed most relevant to our current study. Sabir et al. (personal communication) communicated that there were 50 deaths within 30 days from 192 patients where data were fully known in the sepsis-3-defined cohort, resulting in a 30-day mortality rate of 26%. As this study was deemed the most relevant population this was chosen to be used in our model.

Health-related quality of life

Health-related quality of life after sepsis was required to estimate the QALY gains associated with reductions in mortality due to prehospital sepsis alerts/antibiotics. Only two of the identified economic evaluations in Search 1 provided utility values that were sepsis-specific, published as full papers and relevant to the UK. Therefore, the utility values provided in Stevenson et al. 58 (taken from Cuthbertson et al. 63) and Mouncey et al. ,46 as part of the PROmisE trial, were considered the most applicable to be used. The utility values for sepsis survivors used in the model are shown below:

• Initial 90 days: average of intervention (0.609, SD 0.319) and comparator group (0.603, SD 0.312) in PROmisE trial at 90 days = 0.611.

• 91 days to 1 year: average of intervention (0.620, SD 0.307) and comparator group (0.653, 0.323 SD) in PROmisE trial at 1 year = 0.637.

• After 1 year: 0.676 calculated from the value used between 91 days and 1 year (0.636) increased by a multiplier of 1.0625, which was calculated from dividing the value at 5 years post sepsis and 3 years post sepsis reported in Cuthbertson et al. (0.68/0.64). Utilities are switched to age–sex-matched general population utility values22 where these were below 0.676.

The quality-adjusted life-year gains associated with preventing 30-day mortality

Following the methods used in Stevenson et al.,58 it is assumed in the model that each 30-day mortality prevented due to correct prioritisation of sepsis patients is associated with a QALY gain of 5.94. This figure is based on the following:

1. The estimated number of discounted life-years for a typical patient. The remaining life expectancy of each patient with an assumed age of 77 years and 39.8% (female)/60.2% (male) gender split2 was calculated using National life tables for England and Wales (2018–20). 64 The model assumes that life expectancy following sepsis for those immediately treated is equal to that of the general population.

2. Quality of life post sepsis. The utility values used are those described in Health-related quality of life.

Costs associated with false positives

The only costs included in the model are in relation to patients being incorrectly prioritised (false positives). These include unnecessary antibiotic treatment and a penalty cost for taking up a resuscitation bay as opposed to a general ED bay.

Following advice from clinical experts, it was advised that Tazocin (piperacillin/tazobactam) was the most likely antibiotic to be used in a prehospital setting for sepsis and is also used in the ED in Sheffield Northern General Hospital; therefore, it is able to represent either prehospital treatment or immediate treatment on arrival. The cost of Tazocin 4 g/0.5 g powder solution for infusion vials (Pfizer Ltd) was sourced from the British National Formulary65 with a cost of £15.17 applied to all false-positive patients to represent the first dose of antibiotics.

Patients incorrectly prioritised will take up resources in ED as they are likely to be immediately sent to a resuscitation bay rather than a general ED bay. As NHS Reference Costs do not provide estimates of the costs of resuscitation and general ED bays, the costs per day of an ICU (critical care) bed and general ward bed were used to represent resuscitation bay and general ED bay, respectively, and assumed to reflect the differences in staffing requirements associated with each. After discussions with clinical experts, it was assumed that the penalty cost of incorrectly prioritising patients would equal the difference between 4 hours in an ICU bed and 4 hours in a general ward bed. The main limitation of this assumption is that it assumes there is capacity to increase resuscitation bays and the associated staffing to accommodate the prioritised patients. Sources and costs used to reflect the penalty cost are shown in Table 22. No further analyses were undertaken to evaluate the potential harm to non-prioritised patients who have care delayed due to prioritised patients as it is assumed that this would be incorporated within the additional costs associated with the additional resuscitation bed cost.

Effectiveness of interventions (prehospital sepsis alerts or prehospital antibiotics)

Seven studies found in Search 2 assessed the impact of a prehospital alert/documentation by emergency medical staff (EMS) in the ambulance to warn arriving hospitals of a potential sepsis case (Table 23). Four of these studies were based in the USA (Borrelli et al. ;27 Hunter et al. ;69 Mixon et al. ;70 Guerra et al. 37), while the remaining three were based in Europe; UK (McClelland and Jones71), the Netherlands (Alam et al. 72) and Germany (Floer et al. 73). None of these studies were RCTs and all used retrospective data with conflicting results for the interventions in terms of impact on LoS and mortality. Two out of the seven studies found a significant difference in mortality (McClelland and Jones;71 Guerra et al. 37). Three studies assessed the impact of prehospital alerts on hospital LoS and only one of these (Borrelli et al. 27) found a significant difference in LoS with the intervention group (sepsis screening tool and prehospital alert) having a significantly lower median LoS of 5 days (IQR 3–6) versus 8 days (IQR 5–12), p = 0.01. McClelland and Jones was the only UK study looking at prehospital sepsis alerts and found a significant impact on 3-month mortality rates; however, this was a small pilot study with only 49 patients included in total. 71 As these studies were retrospective they could be subject to confounding bias, as patients’ characteristics such as age or SOFA score may have differed between the groups, as in Mixon et al. 69 and Hunter et al. 70 These studies did, however, show that pre-alerts have significant improvement in the time to antibiotics, reducing time to antibiotics to below 1 hour. However, it is unknown from these current studies if this has an impact on mortality and morbidity.

Five studies in Search 2 assessed the impact of prehospital antibiotic treatment on sepsis outcomes. This also showed conflicting evidence on mortality and LoS, with two out of the five studies showing a significant difference in mortality (Jouffroy et al.;74 Chamberlain et al. 75). These same studies were the only ones to also show a significant difference in ICU LoS. The PHANTASi trial is the only large-scale multicentre RCT to look at the impact of prehospital antibiotic treatment. 33 However, the results of this study did not show a significant impact on either mortality or LoS. Only one of the five studies was based in the UK (Jones et al. 77). This RCT feasibility study (PhRASe study) was designed to assess the viability of paramedics recognising and screening patients for severe sepsis and collecting blood cultures and administering intravenous antibiotics. The results showed no significant difference in mortality and ICU admission between the control and intervention groups. However, this study was not powered to assess significant differences in these outcomes.

Due to the large variability in results found, there was deemed to be no single estimate that could be chosen to accurately represent the effectiveness of either prehospital sepsis alerts or prehospital antibiotics on mortality or LoS outcomes. The gains associated with earlier treatment (if any) therefore cannot be currently quantified. In order to provide information for decision-makers, threshold analyses were performed to provide, in isolation, the reduction in: mortality; general ward LoS; ICU LoS; and the gains in QALYs, in isolation, that would be required for a strategy to be cost-effective at a standard NICE cost per QALY threshold of £20,000 compared with no prioritisation. 47

The number of prehospital alerts by each strategy

The operational consequences of each strategy need to be considered. Figure 10 provides the estimated total number of alerts for each strategy for a large hospital which is assumed to have, on average, 93.5 patients arriving by ambulance each day and also the number of times of prioritisation when the patient had sepsis. Figure 11 provides the estimated total number of alerts for each strategy for a small hospital which is assumed to have, on average, 51.3 patients arriving by ambulance each day and also the number of times of prioritisation when the patient had sepsis. Table 24 shows the number prioritised, number with sepsis correctly prioritised and number with sepsis not prioritised at a large and a small hospital. It is seen in both Figures 10 and 11 that the ratio of correctly prioritised patients to the total number of prioritised patients varies widely by strategy.

FIGURE 10.

The number of cases prioritised each day in a large hospital using each strategy.

FIGURE 11.

The number of cases prioritised each day in a small hospital using each strategy.

Managers and clinicians can use Figures 10 and 11, and Table 24, to predict the consequences of using each strategy at their hospital. They can thus predict the consequences of implementing national recommendations or guidelines. For example, the Academy of Medical Royal Colleges clinical decision support framework recommends using NEWS2 > 6 to identify presentations with evidence of infection for urgent treatment within 1 hour. Table 24 suggests that this would result in 4.11 cases being prioritised per day at a large hospital, including 1.13 with sepsis, while 1.40 people with sepsis were not prioritised. At a small hospital 2.26 cases would be prioritised per day, including 0.62 with sepsis, while 0.77 people with sepsis were not prioritised.

Threshold analysis based on length of stay

Figure 12 shows the average reduction in LoS for patients correctly prioritised that would be required for each strategy to be estimated to be cost-effective assuming a willingness to pay of £20,000 per QALY compared to no prioritisation. These are provided in isolation for general ward stay and ICU stay. These values would be lower if there was a reduction in both general ward stay and ICU stay.

FIGURE 12.

The threshold levels for reduction in general ward LoS and reduction in ICU LoS at which each strategy becomes cost-effective assuming a willingness to pay £20,000 per QALY gained compared with no prioritisation.

Threshold analysis based on relative risk of mortality

Figure 13 provides the relative risk of mortality for patients with sepsis who have been correctly prioritised at which each strategy would be cost-effective assuming a willingness to pay of £20,000 per QALY compared to no prioritisation. The y-axis starts at 0.975, indicating that only minor gains in mortality are required in order for even the most sensitive strategies to be cost-effective.

FIGURE 13.

The threshold levels for relative risk of mortality at which each strategy becomes cost-effective assuming a willingness to pay £20,000 per QALY gained compared with no prioritisation.

Threshold based on QALYs gained

Figure 14 provides the total number of QALYs gained per patient with sepsis correctly prioritised at which each strategy is estimated to become cost-effective assuming a willingness to pay of £20,000 per QALY compared to no prioritisation. It is seen that these numbers are very low, with all strategies having a value below 0.0006.

FIGURE 14.

The threshold levels for QALYs gained per patient at which each strategy becomes cost-effective assuming a willingness to pay £20,000 per QALY gained compared with no prioritisation.

The threshold analyses only show the threshold of effectiveness required for each strategy to be cost-effective compared to no prioritisation. We used this approach because none of the alternative strategies represent routine practice. The strategies can be compared to each other by examining the difference in the threshold level between the strategies. Given that the threshold relative risk reduction for mortality required for even the most sensitive strategy to be cost-effective was very small, we can conclude that more sensitive (effective) strategies are likely to be cost-effective compared to less sensitive strategies, if we believe that prioritisation results in a meaningful reduction in mortality. It is also worth noting that we excluded the most sensitive strategies from the decision-analytic modelling on the basis that they would prioritise an unmanageable number of cases. These strategies would be cost-effective if prioritisation reduced mortality and hospitals could release resources to increase capacity for prioritisation.

Incremental analyses comparing all strategies while varying the benefit associated with prioritisation

Figure 15 provides the iNMB for all strategies assuming a willingness to pay of £20,000 per QALY. At all relative risks, there are multiple strategies with similar iNMB values.

FIGURE 15.

The iNMB compared with no prioritisation assuming a willingness to pay £20,000 per QALY gained at different benefits associated with prioritisation.

The strategy with the greatest iNMB associated with prioritisation between ranges of relative risks of mortality is shown in Table 25 assuming a willingness to pay of £20,000 per QALY gained. The strategy with the highest iNMB changes multiple times between a range of 0.900 and 1.000. The results suggest that strategies prioritising patients at relatively low thresholds of NEWS2 (> 2 or > 3) are the most cost-effective strategies when the relative risk of mortality is 0.9–0.946. These strategies would prioritise large numbers of patients that would potentially exceed the capacity of the ED.

Incremental analyses comparing all strategies whilst varying the benefit associated with prioritisation having increased the costs of false positives

Figure 16 provides the iNMB, assuming a willingness to pay of £20,000 per QALY, for all strategies compared with no prioritisation assuming that the costs of false positives was increased from £124 to £500. The iNMBs for all strategies were noticeably reduced, with many strategies having a negative iNMB even at a relative risk of 0.980.

FIGURE 16.

The iNMB compared with no prioritisation assuming a willingness to pay £20,000 per QALY gained at different benefits associated with prioritisation.

The strategy with the greatest iNMB associated with prioritisation between ranges of relative risks of mortality is shown in Table 26 assuming a willingness to pay of £20,000 per QALY gained. The strategy with the highest iNMB changes multiple times between a range of 0.900 and 1.000. This suggests that the findings are sensitive to our estimate of the cost of false-positive prioritisation. Strategies using relatively low thresholds of NEWS2 (> 2 or > 3) are not the most cost-effective at relative risk estimates from 0.900 to 0.946 if the cost of false positives is markedly higher than our baseline estimate.