Sepsis: the LightCycler SeptiFast Test MGRADE®, SepsiTest and IRIDICA BAC BSI assay for rapidly identifying bloodstream bacteria and fungi a systematic review and economic evaluation

Electronic databases

Studies were identified by searching the following electronic databases and research registers from January 2006 to May 2015:

• MEDLINE In-Process & Other Non-Indexed Citations and MEDLINE (via OvidSP)

• EMBASE (via OvidSP)

• Cochrane Database of Systematic Reviews (via Wiley Online Library)

• Cochrane Central Register of Controlled Trials (via Wiley Online Library)

• Health Technology Assessment Database (via Wiley Online Library)

• Database of Abstracts of Review of Effects (via Wiley Online Library)

• Science Citation Index Expanded (via the Web of Science)

• Conference Proceedings Index-Science (via the Web of Science)

• World Health Organization’s International Clinical Trials Registry Platform

• Current Controlled Trials

• National Institutes of Health ClinicalTrials.gov

• Manufacturer and User Facility Device.

Sensitive keyword strategies using free text and, where available, thesaurus terms using Boolean operators and database-specific syntax were developed to search the electronic databases. Synonyms relating to the condition (e.g. sepsis) were combined with terms for the test (i.e. SeptiFast, SepsiTest and IRIDICA). No language restrictions were used on any database; however, the clinical effectiveness searches were date restricted. To date, all included rapid molecular tests (SeptiFast, SepsiTest and IRIDICA assay) have received a CE mark for use on whole-blood samples. For the SeptiFast test, clinical studies on whole-blood samples were first published in abstract form by Raglio et al. 44 in 2006 with subsequent full-text peer-reviewed publications by Mancini et al. 45 and Louie et al. 46 in 2008. The SeptiFast test gained its CE mark in 2006. For the SepsiTest assay, studies evaluating the use of SepsiTest on whole-blood samples in the clinical setting were first published in abstract form by Disqué et al. 47 in 2008, with a subsequent full-text peer-reviewed publication by Wellinghausen et al. 48 in 2009. SepsiTest received a CE mark in 2008. For the IRIDICA assay, studies evaluating the use of IRIDICA on whole-blood samples in the clinical setting were first published by Bacconi et al. ,49 in 2014, who used an IRIDICA–PLEX-ID hybrid system. The final version of the IRIDICA platform received a CE mark in 2014 and has been available for purchase by the NHS since 16 November 2014. Based on these data, the clinical effectiveness searches were limited by date from 2006 to May 2015. The search strategy of the current review updated the search strategy of an existing review on SeptiFast50 and amended it within the scope of the current review (i.e. the search strategy was amended to include generic, trademark or other product names of all the relevant index tests, other bacterial or fungal gene terms were added and were combined with PCR and population terms, and a limit to exclude all animal-only studies was introduced). An example of the MEDLINE search strategy is provided in Appendix 1.

Other resources

To identify additional published, unpublished and ongoing studies, the reference lists of all relevant studies were checked and a citation search of relevant articles (using the Web of Science Citation Index Expanded and Conference Proceedings Citation Index – Science) was undertaken to identify articles that cite the relevant articles. In addition, systematic keyword searches of the World Wide Web were undertaken using the Google search engine (Google Inc., Mountain View, CA, USA), key experts in the field were contacted and company submissions were screened for published or unpublished data additional to those identified in studies retrieved from the literature search.

All identified citations from the electronic searches and other resources were imported into, and managed using, the Reference Manager bibliographic software (version 12.0; Thomson ResearchSoft, San Francisco, CA, USA).

Study design

All clinical diagnostic accuracy studies that evaluated the index test with standard culture results (with or without MALDI-TOF MS) on patients’ whole-blood samples during the management of suspected sepsis were included. In reviews of test accuracy the ‘index test’ (the test of which the performance is being evaluated) can be viewed as the intervention.

Reviews of primary studies were not included in the analysis but were retained for discussion and identification of additional studies. Moreover, the following publication types were excluded from the review: animal models; biological studies; narrative reviews, editorials and opinions; case reports; non-English-language papers; and reports published as meeting abstracts only when insufficient methodological details are reported to allow critical appraisal of study quality.

Population

All studies of adults and children (of any age) with suspected bloodstream infections in secondary care (i.e. departments and wards providing care for acutely unwell patients and/or critical care units) who required blood cultures were included. Potential subgroups of interest included people with a suspected health-care-associated infection, people with a suspected community-acquired infection, children and neonates, people who are immunocompromised and people exposed to antibiotics prior to blood sample collection. Following clinical advice, people with febrile neutropenia were also considered as potential subgroup of interest. This group of patients usually undergo blood culture testing as their ability to show the classical signs of sepsis are impaired and failing to treat an underlying infection can result in mortality. This practice is supported by a recent large, retrospective study by Kaukonen et al. ,3 which found that a significant number of poor outcomes from severe systemic infection occurs in the absence of SIRS criteria at inception.

Target conditions

Suspected sepsis, including severe sepsis and septic shock as defined by Levy et al. 17

Interventions (index test)

The following tests (in conjunction with clinical assessment) performed on whole-blood samples for the detection of bloodstream bacterial and fungal pathogens were included:

• SeptiFast

• SepsiTest

• IRIDICA assay (extended to include preceding versions of the test if the authors believed that the data were likely to be generalisable to the IRIDICA assay).

Comparator test (reference standard)

The reference tests included current standard care to define the target condition, which included blood culture (in conjunction with clinical assessment) for the identification of bloodstream bacterial and fungal pathogens with or without MALDI-TOF MS. Where studies were identified that included more than one intervention, these would also form comparators for each intervention.

Outcomes

The outcomes of the review included a range of intermediate measures (such as diagnostic accuracy, discordant results with blood culture, time to result, time to treatment, test failure rates, duration of ICU and/or hospital stay, duration of broad- and narrow-spectrum antimicrobial therapy, readmission rate and change in antimicrobial treatment plan) and clinical outcome measures {such as side effects associated with broad-spectrum antimicrobial use, morbidity and mortality, severity of disease [as measured by scoring systems such as the Acute Physiology and Chronic Health Evaluation (APACHE) II, Simplified Acute Physiology Score (SAPS) II and the Sequential Organ Failure Assessment (SOFA)], rates of superinfection (including C. difficile), rates of resistant infections and health-related quality of life}, where available.

Meta-analysis

Where sufficient data existed, a meta-analysis was undertaken to generate pooled estimates of diagnostic parameters. The number of true positives, false negatives, false positives and true negatives from each study was meta-analysed to estimate sensitivity and specificity under the assumption that blood culture was 100% sensitive and specific. In brief, a bivariate normal model was used to model the population logit sensitivities and specificities in each study to account for correlation between sensitivity and specificity within studies. 52 We assumed that the observed number of true positives in study i, TP_i, was binomially distributed, with parameter, π_Ai, representing the study-specific sensitivity given the total number of positives on the reference test such that:

Similarly, we assumed that the observed number of true negatives in study i, TN_i, was binomially distributed with parameter, π_Bi, representing the study-specific specificity given the total number of negatives on the reference test such that:

We transform the parameters to the real line using the logit transformation such that:

Sensitivity and specificity are correlated within each study such that higher values for sensitivity tend to be associated with lower values for specificity, and vice versa. We model this by assuming that the study-specific logits for sensitivity and specificity arise from a bivariate normal distribution with population logits for sensitivity and specificity, (µ_A,µ_B)^T, respectively, and variance–covariance matrix, Σ_AB, such that:

σA2 represents the variability in the logit sensitivities between studies, σB2 represents the variability in the logit specificities between studies and σAB represents the covariance of the logit sensitivity and logit specificity.

The model was completed by giving the uncertain parameters the following prior distributions:

IW represents the inverse Wishart distribution on υ degrees of freedom.

This prior distribution has a between-study standard deviation (SD) of 1.5 [95% credible interval (CrI) 0.4 to 32.4].

Where there were relatively few studies to estimate the variance–covariance matrix, Σ_AB, a weakly informative prior distribution was used such that:

This prior distribution has a between-study SD of 0.5 (95% CrI 0.3 to 1.4).

Reasons for the heterogeneity in sensitivity and specificity between studies were explored using metaregression. Models with and without covariates were compared using the deviance information criterion, which provides a relative measure of goodness of fit that penalises complexity and can be used to compare different models for the same likelihood and data. 53

All parameters were estimated using Markov chain Monte Carlo simulation implemented using the WinBUGS software package (version 1.4.3; MRC Biostatistics Unit, Cambridge, UK). 54 Analyses were conducted in R using the R2WinBUGS interface package (version 3.1.2; R Foundation for Statistical Computing, Vienna, Austria). 55 Convergence was assessed using the Gelman–Rubin convergence statistic. Convergence was achieved relatively quickly and generally within 5000 iterations; in practice, a burn-in of 10,000 iterations was used. There was no evidence of high autocorrelation between successive samples of the Markov chains. Results were displayed as forest plots and summary receiver operating curve plots with 95% CrIs and 95% prediction intervals for sensitivity and specificity.

Narrative synthesis

A meta-analysis was not conducted on a range of intermediate measures (i.e. time to result, time to treatment, test failure rates, duration of ICU and/or hospital stay, duration of broad- and narrow-spectrum antimicrobial therapy, readmission rate and change in antimicrobial treatment plan) and clinical outcome measures (e.g. side effects associated with broad-spectrum antimicrobial use, morbidity and mortality, severity of disease, rates of superinfection, rates of resistant infections and health-related quality of life), as the necessary data were not available or it was inappropriate to statistically pool studies because of their variability in reporting outcome data. Therefore, as suggested by the guidance produced by the Cochrane Collaboration56 and the Centre for Reviews and Dissemination for undertaking systematic reviews,41,57 a narrative synthesis of included studies (grouped by outcome) was undertaken.

Number of studies identified/included

The literature searches identified 2892 citations. Of these, 66 studies met the inclusion criteria. A flow chart describing the process of identifying relevant literature can be found in Figure 2.

FIGURE 2.

Study flow chart (adapted from Moher et al. 58): clinical effectiveness review. a, Two studies included both SeptiFast and SepsiTest and are counted as individual studies in each test comparison (meta-analysis) with the reference standard.

Number and type of studies excluded

A total of 111 full-text articles were excluded, as they did not meet all the prespecified inclusion criteria. The majority of the articles were excluded primarily on the basis of having insufficient information to allow calculation of a diagnostic 2 × 2 metrics table (which includes data for true positives, false negatives, false positives and true negatives), incorrect population or interventions, or data reported in abstract form that were replaced by published full-text papers. A full list of excluded studies with reasons for exclusion is presented in Appendix 3.

Description of included studies (design and patient characteristics)

Quality characteristics

The QUADAS-2 tool,51 designed to evaluate the methodological quality of diagnostic accuracy studies, comprises four key domains: patient selection, index test, reference standard, and flow and timing. Using a set of signalling questions, each domain is assessed in terms of risk of bias [low, high or unclear risk (in the event of insufficient data in the publication to answer the corresponding question)] and the first three domains are also assessed in terms of applicability (no, yes or unclear concerns).

The overall methodological quality of the 66 included studies is summarised in Figure 3 and Table 7. The methodological quality of the included studies, as assessed using the QUADAS-2 tool, was variable. With the exception of Warhurst et al. ,111 all other studies were considered to be at risk of bias and to have concerns regarding applicability. 121

FIGURE 3.

Risk of bias and applicability concerns graph: review authors’ judgements about each domain presented as percentages across included studies.

(Confidential information has been removed.)

(Confidential information has been removed.)

Effectiveness of the interventions

This section presents the results of the following separately:

• an assessment of diagnostic test accuracy (meta-analysis, where applicable) of each diagnostic tests (i.e. SeptiFast, SepsiTest and IRIDICA in conjunction with clinical assessment) for rapidly identifying bloodstream bacteria and fungi

• an assessment of each diagnostic test on a range of other intermediate and clinical outcome measures (narrative synthesis).

Analyses were undertaken to assess the sensitivity of the results to alternative priors but these made little difference and thus only the results using the priors detailed in Meta-analysis have been presented.

Diagnostic test accuracy

A total of 62 studies contributed to the meta-analysis of sensitivity and specificity, including two studies118,119 that were three-arm (two index tests) studies. For simplicity, the correlation between tests was ignored in the analyses.

In total, 54 studies44–46,59–65,67–88,90–99,101,102,104–111,118,119 evaluated SeptiFast compared with blood culture, four studies48,113,118,119 evaluated SepsiTest compared with blood culture and four studies49,115–117 evaluated IRIDICA compared with blood culture. Separate meta-analyses are presented for each of these three tests in SeptiFast test, SepsiTest and IRIDICA assay compared with blood culture. In addition, one study112 evaluated SeptiFast compared with blood culture plus MALDI-TOF MS and one study114 evaluated SepsiTest compared with blood culture plus MALDI-TOF MS. As there was only one study for each of these comparisons, no meta-analysis was conducted and the data were summarised narratively.

SeptiFast test SeptiFast test compared with blood culture

The pooled sensitivity and specificity of SeptiFast compared with blood culture (54 studies) were 0.65 (95% CrI 0.60 to 0.71) and 0.86 (95% CrI 0.84 to 0.89), respectively (Figure 4). The 95% prediction intervals of 0.29 to 0.90 (sensitivity) and 0.62 to 0.96 (specificity) suggest considerable uncertainty in predicting the sensitivity and specificity of a new study. The between-study SDs for logit sensitivity and specificity were estimated to be 0.76 (95% CrI 0.57 to 1.01) and 0.66 (95% CrI 0.53 to 0.85), respectively, with a correlation of –0.05 (95% CrI –0.38 to 0.28). Figure 5 presents the joint distribution for sensitivity and specificity. The proportion of discordant results with blood culture (i.e. cases of disagreement between the reference standard and the index test) varied across studies from 6% to 46%, with a median of 17%.

FIGURE 4.

Sensitivity and specificity of SeptiFast compared with blood culture. FN, false negative; FP, false positive; PrI, prediction interval; TN, true negative; TP, true positive.

FIGURE 5.

Summary receiver operating curve plot of SeptiFast compared with blood culture (all studies). The 95% prediction interval indicates the extent of heterogeneity between studies. The green circles represent the sensitivity and specificity estimates from each study, with the circle size reflecting the study sample size.

Additional analyses were undertaken for the following subgroups only: neonates and children, antibiotic use prior to blood sample collection, suspected community- or health-acquired infection, patients with febrile neutropenia and studies that included/excluded contaminants in the data analysis. There was insufficient information on studies at low risk of bias (see Quality characteristics) and people who were immunocompromised to allow a meaningful estimate of test accuracy. None of the subgroups analysed was shown to significantly affect the results.

• Neonates and children Six studies provided data on children and neonates. Of these, three studies included neonates only,69,73,98 one included children only aged 1 month to 17 years105 and two included both neonates and children. 86,109 Of the remaining studies, six were conducted in adults and children,70,85,93,96,99,111 28 were conducted in adults45,46,63,64,67,68,71,74–76,78,80,81,83,84,87,90–92,94,95,97,101,102,104,106,110,119 and 14 did not report the age of participants. 44,59–62,65,72,77,79,82,88,107,108,118 Based on comparison of models with and without covariates for an age category, there was no evidence that sensitivity and specificity was affected by the age of the subjects (see Appendix 5).

• People exposed to antibiotics prior to blood sample collection The proportion of patients receiving antibiotics prior to blood draw was recorded in 24 studies24,62,64,65,68,70,71,74,77,80,83,90–94,97–99,104,106,110,111,119 and ranged from 0% to 100% with a median of 72%. The remaining studies either did not report prior exposure to antibiotics or provided only limited information and were, therefore, excluded from the analysis. There was no evidence that exposure to antibiotics prior to blood sample collection affected the estimates of sensitivity and specificity (see Appendix 5).

• People with suspected community- or health-acquired infection Clinical setting was used as a proxy for suspected community- or health-acquired infection. Studies were grouped according to whether infection was diagnosed in hospital (38 studies45,59–64,67,68,70,71,73–79,81,83,85,86,90–97,99,102,104,107–109,111,119), emergency department (three studies80,84,106), mixed setting of emergency or other hospital department (four studies46,82,87,101) or not recorded (nine studies44,65,69,72,88,98,105,110,118). Based on comparison of models with and without covariates for the clinical setting, there was no evidence that this affected sensitivity and specificity (see Appendix 5).

• People with febrile neutropenia In total, eight studies provided data on patients with febrile neutropenia. 45,65,71,75,91,99,108,110 Of these, six studies included patients (100%) with febrile neutropenia only. 65,71,75,91,99,108 Studies by Mancini et al. 45 and Idelevich et al. 110 reported that 92% and 98% of patients had febrile neutropenia, respectively. Based on a comparison of models with and without covariates for the presence of patients with febrile neutropenia, there was no evidence that this affected sensitivity and specificity (see Appendix 5).

• Studies with inclusion/exclusion of contaminants In total, 32 studies44,45,64,67–70,73,76,78,81–85,87,91–98,105–110,118,119 included contaminants in the reported results, eight studies46,72,80,86,99,101,102,111 excluded contaminants and 14 did not report on handling of contaminants. 59–63,65,71,74,75,77,79,88,90,104 Based on a comparison of models with and without covariates for the inclusion/exclusion of contaminants, there was no evidence that this affected sensitivity and specificity (see Appendix 5).

SeptiFast test compared with blood culture plus MALDI-TOF MS 

Only one study,112 which compared the SeptiFast test with MALDI-TOF MS, provided data on diagnostic test accuracy. This study reported a sensitivity and specificity of 0.58 (95% CI 0.30 to 0.86) and 0.74 (95% CI 0.64 to 0.85), respectively.

SepsiTest SepsiTest compared with blood culture 

The pooled sensitivity and specificity of SepsiTest compared with blood culture was 0.48 (95% CrI 0.21 to 0.74) and 0.86 (95% CrI 0.78 to 0.92), respectively (Figure 6). The 95% prediction intervals of 0.07 to 0.90 (sensitivity) and 0.66 to 0.95 (specificity) suggest considerable uncertainty in predicting the sensitivity and specificity of a new study. The between-study SDs for logit sensitivity and specificity were estimated to be 0.90 (95% CrI 0.50 to 1.92) and 0.45 (95% CrI 0.27 to 0.90), with a correlation of –0.03 (95% CrI –0.73 to 0.68). Figure 7 presents the joint distribution for sensitivity and specificity, and highlights the extent of the heterogeneity between studies (as indicated by the 95% prediction interval). Owing to insufficient information provided in the included studies, planned subgroup analyses were not conducted and there were insufficient studies to conduct meaningful analyses.

FIGURE 6.

Confidential information has been removed.

FIGURE 7.

Confidential information has been removed.

SepsiTest compared with blood culture plus MALDI-TOF MS

 Only one study114 that compared the SepsiTest assay with MALDI-TOF MS provided data on diagnostic test accuracy. This study reported a sensitivity and specificity of 0.11 (95% CI 0.00 to 0.23) and 0.96 (95% CI 0.92 to 1.00), respectively.

IRIDICA assay compared with blood culture

The pooled sensitivity and specificity of IRIDICA compared with blood culture was 0.81 (95% CrI 0.69 to 0.90) and 0.84 (95% CrI 0.71 to 0.92), respectively (Figure 8). The 95% prediction intervals of 0.55 to 0.94 (sensitivity) and 0.50 to 0.96 (specificity) suggest considerable uncertainty in predicting the sensitivity and specificity of a new study. The between-study SDs for logit sensitivity and specificity were estimated to be 0.46 (95% CrI 0.28 to 0.93) and 0.65 (95% CrI 0.39 to 1.27), with a correlation of 0.06 (95% CrI –0.71 to 0.75). Figure 9 presents the joint distribution for sensitivity and specificity and highlights the extent of the heterogeneity between studies (as indicated by the 95% prediction interval). Owing to insufficient information provided in the included studies, planned subgroup analyses were not conducted and there were insufficient studies to conduct meaningful analyses.

FIGURE 8.

Confidential information has been removed.

FIGURE 9.

Confidential information has been removed.

Electronic resources

A systematic search was undertaken of the existing published literature evaluating the economic impact of the SeptiFast, SepsiTest and IRIDICA tests to rapidly detect and identify bacterial and fungal DNA that may be present in the bloodstream in people who are suspected of having sepsis.

Studies were identified by searching the following electronic databases and research registers:

• MEDLINE In-Process & Other Non-Indexed Citations and MEDLINE (via OvidSP), searched from 1948 to May 2015

• EMBASE (via OvidSP), searched from 1980 to May 2015

• Cochrane Database of Systematic Reviews (via Wiley Online Library), searched from 1996 to May 2015

• Cochrane Central Register of Controlled Trials (via Wiley Online Library), searched from 1898 to May 2015

• Health Technology Assessment Database (via Wiley Online Library), searched from 1995 to May 2015

• Database of Abstracts of Review of Effects (via Wiley Online Library), searched from 1995 to May 2015

• NHS Economic Evaluation Database (via Wiley Online Library), searched from 1995 to May 2015

• Science Citation Index Expanded (via the Web of Science), searched from 1899 to May 2015

• Conference Proceedings Index-Science (via the Web of Science), searched from 1990 to May 2015

• World Health Organization’s International Clinical Trials Registry Platform, searched from 2007 to May 2015

• Current Controlled Trials, searched from 2000 to May 2015

• NIH ClinicalTrials.gov, searched from 2000 to May 2015

• Manufacturer and User Facility Device, searched from 1991 to May 2015

• MEDION database.

Sensitive keyword strategies using free text and, where available, thesaurus terms using Boolean operators and database-specific syntaxes were developed to search the electronic databases. Synonyms relating to the condition (e.g. sepsis) and the test (i.e. SeptiFast, SepsiTest and IRIDICA) were combined with a search filter aimed at restricting results to economic and cost-related studies (used in the searches of MEDLINE and EMBASE). No language restrictions were used on any database; however, the searches were restricted by date (see Chapter 2, Methods for reviewing effectiveness, for further details). In brief, CE approval for the oldest rapid molecular test (i.e. SeptiFast) was obtained in 2006. As a result, no relevant economic evaluations were expected to be published prior to this date. An example of the MEDLINE search strategy is provided in Appendix 6.

Other resources

To identify additional published, unpublished and ongoing studies, the reference lists of all relevant studies were checked and a citation search of relevant articles (using the Web of Science Citation Index Expanded and Conference Proceedings Citation Index – Science) was undertaken to identify articles that cited the relevant articles. In addition, systematic keyword searches of the World Wide Web were undertaken using the Google search engine, key experts in the field were contacted and company submissions were screened for published or unpublished data additional to those identified in studies retrieved from the literature search.

All identified citations from the electronic searches and other resources were imported into, and managed using, the Reference Manager bibliographic software.

Studies were selected for inclusion according to predetermined inclusion and exclusion criteria. A summary of these criteria is provided in Table 13.

Studies were selected for inclusion through a two-stage process:

• Level 1 screening: titles and abstracts were independently examined for inclusion by two reviewers (RR and MS). Any disagreements in the selection process were resolved through discussion.

• Level 2 screening: full manuscripts of selected citations were then retrieved and assessed by one reviewer (RR). A second reviewer (MS) performed an independent quality check to ensure that the inclusion criteria were applied correctly. Any disagreements in the selection process were resolved through discussion.

No formal quality assessment was conducted. When assessing the methodological quality of the economic literature, a number of checklists are available; however, quality assessment checklists for assessing economic evaluations of diagnostic tests are limited. Similarly, the majority of checklists focus on the quality of reporting rather than the methodological quality of a study. Owing to these limitations the relevance of each study to the decision problem is discussed within Descriptive summary of the study included in the review, Critique of the studies included in the review and Relevance of existing economic evaluations for National Institute for Health and Care Excellence decision-making.

Identified studies

A total of 89 citations were retrieved. Of these, 77 citations were identified via database searching and an additional 12 citations were retrieved through other sources (Figure 10).

FIGURE 10.

Study flow chart (adapted from Moher et al. 58): economic review.

Eighty references were excluded at title and abstract stage. Nine references related to eight studies were examined at full-text level and four studies (corresponding to four references) were identified as meeting the inclusion criteria of the systematic review of economic evaluations. 89,103,128,129 These included an economic evaluation of the IRIDICA–PLEX-ID hybrid assay reported in a poster presentation (submitted by the company). 128 It was highlighted that the system evaluated in the poster presentation was not the final IRIDICA BAC BSI assay but was an earlier version that used components of PLEX-ID and is assumed to be equivalent (see Chapter 1, IRIDICA BAC BSI).

Five papers were excluded after retrieval of the full papers,130–134 the rationale being that the results were published in full elsewhere,130 they were other interventions131–133 and there was an absence of an economic evaluation and inappropriate intervention. 134

Descriptive summary of the study included in the review

A tabulated summary of the key characteristics of the studies included in the economic review, as determined by the authors of this report, is presented in Table 14. It was not possible for the External Assessment Group to check the economic models, as only the publications were available in the public domain.

Of the four identified economic studies (corresponding to three full texts89,103,129 and one poster presentation128), three economic evaluations (full text) compared the addition of SeptiFast to blood culture with blood culture alone89,103,129 and one compared the addition of the IRIDICA–PLEX-ID hybrid assay to blood culture with blood culture alone (poster presentation). 128 No economic evaluations of SepsiTest were identified. None of the four published economic evaluations was conducted in a UK setting. However, the RADICAL study,116 used for the impact of the IRIDICA–PLEX-ID hybrid assay on treatment modification, included two UK (out of nine) sites. 116

Two103,129 out of the three SeptiFast studies were funded by Roche Diagnostics; it was unclear whether or not the third study89 was funded by the company. The IRIDICA–PLEX-ID hybrid cost minimisation study was funded by Abbott Diagnostics.

The target population, condition and setting varied between the four identified economic studies. Mancini et al. 103 included haematological patients with signs of systemic inflammatory response syndrome with suspected sepsis (SIRS-SS). Alvarez et al. 89 included patients diagnosed with severe sepsis and septic shock. Lehmann et al. 129 included all post-surgical and ICU patients with a sepsis episode (predominantly hospital-acquired infection), while Bilkovski et al. 128 included critically ill patients with suspected bloodstream infection.

Three studies were cost-minimisation studies. 89,103,128 Two were conducted within studies: a non-matched retrospective study evaluating SeptiFast89 and a propensity score-matched study evaluating SeptiFast against blood culture. 103 The Alvarez et al. study89 justified the use of cost minimisation given the absence of mortality data associated with the use of SeptiFast. The third cost minimisation study was undertaken using a decision tree model128 and evaluated the IRIDICA–PLEX-ID hybrid assay by combining evidence from the RADICAL study116 on the impact of the test in terms of treatment decision, and evidence from MALDI-TOF MS studies123,124 on the impact of rapid identification on the reduction in hospital and ICU length of stay. The main assumptions within the model were that (a) all patients start on empiric antimicrobial therapy and (b) only patients testing positive using the IRIDICA–PLEX-ID hybrid assay experience a reduction in length of stay.

Only one study was a cost-effectiveness (cost–utility) analysis and estimated the ‘cost per incremental survivor’ and the cost per quality-adjusted life-year (QALY) gained of introducing SeptiFast. 129 An algebraic model was constructed, which independently estimated the potential cost impacts and clinical outcomes associated with a change in treatment plan as a result of the earlier identification of inadequate treatment through the use of SeptiFast. Only positive SeptiFast results were considered to provide sufficient evidence to allow a treatment change, with the authors concluding that ‘withdrawal of antimicrobial treatment upon a PCR negative result is not recommended’. 129 Cost savings and increased health associated with quicker adequate treatment were estimated assuming a relationship between a reduction of 1 day in inadequate treatment and changes in both length of stay and mortality. This study used evidence collected prospectively from five hospitals to inform the change in treatment decision: two based in Germany, one in Spain, one in Italy and one in the USA. 66 In addition, the modelling uses pooled data on the impact of inadequate treatment on outcomes from two previously published studies135,136 conducted in the USA.

Although not clearly stated by the authors, it is believed by the authors of this report that Mancini et al. 103 and Alvarez et al. 89 report results from an Italian and Spanish setting, respectively. In the case of the studies by Lehmann et al. 129 and Bilkovski et al.,128 the country on which the analysis was based is unclear, as the data came from multicentre studies.

The economic evaluation was conducted in patients in the ICU in three studies. 89,128,129 All studies appear to use a hospital (health-care payer) perspective, although this was not explicitly stated in two studies. 103,128 Uncertainty was examined in two studies. 89,129 Only one study used QALYs as a measure of benefit. 129

The cost of performing SeptiFast varied between studies and ranged from €178.75103 (£128.70 assuming an exchange rate of €1 to £0.72137) to €300129 (£216). One study evaluated the IRIDICA–PLEX-ID hybrid assay and assumed a cost of US$250 per test128 (£160.67 assuming an exchange rate of US$1 to £0.64). 137

Two studies89,103 considered antibiotics costs, and both reported a reduction in antibiotics costs associated with the use of SeptiFast. One study included the savings in classical diagnostic assays103 with the use of SeptiFast. Three of the four studies considered the costs associated with ICU/hospital stay, and all reported a reduction in hospital/ICU stay with the use of the test. 76,89,128 None of the studies identified considered the impact on costs associated with the potential reduction in antibiotic resistance.

Overall, all three cost-minimisation studies reported a reduction in total costs, with the additional cost of SeptiFast and the IRIDICA–PLEX-ID hybrid assay being outweighed by the savings in antibiotics and/or hospital costs. 89,103,128 Mancini et al. 103 reported an overall saving of €430.73 per patient receiving a SeptiFast test and blood culture compared with those diagnosed using only blood culture (€1579.80 vs. €2010.53), considering savings in diagnostic and instrumental assays, and medications (including antibiotics, antimycotics, antiviral agents and other drugs). Alvarez et al. 89 reported a saving of €9970 per SeptiFast test, with the majority of savings achieved based on a reduction in ICU length of stay. Bilkovski et al. 128 reported a total saving of US$1,123,372 over 422 patients tested (equating to a saving of £2662 per patient tested), with the IRIDICA–PLEX-ID hybrid assay associated with a reduction in hospital or ICU length of stay. Finally, Lehmann et al. 129 reported that the cost of the SeptiFast test could be recovered if the daily medical costs were above €717 and suggested that this was likely to be the case. 129 The authors reported the cost per incremental survivor and cost per QALY gained to be €11,477 (95% CI €9321 to €14,977) and €3107 (95% CI €2523 to €4055), respectively.

Critique of the studies included in the review

The Lehmann et al. 129 economic evaluation, which compared the use of SeptiFast and conventional blood culture, appears to be a reasonably well-conducted cost-effectiveness analysis based on the description provided by the authors. However, this study has a number of limitations that may restrict the generalisability to UK practice, not only because none of the hospitals contributing to the study was located in the UK. The data used in the Lehmann et al. 129 cost-effectiveness model on the potential impact of the test in terms of treatment modification are relatively out of date and were collected prior to recent guidelines on the diagnosis and treatment of sepsis. 16,18–20 Therefore, the extent to which the inadequate treatment observed in Lehmann et al. 66 is generalisable to current practice in the UK is not known. Furthermore, the studies on how earlier adequate treatment translates into reduced morbidity and mortality were dated and cohort based and, as acknowledged by Lehmann et al. ,129 could be potentially confounded. Lehmann et al. 129 used a relative risk of non-survival between immediate and delayed adequate antimicrobial treatment of 2.32, but report in their discussion section that a value of 1.8 estimated in a clinical trial of immunomodulating therapy for severe sepsis138 would have been a better estimate of the relative risk of non-survival associated with inadequate treatment. Using this value led to an increase in the cost per incremental survivor from €11,477 to €14,670 (an alternative cost per QALY gained value was not reported). It should be noted that the relative risk is still relatively dated as it was published in 2003. 138 The relative risk of mortality was pooled from three studies, rather than the more appropriate method of meta-analysing. The same limitations of the relationship between inadequate treatment and mortality also apply to the relationship between inadequate treatment and length of hospitalisation, which uses data from the two old studies. It is noted that, in the original Lehmann et al. 66 paper, data were collected from two separate sites of attendance: ICU or surgical ward, and emergency room or other. The estimated gainable days of adequate treatment per 100 SeptiFast tests were 36.4 days for the ICU or surgical ward group and 10.6 days for the remaining attendance method. It is unclear if the evaluation by site was preplanned or if the analysis on ICU patients alone could be viewed as data dredging; ideally a replication of the study within the ICU would provide more conclusive data. The failure rate of the SeptiFast test was also not considered. In the model, patients receiving SeptiFast are assumed to experience a mortality benefit associated with rapid identification. Although this may be plausible, so far, evidence on the impact of SeptiFast on clinical outcomes of mortality and hospital length of stay is inconclusive from five comparative studies. 89,103,110,112,139

The cost-minimisation study conducted by Alvarez et al. 89 is based on a retrospective non-matched comparison of the costs in a cohort prior to the introduction of SeptiFast and a cohort following the use of SeptiFast. Alvarez et al. 89 reported a non-significant increase in mortality in the cohort receiving the intervention at 28 days (29% vs. 24%, non-significant; p-value not reported), and at 6 months (41.6% vs. 37%, non-significant; p-value not reported). In contrast, the authors reported a significant reduction in ICU length of stay for survivors (31.0 ± 19.4 vs. 22.9 ± 29.9 days; p < 0.05), hospital length of stay (21.3 ± 23.4 vs. 18.3 ± 21.4 days; p < 0.05) and ICU length of stay (24.1 ± 21.9 vs. 18.3 ± 11.4 days; p < 0.05). The authors calculated the total cost to be €42,198 for the control group and €32,228 for the intervention group, corresponding to a saving of €9970 (with more than 85% of savings attributable to a reduction in ICU length of stay). This study has serious limitations that need further consideration. The results need to be interpreted with caution because of the retrospective nature of this study, the small sample size of each arm (48 patients in the intervention group and 54 patients in the control group) and the imbalances between patient characteristics, as shown in Table 15. It is also unclear whether patients were randomly selected to receive SeptiFast or whether these patients were selected based on their characteristics. Although the authors reported no differences in terms of patients’ ages, sex distribution or risk indices, it is unclear from the paper how these risk indices were calculated. There appear to be large imbalances in the initial diagnoses that could impact on ICU length of stay (main source of savings). It should be noted that no difference in ICU or hospital length of stay was observed in other comparative studies (RCT or propensity-matched score studies). 103,110,112,139 The authors report a saving in antibiotics (excluding cost of the test) of €764 per patient [as a result of a reduction in the number of antibiotics used per patient of 0.9 (from 5.1 to 4.2)]. The authors do not say which antibiotics were used in the study, or for how long. It should be noted that, in England, antibiotics are typically given as a 5- to 7-day course, with a daily cost around £50. The failure rate of the SeptiFast test was not considered.

The second cost-minimisation study, by Mancini et al. 103 evaluated the use of SeptiFast with blood culture and is based on a propensity score-matched approach with 101 matched SIRS-SS episodes. The authors reported a trend (although not significant) towards a reduction in SIRS-SS-related mortality associated with SeptiFast. The authors reported that the differences in mortality reached statistical significance when the tolerance (calliper) used for the propensity matching was reduced to only 5%, corresponding to 77 matched episodes (64 patients in the intervention group and 68 in the control group). Although the use of a lower calliper improved the quality of matching, it reduced the precision by decreasing the sample size and could also introduce bias, as the estimate is no longer the effect of treatment in the treated subjects, but the effect of treatment in those treated subjects for whom a control was found. 140 Using the tight calliper of 0.05, the authors found a mortality rate of 3.13% in the intervention group and 14.71% in the control group (i.e. a difference in the mortality rate of 11.58%). Based on a pilot study, the authors reported that 278 pairs of SIRS events were needed to demonstrate a 10.6% difference in mortality (9.7% in the intervention group vs. 20.3% in the control group), which is greater than the numbers analysed in Mancini et al. 103 The results from this study therefore need to be interpreted with caution. It should also be noted that the PCR test was implemented under optimal conditions. The authors found no reduction in SIRS-SS episode length, even under a tight strict matching criterion. In this study, the SeptiFast test was estimated to lead to a reduction in medication. Based on information provided in a supplementary appendix associated with the paper, the cost of antibiotics was higher in patients receiving SeptiFast (€11.78). In contrast, the costs of antimycotics (commonly called antifungals in England) (€376.62), antiviral agents (€1.65) and other drugs (€91.39) were reduced. Although the two cohorts were propensity matched, it is unclear whether or not the suspected organisms in the two cohorts were matched, which may affect the antibiotics prescribed. In the control arm, antimycotics represented approximately 53% of total medication costs. This figure appears relatively high compared with that expected in England based on Warhurst et al. 10 It is also unclear how the cost of antibiotics and antifungals differ between England and Italy. The failure rate of the SeptiFast test was not considered, nor was it clear whether patients were randomly selected to receive SeptiFast or were selected based on their characteristics.

Finally, the Bilkovski et al. study128 is a decision tree using evidence from the RADICAL study116 on the impact of the IRIDICA–PLEX-ID hybrid assay (referred to by the authors as the PCR/ESI-MS) in terms of treatment decision and studies on MALDI-TOF MS123,124 for the impact of the test on clinical outcomes. This study was available only as a poster presentation and, therefore, it is difficult to judge its quality. However, the External Assessment Group notes some inconsistencies. In the RADICAL study, Vincent et al. 116 reported that, among the 529 patients included, the overall mortality rate was 29%, suggesting that about 375 patients survived hospital discharge. However, the economic evaluation conducted by Bilkovski et al. 128 is based on 422 patients and includes only 290 patients surviving to hospital discharge. Patients who tested positive using the IRIDICA–PLEX-ID hybrid were assumed to experience a reduction in hospital and ICU length of stay because of the earlier availability of results and potential therapy changes based on data from two previous studies of MALDI-TOF MS. 123,124 The validity of this assumption is uncertain as MALDI-TOF MS and the IRIDICA–PLEX-ID hybrid assay are used differently and provide different information. Notably, studies of MALDI-TOF MS are conducted in patients with a positive blood culture only. Therefore, the impact of treatment changes when blood culture is negative is unknown and there may be unintended consequences. Importantly, studies on MALDI-TOF MS123,124 used in the economic evaluation evaluated the impact of MALDI-TOF MS in combination with an antimicrobial stewardship programme compared with blood culture prior to the programme. Hence, the impacts of MALDI-TOF MS in these studies are highly likely to be confounded by the simultaneous introduction of an antimicrobial stewardship programme. These studies were also non-randomised in nature and, therefore, subject to potential biases. Bilkovski et al. 128 predicted that the use of the IRIDICA–PLEX-ID hybrid system would lead to a reduction in hospital and ICU length of stay of 1.6 days and 0.7 days, respectively, in all patients and of 4.2 days and 1.8 days, respectively, in patients with positive IRIDICA-PLEX-ID results. It is unclear how the model estimate relates to the impact of the test in practice and whether or not such savings would be observed. As previously stated, evidence from five comparative studies of SeptiFast on the impact of a similar test in terms of mortality or ICU and hospital length of stay is inconclusive. 89,100,103,110,112 Test failure was not considered in the analysis.

The number of blood samples that need analysing per day

Three broad scenarios were undertaken regarding the number of blood samples that need to be analysed each day: based on the number observed in a recent clinical study10 assuming an increase for community-acquired infection (2.4 blood samples); assuming 17 blood samples; and assuming 68 blood samples. These values were thought to provide a wide range of plausible values that could be undertaken within units of different sizes.

The estimated costs of the interventions and matrix-absorbed laser desorption/ionisation time-of-flight mass spectrometry

For all tests there is a fixed cost and a variable cost that is determined by the numbers of tests undertaken. Following advice obtained from our clinical advisors, we assumed that the purchase cost of a machine could be equally divided over an assumed 7-year period of use. Annual costs associated with the maintenance of the machines were incorporated.

For each intervention, the cost will be dependent on whether or not the equipment required for the specific test (SeptiFast, SepsiTest, IRIDICA and MALDI-TOF MS) is already available within the microbiology laboratory. SeptiFast and IRIDICA have their own bespoke PCR machines, whereas SepsiTest can be run on generic PCR machines. This led to two scenarios being evaluated for each intervention. These, in order of cost, are (1) purchasing of the required machine for the specific test or a generic PCR machine; and (2) no additional machinery needing to be purchased. In both cases consumable costs would be incurred.

The estimation of the cost of an average test is far from robust, relying on uncertain assumptions in the number of tests per day and machinery costs, and therefore adds additional uncertainty.

Note that, in accordance with the NICE reference case for diagnostic evaluations, value-added tax (VAT) is not included in economic evaluations. 141 Based on advice from our clinical experts, it was assumed that no additional staff costs or room costs would be incurred if any of the interventions were purchased, although this is less likely to be the case should it be assumed that 17 or 68 samples need to be analysed per day. For simplicity, neither transport costs associated with sample testing nor additional staff training was included.

No discounts associated with bulk purchasing of equipment has been assumed for any intervention. It is possible that these exist in reality; if the costs of the test are lower than those assumed in the analyses, then the incremental cost-effectiveness ratio (ICER) would be lower than those presented with all other variables held constant.

The assumed failure rates of the interventions

Both SeptiFast and IRIDICA use internal controls that could be subject to failure, as could the controls on a PCR machine used by SepsiTest. For SeptiFast, a 6.9% failure rate was reported in Warhurst et al. ,10 although a greater value of 22.9% was reported in Paolucci et al. 99 Data on the failure rate of IRIDICA have been reported in Metzgar et al,117 and indicate a rate of (confidential information has been removed). No data on the failure rate of SepsiTest were identified.

In the base case it has been assumed that SeptiFast has a failure rate of 6.9%. In sensitivity analyses a failure rate of 11.7% was assumed for SeptiFast based on a naive pooling of failure results from Warhurst et al. 10 (69 failures) and from Paolucci et al. 99 (100 failures) divided by the numbers of samples analysed in all of the SeptiFast versus blood culture trials combined (11,659 samples), assuming no failures in any SeptiFast study. This results in an estimated failure rate for SeptiFast of 1.4%. For the base case for IRIDICA, a failure rate of (confidential information has been removed) was assumed based on a naive pooling of data from all of the studies assuming no failures in any study but Metzgar et al. 117

Although Warhurst et al. ,10 Paolucci et al. 99 and Metzgar et al. 117 explicitly stated that any failures were excluded from analyses of diagnostic accuracy, it is not clear if failures occurred in the remaining identified studies but were not reported as such and were treated as a negative result. If failures had been excluded from the analysis of diagnostic accuracy but not reported, then this would be beneficial to the relevant intervention.

No data were identified on the failure rate of SepsiTest. Rather than assign an arbitrary value to this parameter, the value was set to zero with the acknowledgement that this was likely to be favourable to SepsiTest.

The quality-adjusted life-year gains associated with preventing 30-day mortality

It was assumed that each 30-day mortality avoided is associated with a gain of 11.32 discounted QALYs. This value was calculated based on (a) the estimated number of discounted life-years for a typical patient and (b) the estimated quality of life after a sepsis episode to account for the possible reduced quality of life in sepsis survivors. It should be noted that the discounted QALY gains for neonates and children would be greater than those for adults because of their longer life expectancy, although the exact increase is uncertain.

Although there is evidence that the survival after a sepsis episode may be lower than in the general population,142–144 for simplicity we assumed that survival of patients with sepsis was comparable with that of the general population. National life tables for England and Wales for the period 2011–13145 were used to estimate the life expectancy of a typical patient, assuming an age of 58 years and a sex split of 60%/40% (male/female) as reported in Warhurst et al. 10

Patients were assumed to have a utility value of 0.68 based on the European Quality of Life-5 Dimensions score reported by Cuthbertson et al. 142 at 5 years after a severe sepsis episode, which is similar to the value reported by Drabinski et al. 146 If the utility predicted for the general population for an age and sex profile147 was lower than 0.68, this value was used instead. A discount rate of 3.5% per annum was used, as recommended in the NICE reference case. 141

The assumed cost of 1 day’s treatment in an intensive care unit

This value was calculated from NHS reference costs. 148 Service code CCU03 [medical adult patients (unspecified specialty)] was assumed to be representative of treatment for sepsis patients. This service code is subdivided by the number of organs supported, ranging from 0 to ≥ 6, and an average of the reported average unit costs weighted by activity levels was calculated. This resulted in a cost of ICU care of £1057 per day. This is slightly more than the weighted average for service code CCU002 [surgical adult patients (unspecified specialty)] selected, which was £987 per day.

The assumed cost of 1 day’s treatment in a standard hospital ward

This value was calculated from NHS reference costs,148 assuming that the average excess bed-day cost per non-elective impatient of £275 was appropriate.

The assumed cost of typical empirical antimicrobial treatment for sepsis

Based on the advice of our clinical advisors, it was assumed that 7 days’ treatment with either 18 g of piperacillin/tazobactam per day or 3 g of meropenem per day was an appropriate empirical treatment for typical sepsis patients. Using British National Formulary costs,149 these prices were estimated to be £51.60 per day (assuming 4.5 g every 6 hours) for piperacillin/tazobactam and £48.00 (assuming 1 g every 8 hours) for meropenem. Given the uncertain proportion of the drugs used in England, it was assumed that a cost per day of £50, equating to a cost for a course of treatment of £350, was not unreasonable. However, an expert clinician on the Diagnostic Appraisal Committee commented that these costs may be high for adults admitted to regular hospital wards or for children. If this was the case, then the value used for a course of treatment would be favourable to the interventions.

The assumed 30-day mortality rate for those with suspected sepsis

It was assumed that the 28-day mortality rate reported in a recent health technology assessment,111 set in England, could be generalised to a 30-day mortality rate. This value was 13% (95% CI 11% to 16%),10 with the rate of hospital mortality being 21% (95% CI 17% to 23%). This value has some support from data provided by Kaukonen et al. ,3 which analysed hospital mortality rates in patients with severe sepsis in Australia and New Zealand and showed a decrease across time, with values of approximately 10% for SIRS-positive sepsis and 20% for SIRS-negative sepsis.

An alternative value of 29%, as reported in Mouncey et al. 9 (albeit for 90-day mortality), in patients with early septic shock was tested in sensitivity analyses. This value was supported by data from Levy et al. ,6 who reported hospital mortality rates of 29% when there was high compliance with a resuscitation bundle, although the patients included in this study were those with severe sepsis and septic shock and would be likely to have a worse prognosis.

Results from base case 2 using the average clinician values and assuming a 30-day mortality rate of 13%

The estimated costs and estimated QALYs for each intervention and MALDI-TOF MS compared with blood culture when it is assumed that the machinery required for the intervention needs to be purchased are shown in Table 39. The results when no additional machinery needs to be purchased are provided in Table 40.

It is seen that the cost per QALY values are relatively low for all interventions. It is assumed that any costs associated with allowing machines to be run on a 7 days per week, 24 hours per day basis could be subsumed into the intervention costs while still producing ICERs that are below £20,000 per QALY gained. To illustrate this, if it was assumed that there were additional costs of £100,000 to operate SeptiFast continuously, then the ICER, assuming 17 samples to be analysed per day and machine purchase, would be calculated as:

Incremental cost (see Table 39): £201,782 + £100,000 = £301,782.

Incremental QALYs (see Table 39): 48.81.

ICER: £301,782/£48.81 = £6183.

Results from base case 2 using the average values from the clinician survey and assuming a 30-day mortality rate of 29%

The estimated costs and estimated QALYs for each intervention test compared with blood culture, when it is assumed that the machinery required for the intervention needs to be purchased, are shown in Table 41. The results when no additional machinery needs to be purchased are provided in Table 42.

Results from base case 2 using the average values from the clinical survey, using the results from the data synthesis for SeptiFast

In the base case, for SeptiFast, it is assumed that data from Warhurst et al. 10 are the most appropriate, as these data were taken from a large study (853 patients) in an English population that was believed to be the study with the highest quality (see Table 6). The only study with a similar level of quality was that by Tafelski et al. 112 but, as that was a small study (88 patients) set in Germany, the results are believed to be not as generalisable as those from Warhurst et al. 10

In an alternative scenario, the impact of using the estimated results from the synthesis of diagnostic accuracy data (see Figure 3) was explored. The mid-point values of 65% sensitivity and 86% specificity were used, with the prevalence of sepsis identified by blood culture by episode assumed to be that reported in table 6 of Warhurst et al. ,10 of 80 out of 922 (8.7%). These data result in an estimated positivity rate of 18.4% for SeptiFast with 30.6% of subsequent blood cultures also being positive and 67% being negative. In this analysis, a failure rate of 1.4% was assumed for SeptiFast, based on a naive pooling of failure results from Warhurst et al. 10 (69 failed samples) and from Paolucci et al. 99 (100 failed samples) divided by the number of samples analysed in the SeptiFast versus blood culture trials (11,659 samples).

The estimated cost per QALY values for SeptiFast, obtained using the results of the synthesis of diagnostic accuracy, are shown in Table 43 (assuming a 13% mortality rate) and Table 44 (assuming a 29% mortality rate).