Multiplex tests to identify gastrointestinal bacteria, viruses and parasites in people with suspected infectious gastroenteritis: a systematic review and economic analysis

Search strategies for clinical effectiveness

The search strategy for the clinical effectiveness review is detailed in Appendix 1. Briefly, the search strategy included:

• databases: Ovid MEDLINE 1946 to November week 3 2015; Ovid MEDLINE In-Process & Other Non-Indexed Citations 31 December 2015; Ovid EMBASE 1980 to 2015 week 52; Web of Science 1980 to 31 December 2015; Cochrane Database of Systematic Reviews issue 5 of 12, May 2016 – all sections. Weekly auto-alerts for emerging evidence were run in Ovid MEDLINE, Ovid EMBASE and PubMed from 1 January 2016 to 30 April 2016

• reference lists of all reviews and included studies

• websites of NICE, PHE, Food and Drug Administration (FDA) and the manufacturers of the multiplex PCR tests

• ongoing studies of the following sources: National Institutes of Health ClinicalTrials.gov, Current Controlled Trials, World Health Organization International Clinical Trials Registry Platform and UK Clinical Trials Gateway.

Inclusion and exclusion of relevant studies

Patient selection domain

Addition of one signalling question: was only one sample per episode of diarrhoea included in the study?

It is important that studies do not include more than one sample per patient per episode of diarrhoea to avoid double counting. Many studies reported samples rather than patients, or studies reported greater numbers of samples included than numbers of patients, where it would be important to ascertain that samples are representative of one episode of diarrhoea. We assumed that a period of 1 month between samples was required to define separate episodes of diarrhoea.

Index test domain

The addition of one signalling question: was the index test undertaken as recommended by the manufacturer?

Bias may occur if the study deviates from the manufacturers’ recommendations in the conduct and interpretation of the GPP tests. Manufacturers provide clear package inserts for the handling of samples and equipment. Deviation in terms of type of sample (fresh, frozen, pure, in Cary–Blair medium), amount of sample and handling of sample if not tested immediately may result in systematic differences in the performance of the test.

Comparator domain

The addition of a domain for the comparator.

The eligible studies compared GPP testing with a comparator that consisted, broadly speaking, of a range of conventional microbiological tests that cannot be classed as the reference standard because GPP testing may be superior to conventional methods. Therefore, the comparator was assessed in addition to the index test and the reference standard. Similar signalling questions in terms of blinding and thresholds were considered for the comparator. In addition, one signalling question was added: was culture performed on fresh (not previously frozen) samples? Culture from frozen samples appear to be less reliable than culture from fresh samples, which will result in greater discrepancies between conventional and GPP methods for pathogens that are confirmed by culture. 11,12

Reference standard domain

The addition of one signalling question: was the reference standard independent and unbiased?

As there is no independent and reliable reference standard for the assessment, we looked for an independent and unbiased umpire test to be used in the studies to assess discordant results between index test and conventional methods [e.g. exposure, treatment effect (pathogen-specific treatment) or self-reported symptoms, previous antibiotic treatment, over-the-counter medicine].

Flow and timing domain

The adjustment of one signalling question: did all discordant samples receive a reference standard?

For a fair umpire test to be a valid verification method, all discordant results rather than a proportion should be tested.

The addition of one signalling question: did all samples receive the comparator methods for all pathogens considered in the study?

The risk of bias is high if patients received conventional tests only on the basis of prior assessments by physicians and according to symptoms.

Summary of strategies for analysis

Methods of analysis

In the absence of any test–treat trials, a suitable reference standard or any fair umpire tests in any included studies, we followed the FDA recommendations on reporting results from studies evaluating diagnostic tests. 17 In the absence of a reference standard, the guidance discourages the reporting of sensitivity and specificity and recommends reporting measures of positive and negative test agreement.

We calculated positive agreement (a/a + c) and negative agreement (d/b + d) when benchmarked against the comparator, and positive agreement (a/a + b) and negative agreement (d/c + d) when benchmarked against GPP for each pathogen to determine the range of feasible outcomes of test agreement.

The resulting measures of agreement are not measures of performance, as it is not known which test is correct. Agreement could be poor because one test is more accurate than the other or because they are both poor. There is no statistical method to determine which scenario is correct. As recommended in the guidance,17 we used the original 2 × 2 table data without updating results from discrepant analyses because, when attempted in a minority of studies, the resolving test used was not a reference standard.

Positive and negative agreement benchmarked against conventional methods and positive and negative agreement benchmarked against GPP were then meta-analysed by pathogen if the denominator was ≥ 20 to achieve agreement estimates and explore heterogeneity.

We undertook a random-effects meta-analysis of proportions using the metaprop command in Stata SE 14.1 (StataCorp LLC, College Station, TX, USA). 18 The analysis was exploratory and no adjustment was made for the interdependence of positive and negative values. Exact binomial methods were used to estimate 95% confidence intervals (CIs) using the Freeman–Tukey transformation of proportions, and the I²-statistic of between-study heterogeneity was computed.

The analysis was undertaken for each pathogen at the sample level, assuming independence between samples within studies and between studies, that is, having one pathogen does not affect the likelihood of having another pathogen.

Definition of the Public Health England algorithm

The PHE algorithm8 consists of primary testing for Salmonella, Campylobacter, Shigella, E. coli O157, Cryptosporidium and C. difficile with additional tests for children aged < 5 years for rotavirus, adenovirus 40/41 and norovirus. Secondary testing in hospitalised adults is recommended for norovirus and ova, cysts and parasites including Giardia, whereas people in the community should not be tested for norovirus and children aged < 5 years should also be tested for astro- and sapovirus. Furthermore, our assessment included travellers as a subgroup of interest for which additional tests for Vibrio and Plesiomonas species are considered if they are returning from areas other than Western Europe, North America, Australia or New Zealand. Physicians following this algorithm use local tests that vary across the UK and may or may not request all tests detailed in the PHE algorithm, depending on symptoms and suspicion following patient assessment. In order to compare GPP testing with microbiology techniques, as outlined in the PHE syndromic algorithm for routine testing, a clear definition of the comparator is needed.

For the purpose of this review, no differentiation is being made between primary and secondary testing or differences in terms of population testing (children, adults, travellers) and seasonality (including norovirus testing in winter months only). For a sufficiently equivalent comparator, all pathogens mentioned in the syndromic PHE pathway8 for any of the populations of interest were included. A pragmatic judgement was required regarding whether or not the number of tested pathogens in the studies is sufficiently similar to the PHE algorithm [i.e. at least 75% (≥ 11 of the above pathogens) included in study].

In terms of tests, the following was assumed: testing is classed as sufficiently equivalent to the PHE algorithm if the comparator in the studies includes:

• culture or PCR for bacteria (see final bullet point for C. difficile)

• PCR or enzyme immunoassay for viruses

• microscopy or enzyme immunoassay for parasites

• PCR and/or enzyme immunoassay (plus toxin test) for C. difficile.

It is noteworthy that even if the type of method is equivalent, there are a huge number of different tests and testing kits currently used within the NHS for the various pathogens. Therefore, we have included studies using a wide range of conventional microbiology approaches, reflecting variation in health services practice.

Studies identified after the review deadline

Three studies were identified after the deadline for inclusion of studies and were therefore not included in the review: Stockmann et al. ,45 Eibach et al. 46 and De Rauw et al. 47

Participants

The included patient population and the setting of studies were insufficiently characterised, possibly as result of the inclusion of samples rather than patients in the majority of studies. Only one study11 differentiated between hospital-based samples (from children admitted with suspected viral gastroenteritis and children presenting to the emergency room) and community-based samples [adult and child travellers whose samples were submitted by general practitioners (GPs)]; the study population consisted of children and travellers only. Seven studies19,22,26,27,29,35,39 reported the setting sufficiently to allow judgement on the origin of the infection (hospital vs. community). The majority of studies11,12,20,21,23–25,28,30–34,36–38 reported recruitment of hospitalised patients, for which a mixed population should be assumed (in terms of origin of infection) as it is unclear at what point in time during hospitalisation each infection occurred. Only one study was identified as a community study, set in Côte d’Ivoire. 19 Subgroups of interest were reported in 10 studies11,19,20,22,25,26,29,35,36,39 and, of these, one considered travellers only39 and two considered immunocompromised patients only. 22,26 Children aged < 5 years were not considered separately in any study, but the proportion of children aged < 5 years was reported in six studies11,19,20,25,35,36 and ranged from 2% to 40% of the total study population. These studies did not report outcomes by setting or subgroup at the pathogen level, which is why reported outcomes should be regarded as applicable to a mixed population. However, one study reported study-level results for hospital- and community-acquired infections. 27

Country of study

Eleven studies11,22,27–30,32,35,37–39 included participants from European countries, including three studies from the UK. 27,30,32 One study was multinational, covering North America and Europe,21 eight studies were from North America,12,20,25,26,31,33,34,36 two were from Asia23,24 and one was from Africa. 19 Therefore, the applicability of the study population in terms of prevalence of pathogens is questionable in 3 out of the 23 studies (see Quality considerations of included studies).

Index test

Overall, 17 studies11,19,21–25,27–33,37–39 evaluated xTAG and four studies20,34–36 evaluated FilmArray. Two studies12,26 compared both tests but no study was identified that assessed the Faecal Pathogens B assay. The majority of studies11,12,19,21,23,24,26–30,32,33,37–39 investigated xTAG for all 15 pathogens, of which two studies12,33 specified the research use only version (pre-FDA approval). However, two studies considered either 1131 (excluding adenovirus, Entamoeba histolytica, Vibrio cholerae and Yersinia) or 1425 pathogens (excluding Yersinia). Initial FDA approval of xTAG only covered 11 pathogens but was extended to 14 pathogens following an additional submission by Luminex requesting to extend reporting to a further three pathogens in 2014. 40 However, CE marking for xTAG covers all 15 pathogens. 48 Therefore, variation of the number of pathogens included in the studies exists. The assay methods are identical in the different versions. 40 Similarly, an investigational use only version of the FilmArray panel test included a test for Aeromonas. This pathogen is not included in the FDA-cleared version of the panel test. Therefore, the majority of FilmArray studies included 23 pathogens12,20,26,34,36 using the investigational use only version rather than the CE-marked 2235 pathogen panel.

Comparator

A methodological limitation of studies included in this review concerns the varying ways in which comparator tests are implemented between studies. Relatively few studies have used a mix of microbiological and PCR methods reflected in the PHE pathway, with an over-reliance on PCR methods and sequencing. A further difficulty is the inconsistent use of the full comparator method. In most studies the GPP system is compared with a subset of conventional pathogen tests rather than assessing the full PHE algorithm.

According to our definition of the PHE algorithm (see Definitions for the review), only six studies12,21,26,27,29,36 can be classed as having used a comparator sufficiently similar to the routine screening pathway recommended by PHE in terms of pathogens included (at least 11) and methods used. None of the studies tested patients or samples with all conventional tests. Pathogens were tested according to the physician’s request only. Two studies19,34 considered as few as two pathogens in common with the PHE algorithm, whereas two restricted conventional testing to only five pathogens28 and four pathogens. 30 Rand et al. 34 included samples negative for rotavirus and C. difficile only and Beckmann et al. 11 tested children only for viruses and travellers only for parasites with conventional methods. The two main clinical studies by the companies20,25 used a comparator that was significantly different (PCR and sequencing for the majority of pathogens) to typical conventional methods, whereas one study from the Netherlands38 also used methods different from the PHE methods, namely multiplexed PCR for parasites and viruses. Although multiplexing is emerging in UK laboratories for individual pathogens or types of pathogens, this is not specified in the PHE algorithm. 8 The assessment of the comparator in terms of equivalence to the algorithm was not possible for one multinational study35 in which conventional methods were described as routine tests undertaken at local laboratories in 10 different countries.

Outcomes

Ten studies20–27,29,30 reported results of association between the comparator and the GPP test in sufficient detail by pathogen to allow the construction of 2 × 2 tables by pathogen. These studies are further characterised in Table 7 and contributed data to the meta-analysis in Pathogen-level positive and negative agreement.

An attempt to verify at least some of the discrepant results between GPP assay and conventional methods was undertaken in 15 studies. 11,12,20–23,25,29–34,38,39 However, only four studies reported outcomes of verification by pathogen. 20,23,25,30

Study design

Table 7 characterises the design heterogeneity among the studies reported, which should be noted when considering the pooled study outcomes. Studies retrospectively included samples with confirmed gastroenteritis and negative controls30 or prospectively included samples from patients with suspected gastroenteritis. 20–27,29 Comparator and GPP tests were both undertaken at the study site(s) in seven studies,22–27,29 whereas one study sent the samples to reference laboratories for conventional testing,30 one study sent the samples to Luminex for GPP testing21 and one study sent the samples to BioFire for comparator testing. 20 Verification was undertaken on all discrepant samples by four studies,20,23,25,30 on a subset of samples by Claas et al. ,21 on positive samples for enteric viruses and Campylobacter as well as 23 negative samples by Coste et al. ,22 and on samples with discrepant adenovirus results only by Mengelle et al. 29 The remaining three studies,24,26,27 which did not undertake verification, assumed either that the comparator methods are the reference standard to calculate sensitivity and specificity24,26 or assumed that the GPP assay is correct and did not calculate sensitivity and specificity because of the lack of a resolving assay. 27 The verification results are summarised in Analysis of discordant results.

Comparator assays varied considerably between studies. Testing included different pathogens and different assays according to the routine diagnostic algorithms in place at the study sites. Pankhurst et al. 30 only included the most common four pathogens for conventional testing by selecting fixed numbers of samples positive for C. difficile, Campylobacter, Salmonella and norovirus. The xTAG GPP Luminex trial25 did not evaluate test samples using traditional conventional methods within routine laboratory analysis, instead testing all samples at a reference laboratory; the FilmArray BioFire trial20 only considered conventional culture results as undertaken by a routine laboratory while the presence of other pathogens was tested at BioFire with various methods and a combination of methods. Gu et al. 26 added multiplex PCR testing to the in-house methods for four viruses that were not routinely tested for, creating a comparator of in-house methods consisting of conventional and multiplex PCR assays. In three studies21,27,29 participants received conventional testing according to the physician’s request not to provide results for all pathogens, for all included patients. Only Deng et al. 23 and Coste et al. 22 tested all patients with conventional methods, irrespective of whether or not they were requested by the physician. Two studies tested all samples with comparator methods, but these were not equivalent to conventional methods of the PHE algorithm. 20,25 In three studies,22,24,30 samples had received all conventional methods, but the studies only looked at a limited number of pathogens, and Gu et al. 26 constructed a comparator for which conventional methods were undertaken according to the physician’s judgement, but additional testing for astrovirus, norovirus and sapovirus was undertaken for all samples.

Use of GPP testing showed less variation across studies; 2 × 2 data were available from nine studies21–27,29,30 for xTAG and from two studies20,26 for FilmArray. Of interest is the batching method reported by Halligan et al. ,27 who batched samples for DNA extraction at 16.00 Monday–Thursday and subsequent analysis of samples the following morning with results available at 15.00. Alternative runs took place on Friday at 10.00 for late evening or Saturday reporting. This is the only study reporting details of batching of samples.

The majority of studies21–26,30 reported sensitivity and specificity using either GPP or conventional tests as a reference standard; these estimates were not used within this review, which instead used the 2 × 2 table data from the studies to estimate the positive and negative agreement between the GPP and conventional tests.

Assessment of risk of bias and applicability of study outcomes to the research question

The assessment of risk of bias and applicability for the 23 studies included in the review using the QUADAS-2 tool are summarised in Table 10 and Figure 4.

FIGURE 4.

Concerns regarding (a) bias; and (b) applicability in included studies. NA, not applicable.

Pathogen-level positive and negative agreement

Pathogen-level meta-analysis of positive and negative agreement was undertaken. The 2 × 2 data of test agreement, by pathogen, for the studies eligible for inclusion in the meta-analysis are reported in Appendix 7. The meta-analytic outcomes reported in Agreement between xTAG and the comparator and Agreement between FilmArray and conventional methods are of an exploratory nature, summarising the available evidence and illustrating patterns in the data as well as heterogeneity. When considering the results, the following caveats apply.

• The results do not reflect the performance of the tests, but the agreement between the GPP test and the comparator.

• These findings are typically heterogeneous, probably reflecting both methodological and statistical heterogeneity and drawing from studies of variable quality.

• Analyses are presented at the pathogen level, requiring independence assumptions both within and between pathogens (i.e. repeat samples of the same patient are not included and having one pathogen does not affect the likelihood of having another pathogen).

• The comparator used in studies does not sufficiently align with the conventional methods described in the PHE algorithm8 in the majority of studies, as described in Definitions for the review.

• Pooled summary estimates (across pathogens) have been included for information; however, these pooled estimates are not weighted by the prevalence of the different pathogens and include varying multiple usage of samples for which studies have tested samples with varying components of the conventional panel of tests, thus violating the independence assumption.

Agreement between xTAG and the comparator

Table 11 reports the positive (a/a + c) and negative (d/b + d) agreement between xTAG and conventional methods when conventional methods are the benchmark. When conventional testing provides the benchmark, virtually all positive cases found by xTAG are confirmed by conventional testing, leading to generally high levels of positive agreement findings. There are few additional positives identified by xTAG compared with the vast majority of specimens that are pathogen negative; thus, negative agreement remains high.

TABLE 11 - Positive and negative agreement: xTAG vs. conventional testing (benchmark)

ETEC, enterotoxigenic E. coli; LCI, lower confidence interval; RE, random-effects estimate, measure of agreement; STEC, shiga toxin-producing E. coli; UCI, upper confidence interval.

Reproduced from Freeman et al. 2017. 49 © 2017 Freeman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

	RE	LCI	UCI	Number of studies contributing	Q	p-value	I 2
Positive agreement
C. difficile	0.959	0.933	0.980	5	5.9	0.207	32%
Campylobacter	0.959	0.924	0.985	6	8.0	0.157	37%
E. coli O157	–	–	–	–	–	–	–
ETEC	–	–	–	–	–	–	–
STEC	–	–	–	–	–	–	–
Salmonella	0.818	0.666	0.934	5	30.8	0.000	87%
Shigella	0.989	0.949	1.000	3	3.6	0.164	45%
V. cholerae	–	–	–	–	–	–	–
Yersinia enterocolitica	–	–	–	–	–	–	–
Adenovirus	0.558	0.413	0.699	–	–	–	–
Norovirus	0.927	0.893	0.956	7	10.9	0.093	45%
Rotavirus	0.958	0.920	0.985	3	2.9	0.240	30%
Cryptosporidium	0.914	0.794	0.989	1	–	–	–
E. histolytica	–	–	–	–	–	–	–
Giardia	1.000	0.935	1.000	1	–	–	–
Negative agreement
C. difficile	0.968	0.933	0.991	7	128.0	0.000	95%
Campylobacter	0.968	0.950	0.982	10	83.7	0.000	89%
E. coli O157	0.995	0.990	0.998	6	10.8	0.055	54%
ETEC	0.988	0.964	1.000	4	23.7	0.000	87%
STEC	0.990	0.984	0.995	4	2.8	0.418	0%
Salmonella	0.940	0.866	0.986	10	726.0	0.000	99%
Shigella	0.985	0.965	0.997	8	120.0	0.000	94%
V. cholerae	1.000	0.998	1.000	4	0.1	0.988	0%
Y. enterocolitica	1.000	1.000	1.000	4	0.4	0.933	0%
Adenovirus	0.990	0.983	0.996	2	–	–	–
Norovirus	0.969	0.944	0.987	12	239.0	0.000	95%
Rotavirus	0.991	0.979	0.999	8	36.7	0.000	81%
Cryptosporidium	0.989	0.954	1.000	5	77.4	0.000	95%
E. histolytica	0.991	0.979	0.998	5	20.5	0.000	81%
Giardia	0.989	0.970	0.999	7	46.4	0.000	87%
Overall agreement
Positive	0.929	0.898	0.955	33	188.3	0.000	83%
Negative	0.982	0.976	0.988	101	2080.8	0.000	95%

Overall, more studies contributed to the calculation of negative than positive agreement as only studies with sufficient numbers (denominator ≥ 20) were included in the analysis. For a number of pathogens, E. coli O157, enterotoxigenic E. coli, shiga toxin-producing E. coli, V. cholerae, Yersinia enterocolitica and E. histolytica (rare pathogens or no test requested by physician and marked as empty rows in Tables 6 and 7), limited data were available and, therefore, no positive agreement was estimated. Overall, agreement is high (positive agreement 0.929 and negative agreement 0.982), with little variation between pathogens for both positive and negative agreement. Positive agreement for adenovirus was an exception, as positive agreement was considerably lower at 0.558. This is visualised in the forest plot in Figure 5. Gu et al. 26 reported that an additional 20 samples positive for adenovirus detected by comparator were a result of the use of multiplex PCR, which detected all serotypes, whereas xTAG only detected adenovirus 40/41, resulting in the poor agreement of tests for this virus. However, it is important to note that for enteric infections, adenovirus 40/41 is commonly implicated. In situations when a patient may have a systemic infection affecting, for example, the respiratory or urinary tracts, then all adenovirus serotypes should be looked for by multiplex PCR. Therefore, it is difficult to come to any firm conclusions about the positive agreement because of the different parameters (type 40/41 vs. multiplex) that were assessed. The outlying finding for Salmonella (Pankhurst et al. 30), which is caused by a worryingly high number of missed Salmonella infections by xTAG, is not as easily interpreted. Generally, both tests agreed in terms of absence of pathogens, masking the relatively small number of disagreements. A forest plot in Appendix 8 (see Figure 43) shows, however, that there are a few outliers when studies report a significantly higher number of positives for certain pathogens with xTAG than conventional methods, specifically Campylobacter and norovirus in a small study of 49 adult kidney transplant recipients22 and Salmonella in a study in which bacteria were tested by PCR as well as culture. 24

FIGURE 5.

Positive agreement: xTAG vs. conventional testing (benchmark). G2, genogroup 2; RE, random-effects estimate. Reproduced from Freeman et al. 2017. 49 © 2017 Freeman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Used as a measure of overall heterogeneity of estimates, I² is moderate (for positive agreement) to high (for negative agreement) at the pathogen level.

Table 12 shows the positive (a/a + b) and negative (d/c + d) agreement when xTAG is considered the benchmark. Strikingly, the positive agreement between xTAG and conventional methods is considerably smaller when xTAG provides the benchmark.

TABLE 12 - Positive and negative agreement: conventional testing vs. xTAG (benchmark)

ETEC, enterotoxigenic E. coli; LCI, lower confidence interval; RE, random-effects estimate, measure of agreement; STEC, shiga toxin-producing E. coli; UCI, upper confidence interval.

Reproduced from Freeman et al. 2017. 49 © 2017 Freeman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

	RE	LCI	UCI	Number of studies contributing	Q	p-value	I 2
Positive agreement
C. difficile	0.801	0.594	0.948	5	124.0	0.000	97%
Campylobacter	0.639	0.398	0.849	7	167.0	0.000	96%
E. coli O157	0.750	0.534	0.920	1	–	–	–
ETEC	–	–	–	–	–	–	–
STEC	–	–	–	–	–	–	–
Salmonella	0.484	0.278	0.693	8	173.0	0.000	96%
Shigella	0.734	0.381	0.971	3	61.6	0.000	97%
V. cholerae	–	–	–	–	–	–	–
Y. enterocolitica	–	–	–	–	–	–	–
Adenovirus	0.570	0.425	0.710	2	–	–	–
Norovirus	0.774	0.584	0.920	8	215.0	0.000	97%
Rotavirus	0.924	0.853	0.975	3	6.5	0.039	69%
Cryptosporidium	0.508	0.407	0.608	2	–	–	–
E. histolytica	–	–	–	–	–	–	–
Giardia	0.337	0.237	0.444	2	–	–	–
Negative agreement
C. difficile	0.996	0.992	0.999	7	11.1	0.084	46%
Campylobacter	0.998	0.994	1.000	10	37.4	0.000	76%
E. coli O157	1.000	1.000	1.000	6	3.4	0.637	0%
ETEC	0.999	0.996	1.000	4	3.0	0.393	0%
STEC	1.000	0.998	1.000	4	4.9	0.182	38%
Salmonella	0.992	0.980	0.999	10	94.4	0.000	91%
Shigella	1.000	0.999	1.000	8	12.0	0.099	42%
V. cholerae	1.000	0.999	1.000	4	3.5	0.326	13%
Y. enterocolitica	1.000	0.999	1.000	4	0.4	0.933	0%
Astrovirus	0.989	0.971	0.999	7	68.7	0.000	91%
Norovirus	0.995	0.990	0.998	12	34.9	0.000	69%
Rotavirus	0.998	0.992	1.000	8	29.3	0.000	76%
Cryptosporidium	1.000	0.999	1.000	5	2.0	0.743	0%
E. histolytica	1.000	0.999	1.000	5	3.5	0.481	0%
Giardia	1.000	1.000	1.000	7	1.8	0.941	0%
Overall agreement
Positive	0.678	0.580	0.770	41	1340.5	0.000	97%
Negative	0.998	0.997	0.999	101	429.2	0.000	77%

The forest plot in Figure 6 visualises the inconsistency and variation in positive agreement between studies across all pathogens. When xTAG is the benchmark, the positive cases ‘missed’ by conventional testing have a considerable impact of the positive agreement findings.

FIGURE 6.

Positive agreement: conventional testing vs. xTAG (benchmark). G1, genogroup 1; G2, genogroup 2; RE, random-effects estimate. Reproduced from Freeman et al. 2017. 49 © 2017 Freeman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

As the positive agreement overall is 0.678 (95% CI 0.580 to 0.770), inverting these figures means that xTAG finds about 1.5 times more pathology (95% CI 1.3 to 1.7), although this estimate does not reflect the prevalence of individual pathogens. Negative agreement was consistently very high across studies and pathogens (see Appendix 8, Figure 44) with the exception of Gu et al.,26 discussed previously. Heterogeneity is moderate to high when considering I², and is higher for positive agreement than for negative agreement.

Agreement between FilmArray and conventional methods

Only two studies contributed to the evaluation of FilmArray; therefore, the agreement between FilmArray and conventional methods is based on one or two studies across the range of pathogens. Table 13 reports the positive (a/a + b) and negative (d/c + d) agreement between FilmArray and conventional methods when benchmarked against conventional methods. Positive agreement is very high for all pathogens, with the same study (Gu et al. 26) contributing an outlier value for adenovirus for FilmArray, as was the case for xTAG (Figure 7). Negative agreement is consistently high for all studies across all pathogens (see Appendix 8, Figure 45).

TABLE 13 - Positive and negative agreement: FilmArray vs. conventional testing (benchmark)

EAEC, enteroaggregative E. coli; EPEC, enteropathogenic E. coli; ETEC, enterotoxigenic E. coli; LCI, lower confidence interval; RE, random-effects estimate, measure of agreement; STEC, shiga toxin-producing E. coli; UCI, upper confidence interval.

	RE	LCI	UCI	Number of studies contributing	Q	p-value	I 2
Positive agreement
C. difficile	0.967	0.937	0.989	2	–	–	–
Campylobacter	0.971	0.881	1.000	1	–	–	–
E. coli O157	–	–	–	–	–	–	–
EAEC	0.988	0.949	1.000	1	–	–	–
EPEC	0.991	0.976	0.999	1	–	–	–
ETEC	1.000	0.923	1.000	1	–	–	–
Plesiomonas shigelloides	–	–	–	–	–	–	–
STEC	1.000	0.949	1.000	1	–	–	–
Salmonella	1.000	0.945	1.000	1	–	–	–
Shigella	0.959	0.881	0.999	1	–	–	–
Vibrio (parahaemolyticus, vulnificus and cholerae)	–	–	–	–	–	–	–
V. cholerae	–	–	–	–	–	–	–
Y. enterocolitica	–	–	–	–	–	–	–
Adenovirus	0.722	0.606	0.826	2	–	–	–
Astrovirus	–	–	–	–	–	–	–
Norovirus	0.932	0.866	0.979	2	–	–	–
Rotavirus	–	–	–	–	–	–	–
Sapovirus	1.000	0.963	1.000	1	–	–	–
Cyclospora cayetanensis	–	–	–	–	–	–	–
Cryptosporidium	–	–	–	–	–	–	–
E. histolytica	–	–	–	–	–	–	–
Giardia	1.000	0.916	1.000	1	–	–	–
Negative agreement
C. difficile	0.971	0.962	0.979	2	–	–	–
Campylobacter	0.987	0.981	0.993	2	–	–	–
E. coli O157	0.968	0.930	0.993	2	–	–	–
EAEC	0.982	0.974	0.988	1	–	–	–
EPEC	0.972	0.961	0.980	1	–	–	–
ETEC	0.994	0.990	0.997	1	–	–	–
P. shigelloides	0.990	0.985	0.995	1	–	–	–
STEC	0.997	0.993	0.999	1	–	–	–
Salmonella	0.998	0.994	1.000	2	–	–	–
Shigella	1.000	0.998	1.000	2	–	–	–
Vibrio (parahaemolyticus, vulnificus and cholerae)	0.999	0.997	1.000	1	–	–	–
V. cholerae	1.000	0.997	1.000	2	–	–	–
Y. enterocolitica	1.000	1.000	1.000	2	–	–	–
Adenovirus	0.993	0.987	0.997	2	–	–	–
Astrovirus	1.000	0.998	1.000	2	–	–	–
Norovirus	0.991	0.985	0.995	2	–	–	–
Rotavirus	0.995	0.990	0.998	2	–	–	–
Sapovirus	0.994	0.989	0.997	2	–	–	–
C. cayetanensis	1.000	0.999	1.000	1	–	–	–
Cryptosporidium	1.000	0.997	1.000	2	–	–	–
E. histolytica	1.000	0.999	1.000	1	–	–	–
Giardia	1.000	0.997	1.000	2	–	–	–
Overall agreement
Positive	0.954	0.897	0.991	14	129.4	0.000	89%
Negative	0.996	0.993	0.998	35	295.6	0.000	88%

FIGURE 7.

Positive agreement: FilmArray vs. conventional testing (benchmark). EAEC, enteroaggregative E. coli; EPEC, enteropathogenic E. coli; ETEC, enterotoxigenic E. coli; RE, random-effects estimate; STEC, shiga toxin-producing E. coli.

When benchmarked against FilmArray, positive agreement between FilmArray and conventional methods is lower (0.82 vs. 0.95) (Table 14) with greater variability across pathogens (Figure 8), whereas negative agreement (see Appendix 8, Figure 46) is consistently high, with the exception of Gu et al. 26 once again. Heterogeneity is not reported, as only one or two studies contributed data per pathogen.

TABLE 14 - Positive and negative agreement: conventional testing vs. FilmArray (benchmark)

EAEC, enteroaggregative E. coli; EPEC, enteropathogenic E. coli; ETEC, enterotoxigenic E. coli; LCI, lower confidence interval; RE, random-effects estimate, measure of agreement; STEC, shiga toxin-producing E. coli; UCI, upper confidence interval.

	RE	LCI	UCI	Number of studies contributing	Q	p-value	I 2
Positive agreement
C. difficile	0.817	0.766	0.864	2	–	–	–
Campylobacter	0.586	0.456	0.710	1	–	–	–
E. coli O157	–	–	–	–	–	–	–
EAEC	0.752	0.667	0.829	1	–	–	–
EPEC	0.902	0.869	0.931	1	–	–	–
ETEC	0.710	0.536	0.858	1	–	–	–
Plesiomonas shigelloides	–	–	–	–	–	–	–
STEC	0.868	0.739	0.961	1	–	–	–
Salmonella	0.838	0.699	0.942	1	–	–	–
Shigella	0.959	0.881	0.999	1	–	–	–
Vibrio (parahaemolyticus, vulnificus and cholerae)	–	–	–	–	–	–	–
V. cholerae	–	–	–	–	–	–	–
Y. enterocolitica	–	–	–	–	–	–	–
Adenovirus	0.764	0.641	0.868	1	–	–	–
Astrovirus	–	–	–	–	–	–	–
Norovirus	0.851	0.771	0.917	2	–	–	–
Rotavirus	–	–	–	–	–	–	–
Sapovirus	0.780	0.664	0.877	1	–	–	–
Cyclospora cayetanensis	–	–	–	–	–	–	–
Cryptosporidium	0.750	0.555	0.906	1	–	–	–
E. histolytica	–	–	–	–	–	–	–
Giardia	0.741	0.557	0.891	1	–	–	–
Negative agreement
C. difficile	0.998	0.994	1.000	2	–	–	–
Campylobacter	1.000	0.999	1.000	2	–	–	–
E. coli O157	1.000	0.989	1.000	2	–	–	–
EAEC	0.999	0.997	1.000	1	–	–	–
EPEC	0.997	0.994	1.000	1	–	–	–
ETEC	1.000	0.999	1.000	1	–	–	–
P. shigelloides	1.000	0.999	1.000	1	–	–	–
STEC	1.000	0.999	1.000	1	–	–	–
Salmonella	1.000	1.000	1.000	2	–	–	–
Shigella	1.000	0.998	1.000	2	–	–	–
Vibrio (parahaemolyticus, vulnificus and cholerae)	1.000	0.999	1.000	1	–	–	–
V. cholerae	1.000	1.000	1.000	2	–	–	–
Y. enterocolitica	1.000	1.000	1.000	2	–	–	–
Adenovirus	0.997	0.993	0.999	2	–	–	–
Astrovirus	1.000	0.998	1.000	2	–	–	–
Norovirus	0.998	0.995	1.000	2	–	–	–
Rotavirus	1.000	1.000	1.000	2	–	–	–
Sapovirus	1.000	0.999	1.000	2	–	–	–
C. cayetanensis	1.000	0.999	1.000	1	–	–	–
Cryptosporidium	1.000	1.000	1.000	2	–	–	–
E. histolytica	1.000	0.999	1.000	1	–	–	–
Giardia	1.000	1.000	1.000	2	–	–	–
Overall agreement
Positive	0.820	0.761	0.872	15	72.3	0.000	81%
Negative	1.000	0.999	1.000	36	171.5	0.000	80%

FIGURE 8.

Positive agreement: conventional testing vs. FilmArray (benchmark). EAEC, enteroaggregative E. coli; EPEC, enteropathogenic E. coli; ETEC, enterotoxigenic E. coli; RE, random-effects estimate; STEC, shiga toxin-producing E. coli.

As positive agreement overall is 0.820 (95% CI 0.761 to 0.872), inverting these figures means that FilmArray finds about 1.2 times more pathology than conventional testing (95% CI 1.1 to 1.3), although this estimate does not reflect the prevalence of individual pathogens.

Aetiology of additional positive findings

Studies typically reported overall levels of detection of pathogens in addition to providing pathogen-level analysis (several studies only provided this overall analysis). Additional pathogens could arise in part because of a greater ‘sensitivity’ to detect specific pathogens, but also because of a greater coverage of pathogens within the GPP system than in the conventional method. Studies were reviewed to ascertain the extent of these two sources of additional positive findings, because simply reporting the total number of pathogens detected may mislead by confounding these issues.

Table 15 summarises the ascertainment of positive results. Studies reported this issue in a varying manner, but the agreement findings illustrate that GPP testing produces a greater number of pathogen-positive findings than conventional testing, and the extent to which greater overall pathology arises from the number of tests within the GPP system or a higher level of pathogen detection is inconclusive.

Even in the studies in which the comparator was largely PCR based, GPP methods identified more pathogens. 25 Only three studies reported more additional samples with conventional testing than with GPP. 11,26,38 Gu et al. 26 reported that 20 of the additional samples with conventional testing were positive for adenovirus by multiplex PCR, which detected all serotypes, whereas GPP assays detected only adenovirus 40/41. Beckmann et al. 11 detected parasites in travellers using microscopy and detected additional parasites that were not available on the GPP test. Similarly, in Wessels et al. ,38 51 out of 55 additional positives by conventional methods were pathogens not present on the panel test.

Eight studies characterised the additional GPP positives by judging whether they were primarily a result of greater ‘sensitivity’ or greater coverage. 11,21,22,28,29,32,34,38 Proportions varied across studies and clearly depended on how well the conventional methods and GPP test agreed in terms of number and type of pathogens tested for in individual samples.

Using two methods for the comparator, Duong et al. 24 reported considerably more additional positives when GPP was compared with culture rather than PCR. Conducted in Vietnam, this study reported a coinfection rate of 58%, much higher than other studies (see Frequency and characterisation of multiple infections), resulting in a number of additional positives.

There is some suggestion from the meta-analyses that additional positive case findings (above conventional testing) may be greater with xTAG and FilmArray. However, this suggestion does not arise from a direct comparison: only one or two studies contribute to evidence for FilmArray, there is considerable heterogeneity between studies and the methods of pooling across pathogens do not reflect pathogen prevalence. Without validation with a reliable reference standard, it remains unknown whether additional positives are true or false positives; however, there is concern that at least some of the additional positives by panel testing are false positives because they identify non-viable pathogens.

Analysis of discordant results

In the absence of a reference standard to verify test results, a small number of studies attempted to verify samples that did not agree when tested with GPP and conventional methods (Table 16). In four studies,20,23,25,30 discordant samples were verified and results reported by pathogen, whereas one additional study only reported that verification produced similar results to GPP testing. 29 Verification methods were PCR based. Buss et al. 20 aimed to analyse discordant samples using PCR and sequencing assays that targeted DNA sequences different from both the FilmArray panel and the comparator method. Alternatively, if no distinct PCR assay was available, discordant samples were retested by the original molecular tests using enhanced methods (including up to 10 additional PCR cycles and up to 10 additional replicate samples) followed by sequencing. If still discrepant, samples were tested with a bench-top version of the FilmArray panel followed by sequencing. Deng et al. 23 sourced primers from the published literature for individual pathogens for single-plex PCR and followed up results by sequencing. Similarly, the xTAG study in the FDA report25 used validated primers that targeted genomic regions distinct from the xTAG GPP and followed up with bidirectional sequencing analysis to confirm pathogen identification. Mengelle et al. 29 used in-house real-time PCR to verify discordant results, and Pankhurst et al. 30 used single quantitative PCR for which the primers were probably different from xTAG.

Even though the verification methods differed from conventional methods and used different molecular targets to GPP assays, they are not independent of the PCR methods used for either conventional methods or GPP assays, and they would be expected to resolve discordant results in favour of PCR-based methods, such as the GPP assays. Therefore, they are biased towards GPP testing. The findings are described briefly and are not considered further in the assessment of the GPP assays or economic modelling. Discordant analysis of GPP positive/conventional testing negative samples was generally in favour of GPP as anticipated; however, analysis of GPP negative/conventional positive outcomes more often favoured conventional testing (see Table 11). Discordant analysis failed to resolve all discordant samples and a considerable number of discordant results remained inconclusive following testing.

No particular pattern for any specific pathogen emerges from the discordant analysis. Finally, no study attempted independent verification of samples found to be pathogen positive by both methods.

Run failures/invalid test results

Gastrointestinal pathogen panel run failure results were reported in nine studies (Table 17). 20,25,26,29,31,33,35,38,39 In six studies,20,26,29,33,35,38 the run failure rate ranged from 0.8%20 to 7.8%. 29 Some of the reasons for these failures were ‘no call’, software error, loss of vacuum pressure in pouches or failed internal control. In one study,29 there was a repeated pattern of 34 failures (7.8%) for each pathogen, suggesting that these failures were test related rather than pathogen related. Likewise, reported failure rates were similar for each pathogen in the FDA report. 25 GPP failures appear much greater than for conventional testing when considering failures reported in the FDA report, which recorded failures only for C. difficile with conventional testing. Two other studies reported GPP run failure rates of ≥ 14%,25,31 and no explanations or reasons for these failures were provided. Only one study using two GPP tests (xTAG and FilmArray) reported comparative failure rates, with FilmArray producing no run failures whereas xTAG returned about 5%. 26

Test failures for conventional tests were reported for one study only,25 in which, for C. difficile toxin, the test failure rate was 6.8% (95/1407), whereas for the remaining pathogens no failure was recorded (Table 18).

Frequency and characterisation of multiple infections

Table 19 summarises multiple infections as a proportion of positive GPP samples in the included studies. Multiple infections observed in the studies varied from 4%28,32 to 58%. 24 The studies with the greatest reported rate of multiple infections were from Vietnam (58%)24 and Côte d’Ivoire (54%). 19 Although the majority of multiple infections were double infections, Becker et al. 19 identified three samples with 10 pathogens, and the FDA study25 reported one sample with seven pathogens. Spina et al. 35 investigated stool samples from in- and outpatients and concluded that multiple infections were mainly identified in children aged < 5 years and outpatients. The three most frequent pathogens involved in multiple infections varied between studies, with no clear pattern emerging.

Multiple infections in positive samples identified by the comparator were less frequently reported and the number of pathogens detected was generally lower. Spina et al. 35 reported that 5.4% (7/128) of positive samples were positive for two pathogens using conventional methods, Stockmann et al. 36 reported that 3.4% (6/175) of positive samples were positive for multiple infections with conventional methods and Gu et al. 26 reported that 2% (1/47) of samples tested positive for two pathogens using conventional methods. Deng et al. 23 reported a considerably higher rate of 16% (29/178) of multiple infections with conventional methods.

Patient isolation

Patient isolation as an outcome was reported in two studies (Table 20). 27,34 In the study by Halligan et al. ,27 a higher proportion of community-acquired inpatient cases versus hospital-acquired inpatient cases were isolated (69.0% vs. 52.1%), but subsequently removed from isolation (60.1% vs. 41.6%; p < 0.01). The same study found a tendency for the median number of days spent in isolation to be higher for patients with hospital-acquired infection (range 0–13.5 days) than for those with community-acquired infection (range 1–4 days). Rand et al. 34 reported that isolation could have been averted in the 25 patients (out of the 102) who tested negative on GPP (24.6%). The median isolation time for these patients was 3.8 days (interquartile range 1.8–11.0 days).

Turnaround times

The turnaround times for GPP testing were reported in seven studies (Table 21),12,20,23,26–28,35 of which two compared timings with conventional methods. 27,28 In two studies12,26 the results tended to demonstrate a longer turnaround time for xTAG than for FilmArray. However, turnaround times may apply to different throughputs, with FilmArray able to test up to eight samples (using eight units) at the same time and xTAG able to run up to 96 samples. One study found that median turnover time was significantly shorter for xTAG than for conventional methods (1 day vs. 3 days; p < 0.0005). 28 One study reported a wide variability in turnaround test times across different pathogens when using conventional methods. 27 This study reported separately the clinical and laboratory turnaround times for GPP testing, with clinical turnaround times being much longer (41.8 hours vs. 26.6 hours), but suggested that xTAG pragmatically may provide a next-day service with test outcomes, only available the following day, having allowed for batching. One study only reported time taken using conventional methods, reflecting the common experience of conventional testing taking up to 3 days to result and report. 37

Costs

Three studies reported test-related costs (Table 22). 12,27,31 GPP testing was reported to be more costly than conventional tests (£150,641 for 2187 samples vs. £63,431 for 4467 samples). 27,31 Among GPPs, FilmArray was reported to be more costly than xTAG (£22,910 vs. £21,460 list price per instrument). 12 These costs reflect different fixed and variable cost assumptions and are reported for information only.

Other outcomes

Other outcomes (e.g. clinical management change, user acceptance, length of stay, technical hands-on time) were reported for four studies and are summarised in Table 23. 22,27,31,34

Interpretation of findings

In the absence of an adequate reference standard or alternative methods of assessing test accuracy, positive and negative agreements have been produced against a benchmark, as recommended by FDA guidance. If conventional methods are an accurate determinant of important disease, then meta-analytic results (benchmarked against conventional testing) suggest that GPP testing is reliable and could replace current microbiological methods, although there would be an increase in false-positive reporting and potential overtreatment. However, if GPP testing is considered accurate (in the sense that current testing practices are missing clinically important pathology), then it would identify missed infections and could potentially result in more appropriate treatment. Currently, the clinical importance of the additional pathogens identified by GPP testing is uncertain, apart from, for example, enteroaggregative E. coli, which is not detected by routine culture.

A further uncertainty is the value of more rapid testing, achieved to a varying extent by each GPP system. In the vast majority of cases, hydration, hygiene and watchful waiting are required. Only for a select few pathogens is specific treatment recommended [C. difficile, some strains of Salmonella, Shigella, E. coli (non-shiga toxin-producing E. coli), Campylobacter and Giardia], although not all patients are treated (see Chapter 5, Treatment). If conventional methods do accurately determine important disease, then the introduction of GPP testing could potentially result in some earlier overtreatment of unnecessary cases, which might spontaneously resolve.

Agreement measures are not measures of test performance in a conventional sense, and only adequately designed research will resolve uncertainties about the introduction of GPP testing. It may not be possible to design a study with an adequate independent reference standard. Molecular methods may not be the best option to address the problem, as the presence of pathogen DNA does not necessarily answer the question of the clinical importance of the identified pathogens. A randomised controlled trial may be the best option, with patients randomised to conventional and GPP testing, and clinical effectiveness and cost-effectiveness outcomes used to determine the value of GPP testing. Currently, there is no adequate evidence of the impact of GPP testing on the management, treatment and outcome of patients as no test-and-treat trials have been conducted.

The NICE scope identified the PHE algorithm8 as the relevant comparator against which to consider GPP tests. However, the PHE algorithm was published in 2013 and may require updating to reflect progress in routine microbiological laboratories. For instance, the new PHE guidance for the interpretation of PCR assays for gastrointestinal pathogens states that a growing number of PHE and NHS virology laboratories use PCR assays for detection of viral gastrointestinal pathogens. 50 The same guidance recommends PCR as a first-line diagnostic test for the intestinal parasites Giardia lamblia, Cryptosporidium spp. and E. histolytica. This is not reflected in the national standard of practice used in this review. Similarly, this review has evaluated GPP systems according to their current specification, but it is anticipated that the coverage of these systems will continue to evolve in response to changing pathogen prevalence, hence the evaluation problem is a dynamic one.

Current evidence should be regarded as applicable to a mixed population of community and hospitalised patients, as the studies do not adequately report outcomes by setting or subgroup. Only one study from Switzerland11 reported receiving samples both from GP practices and hospitals (emergency room and admitted patients). However, the GP samples were from travellers returning from the tropics only and the hospital samples were paediatric samples with suspected viral gastroenteritis; the study sample may have low applicability to a routine clinical setting. One study27 reported that community-acquired infections show a much greater spectrum of pathogens present than patients with hospital-acquired infections. Thus, a goal for a future trial might include adequate stratification of different populations (e.g. community managed, community acquired and hospital managed, hospital acquired, children, travellers and immunocompromised patients).

A paucity of head-to-head comparative data of different GPP tests limits conclusions. Khare et al. 12 directly compared FilmArray with xTAG, and concluded that both assays identified more pathogens than conventional methods, but may not perform uniformly well against different pathogens. Gu et al. 26 reported that both assays detected similar numbers of pathogens, with differences between the two tests not reaching significance. Failure rates may vary between GPP systems, and further research is needed to inform this issue.

Discrepancies between GPP tests and conventional methods may result from differences not only in accuracy but also in their targets. For example, discrepancies reported by Gu et al. 26 for adenovirus may have been a result of the comparator PCR identifying all adenovirus strains, whereas the GPP system identified only adenovirus 40 and 41. Similarly, Pankhurst et al. 30 reported that xTAG has two targets for C. difficile (genes for toxins A and B) and includes two primers against norovirus GI and GII strains. However, the single quantitative PCRs used as comparators in the study targeted only the gene for toxin B or the GII strain. They classed a positive result on either target with GPP as indicating the presence of the organism. This misalignment problem may be exacerbated when comparing different GPP systems with their varying coverage.

Implications for health economic model

Uncertainties about how to interpret available evidence for the performance of GPP systems and the impact they may have on clinical care make any attempt to model the economic consequences of introducing GPP systems tentative. Consequently, an economic model has been developed to explore these uncertainties. In the model, conventional testing using the PHE algorithm is considered to be accurate, providing the benchmark for comparison. The consequence of replacing conventional testing with GPP systems is explored. Different populations and a range of assumptions are explored regarding the impact of adopting GPP systems on clinical care. These assumptions have been variously informed by the direct evidence, epidemiological sources and expert opinion.

Search strategy

A comprehensive review of the published literature including economic evaluations, economic models, cost studies and quality-of-life studies was performed. The systematic search included the following electronic databases on 29 January 2016:

• Ovid MEDLINE – 1946 to January week 3 2016

• Ovid MEDLINE In-Process & Other Non-Indexed Citations: 27 January 2016

• EMBASE: 1974 to January 27 2016

• Web of Science Core Collection.

The search terms included economic and quality-of-life terms combined with gastroenteritis and the different GPP tests. The search was limited to studies published in the English language and studies involving humans. The search strategy was developed, based on the clinical effectiveness review and with health economist input. Details of the search strategies are provided in Appendix 1.

Review strategy

Citations and abstracts from the electronic online databases were exported into a citation software package (EndNote X7) and duplicate citations were removed. Two reviewers independently reviewed titles and abstracts to identify potentially relevant papers for inclusion. Any discrepancies were resolved by discussion.

Inclusion criteria

Studies meeting the following criteria were included in the review:

• study type – fully published economic evaluations (including any economic models)

• population – patients with suspected infectious gastroenteritis

• intervention – integrated multiplex PCR tests, which include xTAG, FilmArray and Faecal Pathogens B

• comparator – standard microbiology techniques as outlined in the PHE pathway for routine testing in cases of gastroenteritis and diarrhoea

• outcomes – cost–benefit, cost–consequences, cost-effectiveness or cost–utility studies reporting outcomes as clinical effectiveness measures or utility measures [utility, EuroQol-5 Dimensions (EQ-5D) or Short Form questionnaire-6 Dimensions score, or quality-adjusted life-years (QALYs)].

Exclusion criteria

Studies meeting the following criteria were excluded from the review:

• non-English-language publications

• studies not in humans

• studies not in gastroenteritis and diarrhoea

• studies that were not full economic evaluations (incremental costs and incremental benefits).

Studies that provided useful information for the economic model, such as resource use, costs, utilities and transition probabilities, were retained but were not included in this review.

Data extraction strategy

Data extraction was carried out by one reviewer using standardised data extraction sheets (www.journalslibrary.nihr.ac.uk/hta/hta19100) and was then checked by a second reviewer. Data extracted included the following information:

• study details – study title, author names, source of publication, language and publication type

• baseline characteristics – population (and subgroups), intervention, comparators, outcomes, study design, and setting and location

• methods – study perspective, time horizon, discount rate, measurement of effectiveness, measurement and valuation of preference-based outcomes, resource use and costs, currency, price date and conversion, model type, assumptions and analytical methods

• results – study parameters, incremental costs and outcomes, and reporting of uncertainty

• discussion – study findings, limitations, generalisability and conclusions

• other – sources of funding, conflicts of interest and any comments.

Quality assessment strategy

We critically appraised studies classed as full economic evaluations against the framework of quality assessment for economic evaluation studies developed by the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) group. 51 The CHEERS framework comprises six dimensions (i.e. title and abstract, introduction, methods, results, discussion and other), including a series of questions check whether or not the criteria have been clearly reported. If the studies included any model-based economic evaluations, they were further critically appraised using the framework on quality assessment for economic modelling developed by Philips et al. 52 The framework assesses decision-analytic models used in economic evaluations under the dimensions of structure, data and consistency.

Search results

The literature search identified 377 records through electronic database searches and other sources:

• Ovid MEDLINE – 1946 to January week 3 2016 (n = 132)

• Ovid MEDLINE In-Process & Other Non-Indexed Citations – 27 January 2016 (n = 12)

• EMBASE – 1974 to January 27 2016 (n = 203)

• Web of Science Core Collection (n = 30).

After removing duplicates, 222 records were screened for inclusion. On the basis of title and abstract sift only, 217 records were excluded. The remaining five records were included for full-text screening. 27,53–56 A further four articles were excluded at the full-text stage as these studies did not contain a full economic evaluation (see Appendix 9). 27,53–55

The literature search identified only one study,56 classed as a full economic evaluation, that simulated the GPP pathway of care and presented a cost–consequence type model (although not explicitly described) (Figure 9). Two further reports were identified in our search (and also by NICE) but were not included in our review: Abubakar et al. 57 did not look at the same interventions as outlined above in our inclusion criteria and Pankhurst et al. 30 did not provide a cost-effectiveness analysis. However, both studies contained some useful information for our economic model and so have been summarised in Additional information.

FIGURE 9.

A PRISMA flow diagram for the economic review.

Quality assessment

The quality of the reporting of the economic analysis provided by the Goldenberg et al. 56 study was assessed using the 25-point CHEERS checklist51 and is provided in Appendix 10.

The study was comprehensively completed, with 17 out of the 25 statements (68.0%) being completed, three statements (12.0%) partially completed, three statements (12.0%) not completed and two statements (8.0%) not applying. The quality of the reporting of the cost–consequence model, using the Philips et al. 52 checklist, is presented in Table 7. The article was not comprehensively reported, with just 11 out of the 57 statements (19.3%) completed, 7 statements (12.3%) partially completed, 11 statements (19.3%) not completed, 18 statements (31.6%) not applying and 10 statements unclear (17.5%). The poor quality of reporting was related to the authors conducting a partial rather than full economic evaluation.

Overview of included studies

Isolated patients

In the following pathways, ‘treatment’ refers to pathogen-specific treatment over and above general care, hydration and hygiene management received by all patients. In the case of isolated patients in whom a pathogen has been detected, there are four options:

1. discharge the patient because symptoms have resolved

2. continue to isolate the patient and provide treatment

3. continue to isolate the patient and do not provide any treatment

4. deisolate the patient and do not provide any treatment.

With the final three options, symptoms can either resolve naturally, or can persist; in both cases, the patient is subsequently discharged when appropriate.

In the case of isolated patients in whom no pathogen has been detected, there are three options:

1. discharge the patient because symptoms have resolved naturally

2. keep the patient in isolation because symptoms persist

3. deisolate the patient and do not provide any treatment.

Patients in whom symptoms persist undergo retesting. (These patients remain in isolation.) The second test results either will or will not detect a pathogen. Patients in whom a pathogen is detected follow the same care pathway as isolated patients in whom a pathogen has been detected. In the case of patients in whom no pathogen is detected there are two options:

1. discharge the patient because symptoms have resolved

2. continue to isolate the patient and do not provide any treatment.

In the latter case, symptoms can either resolve naturally or persist; in both cases, the patient is subsequently discharged. Among patients who have been deisolated and who have received no treatment, symptoms can either resolve naturally or can persist; in both cases, the patient is subsequently discharged.

Non-isolated patients

For non-isolated patients in whom a pathogen has been detected, there are four options:

1. discharge the patient because symptoms have resolved

2. isolate the patient and provide treatment

3. isolate the patient and do not provide any treatment

4. do not provide any treatment.

With the final three options, symptoms can either resolve naturally, or can persist; in both cases, the patient is subsequently discharged when appropriate.

Among non-isolated patients in whom no pathogen has been detected, symptoms can resolve naturally or persist. Patients whose symptoms have resolved are discharged. Patients with persistent symptoms are retested. Those patients in whom a pathogen is detected follow the same care pathway as non-isolated patients in whom a pathogen has been detected. For patients in whom no pathogen is detected, there are two options:

1. discharge the patient because symptoms have resolved

2. do not provide any treatment.

In the case of the latter option, symptoms can either resolve naturally or persist; in both cases, the patient is subsequently discharged when appropriate.

Test costs

The test costs for the conventional arm were based on the study by Goldenberg et al. 56 The tests included routine culture methods, which can potentially detect Campylobacter, Salmonella, Shigella and E. coli O157; light or fluorescent microscopy, which can detect Giardia and Cryptosporidium; enzyme immunoassays [glutamate dehydrogenase (GDH)] or PCR for detecting C. difficile; PCR tests for detecting norovirus; and the combined immunochromatographic tests for detecting adenovirus and rotavirus. All test costs included consumables, labour and overheads. The costs for conventional tests are shown in Table 29.

The base model assumes an average throughput by GPP of 24 samples per day, as representative of typical use, although some larger facilities may have considerably higher average throughput. The configurations chosen allow flexibility to manage peak loads within facilities.

To calculate staff costs per hour associated with processing the samples, we have used the cost of a medical laboratory assistant using Agenda for Change pay rates for 2015 band 4 mid-point (point 14: £20,844 plus 17.5% on-costs). 60 This was based on working an average of 42 weeks per annum and 37.5 hours per week.

For xTAG, to run 24 samples would take an average of 5–6 hours. Using the information above, the staff cost associated with each individual sample would be £3.43. The MAGPIX system (Luminex, Toronto, ON, Canada) costs £25,000 and, taking into account a 10-year lifespan and using a 3.5% discount rate,61 assuming that the system is used every day for 24 samples, the cost would be £0.34 per sample. Assuming that the annual maintenance cost is £4190, the cost of the nucleic acid extraction system is approximately £35,000 and the Thermal cycler costs £7500 (assuming for the final two products a 5-year lifespan and using a 3.5% discount rate), the total cost per sample would be £4.67. We have also included the cost per well, which equates to £20, and the cost of consumables of £8.66. In total, the cost per sample for xTAG is £37.10 (see Table 29).

For FilmArray, to run 24 samples would take an average of 3 hours (using FilmArray 2.0 with eight modules, which is run three times). Using the information above, the staff cost associated with each individual sample would be £2.01. The capital cost of the system is £28,000, including the costs of a further seven modules, taking the total capital cost up to £203,000. Taking into account a 5-year lifespan and using a 3.5% discount rate,61 assuming that the system is used every day for 24 samples, the cost would be £5.13 per sample. The annual maintenance cost is £3400 for one module, the cost for one gastrointestinal panel kit is £85.00 and the Cary–Blair medium cost is approximately £1. Using these costs, the total cost per sample would be £93.53 (see Table 29).

For Faecal Pathogens B, to run 24 samples would take an average of 2–3 hours. Using the information above, the staff cost associated with each individual sample would be £2.73. The capital cost of the system is £28,320, and taking into account a 5-year lifespan and using a 3.5% discount rate,61 assuming that the system is used every day for 24 samples, the cost would be £0.72 per sample. The annual maintenance cost is £3172; the costs for tubes (step 1), plates (step 2), plus the master mix is, in total, £10.26 per sample; and consumables and extraction come to £4.16 per sample. Using these costs, the total cost per sample would be £17.86 (see Table 29).

After GPP testing, for confirmatory purposes we have included the cost of enzyme immunoassay (GDH) or PCR test for screening (£26.05) when C. difficile has been detected and the cost of routine culture (£11.30) when E. coli O157, Salmonella and Shigella have been detected.

We have not added in the cost of waste disposal associated with culture of faecal samples. The cost of disposing of samples can be substantial; however, as confirmed by our clinical experts, this cost is common to both arms.

In addition, we have not added any further costs for samples that are then sent to PHE for subsequent testing. As confirmed by our clinical experts, sending these samples will not affect subsequent management for these patients.

Unit costs for bed-days, cleaning, any other tests or investigations, blood tests, medications and rehydration costs are presented in Table 30.

Number of bed-days before obtaining test results

It has been assumed that the results of conventional testing will come back in 3 days and the results of GPP tests will come back more quickly: xTAG in 1 day and FilmArray in half a day. The use of bed-days takes into account the time taken to collect a stool sample, to send the stool sample to the laboratory, to process the test and obtain the results, to communicate the results and to decide, depending on the test results, if any action needs to be taken.

Patients in whom a pathogen was not detected in the first instance, and who are retested because of persisting symptoms, are assumed to consume the same number of bed-days as above before the second set of test results is obtained.

Number of bed-days after obtaining test results

The total number of bed-days spent in hospital by patients with suspected gastroenteritis was based on a weighted average obtained from the Admitted Patient Care spreadsheet within the Hospital Episodes Statistics data for 2014/15. 67 The primary diagnosis Healthcare Resource Group code for intestinal infectious diseases was A00–A09. During the time period 1 April 2014 to 31 March 2015, there were 201,141 finished consultant episodes relating to intestinal infectious diseases, and mean length of stay was 4 days (this is based on all patients regardless of age or sex). Further breakdown of codes A00–A09 enabled a calculation of the weighted average number of bed-days for the different pathogens (Table 31).

Once test results have been obtained, whether by conventional testing or by one of the GPP tests, it has been assumed that the number of bed-days listed above will apply for the total number of bed-days (before and after test results) for both isolated and non-isolated patients, depending on the care pathway before the patient is discharged. However, when a pathogen was missed at the first test but identified at the second test, the patient stay would be increased by the time delay to the second test.

A number of further assumptions were also made with regard to bed-days. First, we have assumed that the total minimum number of bed-days for both arms would be 3 days (even though the results may come back earlier for GPP testing). This was so that we could be consistent with what happens with conventional care patients, as there is currently no evidence that patients would be discharged earlier. Some patients are already discharged earlier than the 3 days (as this is an average), presumably because of symptom resolution. Thus, the base model assumes that patients are essentially treated symptomatically (with simple hygiene and rehydration advice) independent of the method of testing. Second, we have assumed that, if patients were in isolation, they might remain in isolation without moving to an open ward/bay before being discharged. Third, for patients in whom C. difficile was detected, the number of bed-days after the test result was split evenly between an isolation room and an open ward/bay.

Costs for bed-days were obtained from NHS Reference Costs62 (see Table 30). For stay on an adult general ward, this cost was a weighted average based on patients with gastrointestinal infections without interventions from the elective inpatient spreadsheet. For patients who stayed in isolated wards, an additional cost of £96.50 was added. This cost was based on the additional cost of patients with meticillin-resistant S. aureus infection staying in single beds in single room isolation rather than open wards. 63

Daily cleaning and spot cleaning

For each day the patient spent in hospital, a cost for daily cleaning was included. This cost included time for domestic cleaning staff, specialist cleaning agents and special cleaning equipment as reported in a previous Health Technology Assessment publication64 looking at diarrhoea in older people who are admitted to hospital. We also assumed that, for every second day spent in hospital, patients would have ‘an accident’ relating to their diarrhoea and spot cleaning would need to take place. This cost, again, included time for domestic cleaning staff, specialist cleaning agents and special cleaning equipment, as well as spot cleaning and changing of beds, laundry and disposables such as gloves and aprons. 64 Unit costs for cleaning are shown in Table 30.

Blood tests and other investigations

We assumed that a full blood count will be carried out in every patient on admission, and that approximately 30% of patients will undergo routine blood chemistry and a C-reactive protein test. Unit costs for blood tests were obtained from the NHS Reference Costs from the Direct Access Pathology Services spreadsheet62 (see Table 30).

We also assumed that 1% of patients in whom a pathogen was detected would undergo flexible sigmoidoscopy, as would 10% of patients in whom no pathogen was detected. We estimated that abdominal radiography would be performed in 10% of patients in whom C. difficile was detected and in all patients in whom no pathogen was detected. It was assumed that 5% of patients in whom conventional testing detected any other pathogen would undergo abdominal radiography, as would 2% of patients in the GPP test arm, reflecting the impact of earlier microbiological diagnosis on other testing. Unit costs for the named tests above were obtained from NHS Reference Costs62 (see Table 30).

Treatment

For most gastrointestinal infections, antimicrobial therapy is not recommended except in certain situations, for example C. difficile infection or immunocompromise. Unit costs for drugs were obtained from the British National Formulary (BNF)65 (see Table 30) and we have assumed that patients complete each course of tablets/capsules prescribed either at hospital or at home. For patients in whom C. difficile infection is identified, either metronidazole (non-severe C. difficile) or vancomycin (severe C. difficile) is administered. For patients with severe C. difficile infection and multiple comorbidities, oral fidaxomicin (Dificlir^®, Astellas) may be considered. 68 A typical course of metronidazole or vancomycin lasts for 10–14 days (and fidaxomicin lasts 10 days), and only 47% of patients complete antibiotic therapy in hospital. 30 In a study of C. difficile-associated diarrhoea in hospitalised patients in Northern Ireland, 55.2% of patients received vancomycin and 44.8% received metronidazole. 69 According to our clinical experts, approximately 4.5% of patients in hospital would be treated with fidaxomicin. The following proportions were applied in our economic model: 52.95% vancomycin, 42.55% metronidazole and 4.5% fidaxomicin. For C. difficile patients, two packs of vancomycin or metronidazole are needed for a 10- to 14-day course.

Certain other pathogens are sometimes treated in the NHS; patients infected with these pathogens are included in the isolate and treat arm of the economic model. For example, patients infected with Campylobacter may receive erythromycin and patients infected with Salmonella, Shigella or E. coli (non-shiga toxin producing) may receive either ciprofloxacin or amoxicillin (on the advice of our clinical experts we have assumed that 75% of patients would receive ciprofloxacin and 25% of patients would receive amoxicillin); and for patients with Giardia they may receive metronidazole. Treatment is generally not recommended for shiga toxin-producing E. coli O157.

In patients in whom Cryptosporidium is detected (this is a very rare occurrence), two drugs [paromomycin or nitazoxanide (Alina^®, Lupin Pharmaceuticals Inc., MD, USA)] can be used. However, there is no standard approved treatment in the UK for Cryptosporidium;65 hence, we have not included any costs in the economic model for this pathogen.

We have not included any cost of administering the drugs, as this is included in the nursing costs, which are included in the cost of the bed-days.

Rehydration methods

Oral rehydration therapy is given only to patients who are clinically dehydrated; however, most hospitalised patients will receive some form of rehydration. We have assumed that, for each day in hospital, 60% of patients would receive oral rehydration and 30% of patients would receive intravenous fluids. We have assumed that each patient receiving oral rehydration would receive 200 ml of Dioralyte^® (Sanofi-Aventis) every 4 hours; one pack of Dioralyte contains six sachets and costs £2.25. 65 We have assumed that patients receiving intravenous fluids would receive 2 l of sodium chloride 0.9% per day, at a cost of £2.31. 66

The cost of nursing staff time to administer rehydration solutions must also be included. These costs were based on the earlier study by Allen et al. ,64 who calculated the time spent administering oral medication as 5 minutes and time spent administering intravenous medications as 20 minutes. The cost of a hospital-based nurse (band 5 with qualifications) was obtained from the Unit Costs of Health and Social Care. 59 The unit costs for rehydration methods and the administering costs are shown in Table 30.

Prevalence and probabilities

Table 33 shows the prevalence of pathogens and conditional agreement probabilities when conventional testing is the benchmark for young children, which was estimated by our clinical experts using the hospital prevalence for adults as a base case. The overall pathogen detection rate was 24.2% for conventional testing, 21.4% for xTAG and 22.3% for FilmArray testing.

The proportions of young children treated in each scenario are assumed to be the same as reported in Table 25. The overall conditional probabilities for patients in whom a pathogen is detected are shown in Table 34.

All common probabilities, as shown in Table 27, remain the same. The other probabilities for model 2 are shown in Table 35.

Resource use and costs

Conventional test costs and GPP costs, as shown in Table 29, remain the same.

We assumed that the number of days spent in hospital is the same for children as for adults infected with the same pathogen (obtained from Hospital Episodes Statistics67), as hospital stay did not differ by age or sex. The costs for bed-days were obtained from NHS Reference Costs62 based on a weighted average of paediatric, infectious and non-infectious gastroenteritis stay, and are shown in Table 36.

We have assumed that cleaning took place each day, as in the base model, and that spot cleaning took place every day, rather than every second day as in the base model.

We have assumed that the same percentage of patients as in the base model would undergo blood tests. We also assumed that 3% of patients in whom no pathogen was detected would undergo flexible sigmoidoscopy (patients in whom a pathogen was detected would not undergo flexible sigmoidoscopy). We estimated that 5% of patients in whom C. difficile was detected and all patients in whom no pathogen was detected would undergo abdominal radiography. It was assumed that 3% of patients in the conventional arm and 2% of patients in the GPP arm in whom any other pathogen was detected would undergo abdominal radiography.

The proportions of patients with C. difficile, Campylobacter, Salmonella, Shigella, E. coli (non-shiga toxin producing) or Giardia infection who received different treatments were assumed to be the same as in the base model. The costs of the drugs were taken from the BNF for Children80 (see Table 36). Patients infected with C. difficile require three vials of intravenous injections of vancomycin or two packs of metronidazole for the 10- to 14-day course.

We have assumed that, for each day in hospital, 75% of young children would receive oral rehydration and 20% of young children would receive intravenous fluids. The unit costs for the rehydration methods were obtained from the BNF for Children80 and are shown in Table 30 along with the administration costs.

Utilities

Utilities for the different pathogens remained the same as the base model, except for rotavirus, in which case the mean QALY loss value changed from 0.0032 to 0.0022. This figure was derived from UK-based studies of rotavirus in children. 70,72,75,78

Prevalence and probabilities

Table 37 shows the prevalence of pathogens and conditional agreement probabilities when conventional testing is the benchmark for the community. Prevalences for conventional testing were obtained from the Second Study of Infectious Intestinal Disease in the Community. 1 The overall pathogen detection rate was 34.9% for conventional testing, 31.6% for xTAG and 32.5% for FilmArray testing. Probabilities for positive and negative agreements for each pathogen were drawn from the clinical effectiveness review.