A cost-effectiveness modelling study of strategies to reduce risk of infection following primary hip replacement based on a systematic review

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 08/13/02. The contractual start date was in November 2009. The draft report began editorial review in August 2015 and was accepted for publication in January 2016. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors” report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

none

Copyright statement

© Queen’s Printer and Controller of HMSO 2016. This work was produced by Graves et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Incidence of infections and revisions among hip replacements

Data from 235 NHS and independent sector treatment centres are shown in Table 1. The information was submitted as part of the Public Health England (PHE) national surveillance of SSI service. 3 Data were reported on 549,495 procedures and SSIs organised into 17 surgical categories from April 2008 to May 2013. Fifty-six per cent of hospitals that perform total hip replacement (THR) contributed data for patients who underwent hip prosthesis surgery. They show a low cumulative incidence of SSI of 0.7% within 1 year of hip prosthesis surgery. No clear trend was detected in the incidence over the 5-year period (Figure 2).

FIGURE 2.

Trends in the annual cumulative incidence of SSI (%) in the orthopaedic surveillance categories, with lower and upper 95% CIs, within NHS hospitals in England. (a) Repair of neck of femur; (b) hip prosthesis; (c) reduction of long bone fracture; and (d) knee prothesis. Source: Elgohari et al. 3 Crown copyright and reproduced with the permission of Public Health England under delegated authority from the Controller of HMSO.

The National Joint Registry (NJR)11 collects data that quantify some of the long-term consequences of deep infection, such as revisions. In 2012, 10,040 hip revision procedures were recorded, of which 12% were a result of infection, compared with 11% in 2011. 1 The burden of revision operations lies primarily with NHS hospitals, which carried out 83% of revision procedures, compared with 15% in independent sector hospitals. These statistics contrast with primary hip replacements, 69% of which were carried out in NHS hospitals and 27% in independent hospitals. Patients who had a revision operation were also less fit than patients undergoing primary hip replacement, with one-third of patients for revision surgery being identified as grade 3 under the American Society of Anaesthetists scoring system. 1

Definitions and diagnosis of infection

We excluded superficial infections from this analysis as they are generally easy to treat and do not present serious risks for patients. Reducing deep incisional and organ/space infections is the primary focus. Definitions used have been taken from the Surgical Site Infection Surveillance Service (SSISS) of PHE (formerly the Health Protection Agency). 3

Diagnosis and treatment

The decision regarding the selection of an adequate treatment strategy for SSI after a hip replacement should be well planned and made by a team of orthopaedic surgeons and infectious disease/microbiology specialists. The goal is not only to eradicate infection but also to preserve function of the hip joint. 19 As a number of different treatment options exist, individual circumstances have to be taken into consideration before treatment can begin. The main factors that influence the treatment and its success are the nature of the organism, site of infection, local factors relating to the bone and tissue condition, the time of infection onset and the patient’s general health status. 19–21

The first step should be diagnosis of the infection and its microbiological properties, especially when antibiotics are considered for treatment. 20 Only after the virulence and antimicrobial susceptibility of the organism and immune status of the host have been assessed can an appropriate antibiotic agent be selected. 22 There is no set approach for diagnosing SSIs or prosthesis-related infections, but a physical examination and a discussion of the patient’s history and symptoms should always be incorporated. 22 The following list of strategies can be regarded as a gold standard, but in practice a combination of only some of these individual strategies is pursued. 21,23,24

• Imaging [e.g. standard radiography (X-ray), contrast autography, computerised tomography, ultrasonography, magnetic resonance imaging].

• Laboratory [e.g. repeated measurement of leucocyte count in the synovial fluid (found in the cavities between joints to reduce friction)].

• Histopathology (e.g. testing for neutrophils in tissue specimen).

• Microbiology (e.g. swabs from pre- or intraoperative specimen or from removed fragments of the implant).

Once the diagnosis and type of infection has been established, the treatment options are surgical debridement with retention of the prosthesis, revision surgery, lifelong suppressive antibiotics or permanent resection. 19,25,26

Impact on patients and costs to health services

Deep SSIs are associated with substantial costs. Direct costs include a prolonged hospital stay, hospital readmission, outpatient visits, reoperation, additional antibiotic treatment, radiological and laboratory tests, home visits by health services and medication costs, as well as additional mortality and morbidity. Indirect costs include lost productivity of the patient and their family but could also comprise the damage to a clinician’s or hospital’s reputation or loss of staff morale. 38 Other consequences of SSIs are a serious impact on the quality of life and mental state of the generally elderly patients having to face further major surgery. Orthopaedic SSI has been reported to increase the mortality risk by 50% and the hospital stay by, on average, 11.5–14 days and to double the rehospitalisation rates. 4,5,39 A UK study by Vanhegan et al. 40 compared the costs of revision operations for aseptic loosening, dislocation, deep infection and periprosthetic fracture. Clinical, demographic and economic data were collected for 305 consecutive-revision THRs in 286 patients in a single tertiary referral unit. The authors found that the mean inpatient stay for patients with deep infection was significantly longer. Mean total costs were £21,937 in deep infection cases, compared with £11,897, £18,185 and £10,893 for aseptic, dislocation and periprosthetic cases, respectively. A recent cost-analysis study in France found that costs of septic revisions were 3.6 times higher following a THA than following a primary THA, mainly because of longer hospitalisation periods and rehabilitation after hospital discharge. 41 The treatment or revision of an infected joint can be very time-consuming and resource intensive. 42,43 Costs of SSIs usually increase with the depth of infection (i.e. superficial infections incur fewer costs than deep infections). 38 The economic burden of SSI after hip arthroplasty was reported to increase health-care costs by more than 300%, but dollar estimates differ widely, from US$400 to US$60,000 per SSI treated. 5,38,41,44,45 Apart from obvious reasons for the wide range of estimated costs, such as severity of infection or differing hospital fees in different countries, there is no consistent methodology used to calculate costs of infection. 46

The measurement of cost requires identifying the quantity of resource use and the unit costs or prices for using the resources. 47 However, in this context not only costs attributed to the occurrence of infections and related resources are relevant. The implementation of additional infection prevention will also incur costs, details of which are important for an accurate representation of cost outcomes. Often cost savings resulting from infection prevention programmes are overstated, making them appear more desirable than they actually are. Using hospital accounting methods rather than an economic approach biases the results in this context; they include fixed costs, such as wages, which are not affected by the incidence or control of infection. 48 Similarly, costs of infection are overstated when total hospital costs and total length of stay of infected patients are compared with uninfected patients. Patients who are more susceptible to infection often display severe cases of illness and are thus likely to incur higher costs throughout their hospital stay. Only costs incurred after the onset of infection should be attributed to infections. 49

For the economic evaluation in this project, the costs of infection prevention interventions are direct costs of implementation and continuous use of each strategy. Ultraclean air surgery, for example, would require the installation of specialised air handling systems. These capital costs would be valued by calculating the equivalent annual costs that integrate opportunity costs, as well as depreciation of the investment. 47 These costs, along with resource costs, such as additional electricity consumption per procedure or increased staffing requirements, would have to be taken into consideration. Each of the competing infection prevention strategies may be associated with different costs. The cost estimates for the occurrence of infection include direct in-hospital and post-discharge costs. Direct in-hospital costs can be measured as treatment costs, resource consumption and extra mortality and morbidity resulting from infection, whereas direct post-discharge costs are measured as out-of-pocket costs for patients and the use of primary care services. Treatment costs refer to the different treatment alternatives described in the model.

Current general infection prevention measures in the NHS

Guidelines on infection prevention precautions have been developed for NHS professionals to ensure the safety of patients and health-care personnel, as well as those who visit the health-care environment. General infection prevention measures include hand hygiene, respiratory hygiene, personal protective equipment, occupational exposure, management of care equipment, safe care of linen, control of environment and safe waste disposal.

• Hand hygiene: good hand hygiene is very important for reducing the transmission of infectious agents.

• Respiratory hygiene: respiratory hygiene has been added to the guidelines to reduce the risk of an influenza pandemic.

• Personal protective equipment: the use of gloves, gowns, masks or goggles is essential for the health and safety of both patients and those caring for them.

• Occupational exposure management: health-care staff must report incidents of exposure to infectious agents or needle-stick injuries and deal with incidents promptly.

• Management of care equipment: care equipment needs to be carefully managed to limit the risk of contamination with micro-organisms.

• Safe management of linen: appropriate handling of soiled fabric/linen is important to avoid transmission of micro-organisms.

• Control of environment: the term ‘environment’ refers to any general horizontal or frequently touched surfaces in the environment. Routine cleaning is required to minimise transfer of micro-organisms from the environment to patients/clients.

• Safe waste management: disposing of waste appropriately can minimise the risk of transmitting micro-organisms.

Specific guidelines on control of SSIs are available in UK, providing advice on the preoperative, intraoperative and postoperative phase (Table 2). 50–52

Current cost-effectiveness evidence

The following databases were searched for relevant cost-effectiveness evidence: PubMed, Health Technology Assessment Database, NHS Economic Evaluation Database, MEDLINE, Academic Search Elite, Australia/New Zealand Reference Centre, Cumulative Index to Nursing and Allied Health Literature and EconLit. Searches were initially based on applicable medical subject headings/subject terms for research undertaken in this field (‘Arthroplasty, Replacement, Hip’, ‘Surgical wound infection’, ‘Decision Support Techniques’, ‘Models, economic’, ‘Costs and Cost Analysis’). In order to make the searches more specific, and to find further references, individual search terms were combined with medical subject headings (‘infection’, ‘cost-effectiv*’, ‘intervention’, ‘control, ‘antimicrobial’, ‘prophylaxis’, ‘antibiotic’, ‘cement’, ‘gentamicin’, ‘pre-operative’, ‘showering’, ‘antisep*’, ‘skin preparation’, ‘hair’, ‘nutrition’, ‘sterile’, ‘gown’, ‘surgical attire’, ‘mask’, ‘ultra-clean air’, ‘laminar’, ‘operating room’, ‘oxygen’, ‘suction drain’, ‘patient warming’, ‘UV radiation’, ‘wound dressing’, ‘surveillance’).

Studies were screened for relevance using the eligibility criteria outlined in Table 3.

Searches were limited to match these criteria as much as possible; for example, searches were restricted to the relevant time frame and English language only.

Studies were selected in a four-step process, illustrated in Figure 3. First, all titles in the selected search results (n = 199) were screened for inclusion. Titles not meeting the eligibility criteria at this stage were excluded (n = 133), for example because the study focus was on a different type of surgical procedure, such as knee or shoulder arthroplasty, or had a different objective, such as prevention of thromboembolism. Second, the abstracts of relevant studies (n = 66) were screened for inclusion.

FIGURE 3.

Review process of articles retrieved in the medical literature.

The most common reasons for exclusion were partial economic analysis, especially cost analysis, or studies for infection treatment or diagnosis rather than control. If decisions about inclusion were unclear, full articles were retrieved and reviewed. Studies were included if they met all of the above-described criteria. References of eligible studies (n = 4) and excluded literature reviews (n = 3) were hand-searched for further relevant studies.

Quality assessment

The quality of eligible studies was assessed on aspects of structure, data and consistency, using a checklist adapted from Philips et al. 59 (Table 5).

TABLE 5 - Quality assessment of existing cost-effectiveness evidence

Dimension of quality	Study
	Cummins et al. (2009)54	Elliott et al. (2010)55	Courville et al. (2012)56	Merollini et al. (2013)57
Structure
Statement of decision problem	Yes	Yes	Yes	Yes
Statement of perspective	No	Yes	Yes	Yes
Definition of comparators	Yes	Yes	Yes	Yes
Statement of model assumptions	Yes	Yes	Yes	Yes
Illustration of model structure	Yes	Yes	Yes	Yes
Appropriate time horizon	Yes	Unclear	Yes	Yes
Data
Data identification process transparent	Yes	Yes	Yes	Yes
Sources of all data given in detail	No	No	Yes	Yes
Quality of model input parameters	see Figure 4	see Figure 5	see Figure 6	see Figure 7
Assessment of uncertainty	Yes	Yes	Yes	Yes
Identification of key parameters	Yes	Yes	Yes	Yes

The structural quality of all four economic evaluations was high, although Cummins et al. 54 failed to clearly state the perspective of the evaluation and Elliott et al. 55 did not clearly define the time horizon. Dimensions of data quality were fulfilled by most studies, but two, by Cummins et al. 54 and Elliott et al. ,55 failed to provide sources of data for at least one parameter. Owing to their importance for the quality of model, input parameters were assessed in more detail.

Model parameters were ranked using a hierarchy developed by Cooper et al. 60 for evaluating decision model parameters. Using this hierarchy, evidence levels range from 1 to 6, with higher numbers indicating better quality. The quality of baseline clinical data, treatment effect, costs and utilities is summarised for the economic evaluation by Cummins et al. 54 in Figure 4 and for Elliott et al. 55 in Figure 5. If the evidence used did not match the specified description of evidence, the next most suitable category was chosen. Neither of the studies commented on the assessments of internal consistency of the decision model. Therefore, it is unclear whether or not the mathematical logic of the model has been evaluated in these publications.

FIGURE 4.

Quality of model input parameters used in decision model by Cummins et al. 54

FIGURE 5.

Quality of model input parameters used in decision model by Elliott et al. 55

Figure 4 shows that the quality of baseline clinical data used by Cummins et al. 54 was medium to high, low for treatment effect and high for cost estimates, whereas the quality of utilities used ranged from low to high.

Cummins et al. 60 derived baseline infection rates from the Norwegian Arthroplasty Registry, which can be classed as a recent ‘reliable administrative database covering patients solely from another jurisdiction’,60 representing a quality score of 3. 60 Baseline surgical mortality was assumed to be the same for primary THA and revision surgery, and was also taken from the Norwegian Arthroplasty Registry. Underlying mortality was informed by recent US life tables, earning a quality score of 2. Owing to a lack of high-quality data, the clinical effectiveness (treatment effect) of antibiotic-impregnated bone cement was derived from a non-analytic study and is, therefore, classified as low quality. Cost parameters assigned to primary THA, increased cost of aseptic revision and revision resulting from infection were adopted from recently published cost calculations of high quality from the same jurisdiction. The quality of data for the extra costs of using antibiotic cement was also high, as these estimates were based on original figures from the authors’ institution. Utility parameters for primary THA and aseptic revision were of high quality, adopted from a study designed for the patient group of interest. The utility for aseptic revision, on the other hand, was estimated by the authors as no published estimate was found, representing the lowest level of data quality according to the classification system used. It was unclear how disutility was measured, as no reference was stated and no measurement method was described. However, the results of the Markov model showed that, in the sensitivity analysis, disutility values had no effect on the model. 54

Figure 5 shows that the quality of baseline clinical data used by Elliott et al. 55 was high. The quality of treatment effect data was rated as medium, whereas cost parameters data were a mix of high and low quality and utility inputs data were of low quality. Elliott et al. 55 used a possible range of infection rates for their baseline estimate, based on two different high-quality sources from the literature. Mortality estimates for superficial and deep SSIs were informed by a recent case series from the same jurisdiction. Underlying mortality rates were taken from UK life tables, which are high-quality data. As no treatment effect describing antibiotic prophylaxis for MRSA prevalence was available in the orthopaedic setting, results from a cardiac clean surgery trial were used instead. Using data from a randomised controlled trial (RCT) conducted in a different setting may compromise the credibility of model outcomes but, in this case, as both procedures are clean surgery, the most appropriate ranking for this input is 4. Different costs for antibiotic prophylaxis and treatment costs of superficial, deep MRSA and non-MRSA infections were derived through personal communication with the British Medical Association and Royal Pharmaceutical Society of Great Britain, all representing high-quality data inputs. Recent UK case series and a previously published economic evaluation informed costs of barrier nursing and inpatient days, and administration costs were set at local NHS contract costs. The reference specified for wound exploration costs did not describe any costs but stated that this strategy can be used to treat infections and hence this parameter was classed as low quality. 61 QALYs for infected joints were derived from a previous study that used expert opinion and was therefore given the lowest quality categorisation. 55,62

Figure 6 shows that the baseline clinical data used by Courville et al. 56 was a mix of low and high quality: the quality of treatment effect data was medium, cost parameters data were of high quality and utility inputs data were a mix of low and medium quality. For the base case, Courville et al. 56 used the S. aureus colonisation rate from their local population of patients with total joint arthroplasty. This is high quality according to the Cooper et al. 60 classification system. However, the specificity and sensitivity of the screening test for S. aureus were estimated based on personal communications and unpublished data in the original source. The treatment effect of mupirocin in reducing SSIs was derived from a meta-analysis that included both RCT and before-and-after trials for patients undergoing different surgical procedures and is therefore ranked at level 3 for quality. The probabilities of SSI among mupirocin-treated carriers and untreated non-carriers were sourced from a single case–control study, representing medium quality. The quality of data for costs of screening tests and treatment was high, being based on fees from the authors’ medical centre. Similarly, the costs of primary total joint arthroplasty and revisions were estimated from published papers where cost calculations were based on reliable data sources in the USA, and were therefore also of high quality. Utility scores for primary total knee replacement, septic knee revision and THA were based on previous studies using the time trade-off method. The quality of utility data for septic hip revision was classified as low, having been taken from the expert opinion estimates used by Cummins et al. 54

FIGURE 6.

Quality of model input parameters used in decision model by Courville et al. 56

Figure 7 shows that the quality of baseline clinical data used by Merollini et al. 57 was mostly high and that treatment effect data were of medium to high quality, whereas cost parameters data were a mix of high and low quality and utility inputs data were a mix of low and medium quality. Merollini et al. 57 sourced the occurrence of deep infection and other transition probabilities from high-quality hospital records in Queensland, Australia. Mortality probabilities for deep infection were taken from a retrospective review of surveillance data in the UK, which is classified as medium quality. Mortality probabilities for revision surgery were calculated based on data from the Australian Orthopaedic Association National Joint Replacement Registry by Merollini et al. ,57 giving a quality score of 1. The quality of underlying mortality probabilities was also high, as Australian life tables were used. The clinical effects of antibiotic prophylaxis and antibiotic-impregnated cement were taken from meta-analyses and are therefore classified as high quality. A large cohort study provided medium-quality data on clinical effect on laminar air systems in operating theatres. Costs of antibiotic prophylaxis and ultraclean air systems were based on expert opinion and conservative estimates by the authors, and are therefore of low quality. On the other hand, the cost assigned to the additional use of antibiotic cement was obtained through personal communication with the Prince Charles Hospital in Brisbane, QLD, Australia, which is classified as high quality. Utility scores for patients with no infection, revision operations and successful treatments were derived from a study that estimated quality of life using the 15D health-related quality-of-life (HRQoL) survey (a 15-dimensional, standardised questionnaire) and an observational study (using the Short Form questionnaire-36 items) that was referred to estimate utility for deep infection. However, the utility of permanent and temporary resection was based on expert opinion and was therefore classified as low-quality evidence.

FIGURE 7.

Quality of model input parameters used in decision model by Merollini et al. 57

Research goals and objectives

The economic paradigm is that enhanced infection prevention in NHS hospitals will change cost and health outcomes. Costs will rise with more aggressive infection prevention programmes, but savings will accrue when cases of infection are prevented. Infection prevention strategies will either increase or decrease total costs depending on their cost and effectiveness. Health outcomes will only improve as infection-related morbidity and mortality risk are avoided. The goal of this research is to assess the cost-effectiveness of infection prevention strategies for hip replacements performed in NHS hospitals.

The information generated will address a gap in the scientific knowledge about how the risks of infection following THR should be managed. The major benefit of this work will be that infection prevention arrangements for THR will be improved and this will improve NHS efficiency.

There are five tasks:

1. Synthesise evidence on the effectiveness of strategies to reduce risk of infection after primary hip replacement (see Chapters 2 and 3).

2. Design a decision-analytic model to predict cost and health outcomes from infection prevention strategies (see Chapter 4).

3. Identify, select and synthesise the remaining evidence required to update the model (see Chapters 5 and 6).

4. Evaluate the model and characterise uncertainty among the predictions of cost-effectiveness (see Chapter 6).

5. Interpret the findings with the needs of policy-makers in mind (see Chapters 7 and 8).

A formal modelling framework will be used and updated with existing data. This will inform decision-making for infection prevention and risk reduction. This represents an advance on the existing research in which either the effectiveness or cost-effectiveness of single interventions to manage risks is studied in isolation. Answering the research questions with a prospective clinical trial would be too complex and costly, and unlikely to survive review by an ethics committee.

Mixed-treatment comparison models

We used a mixed-treatment comparison (MTC) model for the evidence synthesis, as it allows coherent judgement to be made on which of multiple treatments is the most effective and produces estimates of the relative effects of each treatment compared with every other treatment in a network. 77,78 A MTC enables simultaneous comparison of multiple treatments from trials that individually do not compare all treatment options under consideration. 78,79

A MTC is achieved by pooling direct and indirect evidence on each relative treatment effect from all trials in the evidence network for comparison. 77 The model assumes that the relative effect of one treatment compared with another is the same across the entire set of trials for the fixed-effect analysis,80 and the odds ratios (ORs) in each trial are different, but form a single common distribution that is the same across all sets of the trials for the random-effects analysis. 77,79 It is under such an assumption that models have been developed to simultaneously synthesise all available evidence using an extended meta-analysis model without breaking randomisation. 79,81

Regression-based methods have been developed to fit MTC models. 82–85 The basic model specification for MTC methods is an extension of the Bayesian specification for standard pairwise meta-analysis of binary data using a logistic regression model:86

where p_jk is the probability of the event for treatment k in trial j; µ_jb is the log-odds of the event for the reference (baseline) treatment b in trial j. The study effects µ_jb are treated as unrelated nuisance parameters. 77 δ_jbk is the trial specific log-OR of treatment k relative to the reference treatment b in trial j (k > b signifies that k is numerically after b).

Evidence search strategy

There is a wide range of infection prevention measures in current clinical practice, yet we focused on key infection prevention strategies with critically important clinical and health-care cost implications. These strategies include antibiotic prophylaxis, antibiotic-impregnated cement and laminar airflow systems. We followed the systematic review guidelines as outlined in the Quality of Reporting of Meta-Analyses statements87 and the guide to the methods of technology appraisal by NICE88 for the evidence synthesis.

The first stage of the literature search used existing systematic reviews on antibiotic prophylaxis, antibiotic-impregnated cement and laminar airflow systems to locate relevant studies for inclusion in the current evidence synthesis. The second-stage literature search was designed to update existing systematic reviews by identifying new primary interventions for inclusion in the evidence synthesis. The major electronic databases searched included MEDLINE (via EBSCOhost), EMBASE, Cumulative Index to Nursing and Allied Health Literature and the Cochrane Central Register of Controlled Trials.

Relevant journals, conference proceedings and bibliographies of retrieved papers were hand-searched. Orthopaedic surgeons and infection prevention experts were also consulted. Owing to language resource constraints, the search was limited to only English-language papers. The selection of evidence was conducted by two independent reviewers and discrepancies were resolved by consensus. The search terms and electronic search strategies used can be found in Appendix 1.

Definition of the outcome measure

The outcome measure used for the current evidence synthesis was deep SSI following primary THR. Failure to use objective criteria to define SSIs has been shown to substantially affect reported SSI rates. 89 For this review process, which included international evidence, we relied on the definition used in the Guideline for Prevention of Surgical Site Infection by the US Centers for Disease Control and Prevention. 12 This definition is described in Chapter 1.

Inclusion criteria

Studies were included if all of the following applied:

1. They were primary interventions with THR-related deep SSIs reported as an outcome.

2. Antibiotic prophylaxis, antibiotic-impregnated cement or laminar airflow system was a trial arm.

3. Antibiotic delivery methods were specified (systemically, via cement or both).

4. The type of ventilation system used in the operating theatre was indicated; otherwise, a conventional ventilation system was assumed.

Some studies defined early deep infection as those requiring a revision procedure within 6 months of the initial operation and used revision rates as early deep infection rates. Given that deep infections that developed within this time frame were most likely caused by bacterial contamination at the time of surgery, studies that only reported revision rates resulting from primary THR-related infection were also included.

Exclusion criteria

Studies were excluded if any of the following applied:

1. Only superficial infection following THR was reported as an outcome measure, or superficial and deep infection were treated as one outcome measure without separating one from the other.

2. Only joint replacement-related infection was reported as an outcome measure without separating THR-related infection from knee replacement-related infection.

3. The outcome measure was revision not caused by SSI.

We treated antibiotic prophylaxis as one intervention arm without differentiating between types, doses and durations of the administration of different antibiotic regimens. This is because there is no convincing evidence to suggest that one type of antibiotic regimen is more effective than another, that extending the duration of an antibiotic regimen beyond 24 hours postoperatively further reduces THR-related SSIs, or that single-dose or short-term administration is not as effective as long-term administration. 65 Therefore, primary interventions that compared different types, doses or durations of antibiotic regimens were treated as one-arm trials and excluded from the network meta-analysis, as MTC relies on there being at least one comparison of two arms that can become part of the connected network. 79 The process used for the literature search is shown in Figure 8.

FIGURE 8.

Two-stage literature search flow chart. CINAHL, Cumulative Index to Nursing and Allied Health Literature. Reproduced from Zheng H, Barnett AG, Merollini K, Sutton A, Cooper N, Berendt T, et al. Control strategies to prevent total hip replacement-related infections: a systematic review and mixed treatment comparison. BMJ Open 2014;4:e003978. 9 This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.

Data extraction

Data were extracted by two independent reviewers and consensus was reached. The key data extracted from the included studies for the evidence synthesis included the total number of THRs performed and the total number of deep SSIs diagnosed following the THR operations in each trial; information regarding the use of antibiotic prophylaxis and its delivery mode, for example systemically, via cement or both; information about the ventilation system used in the operating theatre; and special surgical clothing used, for example exhaust body suits. Where information regarding the use of the type of cement, antibiotic impregnated or plain, was unavailable, plain cement was assumed. If a study reported both early and late deep infection as outcome measures, the former was chosen over the latter in line with the Centers for Disease Control and Prevention definition of SSI. 12

Quality assessment

We conducted a quality assessment of the included studies. The method for quality grading was adapted from NICE’s Methods for the Development of NICE Public Health Guidance (Third Edition)93 and NHS Centre for Reviews and Dissemination guidance. 94 Each study was categorised by study type and graded based on the extent to which the potential sources of bias were minimised. The type of study coupled with the quality evaluation decided the level of evidence. The level of evidence table was based on the checklist of NICE’s Methods for the Development of NICE Public Health Guidance (Third Edition) that was adapted from the Scottish Intercollegiate Guidelines Network. We also assessed the internal validity of the included studies against the critical appraisal criteria based on the checklists of NICE’s Methods for the Development of NICE Public Health Guidance (Third Edition) and NHS Centre for Reviews and Dissemination guidance. Quality scores were allocated to the included studies using quality scoring systems adapted from that used by the Cochrane Musculoskeletal Injuries Group for RCTs and by Tooth et al. 95 for observational studies (see Tables 62–66 in Appendix 3 and see Table 6 for results of the quality assessment).

The choice of reference treatment and testing the statistical approach

We chose treatment (T) 1 to be ‘no antibiotic prophylaxis, plain cement and conventional ventilation without laminar airflow system’. This was the reference treatment as it was trialled against the highest number of other infection prevention strategies. The choice was made to minimise correlations that may otherwise be induced between mean treatment effects for each pair of treatments compared. A number of issues required investigation and the use of some diagnostics: variation in follow-up among the studies, model fit and deviance, and consistency. The models were fitted within a Bayesian framework using WinBUGS (MRC Biostatistics Unit, Cambridge, UK) and relevant code by Dias and colleagues. 77 The absolute and relative treatment effects and the ranking of infection prevention strategies were generated and analysed.

Addressing heterogeneity

Between-study variability in intervention effects is broadly termed ‘heterogeneity’. It can be induced by clinical diversity in terms of patient population, intervention or setting, or by variability in study design and risk of bias. The former is commonly known as clinical heterogeneity and the latter as methodological heterogeneity. We attempted to address both forms of heterogeneity with metaregression and bias adjustment.

Description of the evidence and interventions

From the 12 studies identified by the process shown in Figure 8, six were RCTs98–103 and six were observational studies. 104–109 They included 123,788 THRs and nine infection prevention strategies, as shown in the MTC network (Figure 9). The data from the included papers are shown in Table 6.

FIGURE 9.

Mixed-treatment comparison network consisting of 12 studies, with nine infection prevention strategies. Note that the lines represent direct evidence comparisons, boxes represent infection control strategies involving multiple infection control measures and the numbers on the lines represent the numbers of comparisons. The three-way loops in bold lines represent loops only formed by a multiarm trial. Reproduced from Zheng H, Barnett AG, Merollini K, Sutton A, Cooper N, Berendt T, et al. Control strategies to prevent total hip replacement-related infections: a systematic review and mixed treatment comparison. BMJ Open 2014;4:e003978. 9 This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.

The quality of evidence summarised in Table 6 was variable, with quality score between 11 and 21 (see Tables 65 and 66 in Appendix 3 for the scoring tools used). Five of the six RCT studies99–103 provided no information on random sequence generation, four100–103 provided no information on blinding assessors and only one98 reported prior calculation of the sample size. The statistical power for most RCTs was generally low. Only one98 RCT reported primary analysis based on all randomised cases, whereas the rest did not report intention to treat. Of the six observational studies, three105,108,109 identified and adjusted for confounding variables, one106 reported that cases and control groups were comparable on diagnostic confounding factors and two108,109 described and included in the analysis the outcomes of the patients who withdrew. Three studies101,108,109 used objective measures to assess the outcomes and were adequately powered with large sample size ranging from 10,905 to 51,485.

Effectiveness outcomes

For every strategy in the connected network a relative effect was estimated against another infection prevention strategy using an OR of SSI. We chose ‘no systemic antibiotics, plain cement and conventional ventilation’ as the reference strategy (T1), as it was compared with the greatest number of other strategies.

Thirty-six relative effects involving nine infection prevention strategies were estimated in the MTC network using models that did and did not adjust for duration of follow-up (Table 7). The results from both models were almost identical, as were estimates of the model fit; therefore, the differences in follow-up duration had little effect on the effectiveness of the infection strategies. Therefore, we report the results of the model without adjustment for follow-up from now on. The 36 ORs for all pairwise comparisons are presented in the forest plot in Figure 10. The probability and median rank of a strategy being the most effective strategy is shown in Table 8. The models were fitted in a Bayesian framework using WinBUGS and code by Dias et al. 110

FIGURE 10.

Forest plot of ORs of SSI for infection prevention strategies (random effects). Reproduced from Zheng H, Barnett AG, Merollini K, Sutton A, Cooper N, Berendt T, et al. Control strategies to prevent total hip replacement-related infections: a systematic review and mixed treatment comparison. BMJ Open 2014;4:e003978. 9 This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.

The five infection prevention strategies associated with a statistically significant reduction in THR-related SSI compared with the reference strategy (T1) were:

1. T6 with an OR of 0.13 [95% credible interval (CrI) 0.03 to 0.35]; systemic antibiotics, antibiotic-impregnated cement and conventional ventilation

2. T4 with an OR of 0.25 (95% CrI 0.06 to 0.66); systemic antibiotics, plain cement and laminar airflow

3. T3 with an OR of 0.26 (95% CrI 0.03 to 0.95); no systemic antibiotics, plain cement and laminar airflow

4. T7 with an OR of 0.27 (95% CrI 0.03 to 0.93); systemic antibiotics, antibiotic-impregnated cement and laminar airflow

5. T2 with an OR of 0.31 (95% CrI 0.12 to 0.65); systemic antibiotics, plain cement and conventional ventilation.

Statistically non-significant reductions in THR-related SSI compared with the reference strategy, T1, were:

• T5 with an OR of 0.38 (95% CrI 0.09 to 1.12); no systemic antibiotics, antibiotic-impregnated cement and conventional ventilation

• T8 with an OR of 0.52 (95% CrI 0.03 to 2.12); systemic antibiotics, antibiotic-impregnated cement, conventional ventilation and body exhaust suit

• T9 with an OR of 0.74 (95% CrI 0.05 to 2.69); systemic antibiotics, antibiotic-impregnated cement, laminar ventilation and body exhaust suit.

When T7 was compared with T6, the OR of SSI was 1.96 (95% CrI 0.52 to 5.37), suggesting that laminar airflow could increase infection risk. Similarly, when T8 was compared with T6, the OR was 3.72 (95% CrI 0.38 to 13.75), suggesting that body exhaust suits may also increase infection risk, at least where there is conventional ventilation. There was no high-quality evidence that antibiotic-impregnated cement without systemic antibiotics was more effective in reducing infection than plain cement and systemic antibiotics (T2 vs. T5; OR 1.28, 95% CrI 0.38 to 3.38). All comparisons and interpretations are summarised in Table 9.

Model fit and evidence consistency

The model fit statistics indicate that the fit was less than adequate (see Table 7). This was confirmed by diagnostic plots that showed infection prevention strategies T2 and T5 of study 4102 and the T1 strategy of study 10109 were outliers contributing to the inadequate model fit (Figure 11).

FIGURE 11.

Leverage vs. deviance residual superimposed on curves y = –x² + c, where c = T1, T2, T3 and T4, representing the amount contributed to DIC. Reproduced from Zheng H, Barnett AG, Merollini K, Sutton A, Cooper N, Berendt T, et al. Control strategies to prevent total hip replacement-related infections: a systematic review and mixed treatment comparison. BMJ Open 2014;4:e003978. 9 This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.

Curves of the quadratic function were plotted as they represented the lines of each contribution to DIC. Points lying outside the line c = 3, were identified as contributing to the inadequate model fit. The plot shows the first and second arms, strategies T2 and T5, of study 4102 are outliers contributing to the inadequate model fit.

After exclusion of both the first and second arms of study 4102 (T4 and T1, and T4 and T2, respectively), the model fitted the data well and the heterogeneity was significantly reduced, but the results were little changed (Figure 12).

FIGURE 12.

Sensitivity analysis excluding the first and second arms of study 4102 (T4 and T1, and T4 and T2, respectively). Leverage vs. deviance residual superimposed on curves: y = –x² + c, where c = T1, T2, T3 and T4, representing the amount contributed to DIC. Reproduced from Zheng H, Barnett AG, Merollini K, Sutton A, Cooper N, Berendt T, et al. Control strategies to prevent total hip replacement-related infections: a systematic review and mixed treatment comparison. BMJ Open 2014;4:e003978. 9 This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.

A further sensitivity analysis was undertaken that removed the first arm of study 10109 (T1 and T10) (Figure 13). After excluding study 10109 from the network, all the remaining data points lay below the quadratic curve with c = 3, suggesting that the contribution of the remaining data points to the DIC was unimportant, which in turn would improve the model fit.

FIGURE 13.

Sensitivity analysis by further excluding the first arm of study 10109 (T1 and T10). Leverage vs. deviance residual superimposed on curves: y = –x² + c, where c = T1, T2, T3 and T4, representing the amount contributed to DIC. Reproduced from Zheng H, Barnett AG, Merollini K, Sutton A, Cooper N, Berendt T, et al. Control strategies to prevent total hip replacement-related infections: a systematic review and mixed treatment comparison. BMJ Open 2014;4:e003978. 9 This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.

Following these sensitivity analyses, infection prevention strategy T6 remained dominant, with the highest probability (64%) and highest median rank of being the most effective strategy (Table 10).

The MTC results are shown in Table 11. The sensitivity analysis that excluded studies 4102 and 10109 from the MTC network showed that model fit was improved; the DIC was reduced from 180.6 to 141.8. The posterior mean residual deviance was also reduced from 34.3 to 25.3. Heterogeneity measured in between-study standard deviation across the MTC network also reduced from 0.63 to 0.43.

The direct evidence from all conventional pairwise meta-analyses is presented in Table 12. There was broad agreement among the direct evidence from conventional pairwise meta-analyses, the direct and indirect evidence from node splitting and the evidence from the MTC model. Tests for inconsistency between direct and indirect evidence from node splitting suggested that there was no statistically significant evidence of inconsistency. The model fit statistics for the node splitting and the MTC models were similar, implying that there was no conflict between the direct and indirect evidence. It is worth noting that the 95% CrIs for some pairwise comparisons widened greatly following node splitting. This is explained by the node splitting reducing the evidence available to inform the variance.

A test of interaction between RCTs and observational studies was not statistically significant, suggesting that combining these study types was not inappropriate (Table 13).

The results were little changed by excluding the RCT by Hill et al. 98 (Table 14) or by including the RCT by Lidwell et al. 111 (Table 15).

Strategy T6 (systemic antibiotics, antibiotic-impregnated cement and conventional ventilation) remained dominant in both these further analyses, with the highest probability of being a cost-effective decision (63% and 83%, respectively) and highest median rank of being the most effective strategy (see Tables 14 and 15).

Background

The purpose of this cost-effectiveness study is to evaluate change in cost and health outcomes arising from each of the identified infection prevention strategies. The main decision-making group in the NHS is NICE, and it uses cost-effectiveness information to compare the health benefits forgone from competing configurations of NHS services. This approach meets the social objective of making improvements to health when the budget for services is fixed. It requires a systematic assessment of the changes to costs and the changes to health from alternate configurations of services. 112

Costs

The change in costs is shown as ΔC and includes the positive costs from implementing an infection prevention strategy and all the cost savings that arise from the avoided infections, such as reduced length of stay and revisions. Costs are estimated as quantities of resource used, q, and prices, p. The latter are estimated using existing market prices or shadow prices. In processes of production, costs can be fixed or variable. Fixed costs cannot be changed short term and do not vary with fluctuations in productivity or output, whereas variable costs are flexible and change with productivity levels. 47 The sum of both these costs is the total cost. For the interpretation of costs of competing health-care alternatives we will estimate the incremental change to total costs. Incremental costs are more helpful for decision-making as they show the costs of achieving an additional unit of health benefit. 113

The perspective for the analysis determines the scope of the costs included, and for this project we assume only those costs incurred by the NHS are relevant. Private costs are excluded because of the difficulty in assigning a shadow price to certain items, such as volunteer time.

Costs that occur in the present are usually given more value than future costs and hence are discounted. To avoid inconsistencies in the conclusions, both costs and benefits arising in the future need to be discounted, typically at a rate of 3%; this is the rate used for this research. Without discounting, inconsistencies in the overall allocation of resources could arise, as we would be treating the value of health care differently compared with economic benefits of other sectors in the economy. 47

Health outcomes

The change in health outcomes/effects are shown as ΔE. This summarises the gain in health outcomes from a decision to adopt a treatment strategy. In this case we use the QALY, which serves as a generic measure of health outcomes. It is widely used since it allows a combination of both morbidity, measuring quality of life, and mortality, measuring length of life, into a single score. 114

Utility scores describe the quality of life in a certain health state where perfect health is valued 1 and death 0. A QALY is the product of a utility score and the time spent in that health state. In Figure 14 an example of two competing alternatives with different health outcomes is shown. Alternative A generates two additional life-years at a utility of 0.8, whereas alternative B offers four additional life-years at a lower utility of 0.6. The calculation of QALYs below shows that alternative B generates 0.8 extra QALY outcomes than alternative A. Health benefits are not discounted in this example.

FIGURE 14.

Example of a QALY calculation of two alternatives.

A widely used method for valuing the utility of different health states is a multiattribute utility scale. 115 Examples of instruments using a multiattribute utility scale include the European Quality of Life-5 Dimensions (EQ-5D) questionnaire, developed by the EuroQoL Group, and the Assessment of Quality of Life and Short Form questionnaire-6 Dimensions. 47 These classification systems allow the derivation of preferences for health states using prescored measurements called utility weights. The utility weights have been valued previously, by a representative sample of the population for which it is used. The EQ-5D is one of the most widespread instruments and measures five dimensions – mobility, self-care, usual activities, pain/discomfort and anxiety/depression – with three levels for each and so defines a total of 243 health states. 115 People with a certain disease or health outcome are asked to respond to a question for each dimension and their answers are used to determine which health state best fits their condition. The UK value set generated by Dolan116 for the EQ-5D was used for this project.

Incremental cost-effectiveness

The ICER is shown below as the cost incurred to gain 1 QALY:

It arises by comparing the costs and health outcomes of novel treatments or health programmes (C_T) with the costs of the baseline comparator (C_C). In Table 16 we show hypothetical outcomes of current practice compared with a novel infection prevention strategy, and how the resulting ICER is calculated.

A decision to adopt a new treatment or service depends on a number of factors, most importantly the ability and willingness to pay for marginal health benefits. This value is reflected by the ceiling or threshold ratio, such as £25,000–£35,000, traditionally used by NICE. 117 An evidence-based approach to finding a threshold was presented recently by Claxton et al. ,118 who found that values of around £18,000 were more appropriate. Combined uncertainty in the estimates showed that the probability that the threshold is < £20,000 per QALY is 0.64 and the probability that it is < £30,000 is 0.92. 118 Other factors influencing health policy decisions are social value judgements, such as equity and efficiency, the level of need in the community or the severity of a given outcome. 117,119

Cost-effectiveness planes are used to summarise results, and Figure 15 shows the ICER from the example in Table 16. Current practice is at the origin, the x-axis represents the effectiveness and the y-axis represents the costs associated with the infection prevention alternative. The ceiling ratio in green indicates the maximum willingness to pay for one extra unit of outcome (i.e. 1 QALY).

FIGURE 15.

Cost-effectiveness plane showing an ICER example of a novel strategy in the context of a ceiling ratio.

Strategies can fall into one of four quadrants describing possible combinations of incremental costs and benefits. Strong dominance exists for strategies that fall in quadrants II and IV. As infection prevention alternatives in quadrant IV cost less and are more effective, they should always be accepted and are described as dominant. On the other hand, strategies falling into quadrant II cost more and give less benefit. These are described as dominated and should always be rejected by decision-makers. For strategies in quadrant I or III, the decision is more complex. Strategies falling into quadrant III are less costly but also have a lower effectiveness, whereas those falling into quadrant I are more costly but return better health outcomes. 120,121 For these cases the ceiling ratio is crucial for making a decision about accepting or rejecting an alternative strategy. Strategies with an ICER below the ceiling ratio are likely to be accepted, and those above are likely to be rejected.

The novel strategy in the hypothetical example shows higher incremental costs and benefits than current practice and lies in quadrant I. At £5000 per QALY gained, it falls below the ceiling ratio and we would therefore expect the decision-maker to adopt the new treatment strategy.

Models versus prospective trials

Cost-effectiveness studies can be undertaken either as decision models or alongside prospective clinical trials. The latter are called ‘piggyback’ economic evaluations, meaning that economic data are collected alongside trial-specific clinical data. The positive aspects of this approach are low marginal costs of collecting economic data and the availability of patient-specific data for both costs and outcomes. But the fact that clinical trials are designed to assess clinical issues, such as treatment effects, can be problematic. Economic models focus on informing a specific decision, whereas trials never include all treatment alternatives relevant to a health-care decision and often compare new treatment with a placebo alternative. Economic models can include all relevant options, represented in a model structure. It is important to always compare current practice with the alternative of ‘doing nothing’, which can be simulated in a hypothetical model without randomising real patients. 122,123

Randomised trials allocate patients into groups, have relatively small sample sizes and are designed to capture short to intermediate outcomes. Follow-up of patients usually ends at a clinically significant time point, often when the efficacy of a treatment or medication is established. But other long-term effects, such as mortality beyond study end points, may nevertheless play an important role for the implementation of an alternative. Economic models have the advantage that they can simulate large patient cohorts over their lifetimes and, therefore, estimate long-term health outcomes without the time and money constraints of clinical studies. 47 Further advantages of models are that they can make indirect comparisons between alternatives for which no RCT is available. They can also be used to observe how outcomes change when certain parameters or assumptions are varied. 122

Randomised controlled trials are a gold standard measurement of outcomes and may differ from real-world scenarios. Economic models, on the other hand, synthesise data from different sources, including RCTs. Therefore, they can integrate data specific to the country of interest, such as particular occurrence and management strategies of a disease. 118 Consequently, RCTs often cannot be generalised, whereas economic models are better at reflecting real-world settings using real-world data inputs. 123

Evidence used in decision models

Decision models employ data from a range of sources to inform the model structure and parameters. Economic analysis based on data from a single RCT could introduce bias because of the method of analysis, confounding or patient selection. 60,123 In decision models, evidence is commonly required for model pathways with related probabilities, clinical effect size, baseline clinical data, resource use, costs and utilities. Possible data sources have been categorised as research based, real world based and reference based. 118,124 Research-based sources of evidence include RCTs, meta-analyses of RCTs, observational studies and economic models. Examples of real-world data sources are administrative databases, hospital or other health-care statistics and disease registers. Reference-based data sources refer to established standards in a decision-making context, such as guidelines, disease classifications or drug formularies. 125

Depending on the model component, different data sources might be more suitable than others. For the assessment of treatment effects, well-conducted clinical trials or systematic reviews of clinical trials including meta-analyses are favoured. 122,126 However, systematic reviews can very rarely be performed for each single model input because of time and resource constraints. In this case, alternative search techniques can be used, known as ‘pearl growing’ and ‘berrypicking’, which start with reviewing relevant studies and studies related to these studies until enough evidence is collected. 127,128 Although it is hard to determine when sufficient evidence has been identified, economic models can be used to establish the value of collecting more information. 129 Furthermore, systematic reviews are not always required or the ideal source of data. 118 Cost and resource use are often informed by local sources, by primary data collection or routinely collected data, as they can vary greatly in different settings.

A good model will include the most important events that impact on costs and health outcomes, rather than all possible events. 125 The model structure is typically developed and validated with the help of experts in the field, and the structure of models can be updated as new evidence becomes available. 130

Potential hierarchies of data sources have been developed to facilitate the selection of appropriate evidence. 60 Rankings of data components included in the hierarchy are clinical effect sizes, adverse events and complications, baseline clinical data, resource use, costs and utilities. For baseline clinical data, for example, the best available evidence would be a case series or a reliable database specifically developed for the study of interest, with patients from the jurisdiction of interest. Evidence of poorer quality would be recent case series not developed for the study or include patients from another jurisdiction. Further down in the hierarchy are old case series, estimates from RCTs, estimates from previously published economic analyses and, finally, expert opinion.

It is essential that the best available evidence is used to inform decision models as the results are only as reliable as the data input with the lowest quality. 60 However, the identification of all relevant evidence is very demanding and most search strategies practised lack uniformity and transparency. 124 There are no clear guidelines on this process and a recent review of modelling guidelines reported inconsistent advice. 59 This methodological issue has been recognised and more guidance on searches for model parameters is anticipated. 59,125

Uncertainty in parameters

Model parameters such as probabilities, relative risk, costs and utilities are estimates and their true value is usually unknown. In order to reduce the risk of bias and improve the usefulness of a model for decision-making, it is crucial to assess the extent of uncertainty around parameter estimates and its overall effect on model outcomes. A semi-Bayesian framework is suitable to capture uncertainty in model parameters. 131 Instead of using only point estimates the whole distribution of a parameter can be used in probabilistic sensitivity analyses (PSAs). The first step is to fit distributions around individual model parameters. Some distributions are more suitable for certain parameters because of their natural features, for example cost parameters are always positive and skewed, hence gamma distributions are a good fit (Figure 16).

FIGURE 16.

Examples of gamma distributions.

Possible shapes of beta distributions are shown in Figure 17. Beta distributions take values between 0 and 1 and are therefore suited to distributions of probabilities in the model.

FIGURE 17.

Examples of beta distributions.

Briggs et al. 132 summarises the following types of distributions, typically used to fit data likelihood functions to model parameters (Table 17).

Once distributions are fitted around model parameters a Monte Carlo simulation is used in which values are drawn randomly for each parameter from the assigned distribution. The model is run multiple times, each time with new values selected at random. The result is a joint posterior distribution of costs and effects. 133 This output distribution can be plotted on the cost-effectiveness plane illustrating incremental costs and incremental effectiveness for each of the simulations.

Figure 18 shows the PSA results for the novel infection prevention strategy example. Instead of giving the results of thousands of simulations, only 10 simulations are illustrated here, for simplification purposes. The initial ICER is surrounded by possible outcomes generated by the Monte Carlo simulation, represented by the blue circles. The area of uncertainty forms a cloud of point estimates.

FIGURE 18.

Example of uncertainty cloud on the cost-effectiveness plane.

Whether or not a strategy is cost-effective depends on the proportion of the point estimates generated by the Monte Carlo simulation that fall in the cost-effective part of the plane, under the ceiling ratio line. As seven point estimates in the example are below the given ceiling ratio, there is a 70% chance that the novel strategy is cost-effective and, therefore, optimal for decision-makers. However, this approach is only useful when there is one choice compared with a current practice comparator. When multiple options are available to decision-makers, another interpretation of the results is required.

The concept of incremental net monetary benefit (NMB) is appropriate and represents a single-figure summary of the outcome measures of cost and health benefits (QALYs). 134 It is derived by rearranging the formula for calculating ICERs and incorporating the ceiling ratio, λ.

The NMB is calculated by multiplying the ceiling ratio by the average incremental health benefits and, subsequently, deducting average incremental costs:

Employing the NMB simplifies the statistical interpretation of decision model results. The alternative with the highest mean NMB is the best option for decision-makers as it maximises health benefits under conditions of scarce resources. 135 This may not be the outcome with the highest probability of being the best decision, and the cause of the discrepancy is worth discussing. Halton et al. 136 shows exactly why this occurs via an example reproduced in Table 18.

These results show that treatment A is optimal only once out of the five simulations from the model parameters, giving a 20% chance it is the best choice and an 80% chance it is the wrong choice. Treatment B or remaining with baseline reveals higher chances of being an optimal decision, yet treatment A has the highest NMB at 126 and so is the best decision if the objective is to maximise health benefits under conditions of scarce resources. The reason for this apparent contradiction is that the distribution of NMBs is skewed. There could be a few large outlier values for treatment A that drag the mean NMB statistic to a higher value; in this case the value of 150 shown twice in Table 18 is fulfilling this. The only way to interpret the information in Table 18 is to say that treatment A is best, but there is great uncertainty in this conclusion with an 80% chance it is incorrect. In this case more research might be required to reduce this uncertainty. 135 If the NMB outcome were normally distributed, then the treatment with the highest probability of being cost-effective would also be the one with the highest NMB. 135 The generalisation of the decision rule to multiple comparators implies we choose the option with the maximum incremental NMB.

Other sources of uncertainty

In addition to parameter uncertainty, other types of uncertainty should also be considered: uncertainty in model structure or assumptions, patient heterogeneity and variability. An approach favoured by NICE to test model assumptions is called scenario analysis and can be performed as part of the PSA. In a scenario analysis, the model assumptions are varied and the PSA run for each change. The different scenarios are compared and the most robust one is selected. 137 Patient heterogeneity is a type of uncertainty that can be explained, for example, by different age or sex of patients. Variability, on the other hand, refers to random differences between patients that cannot be explained, that is, differences occurring by chance. 132 In order to deal with heterogeneity between patient groups, multiple acceptability curves can be employed, for example for different age groups or according to sex. Decision-makers can subsequently identify which infection prevention strategies are cost-effective for certain patient subgroups (e.g. 80-year-old males). A scenario analysis can also be performed for variability, such as different types of hospitals or levels of surgeon experience, but this depends on the information available.

Markov models

The mechanism of Markov models is explained in this section using a very simple example: a model with only two health states and a death state. Model health states are mutually exclusive and death is an absorbing state, because once an individual has entered the state, they must remain there. 138 As Markov models generally start with a cohort of people at risk of a disease, the example starts with a cohort of patients at risk of developing an infection. The model can capture the probability of patients remaining in the initial state (‘no infection’) or moving into one of the other health states (‘infection’ or ‘death’). These transitions occur within a previously defined time period, called a ‘Markov cycle’. 132,138 The length of model cycle depends on the research question but could be hours, days, months or years and can be run for any period of time, up to the full lifetime of a patient or cohort of patients. In each model cycle individuals have a certain probability of moving between health states, shown by the arrows in Figure 19.

FIGURE 19.

Possible transitions between Markov model health states in a given cycle.

In this example it is possible for an individual to remain in their current health state, move to the other one or die in any cycle. The fictional transition probabilities for this example are summarised in a transition matrix, illustrated in Table 19.

This table is to be read from left to right; for example, the probability of moving from ‘no infection’ (first column) to ‘infection’ (third column) is 0.15, representing 15% of patients. All transition probabilities sum to 1, as it is not possible to leave the model. As indicated by the ‘0’ in the matrix, transitions from the death state to other health states are not possible. The transition probabilities in this simplified example are constant but they can be time dependent and adjusted or varied for different cycles. Commonly, the probability of dying is adjusted as the mortality rate increases with age.

A Markov trace can be computed using the probabilities in the transition matrix for all model cycles. The Markov trace for the first five cycles of the example model is shown in Table 20. The model simulation starts with a cohort of 1000 patients free of infection. Over the five cycles, patients move between health states in accordance with the transition probabilities specified in Table 19.

The transitions in cycle 1 are straightforward: 800 patients remain without infection (1000 × 0.8), 150 patients developed an infection (1000 × 0.15) and 50 patients die (1000 × 0.05). Cycle 2 becomes more complex as patients are now moving from both the ‘no infection’ and the ‘infection’ states. The total number of people without infection in this cycle is the sum of patients moving from both health states in the previous cycle: (800 × 0.8) + (150 × 0.6) = 730. Typically, cohort simulations are run until all patients are absorbed by the ‘death’ state.

Model description and comparisons

The model takes the health services perspective. The final model structure is outlined in Figure 20 and was based on the following assumptions.

• No infection is the primary state for the model.

• Deep SSIs occur within 12 months following a THA.

• Patients remain in the ‘deep infection’ state until they receive treatment.

• Treatment options for patients with deep infection are usually debridement, antibiotics and implant retention (DAIR), one-stage revision or two-stage revision.

• A last resort option is ‘permanent resection’ which patients will remain in until death.

• Patients either experience successful treatment outcomes or treatment failure.

• Patients move into the ‘successful treatment’ state after treatment for infection.

• If treatment was unsuccessful, patients can undergo further treatment (i.e. move directly into treatment health states DAIR, one-stage revision, two-stage revision or permanent resection).

FIGURE 20.

Structure of an economic decision model: a Markov model.

The model begins with a cohort of infection-free patients who underwent THA. In each model cycle, patients can move between health states. Patients remain in the ‘no infection’ state, die or develop an infection. During the following cycles, all patients diagnosed with a deep infection within 12 months move to the health state ‘deep infection’. Patients stay in this health state until they receive treatment or die. The standard treatment options are DAIR, one-stage revision or two-stage revision. A less common treatment option is permanent resection of the prosthesis, used for severe cases. After the initial treatment for deep infection, patients are assumed to be infection free and move to the ‘successful treatment’ state. Patients remain in this health state unless further treatment is required because of recurring or persisting symptoms of infection. In this case, patients can undergo any of the four treatment options multiple times until the treatment is successful or until they are absorbed by death or permanent removal of the prosthesis. The model we have developed will be used to estimate the costs and health outcomes, measured by QALYs, arising from nine competing approaches to managing risk of SSI among THR.

NHS Hospital Episode Statistics

The NHS Hospital Episode Statistics (HES) is a data warehouse containing records of all patients admitted to NHS hospitals in England. It includes private patients treated in NHS hospitals, patients who were resident outside England and care delivered by treatment centres, including those in the independent sector, funded by the NHS. It processes over 125 million admitted patient, outpatient and accident and emergency records each year. The HES data are designed to enable secondary use (i.e. for non-clinical purposes). Each HES record contains a wide range of information about an individual patient admitted to a NHS hospital, including clinical information about diagnoses and operations; information about the patient, such as age group, sex and ethnicity; administrative information, such as waiting times, and dates and methods of admission and discharge; and geographical information, such as where patients were treated and the area in which they live.

A data extract was obtained from the Health and Social Care Information Centre (HSCIC) for a cohort of patients, with specific Office of Population, Censuses and Surveys, Classification of Surgical Operations and Procedures (OPCS) codes relating to THR for the financial years 2008–12. The HES records for the episode of operation were extracted and subsequently all other records of hospital episodes relating to this same cohort of patients were extracted within the given time period, including critical care data (see Appendix 5). These included patient identifiers such as NHS number, date of birth and provider number, as well as HSCIC identifiers ‘encryptedhesid’, ‘epikey’ and ‘susrecid’. The NHS number, date of birth and HSCIC identifiers were used to link the HES admitted patient care (APC) data to other data sets, including:

• critical care

• patient-reported outcome measures (PROMs)

• the Office for National Statistics (ONS) death registrations

• NJR data from the Healthcare Quality Improvement Partnership

• PHE SSISS data.

Office for National Statistics

The ONS is the UK’s largest independent producer of official statistics and is the recognised national statistical institute for the UK. It is responsible for collecting and publishing statistics related to the economy, population and society at national, regional and local levels. It also conducts the census in England and Wales every 10 years. The ONS plays a leading role in national and international good practice in the production of official statistics. An extract of ONS mortality data for the cohort of hip replacement patients described above was obtained from the HSCIC. The extract contained the variables listed in Appendix 5 and included the identifier ‘encryptedhesid’ to link these records to HES APC.

Patient-reported outcome measures

Patient-reported outcome measures data sets hold information on the effectiveness of care delivered to NHS patients, as perceived by the patients themselves. Patients rate their health by scoring the severity or difficulty in completing certain tasks or routine activities. The tools used are the EQ-5D index and the EQ-5D visual analogue scale. The Oxford hip score is also collected for those undergoing THR. PROMs data collection is co-ordinated by NHS England, but delivered by organisations such as hospital trusts that perform relevant procedures and the HSCIC. An extract of PROMs data was obtained from the HSCIC for the period 2009–12 for the cohort of hip replacement patients obtained from HES records. The PROMs extract included the variables listed in Appendix 5, in addition to the identifier ‘epikey’ which was used to link these records to HES APC.

Surgical Site Infection Surveillance Service

The SSISS holds data on SSIs reported by NHS hospitals and independent sector NHS treatment centres. All hospitals participating in national SSI surveillance are required to follow the surveillance protocol outlining the follow-up methods and case definitions. Each hospital collects data prospectively on all eligible patients in a surgical category over a 3-month period. The follow-up period is set at 30 days for non-implant procedures and 1 year for implant procedures.

Baseline risks for SSI and rates of revision, by patient characteristics, are available from the surveillance data. Acute care hospitals in England that participated in mandatory SSI surveillance from 1 April 2004 to 31 March 2014 have contributed data for patients who underwent hip replacement. An extract of hip replacement data was downloaded from the PHE SSISS database for the period 2008–12. The SSISS extract contained the variables listed in Appendix 5 and included the identifiers ‘patient name’, ‘NHS number’ and ‘date of birth’ to link these records to HES APC. When NHS numbers were unavailable, these data were obtained from the Demographic Batch Service.

National Joint Registry for England, Wales and Northern Ireland

The purpose of the NJR data collection is to provide an early warning system for patient safety issues. Furthermore, in an effort to continue to improve quality and cost-effectiveness of care, the high-quality data collected about joint replacement surgery in the UK are used to report on, and monitor, patient outcomes and to support research. NJR data from 2008 to 2012 were provided by the Healthcare Quality Improvement Partnership and included the variables listed in Appendix 5. The patient identifiers, NHS number and date of birth were used to link these records to HES APC.

Data linkage and pseudonymisation

The data sets were linked to the HES APC data set using the patient identifiers and HSCIC identifiers described above for each data set. Prior to analysis the data were anonymised to reduce the chance of disclosure by transforming NHS number and HSCIC identifiers to new identification numbers that bore no relationship to the originals. Data, such as date of birth, were ‘blurred’ by calculating age in whole years, rounded down, or time intervals between two incidents in units larger than days, generating a new code for providers/institutions to distinguish between but not identify institutions. All patient identifiers were then removed from the linked data set. Access to the data was restricted to a finite number of named individuals. The data linkages are shown in Figure 21.

FIGURE 21.

Data linkage of national data sets.

There are large numbers of data in HES and ONS data sets that have little value for this project, as they do not relate to THR. The identification of a cohort of patients by the HSCIC, using OPCS codes for THR, has reduced this to a minimum.

In Figure 21, the dark-green section (A) is where we focus our efforts, as these patients can potentially be linked across HES, PROMs, NJR, SSISS and ONS mortality data sets. The HES APC data included 316,208 distinct NHS numbers. The mid-green section (B) includes a subset of 276,858 patients of interest from HES who are not linked to PROMs data, as PROMs data were not available prior to 2009; however, they can potentially be linked to the NJR and SSISS. Whether or not they can be linked to ONS data will depend on their mortality status. The light-green section (C) is a subset of 3117 NJR and 57,017 SSISS patients who were unable to be linked to HES inpatient data. The dark-blue section (D) represents a hypothetical subset of patients whose data are present in PROMs and link to NJR and SSISS databases, but who cannot be linked to HES.

Only about 65% of patients in SSISS had a NHS number. This increased to 80% after tracing by the Demographics Batch Service. The light-blue section (E) represents patients with SSISS data for whom a missing NHS number could not be traced and, therefore, whose record could not be linked to other data sets. The NHS number could not be traced for 36,096 patients (19.23% of 187,673) patients in the SSISS data extract. The final linked data sets may, therefore, be compromised if missing data from patients in sections C, D and E were excluded for non-random reasons (e.g. if these patients were different in some way to the included patients in their demographics, treatment pathways or prognosis).

Transition probabilities

We simulated a cohort of 77,321 patients who had a primary THR in 2012. 11 A rate of deep infection to reflect risks under strategy T1 (no systemic antibiotics, plain cement and conventional ventilation) for a time period of 2.5 years was estimated by Lidwell et al. 111 This study of 8055 operations for replacement of hips (84%) and knees (16%) was undertaken in 19 hospitals in England, Scotland and Sweden between 1974 and 1979. The conditions relevant to T1 were met for 1161 of these procedures and there were 39 cases of deep infection. The rate of infection under T1 conditions for 2.5 years was therefore 3.4%. To adjust this for a 12-month follow-up appropriate to this model, we used information from Ong et al. 139 Ong et al. 139 evaluated the incidence of early-onset (< 2 years) and late-onset (> 2 years) deep joint infection after primary THA for the period 1997–2006. At 1 year post surgery 66.7% of all deep infections had been identified. This information was used to predict risks of infection for 12 months following primary surgery under conditions of T1 for the model.

We accounted for the fact that 32.9% of patients had a cemented hip replacement11 and so the remainder would have an uncemented procedure. The impact was to change the numbers or individuals in the cohort who would have incurred costs and gained health benefits from any strategy that included antibiotic-impregnated cement (T5–9).

The average risks among the cohort of 77,321 patients under conditions of T1 were used to predict 1887 cases of deep infection for 12 months following primary THR (95% uncertainty interval 1253–2621 cases). These patients then progressed through treatment pathways shown in Figure 20, which is a Markov model that simulates patients daily. We used the information described in Chapter 5 to estimate the probabilities of progressing through all model states for 5 years. A major advantage of accessing original data is that treatment probabilities are time dependent. A hypothetical cohort of patients will move through the model proportionally to the real patient data. It is not necessary to assume constant transitions, for example average yearly transitions. Accurate ‘real-world’ transitions between model health states can be used and the model cycle length adjusted to reflect this. Transition probabilities were calculated for each cycle for the occurrence of deep infection, first treatment with DAIR, one-stage revision, the first stage of a two-stage revision followed by the second stage two-stage revision and permanent resection, and further treatment if the initial treatment failed. Each simulation continued until every simulated patient had died. The means of the daily transition probabilities are shown in Table 21.

All patients started in the ‘primary’ state. Deep infections and their subsequent treatments were identified using the International Classification of Diseases, Tenth Edition (ICD-10) codes listed in Appendix 6, and the presence of a subsequent revision procedure listed in Appendix 7. All deep infections must have occurred within 12 months of the primary THR, in accordance with SSISS follow-up periods. The ICD-10 codes listed in Appendix 6 were used to identify infections leading to any of the treatment strategies. Revision resulting from deep SSI could be performed on one or more components of the prosthesis. ICD-10 organism codes were used to indicate the presence of infection but not the severity of infection, such as deep or superficial (see Appendix 8). The probability of death was taken from UK life table data for the years 2010–12 and was downloaded from the ONS website (see Appendix 9). Each patient’s starting age was randomly selected from the distribution of ages of primary THRs in the hip data. To acknowledge the uncertainty in the transition probabilities we randomly sampled probabilities using a beta distribution for each simulation. Hence our simulations include uncertainty about the parameter estimates as well as stochastic uncertainty. The plot of daily death probability by age is shown in Figure 22.

FIGURE 22.

Plot of daily death probabilities by age.

Effectiveness evidence

Estimates of the clinical effectiveness of the competing infection prevention strategies are available from the material included in Chapters 2 and 3. Descriptions of the strategies are provided in Table 22.

Probability ratios are the estimated probabilities of deep infection for strategies T2–9 divided by the probability of deep infection for T1. The simulations were conducted using Queensland University of Technology’s high-performance computer. Probability ratios of deep infection using T1 as the reference strategy are shown in Figure 23. The vertical lines in blue show a probability ratio of 1 (i.e. no change in probability of infection compared with T1). The probability ratio for deep infection for T2 compared with T1 has a mode near to 0.5.

FIGURE 23.

Probability ratios of deep infection using T1 as the reference strategy. (a) T2 vs. T1; (b) T3 vs. T1; (c) T4 vs. T1; (d) T5 vs. T1; (e) T6 vs. T1; (f) T7 vs. T1; (g) T8 vs. T1; and (h) T9 vs. T1.

The descriptive statistics for the probability ratios are shown in Table 23.

The probability ratio for deep infection for T2 compared with T1 has a mean of 0.46. Therefore, the average reduction in deep infection for this strategy is around half of the probability of deep infection for T1.

Membership of model states for 5 years after primary procedure

For the starting cohort of 77,321, the number of patients with deep infection for each of the treatments T1–9, and their subsequent occupancy of model states, is shown in Table 24.

These results are for the first 5 years following primary procedure and show that T6 prevented the most cases and had the fewest treatments.

We used the programing software R version 3.0.3 (The R Foundation for Statistical Computing, Vienna, Austria) for all analyses.

Costing treatments

The cost/price year for all estimates is 2012. The positive incremental costs of implementing each strategy are estimated from unit cost values for each component of the strategy. Unit costs were estimated for antibiotics used for prophylaxis, antibiotic-impregnated cement, laminar airflow systems and the use of exhaust body suits.

Conventional ventilation, laminar airflow and body exhaust suit costs

The only information found regarding the costs of different ventilation systems was published by Evans,8 who acknowledged laminar airflow was historically a high-cost technology. US data144 showed that, in 1974, the costs of installation were US$50,000, which is equivalent to US$232,853 in 2012 prices.

The technology has matured, and Evans8 estimates that the costs of construction and installation of an exponential laminar airflow system into a new operating room for 2011/12 range from US$60,000 to US$90,000. A lifetime of 5 years was assumed for the installation. Based on an exchange rate of US$1 to £0.66 (source OANDA;145 New York, NY, USA), this estimates annual capital costs between £7920 and £11,880. Using a typical caseload of five surgeries per day for a 5-day week, 50 weeks per year, the mean cost per case is between £6.33 and £9.50.

The costs of body exhaust suits were based on US prices as no UK data could be found, and prices were converted to Pounds Sterling as above. The Stryker Steri-Shield system (Stryker, Kalamazoo, MI, USA) was used as the case study146 and costs were included for the helmet without lights (US$3515), disposable hoods (US$86 for a pack of 32), power pack and cord (US$750), and charger (US$7000). Based on the same volume assumptions, the cost per surgical patient would be £7.72 (Table 32).

Incremental per patient cost of infection prevention strategies

The components of and total costs for each infection prevention strategy are shown in Tables 33–41. It was assumed that plain cement and conventional ventilation would have been used regardless, so these items attract zero cost.

Cost outcomes for each model state

The costs of a patient spending one cycle in each model state is estimated from the linked data set assembled for this project. Data for patients who went on to develop an infection following a THR were analysed with information relating to their THR and subsequent treatments for the infection, informing cost estimates.

Each patient in the data set has OPCS – version 4 (OPCS-4) codes assigned to each of the procedures they received. OPCS refers to the former Office of Population, Censuses and Surveys, originally responsible for the classification system, which has since been merged into what is now the ONS. The OPCS-4 classification system is designed to translate operations and surgical procedures into codes for statistical and epidemiological analysis. 147

The NHS mandatory standard for data collection is the OPCS-4 classification system. It is the required classification system for Admitted Patient Care Commissioning Data Sets and other national data sets required for secondary use.

Case-mix classifications are used to categorise patients based on their expected health-care resource use, primarily for the purposes of reimbursing providers. The NHS in England primarily uses Healthcare Resource Groups (HRGs). 148 HRGs are derived from OPCS-4 and ICD-10 codes in patient records and represent clinical groupings of patient activity.

Each patient’s OPCS-4 codes were assigned to a HRG using the HRG4+ Code to Group spreadsheet, which is available online. 147–149 It was possible to assign a HRG to each OPCS-4 code; however, each HRG has different levels based on diagnoses scores or complication and comorbidity information. In the absence of information that would allow us to specify each patient’s HRG level, it was assumed that patients had intermediate diagnoses and complication/comorbidity.

The HRG costs were taken from the NHS Reference Costs 2012 to 2013. 150 HRGs associated with trauma and orthopaedics for elective inpatients were used. Each model state was associated with an expected cost calculated as the weighted average of the cost of the treatments received (Tables 42–53).

Health benefits: attaching preference-based utilities to each health state

Despite having EQ-5D data from the PROMs data set, none of the scores was collected close enough in time to the health events included in the model. We sought an EQ-5D score within 14 days of the date of infection, or within 3 months of DAIR or one- or two-stage revision. In the absence of these primary data we used estimates from published sources, and these are shown in Table 54.

These estimates emerged from a reproducible review that was not published, but formed part of a doctor of philosophy thesis (see Merollini et al. 57). The relevant sections have been reproduced in Appendix 10, and the entire thesis is available online.

Cost-effectiveness evaluation

Simulations were made from all prior distributions specified in the model to make 1000 estimates of the change to total cost and QALYs, arising from a decision to adopt each of the treatments among the starting cohort of 77,321 patients. The most effective strategy is T6 and the least effective is T9, shown by the expected change to the number of cases of deep infection (Table 55).

The impacts on cost and QALY outcomes of choosing another treatment compared with T1, with all parameter uncertainties included, are shown in Figures 24 and 25, respectively.

FIGURE 24.

Change to total costs from a decision to adopt T2–9 compared with T1.

FIGURE 25.

Change to total QALYs from a decision to adopt T2–9 compared with T1.

Costs are lower than T1 for every other treatment except T9. Costs among patients in the cohort are reduced most for T6, with estimated savings of £8,325,749; however, T9 increases actual costs by £781,075. The uncertainty in these estimates is large, shown by the error bars in Figure 24. For all treatments the error bars rise above zero change in cost, indicating a chance that costs will increase as a result of a decision to adopt a treatment strategy different from T1. The probability that any of the treatment strategies will be cost saving is high (Table 56).

The change to total QALYs when other treatments are compared with T1 is shown in Figure 25.

This shows that QALYs for the cohort are likely to increase, compared with T1, regardless of the treatment strategy chosen. The mean change to QALYs is greatest for T6, at 147 QALYS gained, and least for T9 at 62 QALYs gained. There is large uncertainty in these estimates and the probability that treatment strategies will increase health benefits is moderate (Table 57).

The mean of the joint distributions of the change to costs and QALYs for each strategy, compared with T1, are shown in Figure 26.

FIGURE 26.

The mean of the joint distributions of the change to costs and QALYs for each strategy compared with T1.

The joint distributions for each comparison are shown in Figures 27–33. For Figures 26–34, T1 is at the origin of both axes.

FIGURE 27.

T2 compared with T1, change to cost and QALY outcomes.

FIGURE 28.

T3 compared with T1, change to cost and QALY outcomes.

FIGURE 29.

T4 compared with T1, change to cost and QALY outcomes.

FIGURE 30.

T5 compared with T1, change to cost and QALY outcomes.

FIGURE 31.

T6 compared with T1, change to cost and QALY outcomes.

FIGURE 32.

T7 compared with T1, change to cost and QALY outcomes.

FIGURE 33.

T8 compared with T1, change to cost and QALY outcomes.

FIGURE 34.

T9 compared with T1, change to cost and QALY outcomes.

Interpretation of results for decision-making

To help decision-makers understand these results, we calculated the expected incremental NMB from adopting each treatment compared with T1. Deriving this statistic is discussed in Chapter 4. It is a linear measure of the change to QALYs expressed in monetary units less the change to costs, and so it reflects a net profit or loss for society from any adoption decision.

Positive NMB suggests an adoption decision is a good outcome for society and negative values suggest the opposite. QALYs were valued at £18,000 each, in line with recent research. 118 In Figure 35 the mean incremental NMB for each treatment option compared with T1 is shown.

FIGURE 35.

Mean incremental NMB for each treatment option compared with T1.

Treatment 6 shows the highest expected incremental NMB at £10,964,883 for the cohort and reflects the large cost savings and the value of the incremental QALYs gained. T9 shows the lowest expected incremental NMB at £342,619. The large uncertainty in these conclusions remains, with error bars overlapping for every treatment and all the time crossing zero.

The probability that each treatment is cost-effective and has positive incremental NMBs is shown by Table 58.

The probabilities in Table 58 were calculated by comparing 1000 simulations of the NMB for each treatment strategy and choosing, for each simulation, the treatment that returned the highest value. T9 returned the highest NMB value in only 10 out of 1000 simulations, or 1% of the time. Therefore, we interpret this decision (T9) as having a 99% chance of being incorrect. T6 returned the highest NMB value 32% of the time, making this treatment the most likely to be cost-effective. T6 also prevented the greatest number of cases of deep infection (see Table 55), saved the most costs and generated the greatest QALY gains (see Figures 24 and 25).

Summary and conclusions

We used a MTC analysis to compare the effectiveness of nine competing strategies for reducing the risk of infection following THA and applied the results to a Markov model of the treatment pathways for patients who develop an infection following THA. This enabled an assessment of the costs and health effects of the nine treatment strategies in a single coherent framework. To evaluate uncertainty in the estimates of effectiveness and parameters in the Markov model, a Monte Carlo simulation of prior uncertain parameters was undertaken. The output of the simulation gives an estimate of the probability of cost-effectiveness of each of the treatment strategies and can be used as a decision-making tool in the context of the resource-limited NHS.

The results show that, compared with T1, T6 is the strategy that prevents the greatest number of deep infections following THA. T6 is a treatment of systemic antibiotics, antibiotic-impregnated cement and conventional ventilation. It prevents an extra 1481 cases of deep infection and leads to the largest annual cost savings of –£8,325,277 (95% uncertainty interval –£17,981,040 to £5,765,832). The mean gains to health benefits measured by QALYs are greatest at 147 (95% uncertainty interval –585 to 1157). T6 is the optimal decision with the highest probability of being cost-effective, at 32%. T2 competed closely with T6, delivering cost savings of £7,226,732 and QALY gains of 101. The NMB of a decision to adopt T2 was second highest, at £9,039,481. The only difference between T6 and T2 was the removal of antibiotic-impregnated cement for the cemented procedures. This suggests that adding antibiotic-impregnated cement is worthwhile. Two poor treatments for cost and clinical effectiveness outcomes were T9 and T7, which involved adding in laminar ventilation and a body exhaust suit. The addition of laminar air only (T7) drives costs upwards by £5,032,528 each year and reduces QALYs by 23. The further addition of a body exhaust suit increases costs by £9,106,352 and reduces QALYs by 84. This happens because laminar ventilation and body exhaust suits were found to increase risk of infection.

T6 dominates all other options by both cost and effect. These findings have to be considered against the limitations of the research method, data used and large uncertainties shown.

Limitations and strengths

As a relatively new evidence synthesis tool, MTC provides a powerful analytical tool that enables all available evidence, direct and indirect, to be included in an evidence base, synthesised and ranked according to its relative efficacy with associated probability statements. 78 Like any analytical model, MTC is based on a range of assumptions and the use of non-informative priors within a Bayesian framework. A key assumption of the MTC model used is that intervention effects are exchangeable across all trials in the network. In other words, both direct and indirect sources of evidence in each pairwise comparison are assumed to estimate the same treatment effects across the network of all trials. However, the assumption is appropriate only provided that the baseline characteristics of patient populations and intervention protocols are homogeneous. Variation in these parameters inevitably poses certain threats to both the internal and external validity of MTC models. In order to assess and adjust for the risk of estimation bias, we assessed the model fit and checked the consistency of intervention effect estimates from direct and indirect evidence sources. We conducted a metaregression on previous surgery and follow-up duration as potential sources of heterogeneity. We also estimated and adjusted for bias associated with studies with mixed quality. We were limited to English-language literature, and this may cause a bias in the results, but we were unable to describe the size of it or direction it might take. There were limited data available, and the MTC model was unable to adjust for potential confounders such as case mix, particularly patient comorbidity, in different hospital settings, different types of laminar airflow systems used and temporal changes in clinical practices, infection control technology and patient profiles that may have taken place over the past several decades. Despite the inherent limitations of MTC models and the limited number of studies available, the current evidence using MTC has the potential to change our understanding about the effectiveness of infection prevention strategies or a combination of multiple infection prevention strategies. This in turn may help contribute to the identification and implementation of evidence-based best infection prevention strategies for reducing the risk of, and preventing, THR-related SSI.

The baseline risks of infection for T1 were based on the Lidwell et al. 111 study, conducted in the 1970s and reporting 39 cases of deep infection among 1161 patients who had surgery under conditions that were closely correlated with T1. The 3.4% rate of infection describes 2.5 years, yet the model we ran counted infection outcomes for 12 months only. The rate of 3.4% was reduced based on US data by Ong et al. ,139 who showed that 66.7% of all infections they found over a 9-year period presented in the first 12 months. Whether or not these assumptions lead to an estimate that reliably describes the baseline risk of infection for T1 in NHS hospitals is unknown. If this estimate is inaccurate, however, it will not impact on the conclusions drawn, as the incremental change to outcomes is the key for decision-making. Absolute values are not particularly relevant.

A large effort was made to include parameter uncertainties in the model, and these large uncertainties can be seen in the results tables and figures, especially Figures 24, 25 and 35. We assumed that there was no structural uncertainty arising from the design of the Markov model. We did, however, spend time consulting with orthopaedic surgeons and infection prevention experts. A process was completed to identify the events most likely to impact cost and health outcomes while attempting to keep the model as simple as possible.

A further limitation of the model relates to the data used to inform patient outcomes. PROMs data could not be linked to the relevant time points for primary THA, deep infection or revision procedures. As a result, we made use of utility values from a previous review of health outcomes relating to THA, to attach health benefits to each state in our Markov model. These had been used in a previous model, in an Australian context. There are no strong reasons to believe that these values are not appropriate to the NHS setting. A further limitation is the scope of costs included. The data sets that were available for this project describe only the acute sector, and the costs of primary care, private out-of-pocket expenses and personal social services costs were excluded. We do feel that the costs of revisions of the primary hip and other surgical interventions were appropriately quantified and are likely to be the largest component of total costs.

Perhaps the biggest limitation of our model is the uncertainty regarding the effectiveness of the different treatment options. The best possible data were sourced and appropriate methods used to interpret them. A prospective randomised trial to address the research questions addressed here would be impossible; the cost would be prohibitive and it would not be passed by an ethics committee.

A major strength of this analysis, therefore, is the ability to compare nine different treatment strategies for the prevention of infection, together, using currently available evidence, without the need for a costly and time-consuming clinical trial. Large numbers of patient-level data were available, which are routinely collected in the NHS. This enabled a thorough analysis of THA patient demographics and treatment strategies as a result of deep infection following THA. Similarly, effectiveness data from a number of trials were available and were able to be synthesised using MTC techniques into one comparative analysis of all relevant treatment strategies. In addition, modelling studies have significant advantages over RCTs and other clinical studies in that they enable estimates of long-term cost and effectiveness of treatments. Similarly, findings from modelling studies are generalisable as they are not restricted to controlled trial criteria, and real-world populations are considered.

Contributions of authors

Nicholas Graves wrote the grant, designed the methods, managed the research and drafted sections of the report.

Catherine Wloch managed the process of data linkage from the NHS databases used.

Jennie Wilson contributed to study design, data collection, analysis and drafted sections of the report.

Adrian Barnett undertook the modelling and wrote the results sections.

Alex Sutton designed the methods and managed the research.

Nicola Cooper designed the methods and managed the research.

Katharina Merollini designed the methods and managed the research.

Victoria McCreanor undertook data collection and wrote sections of the report.

Qinglu Cheng undertook data collection and wrote sections of the report.

Edward Burn undertook data collection and wrote sections of the report.

Theresa Lamagni managed the process of data linkage from the NHS databases used and wrote sections of the report.

Andre Charlett designed the methods, managed the research and managed the process of data linkage from the NHS databases used.

Publication

Zheng H, Barnett AG, Merollini K, Sutton A, Cooper N, Berendt T, et al. Control strategies to prevent total hip replacement-related infections: a systematic review and mixed treatment comparison. BMJ Open 2014;4:e003978.

Data sharing statement

Data can be obtained from the corresponding author after approvals have been gained from the data custodians (PHE and ONS).

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health.

MEDLINE search strategy and output

Date of search: August 2011.

Date range searched: January 1966 to June 2011.

Cumulative Index to Nursing and Allied Health Literature search strategy and output

Date of search: August 2011.

Date range searched: January 1966 to June 2011.

The Cochrane Central Register of Controlled Trials search strategy and search output

Date of search: August 2011.

Date range searched: January 1966 to June 2011.

EMBASE search strategy and search output

Date of search: August 2011.

Date range searched: January 1966 to June 2011.