Sacral nerve stimulation versus the magnetic sphincter augmentation device for adult faecal incontinence: the SaFaRI RCT

Current treatment options

Current treatment strategies for adult FI are summarised in the National Institute for Health and Care Excellence (NICE) 2014 guidance. 2 All patients should undergo a thorough history and physical examination to determine the nature and severity of the problem, and to identify a probable aetiological cause. Initial management consists of a combination of patient education, dietary modification and antidiarrhoeal medication. If this is unsuccessful, investigation in the form of endoscopic visualisation of the colorectum, anorectal manometry (pudendal nerve testing optional), and endoanal ultrasound is performed to further characterise the underlying disorder and inform treatment options.

Rationale for the SaFaRI trial

New technologies have often been introduced into clinical practice without rigorous evaluation of safety, efficacy and cost-effectiveness. Objective assessment has been overlooked because of the intrinsic appeal of new innovation, the need to be a part of a ‘pioneering group’ or, worse, because of the financial incentives from industry. Once introduced, low-grade observational evidence is often used to keep practices going. As a result, it has often been easier to ‘stop them starting’ than to ‘start them stopping.’18 Ideally, any new technology introduced into clinical practice should be simultaneously evaluated, and in most cases the best way of doing this is by randomised comparison with an already established technique. The National Institute for Health Research Horizon Scanning Centre (NIHR HSC) was established to ‘supply timely information to key health policy and decision-makers within the NHS about emerging health technologies that may have a significant impact on patients or the provision of health services in the near future.’19 In May 2012, the NIHR HSC reported on the FENIX Continence Restoration System (FENIX MSA) and concluded that ‘in order to determine its potential place in the pathway of care for FI larger long term studies of the safety, effectiveness and cost-effectiveness of FENIX in comparison to existing treatments are needed.’20 Therefore, although FENIX MSA may have a role to play in the treatment of FI, the evidence was not robust enough to support its widespread adoption.

The SaFaRI trial was thus designed to undertake a rigorous, prospective assessment of the new FENIX MSA as it was adopted into the NHS. The aim was for reliable data, collected independently from commercial interests, to be made available on the safety and efficacy of the device. This would include information on safety, efficacy, QoL and cost-effectiveness. Important information would be gained on the costs associated with the device, enabling the ICER per QALY to be determined. This would allow health-care providers to make informed decisions about value for money and future provision of the technology.

Sacral nerve stimulation was chosen as the comparator to FENIX MSA as SNS is currently the preferred, and NICE-recommended,2 surgical intervention for FI that is resistant to conservative therapies; the NIHR HSC report19 from May 2012 also identified SNS as the preferred comparator for any randomised comparison with FENIX MSA.

Furthermore, the SaFaRI trial was designed to collect additional, important data about SNS. SNS is a costly yet effective treatment for FI; however, concerns have been expressed about the lack of efficacy when analysed on an intention-to-treat basis and the loss of efficacy on longer-term follow-up. The SaFaRI study provided an additional opportunity to clarify the indications for SNS and the indicators of success.

The opportunity also presented itself to comprehensively document, for the first time, the treatment and associated costs for patients for whom either SNS or FENIX MSA is not successful. In effect, these patients would provide comparative, longitudinal data of the patient pathway where FENIX MSA or SNS is either unsuitable or unavailable.

In addition to the costs detailed above, the health economics would provide data on the short- and long-term cost-effectiveness of FENIX MSA compared with SNS. Within the analyses, use of two measures of health-related QoL to produce QALYs, the Short Form questionnaire-12 items (SF-12) together with the EuroQol-5 Dimensions (EQ-5D), would allow assessment of the sensitivity of the EuroQol-5 Dimensions, five-level version (EQ-5D-5L) to detect changes in FI, which is to date unproven. The disease-specific questionnaire chosen to assess QoL, the FIQoL questionnaire, collects important information on many social and psychological aspects of FI (shame, depression, enjoyment, etc.). These aspects of FI have received little previous recognition in the literature and remain poorly defined.

Aim and objectives

The overall objectives of the study were to:

• determine the short-term safety and efficacy of FENIX MSA and SNS in adult FI

• assess FENIX MSA and SNS in terms of impact on QoL and cost-effectiveness.

Trial design

SaFaRI was a prospective, UK multisite, parallel-group, randomised clinical study investigating the safety and efficacy of the FENIX MSA for adult FI. The comparator was SNS, a preferred treatment recommended by NICE for the treatment of FI that is resistant to conservative therapies. 2 Participants were randomised on a 1 : 1 basis to receive either FENIX MSA or SNS.

Prior to randomising participants, all participating surgeons had to have performed a minimum of 10 permanent SNS implantations, observed a minimum of one FENIX MSA procedure and performed two FENIX MSA procedures under proctorship.

A registration phase was incorporated into the study design to enable surgeons without the required FENIX MSA experience prior to study participation to gain the relevant experience within the scope of the study. Within the registration phase, the first two eligible patients providing consent were registered to the study under the training surgeon’s name and received FENIX MSA implants (there was no randomisation in the registration phase); these two operations, which were performed under proctorship, were considered study training cases and were not included in the main study/main trial analysis. Once the required FENIX MSA experience had been obtained, the surgeon could progress to the randomisation phase.

The trial received national ethics approval in the UK. The trial conduct was overseen by an independent Trial Steering Committee (TSC) and Data Monitoring and Ethics Committee (DMEC). The trial had public and patient involvement during the trial design phase and throughout the course of the study. The trial was registered on the International Standard Randomised Controlled Trial Number (ISRCTN) register (16077538).

Participants

The inclusion criteria were as follows:

• aged ≥ 18 years

• able to provide written informed consent

• FI for > 6 months

• incontinent episodes of ≥ 2 per week

• suitable candidate for surgery, as judged by the operating surgeon

• suitable for either FENIX MSA or SNS (unless the patient was being registered as a training case, in which event they only needed to be suitable for the FENIX MSA)

• anal sphincter defect < 180° as documented on endoanal ultrasound scan

• able and willing to comply with the terms of the protocol including QoL questionnaires.

The exclusion criteria were as follows:

• previous interventions for FI (i.e. SNS, FENIX MSA or ABS) (unless the patient was registered as a training case, in which event they could have had previous interventions for FI)

• chronic gastrointestinal motility disorders causing incontinence due to diarrhoea

• obstructed defaecation, defined as an inability to satisfactorily evacuate the rectum [it was recommended that the Obstructed Defecation Score (ODS) was calculated and was ≤ 8 for trial inclusion]

• anal sphincter defect ≥ 180°, as documented on endoanal ultrasound scan

• an electric or metallic implant within 10 cm of anal canal

• co-existent systemic disease (e.g. scleroderma) affecting continence

• active anorectal sepsis

• diagnosis of colorectal or anal cancer within previous 2 years

• external rectal prolapse

• significant scarring of the anorectum that, as judged by the treating surgeon, would prohibit FENIX MSA implantation or put the patient at high risk of implant erosion

• pregnancy (it was the local surgeon’s responsibility to assess pregnancy in women of childbearing potential)

• immunocompromised, including haematological abnormalities and treatment with steroids or other immunomodulatory medicines

• congenital spinal abnormalities, preventing SNS implantation

• known requirement for future magnetic resonance imaging surveillance, which would be contraindicated in the presence of metallic implant

• suspected or known allergies to titanium.

Interventions

Preoperative investigation and preparation were as per institutional protocol, which included, as standard practice, visualisation of the colorectum (flexible sigmoidoscopy as a minimum), anorectal manometry (pudendal nerve testing optional) and endoanal ultrasound.

Summary of protocol changes

A summary of all substantial amendments to the SaFaRI protocol can be found in Table 1.

Early trial closure

On 23 March 2017, the SaFaRI trial team received formal notice from Torax Medical (the manufacturer of the FENIX MSA device) that the decision had been made to suspend the commercial sale of the FENIX MSA device in the UK and other European countries for ‘strategic and business reasons’. As a result of this, recruitment into the SaFaRI trial, regrettably, had to cease with immediate effect.

Following the withdrawal of the FENIX MSA device and with approval from the Research Ethics Committee, all patients who had consented for the trial up to 23 March 2017, and had been randomised to the FENIX MSA arm but had not yet had surgery, were given the chance to have the FENIX MSA device implanted if they still wished to proceed with the operation as randomised. Any patients not wishing to undergo a FENIX MSA implantation, in the light of the fact that the device had been withdrawn, were offered alternative FI treatment as per local standard practice (this included the option of undergoing the alternative study intervention, SNS).

In total, 99 participants were randomised into the SaFaRI trial and 23 participants were registered as FENIX MSA training cases. A consequence of recruiting only 99 patients out of the target sample size of 350 is that the study is substantially underpowered to detect differences between the treatment arms, in particular with respect to the primary end point. Although the recruitment total was significantly less than the originally planned sample size of 350 participants, it was felt that continuing to follow up all randomised participants until the end of the planned follow-up period (i.e. 18 months post randomisation) would still provide valuable data that, at the very least, could provide some initial evidence; at the time of trial closure, SaFaRI was also the largest randomised trial of SNS to date.

The National Institute for Health Research was amenable to the trial team’s proposal to continue with the planned follow-up period, and, as a result, all patients who had been randomised into the SaFaRI trial continued to be followed up until 18 months post randomisation. The last participant’s final follow-up took place in September 2018.

End points

Sample size

A total of 350 participants were required to detect at least a 20% difference in the percentage of successes at 18 months post randomisation (where success was defined as device in use and ≥ 50% CCIS improvement from baseline) between FENIX MSA and SNS at a 5% level of significance, with 90% power, assuming approximately 40% success in the SNS arm and allowing for 20% loss to follow-up. However, the number of patients recruited was 99.

Randomisation

Following confirmation of written informed consent and eligibility, patients were randomised into the trial by authorised members of staff at the trial sites. Randomisation was performed centrally using the Clinical Trials Research Unit (CTRU) automated 24-hour telephone randomisation system. Authorisation codes and personal identification numbers (PINs), provided by the CTRU, were required to access the randomisation system.

Participants were randomised on a 1 : 1 basis to receive either FENIX MSA or SNS, and were allocated a unique study number. A computer-generated minimisation programme that incorporated a random element was used with the following minimisation factors:

• treating surgeon

• participant sex (male or female)

• severity of incontinence (CCIS)

  • mild to moderate: CCIS ≤ 10 points

  • moderate to severe: CCIS > 10 points.

• degree of anal sphincter defect on endoanal ultrasound

  • no anal sphincter defect

  • anal sphincter defect ≤ 90°

  • > 90° anal sphincter defect < 180°.

Blinding

The study was not blinded to participants, medical staff or clinical trial staff because of the difference between the two devices being compared (SNS treatment requires a temporary implant followed by a permanent implant if successful and involves patient input to function).

Statistical methods

Unless otherwise stated, all analyses were prespecified and conducted on the intention-to-treat population (i.e. all randomised participants were categorised into treatment groups based on their randomisation regardless of what treatment they subsequently received). All hypothesis tests were two-sided and conducted at the 5% level of significance. Estimates and their corresponding 95% confidence intervals (CIs) and p-values are presented for fixed effects. For all end points, missing outcome data were assumed to be missing at random, and the treatment effect was therefore estimated via maximum likelihood estimation using all participants with non-missing outcome data for non-longitudinal end points (this is referred to as a complete case analysis for the remainder of the report). All models were fitted using SAS^® v9.4 (SAS Institute Inc., Cary, NC, USA). (SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ^® indicates USA registration.)

Sensitivity analysis: additional covariates

Owing to the small number of ‘successes’, performing the sensitivity analysis with additional covariates that was described in the analysis plan was inappropriate.

Longitudinal analysis

As the primary end point was measured at 6, 12 and 18 months post randomisation, a longitudinal analysis using the data at each time point was performed. The model did not adjust for the stratification factors because of the added complexity and the number of missing data. In addition, the stratification factors caused the model to fail to converge.

The fixed effects of the model can be seen in Table 8. The estimated random effect caused by the operating surgeon was 1.52 (SE 0.99) and the estimated random effect caused by within patient measurements was 1.19. The model does not show a significant difference between the treatment arms (p = 0.20) or in change over time (p = 0.17), although, again, it is worth noting that the estimates have large SEs due to the small sample size.

The model results can be seen in Figure 4.

FIGURE 4.

Longitudinal model results.

Health utility and quality-adjusted life-year calculation

Health-related quality-of-life (HRQoL) data to enable QALY calculations were collected during the trial at baseline and at 6, 12 and 18 months post randomisation. Data were available on the EQ-5D-5L27,28 and the SF-12 [Short Form questionnaire-6 Dimensions (SF-6D)]. 29

The primary analysis is based on the EQ-5D-5L direct valuation. 30 The secondary analysis estimated QALYs based on the SF-12 (SF-6D) instrument. A supplementary analysis reported results based on the EQ–5D-5L crosswalk (to the EQ-5D-3L). 31

The QALY calculation was estimated on an area under the curve approach, adopting the following assumptions:

• All follow-ups were carried out at the exact, prespecified time period (0, 6, 12 and 18 months).

• A patient who died with their last EQ-5D-5L observation as positive had a linear fall in HRQoL from this point until death.

• A patient who died with their last EQ-5D-5L observation as negative had this constant level of HRQoL from this point until death.

Notation: E₀ = baseline utility, E₆ = utility at 6 months, E₁₂ = utility at 12 months, E₁₈ = utility at 18 months, t = duration in each health state in days. If EQ-5D-5L was present at baseline, 6, 12 and 18 months’ follow-up, QALYs were calculated by:

Resource use and costs

The perspective for the costs was the health and social care provider (i.e. wider costs, such as productivity loss or out-of-pocket expenses, were not included).

We used a combination of patient-reported data, CRF data capture and published estimates to derive costs during the evaluation. The general approach, individual unit costs and costing assumptions were verified with and refined following input from the lead clinician on the trial. The cost categories and overall approach to establishing the cost of each intervention over the trial period and the associated source of resource use/costs is as follows:

1. device costs – taken from publications and online resources

2. procedure costs – taken from national tariffs and online resources

3. medications – based on patient report

4. primary/social care – based on patient report

5. secondary care after the initial procedure – generated by attributing costs to explants, complications and further treatments/surgery.

The strategy for secondary care resource use capture deviates from the strategy that was planned. We had originally aimed to request NHS digital data to capture this aspect of resource; however, as a result of the early stopping of the trial, the associated reduction in research funding and the resultant limited sample size, this was considered to be impractical and was not pursued. This decision was made following consultation with the trial team. Since secondary care costs (hospital visits and stay) were not captured directly elsewhere in the trial, we elected to capture them via the length of stay of the original procedure and by costing subsequent explants, complications and surgeries.

The combined device and procedure costs are included in Table 22 and include the device-specific cost and a day-case rate for insertion of a generic neurostimulator for treatment of FI costs. These costs are fixed, except for the cost of the hospital stay. Any nights spent in hospital (including the first night) were costed using the unit cost shown for an excess bed-day. Data collected in the trial (e.g. relating to theatre time during procedure) may have allowed for a more granular approach to costing the procedure; however, given the small sample sizes, it may be prone to undue bias from outliers; thus, we opted to use NHS reference cost tariffs.

An operative form was completed for each patient following the procedure. The form captures operation details, including which procedure the patient received (temporary SNS and/or permanent SNS, or FENIX MSA) and any intraoperative complications that may occur. The form also reports whether patients receive a local, general or spinal anaesthetic, and whether or not the procedure has been completed. Furthermore, a 2-week postoperative review was issued for each patient to gather information on whether or not patients progress to permanent SNS, if they were randomised to receive temporary SNS, whether or not the device they received is still in situ and a reason provided if the device has been explanted. This discharge form was also used to record whether or not patients who have been discharged receive further surgery or suffer from any postoperative complications. Follow-up forms were completed by the local research teams at 6, 12 and 18 months following randomisation and asked for similar patient information. Individual complications and further surgeries were costed using the unit costs given in Appendix 1, Tables 39 and 40, respectively. With the exception of explantation, we assumed that the costs of any intraoperative complications were captured in the length of stay of the initial procedure.

At baseline and at 6-month, 12-month and 18-month follow-up, patients were provided with a specially designed form to report on their (primary and social) health-care resource use in the past 3 months: general practitioner (GP), practice nurse, district nurse, physiotherapist, occupational therapist, counsellor in the GP surgery visits, clinic (non-hospital) by telephone/e-mail or at home. They were also asked if they received support from social services, including meals-on-wheels deliveries, laundry services, care workers/help at home or social worker visits, and the frequency of this support. Furthermore, they were asked to provide information on the use of community/residential services, including any visits and overnight stays at convalescent homes, nursing homes and day centres. Patients were also asked to note what medication relating to their FI they were taking on a regular basis. Unit costs for medications are listed in Appendix 1, Table 41, and for primary care/social care in Appendix 1, Tables 42 and 43. We assumed that any follow-up care costs associated with the interventions are captured in the self-reported primary care use data. The 3-month recall data were multiplied by two to obtain 6-month values.

In a majority of cases, unit costs were based on national resources such as the Personal Social Services Research Unit (PSSRU) report, NHS Reference Costs and the Electronic Market Information Tool (eMIT). Elsewhere, we used other web resources and previous publications as the basis for unit costs for health-care resources. All costs are presented in Great British pounds and the price year is 2018 (costs are inflated to 2018/19 prices where necessary using the Consumer Price Index for health).

Analysis

The main analysis was based on an intention-to-treat principle. The trial analysis produced ICERs over the trial period (18 months). We used seemingly unrelated regression34,35 to account for the expected correlation between costs and QALYs while controlling for any baseline imbalances. Although we had planned to use the same covariates as the statistical analysis, analyses were run with no adjustment, and subsequently adjusting for baseline EQ-5D-5L and (primary/social care) costs given the data limitations. The regression models estimated costs and QALYs that will allow the calculation of ICERs.

Our primary analysis incorporated multiple imputation (MI) to impute missing data; however, we also conducted a complete-case analysis. Individuals with missing EQ-5D-5L items were not allocated a utility index score. Where missing values could not be dealt with in the manner described above (see Health utility and quality-adjusted life-year calculation), we conducted MI as part of the model estimation procedure. Costs were imputed for each individual at each time point for the following categories: (1) total primary care and social care costs, (2) total secondary care costs and (3) total medication costs.

We present a range of sensitivity analyses to explore the impact of the assumptions made in the base case analysis on the cost-effectiveness estimates. These include the methods used to assess utility in the trial and model covariates. We also conducted non-parametric bootstrapping to calculate the probability that FENIX was cost-effective at the trial end.

The final cost and QALY estimates at 18 months were added to the beginning of the decision model time horizon, and additional costs and benefits were estimated in the model from 18 months over a lifetime.

Model type, cycle length, structure and health states

The model is a Markov model describing a simplified patient pathway from the end of the trial period (18 months) over a lifetime in terms of discrete health states. The average age of the patients at the model start reflected that of the trial participants (62 years). Patients move between these health states at the end of every model cycle (in this case, 1 year) according to prespecified transition probabilities.

The structure of the decision model is shown in Figure 6. The structure was informed by other models used in this population32,37 and was finalised following input from the lead clinician.

FIGURE 6.

Decision model structure.

In the model, we defined the health states as being related to effectiveness according to the percentage improvement on baseline CCIS (100–76% improvement, 75–51% improvement, ≤ 50% improvement) and whether or not the device is still in situ (device explant). ‘Better’ health states (i.e. higher percentage of improvement) are associated with lower costs and higher QoL. Although we did not a priori expect the interventions to have an impact on survival, because the model is over a lifetime horizon, we also modelled background, all-cause mortality (dead health state).

Each health state is associated with a specific average cost and HRQoL (utility value). A hypothetical cohort of FI patients for each treatment arm was distributed between these health states at model start with the distribution reflecting that of the trial sample and intervention effectiveness/functionality at 18 months. These patients moved between health states at the end of each model cycle, with the likelihood of this movement determined by transition probabilities that reflected treatment effectiveness – more specifically, loss of effectiveness over time – and mortality rates.

We assumed that 18 months after initial implant, effectiveness could only deteriorate, and thus patients would be transited downwards in terms of percentage improvement from baseline. We also assumed that once explantation had occurred, another device (either SNS or FENIX) would not be implanted. Patients receiving FENIX initially are unlikely to receive another FENIX device following explantation owing to clinical viability. While it is possible that initial FENIX implants could be replaced by SNS devices and that initial SNS devices could be replaced by either another SNS device or a FENIX device, the patient numbers on these pathways would be relatively small. Incorporating these possibilities would require additional modelling complexity, and the additional burden of data requirements (e.g. effectiveness of second implants) was unlikely to be met, and thus additional assumptions would be required. We also included in the explant health state those in the SNS group who did not have a permanent implant. The explant health state was therefore capturing ‘conservative management’ for those without a device in situ.

The durability of the FENIX device is unknown and the device will in theory continue to function in the long term until a fault or infection occurs. However, the SNS device has a finite battery life (approximately between 5 and 7 years), at which point explantation, battery replacement and implantation is required. We reflected this in a submodel for SNS. After battery replacement, patients were subject to the same distribution among effectiveness categories as observed in the trial arm at 18 months (rather than immediately post procedure) for those that had the permanent device implanted. We did not include another battery replacement at 14 years in the base-case model because of the added modelling complexity and the small proportion likely to have a device in situ but we tested this in simplified sensitivity analyses. Other modelling assumptions are set out in the parameter tables.

Parameter values: transition probabilities

The decision model required a set of parameter values that defined the movement of the cohort around the model (transition probabilities are shown in Table 23). Where possible, the transition rates for the model were estimated from the randomised controlled trial (RCT) data (observed data rather than imputed). We used the end distribution of trial patients per arm across percentage improvement groups as the starting proportions in these groups at model start. We used trial data on transition rates between these percentage improvement groups during the trial to specify a rate beyond 18 months. As a result of the limited sample size, we were unable to do this in a robust way per intervention arm, so we assumed these rates to be the same across arms.

We applied parametric survival curves to the time-to-explant data from the trial and extrapolated this beyond 18 months. The survival analysis indicated no increase in hazard over time, and thus we estimated a constant hazard rate using the exponential distribution in Stata^® version 15 (StataCorp LP, College Station, TX, USA). There were no explants in the SNS group, but a clinical expert felt that this was an artefact of the sample size; hence, we used the same explant rate for both intervention groups. Following input from the same clinical expert, we assumed that survival in the FI population was not significantly different from survival in the general population and, thus, used Office for National Statistics mortality data38 to model this. We also assumed that neither the interventions nor the percentage improvement had an impact on survival.

Parameter values: health state utilities

Economic evaluations are designed to inform resource allocation decisions, thus the model used the QALY outcome measure as prescribed by NICE. The estimation of QALYs requires the production of utility weights for health states; these data were derived from the RCT (Table 24). We used linear regression predicting EQ-5D-5L utility weights and health state groups as an explanatory categorical variable. The regression used multiple observations per person and robust SEs to reflect this approach.

As the interventions were assumed not to influence survival, it was not considered necessary to adjust utility values for age over the time horizon. There was no utility decrement for battery replacement as this was assumed to be a ‘tunnel state’ straight to percentage CCIS improvement groups (i.e. patients did not stay in this state for a sufficient period to incur substantive HRQoL loss).

Parameter values: costs

We generated health state costs from the trial data (see Table 24). As we expected, costs were at their highest immediately after the initial procedure and then fell over time (with the exception of the battery replacement for SNS). We used the 12-month and 18-month resource use data to estimate ongoing health-care costs in the model. As it was unclear whether or not the trial forms captured ongoing (conservative management) costs for the no device group, we generated a monthly cost following discussion with the lead clinician (detailed in Table 24) and tested this in a sensitivity analysis. We used the full set of cost data to estimate costs immediately post battery replacement, and this was anticipated to capture any complication and additional surgery costs. The explantation of a device or replacement of a battery led to the procedure costs outlined in Table 24.

Relevant costs for the evaluation include the FENIX device/SNS acquisition and procedure (including length of hospital stay), costs of treating adverse events/complications, costs of device explants and other surgery, and (for SNS) the cost of battery replacement. These are in addition to the health state costs that were obtained from the trial data. All costs relating to other surgeries, explants and complications within 18 months were observed/captured in the trial data, and thus were not modelled here. After 18 months, complication costs were assumed to be captured within the other health states.

Unit costs were obtained from national sources including the PSSRU,41 the British National Formulary42 (BNF) and the NHS Reference Cost33 database. Internet searches were conducted for other cost elements.

Validation

We validated the model by checking the face validity of the model structure and parameter values with other health economists and with the lead clinician. We tested the internal validity of the model by checking that model outputs followed expectations when changes were made to certain parameter values (e.g. that the cost-effectiveness of FENIX improved following an increase in the SNS device cost or reduced following an increase in the probability of explant in the FENIX arm).

Analysis

The average costs and QALYs accrued during the trial period were inputted at the start of the model, with further costs and benefits added to these over the model horizon. We generated a deterministic ICER per QALY gained and a net monetary benefit (NMB) estimate at the model end. We tested these estimates using prespecified sensitivity analyses to identify drivers of cost-effectiveness. We specified distributions for certain parameter values (denoted in the parameter tables) and ran a PSA where random parameters were selected from these distributions and inputted into the model over 10,000 Monte Carlo simulations. The 10,000 costs and QALY estimates generated from the PSA were plotted on a cost-effectiveness plane and the average of these was used to calculate the probabilistic ICER. We used estimates of NMB to determine the probability that FENIX was cost-effective compared with SNS over a range of willingness-to-pay values on the CEAC. Since the CEAC indicates the probability of cost-effectiveness, we can calculate the probability that the optimal treatment will not be cost-effective. Using this information and the NMB provided by the alternative treatment (the net benefit loss), we used the value of information framework43 to estimate the total population expected value of perfect information (EVPI). The population EVPI is estimated using the expected net benefit loss per person from the decision, the number of people with FI that would be eligible for the treatments per year (annual incidence is estimated to be 13,126) and the period for which the FENIX versus SNS decision is still relevant (here, we assumed this to be 10 years). The higher the EVPI value, the greater the potential cost of uncertainty and the greater the incentive to conduct additional research (e.g. further trials) to reduce that uncertainty prior to making a decision.

We used a NICE willingness-to-pay threshold of £20,000 per QALY to determine cost-effectiveness (or NMB value of > 0). We used a half-cycle correction to reflect the expectation that health state transitions can happen at any time during a model cycle. In accordance with current NICE recommendations, costs and outcomes post year 1 were discounted at 3.5% per annum. The perspective remained that of the health and personal social services provider.

Trial analysis

Descriptive results for the costs (and cost category) are shown by arm in Table 25. There is a relatively modest difference in total cost between arms. However, there are notable differences in the individual cost categories, with the FENIX procedure being much cheaper than the SNS procedure, but being associated with significantly higher secondary care costs (for other complications and surgeries). Descriptive statistics on costs (and by cost category) and utility by arm, time-point and sample are included in Appendix 1, Table 44.

The length of stay varied between intervention arms (SNS: range 0 to 59 days, median 0 days, mean 1.43 days; FENIX: range 0 to 15 days, median 1 day, mean 1.39 days).

The economic evaluation results for the 18-month trial period are included in Table 26. Results are presented for complete case and following MI using EQ-5D-5L valuation, EQ-5D-5L to 3L cross-walk and SF-6D utility scores for alternative adjusted models.

For the EQ-5D-5L and SF-6D analyses, there were a total of 47 cases across both arms that met the complete-case criteria (of which 42 cases were common across both the EQ-5D-5L and the SF-6D complete cases), whereas the MI analysis sample total was 94 cases. Five of the original 99 cases were dropped because they had no baseline data or only baseline data. In all analyses, FENIX was more expensive in the trial period than SNS, although this was reduced in the imputed analyses. In all of the EQ-5D-5L analyses, except for the complete-case, 3L cross-walk with adjustment (for age and baseline utility), FENIX led to a QALY benefit compared with SNS. We present the results for adjustment for operating surgeon, but because of the small sample sizes these figures are highly variable and cannot be considered robust.

The ICER for the primary analysis (MI, direct EQ-5D-5L valuation with adjustment for age and baseline utility) is £44,785.62 per QALY gained and is well above the £20,000 threshold. In this analysis, FENIX was associated with modest additional costs and negligible incremental QALYs compared with SNS. Uncertainty in the results was represented in the CEAC (Figure 7). At a willingness-to-pay threshold of £20,000 per QALY gain, FENIX has a 41.3% chance of being cost-effective.

FIGURE 7.

Cost-effectiveness acceptability curve.

The supplementary analysis using MI and a cross-walk to derive EQ-5D-3L values yields a much higher QALY gain and lower ICER for FENIX. The analyses based on the SF-6D indicated a QALY loss associated with FENIX, which was therefore dominated.

The proportion of patients distributed between the CCIS percentage improvement groups is shown in Table 23 (parameter table). For both arms, a majority of patients were either in the < 50% improvement group or did not have a device in situ. Where the device was in situ, the results appear to indicate a slight benefit for the SNS group, with more patients in the 50–75% and > 75% improvement groups. One patient in the FENIX arm had died by 18 months and, although this is unlikely to be attributable to randomisation arm, we retained this patient in the analysis.

The costs and benefits estimated during the trial period were added to the start of the model and assumed to be fixed in the base case deterministic analysis. In each simulation of the probabilistic analysis, we used a bootstrapped cost and QALY generated from the trial data.

Decision modelling

The model Markov trace is included in Figure 8. This gives an indication of the predicted intervention effectiveness and survival over time. There are very few noticeable differences in the curves over the lifetime horizon. The kinks in the SNS figure show where those patients with a device in situ receive a battery replacement at 7 years and the redistribution among CCIS percentage improvement groups.

FIGURE 8.

Markov model traces: (a) SNS; and (b) FENIX.

The deterministic model results, including lifetime costs, QALYs and ICER, are presented in Table 27. Estimated lifetime costs are higher in the SNS group, as are the QALYs. The ICER and NMB values indicate that FENIX is not cost-effective over this time horizon (SNS is the optimal strategy). These results are mirrored by the probabilistic results, where the ICER for SNS versus FENIX drops.

Figure 9 plots the 10,000 costs and QALYs from the Monte Carlo simulations. The results highlight significant uncertainty in the model outputs. The ICER of the mean value falls just below the willingness-to-pay threshold (for SNS minus FENIX). The uncertainty in the results is further represented in the CEAC (Figure 10). At a willingness-to-pay threshold of £20,000 per QALY gain, SNS has a 55% chance of being the optimal strategy (and, therefore, FENIX has a 45% chance of being the optimal strategy).

FIGURE 9.

Cost-effectiveness plane: SNS vs. FENIX. The orange dot represents the ICER using the mean from the cost and QALY simulations.

FIGURE 10.

Cost-effectiveness acceptability curve.

Table 28 includes a range of deterministic sensitivity analyses. The analyses include using complete-case costs and QALYs from the trial, changing the year at which battery replacement is conducted in the SNS group, changes to health state costs and utilities, and assuming equivalent effectiveness across interventions at 18 months. In most analyses, the decision is unchanged (i.e. SNS remains the optimal strategy). A significant drop in the utility estimates for those without a device in situ substantially reduces the QALY gain from SNS over time and changes the decision such that FENIX would be the optimal strategy. FENIX also becomes the optimal strategy (cost saving but marginally less effective) when the proportions of patients across CCIS percentage improvement and explant groups are assumed to be equivalent at the trial end.

Table 29 includes the per patient and population EVPI. These values reflect a monetary cost (attaching a value of £20,000 to a QALY) of the uncertainty in the current decision problem. At the NICE willingness-to-pay per QALY threshold of £20,000, there was a 55% and a 45% chance that SNS and FENIX, respectively, were the optimal intervention and, hence, there was significant uncertainty remaining in the decision. Given the large number of potential intervention recipients, the cost of this uncertainty over 1 year and 10 years is considerable. The EVPI estimates indicate that further research in this area is warranted to reduce the level of decision uncertainty.

Cleveland Clinic Incontinence Score

There were 96 out of 99 (97%) patients included in the analysis, meaning that they each provided their CCIS at least once at one of the baseline, 6-, 12- or 18-month time points.

The CCIS data were modelled using a three-level multilevel model, with time points clustered within patients and patients clustered within surgeons. The estimates of the fixed effects can be seen in Table 34. The random effect caused by surgeon was 0 and the random effect caused by within-patient measurements was 5.93 (SE 1.26). The residual of the random effects was 7.13 (SE 0.71).

Patients who were randomised to the FENIX MSA arm have a baseline CCIS 0.11 points lower than patients randomised to SNS; however, this result is not statistically significant (p = 0.9). Patients having a device in use reduces the CCIS by 3.04 points, a result that is statistically significant (p < 0.0001); however, the interaction term is not statistically significant (p = 0.9). The estimated change in CCIS over time is small; there is a reduction of 0.078 points per month on average, although the CI around this estimate is large owing to the small number of patients. The baseline CCIS (≤ 10 points or > 10 points) has a significant effect on the CCIS (p = 0.005), as would be expected due to them using the same measure; however, the interaction between baseline CCIS and time was not significant (p = 0.8).

The results of the model can be seen in Figure 11. This figure shows the difference between two identical patients, with the only difference being the randomised treatment, and assuming they received the device after 6 months and it remained in use at 12 and 18 months post randomisation.

FIGURE 11.

Cleveland Clinic Incontinence Score model results.

EuroQol-5 Dimensions, five-level version

There were 96 out of 99 (97%) patients included in the analysis, meaning that 96 patients provided a score at least once at one of the baseline, 6-, 12- or 18-month time points.

The EQ-5D-5L score was modelled using a three-level multilevel model, with time points clustered within patients and patients clustered within surgeons. The fixed effects of the model are presented in Table 35. The random effect caused by surgeon was 0.00 (SE 0.003) and the random effect caused by within-patient measurements was 0.04 (SE 0.007). The residual of the random effects was 0.03 (SE 0.003). The model did not show a significant difference in the score between treatment arms (p = 0.76) or over time (p = 0.18). The device being in use has a statistically significant difference of 0.13 (p = 0.004), meaning that patients with their device in use have an EQ-5D-5L score 0.13 points higher than patients without the device in use, and this difference is the same for patients randomised to both treatment arms, as can be seen by the non-significance of the device in use and treatment interaction (p = 0.2). There is no indication that there is a clustering effect caused by the randomising surgeon.

The results of the model can be seen in Figure 12, which presents estimates and 95% CIs for the EQ-5D-5L score at 6, 12 and 18 months post randomisation for two patients who are identical except for their randomised treatment, and assuming that they received the device after 6 months and it remained in use at 12 and 18 months post randomisation.

FIGURE 12.

EQ-5D score model results.

The EQ-5D-5L questionnaire also records the respondent’s self-rated health on a vertical VAS ranging from 0–100, in which higher numbers indicate better health.

A total of 97 out of 99 (98%) patients were included in the VAS analysis, meaning that they provided a VAS score at least at one time point.

The VAS was modelled using a three-level multilevel model with time points clustered within patients and patients clustered within surgeons. The fixed effects of the model can be seen in Table 35. The random effect caused by surgeon was 0 and the random effect caused by within-patient measurements was 233.05 (SE 44.06). The residual of the random effects was 169.71 (SE 16.62). The randomised treatment has no significant effect on the VAS score (p = 0.25) and there is no significant difference over time between the two treatment arms (p = 0.9). The baseline CCIS minimisation factor has a statistically significant effect (p = 0.023); however, the effect over time is not significant, meaning that the baseline VAS score is higher for patients who have a baseline CCIS ≤ 10 points, but the VAS score changes at the same rate for patients with a baseline CCIS ≤ 10 points and > 10 points. The device being in use does not have a significant effect on the VAS score (p = 0.1) and the effect is not significantly different between the treatment arms (p = 0.3) There is no indication of a clustering effect caused by the randomising surgeon.

The results of the model can be seen in Figure 13, which presents estimates and 95% CIs for the EQ-5D-5L VAS score at 6, 12 and 18 months post randomisation for two patients who are identical except for their randomised treatment, and assuming that they received the device after 6 months and it remained in use at 12 and 18 months post randomisation.

FIGURE 13.

EQ-5D VAS model results.

The model diagnostics can be seen in Figures 14 and 15.

FIGURE 14.

Model residuals for EQ-5D-5L score. (a) Raw residuals vs. linear predictor value; (b) histogram of the raw residuals with normal density overlay for reference; (c) normal Q-Q plot of the raw residuals; and (d) box plot of the raw residuals.

FIGURE 15.

Model residuals for EQ-5D VAS. (a) Raw residuals vs. linear predictor value; (b) histogram of the raw residuals with normal density overlay for reference; (c) normal Q-Q plot of the raw residuals; and (d) box plot of the raw residuals.

Faecal incontinence quality of life

The results of the models fitted are summarised in Table 36. None of the domains shows any significant difference between the treatment arms. All of the domains show a statistically significant improvement in FIQoL when the device is in use and there is no significant difference in the improvement between the two treatment arms.

In the lifestyle domain, the random effect caused by surgeon was 0 and the random effect caused by within-patient measurements was 0.61 (SE 0.10). The residual of the random effects was 0.25 (SE 0.03).

In the coping domain, the random effect caused by surgeon was 0 and the random effect caused by within-patient measurements was 0.32 (SE 0.06). The residual of the random effects was 0.22 (SE 0.03).

In the depression domain, the random effect caused by surgeon was 0 and the random effect caused by within-patient measurements was 0.60 (SE 0.11). The residual of the random effects was 0.27 (SE 0.03).

In the embarrassment domain, the random effect caused by surgeon was 0 and the random effect caused by within-patient measurements was 0.38 (SE 0.07). The residual of the random effects was 0.21 (SE 0.02).

The results of the models can be seen in Figures 16–19, which present estimates and 95% CIs for each of the FIQoL domains at 6, 12 and 18 months post randomisation for two patients who are identical except for their randomised treatment, and assuming that they received the device after 6 months and it remained in use at 12 and 18 months post randomisation.

FIGURE 16.

The FIQoL lifestyle model results.

FIGURE 17.

The FIQoL coping model results.

FIGURE 18.

The FIQoL depression model results.

FIGURE 19.

The FIQoL embarrassment model results.

The model diagnostics corresponding to models of the lifestyle, coping, depression and embarrassment domains of the FIQoL can be seen in Figures 20–23, respectively.

FIGURE 20.

Model residuals for FIQoL: lifestyle. (a) Raw residuals vs. linear predictor value; (b) histogram of the raw residuals with normal density overlay for reference; (c) normal Q-Q plot of the raw residuals; and (d) box plot of the raw residuals.

FIGURE 21.

Model residuals for FIQoL: coping. (a) Raw residuals vs. linear predictor value; (b) histogram of the raw residuals with normal density overlay for reference; (c) normal Q-Q plot of the raw residuals; and (d) box plot of the raw residuals.

FIGURE 22.

Model residuals for FIQoL: depression. (a) Raw residuals vs. linear predictor value; (b) histogram of the raw residuals with normal density overlay for reference; (c) normal Q-Q plot of the raw residuals; and (d) box plot of the raw residuals.

FIGURE 23.

Model residuals for FIQoL: embarrassment. (a) Raw residuals vs. linear predictor value; (b) histogram of the raw residuals with normal density overlay for reference; (c) normal Q-Q plot of the raw residuals; and (d) box plot of the raw residuals.

Obstructed Defecation Score

The results of the model fitted to the ODS data can be seen in Table 37. There is no significant difference in the ODS between the two treatment arms (p = 0.64). Whether or not the patient has the device in use does not have a statistically significant difference on the ODS (p = 0.58).

The random effect caused by surgeon was 0 and the random effect caused by within patient measurements was 8.63 (SE 1.73). The residual of the random effects was 8.08 (SE 0.83).

The results of the model can be seen in Figure 24, which presents estimates and 95% CIs for each of the ODSs at 6, 12 and 18 months post randomisation for two patients who are identical except for their randomised treatment, and assuming that they received the device after 6 months and it remained in use at 12 and 18 months post randomisation.

FIGURE 24.

Obstructed Defecation Score model results.

The model diagnostics can be seen in Figure 25.

FIGURE 25.

Model residuals for ODS. (a) Raw residuals vs. linear predictor value; (b) histogram of the raw residuals with normal density overlay for reference; (c) normal Q-Q plot of the raw residuals; and (d) box plot of the raw residuals.

Short Form questionnaire-12 items version 2

The fixed effects of the multilevel model fitted to the data can be seen in Table 38. Both models show that there is no difference between the PCS/MCS for patients randomised to the two treatment arms. The PCS model shows a statistically significant increase in score of 3.07 if the patient has the device in use (p = 0.04), and this difference is not significantly different for the treatment arms (p = 0.6). The MCS model does not show any statistically significant difference in the score if the patient has the device in use. The MCS model shows a significant difference between the scores for patients who have mild to moderate CCIS versus moderate to severe CCIS (p = 0.004); however, this effect is not significantly different over time (p = 0.48).

For the PCS model, the random effect caused by surgeon was 2.45 (SE 5.84) and the random effect caused by within-patient measurements was 61.14 (SE 11.45). The residual of the random effects was 30.52 (SE 2.93).

For the PCS model, the random effect caused by surgeon was 0 and the random effect caused by within-patient measurements was 82.30 (SE 15.16). The residual of the random effects was 58.73 (SE 5.62).

The results of the models can be seen in Figures 26 and 27, which present estimates and 95% CIs for each of the PCS and MCS components of the SF-12 at 6, 12 and 18 months post randomisation for two patients who are identical except for their randomised treatment, and assuming that they received the device after 6 months and it remained in use at 12 and 18 months post randomisation.

FIGURE 26.

Physical Component Summary model results.

FIGURE 27.

Mental Component Summary model results.

The model diagnostics can be seen in Figures 28 and 29.

FIGURE 28.

Model residuals for SF-12: PCS. (a) Raw residuals vs. linear predictor value; (b) histogram of the raw residuals with normal density overlay for reference; (c) normal Q-Q plot of the raw residuals; and (d) box plot of the raw residuals.

FIGURE 29.

Model residuals for SF-12: MCS. (a) Raw residuals vs. linear predictor value; (b) histogram of the raw residuals with normal density overlay for reference; (c) normal Q-Q plot of the raw residuals; and (d) box plot of the raw residuals.