Subacromial spacers for adults with symptomatic, irreparable rotator cuff tears: the START:REACTS novel group sequential adaptive RCT

Primary clinical objective

Our primary clinical objective was to quantify and make inferences on observed differences between arthroscopic debridement using an InSpace balloon with arthroscopic debridement alone 12 months after surgery, using the OSS as the primary outcome measure. 63,64 A 12-month time point was selected based on our meta-analysis of outcomes for randomised trials which found that shoulder scores typically reach a plateau at 12 months after any intervention for a rotator cuff tear. 42 Although we are collecting 24-month scores, they do not provide sufficient additional value to justify the increase in costs and delay in the trial result that would be required had they been used as the primary outcome.

Secondary clinical objectives

Our secondary clinical objectives were to:

1. quantify and make inferences on observed differences between the two intervention groups on the following measures; all were assessed at baseline, 3, 6 and 12 months:

   • the Constant score38,39

   • shoulder pain-free range of motion (ROM) in abduction and flexion

   • strength in scapular-plane abduction measured using a hand-held dynamometer

   • the OSS63,64

   • the WORC index65

   • the EuroQol-5 Dimensions, five-level version (EQ-5D-5L)66,67

   • adverse events.

2. Perform an economic analysis to assess the comparative cost-effectiveness of the two interventions (see Chapter 5).

3. Perform a magnetic resonance imaging (MRI) substudy to compare the acromiohumeral distance (AHD) on MRI scans in a sample of participants with and without the balloon at 8 weeks and at least 6 months after treatment. This was to assess the proposed mechanism of action of the InSpace balloon while still inflated (8 weeks) and to determine any persistent effects following deflation (after 6 months).

Methodological objectives

The methodological objectives of the study were to:

• develop and implement appropriate statistical tools for an efficient adaptive clinical trial design, with the potential to stop early either for futility or efficacy, using outcome data available at 3, 6 and 12 months61

• compare the use of frequentist and Bayesian design and analysis on the conduct and interpretation of an adaptive surgical clinical trial with reference to decision-making by data monitoring committees (DMCs) during the study and by clinicians, commissioners and other stakeholders at the conclusion of the trial

• explore the challenges of supporting adaptive design decision-making with net benefit and expected value of information approaches to health economic analyses.

Primary outcome

Secondary outcomes

Participant identification, screening and withdrawals

Potential participants were identified by the attending clinical team in intermediate or secondary care clinics, or from surgical waiting lists. The attending clinician confirmed each patient’s eligibility for the study based on clinical assessment and standard care preoperative imaging for each site (e.g. MRI or ultrasound).

All potential participants meeting study entry criteria were checked for eligibility and their details recorded on a screening log. Potential participants suitable for inclusion and willing to be approached were given verbal and written information about the study by a suitably trained member of the research team, were given adequate time to consider the information and were invited to ask questions and discuss the study further prior to informed consent being obtained. Trial procedures including baseline assessments were not undertaken until the consent form was signed and dated by the participant.

All participants provided informed, written, signed and dated consent. As there was a delay of a number of weeks before randomisation (the waiting list for surgery), people entering the study still had the option to withdraw before treatment commenced if for any reason, they changed their mind. Any new relevant information arising during the trial was discussed with the participants and, where applicable, continuing consent was obtained using an amended consent form.

Sites agreeing to take part in both the main study and the additional MRI substudy were provided with a combined main and substudy patient information sheet and consent form.

Eligibility was also confirmed by the operating surgeon intraoperatively and participants were excluded (withdrawn) at this stage if it was discovered that the rotator cuff tear was repairable. These excluded participants were informed by letter that they were no longer taking part in the study. Baseline data were, however, retained to explore any differences between those who were deemed ineligible intraoperatively to those who were randomised. Participants randomised into the study were allowed to withdraw from follow-up at any time, without prejudice and without any effect on their current or future care.

Inclusion criteria

Patients were eligible to be included in the trial if they met the following inclusion criteria:

• Rotator cuff tear deemed by the treating clinician to be technically irreparable. Multiple factors apart from size can influence whether a tear can be repaired (i.e. chronicity, retraction of the tendon ends and fat infiltration in muscle). However, patients with tears thought to be technically repairable, such as a small tear, but considered unsuitable for repair due to age or comorbidities, were not eligible for this study.

• Experiencing intrusive symptoms (pain and loss of function) which in the opinion of the treating clinician warrants surgery.

• Had undergone unsuccessful nonoperative management, as determined by the treating clinician.

Exclusion criteria

Patients were considered ineligible if they met any of the following criteria:

• Advanced glenohumeral osteoarthritis on pre-operative imaging (in the opinion of the treating clinician). This was interpreted as Kellgren–Lawrence grade 3 or 4 changes on routine preoperative radiographs73 or the MRI equivalent if radiographs were not available.

• Subscapularis deficiency,¹ defined as a tear involving more than the superior 1 cm (approximately) of the subscapularis, if repaired, or any tear that is not repaired.

• The treating clinician determines that interposition grafting or tendon transfers are indicated.

• Pseudoparalysis (an inability to actively abduct or forward flex up to 20 degrees), as determined by the treating clinician.

• Having any unrelated, symptomatic ipsilateral shoulder disorder that would interfere with strength measurement or ability to perform rehabilitation.

• Other neurological or muscular conditions that would interfere with strength measurement or ability to perform rehabilitation, in the opinion of the treating clinician.

• A previous proximal humerus fracture that could influence shoulder function, as determined by the treating clinician.

• Previous entry into the present trial (i.e. other shoulder).

• Unable to complete trial procedures.

• Age under 18 years.

• Unable to consent to the trial.

• Unfit for surgery as defined by the treating clinician.

Randomisation

Participants were randomly allocated on a one to one basis to the two treatment groups via a central computer-based randomisation system provided by Warwick Clinical Trials Unit (WCTU), independent of the study team. A minimisation algorithm was used to determine participant allocation, using site, sex (used rather than gender as we were interested in the structural effect), age group (< 70 and ≥ 70 years) and cuff tear size (as assessed by the operating surgeon, ≥ 3 or < 3 cm, commonly used as the definition between small/medium and large/massive cuff tears) as factors, with a random element included to provide a 70% chance that the participant would receive the treatment that minimises imbalance, to ensure unpredictability.

Randomisation was performed by theatre staff, after intraoperative findings were checked (including cuff tear size) and eligibility confirmed. To maintain blinding, staff used an online system in a separate room (a 24-hour back-up automated telephone system was also available). Participants were randomised strictly sequentially at site level. Allocation concealment was maintained by the independent randomisation team who were responsible for the generation of the sequence and had no role in the randomisation of participants beyond this.

Standard arthroscopic debridement (control)

The control group were randomised to arthroscopic debridement of the subacromial space with removal of inflamed tissue (bursectomy) and unstable remnants of the torn tendon, limited bone resection of the acromion, retention of the coracoacromial ligament and biceps tenotomy (if not already torn). The anaesthetic (general or local) was decided by the anaesthetist. Surgeons were permitted to use their normal surgical technique within the confines described in a trial-specific surgical guideline, available on the Warwick Research Archive Portal (http://wrap.warwick.ac.uk).

Standard arthroscopic debridement plus insertion of InSpace balloon (intervention)

Those allocated to the intervention underwent the same arthroscopic debridement as the control group plus insertion of the InSpace Balloon by a subspecialty trained shoulder surgeon. The recommended surgical technique was followed for sizing, insertion, and deployment of the balloon, as documented in the surgical manual. The content of the surgical manual was confirmed with the manufacturers (OrthoSpace at the time) prior to interventions starting in the study, to ensure the study followed recommended techniques.

For both groups, fidelity was assessed with an operative record form, arthroscopic photographs (posterior and lateral portal photographs after debridement for both groups, and a photograph from the posterior portal just before balloon inflation in the balloon group to demonstrate balloon position). In addition, the number of physiotherapy visits for each participant was also documented in both groups.

Rehabilitation

The postoperative rehabilitation programme was developed during the set-up phase of the trial. The process involved collecting current NHS rehabilitation protocols from participating sites, manufacturer protocols, scoping the literature and ratification among a panel of expert physiotherapists. As with the surgical technique, a copy was forwarded to the manufacturer, OrthoSpace, prior to starting interventions to ensure that the study did not conflict with manufacturer’s recommendations. A physiotherapy trial manual was produced and followed, to standardise rehabilitation progression and is available from: https://warwick.ac.uk/fac/sci/med/research/ctu/trials/startreacts/links/healthlinks/physiotherapy_manual_v2.2_10072018.pdf.

Postoperative rehabilitation for both groups was blind to treatment allocation and included standardised postoperative information, home exercises and the offer of a minimum of three face-to-face physiotherapy appointments. Additional physiotherapy was arranged at the discretion of the trial sites.

Power and sample size

Initial sample size calculations were based on the Constant score, with a target difference of 10 units, as widely used for other trials and a standard deviation (SD) of 20, giving a moderate standardised mean difference of 0.5. 20,27,75,76 In anchor-based studies the estimated target difference for the OSS is 6 with an SD of 12 being observed in multiple studies. 41,68,69,76 For an expensive invasive procedure of this nature, a smaller standardised mean difference was considered unlikely to be worthwhile. Therefore, a moderate standardised mean difference of 0.5 was considered appropriate. For a power of 90% and a (two-sided) type I error rate of 5% a study with a conventional fixed design (i.e. with no possibility of stopping early), assuming an approximate normal distribution for the score data, would require 170 participants.

The design characteristics and required sample size for the planned adaptive design were assessed and estimated in a large simulation study (see below). 61 We anticipated correlations between time points (based on Karthikeyan et al.)76 and effect sizes to be equivalent between the Constant score and OSS; thus, these simulations remained valid, despite the change of primary outcome. 68,76 The simulations showed that an adaptive design that allowed the possibility of early stopping for efficacy and/or futility, was feasible for the START:REACTS study.

Based on an assumed modest correlation between 3-, 6-, and 12-month OSS equal to 0.5, and an SD of 12 for both 3- and 6-month scores, the simulations for the selected adaptive design indicated that follow-up data from a maximum of 188 participants would be needed to detect a six-point difference in the OSS between treatment arms with 90% power and 5% (two-sided) type I error rate. Allowing for 15% lost to follow-up, while attempting to keep this below 10%, a maximum study sample size of 221 was calculated.

Primary outcome analysis

All data have been analysed and reported in accordance with the Consolidated Standards of Reporting Trials guidelines. 77,78 A detailed statistical analysis plan and a data-sharing plan were agreed with the DMC prior to any formal analyses being conducted. 79

The primary analysis investigated differences in the OSS 12 months after surgery between the two treatment groups on an intention-to-treat basis following the methods, test statistics and boundaries described by Parsons et al. 61 To preserve the integrity of the study, the exact boundaries used for testing were specified in an adaptive charter known only to the study team and independent DMC. In brief, if the study recruited to target, Parsons et al. ’s61 methods would be used to calculate the boundaries. As the study stopped early, testing proceeded using boundaries calculated by the deletion method for over-running analysis,80 with inferences following directly from widely used methods for unbiased estimates and confidence intervals (CIs) in group sequential trials. 80–82

Secondary outcome analyses

Descriptive statistics of all outcome measures data (i.e. the OSS, WORC, EQ-5D-5L, PGIC and analgesia usage) at each time point were constructed.

Adjusted estimates of treatment group differences (with 95% CIs) for the OSS at 12 months post-randomisation were calculated using a mixed-effects model including a random effect for the recruiting centre and fixed effects for variables of interest including patient age, sex and size of tear. Estimates of efficacy for the other outcome measures followed this approach to analysis. Adverse and SAEs were analysed using Fisher’s exact test. Patient-reported change in symptoms and the PGIC were analysed using adjusted proportional odds ordered regression models.

Owing to the low level of missing data in the primary outcome, no imputation was undertaken.

Exploratory analyses

To assess the impact of changing the primary outcome measure from the Constant score to the OSS, we assessed the relationship between the two scores by fitting a simple linear model between the two outcomes at each time point.

Based on discussions with clinicians at sites, proximal humeral migration on a preoperative X-ray was used by some shoulder surgeons to understand the severity of the rotator cuff tear. 83,84 Explanatory models were used to assess the relationship between acromiohumeral position and the OSS data in the trial. The routine clinical images at baseline (X-rays and MRIs) were used to measure AHD for all participants recruited (registered) into the study. To assess the effects of AHD on the primary outcome, AHD was added as an additional confounder in the adjust mixed-effects model (described above).

Subgroup analyses

Prespecified subgroup analyses were undertaken to assess whether there was evidence that the intervention effect differed with respect to:

• the size of the rotator cuff tear as measured at the start of surgery, defined as large or massive cuff tear (≥ 3 cm) or moderate to small tear (< 3 cm)

• sex (male or female)

• age (≥ 70 or < 70 years).

These variables were selected as they are either important to the function of the intervention (cuff tear size) or to interpretation (sex and age). The subgroup analyses followed the methods described for the mixed-effects model for the primary analysis, with additional interaction terms incorporated into the mixed-effects regression model to assess the level of support for these hypotheses. The study was not powered to formally test these hypotheses, so they are reported as exploratory analyses only, and are secondary to the analysis reporting the main effects of the intervention in the full study population. 85

Initial simulation study

A 3-month work package was undertaken at the start of study to develop appropriate methodological tools, implement a series of simulation studies using synthetic data to understand the properties of the selected design and its sensitivities to model assumptions and, in collaboration with the study DMC, agree futility and efficacy stopping boundaries. 61

Study end points for the OSS were at 3, 6 and 12 months. For the adaptive design, the 12-month primary outcome would have been too late to be used for determining futility before the study finished recruiting, and therefore, available early and late outcome data were used to determine early stopping.

A 3-month outcome was chosen, particularly as the balloon degrades after this time, with the predictive strength of the model strengthened further by the addition of 6- and 12-month outcomes. A decision to stop for futility would only be considered when there was sufficient confidence (based on this comprehensive simulation work) using all available 3-, 6- and 12-month data. A decision to stop because of clear evidence of efficacy was also considered in the modelling but especially strong evidence of efficacy (and the relationship to later outcomes) was required for early stopping.

Boundary setting

Futility stopping boundaries based on early and late observations of a single study end point with a final analysis adjusted for futility stopping have been suggested previously by Stallard. 58 Methodology developed previously by the group for two time points was extended to three time points (and also made more general) to allow the totality of observed data at planned interim analyses to be used to inform study progress (e.g. stopping).

Results from our systematic review suggested a strong association between early and late outcomes based on data from trials of interventions for rotator cuff tears. 42 A strong positive correlation between 3- and 6-month outcomes and 12-month outcomes indicated that information on the former early outcomes were strongly indicative of later outcomes, and consequently intervention efficacy or futility.

Sequential stopping boundaries were constructed that allowed stopping for futility or stopping to reject the null hypothesis (efficacy), with interim analyses determined by the results of the simulations and agreed with the TSC and DMC. 86 It was agreed that the stopping boundaries would be binding. Stopping boundaries for safety were also agreed and set as a separate criterion for early stopping.

Simulated data that replicate the metric properties of the primary outcome were then used to explore the characteristics of the design and sensitivities to likely treatment effect sizes and correlations between early and late outcomes. The aim was to understand changes in study power and rates of early stopping for a range of likely treatment effect sizes. The results of these simulation studies were presented to the TSC and DMC, and rules were agreed such that it was clearly defined as to when interim analyses should happen, what the thresholds were for early stopping, and how decisions should be communicated within the study team.

Preliminary simulations indicated that a single interim analysis using 3-month data on 53 patients per arm with the probability of early stopping under the null hypothesis set to 50% would result in only a small reduction in power to 88% for modest correlations between 3- and 12-month data. Given the findings of our systematic review of a good relationship between early and late outcomes in trials of interventions for rotator cuff tears42 and the fact that correlations were improved by considering all of the available time points, there is likely to be little loss of power.

As we expected correlations between time points (based on Karthikeyan et al.)76 and effect sizes to be equivalent between the Constant score and the OSS, these simulations remained valid, despite the change of primary outcome score. 68

As with the primary analysis, the interim analyses were performed using a frequentist approach although a Bayesian framework was explored as a substudy (see Chapter 8).

The Constant score

Further details of the Constant scores can be found in Appendix 1.

The Constant score had high levels of missingness due to the difficulty of obtaining data during the lockdown due to COVID-19. Insufficient data were collected to allow for the planned linear regression model to be calculated. However, as earlier time points were less affected, some inferences could be made on the 3- and 6-month data. The collected data showed that the debridement-only group had better outcomes at these early time points.

Western Ontario Rotator Cuff Index

Further details of the WORC scores can be found in Appendix 1. Here it can be seen that the effect of the InSpace balloon again has a reduction on function (mean −8.4 points, 95% CI −16.7 to −0.1; p = 0.055), following the results of the OSS. Similarly, to the adjusted OSS model, smaller tear size and higher ages were associated with non-significant increases in function. Female sex was also associated with an in increase in function, but this was small (mean 1.5, 95% CI −7.3 to 11.1; p = 0.74).

EuroQol Five Dimension Five-Level score

Further details of the EQ-5D-5L scores at each time point can be found in Appendix 1. No terms were found to be statistically significantly associated with health-related quality of life (HRQoL). The InSpace balloon was associated with a small non-significant decrease on overall HRQoL (mean −0.056, 95% CI −0.15 to 0.03; p = 0.239). The older age group was associated with an increase (0.015, 95% CI −0.08 to 0.12) while female sex (−0.007, 95% CI −0.10 to 0.09) and smaller tear sizes (−0.059, 95% CI −0.28 to 0.15) were associated with reductions in HRQoL, although all these differences were non-significant.

Patient Global Impression of Change

Participants were asked to report how much better or worse their shoulder felt 12 months after their surgery as well as if there was any change in their activity limitations, symptoms, emotions, and overall QoL. Details are shown in Appendix 1. There were no statistically significant differences in either overall change (OR 0.6, 95% CI 0.3 to 1.2) or in PGIC (OR 0.5, 95% CI 0.3 to 1.1); although both questions favoured the arthroscopic debridement alone group.

Effects of COVID-19

We undertook a sensitivity analysis to assess if COVID-19 could have affected the primary analysis. Data from participants who had completed the 12-month primary outcome before the first lockdown were compared with the remaining participants; 25 participants had completed their 12-month primary end point prior to the first UK lockdown (20 March 2020) and 89 participants were followed up during/after the COVID-19 lockdown period. The results of this analysis (see Appendix 1) confirm that COVID-19 did not significantly change the collected OSS or EQ-5D-5L scores at 12 months, nor did it affect the completion rates. However, since there was a major impact on collecting the 12-month Constant score (97.8% missing during or after COVID-19), this score could not be assessed.

Acromiohumeral distance and rotator cuff pathology

Since more participants than expected were excluded intraoperatively, we did additional exploratory analyses to explore whether there were differences in the pathology between the two groups at registration, prior to surgery. Hence, these analyses should be interpreted with caution and may not be generalisable.

While baseline OSS and Constant scores were both greater in the repairable cuff tear group, these differences were small and not statistically significant (OSS mean difference 2.1, 95% CI −5.0 to 0.9; Constant score mean difference 3.8, 95% CI −1.2 to 8.9). The difference in AHD was statistically significant with those which had a repairable cuff tear having a significantly larger AHD than those with a non-repairable tear (mean 1.1, 95% CI −0.11 to 2.1; p = 0.034). Full details of these analyses can be found in Appendix 1.

Acromiohumeral distance and Oxford Shoulder Score at 12-month follow-up

Including AHD into the adjusted efficacy model showed that the OSS improved by approximately 0.47 points for every 1 mm increase in the AHD; however, this was not a statistically significant result (95% CI −0.3 to 1.2; p = 0.236). The effect of the allocation group remained broadly similar to the main analyses, favouring the debridement alone group.

Relationship between the Oxford Shoulder Score and the Constant score

The original primary outcome was the Constant score (see above). As the study design was dependent upon the observed correlations at each follow-up pint, Pearson’s correlation coefficient between this outcome and the OSS were calculated to confirm that it was a suitable alternative. The OSS is scored between 0 and 48, hence we rescaled scores to a 0–100 range to allow direct comparison to the Constant score (range is 0–100). These results (see Appendix 1) suggest that the Constant score has a strong association with the OSS at each time point.

Costing of the intervention

The economic evaluation focused on assessing intervention implementation costs, including:

1. Health-care resource use: costed using most recently available published national reference costs (see Appendix 2, Table 60), within a common year. 22,98–100 Participants were asked about their health service contacts in connection with their shoulder condition at 3, 6 and 12 months. The numbers of contacts with community and outpatient clinics and use of hospital services were recorded within the trial case report form.

2. Additional medication/prescriptions: collected from a diary to assist participants to record this information. The costs of study medications were included using most recent English prescription cost analysis data (using British National Formulary 2020). 101 Participant-level costs were estimated as the sum of resources used weighted by their unit costs. 101

3. Direct intervention costs: including the costs of delivering arthroscopic debridement with insertion of the InSpace balloon and arthroscopic debridement only.

4. Production losses in terms of the time from work (paid or unpaid) due to receiving the intervention was also recorded (although time from work was not included as a resource from the perspective of the analysis).

Collection of broader resource-use data

Data were collected on health, PSS and broader societal resource inputs between randomisation and 24 months post-randomisation. Trial participants were required to complete resource-use questionnaires at baseline, 3, 6, 12 and 24 months post-randomisation using a modified version of the Client Service Receipt Inventory (version 1.0). Data were reported in terms of the following categories:

1. inpatient care

2. outpatient care

3. community health care

4. personal and social services

5. loss of production/work

6. pain medication.

Medication use was categorised by drug name (or active constituent), mode of administration, dosage frequency and duration.

Treatment by subgroup interaction

A treatment by subgroup interaction term was also tested for sex and age group for each of costs and QALYs, while ensuring costs and QALYs were correlated following a bivariate regression model of the form:

by the planned subgroups: sex and age (≥ 70 years, < 70 years) in terms of QALYs and total costs. The interaction was tested at a two-sided 5% level.

Study population

A total of 117 participants were randomised into the START trial: 56 to the debridement with InSpace balloon group and 61 to the debridement-only group. Complete baseline information was available for 117 participants, so the baseline study population for the bulk of the health economic analyses was 117 participants.

Between 96% and 100% of all health resource-use data were complete at baseline for participants randomised to debridement with InSpace balloon and between 95% and 97% for those randomised to debridement-only group. Similarly, these values ranged between 93% and 100% between 3 and 12 months, depending on the specific health resource item. A complete QALY profile was available over 12 months for 54/56 participants (96%) with debridement with InSpace balloon and 58/61 (95%) for those with debridement-only. Consequently, about 5% of QALY data and between 5% and 7% of the total cost outcomes were missing (and subsequently imputed) for the primary analysis.

Resource use and costs

Health-related quality-of-life outcomes

There were no (statistically significant) differences in the overall EQ-5D-5L utility scores or EQ-5D-5L VAS scores between the intervention and control groups, at each of the follow-up time points. For complete cases, the mean (SE) patient-reported QALY over 12 months was estimated as 0.551 (0.0586) compared with 0.606 (0.0307) for debridement with InSpace balloon and debridement-only, respectively (p = 0.2242). In the absence of a significant effect on the EQ-5D-5L either at the primary outcome time point, or the EQ-5D-5L based QALY, usual care was dominant in health economic terms. Unless the expected costs were lower for the debridement with InSpace balloon group, usual care would almost always dominate.

There was a statistically significant treatment by subgroup interaction in terms of QALYs for sex (p = 0.0025) and age group, which is reflected in estimates of the incremental QALY in these groups (see Table 6).

Cost-effectiveness results

Base case analysis

The incremental cost-effectiveness of arthroscopic debridement with InSpace balloon is shown in Table 14 for the participants with costs and health outcomes data subject to multiple imputation. When a societal (NHS/PSS plus broader societal costs) perspective was adopted (i.e. that adopted for the baseline analysis) and health outcomes were measured in terms of QALYs, the mean incremental cost was £2322. The mean incremental cost-effectiveness of debridement with InSpace balloon was estimated at −£45,351 per QALY; that is, on average, the intervention was associated with a higher net cost and a lower net effect and was dominated in health economic terms.

The associated mean INMB (net loss) at cost-effectiveness thresholds of £15,000, £20,000 and £30,000 per QALY were −£3074, −£3325 and −£3826, respectively (see Table 6). The base case mean INMB was < 0, suggesting that those randomised to debridement with InSpace balloon would result in an average NHS/PSS loss of about £3074 [INMB (£) = −2158, 95% CI −3455 to −969] to [INMB (£) = −3074, 95% CI −4403 to −1761]. The cost-effectiveness plane (see Figure 6) shows that the vast majority of the ICER values lie in the north-west quadrant. These result in a probability of cost-effectiveness close to zero; that is, if decision-makers are willing to pay between £15,000 and £30,000 for an additional QALY, the probability that the debridement with InSpace balloon intervention is cost-effective is very low (< 1%).

FIGURE 6.

Cost-effectiveness plane for DB (experimental: debridement with InSpace balloon intervention) vs. D (control: debridement only): incremental cost (£) vs. incremental QALY – base case.

Sensitivity and subgroup analyses

Several sensitivity analyses were undertaken to assess the impact of uncertainty surrounding key parameters or methodological features on the cost-effectiveness results. The probability that the debridement with InSpace balloon intervention is cost-effective remained relatively static (< 1%) for the majority of the sensitivity analyses (see Table 6). For subgroups, only in the case of sex was the probability of cost-effectiveness higher (at 42%) for males compared with females at a CE threshold of £30,000/QALY. In all cases the average INMB was negative: all analyses showed the average INMB were unlikely to be positive as all upper limits of the 95% CI were below zero (see Table 6; Figure 7). A treatment by subgroup interaction in terms of costs and QALYs.

FIGURE 7.

Forest plot of sensitivity and subgroup analyses (impact on incremental net monetary benefit).

Missing data assumptions

Using a logistic regression model, the response (missing or not) was modelled against covariates age group and sex to assess whether missingness was explained by the covariates. Since the number of missing observations was small [n = 3 (5%) for debridement-only and n = 4 (7%) for the debridement with InSpace balloon group] and similar between groups, missingness was not statistically associated with the covariates (p > 0.05) (see Appendix 2, Table 61).

Conditional power for cost-effectiveness

Conditional power (CP) is a well-established method for conducting futility analyses to stop a trial early due to lack of efficacy using interim data. 110 However, such futility rules can be problematic. Trials of interventions with modest treatment effects may pass futility criteria (high CP) but have little chance of demonstrating value for money from a cost-effectiveness perspective. Conversely, trials may stop prematurely due to modest treatment effects (low CP) that might have proved cost-effective because, for example, safety improved. Futility analyses reflecting both health outcomes and costs might better inform futility decision-making. In this research, we generalise the CP expressions used for efficacy to a cost-effectiveness framework. 110,111

Expressions for the CP based on efficacy for the two-sample case (1 : 1 allocation) are extended to cost-effectiveness. The INMB, willingness to pay threshold, interim and final sample sizes, variabilities and correlations between costs and effects are examined. Data from two clinical trials are used to examine the operating characteristics of the conditional power of cost-effectiveness (CP_CE).

Assuming a two (independent) group trial, the sample size formula for a two group (1 : 1 allocation) randomised trial using the standard two-sample t-test is described in equation 1:

Rewritten, this is:

where Z_α, Z_β are typically the 5% and 80% (or 90%) values from the normal tables referring to the type I and type II errors respectively, σ² is the estimated (population) pooled variance from responses across the two treatment arms and Δ is the clinically relevant difference in means (between groups) in some efficacy outcome (determined from |μ1−μ2 |, where μ₁ and μ₂ are the population mean responses for groups 1 and 2, respectively, and N is the total sample size).

The expressions in equations (1) and (2) can be expressed in the following form:

Where

is the observed test statistic at some fraction t where all parameters are estimated using data at the interim analysis. Ι_Ω in equation (2) is the combined information fraction from, n_t is the total sample size at time t. In equation (3), θ_INMB is the INMB under H₁ (i.e. the target mean INMB, at the end of the trial, which should be > 0). Hence, expanding (Eq 3) to use the average information fraction, we get:

where INMB is the observed mean INMB at some interim period, v²_c1 and v²_c2 are the variances of costs for each of the two groups at some interim period; Nc1 and Nc2 are the planned sample sizes, nc1 and nc1 are the interim sample sizes, k is the cost-effectiveness threshold; s²_c1 and s²_c2 are the observed variances of costs at the interim which are estimates of the unknown v²_c1 and v²_c2; subscripts E and C refer to costs and effects (QALYs) for the respective treatment (experimental and control); The term θINMB is the expected INMB under some alternative hypothesis (usually > 0). If the value of θINMB is set to 0 (i.e. the mean INMB is non-negative), equation (4) can be simplified further. For this example, we will assume the target mean INMB is zero in the worst case, since to plan for a trial yielding a mean INMB of less than 0 would not seem to make sense.

Results of applying the above equation (4) to the START data under the assumption of zero correlation between costs and effects, with the results estimated after 50% of patients using summary statistics only at 6 months, for NHS/PSS perspective, the estimated parameters for the CP are shown in Table 62 in Appendix 2. Hence, P_t(θ) = 2.5% This calculation provides a CP for cost-effectiveness of 2.5%

This means that the chance of cost-effectiveness at the end of the trial, conditional on what was observed during the interim and assuming that future data would behave in a similar way to the interim, would be low enough to warranting terminating the trial for lack of cost-effectiveness.

Methods for developmental work

People with a rotator cuff tear listed for surgical repair were identified from waiting lists at University Hospitals Coventry and Warwickshire and invited to enter one of two studies. While the main START:REACTS study was recruiting patients with irreparable rotator cuff tears, we selected people with repairable rotator cuff tears for the development of the technique described here. The development of the technique was not impeded by selecting a similar yet distinct group of people but allowed for the retention of as many patients for potential recruitment into the main study.

Consent was taken prior to their participation. Ethical approval was obtained on 13 February 2018 (Integrated Research Application 233804). All participants completed a case report form, which included sections on demographics, the severity, duration and impact of symptoms, as well as the investigative work-up and therapy received.

The first group of 11 participants took part in an electromyography (EMG) study to assess the optimal method of activating the deltoid muscle, and then four participants underwent MRI scanning using the pilot methodology. For the EMG component of the study, participants were invited to attend research clinics where deltoid muscle activity was determined using surface EMG while undergoing a series of tasks. This allowed us to determine a technique for deltoid muscle activation that could subsequently be performed by participants in the second part of the study, who were invited for a MRI where humeral head migration was measured. All participants provided written consent after discussion with a research nurse but prior to MRI and EMG studies being done. Participants were free to stop the investigations at any time during the studies if they felt pain.

Magnetic resonance imaging substudy

After the EMG study, a further four participants were invited for an MRI of the shoulder using the novel protocol developed for this study. These were all participants who had been listed by the treating surgeon for a repair of the rotator cuff based on prior imaging. The participants were positioned in the MRI scanner with a custom-made plastic ‘L-shaped board’ placed under the participants back, level with the elbow.

The board was designed with two vertical slats spaced 5 cm apart. Each participant placed their arm against the first slat. A green TheraBand was applied around the participants in the same manner as for the EMG study. The first slat was then removed and the participant was asked to abduct their arm until it was touching the second slat, ensuring that each participant abducted their arm by 5 cm under the tension of the TheraBand thus allowing us to control the force provided under each deltoid activation between participants (see Figure 8).

FIGURE 8.

Images of MRI coil placement and TheraBand positioning. (a) Arm in neutral position against L-shaped device. (b) Slat removed and arm abducted by 5 cm to outer slat. Source: photograph of staff member reproduced with permission from University Hospitals Coventry and Warwickshire.

While 5- and 10-cm abduction were tested for the EMG study, only 5-cm abduction was used, as practically a 10-cm movement was not found to be consistently possible due to the 70-cm bore of the scanner and the variance in body habitus of the participants being scanned.

Methods for magnetic resonance imaging substudy

All participants from across both treatment groups were invited to undergo two research MRI scans: one at 8 weeks post intervention and a further one after 6 months.

MRI scans were preferred to X-ray, as measures taken from X-rays would be prone to error, primarily due to variation in the angle between the shoulder joint and the beam of the X-ray. The proposed mechanism of the balloon was discussed with its inventor (Dr Assaf Dekel), who explained that it was not designed to depress the humeral head passively but to cushion the humeral head from impinging on the acromion during activity. We therefore decided that passive imaging alone was likely to be inadequate to demonstrate the function of the balloon and imaging also needed to be performed when the deltoid muscle was active, producing a proximally directed force on the humerus. 10,11

As described, we developed and piloted a novel dynamic approach to assessing the function of the rotator cuff using MRI under a mild deltoid load specifically to assess the mechanism of the balloon. We followed the methods for positioning and scanning participants described previously. We used conventional MRI with images in the oblique coronal planes as the preferred technique for imaging the rotator cuff. Evidence shows that fat-suppressed, fast spin-echo, T2-weighted images are the most accurate for the assessment of rotator cuff tears and a variation of this sequence was used and applied on the range of MRI machines across the different sites in this substudy. 116,117

Staff carrying out the scans and trial staff assessing images were blinded to participant allocation.

The development study

A total of 11 participants attended for EMG testing while a further 4 attended for an MRI scan (see Table 7). The median age was 64 years (range 53–73 years) for the EMG group and 63 years (range 49–72 years) for the MRI group. The median duration of symptoms was 9 years (range 6–168 years) and 24 months (range 11–36 months), respectively.

In the EMG group, of 11 participants, 7 (64%) had suffered trauma to the shoulder, 8 (73%) had received a steroid injection prior to commencing the study and 7 (64%) had received physiotherapy. In the MRI group three of four participants (75%) had suffered trauma, received steroid injections and physiotherapy.

Magnetic resonance imaging substudy main results

Recruitment was lower than anticipated, with a total of 24 participants recruited to the substudy before randomisation was stopped. This was partly due to the early adaptive stop from the main trial, and partly due to restrictions related to COVID-19. A summary of these participants can be found in Appendix 3. Furthermore, this substudy was greatly affected by the response to the COVID-19 pandemic, as participants could not attend scans during lockdown. Hence, only 17 (71%) had images that could be analysed. Table 10 shows the values of AHD for each scan type at the two follow-up point by allocation groups. A further analysis investigating the change on the AHD between active and passive scans can be found in Appendix 3.

AHD reduced under active loading in all study groups, in a way that was consistent with the findings from the development work. Participants who underwent debridement-only and those who underwent debridement with the InSpace device had comparable findings on both the passive and active images, and the relative differences between passive and active scans were consistent throughout. This relationship was repeated across the time points and scan types.

The development study

The biomechanical action of the deltoid muscle causing proximal humeral migration in the absence of a functional rotator cuff is well understood in the literature. However, this action is hard to measure and define in real patients in a consistent and measurable manner. Our first aim was to produce a reproducible technique for deltoid muscle activation that could be performed within an MRI scanner and used to measure proximal humeral migration.

We found several limitations for movement within an MRI scanner, namely the size of the scanner (bore/diameter) and the ability of the participants to hold the movement under load for up to 3 minutes at a time. If the technique produced called for large movements, these would not be possible within the confined space.

Development of the technique involved us testing shoulder abduction under various tensions at two different distances while simultaneously measuring activation. Our initial thoughts were that deltoid muscle activity would be positively associated with both the distance and the force applied. While this held true for the large part, the differences in the actual measurements achieved in activating the deltoid using the same band at either 5 cm or 10 cm were minimal, except for the lowest tension band.

It became apparent when designing the task that a movement of 10 cm measured from the elbow to abduct the shoulder would not be achievable for larger participants within the confines of the MRI scanner. This combined with the minimal difference in activation scores between 5 cm and 10 cm made us implement the shorter distance for the technique in the MRI study. The scanner used in this study was a wide-bore scanner (70 cm) compared with a standard-bore scanner (55–60 cm). It was hoped that the 5-cm abduction technique would be possible on the majority of participants in standard-bore MRI scanners.

The ability of participants to hold their arm abducted under tension for approximately two and a half minutes during a MRI was also of concern. As tension increased during the EMG study the difficulty of movement became apparent, with participants fatiguing after holding the band for only a few seconds as well as moving their arm, contributing to a large number of the outliers we observed. This was particularly noticed with the black band and so this meant that we could not use this band for the final technique. This resulted in our final recommendation for using the green band at a distance of 5 cm for the activation of the deltoid muscle in a consistent and controllable manner within a MRI scanner.

One MRI participant found it difficult to perform and maintain 5 cm abduction for the allotted time, which resulted in movement artefacts on the images. The technique’s aim was to achieve a reproducible abduction distance, so if 5 cm abduction cannot be maintained, the distance of abduction should be reduced by centimetre decrements until a comfortable abduction position is achieved. The participant’s abduction distance can be noted and reproduced for subsequent visits.

Though surface EMG techniques are less accurate than conventional EMGs, it would have been inappropriate to use an invasive technique for research purposes in this setting. Other inaccuracies as a result of inconsistencies in the positioning of the probes may have contributed to some measurement error in our study. Despite using set anatomic landmarks for the positioning of the probes,19 those with a small deltoid muscle bulk may have had interference from other muscles. We did not account for repeat movements as well, and the impact that muscle fatigue, particularly in those with pre-existing shoulder injury may have on the signals obtained. We also did not control for pain and the impact this would have on the movement profile for different patients.

A small amount of humeral head migration was consistently observed while the deltoid was activated. While the true values may seem small, they are relative to a small space between the acromion and the humerus. In addition, the pilot work was performed in a group of participants who were awaiting rotator cuff repair whose rotator biomechanics may be better preserved than those with irreparable tears, who tend to have larger tears and more chronic pathology. People were recruited for these pilot studies from rotator cuff repair waiting lists so as to not impinge on recruitment to the main trial, but participants in the main study may have much greater migration under deltoid load.

To make the imaging task manageable for participants and to avoid fatigue, it was decided to restrict the number of shoulder sequences under load and use a relatively quick sequence. The use of a plastic former (constructed in house at University Hospitals Coventry and Warwickshire radiotherapy department) meant that it was easy to reproduce the position. The use of the green band gives a constant load that could be maintained by most participants for the duration of the scan. A single series of coronal oblique images was acquired while under load and then a repeat of the same sequence, while relaxed, were obtained for consistency and a consistent, reproducible measurement protocol was developed.

Abduction against and exercise band to a distance of 5 cm can be an effective way of activating the deltoid muscle, that can be applied in an MRI scanner. It can be performed safely and was successful in causing humeral head migration in our early pilot study.

Discussion of magnetic resonance imaging substudy findings

Recruitment to the MRI substudy from the main randomised trial was severely impacted by the COVID-19 pandemic, but also by the early stop in recruitment. When embedding a substudy into an adaptive randomised trial such as this, it would be best to front-load recruitment to the substudy as much as possible. We were able to achieve this in the lead site but many of our early sites were unable to participate in the substudy and it can be challenging to identify sites willing to undertake relatively time-intensive substudies. Despite this, a number of sites were open and recruiting to the substudy by the time of the pandemic, 44 participants had consented to the substudy and we were exploring imaging options at other sites that would have allowed us to resolve these issues. Unfortunately, the COVID-19 pandemic meant that people were unable attend hospitals for research visits and we were unable to increase the sample for this study.

Despite the difficulties in delivering the study during these challenging times, we found that the MRI technique did demonstrate narrowing of the acromiohumeral space under deltoid load, confirming our biomechanical theory that deltoid activation would act to draw the humerus proximally in the absence of a functioning rotator cuff. As with the development study, although these differences were small, they were consistently observed and are relative to an already narrow space.

With this small sample size, we did not observe a statistical difference between the randomised intervention groups. Given the clinical findings of the study, we might not expect a difference in MRI appearances. We did not identify any reason for the debridement performing better in this MRI study, and the reason for the clinical findings presented in Chapter 3 remains uncertain. Stopping early for futility also meant that the rationale for continuing to pursue an associated mechanistic study also changed. In this setting, we can be reassured that we did not see any clear evidence of harm on the images for either group.

Group sequential design

The primary interest of all the RCTs discussed in this substudy was to estimate the effect of the test treatment, compared to the control treatment, on the study outcome at the definitive (final) end point at time t (the primary study end point), which we call β_t hereafter. For all studies, we used the same primary or final end point as was originally used.

To implement the approach suggested by Parsons et al. ,61 we needed a single early outcome or a series of early outcomes in addition to the final outcome for each participant in the trial (e.g. primary outcome at 6 months with early outcome at 3 months or primary outcome at 9 months with early outcomes at 3 and 6 months). To assess whether a trial should be stopped at a particular time point during the course of follow-up, we used data not only from those participants with final outcome data but also from those participants with early outcome data.

The early outcomes provide information on the final outcome due to the correlation between the early and the final outcomes for each participant. A strong correlation between, for instance, 3- and 6-month outcomes suggests that a good (or poor) outcome at three months will be indicative of a good (or poor) outcome at 6 months. Independent of these correlations between successive outcomes for study participants, treatment effects for the early outcomes per se do not provide information on treatment effects for the final trial outcome β_t. This approach has been described previously by a number of authors for one, two and more generally any number of early outcomes. 58,126,127,141 The notation used here for the effect size estimate (β_t) reflects the fact that estimation follows from fitting a longitudinal linear model to the totality of outcome data (i.e. data from all trial time points). Methods for estimating (β_t) and var(β_t), using all the data available at any time point during follow-up, are described by Parsons et al. 61

The test statistic Z = β_t/sd(β_t) was used to make stopping decisions at the interim analyses using estimates of the covariance parameters (i.e. the correlations between outcomes and SDs of the outcomes) without constraints (i.e. the covariances parameters can take any value). In addition, as a means to assess the importance of the early outcome data in modifying estimates of the β_t and var(β_t), an analysis that forces all the correlations to be zero was also undertaken. We designate these parameters, which show the evidence for treatment effects using final outcome data only, as β0_t and var (β0_t), with Z = β0_t/sd(β0_t).

Stopping rules

Boundaries

Given the practical constraints imposed by the need for interim analyses to take place during the window of opportunity (after primary outcome data become available and before recruitment finishes), we restricted this study to a maximum of three interim analyses within any trial. The primary focus of this study was to assess whether and under what circumstances an adaptive design may have resulted in the selected trials stopping early. The selected trials are all pragmatic in outlook, testing interventions that are often already being widely used and, as such, the boundaries we choose needed to reflect the fact that stopping is most likely to be for treatment futility rather than efficacy. In many such pragmatic trials, one might argue that settings for the upper (efficacy) boundaries should favour the strategy of collecting as much information as possible if there is emerging evidence of efficacy; that is, not stopping early for efficacy unless there is very strong evidence.

Parsons et al. 61 suggested a range of possible futility boundaries defined by stopping probabilities in the setting of up to three interim analyses, that represented a sequence of four increasingly aggressive options, from a low probability of stopping for futility, labelled as (a), to a high probability, labelled as (d), with (b) and (c) intermediate to these (see Table 8).

In Table 11, at overall one-sided test level α and under the null hypothesis, α^*_u are the probabilities of stopping and rejecting the null hypothesis (H₀) in favour of alternative (efficacy), and α^*_l are the probabilities of stopping without rejecting H₀ (futility). All the futility scenarios (a) to (d) in Table 11 were tested for every trial, but for any trial stopping boundaries were defined by one or more of α^*_l and α^*_u.

The stopping probabilities from Table 11 were used to construct appropriate boundaries for standardised test statistics at each of the planned interim analyses for each trial. This required us to make some assumptions, based on what we believed the trial team may have thought prior to the commencement of recruitment, about (1) the number of possible interim analyses, (2) the covariances between the primary end points and (3) the number of data points that may have been available at each of the interim analyses. Using these data, we calculated the information necessary to trigger each interim analysis, which allowed us to define stopping boundaries for the observed test statistics.

Information monitoring

Data were generated using the observed trial outcome data and the observed recruitment and follow-up patterns, such that they represented the order that data would have accumulated in real time (i.e. the outcome data in the order they accumulated in the original trial). Information monitoring begins, after sufficient data are available to estimate accumulated information, and continues on a regular basis (every 2 weeks) to reflect what would likely have happened in the trial, if an adaptive design had been implemented. Once the required information level is reached, the test statistic is calculated and compared to the stopping boundaries, and decisions about whether the trial would have stopped or continued to the next interim analysis are recorded together with summary information on the current recruitment, time point, and outcome data summaries. This process is continued for subsequent interim analyses if necessary.

In summary, the simulated group sequential design for a trial proceeds as follows:

• Trial data are accumulated, in the order they were in the original study, and the observed information is estimated as I^* = 1/var(β_t).

• When I^* reaches the first preset threshold I₁, parameters β_t1 and var(β_t1) are estimated to give the test statistic Z₁ = β_t1/sd(β_t1).

• Data accumulation continues until I^* reaches the next and subsequent information thresholds and test statistics Z_w calculated.

• At the study end the observed information I^* is noted and data recorded.

Decisions about whether the simulated trial would have stopped, either for efficacy or futility, were made by comparing the estimated test statistics at each interim analysis to the stopping boundaries for the four scenarios (a) to (d). Also, at each interim analysis data on all those study participants recruited up to that point were used to estimated model parameters for the over-running analysis. 80,142 This analysis comprised all the data that would eventually have accumulated on those participants already recruited and is the definitive analysis that would have been reported, had the trial stopped recruiting at the interim analysis. In the over-running analysis, boundaries were recalculated based on the observed information at the final analysis and used to draw interference on the observed treatments effects.

Trial data

Each of the selected seven RCTs are briefly described in Appendix 4 With full details available from the published protocols and papers describing the main clinical results.

Group sequential designs

WOLLF: simulated group sequential trial

Figure 10 shows the observed number of participants recruited and followed up at 3, 6, 9 and 12 months for WOLLF, the window of opportunity available for interim analyses and the times when the interim analyses would have occurred in the simulated adaptive design.

FIGURE 10.

Recruitment (―), follow-up (―) and numbered interim analyses (+) for WOLLF; window of opportunity for interim analyses is shaded.

The trial hit the recruitment target of n = 460, and all the three planned interim analyses could feasibly have occurred. The numbers of observed trial participants providing data at each interim analysis is shown in Table 12. Estimated model parameters and test statistics are shown in Table 13.

The estimated correlations and SDs at each interim analysis are shown in Appendix 4. The estimated correlations were all generally larger than those used in the model to determine the timings of the interim analyses (i.e. ρ = 0.5 for all pairs of times) and SDs were much as expected (i.e. σ = 25 for all times), at all interim analyses.

Figure 11a shows the stopping boundaries and Z and Z0 from Table 13, at each interim analysis and study end. Test statistic Z is in the continuation region (between upper and lower boundaries) at interim analysis 1, for all boundary settings a to d. However, at interim analysis 2, Z falls below the lower boundaries for settings b to d, indicating that the study would have been stopped for futility. Z does not fall below the lower boundary or above the upper boundary for setting a, so the study would not have stopped for this setting.

FIGURE 11.

Stopping boundaries and Z and Z0 for decision-making. (a) For stopping decisions using data available at each interim analysis. (b) Using over-running data.

The over-running analysis (see Figure 11b) confirms the results of the stopping decisions based on the interim data and indicate stopping for futility at settings b to d (see Appendix 4 for full results of the over-running analysis).

Estimates of the treatment effects β_t, from the longitudinal model, are much closer to raw differences in means (Δ) for the over-running analysis as there is less information available from the early outcomes at this analysis. If there were complete follow-up data available for all participants, then (β_t) would be equal Δ. However, in practice it is almost always the case that some participants who did not provide 12-month outcomes had early outcomes (at one or more than one of 3, 6 and 9 months), which provide some information on the 12-month outcomes. Estimates of treatment effects β0_t, from the models that force correlations (between all outcomes) to be zero, are always equal to Δ and have larger variances than (β_t).

WOLLF: summary

The results of the simulated group sequential design for WOLLF can be summarised as follows:

• For three of the four boundary settings tested, the WOLLF study would have stopped at the second interim analysis, when data were available from n = 74 participants with 12-month outcomes, n = 85 with 9 months, n = 136 with 6 months and n = 188 with 3-month outcomes.

• At this interim analysis, N = 293 participants had been recruited into the study and follow-up would have been completed in 27 months; this compares with N = 460 and 50 months for the original study.

• The estimated treatment effect, for 12-month DRI, after completing follow-up for the stopped study (over-running analysis) was −3.5 favouring the control treatment; the estimate of the treatment effect in the original (fixed design) study was −3.1 (95% CI −8.5 to 2.2).

• At the second interim analysis when the study was stopped, the estimated treatment effect from the model was −2.8 (favouring the control), and the raw difference between groups using 12-month data only was 4.1 [favouring negative pressure wound therapy (NPWT)].

• There were extremely strong correlations between the early outcomes (3, 6 and 9 months) and the primary outcome (12 months) for DRI, meaning that at interim analysis inferences based on the modelling approach, that used all the data, gave much better estimates of the true end of trial treatment effect, than simple differences in primary outcome (12 months) group means.

• The stronger than expected correlations also meant that the interim analyses generally occurred early than expected. That is, the observed number of participants providing 3-, 6-, 9- and 12-month outcome data at each interim analysis was less than the expected number.

DRAFFT: simulated group sequential trial

Figure 12 shows the observed number of participants recruited and followed up at 3, 6 and 12 months for DRAFFT, the window of opportunity available for interim analyses and the times when the interim analyses would have occurred in the simulated adaptive design.

FIGURE 12.

Recruitment (―), follow-up (―) and numbered interim analyses (+) for DRAFFT; window of opportunity for interim analyses is shaded.

The study surpassed the recruitment target of n = 390, and the planned interim analysis could feasibly have occurred. The numbers of study observed trial participants providing data at the interim analysis is shown in Table 14. Estimated model parameters and test statistics are shown in Table 15.

The estimated correlations and SDs at each interim analysis are shown in Appendix 4. The estimated SDs for the 12-month outcome were much smaller than expected when planning the study; σ_12m was expected to around 20 but was actually nearer to 15. This explains why the numbers of participants at the interim analysis was much smaller than planned, as it took fewer participants to accumulate the required information, due to the smaller than expected value for σ_12m. Also, the fast rate of recruitment and small number of 12-month outcomes meant that it was problematic to hit the required information level exactly; the value for I^*₁ is larger than I₁.

Figure 13a shows the stopping boundaries and Z and Z0 from Table 15, at each interim analysis and study end. Test statistic Z is in the continuation region (between upper and lower boundaries) at interim analysis 1 for all boundary settings a to d, indicating that the study would not have stopped.

FIGURE 13.

Stopping boundaries and Z and Z0 for decision-making. (a) For stopping decisions using data available at each interim analysis and (b) using over-running data.

The results of the over-running analysis (see Figure 13b) confirm the stopping decisions based on the interim data, that the study would not have stopped at the interim analysis (see Appendix 4 for full results of the over-running analysis).

DRAFFT: summary

The results of the simulated group sequential design for DRAFFT can be summarised as follows:

• For all four boundary settings tested, the DRAFFT study would not have stopped at the interim analysis, when data when available from n = 26 participants with 12-month outcomes, and n = 135 with 6-month outcomes and n = 205 with 3-month outcomes.

• At this interim analysis N = 294 participants had been recruited into the study and follow-up would have been completed in 15 months; this compares to N = 461 and 34 months for the original study.

• The estimate of the SD of the primary outcome (σ_12m), used in the original sample size calculation, and used to build the group sequential design, was smaller than the true value (15 vs. 20). This caused the interim analysis to take place at a much earlier time than planned (i.e. with fewer participants with 12-month outcomes than expected; N_12m = 26 vs. N_12m = 100).

• In reality, the original DRAFFT study recruited at an even faster rate than expected, and the sample size was increased from n = 390 to n = 461. This gave greater precision in the estimate of the treatment effect, which was important for inferences and the health economics analysis particularly. 145

FixDT: simulated group sequential trial

Figure 14 shows the observed number of participants recruited and followed up at 3, 6, and 12 months for FixDT, the window of opportunity available for interim analyses and the times when the interim analyses would have occurred in the simulated adaptive design.

FIGURE 14.

Recruitment (―), follow-up (―) and numbered interim analyses (+) for FixDT; window of opportunity for interim analyses is shaded.

The trial hit the recruitment target of n = 320, and the planned interim analyses could both feasibly have occurred. The numbers of observed trial participants providing data at each interim analysis is shown in Table 16. Estimated model parameters and test statistics are shown in Table 17.

The estimated correlations and SDs at each interim analysis are shown in Appendix 4. The estimated SDs for the 6-month outcome were much larger than expected when planning the study; σ_6m was expected to around 20 but was actually nearer to 24. This explains why the numbers of participants at the interim analysis were larger than planned, as it took more participants to accumulate the required information, due to the larger than expected value for σ_6m.

Figure 15a shows the stopping boundaries and Z and Z0 from Table 17, at each interim analysis and study end. Test statistic Z is in the continuation region (between upper and lower boundaries) at interim analysis 1, for all boundary settings a to d. However, at interim analysis 2, Z falls below the lower boundaries for settings b to d, indicating that the study would have been stopped for futility.

FIGURE 15.

Stopping boundaries and Z and Z0 for decision-making. (a) For stopping decisions using data available at each interim analysis and (b) using over-running data.

The over-running analyses (see Figure 15b) confirms the results of the stopping decisions based on the interim data, and indicate stopping for futility at settings b to d. In fact, the over-running analyses provided stronger evidence in favour of stopping at all interim analyses and settings a to d (see Appendix 4 for full results of the over-running analysis).

FixDT: summary

The results of the simulated group sequential design for FixDT can be summarised as follows:

• For three of the four boundary settings tested, the FixDT study would have stopped at the second interim analysis, when data when available from n = 146 participants with 6-month outcomes, and n = 178 with 3-month outcomes.

• At this interim analysis, N = 243 participants had been recruited into the study and follow-up would have been completed in 27 months; this compares with N = 321 and 42 months for the original study.

• The estimated treatment effect after completing follow-up for the stopped study (over-running analysis) was −6.2 favouring the control treatment; the estimate of the treatment effect in the original (fixed design) study was −4.0 (95% CI −9.6 to 1.6).

• At the second interim analysis when the study was stopped, the estimated treatment effect was −2.9 (favouring the nail), and the raw difference between groups was −0.5 (favouring the nail).

• There was a strong correlation between the early outcome (3 months) and the primary outcome (6 months) for DRI.

• The estimate of the SD of the primary outcome (σ_6m) used in the original sample size calculation, and used to build the group sequential design, was a marked underestimate of the true value (20 vs. 24). This caused the study to be underpowered and the interim analyses to be at later times than planned (i.e. with more participants than expected).

FASHIoN: simulated group sequential trial

Figure 16 shows the observed number of participants recruited and followed up at 3, 6 and 12 months for FASHIoN, the window of opportunity available for interim analyses and the times when the interim analyses would have occurred in the simulated adaptive design.

FIGURE 16.

Recruitment (―), follow-up (―) and numbered interim analyses (+) for FASHIoN; window of opportunity for interim analyses is shaded.

The trial hit the recruitment target of n = 344, and the planned interim analyses could both feasibly have occurred. The numbers of observed trial participants providing data at each interim analysis is shown in Table 18. Estimated model parameters and test statistics are shown in Table 19.

The estimated correlations and SDs at each interim analysis are shown in Appendix 4. The estimated SDs for the 12-month outcome are slightly larger than expected when planning the study; σ_12m was expected to around 24 but was actually nearer to 26. This explains why the numbers of participants at the interim analysis were slightly larger than planned, as it took more participants to accumulate the required information, due to the larger than expected value for σ_12m.

Figure 17a shows the stopping boundaries and Z and Z0 from Table 19, at each interim analysis and study end. Test statistic Z is in the continuation region (between upper and lower boundaries) at interim analysis 1 and 2, for all boundary settings a to d. Figure 17b shows that the interim analyses based on the over-running data indicate stopping for efficacy for all settings a to d, at the second interim analysis. This is consistent with the final data analysis of the original fixed design study (see Appendix 4 for full results of the over-running analysis).

FIGURE 17.

Stopping boundaries and Z and Z0 for decision-making. (a) For stopping decisions using data available at each interim analysis. (b) Using over-running data.

FASHIoN: summary

The results of the simulated group sequential design for FASHIoN can be summarised as follows:

• The recruitment and follow-up profiles (see Figure 16) for FASHIoN were unusual and reflected the phased approach to the study with an initial feasibility/pilot stage at a small number of sites, followed be more rapid recruitment as many more sites were opened.

• This resulted in relatively little benefit available from the early outcomes at the interim analyses – for example, at the first interim analysis 12-month data were available from 62 participants and 6-month data from 86 participants.

• Of the four boundary settings tested, the FASHIoN study would not have stopped at either interim analysis for futility or efficacy.

• The estimated treatment effect after completing follow-up for the second interim analysis (over-running analysis) was consistent with result of the original study.

• The estimate of the treatment effect in the original (fixed design) study was 9.1 (95% CI 3.3 to 14.9), which is consistent with the over-running analysis from the second interim analysis (9.1) suggesting that the study could legitimately have stopped at the second interim analysis.

WAT: simulated group sequential trial

Figure 18 shows the observed number of participants recruited and followed up at 6 weeks, 3, 6, and 12 months for WAT, the window of opportunity available for interim analyses and the times when the interim analyses would have occurred in the simulated adaptive design.

FIGURE 18.

Recruitment (―), follow-up (―) and numbered interim analyses (+) for WAT; window of opportunity for interim analyses is shaded.

The trial hit the recruitment target of n = 120, and the planned interim analysis could feasibly have occurred. The numbers of observed trial participants providing data at each interim analysis is shown in Table 20. Estimated model parameters and test statistics are shown in Table 21.

The estimated correlations and SDs at each interim analysis are shown in Appendix 4. The estimated SDs for the 12-month outcome at the interim analysis was much smaller than expected; σ_12m was expected to around 9 but was actually nearer to 5. Also, the correlations were generally larger than expected. These factors explain why the numbers of participants at the interim analysis was much smaller than planned, as it took many fewer participants to accumulate the required information, due to the larger than expected correlations and smaller than expected value for σ_12m.

Figure 19a shows the stopping boundaries and Z and Z0 from Table 21, at each interim analysis and study end. Test statistic Z is in the continuation region (between upper and lower boundaries) at the interim analysis, for all boundary settings a to d.

FIGURE 19.

Stopping boundaries and Z and Z0 for decision-making. (a) For stopping decisions using data available at each interim analysis. (b) Using over-running data.

The test statistic from the over-running analysis (see Figure 19b) confirms the results of the stopping decisions based on the interim data and indicates that the study would not have stopped for any of the settings a to d (see Appendix 4 for full results of the over-running analysis).

Estimates of the treatment effects β_t, from the longitudinal model, are close to raw differences in means (Δ) for the over-running analysis as there is less information available from the early outcomes at this analysis. If there were complete follow-up data available for all participants, then β_t would be equal Δ. However, in practice it is almost always the case that some participants who did not provide 12-months outcomes had early outcomes (at one or more than one of 6 weeks, 3 months and 6 months), which provide some information on the 12-month outcomes. Estimates of treatment effects β0_t, from the models that force correlations (between all outcomes) to be zero, are always equal to Δ, and have larger variances than β_t).

WAT: summary

The results of the simulated group sequential design for WAT can be summarised as follows:

• For all four boundary settings tested, the WAT study would not have stopped at the interim analysis, when data were available from n = 10 participants with 12-month outcomes, n = 29 with 6-month outcomes, n = 43 with 3-month outcomes and n = 49 with 6-week outcomes.

• At this interim analysis N = 75 participants had been recruited into the study and follow-up would have been completed in 17 months; this compares with N = 126 and 48 months for the original study.

• The estimate of the SD of the primary outcome (σ_12m = 9), used in the original sample size calculation, and used to build the group sequential design, was much larger than the observed value at the interim analysis. Also, many of the correlations were significantly larger (e.g. ρ_3m,12m = 0.71, ρ_3m,6m = 0.90, ρ_6w,3m = 0.86) than anticipated (e.g. ρ_3m,12m = ρ_3m,6m = ρ_6w,3m = 0.5).

• This caused the interim analysis to take place at a much earlier time than planned (i.e. with fewer participants with 12-month outcomes than expected; n = 10 vs. n = 40).

• Given the small, but not clinically significant, result observed in the original study, it seems unlikely that any sensible stopping rule would have caused the WAT study to stop early.

CSAW: simulated group sequential trial

Figure 20 shows the observed number of participants recruited and followed up at 6 months for CSAW, the window of opportunity available for interim analyses and the times when the interim analyses would have occurred in the simulated adaptive design.

FIGURE 20.

Recruitment (―), follow-up (―) and numbered interim analyses (+) for CSAW; window of opportunity for interim analyses is shaded.

The trial hit the recruitment target of n = 200, and the planned interim analyses could feasibly have occurred. The numbers of observed trial participants providing data at each interim analysis is shown in Table 22. Estimated model parameters and test statistics are shown in Table 23.

The estimated SDs at each interim analysis were as follows: σ_6m = 12.2, 11.7, 11.8. The estimated SDs for the 6-month outcome at the interim analysis was larger than expected; σ_6m was expected to around 9 but was actually nearer to 12. This explains why the numbers of participants at the interim analysis were larger than planned, as it took more participants than expected to accumulate the required information.

Figure 21a shows the stopping boundaries and Z and Z0 from Table 23, at each interim analysis and study end. Test statistic Z is in the continuation region (between upper and lower boundaries) at the interim analysis, for all boundary settings a to d. Note that because we have no early outcome data for CSAW, Z is equal to Z0 at all analyses.

FIGURE 21.

Stopping boundaries and Z and Z0 for decision-making. (a) For stopping decisions using data available at each interim analysis. (b) Using over-running data.

The test statistics (Z) from the over-running analysis (see Figure 21b) confirm the results of the stopping decisions based on the interim data and indicate that the study would not have stopped for any of the settings a to d.

There is no evidence in the simulated study to believe that an adaptive design would have caused CSAW to stop early. Indeed, there is some reason to believe that an adaptive design may have complicated the inferences as in the simulated designs described here, in contrast to the original result of the study,69 the final (over-running) analysis does not reject the null hypothesis (although Figure 21b shows that the test statistic is close to the threshold). This is because by spending some of the overall error testing at interim analyses, we make it to harder to reject at the end of the study.

CSAW: summary

The results of the simulated group sequential design for CSAW can be summarised as follows:

• For all four boundary settings tested, the CSAW study would not have stopped at the interim analyses, when data were available from n = 79 and n = 137 participants with 6-month outcomes.

• At these interim analysis N = 142 and N = 195 participants had been recruited into the study and follow-up would have been completed in 21 and 28 months, respectively; this compares with N = 210 and 39 months for the original study.

• The estimate of the SD of the primary outcome (σ_6m = 9), used in the original sample size calculation and used to build the group sequential design, was smaller than the observed value at the interim analyses.

• This caused the interim analysis to take place at a much later than planned (i.e. with more participants with 6-month outcomes than expected; n = 79 vs. n = 40 and n = 137 vs. n = 80).

• Given the small but not clinically significant result observed in the original study, it seems unlikely that any sensible stopping rule would have caused the CSAW study to stop early for efficacy, unless some early outcome data (e.g. 3 months) had been available.

TOPKAT: simulated group sequential trial

Figure 22 shows the observed number of participants recruited and followed up at 2 months, 1, 2, 3, 4 and 5 years for TOPKAT. The recruitment for TOPKAT completed in June 2013 and the first five-year (primary) outcome data were not available until March 2015. Therefore, the window of opportunity, between some five-year final outcome data being available and recruitment completion was non-existent and, as such, the methodology we are investigating here, assessing possible early stopping of the trial, could not have been used.

FIGURE 22.

Recruitment (―), follow-up (―) for TOPKAT; there is no window of opportunity for interim analyses.

Overview

For five of the selected trauma and orthopaedic trials discussed here (WOLLF, FixDT, DRAFFT, FASHIoN and WAT), the methodology of Parsons et al. ,61 that used early outcome data in addition to final outcome data to inform stopping decisions at interim analyses, proved to be feasible. All of the putative group sequential designs described for these five studies used only information that was known (or thought to be known) or could have reasonably been speculated on (e.g. the numbers and patterns of patient data), at the study design and planning stage. The designs do not knowingly use any information from the observed trial publications or data. For this reason, we believe that the results of the simulated trials that used the observed data and the known dates when data were collected for each trial are a true test of whether the design would have been possible, whether the trial would have stopped early and if so whether the result would have been consistent with that obtained from the original (fixed) design.

The CSAW and TOPKAT studies were different from the other trials discussed here in two key respects that made an adaptive design of the type under discussion here impossible, the lack of early outcome data for CSAW and the lack of a window of opportunity for TOPKAT. For these reasons, these two trials are discussed after the other five studies.

WOLLF, FixDT, DRAFFT, FASHIoN and WAT

Looking at each of these five trials in turn, the WOLLF study would have stopped early at the second interim analysis for three of the four boundary settings tested, when 293 participants had been recruited into the study (c.f. 460 in the original trial) and follow-up would have been completed in 27 months (c.f. 50 months for the original study).

Inferences for the stopped studies would have been very similar to the original study; the over-running analysis gave an estimate of the treatment effect equal to −3.5 (favouring the control), whereas the treatment effect estimate in the original (fixed design) study was −3.1 (95% CI −8.5 to 2.2). Of particular note in WOLLF were the extremely strong correlations between the early outcomes (3, 6 and 9 months) and the primary outcome (12 months) for DRI. The strong correlations are important for two reasons. First, they allowed the modelling approach, that used all the data, to give better (more precise) estimates of the true end of trial treatment effect, than simple between-group mean differences for the primary 12-month outcome. Second, the stronger than expected correlations also meant that information accrued rapidly causing the interim analyses to occur earlier than might have been expected based on the number of participants with 12-month outcomes alone.

At the second interim analysis when the study was stopped, the estimated treatment effect from the model was −2.8 (favouring the control). In marked contrast, the raw difference between groups using 12-month data only was 4.1 (favouring NPWT). As we note above, the strong correlations meant that the model estimate was a much better estimate of the true treatment effect than simple between-group difference in 12-month outcomes. However, it is worth considering how this might have played out if this had been the situation in the real trial. Would the trial data monitoring and safety committee have had the confidence in the model estimate to stop the study for futility when the difference in the means for 12-month data alone was strongly favouring NPWT?

The FixDT trial was of a similar overall design to WOLLF, as it ran concurrently with and was designed by the same research team. As for WOLLF, for three of the four boundary settings tested, the FixDT study would have stopped at the second interim analysis, when 243 (c.f. 321 in the original study) participants had been recruited into the study and follow-up would have been completed in 27 months (c.f. 42 months for the original study). Therefore, as for WOLLF, there would have been a considerable saving in time and cost, if an adaptive design had been used.

The treatment estimate for FixDT after completing follow-up for the stopped study was −6.2 favouring the control (nail) treatment and the estimate of the treatment effect in the original (fixed design) study was −4.0 (95% CI −9.6 to 1.6). The estimate of the SD of the primary outcome (σ_6m) used in the original sample size calculation, and used to build the group sequential design, for FIXDT was a marked underestimate of the true value (20 vs. 24). This caused the original study to be underpowered and the interim analyses to be at later times than planned. That is, when there were more participants than expected (146 vs. 100) with 6-month outcome data.

WOLLF and FixDT both reported results in the original studies favouring the control treatments; intramedullary nail fixation in FixDT and standard dressing in WOLLF. The boundaries for our designs reflected the wish to stop for futility if there were a lack of emerging evidence to support better outcomes for the comparator test treatments (locking-plate fixation and NPWT, respectively). Locking-plate fixation and NPWT both proved unlikely to be a cost-effective compared with intramedullary nail fixation and a standard dressing respectively in reported health economic analyses. 146,147

In contrast to WOLLF and FixDT, the DRAFFT trial provided treatment effects throughout the study that tended to marginally favour the test locking-plate treatment over wire fixation control treatment. Therefore, it is perhaps not surprising given the asymmetry of the boundaries that the DRAFFT study would not have stopped at the interim analysis for any of the four boundary settings tested in the simulated studies.

The estimate of the SD of the primary outcome for DRAFFT (σ_12m), used in the original sample size calculation and to build the group sequential design, was much smaller than expected (15 vs. 20). This caused the interim analysis to take place much earlier than planned when there were many fewer participants with 12-month outcomes than expected; 26 rather than the expected 100. The stronger than expected correlations also in part contributed to the early interim analysis. As did the extremely rapid recruitment to DRAFFT caused by a surge in recruitment due to the harsh winter weather causing a surge in distal radius fracture as a consequence of falls. However, given the consistent positive treatment effects in favour of the plate intervention it seems unlikely that better estimates of the covariance parameters or change of boundaries (with reason) would have caused the study to stop early for futility.

The FASHIoN study had a quite different recruitment profile from the other trials (see Figure 19) due to the phased approach with an initial feasibility/pilot stage at a small number of sites, followed be more rapid recruitment as sites were opened. This caused there to be relatively little benefit available from early outcomes (6 months) in addition to that provided by the final 12-month outcomes.

Of the four boundary settings tested, the FASHIoN study would not have stopped at either interim analysis for futility or efficacy. The two interim analyses both provided evidence in favour of the surgical intervention, but test statistics were not of sufficient magnitude to cause the trial to stop. Although, the estimated treatment effect after completing follow-up for the second interim analysis (over-running analysis) was consistent with result of the original study, which reported a positive result in favour of the surgery intervention. The lack of stopping (for efficacy) for FASHIoN is in large part due to the asymmetric selection of boundaries that made it relatively harder to stop for efficacy. The boundaries reflected the widespread view within trauma and orthopaedic medicine that much stronger evidence is required to cause a trial to stop for efficacy than futility, with many clinicians believing that if there is emerging evidence for efficacy then a trial should complete recruitment to target in order to provide a precise estimate of the treatment effect and capture as much safety information as possible (e.g. adverse events). The trials reported here tested interventions that typically were available within the NHS, so there was no ethical imperative to end trials as soon as possible and offer all patients the superior intervention, as for instance might be case when testing novel pharmaceutical drugs.

Given the relatively small sample size and the small (clinically insignificant) result observed in the original WAT trial, it seems unlikely that any sensible stopping rule would have caused the study to stop early. For all four boundary settings tested, the WAT study would not have stopped at the interim analysis. As for some of the other trials discussed here, the estimate of the SD of the primary outcome, used in the original sample size calculation, and used to build the group sequential design, was much larger than the observed value at the interim analysis. Also, for WAT, many of the estimated correlations were significantly larger than anticipated. These together caused the interim analysis to take place at a much earlier time than planned, when there were many fewer participants with 12-month outcomes than expected (n = 10 vs. n = 40).

CSAW and TOPKAT

For the CSAW and TOPKAT trials it was not possible to use the methodology of Parsons et al. 61 directly as for the former study there was no early outcome data available and for the latter no final outcome data were available prior to recruitment completing. Therefore, for TOPKAT we cannot proceed to simulate an adaptive study based on early assessment of the treatment effect on the final outcome.

For CSAW, although no early outcome data were available, we can simulate how the study would have proceeded via a group sequential design based simply on the final outcome data alone. This is amounts to using the test statistic Z0, rather than Z, to make stopping decisions; Z0 forces all the correlations between the final and early outcomes to be zero, and by so doing in practice uses only final outcome data for decision-making. Using this methodological approach, there was no evidence in the simulated study to believe that an adaptive design would have caused CSAW to stop early. Indeed, there is some reason to believe that an adaptive design may have complicated the inferences as in the simulated designs described here, in contrast to the original result of the study,69 the final (over-running) analysis does not reject the null hypothesis (although Figure 20b shows that the test statistic is close to the threshold). This is because by spending some of the overall error testing at interim analyses, we make it to harder to reject at the end of the study. However, there is reason to think that if an early outcome had been collected (e.g. at 3 months) then given that it correlated reasonably stronger with the final outcome, this may have caused us to stop for efficacy at the second interim analysis. This is not an unreasonable suggestion, as the over-running analysis at this occasion very nearly crossed the upper stopping boundary.

Summary

The results for five of the studies reported here (WOLLF, FixDT, DRAFFT, FASHIoN and WAT) showed that adaptive design using early outcome data would have been feasible and likely to provide designs that were at least as efficient, and possibly more efficient, than the original fixed sample size designs. For WOLLF and FixDT the simulations showed that it was highly likely these studies would have (correctly) stopped early for futility, saving potential considerable effort and resources. WOLLF particularly showed the important part that early outcome data particularly can play, as analyses based purely on the final outcome data alone would have meant that stopping (for any reason) would have been unlikely. The boundaries selected here favoured stopping for futility, at the cost of making stopping for efficacy unlikely, unless there were very strong evidence available. For this reason, the two studies that showed modest effect estimates at interim analyses in favour of the test treatment (WAT and DRAFFT), did not stop early. This was consistent with the final results of these studies. The FASHIoN trial showed good evidence in favour of the test surgical intervention in the final analysis but fell short of stopping at the interim analyses. For this study it would have been possible to select different, but sensible, boundaries that would have resulted in early stopping for efficacy.

For all the studies it was clear that the feasibility and practicality of using the methods proposed by Parsons et al. 61 was determined in large part by: (1) the width of the window of opportunity for stopping; (2) the available of early outcome data and their correlations with final outcomes; (3) recruitment and outcome data follow-up accrual profiles; and (4) the veracity of the estimates of the covariance parameters available at the design planning stage. We assumed that the authors had a high degree of confidence in the primary outcomes, which would be a requisite for the method presented, although it may be argued this is true for conventional sample size calculations too. The timing of the first interim analysis could be chosen to ensure sufficient data are also available on other outcomes (e.g. sufficient economic data in the case of an efficacy stopping rule).

The first of the three issues we highlight here were evident for all the trials. If there were little or no final outcome data available at interim analyses, and little or no early outcome data were available or uncorrelated with final outcomes then decision-making for early stopping were simply not possible. The pattern of data accrual and follow-up were important determinants of the feasibility of the methods used. However, more work is needed to fully understand the impact of different approaches to recruitment and follow-up on the widespread applicability of the methods. For instance, whether limiting or increasing recruitment at certain stages of a trial may be beneficial in certain circumstances. It was also clear for a number of the trials that the times when interim analyses occurred were either much earlier or later than expected. This was largely due to estimates of the covariance estimates used in the initial planning being markedly different from the observed values. For instance, if correlations between outcomes were stronger than expected and variances smaller, then interim analyses would occur sooner than might have been expected. This in itself is not necessarily problematic, as we deliberately motivated stopping based on information rather than purely sample size considerations. However, in instances where interim analyses occurred particularly early (e.g. in the DRAFFT study an interim analysis occurred when there were final outcome data from 26 participants rather than the expected 100), it is likely that in practice it would have been difficult for the trial DMC, TMG and TSC to make and confirm stopping decisions and justify these to the funding body based on so few data. In practice, either minimum sample sizes might have to be prespecified or interim analyses plans be modified as the study proceeds (e.g. by using blinded re-estimation of the covariance parameters as data accumulates to update the trial plans).

In many of the trials, correlations between outcomes were much stronger than expected (e.g. > 0.7). If there are such strong correlations between early and final outcomes, then it may also be that there are strong correlations between treatment effect estimates (e.g. treatment effects for an early outcome at 6 months were much the same as those for the primary outcome at 12 months). If this is the case, then arguably we might want to consider using the 6 months outcome as the primary. If this is the case, then this would be a simpler strategy to shorten the trial and save costs. With an earlier primary outcome, the adaptive designs presented here would also become even more efficient, with more early primary outcome data available at each interim analysis and a better likelihood of a strong correlation between (for example) 3- and 6-month outcomes.

There are a number of limitations to this study. We have chosen to use the same sample size in the adaptive designs as was used in the original fixed sample size designs. In practice to maintain power, the sample size for the adaptive designs would have needed to be increased relative to the fixed designs, to allow for the interim analyses and possibility of early stopping. However, by increasing the sample size it would have not been possible to use just the original data and observed recruitment and follow-up patterns, as we would have been forced to, for instance, simulate some or all the data using a model or resample data. This would have diminished some of the clinical impact of the work, as the simulated trials would have deviated considerably from the original trials. Therefore, given that we were predominantly interested in assessing whether the simulated adaptive trials would have stopped, we decided to not to modify the original sample sizes.

We have tried, as strictly as possible, to avoid using information that was only known after trial data were available when planning the adaptive designs. For instance, by using estimates of variances from the published protocols and, where possible, details of recruitment and follow-up strategies from the trial teams. However, it may be that the results or knowledge of the selected trials may have unconsciously influenced the adaptive designs (e.g. the timing and number of interim analyses). We have used the date when outcome data were ‘collected’ as a proxy for when it would have been available to make stopping decisions. However, in reality in a trial it often takes some time to enter the data on the study database and extract data (e.g. freeze and check data ready for analysis). These data would then need to be sent to the trial statistician to undertake an analysis, circulate to colleagues on a data and safety monitoring committee to meet and discuss the results and make a recommendation to the TMG/TSC to finalise the decision. This would typically take some time – a number of weeks at least. We have not accounted for these delays in the simulation study, so it maybe that our assessments of the savings an adaptive design might have made (in terms of time and cost) may be somewhat optimistic. In reality, however, many of these tasks could be better planned, streamlined and automated to a large extent, if an adaptive design were being used.

In summary, adaptive clinical trials may be applied in a number of settings. The applicability and efficiency of these methods are dependent on trial design, especially in terms of the timing of the relevant outcomes and the correlation between early and late outcomes, as well as the expected speed of recruitment. In two of the trials evaluated here, major time savings could have been made in terms of completing the study and reporting early. This aligns with our experience from the START:REACTS trial presented earlier in this monograph. Clearly not all studies will stop early if the methods are applied correctly, but across a body of trials, major efficiency savings could be made. Such savings could reduce research costs and reduce the number of people exposed to potential harm in trials. However, they may also deliver clinical trial results quicker, resulting in earlier changes in practice that also result in major cost savings for the health service. In the IDEAL classification, it may allow more efficient progress between stages 2b and 3. This may be particularly important in the introduction of new surgical procedures, where risks of harm are more likely than in established procedures, and these may be identified earlier and prevented across the community.

The application of more complicated statistical techniques such as this inevitably adds some cost, although this would be expected to reduce substantially as statistical code become readily available and experience is gained across the statistical community. These development costs would likely be offset not only by reduced overall research costs, but also by major savings made across the health-care community by the earlier delivery of important trial results, although this would be a topic for future research.

Data monitoring committee members

Professor Thomas Jaki (Chair), Professor Stephen Gwilym, Ms Loretta Davies.

Trial steering committee members

Professor Rod Taylor (Chair), Professor Angus Wallace, Dr Christopher Littlewood, Dr Anthony Howard, Mr Geoffrey Abel, Mr John Graham.

Sex

Figure 24 show box plots of the OSS at 12 months postrandomisation by participant sex and allocation group. Table 34 shows the full details of the adjusted model of the OSS at 12 months with the allocation group and sex interaction term.

FIGURE 24.

Box plot of OSS at 12 months by sex and allocation group.

Here, the effects of age and tear size remain in line with the main efficacy model, and while the effect of the InSpace device is much reduced, it still favours the arthroscopy only group. Effects of sex are reversed with females now performing slightly better than males. However, the interaction term suggests that females have poorer outcome when in the device allocation group. This can be seen visually in Figure 25, which shows the group means with their 95% CIs for allocation group and participant sex.

FIGURE 25.

Means and 95% CIs of OSS at 12 months by sex and intervention.

Tear size

Table 35 shows the adjusted model results for the OSS at 12 months for the tear size subgroup and Figure 26 shows these results graphically. These results suggest that participants with small and medium tears have better OSS values at 12 months. However, the number of participants with small or medium tears is very low (n = 6, 5% of randomised participants), leading to an imprecise estimate of effect size.

FIGURE 26.

Interaction effects of the allocation group and tear size on the OSS at 12 months. Owing to the small number of medium or small tears in the device group, this CI could not be computed.

To further explore the effects of tear size; instead of using the tear size category, the two measurements of tear taken during surgery were used as continuous measures. The adjusted model results are shown in Tables 36 and 37 shows the summary statistics of the two tear measurements, which is broadly similar to the model using the categories of tear size.

The two tear sizes were modelled separately as interaction effects; the anteroposterior measurement is shown in Table 38 and the mediolateral in Table 39. Both interaction effect models show large changes to the observed effect of the allocation group, however, the overall effect of the InSpace device remains below that of the debridement only group.

Age group

Table 40 shows the results of the model including the interaction term of age group and allocation group, which is shown graphically in Figure 27. These results remain broadly consistent with the secondary efficacy model.

FIGURE 27.

Interaction effects of the allocation and age groups on the OSS at 12 months.

Constant score

Descriptive statistics for the Constant score at each follow-up point are shown in Table 41 and Figure 28. The high level of missing data is due to the lack of in-person follow-up during the COVID-19 pandemic. Note that while there appears to be a large difference between allocation groups at 12-month follow-up, approximately 80% of the data at that time point are missing. Due to the high level of missing, the Constant score has not been modelled.

FIGURE 28.

Box plot of the Constant score at each follow-up point by allocation group.

Objective shoulder function

While performing the assessment for the Constant score, the exact measurements of participant shoulder function were also recorded. These data are summarised below in Table 42, Figure 29 and Figure 30. Owing to the high level of missing, these objective assessments have not been modelled.

FIGURE 29.

Box plot of shoulder flexion angle by allocation group and time point.

FIGURE 30.

Box plot of shoulder abduction angle by allocation group and time point.

Western Ontario Rotator Cuff Index

The WORC index is a patient-reported quality-of-life outcome measure designed for patients with rotator cuff disease. Scores were calculated on a 0–100 scale such that higher score mean better function. The summaries of the data are shown in Table 43 and Figure 31.

FIGURE 31.

Box plot of WORC score by allocation group and time point.

Table 44 shows the results of the adjusted model of the WORC score at 12 months postrandomisation. The model is consistent with the OSS models, with the effect of the InSpace device reducing shoulder function at the 12-month follow-up point.

EQ-5D-5L

The EQ-5D-5L is a validated quality-of-life utility score, calibrated such that 1 represents perfect health and 0 represents death. Descriptive statistics of the EQ-5D-5L by allocation group and time point are shown in Table 45 and Figure 32. The adjusted model results for the EQ-5D-5L can be found in Table 46.

FIGURE 32.

Box plot of EQ-5D-5L score by allocation group and time point.

Patient reported change

In addition to answering the patient-reported outcome measures, participants were also asked to report about their general shoulder function as compared to before their operation (PGIC). Participants were asked to report their assessment of their change in shoulder function and the more specific assessment of their ‘activity limitations, symptoms, emotions and overall quality of life’ since their operation. Table 47 shows the summary of responses of the change in shoulder function. Table 48 shows the response to the more specific prompt.

Relationship between the OSS and the Constant score

The original primary outcome was the Constant score (see main text). To confirm that the OSS was a suitable alternative, the correlations between the Constant score and the OSS were calculated.

The OSS is scored between 0 and 48, hence we rescaled scores to a 0–100 range to allow direct comparison to the Constant score (range is 0–100). These results suggest that there is a moderate to strong correlation between the two outcomes at each time point (see Table 49).

Participants with repairable tears

This analysis was added (prospectively, before recruitment closed) to investigate if there were any observable differences between participants who were randomised into the study and those who were excluded because their cuff tears were repairable.

Table 50 shows the participants baseline information for the randomised group and the excluded intraoperative group due to having a repairable tear.

A t-test was performed between the randomised group and the participants with repairable rotator cuff tears that were excluded intraoperatively. The result showed that the repairable tear group had higher Constant score by 3.8 points, and a higher score for the OSS by 2.1 points. Neither test results showed statistical difference.

Tables 51 and 52 show the linear regression model results for the OSS and the Constant score, respectively.