Online remote behavioural intervention for tics in 9- to 17-year-olds: the ORBIT RCT with embedded process and economic evaluation

Clinical data and background

The effectiveness of BT for reducing tics is now well established,3 with systematic reviews demonstrating a similar magnitude of effect for HRT/CBIT as for pharmaceutical interventions. 3,11,12 With numerous larger-scale RCTs having been conducted to date, CBIT in particular is supported by a strong evidence base regarding its efficacy and safety. 11 ERP is also endorsed, though ‘to a lesser degree of certainty’11 than HRT/CBIT, owing to its more limited evidence base to date. Systematic reviews of the literature also highlight that psychoeducation, whilst shown to be inferior to BT for tics as a standalone treatment in numerous RCTs, should always be offered as an initial component in BT, regardless of subsequent therapeutic approach. 11

Whilst BT models differ in their therapeutic processes, they share similar rationales, theoretical underpinnings and goals. There are various theories highlighting different mechanisms that may be involved in BT for tics. For instance, one theory suggests that HRT/CBIT and ERP work on an underlying principle that motor and vocal tics are linked to a ‘premonitory urge’ (a somatosensory discomfort that occurs before a tic). Tics are then reinforced over time through their association with the premonitory urge, creating an urge–tic cycle. A core aim of BTs for tics is to disrupt the urge–tic reinforcement cycle. This theory also posits that other internal stressors (e.g. emotional distress) and external or situational factors (e.g. environmental stressors, such as noise or social context) may maintain or worsen tics, and so the therapist will also work with patients to address these factors. However, the exact mechanisms involved in BT for tics remain unclear.

Detailed session-by-session guidance on the delivery of HRT/CBIT and ERP can be found in published treatment manuals, but we provide a brief overview of these approaches below. To date there has been no specific research into which BT is preferable for whom or when either HRT/CBIT or ERP in particular may be indicated. From clinical experience alone, Verdellen et al. 16 posit that patients with a large number of tics may obtain greater benefit from ERP, as the model addresses multiple (all) tics simultaneously. There is also some clinical or theoretical rationale to applying ERP where there is comorbid OCD, as ERP is the primary evidence-based therapy for OCD symptoms. However, more studies are needed to clarify which BT works best for whom and when. In clinical practice, clinicians often report combining approaches.

Habit reversal training

In HRT, the core aim of therapy is to break the urge–tic–relief cycle by developing alternative or ‘competing’ responses to the premonitory urge. The process of HRT comprises two main components: (1) awareness training, which involves strategies and techniques to increase awareness of both premonitory urges and tics themselves and (2) competing response training, where physically incompatible actions are identified and performed to disrupt/block tic expression. Competing response training only commences for each tic once the individual has developed good awareness of the tic occurring and the ‘tic signal’ preceding it. This process is followed for each tic individually, such that tics are treated one by one in a hierarchy, usually starting with the most bothersome. Working sequentially through each tic in the hierarchy, treatment involves competing response practice and mastery in-session, followed by continued practice at home.

Comprehensive behavioural intervention for tics

Comprehensive behavioural intervention for tics, which is supported by the largest trials to date,17 is simply an extended package of HRT with additional therapeutic components. These include relaxation training, contingency management and functional analyses to identify and address contextual factors that may exacerbate tics, as well as working with families/schools to promote social support. Though there is some uncertainty as to the ‘active’ components of CBIT, several RCTs have consistently demonstrated the superiority of CBIT to psychoeducation-based treatment in young people and adults, with reductions in tic severity maintained at up to 6 months. 3,11 An 11-year naturalistic follow-up of the original CBIT trial17 showed reduced tic severity was maintained in those who had received CBIT. 18 A recent pilot trial also provided preliminary evidence for the efficacy of a modified form of CBIT with play-based adaptations and significant parent involvement for young children with tics. 19

Exposure and response prevention

Exposure and response prevention also aims to break the urge–tic–relief cycle of reinforcement, but instead of developing a competing response to individual tics, the patient learns to tolerate premonitory urges and suppress tic expression altogether. As such, all tics are addressed simultaneously. During therapy sessions, the patient is supported in practicing suppressing tics for prolonged periods (i.e. ‘response prevention’), and strategies are then used to increase ‘exposure’ to the premonitory urge and tic-inducing environmental factors. This typically includes practice focusing on the urge and gradually increasing exposure to situations and activities that typically elicit tics, whilst at all times resisting the urge to tic.

Randomised controlled trial evidence for ERP is more limited, though the available studies suggest that it may be as effective as HRT in reducing tics. One study directly compared HRT with ERP for children with tics and found no statistically significant difference in the reduction of symptoms in terms of tic frequency and a slightly favourable response to ERP on the Yale Global Tic Severity Scale (YGTSS). 20 Another study involving both children and adults (n = 43; 7–55 years of age) randomised to either ERP or HRT also demonstrated comparable effects maintained up to the 3-month follow-up. 16 Other naturalistic studies have provided supplementary evidence that ERP can be implemented in clinical settings, with comparable effect sizes to those seen in trials to date. 11 Though larger-scale trials of ERP-based interventions are needed, these findings highlight the potential for BTs as effective and safe first-line treatments for tics.

Access to BTs for tics

Despite an increasingly clear evidence base and guidelines consistently recommending BT as a first-line treatment approach, access to BTs remains limited. Estimates suggest that only around one in five young people with TS are currently able to access BT for tics in the UK,21 contrasting with approximately 50% receiving medication, despite their more significant risks of adverse effects. 12,13 Furthermore, those young people who manage to access BT typically receive four or fewer face-to-face therapy sessions, which is under half the recommended number. 21

Research also suggests that families prefer and request better access to BT for tics and are often unsatisfied with current treatment options. Qualitative analysis collated from interviews with 42 young people with TS and a survey of 295 parents of children with TS identified that many families felt health-care professionals were not knowledgeable about TS. 21 Specifically, respondents noted the struggle to access limited BT resources, with 76% of parents saying they would like BT to be available for their child, highlighting the need for improved access to behavioural interventions for TS.

Though various factors are likely at play, the ongoing lack of expert therapists trained to deliver behavioural interventions for tics is a considerable barrier to provision. At present, Tourettes Action (https://www.tourettes-action.org.uk) lists fewer than 10 endorsed NHS behavioural therapists for young people with TS throughout the UK. In England, this equates to approximately one therapist to every 10,000 CYP with TS. This lack of provision is compounded by an uneven geographical distribution of therapists, with the majority located in London and surrounding areas. As a result, many families face long-distance travel to national specialist centres for support, which is expensive, disruptive and time-consuming, creating further inequity of access. There is therefore a desperate need for solutions to improve access to specialist treatment for tics, including scaling up provision of BTs.

Therapist-guided iCBT

Research has demonstrated that an important factor in the efficacy and cost-effectiveness of iCBT, mediated by engagement/adherence with therapy, is the provision of therapist guidance. Though self-guided programmes may seem superficially attractive due to their very low implementation costs, data indicate that low adherence is a major drawback. 29 Overall, research has shown that therapist-supported platforms perform better in terms of engagement and adherence; moreover, they deliver higher effect sizes and are more cost-effective than pure self-help. 25,30 Supporting a low-intensity model of practitioner involvement, even a ‘minimal’ amount of therapist support can be of significant benefit. 31 Importantly, service users themselves also report a preference for online interventions that integrate some traditional face-to-face or telephone support. 23

One multi-diagnostic, therapist-supported platform of note is the ‘BIP’ [Barninternetprojektet (Child Internet Project; Swedish digital platform)] iCBT programme developed by researchers at the Karolinska Institutet in Sweden. Delivered via a secure, password-protected internet platform that enables the presentation of different treatment content to different paediatric populations, the BIP research platform has been used to deliver iCBT for a range of conditions, including phobias,32 anxiety33 and OCD. 34 Similar to models adopted by improving access to psychological therapies (IAPT) in the UK, where graduate mental health workers support adults through manualised, evidence-based iCBT treatment for mild to moderate depression, anxiety and obsessive–compulsive symptoms, the BIP-iCBT treatment content is presented in chapters, like a self-help book, but with interactive materials and videos.

There are some clinical data to support the use of the BIP system for therapist-guided iCBT. A RCT using the BIP system compared participants who received BIP OCD therapy with a waiting-list control and found a significant reduction in OCD symptoms at 3 months post-treatment. 34 Additionally, there were no adverse events (AEs) reported, and participants were generally satisfied with the delivery of treatment, with only 4% stating they would have preferred face-to-face therapy. Qualitative interviews with participants in this trial also demonstrated support for the online delivery of the therapy. Specifically, they noted that iCBT allowed them to control the pace and intensity of the therapy and facilitated self-disclosure, whilst still allowing them to feel supported by a clinician. 35 Symptom reduction was also noted in a RCT using the BIP system for anxiety,36 social anxiety disorder37 and OCD,38 and in a pilot study using BIP for specific phobia,32 demonstrating the potential diversity of this platform. Whilst the evidence base for the BIP system has primarily derived from Swedish studies, recent research has demonstrated its generalisability to youth populations in the UK and Australia. 39

Study aims and design

The aim of this study was to evaluate the clinical and cost-effectiveness of a therapist-guided, parent-assisted ERP BT intervention for tics in young people with TS/CTDs. The interventions were delivered remotely via the BIP technical platform. Building on previous evidence from a Swedish pilot trial,43 the study compared an online ERP-based behavioural intervention and online tic-related psychoeducation. Our primary hypothesis was that remotely delivered, therapist-supported ERP-based BT would be superior to an active comparator intervention of online tic-related psychoeducation in reducing tic severity.

The study design was a single-blind, parallel-group, randomised controlled superiority trial, with an internal pilot and strict ‘stop–go’ progression criteria. The primary clinical outcome (tic severity) was measured via blind-assessed, clinician-rated YGTSS-TTSS. 4 A range of secondary clinician-, parent- and child-completed outcome measures were also implemented, addressing tic-related impairment, behavioural and emotional difficulties and global improvement. Measures of QoL, service use and treatment credibility and satisfaction were also obtained. Details of all primary and secondary measures, including their psychometric properties, can be found in the ‘Trial methods’ section.

The overarching aim of this study was to address the BT treatment gap for young people with tic disorders in a cost-effective manner that can be feasibly scaled up to provide widespread and equitable access to evidence-based treatment for tics across the NHS. In particular, the study aimed to add to the currently limited evidence base relating to online BT for tics by implementing an adequately powered RCT of an ERP-based, therapist-supported online intervention compared with an appropriate (psychoeducation-based) active control intervention. As the therapist role in guided iCBT is to encourage uptake and adherence to the programme, not to deliver highly specialised therapy, the skill set required is easily acquired, as demonstrated by the successful low-intensity IAPT programme, which uses graduate mental health workers to facilitate use of self-help materials by patients. Hence, if the acceptability and efficacy of the proposed therapist-guided behavioural intervention for tics is demonstrated in this trial, it should be feasible to roll it out and adopt it at scale in the NHS, IAPT BT for CYP with tics.

Trial Management Group

The full Trial Management Group (TMG) consisted of all the co-investigators listed on the protocol and a representative from the patient and public involvement (PPI) group when possible. The TMG met at least every 6 months to discuss study progress and overall conduct of the trial.

Trial Steering Committee

The role of the TSC was to provide overall supervision of the trial on behalf of the Trial Sponsor and Trial Funder and to ensure that the trial was conducted to the rigorous standards set out in the Medical Research Council’s (MRC) Guidelines for GCP. The TSC met every year, with an additional meeting to review the internal pilot.

Data Monitoring Committee

The DMC assessed whether there were any ethical or safety reasons why the trial should not continue. The DMC met annually and reviewed the internal pilot.

Inclusion criteria

1. Aged 9–17 years: patient confirmed through screening.

2. Suspected or confirmed TS/chronic tic disorder:

   1. - Including moderate/severe tics: score > 15 on the YGTSS-TTSS; TTSS score > 10 if motor or vocal tics only: researcher confirms at screening appointment.

3. Competent to provide written informed consent (parental consent for child aged < 16 years): researcher confirms at screening appointment.

4. Broadband internet access and regular PC/laptop/Mac user, with mobile phone SMS: patient confirmed through screening.

Exclusion criteria

1. Previous structured behavioural intervention for tics (e.g. HRT/CBIT or ERP) within last 12 months: patient confirmed through screening.

2. Change to medication for tics (start or stop of tic medication) within the previous 2 months: patient confirmed through screening, and subsequent medication/interventions commenced throughout the trial recorded at each time point for analysis.

3. Diagnoses of alcohol/substance dependence, psychosis, suicidality or anorexia nervosa: confirmed through parent development and well-being assessment (DAWBA).

4. Moderate/severe intellectual disability: confirmed through qualitative judgement of the assessor at the telephone screen [and confirmed at baseline through child and adolescent intellectual disability screening questionnaire (CAIDS-Q)] through questions relating to type of school the child attends and previous diagnoses.

5. Immediate risk to self or others: confirmed through screening questions and DAWBA.

6. Parent or child not able to speak or read/write English: patient confirmed through screening by the assessor.

Participant identification

There were three streams to participant identification. Patients could be identified via previous or current referrals at (1) one of the two study sites or (2) the patient identification centres (PICs). For both instances, a member of the usual care team identified potential participants from the patient records or current referrals held at the two study sites. Patients were provided with a brief information sheet and ‘consent to contact’ (C2C) form, which was passed to the research team once completed.

The third stream was through public recruitment campaigns. Specifically, the study advertised for participant recruitment via the Tourettes Action website (a national charity for people with tics) and a study website. A brief information sheet and a C2C form were hosted online.

Screening and baseline appointment

Once C2C had been established, patients were contacted by a member of the research team to go through the telephone screening questionnaire. The screening questionnaire was developed by the research team to understand the patient’s eligibility for the trial and took approximately 20–30 minutes to complete.

Patients who met the eligibility requirements were invited to attend a screening/baseline appointment. The appointment was held at one of the two study sites. All patients were reimbursed for their travel costs to attend this appointment. Before participants attended the screening/baseline appointment parents were asked to complete an online DAWBA. 62 Further details on the DAWBA are described under the Measures section.

At the screening/baseline appointment the researcher took informed consent and went through the eligibility criteria. As part of the eligibility check, the YGTSS assessment and the CAIDS-Q63 were conducted. Further details on measures are described under the Measures section.

After completion of baseline measures, patients were randomised into the study by the researcher via the web-based system hosted on a secure server by Sealed Envelope. At this appointment participants were introduced to the ORBIT therapy platform and provided with login details. Participants set their own start date for therapy but were encouraged to make it with 24–48 hours of their baseline appointment.

Primary outcome measure

The primary outcome was the severity of tics as measured by the TTSS (0–50) on the YGTSS. 4 The primary end point was 3 months post-randomisation. The primary outcome (YGTSS-TTSS) was measured at baseline (pre-intervention; face-to-face), at 3 months (primary end point) and at 6, 12 and 18 months post-randomisation (online via videoconferencing or telephone where this was not possible).

The YGTSS was administered by a blinded assessor as an investigator-based semistructured interview focusing on motor and vocal tic frequency, severity and tic-related impairment over the previous week.

In this study, four index YGTSS scores were obtained: Total Motor Tic Score, Total Phonic Tic Score, TTSS (primary outcome) and Overall Impairment Rating (secondary outcome). The Total Motor Tic Score is derived by adding the five items pertaining to motor tics (range 0–25); the Total Phonic Tic Score is derived by adding the five items pertaining to phonic tics (range 0–25); the TTSS (range 0–50) is derived by adding the Total Motor Tic Score and the Total Phonic Tic Score.

The YGTSS takes between 15 and 35 minutes to administer.

All outcome assessors underwent training alongside 6-month rater-agreement checks in the YGTSS (see Appendix 1, Table 31).

Secondary outcome measures

Screening measures

Two measures were used as screening measures.

Phase 1 analysis

This analysis has been described and published; sections have been reproduced from the Hollis et al. 78 publication under the CC-BY 4.0 Licence.

The primary outcome was estimated using a linear regression model with YGTSS-TTSS at 3 months as the outcome and study group as the main explanatory variable, adjusting for YGTSS-TTSS at baseline and site (Nottingham/London).

Linear regression models were also fitted to estimate the effect of the intervention on secondary outcomes at mid-treatment and at 3 and 6 months of follow-up (post-randomisation). The statistical model for the CGI-I did not adjust for baseline as this is a measure of change. Using CGI-I to indicate response to treatment, the scale was dichotomised to define response as ‘improved’ or ‘much improved’ versus non-response as ‘minimally improved’, ‘stayed the same’, ‘worse’ or ‘very much worse’. Two unplanned subgroup analyses explored whether the effect of the intervention on the primary outcome was modified by either anxiety diagnosis or ADHD diagnosis. The statistical models were the same as for the main analysis of the primary outcome, with the addition of a fixed effect of the comorbidity (anxiety or ADHD) and an interaction between the comorbidity and the study arm. All statistical analyses were conducted on complete cases. This analysis has been described in Hollis et al. 78

Phase 2 analysis

For the phase 2 follow-up, a sample power calculation was not specified. Outcomes at 12 and 18 months were summarised by randomised group, for continuous data using mean (SD) or for categorical data using count (percentage). A single linear mixed model was fitted for each outcome, with measures from all available time points (at mid-treatment and at 3, 6, 12 and 18 months of follow-up) as the repeated-measures outcome and a random effect of participant to account for correlations between the repeated measures on each individual at different time points. The main explanatory variables were treatment, time and the treatment by time interaction, adjusting for site and the baseline measure of the outcome. Since correlations are commonly smaller over longer time periods, we adjusted for baseline through an interaction with time, which allowed correlations with baseline to differ between follow-up times. The effect of the intervention at 12 and 18 months was estimated from this model.

As with the phase 1 analysis, the statistical model for CGI-I did not adjust for baseline as this is a measure of change. Response to treatment was compared between study arms using separate logistic regression models at 12 and 18 months, adjusting for site. Estimated effects are reported with 95% CIs.

We summarised changes in other (non-trial) tic treatments (medication or therapy) between 6 and 12 months and between 6 and 18 months by study arm and for the sample overall using counts (N) and percentages (%). The phase 2 analysis is currently under peer review with the Journal of Child Psychology and Psychiatry. 79

Losses to follow-up

The sample size calculation allowed for 20% missing data; however, data from the primary outcome measure (YGTSS-TTSS) at the primary end point (3 months) were collected from 99/112 participants (88.4%) in the intervention group and 105/112 participants (93.7%) in the psychoeducation group. Thus, retention to follow-up was better than anticipated. Data from the primary measure at 6 months were obtained from 93/112 participants (84.5%) for each group. The only predictor of missingness was site, which was included as a covariate in the statistical models.

Primary outcome

The mean scores for the primary outcome – the YGTSS-TTSS at 3 months – were lower in the ERP group (23.9, SD 8.2) than in the psychoeducation group (26.8, SD 7.3). When comparing to baseline scores, the mean total decrease in the YGTSS-TTSS was greater in the ERP group (4.5; 16%) than the psychoeducation group (1.6; 6%). Table 5 shows the results of the adjusted (for baseline and site) difference in mean scores when comparing the ERP group with the psychoeducation group. The results showed that the ERP intervention reduced the YGTSS-TTSS by −2.29 points (95% CI −3.86 to −0.71) in comparison to psychoeducation, with an effect size of −0.31 (95% CI −0.52 to −0.10).

At 6 months, this adjusted effect on tics (YGTSS-TTSS) was slightly increased (estimated difference −2.64, 95% CI −4.56 to −0.73), with an effect size of −0.36 (95% CI −0.62 to −0.10).

Effect sizes for the primary and secondary outcomes are presented in Figure 4.

FIGURE 4.

Effect sizes for primary and secondary outcomes up to the 6-month follow-up. Note: Figure published in Hollis et al. 78

Secondary outcomes

Figure 4 shows the forest plot of effect sizes for secondary outcomes.

Other secondary measures

There was no statistically significant difference in young person-reported anxiety (SCAS) at 3 months; however, at 6 months there was a greater reduction in SCAS scores in the ERP group than in the psychoeducation group (−5.10, 95% CI −9.70 to −0.50). Conversely, the young person-reported tic-specific QoL (CandA-GTS-QoL) and the outcome assessor-completed perception of global improvement (CGI-I) showed greater improvement in the ERP group than in the psychoeducation group at 3 months (CGI-I −0.41, 95% CI −0.71 to −0.11; CandA GTS-QoL −4.81, 95% CI −8.79 to −0.83) but no statistically significant difference at 6 months.

FIGURE 5.

Forest plot of standardised effect sizes for primary and secondary outcomes at 12- and 18-month follow-ups. Note: Figure in Hollis et al. 79

There was no statistically significant difference at either 3 or 6 months in the other secondary outcomes of parent-reported general emotional and behavioural functioning (SDQ), young person-reported low mood (MFQ) and outcome assessor-reported overall functioning (CGAS).

Treatment adherence

The minimum treatment was specified as completion of the first four chapters for both groups. This minimum treatment was selected as the first four chapters contained the main principles of the intervention, with the subsequent chapters used to reinforce the theory and strategies. This was achieved for 88% (99/112) in the ERP group and 94% (105/112) in the psychoeducation group, indicating high engagement rate for both treatments. The results of engagement are shown in Table 9.

The numbers of times participants logged in to the intervention were similar across both groups, although there was a slightly higher number of logins for the participants in the ERP group.

There was slightly more therapist time required to support the ERP intervention compared to the psychoeducation intervention by approximately 15 minutes. However, it is noteworthy that the overall therapist contact time per family (combined parent/carer and young person) was 2.5 hours, which equates to 15 minutes a week; this is much lower than that of face-to-face therapy. Parent/carer and young person perceptions of treatment suitability and credibility are explored in Chapter 6, but Table 9 demonstrates that they were high across both groups.

Adverse events

Adverse events were recorded up until 6 months (phase 1). Table 10 shows the number of AEs recorded throughout the trial. There were slightly fewer AEs in the ERP group compared to psychoeducation (359 vs. 431), and fewer participants in the ERP group experienced one or more AEs (n = 88/112; 79%) than in the psychoeducation group (n = 94/112; 84%). The most commonly occurring AEs were low mood, increased tics and anger/irritability.

There were two SAEs throughout the trial, which affected two participants who were both in the psychoeducation group. The SAEs were judged as unrelated to the trial by the TSC and DMC. One of the SAEs was a male participant collapsing and being admitted to hospital as a result of a functional movement disorder. The other SAE involved a female participant attending accident and emergency as a result of a ‘tic attack’. The independent clinicians on the TSC and DMC agreed with the chief investigator that waxing and waning of tics is part of the natural course of the disorder and should not be considered related to the intervention. Both participants were discharged from hospital with no further action. The parents informed the trial team that they had not recently been engaging with ORBIT at the time of the event and they did not consider the events to be related to their participation.

Missing data

There was no primary outcome for the phase 2 results; however, the YGTSS-TTSS (the primary outcome for phase 1) was a key outcome of interest. Completion of this YGTSS-TTSS was good throughout the trial. At 12 months, data were collected from 91/112 participants (81%) in both the ERP group and the psychoeducation group, and at 18 months data were collected from 89/112 participants (80%) from the ERP group and from 90/112 participants (80%) from the psychoeducation group. The only predictor of missingness was site, which was included as a covariate in the statistical models. Other measures that were completed by the outcome assessor, including the CGI-I and the CGAS, also sustained good completion rates (see Table 11).

However, data from measures that the parent/carer or young person completed online into the database, which were not directly collected or supported by the outcome assessors (e.g. PTQ, MFQ, SCAS, SDQ, CandA-GTS-QoL), were less well completed, with a large amount of missing data (ranging from 41% to 61% missing). Thus, the results from these measures should be interpreted with caution. In all cases, the quantities of missing data were similar for each randomised group.

Key outcome of interest

At 12 months, the mean YGTSS-TTSS in the ERP group was 21.7 (SD 8.8) compared to 24.9 (SD 7.3) in the psychoeducation group. The analysis, which adjusted for tic severity at baseline and site, demonstrated that the ERP intervention reduced TTSS by −2.64 points (95% CI −4.48 to −0.79), with an effect size of −0.36 (95% CI −0.61 to −0.11), compared to the psychoeducation group.

At 18 months, the mean YGTSS-TTSS in the ERP group was 21.5 (SD 9.0) and 23.9 (SD 8.4) in the psychoeducation group. The analysis indicated that the ERP intervention reduced the TTSS by −2.01 points (95% CI −3.86 to −0.15), with an effect size of −0.27 (95% CI −0.52 to −0.02), in comparison to the psychoeducation group (see Table 12).

The standardised effect sizes for all outcomes in phase 2 are presented in Figure 5.

Other measures

The parent-reported measure of tics (PTQ) also showed that the ERP group had less tic severity than the psychoeducation group at 12 months (−9.89, 95% CI −16.01 to −3.77), but this was not found at 18 months (see Figure 5 and Table 12). There was no statistically significant difference between the two groups on the YGTSS impairment scale at either time point.

The young person-reported MFQ (mood) and SCAS (anxiety) showed larger reductions in symptoms at 12 months (MFQ −2.93, 95% CI −5.77 to −0.09; SCAS −6.11, 95% CI −10.41 to −1.81) and 18 months (MFQ −4.87, 95% CI −8.00 to −1.75; SCAS −9.41, 95% CI −14.11 to −4.70) in the ERP group compared to the psychoeducation group.

At both time points, young person-reported tic-specific QoL showed greater improvement in the ERP group compared to the psychoeducation group (12 months −5.79, 95% CI −10.28 to −1.30; 18 months −9.00, 95% CI −13.98 to −4.01).

The same was also true for the outcome assessor-completed CGAS (12 months 2.85, 95% CI 0.15 to −5.56; 18 months 3.18, 95% CI 0.47 to −5.90) and CGI-I (12 months −0.43, 95% CI −0.75 to −0.10; 18 months 0.38, 95% CI −0.71 to −0.05). However, there was no statistically significant difference in parent-reported general emotional and behavioural functioning (SDQ) at either time point (see Table 12).

Unplanned analysis

After 6 months (phase 2), participants were allowed to start other tic treatments (behavioural or pharmacological) outside the trial interventions. However, between 6 and 12 months only 11 (6%) started a new tic medication and only 6 (3%) started a new tic therapy. Between 6 and 18 months only 7 (4%) started a new tic medication and only 2 (1%) started a new tic therapy (see Table 13).

Most participants were not using tic medication (90% at 12 months, 92% at 18 months) or other tic therapy (94% at 12 months, 96% at 18 months) during phase 2. The number of participants who changed their medication or therapy status was too small to conduct a sensitivity analysis on the YGTSS-TTSS.

We conducted the same unplanned post hoc analysis conducted in phase 1 that compared treatment response on the CGI-I. Comparable to phase 1, the analysis revealed greater treatment responses with ERP at 12 months (2.27, 95% CI 1.23 to 4.22) and 18 months (1.80, 95% CI 0.99 to 3.27). Furthermore, we demonstrated that participants who responded at 6 months were more likely to continue to respond at later follow-up and less likely to relapse if they were in the ERP group than the psychoeducation group (see Table 14).

Cost of the BIP platform

The cost of the BIP platform is split into a fixed cost per participant and a variable cost based on platform use.

Cost of delivering therapist support

The cost of therapist support is also broken down into a fixed and variable cost.

Perspective, discounting and time horizon

In line with NICE Technology Assessment guidelines,80 both the primary analysis alongside the RCT and secondary decision model analysis took an NHS and personal social services cost perspective. A secondary analysis from a wider societal cost perspective was also conducted for the analysis alongside the RCT. All costs and QALYs after 12 months were discounted at a rate of 3.5% per annum. 80 The analysis alongside the RCT reports costs and QALYs at 6 and 18 months to take into account the naturalistic observation that occurs after 6 months. The decision model has a time horizon of 10 years.

Outcome measures

Both the analysis alongside the RCT and the decision models estimate costs and QALYs.

The primary outcome measure was the incremental cost-effectiveness ratio (ICER), calculated as the mean incremental cost per QALY gained calculated from the parent-completed CHU9D at 18 months.

The analysis alongside the RCT also reports the incremental cost per change in YGTSS-TTSS.

Costs and resource use

The costs that each young person incurred included the cost of the intervention, including the cost of the BIP platform for each participant randomised to ERP therapy, and the cost of delivering therapist support for both ERP therapy and psychoeducation groups (see above) and resource use collected during the trial using an adapted version of the CA-SUS. 82–84

The primary analysis was from a health and social care cost perspective. Resource use was collected on specialist tic services, contacts with professionals in the community, inpatient contacts, emergency contacts and medication use at baseline and at 3, 6, 12 and 18 months asking about resource use since last follow-up or in the previous 3 months at baseline using the CA-SUS. Unit costs used in the health and social care cost perspective analysis are reported in Table 15 and are in 2019 GBP.

Wider societal costs include out-of-pocket costs, the cost of education support, voluntary services and the cost of days off from school, also collected using the CA-SUS. When reporting days off from school we report total days off as well as the number of days off that participants attributed as being related to TS.

Out-of-pocket costs are calculated using the amount reported by parents in the CA-SUS. The costs of days off from school are calculated using a human capital approach based on the cost of child care for each day a participant has reported taking off school. Unit costs used in the wider societal cost perspective analysis are reported in Table 16 and are in 2019 GBP.

Utilities

Health-related QoL utility tariffs are calculated from responses to the CHU9D,85 a child-specific patient-reported outcome measure validated for use in the calculation of QALYs and applying the algorithm developed by Stevens. 86 NICE does not recommend specific measures of HRQoL in CYP but that the choice of measure should be informed by evidence of psychometric performance, evidence that it is valid in the age ranges being studied and the quality and availability of value sets. The CHU9D has been validated for use in child and adolescent mental health services, is designed for those aged 7–17 years in line with our inclusion criteria and has a UK value set.

The CHU9D was completed as a patient-reported outcome by the young person and a proxy reported outcome asking about the young person completed by a parent/carer at baseline and at 3, 6, 12 and 18 months. The parent-completed version of the measure was pre-specified as the primary analysis in the health economics analysis plan (HEAP) based on the assumption that we would have better completion rates for the parent-completed rather than young person-completed version.

Cost–utility analysis

Analyses were pre-specified in the HEAP.

We calculated complete case descriptive statistics for the percentage of participants that used each type of resource and the mean number of contacts for the participants that used them. The mean difference in costs and 95% CIs for each resource use type was calculated using regression analysis adjusting for baseline costs, with site as a covariate and bias-corrected bootstrapping with 1000 iterations for complete cases (complete resource use at baseline and at 3-, 6-, 12- and 18-month follow-ups).

The mean difference in QALYs at 6 and 18 months and 95% CIs were calculated using regression analysis adjusting for baseline utility, with site as a covariate and bias-corrected bootstrapping with 1000 iterations for complete cases (complete CHU9D at baseline and at 3-, 6-, 12- and 18-month follow-ups).

We assumed data were missing at random. Predictors of missingness were explored, with site identified as the only predictor of missingness. In line with the statistical analysis of the primary outcome we have conducted a complete case analysis adjusting for predictors of missingness.

Incremental cost-effectiveness ratio

We explore the mean incremental cost per QALY gained and the mean cost per point reduction in YGTSS-TTSS comparing ERP and psychoeducation from the health and social care cost perspective at 18 months. Costs include total health-care resource use per participant and variable costs per participant for both trial arms. For the intervention arm, costs include the costs for the platform, therapist time, supervision and training. For the control arm, we do not include any platform or training and supervision costs, as we assume that the psychoeducation information could be provided without the need to use the platform (i.e. on a standalone website). However, therapist time is included given that this was provided as part of the active psychoeducation arm. The ICERs were calculated using seemingly unrelated regression (SUR; Stata command SUREG) to account for any potential correlation between costs and outcomes.

Cost-effectiveness acceptability planes and cost-effectiveness acceptability curves

The adjusted, bootstrapped SUR QALYs, TTSS and cost data were used to calculate the probability that ERP is cost-effective compared to psychoeducation for a range of cost-effectiveness threshold values. A cost-effectiveness plane of the bootstrapped results is also reported for both the YGTSS-TTSS and QALY analyses.

Sensitivity analyses

Sensitivity analyses were conducted to explore the impact of different assumptions regarding the cost of the intervention. Three assumptions were changed:

• Applying a licence fee per participant instead of a fixed cost per year based on a UK iCBT digital intervention. 87 We apply different licence fees starting at £25 to explore how this impacts the probability of cost-effectiveness.

• Assuming fortnightly supervision instead of weekly supervision of therapists delivering the intervention.

• Adding the cost of the platform to the psychoeducation group to reflect how this might be rolled out in practice.

Design

In the model, cost-effectiveness is estimated by calculating which of three options – (1) online psychoeducation, (2) online ERP and (3) CBIT – has the highest net monetary benefit (NMB) for a range of QALY cost-effectiveness thresholds. The model was designed following a rapid review of the literature and informed by expert clinical input.

Cycle duration is 6 months, with 20 cycles in total to project the results forward 10 years to estimate how costs and outcomes might change for this patient group going into adulthood. The duration of 10 years was chosen to adequately capture long-term costs and QALYs, but such an analysis does not project so far into the future that the results might no longer hold. In line with NICE guidelines, costs and QALYs have been discounted at the recommended rate of 3.5% per year. 80 There are associated costs and utility values for each of three health states of mild, moderate and severe tics.

Transition probabilities for ERP and psychoeducation were obtained during the trial follow-up between 6 and 18 months to measure the natural progression of tics following the intervention, as using data from baseline to 6 months would have overestimated the long-term impact by including the initial decrease in tic severity in the transition probabilities. Transition probabilities for CBIT were calculated from the literature (Markov model structure; Table 17). The mean utilities and costs associated with each health state were calculated using the patient-level data for the study.

The patient population in the model is assumed to be the same as that in the trial, including in terms of sample size and demographics.

The model was designed in Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA), with inputs calculated in Stata version 17. 81

Markov model structure

Following our rapid review of the literature, searching for tics/Tourette’s, young people/adolescents and economic evaluations, we were unable to identify any previous decision models for the treatment of tics in young people. Our review also identified that there is no official definition for what categorises mild to severe tics. We decided to base the model on the commonly referenced structure of severity suggested by Bloch and Leckman,88 where tics are categorised based on the TTSS on the YGTSS as very mild (0–9), mild (10–19), moderate (20–29), severe (30–39) or very severe (40–50). It is possible for individuals to go from any severity level to another each cycle within a relatively short period of time, although the probability of going from a very mild state to very severe and vice versa is close to zero (see Figure 6).

FIGURE 6.

Markov model structure.

Figure 7 shows tic progression over the course of the trial for online psychoeducation and ERP. Discussions with clinicians led us to conclude that it was feasible to simulate a decrease in tic severity in the CBIT group at 6 months from the trial literature data, shown by the dashed line in Figure 7, and then assume a similar tapering to ERP, shown in Figure 7 after 6 months. For this reason, the transition probabilities applied to the CBIT arm after 6 months are the same as those drawn from the trial for the ERP arm.

FIGURE 7.

Tic progression over time by group.

Figures 8 and 9 show the proportions of participants in each group that fell into each health state at baseline and at 6 months, respectively. These results illustrate patients moving into lower-severity health state and tic scores falling over time. Table 18 reports the proportions of participants in each group when they enter the model and begin moving through the health states using the transition probabilities reported in Table 17. For the probabilistic sensitivity analysis (PSA), transition probabilities were sampled using the Dirichlet distribution. 89 Values for CBIT were obtained by combining data from the available literature on CBIT effectiveness for young people at 6 months. 17,59,90,91

FIGURE 8.

Proportions of participants in each tic severity health state by group at baseline.

FIGURE 9.

Proportions of participants in each tic severity health state by group at 6 months.

Calculating health-care costs and utilities

Health state utilities

The relationship between tic severity and HRQoL is not clear. Some studies have found a positive relationship between tic severity and factors relating to QoL, such as functional impairment and psychosocial scores in children. 92,93 A few studies have found that tic severity only has a small impact on HRQoL, if any, and that a greater contributor is comorbidities. Some studies have found a significant relationship between tic severity and HRQoL. 94–97 We therefore included key significant comorbidities using the DAWBA data from the trial as well as age and sex when calculating CHU9D utility values for tic health states.

Analysis of the trial data showed that transitioning from one of the mild-severity health states to a moderate-severity health state had a significant impact on costs and utility values, but there was no statistically significant difference between a very-mild-severity and a mild-severity health state. Similarly, there was a significant difference in costs and utilities between a moderate-severity health state and the severe health states, but there was no difference between severe and very severe tics. Therefore, costs and utilities are allocated as illustrated by Figure 10. Means, standard errors and distributions calculated from the trial data for use in the model are reported in Table 19. Utilities were divided by two when applied to health states to reflect the 6-month cycle duration.

FIGURE 10.

Allocation of costs and utilities according to health states.

Cost-effectiveness analysis

We calculated the total costs and QALYs for patients treated with online ERP compared with online psychoeducation and face-to-face CBIT. We calculated an individual ICER for ERP compared with psychoeducation and ERP compared with CBIT. We also calculated the NMB of each intervention to be able to more easily compare all three options. Any costs and QALYs obtained after 12 months were discounted at 3.5% in line with NICE guidance. 80

Sensitivity analysis

Deterministic analysis and PSA were used to explore the uncertainty regarding our results.

Costs of the intervention

The fixed and variable costs of the BIP platform and therapist support are reported in Tables 20–23. For variable login costs, the mean cost per patient was £1.25 higher (95% CI £0.46 to £2.04) for the intervention arm compared to the control. For variable therapist support costs, the mean cost per patient was £5.03 higher (95% CI £0.88 to £9.97) for ERP compared to psychoeducation.

Primary analysis: analysis alongside the RCT

Primary analysis: mean incremental cost per QALY gained of ERP compared to psychoeducation at 18 months based on proxy parent/carer responses to the CHU9D

From a health and social care cost perspective at 18 months, accounting for the correlation between costs and QALYs using SUR and adjusting for baseline utility and site, the mean incremental cost per participant of ERP compared to psychoeducation was £662 (95% CI −£59 to £1384), with a mean incremental QALY calculated using the parent-completed CHU9D of 0.040 (95% CI −0.004 to 0.083) per participant and an incremental cost per QALY gained of £16,708. The cost-effectiveness acceptability plane (CEAP) is shown in Figure 11. At a cost-effectiveness threshold of £20,000 per QALY there is a 65% probability that ERP is cost-effective, increasing to 79% at a threshold value of £30,000 per QALY (see Figure 12).

FIGURE 11.

Cost-effectiveness acceptability plane QALY analysis using parent-completed CHU9D. Note: Figure in Hollis et al. 79

FIGURE 12.

Cost-effectiveness acceptability curve QALY analysis using parent-completed CHU9D. Note: Figure in Hollis et al. 79

From a wider cost perspective, the mean incremental cost per participant of ERP compared to psychoeducation was £1080 (95% CI −£351 to £2510), with a mean incremental QALY calculated using the parent-completed CHU9D of 0.035 (95% CI −0.004 to 0.083) per participant and an incremental cost per QALY gained of £16,708. The incremental cost per QALY gained was £30,480, with a 53% probability of being cost-effective compared with the control at a cost-effectiveness threshold of £30,000 per QALY.

Figure 13 reports the CEAP using incremental QALYs calculated from young person self-completed CHU9D responses. The mean incremental cost per participant of ERP compared to psychoeducation was £156 (95% CI −£637 to £950), with a mean incremental QALY calculated using the young person-completed CHU9D of 0.038 (95% CI −0.015 to 0.090) per participant and an incremental cost per QALY gained of £4139. At a cost-effectiveness threshold of £20,000 the ERP intervention has an 84% probability of being cost-effective, increasing to 88% at a threshold of £30,000 per QALY gained. The CEAC for this analysis is reported in Figure 14.

FIGURE 13.

Cost-effectiveness acceptability plane QALY analysis using young person-completed CHU9D. Note: Secondary analysis: mean incremental cost per QALY gained of ERP compared to psychoeducation at 18 months is based on young person self-completed responses to the CHU9D.

FIGURE 14.

Cost-effectiveness acceptability curve QALY analysis using young person-completed CHU9D. Note: Secondary analysis: probability that ERP is cost-effective compared to psychoeducation at 18 months is based on young person self-completed responses to the CHU9D.

From a wider cost perspective, the incremental cost per QALY gained at 18 months is £10,053 (mean incremental cost of £381, 95% CI −£950 to £1713). The probability of the ERP intervention being cost-effective compared to psychoeducation from a wider cost perspective at a threshold of £20,000 per QALY gained is 69%, and it is 79% at £30,000 per QALY gained.

At 18 months, the incremental mean cost per patient of ERP compared with psychoeducation, including intervention costs, was £305 (95% CI −£136 to £746), with a mean reduction of −1.68 points on the YGTSS-TTSS (95% CI −3.79 to −0.43) and an incremental cost per reduction in YGTSS-TTSS (YGTSS-TTSS multiplied by −1) of £182 from a health and social care cost perspective. The ICER is reported in the CEAP in Figure 15. There is a 90% probability that ERP is cost-effective compared with psychoeducation for cost-effectiveness thresholds > £1000 per point reduction in YGTSS-TTSS (see Figure 16).

FIGURE 15.

Cost-effectiveness acceptability plane for TTSS analysis. Note: Figure in Hollis et al. 79

FIGURE 16.

Cost-effectiveness acceptability curve for a cost per point reduction on the TTSS. Note: Secondary analysis: mean incremental cost per unit change on the TTSS of ERP compared to psychoeducation at 18 months. Figure in Hollis et al. 79

From a wider cost perspective, the incremental cost per point reduction in YGTSS-TTSS was £346.3, with an 86% probability of being cost-effective compared with the control at a cost-effectiveness threshold of £1000 per point reduction in YGTSS-TTSS. (Please note the ICER has not been multiplied by −1 so the interpretation of the plane is reversed.)

Sensitivity analysis

In order to explore how different costing models for the platform impact the cost-effectiveness of the intervention we applied a licence fee starting at £25 per participant and increasing by £10 increments up to £165. Figure 17 shows that at a licence fee of £25 per participant ERP has an 82% chance of being cost-effective at a threshold value of £30,000 per QALY gained; this falls to 77% when the licence fee was set to £165 per participant. This small range would suggest that at a threshold value of £30,000 per QALY gained there is little uncertainty around ERP being cost-effective compared with psychoeducation, even if the cost of rolling out the intervention was marginally more expensive than expected from the trial.

FIGURE 17.

Impact of different licence fees on probability of cost-effectiveness at a threshold of £30,000 per QALY gained. Note: Figure in Hollis et al. 79

We see similar results at a threshold value of £1000 per point reduction in YGTSS-TTSS. As there is no official threshold for a point reduction in YGTSS-TTSS we also conducted this for a threshold value of £200 per point reduction in YGTSS-TTSS and found that the probability of cost-effectiveness falls below 50% at a licence fee per participant of £155 or more.

As therapists were trained during supervision, it is feasible to assume that once fully trained and familiar with the delivery of the intervention only fortnightly supervision would be necessary. When applying fortnightly supervision, the fixed cost of the intervention falls from £104 to £71. This results in an ICER of £162 per point reduction in YGTSS-TTSS and a probability of 90% of being cost-effective compared with psychoeducation at a threshold of £1000 per point reduction in YGTSS-TTSS.

Using the parent-completed CHU9D, when supervision is assumed to be fortnightly this results in an ICER of £15,885 per QALY gained, and there is an 81% probability that ERP is cost-effective compared with psychoeducation at a threshold of £30,000 per QALY gained.

Given psychoeducation was also delivered through the platform and the possibility of trying to provide this as an alternative intervention side by side with ERP but without the requirement of therapist support, an alternative sensitivity analysis added the fixed platform cost to the psychoeducation arm. This increased the probability of cost-effectiveness to 71% using parent-completed CHU9D at a threshold value of £20,000 per QALY gained and 83% at a threshold value of £30,000 per QALY gained.

Secondary analysis: 10-year decision analytic model

We report the mean total cost and QALYs accrued per participant over the course of the 10 years in the model for each intervention. After 5000 replications, online ERP cost an additional £70.76 per participant compared with psychoeducation and resulted in an additional 0.009 QALYs per participant. This resulted in an ICER of £8276 per QALY gained. CBIT cost an additional £537.38 per participant compared to online ERP and led to an additional 0.018 QALYs per participant.

Figure 18 shows the probability that each intervention is cost-effective at different cost-effectiveness threshold values. At cost-effectiveness thresholds below than £10,000 per QALY gained, psychoeducation has the highest probability of being cost-effective. ERP is cost-effective compared with CBIT at cost-effectiveness threshold values below £30,000 per QALY gained. Figure 19 shows the CEAC for ERP compared with CBIT alone; there is a > 94% chance that ERP is cost-effective compared with CBIT at threshold values of £20,000 per QALY gained or below.

FIGURE 18.

Cost-effectiveness acceptability curve showing the probability each intervention is cost-effective at a range of threshold values.

FIGURE 19.

Cost-effectiveness acceptability curve showing the probability that ERP is cost-effective compared with CBIT.

Figure 20 shows that there is a > 50% chance that online ERP is cost-effective compared with psychoeducation from threshold values of £10,000 per QALY gained or more.

FIGURE 20.

Cost-effectiveness acceptability curve showing the probability that online ERP is cost-effective compared with online psychoeducation.

Conclusion

The 18-month time horizon is long enough to be able to conduct a full economic evaluation and we found that ERP has a 79% chance of being cost-effective compared with psychoeducation at a cost-effectiveness threshold of £30,000 per QALY gained. This is likely to be conservative given that psychoeducation is already better than what is widely available as treatment as usual. Moreover, the cost of the platform is not included for this group in the trial analysis as it was assumed that this information would normally be available elsewhere. When this cost is included the probability of cost-effectiveness goes up to 83%.

Most of the sensitivity analyses conducted have little impact on the results, suggesting that we can be reasonably confident that these results accurately reflect the cost-effectiveness of ERP compared with psychoeducation at 18 months. The initial naturalistic follow-up suggests that benefits of the intervention can be maintained past the initial intervention.

There was a higher probability that the intervention was cost-effective when the young person self-completed responses to the CHU9D are used to calculate QALYs, with a significant difference in QALYs in favour of ERP at 6 months. We pre-specified in the HEAP that the proxy parent/carer-completed CHU9D would be used to calculate QALYs in the primary analysis. This was due to the assumption that we might have a larger number of young people who might not compete the CHU9D, as did indeed happen. Further work is required to determine the suitability of proxy parent/carer-reported CHU9D for tics. Our work on determining health states for the model demonstrated that there was a significant difference in CHU9D proxy utility scores between mild, moderate and severe YGTSS-TTSS, suggesting that the changes in the CHU9D do correspond to the key clinical mechanism of interest in this trial: reducing tics.

The results of the long-term decision model suggest that online ERP is more than 50% cost-effective compared with the active control of online psychoeducation at threshold values considered acceptable by NICE. When we account for the cost of the platform in the online psychoeducation arm, online ERP becomes cost-effective at threshold values of £5000 per QALY gained, suggesting that if the NICE threshold is higher than this for a QALY gained it is more cost-effective to roll out online ERP than online psychoeducation using a similar platform. Even under the assumption that the cost of the active control is £0, which will be underestimating the cost-effectiveness given that it is still accounting for the benefit of online psychoeducation without the cost, online ERP is over 50% cost-effective from threshold values of £13,000 per QALY gained.

Online ERP has a > 50% chance of being cost-effective compared with CBIT at threshold values below £30,000 per QALY gained and a > 94% probability of cost-effectiveness at threshold values of £20,000 per QALY gained or below. Given research showing that the opportunity cost is likely to be closer to £13,000 than £30,000 per QALY gained, the more cost-effective intervention appears to be online ERP. 98 Moreover, even in a situation in which we were able to pay £30,000 per QALY gained for CBIT, this would not resolve the lack of therapists available in the UK to deliver this intervention. The results of the decision model suggest that the remotely delivered ERP intervention can provide an alternative to face-to-face CBIT therapy, with only minimal impact on HRQoL.

Implementation fidelity

Implementation can refer to how an intervention can be implemented within routine clinical practice. However, this can only be achieved once an intervention has shown efficacy in an outcome evaluation. Implementation can also refer to how the delivery of an intervention was achieved within the context of a RCT and the structures and processes through which an intervention was delivered as intended. 103 This is often termed ‘implementation fidelity’. In short, implementation fidelity refers to the degree to which a study was implemented according to design or protocol. If an intervention is designed according to well-established theoretical and empirical underpinnings, including identifying ‘essential ingredients’ and their subsequent relationships with the intended outcomes, implementation fidelity is seen as crucial. 105

There are multiple benefits to a trial that rigorously assesses implementation fidelity. These include improving the validity of intervention outcomes,106,107 enabling replicability108 and aiding in the understanding of why an intervention succeeded or failed in its intended outcome (e.g. symptom reduction). 109 For example, a study may erroneously determine that the lack of impact of an intervention was caused by elements of the programme itself if no process measures were evaluated (i.e. a type III error). 110 Therefore, it is essential that a RCT that includes a process evaluation should contain a rigorous analysis of implementation fidelity.

Mechanisms of impact

The second key component involves exploring the mechanisms through which the intervention produces change. This is crucial to understanding how the effects of the intervention occurred and how these effects might be replicated in future iterations of similar interventions. 111 By exploring the mechanisms of potential impact of an intervention, a process evaluation can provide a better understanding of the causal pathways and identify any unexpected consequences. 112 Mechanisms of impact include mediators, moderators and contextual factors associated with change.

Context

The final component is context, which refers to any factors external to the intervention that may have acted as barriers or facilitators to the way it is implemented or to the outcomes. The uptake and use of DHIs are largely dependent on context; thus, understanding the context is crucial to interpreting the findings and making generalisations from the results. In conclusion, a process evaluation of a complex intervention such as ORBIT is crucial to explaining trial outcomes and will aid in understanding its overall implementation.

Aims and objectives

The aims of the ORBIT process evaluation were to understand the causes of the observed behaviour change data obtained from the RCT, to explore the fidelity of intervention delivery, acceptability of the intervention and reasons for observed variations in uptake and use and to consider the resources and implementation processes required. 102

Specific objectives were:

1. to assess the fidelity, reach and dose of intervention delivery

2. to explore whether any of the intervention features were tailored for individual needs, enabling potential recommendations for adaptations

3. to explore the intervention from the perspectives of children, parents, therapists and clinicians in order to gain a deeper understanding of potential mechanisms underlying participant behaviour change whilst probing for any unexpected consequences

4. to evaluate any factors external to ORBIT that may have affected delivery (i.e. the environment and its characteristics) or whether its mechanisms of impact worked as intended

5. to consider the resources and implementation processes required for effective implementation, uptake and use of the intervention.

In this chapter, we present a planned, mixed-methods, two-phase process evaluation of the ORBIT trial following the MRC’s 2015 guidelines103 to explore the fidelity of delivery and the contextual factors influencing engagement with the intervention (phase 1) and the impact of the intervention and mediators, contextual factors and moderators of this impact (phase 2). 101

Introduction

The MRC have developed guidelines for conducting process evaluations of complex interventions such as ORBIT. 103 In order to evaluate intervention implementation, MRC guidelines for process evaluations suggest researchers assess (1) reach (the extent to which a target audience encounters the intervention), (2) dose (how much intervention is delivered and received), (3) fidelity (the quality of what was delivered) and (4) adaptations (any modifications made to an intervention in order to achieve better contextual fit). 102 The intended target audience for ORBIT was CYP with tic disorders; however, there were pertinent questions that could be asked, such as whether there were socioeconomic biases in who was reached. In terms of dosage, the ORBIT protocol1 states that the intervention should consist of 10 individual intervention chapters following a suggested frequency and total duration of 10–12 weeks. There were four core chapters (chapters 1–4), and this was deemed the minimum requirement for treatment completion. There were six additional chapters offering information regarding reinforcement, further practise and relapse prevention. For DHIs, the fidelity of delivery of the intervention is assured by the online delivery platform. However, the intervention that is experienced by the user is highly dependent on the extent to which they engage with the intervention and use it as intended. Hence, in this process evaluation, the focus is on usage and the proportion of participants receiving the predefined ‘minimum effective dose’ of four or more chapters. 102 Finally, understanding adaptations to the intended intervention involves exploring whether these improve its contextual fit or compromise its functioning,114 or whether they represent innovation or intervention drift. 115 Participants were able to make modifications to various components of the intervention, such as the ‘tic stopwatch’, which was used to self-time the length of tic control.

Methods

Results

Discussion

This first phase of a two-stage process evaluation used a mixed-methods approach to investigate the extent to which the ORBIT intervention was implemented as planned within the context of an RCT and to explore participants’ experiences with the intervention and the contextual factors influencing children’s engagement. 102 In doing so, this made it possible to identify reasons for variation in uptake, usage and engagement, to reflect on how implementation may ultimately give greater confidence in the outcomes and to outline lessons for potential future implementation within routine care. Uptake of the intervention was high, with nearly 90% of participants receiving the predefined minimum effective dose of the first four chapters completed. The median uptake was eight chapters, and only one child failed to access any chapters. Fidelity of delivery was also excellent, with participants reporting high levels of satisfaction and acceptability. 102

The intended sample of CYP with a diagnosed tic disorder was reached, with 7.1% of families residing in the most deprived areas (IMD quintile 1) and over a quarter (25.6%) of the families residing in the least deprived areas (IMD quintile 5). As over half (53.5%) of the CYP’s mothers had completed graduate-level education against a UK average of 42%,125 it seems that more advantaged families may have been over-represented. This is a concern, as one of the aims of ORBIT was to increase access to evidence-based therapeutic interventions for CYP with tic disorders,100 particularly as access to services is generally limited for those from lower socioeconomic status backgrounds. 126 Although the initial baseline visit with associated travel may have been a disincentive to more disadvantaged families, this would not be relevant if ORBIT was delivered entirely remotely in routine care rather than as part of a RCT. Moreover, there was no evidence that socioeconomic factors influenced CYP’s engagement with ORBIT. Furthermore, a child’s age, severity of tics, well-being and comorbidities did not appear to influence their level of engagement with the intervention, providing further evidence that the intervention would have a wide reach within routine clinical care. However, due to the various factors relating to this RCT specifically as opposed to routine care, caution should be taken when interpreting the results from this study concerning reach. 102

London study site, self-referral and higher parental engagement were all associated with higher levels of child engagement. The London site is a world-renowned centre of excellence for paediatric care, which may have increased parents’ motivation for treatment. However, the only independent predictor of child engagement in the multivariate analysis was level of parental engagement with the intervention, as measured by their chapter completion and by parent time with the therapist. This is consistent with previous literature,127–129 which found that parental involvement was particularly key for younger CYP to assist with their engagement with therapeutic interventions, which in turn leads to better outcomes. 130–132 It has been shown in the literature that parental engagement may impact a provider’s ability to implement parent- and family-focused evidence-based treatment with fidelity. 130 Therefore, it is crucial to understand the role of parental support for the implementation of DHIs for children, as without attention to the key processes of child and family engagement efforts to improve the effectiveness and efficiency of the treatment are less likely to succeed. 100

An interesting finding relates to the usage and interactions with the therapist within this study. Therapists interacted online with their assigned child participants for an average of approximately 6 minutes per child per week, which is lower than the 24‐minute average time per week participants interacted with their therapists in the Swedish pilot trial of the BIP platform. 43 In the UK, study therapists were encouraged to use pre-prepared scripts to respond to participants. Their responsibilities involved reinforcing the ORBIT treatment material, with the aim of spending approximately 6 minutes per week responding to each child, which was in the therapist guidance given by supervisors. It was also apparent from qualitative interviews that many participants felt that the term ‘therapist’ was somewhat misleading. Some participants felt that ‘therapist’ had connotations of a clinically trained individual delivering an intervention. This may have limited their reliance on the therapist. Therefore, in any implementation of this intervention within routine health care, it would be sensible to alter this title to ‘coach’, ‘guide’ or ‘mentor’ to better reflect the role of the therapist. 102

Introduction

Although BT is effective and avoids the unpleasant side effects associated with medication, access to therapy is limited due to insufficient numbers of specialists and the uneven geographical distribution of services relative to demand. 21 DHIs have been shown to be effective for a range of neurodevelopmental disorders in CYP, including tic disorders, and thus offer potential to widen access to evidence-based behavioural treatments. 133 However, despite an expanding body of evidence to support the acceptability and effectiveness of online therapy, uptake of DHIs into clinical practice has been disappointing. 134 Previous research has focused on intervention outcomes, with little attention given to the mechanisms of impact: the way in which the intervention components and participants’ responses to the intervention produce change. Knowledge of how and why a DHI works increases the potential for replication across contexts. 103 Furthermore, understanding mechanisms of impact is crucial for assessing core components of an intervention (e.g. ‘essential ingredients’), which helps with defining the minimum therapeutic dose. 102

There are no studies in the tic disorder literature assessing mechanisms of impact of digital interventions; however, studies of face-to-face BT for tics have suggested that clinical factors can moderate the effectiveness of behavioural treatment. Sukhodolsky et al. 135 found that the presence of tic medication significantly moderated impact. For participants receiving 10 weeks of BT, medication status did not impact on effectiveness. In contrast, participants in the psychoeducation and supportive therapy group who were receiving medication showed significantly greater tic reductions than participants not on medication. Tic phenomenology, age, gender, family functioning, treatment expectancy and comorbidities did not moderate response to treatment. A more recent study that examined moderators of treatment outcome after adolescents with CTD received either individual or group therapy found that a higher level of anxiety and a higher premonitory urge to tic favoured treatment in groups, whereas increased sensitivity and higher depression symptomology favoured individual treatment. 55

This second phase of the process evaluation study uses qualitative and quantitative data to explore the contextual factors influencing effectiveness and the factors moderating and mediating the relationship between implementation of the intervention (level of child engagement) and the impact of the intervention on tic severity and clinical improvement. 102 This provides insight into how and why outcomes occurred under given circumstances and what mechanisms underlie these impacts. To the best of our knowledge, this is the first study to examine potential mediators and moderators of an online intervention delivered to CYP with tic disorders. 102

Methods

Results

Discussion

In this second phase of the process evaluation, we examined the impact of the intervention in terms of reduction in tic severity and global clinical improvement. Within the active intervention group, tic severity was reduced by four points from baseline to 3-month follow-up with a Cohen’s d effect size of 0.5, and 36% were rated as ‘very much improved’ or ‘much improved’ on the CGI-I. A further 37% showed some improvement. This process evaluation, which aimed to understand mechanisms of impact, found that only tic severity at recruitment was associated with reduction of tic severity post-intervention. Level of usage as captured by the child engagement factor score99 was not associated with improvement in tic severity; however, higher levels of child engagement and higher parental chapter completion were associated with higher levels of overall clinical improvement. Only parental chapter completion was independently associated with CGI-I scores. No mediators or moderators were identified for either reduction in tic severity or clinical improvement. There may have been a lack of power to detect mediators and moderators for the relationship between usage and impact of the intervention due to the high level of overall uptake and the relatively modest impact compared to face-to-face therapy. 3

Strengths and limitations

This study marks a comprehensive evaluation of a digital intervention delivered to CYP with tic disorders. This understanding is important in order to fully understand the circumstances under which such interventions are likely to be effective, for whom and maximise their efficacy. 148 In addition to explaining intervention-specific processes, the findings also contribute to the currently limited knowledge regarding how CYP and their parents engage with digital interventions to manage complex symptoms. Furthermore, it provides evidence of the feasibility and utility of digital interventions amongst a youth population. 102

There are several strengths within the design of this process evaluation, which was carried out concurrently with the ORBIT trial. A particular strength was that the methodology was based on a peer-reviewed published protocol,101 which is considered ‘best practice’ within process evaluation research. The sample involved in the semistructured interviews and within the intervention group was comparable in terms of age, gender and baseline YGTSS-TTSS. As stipulated in MRC guidelines,103 a mixed-methods approach was undertaken in the form of an integrative mixed-methods design. Moreover, the qualitative data offered a more in-depth evaluation to participants, therapists and referring clinicians’ perceptions and experiences with ORBIT and were able to capture outcomes that the quantitative data set could not. 102 Furthermore, the analysis of intervention usage data provided a detailed and objective insight into the important features and underlying determinants central to the implementation and effectiveness of this intervention. As recommended,103 process evaluation data were analysed prior to the trial findings being known, thus avoiding a biased interpretation of the data.

Another major strength of this study is the use of an objective, comprehensive measure of a child’s level of engagement with the intervention. By using a principal component analysis of dose of intervention received, we were able to capture an innovative measure of engagement, which could be replicated in future designs of such studies. Several strategies were utilised to strengthen the validity of the qualitative analyses. A multidimensional perspective was achieved by considering both CYP’s and parents’ as well as therapists’ and referring clinicians’ perceptions. 149 A subset of the transcripts was also double coded and disagreements in coding were discussed, strengthening the validity and reliability of the qualitative data. A further strength of the qualitative data was the use of PPI in guiding the semistructured interview questions. 102

Despite these strengths, this study and the subsequent findings should be interpreted with caution due to several limitations. Firstly, it was difficult to recruit participants for interview who had either dropped out of the ORBIT trial early or were not deemed treatment completers, and thus the data may have been skewed towards more positive experiences of the intervention. Limitations relating to the quantitative data set must also be considered. This study was embedded within a trial, which limited the sampling frame. Although the ORBIT trial was one of the largest in online tic disorder research, for the purposes of the mediator and moderator analyses in particular the sample size was somewhat underpowered. Therefore, the lack of statistically significant findings within these analyses may have been due to the lack of power, and so such results should be considered as exploratory. Finally, whilst process evaluations tend not to assess the control arm, doing so might have strengthened the generalisability of the overall findings while giving a more holistic view of ORBIT. 102

Implications for research and practice

The findings of this process evaluation have implications for potential research in this area. Primarily, it would be important for any future work to overcome the limitations highlighted above. 102 Indeed, a more intensive effort to recruit participants who did not engage satisfactorily with digital interventions or dropped out early would be welcome. This would enable researchers to gain a more holistic understanding of the implementation and mechanisms of action and prevent the findings from being skewed towards more positive experiences. 102

Future RCTs of complex interventions should consider the integration of quantitative process data from the outset, which could then provide further insight. More broadly, it would be valuable for future studies of complex interventions for CYP to include process evaluations. These should include objective usage metrics and longitudinal qualitative data. Finally, and most crucially of all, future research should consider how this evidence-based online ERP intervention can be made deliverable in routine NHS care, thus giving more people access to much-desired non-pharmacological treatments. 102

As discussed, tic disorders and associated conditions are highly debilitating, having a profound impact on both CYP and their parents. A range of tic-related difficulties with academic work and social and emotional well-being in CYP have been reported. Considering these tic‐related impairments and implications for future life, knowledge of the best treatment options for tic disorders in CYP is clinically important. 102 Children and their parents generally prefer BT over medication due to the fewer associated adverse effects;21 however, the most widespread mode of treatment is pharmacotherapy. 12

Therefore, the findings from this process evaluation have important implications for practice in a myriad of ways. They show how CYP engage with complex interventions and the importance of parental support and motivation, identify those who benefit the most from such interventions and, more broadly, show how DHIs can be designed and implemented in order to maximise their efficacy. 102 Most importantly of all, the findings demonstrate that an online intervention is effective, can be delivered with high fidelity and is highly acceptable to CYP with tic disorders, and that there is no subgroup who benefits the most. This has important implications for how this intervention could be delivered to patients within the UK. An online intervention that could be deployed to large numbers of patients at a relatively low cost is a much-needed and seemingly acceptable means of providing patients with access to an evidence-based treatment. It could provide immediate access to ORBIT for those who otherwise would not have access due to long waiting lists or their geographical location, which could also potentially free up existing resources and services for those requiring more complex treatment and assessment. 102 Such cutting of costs and waiting times would represent a two-fold benefit for the NHS and patients alike.

In addition, one of the barriers to reach identified was the associated travel to the baseline assessment. It would therefore be sensible to have an initial remote assessment. As the use of remote medical assessments has increased due to the COVID-19 pandemic, this flexible approach would allow even more people to benefit from this intervention. 102 As already discussed, it is also recommended that the term ‘therapist’ should be altered to avoid high expectations of their role. Furthermore, the ‘therapists’ could be employed on a part-time basis and would require very little training, which would cut costs even further. As the findings indicate, ‘therapists’ should engage the parents as much as possible to achieve successful outcomes.

Conclusions

This mixed-methods study is a comprehensive assessment of the processes underlying a complex online intervention delivered to CYP with tic disorders using MRC guidelines as a framework. 150 Overall, ORBIT is an effective and acceptable means of delivering an evidence-based ERP treatment to CYP with tic disorders and supporting them in overcoming barriers to accessing this therapy. Whilst some CYP may require additional support from their parents to enhance their level of engagement with the intervention, there is substantial evidence that this online intervention is a promising means through which these debilitating and complex symptoms can be addressed. 102

Refining the therapy

Once the initial content of the interventions had been finalised, both interventions were sent out to review with two parent–child dyads from the PPI panel and two members of the MindTech Involvement team.

The two parent–child dyads were given a login respectively to the child and parent/supporter interventions. Each PPI member was provided with a feedback form (Microsoft Word format) to complete alongside going through the intervention. They were provided with 4 weeks to review the intervention.

Several changes were made to the two interventions (child and parent/supporter) based on feedback:

• inclusion of a ‘warning’ at the beginning of both interventions that the intervention would be about tics, therefore potentially making the child tic more when engaging with it (suggestibility of tics)

• reducing length of some videos and/or cutting one video into two videos

• ensuring all audio on videos was the same volume

• reducing text on lengthy webpages

• correcting spelling errors/typos and clarifying sentences and text lost in translation

• both interventions having three ‘characters’ with tics – feedback suggested that these were better as cartoons and not as photos.

These changes resulted in the final versions of the ERP and psychoeducation interventions described in Chapter 2.

Equality, diversity and inclusion

Discussion

Patient and public involvement was an integral part of the ORBIT trial, integrated from start to finish and involved in all key steps.

Our two core channels of PPI involvement were through the national charity Tourettes Action and through our panel of PPI parents and young people. We believe combining both was a particularly effective approach in PPI. Tourettes Action allowed us to reach a larger audience and to develop a national profile with regards to the trial. This platform was pivotal in not only facilitating recruitment, but also educating the public on the aims and outcomes of the trial. In contrast, our PPI panel allowed us to get in-depth critical opinions and advice on key study documents and dissemination materials, as well as informing our dissemination strategy. Both approaches bought a different but complimentary angle to our PPI.

The key areas where PPI was particularly important were recruitment, retention and dissemination. Facilitating self-referrals into the study via Tourettes Action contributed over half of the total number of initial referrals into the trial and was a key part in achieving the recruitment target within time. Furthermore, not only did our PPI provide direct input into the patient-facing documents, they also co-developed the strategies for engagement/retention and dissemination. The success of this is likely to be reflected in our excellent recruitment rate and retention to the primary outcome, continuing until the 18-month follow-up.

We would have liked to have had face-to-face communication with our families as well as attendance at our team meetings. Although we did offer this, it was not taken up by any of our PPI members. Reasons for this included issues with time to travel, taking time away from school/work and not feeling comfortable with ‘ticcing’ during meetings or around strangers. However, by being flexible in our communication style with families we were able to keep our original PPI panel throughout the trial. We believe that offering the mode of communication that suited them at a time that suited them (usually outside traditional work hours) and providing sufficient time and feedback on tasks were key to facilitating this. It is also important to note that the members of our PPI group were reimbursed for all of their activities in line with INVOLVE guidelines on PPI payments.

There were some limitations in what we achieved with regards to PPI involvement in terms of analysis. As mentioned, we approached members of our PPI team to analyse a subset of interview transcripts for the process evaluation; however, this was not feasible as a result of the COVID-19 pandemic, lack of training for the young people and time constraints. There have been studies in which PPI members have been formally trained in qualitative analysis and in which the analysis is conducted collaboratively with researchers in face-to-face workshops. Due to the pandemic and time constraints, such an approach was not feasible. In hindsight, we could have conducted workshops via videoconferencing; however, this would have been difficult to achieve due to the complexity of the qualitative analyses.

Despite this, the PPI involvement in ORBIT was largely successful. Dr Seonaid Anderson, who supported the trial as part of her previous role in Tourettes Action as well as her position as a freelance neurodiversity consultant (Neuro-Diverse | Dr Seonaid Anderson, neurodiversity consultant), summarised the PPI on ORBIT:

• HUGE collective research AND clinical expertise in the ORBIT team

• genuinely include people in the different stages of their research, not just to have them as participants

• actively listened to my expertise in terms of how to reach potential participants

• were creative and adaptable in their style of reaching an audience

• innovative in creating videos to explain what the study was about and videos about results.

Dr Seonaid Anderson, December 2021

Reflections

Overall, PPI has been a crucial element of the ORBIT trial, which has been integral to the study’s design, delivery and dissemination. The impact of this involvement was overwhelmingly positive, and there were no negative outcomes associated with PPI; however, we note the limitation regarding the involvement of the PPI in the analysis of the intervention. Having an adequately costed PPI budget was crucial to support the delivery of this important aspect of the work.

Our research team consisted of a combination of researchers with previous experience of PPI (such as the trial manager), alongside more junior members with limited or no experience. Utilising the trial manager as the main point of contact enabled junior members of staff to learn about PPI involvement and thus supported their development as researchers. In the initial application, we had suggested that co-applicant S. Brown would facilitate most of the PPI activities. Brown provided support and guidance on PPI where needed; however, given the level of involvement that our PPI group had, it was more efficient and effective for this to be led by the trial manager, who had a more comprehensive understanding of the trial progress and need for PPI.

Although we did not have any set meeting dates, we still were able to facilitate and benefit from regular PPI input, and the use of monthly newsletters ensured that the PPI families were kept abreast of the trial progress. This approach of more flexibility with meetings/task involvements was novel to our research team, with previous research following more specified meetings and areas for input. However, we feel that ORBIT was a good example of true PPI collaboration that facilitated meaningful involvement. The fluid approach was preferred by our PPI families, which may be a result of the characteristics of our group (i.e. young children with tics and potential anxiety regarding participating in group meetings).

Our PPI partner, Tourettes Action, particularly our key contact, Dr Seonaid Anderson, provided the expertise in the topic area and lived experience of the population to facilitate PPI involvement. The charity played a key role in PPI and in offering a route to identify families to form the PPI panels. This meant that PPI was straightforward, present and meaningful.