Play brick therapy to aid the social skills of children and young people with autism spectrum disorder: the I-SOCIALISE cluster RCT

Primary objective

1. To examine the clinical effectiveness of LEGO^® based therapy groups on the social and emotional competence (specifically perceived social skills) of autistic CYP in schools, compared with usual support for autistic CYP, at 20 weeks after randomisation.

Secondary objectives

1. To examine the clinical effectiveness of LEGO^® based therapy groups on the perceived social isolation of autistic CYP in schools, compared with usual support for autistic CYP, at 20 and 52 weeks.

2. To examine the clinical effectiveness of LEGO^® based therapy groups on the academic competence of autistic CYP in schools, compared with usual support for autistic CYP, at 20 and 52 weeks.

3. To examine the clinical effectiveness of LEGO^® based therapy groups on assertion, social control, externalising and internalising in autistic CYP in the school setting, compared with usual support for autistic CYP, at 20 and 52 weeks.

4. To examine the cost-effectiveness of LEGO^® based therapy groups, compared with usual support for autistic CYP, at 52 weeks.

5. To examine the emotional and behavioural symptoms in those receiving LEGO^® based therapy, compared with usual support for autistic CYP, at 20 and 52 weeks.

6. To determine if the impact of LEGO^® based therapy is sustainable into the next academic year by comparing effectiveness on social and emotional competence (specifically perceived social skills) at 52 weeks after randomisation.

7. To examine the acceptability of the intervention at follow-up points using a purpose-designed questionnaire and telephone interviews at 20 weeks.

8. To examine treatment fidelity through independent observation of treatment sessions across schools and a self-report measure completed by the facilitator [i.e. a trained teacher or teaching assistant (TA)] after each session.

Trial design

The I-SOCIALISE trial was a pragmatic, two-arm, cluster RCT at the school level with an internal pilot designed to examine the clinical effectiveness and cost-effectiveness of LEGO^® based therapy groups for autistic CYP. The internal pilot study assessing recruitment feasibility was carried out over the first 10 months, for which a recruitment target of 120 CYP was set, one-third of whom (n = 40) were expected to have reached the primary end point of follow-up at 20 weeks after randomisation. Stop/go criteria were based on 75% of the recruitment target (n = 90) and 70% of the primary outcome target (n = 28) to assess the feasibility of continuing the trial. The trial also included an assessment of intervention fidelity and acceptability, a qualitative element and a nested economic evaluation.

Important changes to methods after trial commencement

Participants and eligibility criteria

Participants were CYP meeting the inclusion criteria, their parents/guardians, associated teachers/TAs in the CYP’s schools who knew them well and facilitator teachers/TAs who could run the study intervention if the school was allocated to the intervention arm. CYP were asked to provide informed assent and parents/guardians provided informed consent for themselves and their CYP to participate. Optional consent items were included on the consent form regarding video-recording of some of the intervention sessions and future contact about this or other research. Additional CYP were also recruited to the study when schools had fewer than three eligible CYP (i.e. the number needed for a LEGO^® based therapy group). These additional CYP did not need to meet the inclusion/exclusion criteria and a minimal amount of information was collected about these additional CYP, including chronological age, gender, any documented diagnoses and details of any special educational needs (SEN) support. A parent/guardian was asked to provide this information and to complete a consent form for their child or young person’s participation, which also included two optional consent items regarding video-recording of some of the intervention sessions and future contact about this or other research.

Settings and locations where the data were collected

Data were collected from participants in mainstream primary and secondary schools in York, Leeds, Sheffield and surrounding areas in the north of England. Data acquired from CYP and their parent/guardian were collected either in their home or at the child or young person’s school, depending on the family’s preference. Data from associated teachers/TAs and facilitator teachers/TAs were collected in school, via post or through a secure link sent via e-mail. Follow-up data were collected from CYP, parents/guardians and teachers through face-to-face visits at home (families only) or school, via post or through a secure link sent via e-mail.

Intervention

Outcome measures

A range of outcome measures was used during the trial to assess the primary and secondary objectives.

Outcome measures were selected on the basis of (1) relevance for the study parameters and participant abilities, (2) brevity to reduce participant burden, (3) appropriate informant versions for those completing the measures and (4) acceptability to patient and public involvement (PPI) groups. The primary outcome measure, the SSIS, is a behaviour rating scale that can be completed by parents, teaching staff and appropriate students. It is widely used in national portfolio studies and has been shown to be sensitive to change resulting from interventions in autistic children. It has good levels of reliability and validity and is validated for use by teachers.

Measures were completed by CYP, parents/guardians, associated teachers/TAs who knew the CYP well and the facilitator teachers/TAs who delivered the intervention (if the child or young person’s school was allocated to the intervention arm). Data were collected face to face in family homes or at school, by post or online via a secure link sent via e-mail. All measures were completed by participants in both trial arms at baseline, 20 weeks after randomisation and 52 weeks after randomisation unless otherwise stated.

Changes to trial outcomes after the trial commenced (with reasons)

As discussed in Important changes to the methods after trial commencement, the study’s first follow-up time point was changed in March 2018 from 16 weeks post randomisation to 20 weeks post randomisation. This was implemented because of concerns in the study team that the 12 sessions of LEGO^® based therapy would not have been completed by all schools when approached for their 16-week follow-up and that resulting data would not be balanced, with schools at different stages of the intervention. This change was approved and implemented following discussions with the trial DMEC and TSC. The primary outcome measure to be used in ITT analysis was clarified in September 2018 as the social skills subscale of the SSIS (as opposed to the entirety of the SSIS).

The fidelity checklist outcome measure completed by the facilitator teacher/TA after every session of LEGO^® based therapy was altered in September 2018 after discussions around the ease of completion for participants. Question 14 originally included one question followed by a separate, second question. This was followed by an ‘or’ option leading to question 15. For clarity and ease of use by facilitators and study team members, the second question in question 14 was changed to become question 15a and question 15 became 15b.

During our study the country faced the consequences of the COVID-19 pandemic. UK schools closed multiple times over the course of the year (2020–1). However, for this study the impact was limited to the first ‘lockdown’ and school closures in March 2020. This coincided with the final months of data collection, meaning that for a small number of our 52-week follow-ups some CYP responses, for example in relation to levels of anxiety, may have been affected by the pandemic.

There was no impact on intervention delivery or the primary outcome measure because these had been completed by the summer of 2019.

Some additional analyses were conducted to assess the potential impact of the pandemic on the trial; these are discussed in Chapter 3, Recruitment and participant flow, Dates defining the periods of recruitment and follow-up.

Sample size calculations

The sample size used for the study was calculated based on a Cochrane review of five studies assessing the effects of social skills groups on the social competence of participants. 28 Four of the studies29–32 reported were RCTs that used standardised tools to measure social competence and could be synthesised via meta-analysis. When comparing group intervention receipt and usual support, the weighted mean standardised difference in social competence was found to be 0.47 [95% confidence interval (CI) 0.16 to 0.78] and the review stated that this represented a clinically significant change in social competence. When using this standardised effect size to calculate sample size to give 90% power and at the 5% two-sided significance, a total of 194 participants would be needed (i.e. 97 per trial arm). This number was increased to 232 participants (i.e. 116 per trial arm) to allow for an attrition rate of 16%, the highest of the reported rates in the Cochrane review of social skills interventions. 28 The sample size was further inflated to 240 participants to allow for trainer/school effects, assuming equal clusters of size four (i.e. two LEGO^® based therapy intervention groups per school and two CYP participants with a diagnosis of ASD per LEGO^® based therapy group). The intraclass correlation (ICC) was assumed to be 0.01 based on findings from the ASSSIST (Autism Spectrum Social Stories In Schools Trial) feasibility study. 65 This gave a total recruitment target of 240 participants (i.e. 120 participants per trial arm) or 12 participants per month over all recruiting areas.

Explanation of any interim analyses and stopping guidelines

The study included a 10-month internal pilot to assess the feasibility of conducting the trial. The expected recruitment target was 120 participants, based on a recruitment rate of 12 participants per month, one-third (n = 40) of whom were expected to have reached the primary end point of 20 weeks post randomisation. Stop/go criteria were used to assess whether or not the study should continue. This was based on 75% of expected recruitment at 10 months (n = 90) and 70% of the primary outcome target (n = 28). In reality we had only 8 months for recruitment because of school holidays and delays with study approvals leading to recruitment start delays, meaning that the recruitment rate target became 15 participants per month.

Method used to generate the random allocation sequence

The unit of randomisation was school, each of which was randomised using stratified randomisation lists based on two strata: stage of education (i.e. primary or secondary) and number of eligible and participating CYP (i.e. ≤ 6 or > 6 CYP). All consented CYP were included in the intervention clusters. A blinded statistician from the Clinical Trials Research Unit (CTRU) at the School of Health and Related Research (ScHARR), University of Sheffield (Sheffield, UK), generated four randomisation lists (i.e. one for each combination of the two strata) using random blocks in the statistical package R (The R Foundation for Statistical Computing, Vienna, Austria). Allocation to arms was undertaken remotely. The co-ordinating researcher telephoned the CTRU for the next entry on the appropriate list after establishing eligibility, obtaining consent and collecting baseline measures.

Type of randomisation and details of any restriction (such as blocking and block size)

Cluster randomisation was used at school level to avoid CYP in the same school receiving different interventions, thus limiting contamination. The allocation list was generated prior to recruitment using block randomisation, with each group of two or four recruited schools having a 1 : 1 balance between arms. Group membership was determined chronologically (by date of recruitment) in the four strata and the size of each subsequent group was selected at random.

Sequence generation, enrolment and assignation

Following random allocation, schools were informed of their allocation by the trial manager by post and by telephone or e-mail, and families were informed by post. Once schools allocated to the intervention had been informed of this, they were asked to select a member of staff to attend training in LEGO^® based therapy and to run 12 sessions of the intervention before their 20-week follow-up. This method of enrolment was used successfully in a previous study of autistic CYP. 61

Blinding

All RAs in the study team were blind to treatment allocation to limit any potential bias at the two follow-up points (i.e. 20 and 52 weeks post randomisation). All forms were completed by participants themselves, and RAs had very little input, but, when instances of unblinding were reported, subsequent follow-ups were carried out by another blinded study team member, and these were recorded.

Trial statisticians were also blind to treatment allocation throughout the trial. The DMEC had access to unblinded data on request to enable it to investigate SAEs associated with the intervention.

Statistical methods

Intention-to-treat analysis was used to assess whether or not LEGO^® based therapy had any effect on the social competence and perceived isolation of autistic CYP. This approach is appropriate for a pragmatic trial because of its focus on external validity despite dilution bias dangers when the regression slope may be biased towards zero. 62 Per-protocol analysis was also used based on a per-protocol definition of receiving six or more sessions of the LEGO^® based therapy intervention. Comparability between baseline demographic and outcome measures was analysed and all data produced were assessed and reported based on RCT CONSORT guidelines44 and cluster RCT CONSORT guidelines. 45

The primary outcome measure was the social skills subscale of the SSIS completed by the associated teacher at 20 weeks post randomisation. This scale gives a summated score, treated as a continuous variable, with the significance level set at 5%.

As pre-specified in SAP version 2.0, the primary analysis of the primary outcome was performed using a multilevel mixed-effects model, with robust standard errors accounting for clustering within school. Covariates in the analysis were age, gender, baseline score on the social skills subscale of the SSIS, school (random effect), number of eligible and participating CYP (stratified by ≤ 6 or > 6 CYP), and school level (stratified by primary or secondary). The standardised mean difference, a two-sided 95% CI and p-values were presented. The primary analysis was performed on the ITT population.

Differences between arms were also summarised using unadjusted estimates with 95% CIs.

Secondary outcome variables were analysed using the same framework as the primary outcome. The sustainability analysis compared 20-week and 52-week estimates of differences between arms on the social skills subscale of the SSIS reported by the associated teacher.

Subgroup analysis of three levels of baseline symptom severity measured using the SCQ was undertaken in two ways. First, the interaction between subgroup and trial arm was added to the primary analysis model. Second, separate models were fitted for each of the three subgroups.

When data were missing from outcome measures because of withdrawals or attrition at follow-up points, multiple imputation (MI) methods were used in sensitivity analyses to reduce potential bias in the analyses resulting from the missing data.

All standardised effect sizes were calculated by rescaling the model estimated mean difference by the pooled standard deviation (SD) of the outcome at baseline, then multiplying by –1 for those outcomes in which a decrease in the score needs to be observed to find evidence in favour of the intervention arm.

Analysis populations

The following pre-planned analysis populations were studied for the primary outcome (analysis of secondary outcomes used the ITT population only):

• ITT – those with data available for ≥ 42 of the 46 items in the primary outcome measure (social skills subscale at 20 weeks reported by associated teacher) and with recorded consent information

• per protocol – the subset of the ITT population who received six or more of the 12 planned intervention sessions

• complier average causal effect (CACE) – ITT analysis accounting for compliance (i.e. six or more sessions received)

• MI population – imputation of missing social skills subscale scores using chained equations.

Additional analyses

Study oversight and management

The study was managed throughout by two trial managers, one at the ScHARR CTRU and one in the Leeds and York Partnership NHS Foundation Trust (LYPFT), with support from the Chief Investigator and CTRU Director. Oversight took the form of four groups: the TMG, the Operations Group, the TSC and the DMEC. TMG meetings, involving study team members and co-applicants, were held monthly for the majority of the study period until January 2019, when their frequency was changed to quarterly, with Operations Group meetings (involving core study team members) occurring monthly in their place. TSC and DMEC meetings occurred every 6 months for the duration of the study period and involved independent members and stakeholders with relevant expertise. In addition, a 6-monthly study report was submitted to the funder, the National Institute for Health and Care Research (NIHR) Public Health Research (PHR) programme, and regular communication occurred between all members of the study team.

Internal monitoring of the study was carried out in October 2018. At this time, each of the two sites [the Child Oriented Mental Health Intervention Centre (COMIC), University of York (York, UK), and the University of Sheffield CTRU] visited the other to check the study site files for completeness and accuracy. Findings were logged and any errors amended by the study team. In addition, in January 2019, the trial master file kept at the Sheffield CTRU was checked along with a randomly selected 10% sample of the completed CRFs at each site (COMIC, n = 9; Sheffield CRTU, n =  10). These were checked for accuracy of entry onto the bespoke clinical data management system. Data entry accuracy was relatively high.

External monitoring was then conducted at the COMIC research site in June 2019 by a member of the Harrogate and District NHS Foundation Trust’s research and development team. This included checking the site file for completeness and accuracy, assessing the trial’s Good Clinical Practice compliance, and source data verification (SDV) checking 10% of the completed CRFs at the site (n = 14). All errors noted in the monitoring report were amended by August 2019. This monitoring session confirmed the findings of the initial site monitoring in that there was a very small number of data entry errors. Therefore, the study team decided to carry out full SDV checks of all critical data (see Appendix 1, Table 29). Overall SDV error percentages were very low, with a total of eight forms out of 48 exceeding the 1% error rate. All identified errors were corrected on discovery.

Ethics arrangements and regulatory approvals

Positive ethics opinions for the study were obtained from the Health Research Authority (HRA) (18/HRA/0101) and from the University of York’s REC. Approvals for any changes to study documents or other required approvals were sought prior to implementation from the REC, the NIHR PHR programme, the sponsor and the HRA when necessary. The trial was not conducted in, and did not recruit from, the NHS, nor did it involve or identify participants based on their use of the NHS; thus, NHS ethics approval was not required. In addition, clinical trials authorisation was not required, as no medical devices or pharmaceutical elements were used.

Patient and public involvement

Background

The trial design was a cluster RCT to investigate the clinical effectiveness and cost-effectiveness of the LEGO^® based therapy intervention on the social and emotional skills of autistic CYP compared with usual support only. In this design, the health economics component was a cost–utility analysis, measuring the incremental cost-effectiveness ratio of the LEGO^® based therapy intervention over the control arm.

Effectiveness

Effectiveness for the health economics analysis was measured using the EQ-5D-Y and the CHU-9D.

Cost

Economic analysis

The health economics analysis was conducted from a UK NHS and PSS perspective. This took the form of a within-trial cost–utility analysis to compare the LEGO^® based therapy intervention with usual support for autistic CYP. Costs were measured using tailored service use questionnaires. Health outcomes were measured using QALYs based on the parent-/guardian-completed EQ-5D-Y as a health descriptor measure. QALYs were estimated using the area-under-the-curve approach between baseline and each follow-up.

Combining costs and QALYs, an ICER of cost per QALY gained was calculated and then compared with the national WTP threshold of £20,000–30,000 per QALY gained71 to assess the cost-effectiveness of the intervention.

Qualitative methods

Interviews were conducted following the completion of the intervention with a subsample of facilitator teachers/TAs across school types (primary or secondary) who provided consent to be invited to participate in the qualitative substudy. The TFA55 and normalisation process theory77 were used to guide the design of the interview schedule and help frame the data analysis to aid our understanding of acceptability and implementation issues related to the intervention. Sekhon et al. 55 define acceptability as ‘a multi-faceted construct that reflects the extent to which people delivering or receiving a healthcare intervention consider it to be appropriate, based on anticipated or experienced cognitive and emotional responses to the intervention’. Sekhon et al. 55 outline seven component constructs of acceptability. Given that we have used the TFA to structure the measurement of intervention acceptability, the definitions of each construct have been included as a direct quotation for transparency as follows:

• affective attitude – ‘How an individual feels about the intervention’55

• burden – ‘The perceived amount of effort that is required to participate in the intervention’55

• ethicality – ‘The extent to which the intervention has a good fit with an individual’s value system’55

• intervention coherence – ‘The extent to which the participant understands the intervention and how it works’55

• opportunity costs – ‘The extent to which benefits, profits or values must be given up to engage in the intervention’55

• perceived effectiveness – ‘The extent to which the intervention is perceived as likely to achieve its purpose’55

• self-efficacy – ‘The participant’s confidence that they can perform the behaviour(s) required to participate in the intervention’. 55

Normalisation process theory has been proposed as a way of identifying whether or not an intervention is likely to become part of routine practice. 78 It offers an explanatory model for the adoption and embedding of new practices into pre-existing routines based on the concept that changing such established routines is a complex process for individuals. The interview schedule was piloted with two teaching staff with experience of delivering LEGO^® based therapy prior to commencing data collection as part of the trial.

Invitation e-mails were sent to all eligible facilitators who had given prior consent to participate in an interview, which included a short reminder of the interview purpose, and those who were still interested and available to participate responded accordingly.

All interviews were conducted by telephone by a member of the research team (AB, a postgraduate non-clinical RA at the time of study). Facilitators were asked to position themselves in a private location (e.g. work offices/private rooms on school premises) during the interview. All interviews were recorded and transcribed verbatim. Interview data were analysed using the framework analysis approach,79 supported by NVivo version 12 (QSR International, Warrington, UK) software.

Interview data were coded by two independently trained researchers in the research team. The coders met regularly to develop a coding manual and to ensure the grounding in original data of all developed codes. The framework used was amended to allow for new codes and the deletion of unnecessary codes, leading to a final framework representative of the whole data set. The final versions of the coding manuals were presented to the TMG and TSC to confirm validity, coherence and relevance.

Acceptability study

The acceptability of the intervention to CYP was assessed using the number of sessions attended and data collected from the facilitator and parent/guardian, because we did not want to overburden the CYP participants at each session. We designed a bespoke questionnaire based around the TFA55 to assess acceptability of the intervention to parents/guardians and facilitators at the 20-week time point. This questionnaire was piloted with two members of teaching staff with experience of delivering LEGO^® based therapy and with the PPI members of the TMG prior to finalising content and design. All data were summarised descriptively. Results from the survey are presented using descriptive statistics.

Fidelity methods

Intervention fidelity was monitored and assessed using the following mechanisms throughout the study:

• An abridged training manual created by the study team based on the LEGO^® based therapy manual. 35

• A training programme developed with the training manual and delivered by co-applicant and co-author of the LEGO^® based therapy manual, Gina Gomez de la Cuesta.

• Video-recording of a subsample of intervention sessions in schools; full written consent for this was obtained from the parents/guardians of all group members (and assent from group members themselves where possible) and the facilitator teacher/TA. Details are as follows:

  • A fidelity checklist developed from work by Gina Gomez de la Cuesta was used to monitor the content of the intervention sessions. This was also completed by facilitators after each session.

  • The aim was to record 10% (i.e. 72) of all intervention sessions in the study. These were sampled at school level (primary or secondary), with the aim of three session recordings per school, ideally one of the first four sessions, one of the second four sessions and one of the final four sessions.

  • Videos of the intervention sessions were reviewed by two independent observers to assess fidelity to the checklist, and inter-rater reliability was analysed.

  • Established criteria were used to classify the extent of observed fidelity of delivery:80 if ≤ 50% of intended content was delivered, this was classified as ‘low’ fidelity; if 51–79% of intended content was delivered, then this was classified as ‘moderate’ fidelity; and if 80–100% of intended content was delivered, then this was classified as ‘high’ fidelity. Descriptive statistics were produced on all completed fidelity checklists.

• Global assessment of fidelity across all LEGO^® based therapy sessions, including those that were not video-recorded and included in the observed fidelity assessment. Self-reported fidelity checklists were also administered to all intervention arm schools for completion by facilitators on a weekly basis, immediately at the end of each session.

  • The facilitator-completed checklist, which was compared with the recorded sessions when possible.

  • The proportion of intended components of LEGO^® based therapy rated as delivered or not by facilitators. The same criteria were used to classify extent of fidelity as in the observed measurements: if ≤ 50% of intended content was delivered, then this was classified as ‘low’ fidelity; if 51–79% of intended content was delivered, then this was classified as ‘moderate’ fidelity; and if 80–100% of intended content was delivered, then this was classified as ‘high’ fidelity. Variations in proportion of fidelity of delivery were examined across session numbers and intervention groups. Descriptive statistics were produced on all completed fidelity checklists.

Losses and exclusions after randomisation

A small number of participants (n = 48) withdrew from the research following randomisation or were lost to follow-up and so did not complete the study. No participants were excluded. Table 2 presents withdrawal specifications. Withdrawal reasons can be found in Appendix 2, Table 32.

Dates defining the periods of recruitment and follow-up

Recruitment to the trial commenced in October 2017 and continued until March 2019. The 20-week post-randomisation follow-up collection period began in March 2018 and ended in October 2019. The 52-week post-randomisation follow-up collection period began in November 2018 and ended in June 2020. The COVID-19 pandemic had no impact on response rates for the primary end point (20 weeks after randomisation) and a small impact on data collection rates at 52 weeks for the second of the two school years in which we recruited. If the response rate had shown similar patterns in both years, we would have expected 222 responses at 52 weeks, rather than the 203 we received (Table 3).

The number of weeks between randomisation and outcome data collection is summarised in the left hand side of Table 4. Expected windows for the 20-week and 52-week follow-ups were 20–26 weeks and 52–60 weeks, respectively. The ITT analysis included all data irrespective of whether they were collected within or outside these expected follow-up windows. The impact on the analysis of data being collected outside these expected follow-up windows was explored by refitting the analysis models to the subset of data collected within the expected follow-up windows. This analysis is reported in the right hand side of Table 4. The only response returned after the start of the COVID-19 pandemic national lockdown was the 52-weeks follow-up booklet for one participant. Therefore, the sensitivity of the primary outcome to the COVID-19 pandemic lockdown is omitted.

Outcomes

The primary outcome was the social and emotional competence of the child or young person with an ASD diagnosis in a mainstream school setting measured 20 weeks post randomisation using the teacher-completed social skills subscale of the SSIS. The SSIS (teacher) completed at 52 weeks was a secondary outcome. Further secondary outcomes and measures are summarised in Table 9.

Estimation

Summary of findings

The ITT analysis of the primary outcome is of borderline statistical significance (p = 0.06) at the pre-specified 5% level. However, the pre-planned sensitivity analysis of the primary outcome provides consistent evidence of LEGO^® based therapy groups having a modest positive effect (standardised effect sizes between 0.15 and 0.21) on participant social and emotional skills measured using the social skills subscale of the SSIS at 20 weeks (see Table 12).

Only 3 out of the 31 pre-specified secondary outcomes were significant at the 5% level, and there is a high probability that these were observed by chance. However, the majority of the secondary outcome effect sizes favoured the intervention arm, which corroborates the modest impact observed for the primary outcome. Figures 2 and 3 present the outcomes observed at 20 and 52 weeks, respectively.

FIGURE 2.

Secondary outcomes at 20 weeks.

FIGURE 3.

Secondary outcomes at 52 weeks.

A large number of unplanned additional analyses were conducted to explore the potential for observational analysis of the trial data in a subsequent substudy. These analyses show that the effect size is potentially greater for certain subgroups and when there is a critical mass of autistic participants in therapy groups. These additional analyses are not intended to be confirmatory and are not used in drawing the overall conclusion of a modest borderline significant effect.

Primary outcome analysis

The primary analysis comparing LEGO^® based therapy groups (i.e. the intervention arm) with usual support (i.e. the control arm) found a more modest, borderline statistically significant, increase in the social and emotional competence of CYP with an ASD diagnosis in the intervention arm than in the control arm. The difference between arms was 3.74 (95% CI –0.16 to 7.63; p = 0.06) (Table 10). In the original sample size calculation, a standardised effect size of 0.47 of a SD was the minimum clinically important difference (MCID). At baseline, the pooled SD for the primary outcome was 20.4 (Table 11). Hence, the rescaled MCID was 9.6 and the observed difference was smaller than this. Changes in primary outcome are detailed in Appendix 2, Table 37.

We had pre-specified a number of sensitivity analyses (Table 12), including an as-treated analysis to allow for the misallocated school. For the per-protocol analysis (six or more sessions attended) the p-value reduced to 0.036 and the difference between arms increased to 4.23 (95% CI 0.27 to 8.19). A second, post hoc per-protocol analysis (with two or more CYP with an ASD diagnosis in each session on average) also had a reduced p-value (p = 0.038) and an increase in the difference between arms to 4.56 (95% CI 0.24 to 8.87). For the as-treated analysis the p-value was further reduced to 0.026 and the adjusted mean difference again increased; however, the range of values included in the 95% CI remained below the MCID (difference between arms 4.37, 95% CI 0.53 to 8.20). Both per-protocol analysis models showed a statistically significant positive effect. The CACE analysis associated with the first per-protocol analysis reduced the difference between arms to 3.97, with a larger p-value than in the per-protocol analysis (95% CI –0.11 to 8.06; p = 0.056). The CACE analysis associated with the second, post hoc per-protocol analysis increased the difference between arms to 5.68, again with a larger p-value than in the per-protocol analysis but with a CI that includes the MCID (95% CI –0.18 to 11.54; p = 0.058). However, the analysis when using MI to allow for missing data at baseline and 20 weeks diminished the signal somewhat. The MI was undertaken in Stata MP 16.0 to test the robustness of findings for missing values of the primary outcome measure at baseline (n = 1 owing to missing survey items) and 20 weeks (n = 32 owing to withdrawal, non-response or missing survey items). There were no missing data for other baseline covariates. A total of 100 MI data sets were generated using chained equations, assuming missing at random and a multivariate normal distribution. Logistic modelling found no other covariates that were predictive of missing status. School random effects were included in the estimation stage of the MI procedure. Overall, there is consistent borderline statistically significant evidence that LEGO^® based therapy groups (i.e. the intervention arm) had a modest positive effect on the social skills subscale of the SSIS at 20 weeks (see Table 12). The average causal response (ACR) analysis, as specified in the SAP, is omitted because the distribution of the number of sessions attended was negatively skewed and this violates the assumption required for an ACR analysis.

Secondary outcome analysis

Table 9 summarises the five secondary statistical objectives covered in the SAP; these include 33 associated end points (see Chapter 1, Research question, aims and objectives, and Chapter 2, Main trial methods, Important changes to methods after trial commencement). Tables 13 and 14 show full results and summary results, respectively. The health economic analysis and qualitative and fidelity results address the trial’s three additional secondary objectives. Secondary outcomes tended to have slightly higher rates of missing data than the primary outcome, so the analysis lacked some information. However, with respect to the secondary objectives, 2 of the 33 end points showed a significant difference between arms in favour of the intervention arm at the 5% level and four more showed a significant at the 10% level, although these small numbers of significant differences between arms may have occurred by chance. More generally, most of the 33 secondary end points showed a difference between arms in favour of the intervention arm and, although few were statistically significant, this consistent trend strengthens the evidence to support the hypothesis that LEGO^® based therapy groups have some beneficial effects. A summary of secondary outcomes by arm is given in Appendix 2, Table 38. Overall data completeness is detailed in Appendix 2, Table 39.

The first secondary objective (i.e. to examine the clinical effectiveness of LEGO^® based therapy groups on the perceived social isolation of autistic CYP in schools, compared with usual support for autistic CYP, at 20 and 52 weeks) was assessed using the ALS and the MSPSS friends and significant other subscales. CYP in the intervention arm had slightly lower (i.e. worse) ALS scores than those in the control arm, and the gap widened at 52 weeks, although neither gap was significant: at 20 weeks, –1.31 points (95% CI –4.89 to 2.27 points; p = 0.47); at 52 weeks, –2.83 points (95% CI –6.39 to 0.74 points; p = 0.12). For the MSPSS, between-arm differences marginally favoured usual support at 20 weeks and the intervention at 52 weeks, but both differences were very small: at 20 weeks, –0.06 points (95% CI –0.44 to 0.31 points; p = 0.74); at 52 weeks, 0.16 points (95% CI –0.23 to 0.56 points; p = 0.41).

The sixth secondary objective [i.e. to determine if the impact of LEGO^® based therapy is sustainable into the next academic year by comparing effectiveness on social and emotional competence (specifically perceived social skills) at 52 weeks after randomisation] was assessed using the SSIS social skills subscale as completed by the associated teacher at 20 weeks and 52 weeks after randomisation. At 20 weeks, the difference between arms was 3.74 points (95% CI –0.16 to 7.63 points; p = 0.06) (see Table 12) and at 52 weeks the difference was 1.68 points (95% CI –2.51 to 5.87 points; p = 0.43). Compared with the primary outcome analysis, which showed a borderline statistically significant effect in favour of the intervention, the effect at 52 weeks was diminished. There is no evidence that the modest effect of the intervention is sustained.

The second secondary objective (i.e. to examine the clinical effectiveness of LEGO^® based therapy groups on the academic competence of autistic CYP in schools, compared with usual support for autistic CYP, at 20 and 52 weeks) was assessed using the SSIS academic competence scale as completed by the associated teacher at 20 weeks (–0.17 points, 95% CI –1.17 to 0.84 points; p = 0.75) and 52 weeks (–1.57 points, 95% CI –2.91 to –0.24 points; p = 0.02). Although the effect at 52 weeks was statistically significant and in favour of usual support, the effect at 20 weeks was very small and not statistically significant.

The fifth secondary objective (i.e. to examine the emotional and behavioural symptoms in those receiving LEGO^® based therapy, compared with usual support for autistic CYP, at 20 and 52 weeks) was assessed using four completed components of two research instruments: first, the SDQ completed by the associated teacher at 20 and 52 weeks and by the parent/guardian at 20 and 52 weeks; second, the SSIS problem behaviours subscale completed by the associated teacher at 20 and 52 weeks and by the parent/guardian at 20 and 52 weeks. There is a borderline statistically significant effect seen for the SDQ completed by the parent/guardian at 20 weeks. Although this is reduced at 52 weeks and no longer borderline significant, the general trend favours the intervention arm. (For the medications being taken at baseline, see Appendix 2, Tables 40 and 41.)

The third secondary objective (i.e. to examine the clinical effectiveness of LEGO^® based therapy groups on assertion, social control, externalising and internalising in autistic CYP in the school setting, compared with usual support for autistic CYP, at 20 and 52 weeks) was assessed using the SSIS assertion, self-control, internalising and externalising subscales completed by the associated teacher and the parent/guardian at 20 and 52 weeks. There are eight pairs of measurements assessed in this objective; for two of the pairs, the SSIS (teacher) self-control and SSIS (parent) internalising subscales, the effects, which are similar in magnitude, were borderline statistically significant. Again, considering all eight pairs, the general trend favours the intervention arm.

Additional analysis of primary outcome

Intervention compliance

Availability of quality-of-life and cost data

A total of 284 autistic CYP were recruited to the trial. After removing 34 ineligible CYP, 250 autistic CYP were available for analysis (LEGO^® based therapy, n = 127; usual support, n = 123). The ‘complete case’ here refers to the parents/guardians of CYP who completed both EQ-5D-Y82 questionnaires and resource use questionnaires at any time point. Details of the quality-of-life and cost data available at each data collection time point are available in Appendix 3, Table 42. Quality-of-life data were fully available at any time point for 151 (62.1%) CYP, and cost data from the NHS and PSS perspective were available for 140 (57.6%) CYP. Overall, a total of 139 (55.6%) CYP had both EQ-5D-Y and resource use (from the NHS and PSS perspective) data at all three data collection time points. This sample constitutes the complete-case group, which is one of the two groups used for all the subsequent analyses. The second group (imputed-case or base-case group) is discussed in Chapter 2, Health economic methods.

Multiple imputation

To account for the data from the sample other than from the complete-case group (44.4%), missing utility scores and costs were further imputed with MI using chained equations (assuming missingness at random at each time point). The following variables were used in the imputation process to ensure best fit of the imputed results: trial arm, age, gender, site, utility scores and SCQ score at baseline. Because it would be speculative to impute missing utility at baseline, two CYP (0.8%) were excluded from the health economics study. The base-case (imputed) sample size was 248 CYP (intervention arm, n = 126; control arm, n = 123); this is the sample used for the primary cost-effectiveness analysis.

Baseline characteristics

Descriptive statistics of participating CYP’s characteristics and the additional predictors used in MI can be seen in Appendix 3, Table 43. These vary in relation to the main results, because the numbers of subjects included differed. This is based on the definitions of ‘complete case’ and ‘base case’ for the health economic analysis.

More than three-quarters of the CYP in the intervention arm and the control arm were male, and > 50% of the CYP in both arms were primary school age (7–11 years). Differences in SCQ scores at baseline were marginal across arms and samples. Overall, the baseline characteristics are consistent across samples (base case and complete case) and with the main statistical analysis.

Unit costs

Individual-level resource use together with unit costs were used to calculate the total services cost for each autistic child or young person. Unit costs were obtained from published sources (i.e. NHS Reference Costs 2017–201883 and the PSSRU’s Unit Costs of Health and Social Care 2018),84 a national survey (i.e. the Childcare Survey 2018)85 and government departments (i.e. data reported by the Department for Education). 70 Appendix 3, Tables 44–52, present the summary of key unit costs used in the study.

Intervention costs

Intervention costs included both training and intervention session costs. The training cost information was obtained from the study team, and the estimated training costs were allocated to each session and each CYP who received the intervention. Information on intervention delivery costs was obtained directly from the facilitators using the self-reported questionnaires. Intervention sessions were costed on the basis of the salary of the professional involved. The total and average costs of each component that were used to deliver the LEGO^® based therapy intervention can be seen in Appendix 3, Table 53. The main cost driver of training costs was trainer fees, whereas the main cost drivers of intervention delivery costs were the costs for preparation and delivery of the intervention and costs for LEGO bricks. On average, the estimated intervention cost per CYP was £96.96 (£25.29 for training and £71.67 for intervention delivery).

Service use and costs

The summarised service use per child or young person in each trial arm based on the complete cases only is presented in Appendix 3, Table 54. The difference in service use between the intervention and control arms was marginal at all three time points. At baseline, autistic CYP in the intervention arm seemed to have used slightly more NHS and education services (‘in the last 4 months’) than those in the control arm. After the intervention, such differences became smaller. In addition, CYP in the intervention arm were more likely to use fewer services at baseline. Caution is needed in interpreting these results, as these summarised service uses include various items with different unit costs.

The total costs broken down by perspective, type of service, trial arm and before and after imputation are presented in Table 23. Discounting was not applied owing to the short-term nature of the trial. The costs vary substantially from one participant to another, as illustrated by the wide 95% CIs.

In detail, the total mean service costs to the NHS providers (before imputation) were £618.30 (95% CI £428.30 to £808.20) in the intervention arm and £751.50 (95% CI £420.00 to £1083.00) in the control arm. Furthermore, the average total costs for the use of CAMHS-related community-based services, hospital-based services (both mental health and non-mental health-related) and non-mental-health-related medication were higher for the autistic CYP in the control arm than in the intervention arm. It is worth noting that some of the cost differences were likely to have been driven by the high-cost cases. For instance, the higher average total cost of CAMHS-related services in the control arm was driven by two high-cost cases, the first of which had 16 psychologist visits and the other 18 psychologist visits over the preceding 8 months. Because these high-cost cases were entirely plausible, we have decided to retain them in the data set without any adjustment.

The average total costs for the use of education services (before imputation) were £1388.20 (95% CI £989.10 to £1787.30) for the intervention arm and £1632.80 (95% CI £1041.30 to £2224.30) for the control arm. Apart from the school-based health-related services, CYP in the intervention arm used fewer education services than those in the control arm. As with the costs of CAMHS-related services, the higher average cost of school-based health-related services in the intervention arm was driven by one high-cost case, which had five education welfare officer visits per week for 36 weeks. Again, we decided to retain this in the analysis, as the scenario is plausible.

The CYP in the intervention arm also incurred fewer costs in private expenses and parental productivity losses than those in the control arm. Overall, CYP in the intervention arm incurred fewer costs across all the perspectives. This is observed in both the complete case and the base case. However, owing to the high-cost cases, the cost differences need to be interpreted with caution.

EuroQol-5 Dimensions-Youth

Table 24 shows the mean EQ-5D-Y utility scores between the two arms of the trial at each time point when scores were not imputed (complete case) and when scores were imputed (base case). In both arms, there was no significant change in EQ-5D-Y scores from baseline to 52 weeks. The fluctuations between baseline and 20 weeks, and between 20 weeks and 52 weeks, were small in both the base case and the complete case. After calculation using the area-under-the-curve approach, it was found that the LEGO^® based therapy produced marginally higher mean QALYs (0.03 QALYs) than the usual support. Further details for the responses in each domain can be found in Appendix 3, Table 55.

Child Health Utility-9 Dimensions

The change in mean CHU-9D utility scores between the two arms of the trial at any time point is presented in Table 24. CYP in the intervention arm had higher utility scores than those in the control arm at baseline. As in the EQ-5D-Y results, the observed changes at different time points in either arm were marginal. Such marginal differences were observed in both the complete case and the base case. Consistent with EQ-5D-Y results, LEGO^® based therapy produced marginally higher (i.e. by 0.02) mean QALYs than did usual support. Further details for the responses in each domain can be found in Appendix 3, Table 56.

Primary analysis

Table 25 shows the ICERs based on the base case for the primary analysis. On average, and before any adjustments, autistic CYP receiving LEGO^® based therapy incurred £56.20 less in costs from the NHS and PSS perspective and gained 0.03 more QALYs (as measured by the EQ-5D-Y) than those receiving usual support.

To account for the uncertainty and adjust for the imbalanced EQ-5D-Y utility scores at baseline, the estimates of incremental costs and the QALYs from regression were bootstrapped to simulate 5000 pairs of net cost and net outcomes, as recommended by NICE for health technology appraisals. 86 On average, after bootstrapping, autistic CYP receiving LEGO^® based therapy incurred £250.90 (95% CI £272.70 to £228.90) less in costs and gained 0.009 (95% CI 0.008 to 0.010) more QALYs than those receiving usual support, which is equivalent to an extra 3.3 days of perfect health. The calculated ICER was below the NICE-recommended WTP threshold (i.e. £20,000–30,000 per QALY gained) for decision-making in England and Wales, indicating that LEGO^® based therapy is likely to be cost-effective.

Figure 4 shows the cost-effectiveness plane for LEGO^® based therapy compared with usual support based on 5000 bootstrapped estimates of incremental costs and incremental QALYs. The blue line represents the WTP threshold of £20,000 per QALY gained. The simulated estimates were largely below the threshold line and sat in the fourth quadrant, suggesting that LEGO^® based therapy is likely to be cost-effective, although the incremental costs and QALYs were marginal.

FIGURE 4.

Base-case cost-effectiveness plane of LEGO^® based therapy compared with usual support (outcome measure: QALY as measured by the EQ-5D-Y; NHS and PSS perspective).

The CEAC of LEGO^® based therapy compared with usual support is presented in Figure 5. The probability of LEGO^® based therapy being cost-effective is 0.94 if decision-makers are willing to pay £20,000 for one QALY gained and 0.94 if they are willing to pay £30,000 per QALY gained.

FIGURE 5.

Base-case CEAC of LEGO^® based therapy compared with usual support (outcome measure: QALY as measured by the EQ-5D-Y; NHS and PSS perspective).

Sensitivity analyses

To account for uncertainty in the cost and QALY estimations, sensitivity analyses were conducted (Table 26). The mean incremental cost and QALY estimates from the complete case (scenario 1) were in line with the results from the best case (primary analysis), yielding a negative cost per QALY gained. Other sensitivity analyses using different cost perspectives and a different instrument (the CHU-9D) to measure QALYs were also conducted. In all of these sensitivity analyses, the ICER pairs lay below the NICE-recommended WTP threshold (£20,000-30,000 per QALY gained) (see Table 26) and the majority of the bootstrapped estimates sat in the fourth quadrant (Figure 6), suggesting that LEGO^® based therapy is likely to be cost-effective.

FIGURE 6.

Sensitivity analysis: cost-effectiveness planes for LEGO^® based therapy compared with usual support. (a) Base case, NHS and PSS perspective, EQ-5D-Y; (b) scenario 1, complete-case analysis; (c) scenario 2, NHS and PSS perspective, CHU-9D; (d) scenario 3, NHS and education perspective, EQ-5D-Y; and (e) scenario 4, societal perspective, EQ-5D-Y.

Delivery of LEGO^® based therapy sessions

A total of 69% of intervention groups received the required total of 12 LEGO^® based therapy sessions and 93% received the per-protocol minimum of six sessions (see Figure 7). Overall, the median number of sessions delivered was 12 (range 3–12 sessions). A total of 749 LEGO^® based therapy sessions were delivered as part of the trial. Appendix 4, Figure 10, shows the number of sessions delivered, grouped according to geographical location, and suggests that a similar dose of the intervention was delivered in Leeds, Sheffield and York.

Fidelity assessment of LEGO^® based therapy: self-reported

Table 27 shows the proportion of components of LEGO^® based therapy delivered by all facilitators during the trial, based on self-completion fidelity checklists and presented according to the ‘core content’ (items 1–4) and ‘all content’ (all 17 items) across all 12 sessions. The completion rate for the self-completed fidelity checklists was 99%, with only seven missing (from the total of 749). In line with guidance from Borrelli et al. ,80 self-reported fidelity was very high for both core content (99% of all sessions) and all content (91% of all sessions).

Delivery of core content was fairly stable over the 12 sessions (95–100%), with limited drift over time (i.e. at later sessions). Each school was required to deliver only one LEGO^® based therapy group (with some schools delivering two groups). This pattern of high fidelity of delivery was stable for all content over the 12 sessions (89–93%), albeit with the caveat that only 69% of groups completed all 12 sessions.

The least commonly delivered components were as follows:

• Was there an opportunity for each child or young person to show the other group members what they made, or talk about or play with what they made with others? (Yes, 67.5%.)

• When a model was complete, was it appropriately displayed, or were photographs taken of the models? (Yes, 67%.)

• If it was necessary to dismantle the models, was there warning and explanation given and were the children allowed to dismantle the models themselves (rather than the models being dismantled between sessions)? (Yes, 66%.)

• Was there evidence of a reward system (e.g. LEGO points) to promote positive social behaviour, or LEGO certificates for attaining the different group levels (helper, builder, creator, etc.)? (Yes, 78%.)

The first three items are dependent on completion of the LEGO model (which does not necessarily happen in all sessions, for example when the model is complex and straddles sessions), whereas provision of the reward (item 16) should happen at any session. All other checklist items were delivered in > 90% of sessions.

The number of sessions delivered by facilitator can be seen in Appendix 4, Figure 9, and by location in Appendix 4, Figure 10. Appendix 4, Figure 11, shows the proportion of core and all content of the sessions according to individual therapists. A total of 63% of group therapists (n = 44) delivered 100% of core content and a further 24% of therapists (n = 17) delivered 98% of core content across all of the sessions completed. The remaining therapists (n = 9; 13%) delivered between 69% and 96% of core content. A total of 14% of therapists (n = 10) delivered 100% of all content; 63% of therapists delivered > 90% of all content and the remaining therapists (n = 16; 23%) delivered 73–89% of all content.

Fidelity assessment of LEGO^® based therapy: independent rating

In addition to the self-assessed fidelity of intervention delivery, a subsample of LEGO^® based therapy sessions were video-recorded to facilitate content analysis. The number of sessions recorded for each school that was fully consented into the fidelity assessment substudy is available in Appendix 4, Table 57. For a group to be included in the independent assessment of fidelity, consent to be video-recorded was required from all CYP and facilitator participants. In total, 63 sessions from 22 schools were recorded, which is equivalent to 9% of all LEGO^® based therapy sessions in the trial. It was possible to organise recording of three sessions in 19 our of 22 schools, as planned, but logistical issues at other schools meant that it was possible to record only two sessions.

A total of 10 out of the 22 groups were from schools in Leeds, eight groups were from Sheffield schools and four were from York schools. The majority of groups (18/22) were from primary schools, and, accordingly, 51 our of 63 sessions were carried out in primary schools. The mean duration of sessions included in this substudy was 55 minutes (range 30–78 minutes). The recorded sessions comprised 20 early sessions (sessions 1–4), 22 middle sessions (sessions 5–8) and 21 later sessions (sessions 9–12) of LEGO^® based therapy. No first sessions were recorded, with the number of all subsequent recorded sessions ranging from three to nine.

To provide an independent assessment of fidelity, all 63 sessions were independently reviewed and rated against the fidelity checklist by a trained researcher (AB). A subsample of 16 sessions was double rated by two senior researchers/co-applicants (BW and GG, who was also the senior trainer of the LEGO^® based therapy) to ensure accuracy of rating. This double rating was completed at two time points (i.e. at the beginning and at the end of rating exercise). The level of inter-rater reliability was high at 84%,87 and all discrepancies between the two raters were discussed and a consensus reached with a third trained researcher (LC).

Table 28 presents the proportion of components of LEGO^® based therapy delivered in those sessions included in the independent fidelity assessment and presented according to the core content (items 1–4) and all content (all 17 items) across sessions 2–12. In line with guidance from Borrelli et al. ,80 self-reported fidelity was high for core content (83% of all sessions) and moderate for all content (77% of all sessions).

As with self-assessed fidelity, delivery of core content was stable over the 11 (out of 12) sessions (79–95%), with limited drift for later sessions (i.e. dropping to 65–75% in sessions 11 and 12). This pattern of fidelity of delivery was stable for all content over the sessions too, with a range of 65–86% throughout.

According to the independent rating, the least commonly delivered components were 15b and 15a (69%), both of which related to dismantling LEGO models; 10 (41%), which assessed if appropriate greetings took place and if these were prompted/encouraged; and 16 (59%), which related to clear use of rewards in the session. As with self-reported fidelity, it may be that some intervention components, such as displaying or dismantling models (items 15a and 15b), did not take place or, as in the case of the greetings (item 10) may have taken place outside the video-recording. However, evidence of the reward system was present in only 59% of the sessions. All other checklist items were delivered in > 70% of sessions.

Comparison of self-reported and independently rated fidelity assessment

Overall, self-reported fidelity of delivery was higher than independently rated fidelity of delivery, with fidelity for core content rated as 99% and 83%, respectively, and for all content rated as 91% and 77%, respectively. Despite these differences, both types of assessment can be classed as achieving high or high–moderate fidelity, according to Borrelli et al. 80

Appendix 4, Table 58, shows a comparison between the self-reported and independently rated delivery of individual components for all 63 sessions video-recorded as part of the independent assessment. Overall, there was 90% agreement between the self-reported and independent ratings of whether or not a component was delivered in the session. Where there was a conflict, this tended to be the result of facilitators reporting delivery of a component that was not observed independently (9%). For core content, the facilitators, for example, reported positive outcomes on items 3 and 4 (related to highlighting positive social interactions and problem solving by the CYP); for other items, such as group rules (item 7), greetings (item 10) or description of what will happen in the session (item 11), these were reported as present but were not perhaps visible or delivered on camera.

All participants

Table 29 and Figure 7 show a summary of the assessment of acceptability of LEGO^® based therapy to all facilitators and parents/guardians of autistic CYP in the trial (who evaluated this as proxy for their CYP). This was measured on a 5-point scale [from 1 (strongly disagree) to 5 (strongly agree)] across the different constructs of the TFA. This shows that acceptability of the intervention was high, with an overall median acceptability score for facilitators of 5 (range 4–5) and for parents/guardians of 4 (range 3–5). Acceptability to facilitators was generally higher than parents/guardians, and this was statistically significant (p = 0.000).

FIGURE 7.

Acceptability of LEGO^® based therapy: comparison of median scores for acceptability constructs for all participants. Higher scores indicate a greater level of acceptability.

Looking at specific TFA constructs, facilitators rated all of these positively, with median scores of 4 or 5 on each. All facilitators positively rated ‘affective attitude’ (i.e. they liked the intervention) and ‘self-efficacy’ (i.e. they felt confident to deliver the intervention), with high scores on all other constructs, and, overall, 92% rated the acceptability of LEGO^® based therapy positively. The lowest score for facilitators (73% for ‘perceived effectiveness on behaviour’) is still high.

Among parents/guardians, only ‘affective attitude’ received the maximum median score of 5, with 92% rating this positively. Overall, parents’/guardians’ acceptability score was slightly lower, at 71%, and this difference from the facilitators’ acceptability score was statistically significant (p = 0.000). With regards to specific TFA constructs, ‘burden’ (49%), ‘intervention coherence’ (47%) and ‘perceived effectiveness on behaviour’ (55%) were rated least positively by parents/guardians (although this is still fairly acceptable, with half of participants giving positive ratings). There were statistically significant differences between facilitator and parent/guardian ratings of acceptability for ‘burden’, ‘intervention coherence’, ‘self-efficacy’ (p = 0.000) and ‘perceived effectiveness’ (all dimensions) (p = 0.008–0.000).

Appendix 4, Tables 59 and 60, summarise the open-ended qualitative responses to the acceptability questionnaire given by facilitators and parents/guardians, respectively. Provision of the open-ended responses was optional, and the numbers reporting specific points are given for clarification only.

In general, the open-ended responses suggest that facilitators viewed LEGO^® based therapy positively, and this was reportedly viewed positively by the CYP in their groups also. Despite some challenges, LEGO^® based therapy was considered to be a rewarding and valuable experience. Facilitators were able to identify a number of perceived benefits from LEGO^® based therapy during the sessions (e.g. communication, social skills and confidence), but some cautioned that these did not always translate into the classroom. A small number of facilitators highlighted problems with LEGO^® based therapy delivery, including conflict, poor communication between two autistic CYP and CYP becoming fatigued at week 7 onwards. Five facilitators reported that their school had adopted LEGO^® based therapy into practice after the end of the trial delivery, although sometimes adapted to six sessions. There were mixed views on the optimal duration for LEGO^® based therapy, and one facilitator suggested that it was better suited to CYP of primary rather than secondary school age.

A majority of parents/guardians shared that their child had enjoyed taking part in LEGO^® based therapy. Only a small number thought that their child had not enjoyed it, for example because they had lots of prior LEGO^®-building experience or because of the gender composition of the group. Many parents/guardians highlighted the perceived benefits that their child had gained from LEGO^® based therapy (e.g. improved communication, behaviour, patience or confidence), but some clarified that these improvements had not translated into behavioural changes at home. A small proportion of parents/guardians said that their child would have liked to continue with LEGO^® based therapy in the longer-term, and that this might help develop their skills further, or lead to changes at home. A small proportion of parents/guardians highlighted that they lacked insight into their child’s experience of the intervention and had received no feedback from school regarding this.

Primary school participants

A summary of the assessment of acceptability of LEGO^® based therapy to primary school facilitators and parents/guardians of primary school-age autistic CYP in the trial is available in Appendix 4, Table 61 and Figure 12. As with all participants, this shows that acceptability of the intervention was again high, with overall median acceptability scores of 5 (range 4–5) and 4 (range 3–5) for primary facilitators and parents/guardians, respectively. Similarly, acceptability to facilitators was generally higher than to parents/guardians, and this difference was statistically significant (p = 0.000).

In relation to specific TFA constructs, primary school facilitators rated all of these positively, with median scores of 4 or 5 on each. All facilitators positively rated ‘affective attitude’ and ‘self-efficacy’, with high scores on all other constructs, and, overall, 94% rated the acceptability of LEGO^® based therapy positively. Again, the lowest rating from facilitators was for ‘perceived effectiveness on behaviour’, which at 71% is still positive.

For parents/guardians of primary school-age CYP, only ‘affective attitude’ and ‘opportunity costs’ received the maximum median score of 5, with 94% and 85% rating this positively, respectively. Overall, parents’ acceptability score was lower, at 76%, and the difference between parents/guardians and facilitators was statistically significant (p = 0.000). With regards to specific TFA constructs, ‘burden’ (42%), ‘intervention coherence’ (41%) and ‘perceived effectiveness on behaviour’ (56%) were rated least positively by parents. There were statistically significant differences between facilitator and parent/guardian ratings of acceptability on burden, intervention coherence, self-efficacy (p = 0.000), perceived effectiveness on behaviour (p = 0.05), academic confidence (p = 0.025), communication skills (p = 0.009), social skills (p = 0.001) and, correspondingly, overall perceived effectiveness (p = 0.000).

Secondary school participants

A summary of the assessment of acceptability of LEGO^® based therapy to secondary school facilitators and parents/guardians of autistic CYP of secondary school age in the trial is available in Appendix 4, Table 62 and Figure 13. As with the whole group of participants, this shows that acceptability of the intervention was again high, with overall median acceptability scores of 5 (range 4–5) and 4 (range 3–5) for secondary facilitators and parents/guardians, respectively. Again, acceptability to facilitators was generally higher than to parents/guardians, and the difference was statistically significant (p = 0.000).

For specific TFA constructs, secondary school facilitators rated all of these positively, with median scores of 4 or 5 on each. Again, 100% of facilitators positively rated ‘affective attitude’ and ‘self-efficacy’, with high scores on all other constructs, and, overall, 94% rated the acceptability of LEGO^® based therapy positively. Again, the lowest rating from facilitators was for ‘perceived effectiveness on academic confidence’, which is still high at 83%.

For parents/guardians of CYP of secondary school age, each of the TFA constructs was rated with a median score of 4. Overall, parents’ acceptability score was lower, at 68%, and the difference between parents/guardians and facilitators was statistically significant (p = 0.000). With regards to specific TFA constructs, ‘perceived effectiveness on social skills’ (60%) and ‘perceived effectiveness on behaviour’ (52%) were rated least positively by parents/guardians, with many other constructs rated positively by only 64% of participants. There were statistically significant differences between facilitator and parent/guardian ratings of acceptability on burden, intervention coherence (p = 0.005), self-efficacy (p = 0.006), perceived effectiveness on behaviour (p = 0.04), general perceived effectiveness (p = 0.03) and, correspondingly, overall perceived effectiveness (p = 0.000).

Comparison between primary and secondary facilitators

Although primary school facilitators generally rate the acceptability of LEGO^® based therapy more positively, with median scores of 5 (range 4–5) compared with 4 (range 3–5) for secondary school facilitators, this difference was not statistically significant overall or on any of the individual constructs (see Appendix 4, Table 63 and Figure 14).

Comparison between parents

Parents/guardians of CYP of secondary school age generally rated LEGO^® based therapy more positively than parents/guardians of CYP of primary school age, but this difference was not statistically significant overall: both groups had median scores of 4 (range 3–5) (see Appendix 4, Table 64 and Figure 15). There were statistically significant differences between the group ratings of ‘burden’ (p = 0.05) and ‘intervention coherence’ (p = 0.026).

Participants

A total of 16 facilitators from the trial were consented to the qualitative acceptability substudy. The median duration of the interviews was 40 minutes (range 22–61 minutes). The majority of participants were working in primary schools (12/16). A total of 10 participants were working as TAs, four as SENCOs and two as teachers. All participants had previous experience of working with CYP with ASD and other SEN. None was actively delivering LEGO^® based therapy (i.e. a study exclusion criterion) but nine indicated that they were familiar with it. Nine facilitators were from Leeds, two from Sheffield and five from York.

Analysis

On familiarisation with the data transcribed from the telephone interviews, we developed a list of preliminary codes. After several iterations, the final framework was developed: this included 26 codes, which could be distilled under eight categories (see Appendix 4, Table 65). Following this, indexing of the data set was undertaken and charts were produced to summarise the content of each category and code (and subcode). To help with interpretation of the data, these were then mapped to the TFA. 55 This mapping between categories and the seven TFA constructs is shown in Appendix 4, Table 66. The presentation of the themes is structured around the seven TFA constructs.

Affective attitude

Most of the facilitator participants explained that in their opinion the LEGO^® based therapy was a positive experience for the CYP in their groups because these CYP were already familiar with and interested in LEGO. As LEGO^® based therapy was generally seen as easy to explain to CYP and enjoyable to deliver, it was also viewed positively by facilitators:

I think what made it fairly easy to talk to them about it, is because they all had a knowledge and an interest in LEGO prior to the programme.

L/40/I2

Most facilitators said that their group had enjoyed participating in the intervention, and some CYP had asked to carry on with this when the 12 sessions had finished. Some participants found that the CYP in their groups were very excited and enthusiastic about taking part in LEGO^® based therapy and would talk about the intervention in-between sessions in the classroom and the playground. Some participants also gave examples of other CYP who had not been directly involved in LEGO^® based therapy wanting to take part:

I know mine have already come up to me and said ‘Are we doing it next year?’ They are so enthused by it.

S/28/I1

One of the secondary school facilitators highlighted reservations at their school, where it had initially been thought that LEGO would not be suitable for older CYP, but they found that the CYP enjoyed the sessions and found them to be ‘light relief’ from their usual timetable.

Overall, facilitators explained that they felt that LEGO^® based therapy ran smoothly as an intervention and was successful; however, certain common challenges arose. For example, some CYP struggled to participate effectively in the groups if they were accomplished LEGO builders at home. This worsened when sets were perceived to be too small or simple, and the facilitators suggested that these CYP found it difficult to stick to their roles without rushing ahead with building:

Some of them had already made some of the models that were identified in the programme, so it was really guiding them because they had a tendency to run ahead.

L/40/I2

By contrast, other facilitators commented that, for groups of younger CYP, some of the LEGO sets were too complex, and the smaller pieces and intricate building could be difficult to manage for CYP with comparably poorer fine motor skills.

Burden

Most of the facilitators in this substudy did not view LEGO^® based therapy as a major burden. Rather, the intervention groups were described as fitting relatively easily into their existing work schedule and compatible with current working practices in their schools. The study team remained in equipoise, but some schools made the decision to continue LEGO^® based therapy after the trial. These were started only after our trial was competed. For example:

… and when we look at the broad spectrum of kids we’ve got here in this school, a lot of them are now requesting it as something that can go on pupil passbooks and things that this child has accessed this as an intervention. So you know, staff are coming to me and saying ‘do you think you could fit this child in?’ and I think it’s just seen now in school as something that we deliver.

L/35/I1

Overall, most facilitators explained that LEGO^® based therapy had already been introduced more widely with other CYP in their schools (i.e. following completion of the intervention as part of the trial) or there were discussions about the possibility of expanding this. New groups had been set up at varying times (e.g. lunchtime or after school) to fit into the school schedule:

For the school, I think it’s certainly valued by me as I’m the SENCO, and for me it’s been really useful. It’s a skill now that I have, it’s a skill that I can now pass onto other people.

L/05/I1

. . . now we run LEGO club every lunchtime. So we have, I don’t remember the numbers, 50 or so children now engaged in LEGO therapy at some point during the term.

L/35/I1

In these schools, facilitators appeared hopeful that LEGO^® based therapy could become another potential intervention available to help autistic CYP. On a practical level, in terms of delivery of the intervention, it was recognised that having timetabled protected time and the availability of a suitable space made integration into practice easier. The facilitators also noted that a private room or space in school to conduct LEGO^® based therapy was important, as was using the same consistent, familiar space for every session. On occasion LEGO^® based therapy sessions had to be held in a different location, and this change of venue had disrupted the sessions and CYP were ‘thrown’ when in an unfamiliar room. These challenges of ensuring the spatial continuity in schools, which are rapidly changing environments with a high demand for private space, were recognised.

Despite the positive comments about being able to fit the intervention into the school timetable, most facilitators said that the largest burden of delivering LEGO^® based therapy in school was finding the time, and scheduling the sessions on a regular basis into timetables (i.e. accommodating the needs and preferences of CYP and teachers) relative to the facilitator’s timetable:

It’s time, because for me to have done this, I have had to take away an hour of my time every week, so yeah, it’s time.

S/28/I1

It’s primarily been the time, my time really and that’s always going to have a cost, but that’s the only thing really.

Y/21/I1

In some schools, it had been necessary to reorganise staff timetables to cover for those conducting LEGO^® based therapy; however, most facilitators said that this change to staffing had a minimal impact and was not perceived as a major issue:

The class teacher is one member of staff down, so yes, it is, it is quite an impact and it’s hard to maintain an intervention every single week without a break in a school because schools are fluid places and they are flexible and things happen.

L/33/I1

The only difference really had been a re-jigging really of staffing on the playground.

L/35/I1

One facilitator described how they had run sessions after school and others had run the intervention groups during lunchtime. One facilitator felt that 12 sessions was a lot to fit in, and that fewer sessions may have been easier to accommodate into the school calendar.

Ethicality

All facilitators in this substudy explained that LEGO^® based therapy fits well with their own professional values about what helps autistic CYP. Most agreed that the structure of the intervention is well tailored, in their opinion, to the needs of autistic CYP, helping them to communicate and socialise with others. Some facilitators also said that they appreciated gaining the new skill of delivering LEGO^® based therapy, although we are aware that they were expressing these views before the study results were known:

Part of my job is to make these children feel successful and feel valued. And if I can use something like this, I know this will be successful, then why not? It’s gonna work, it’s gonna be a winner.

L/26/I1

The majority said that the intervention fitted well into their schools, as the main aim is to be as inclusive as possible for all CYP, including those with ASD and other CYP on the SEN register. Many suggested that their perceptions of benefits from LEGO^® based therapy have helped to increase acceptance of the intervention in their school. For example, one secondary school participant explained that SEN can sometimes be seen as less of a priority, but the perceived benefits from LEGO^® based therapy (e.g. helping to control anger and increase resilience) have been valued by the school, and it was seen as worthwhile:

So it fits with the goals, we are an inclusive school, we want everyone to achieve their best and we are, we often do a lot of different types of interventions in order to try and achieve those.

L/05/I1

I think [it fits in with school values] because we’re just striving to be a fully inclusive school and to give kids on the SEND [special educational needs and disabilities] register just the greatest opportunity they can.

L/35/I1

Intervention coherence

Opportunity costs

Most facilitators reflected that the benefits that they perceived from LEGO^® based therapy far outweighed any resource implications or opportunity costs (e.g. staff time and planning) experienced in the delivery of the intervention. As already outlined, the main issue faced by the facilitators was timetabling and the juggling act required to deliver the intervention at the same time each week. Some facilitators identified a small cost to their own time, and a need to be more organised, but most felt that their workload was unaffected and that other staff helped them to fit the sessions in when there was a timetable clash (e.g. with school trips):

The teachers have never had a problem with them actually missing that lesson, as they know what those children come back to class like, and they’re very happy that it’s worth it and they’re not losing, they’re not missing anything.

L/43/I1

Related to this, most facilitators noted that, for LEGO^® based therapy to continue in their schools, there would be an initial financial cost for the LEGO sets, which were perceived as expensive, and therefore the intervention must be seen as ‘worth it’ to justify this. Some of the schools had received donations and voluntary contributions from parents/guardians or planned to fundraise, but one school said that this large initial cost was a barrier to the continuation of LEGO^® based therapy beyond the trial:

The other cost would be and that would be the real sticking point is, you know, having to buy the LEGO kits. That is, and that would have to be thought through carefully.

L/33/I1

Some facilitators noted that there was a key staff member (e.g. head teacher, inclusion manager or SENCO) who demonstrated clear enthusiasm for LEGO^® based therapy and supported them doing this work during the trial:

I was allocated the time to do it, my school was behind me and they gave me the hours that I needed to it.

Y/06/I1

Perceived effectiveness

Self-efficacy

Most facilitators felt that the training for LEGO^® based therapy provided as part of the trial adequately prepared them to deliver the manualised intervention in their schools. Most said that they enjoyed the training and felt well equipped to deliver the intervention afterwards. Some noted that receiving the group training format was helpful because they could ‘bounce ideas’ off one another:

It was a really, really good training session day. I really enjoyed it. I felt, after I came, I couldn’t wait to start.

L/07/I1

A couple of the facilitators stated that they would have preferred a little more training on conflict resolution during the sessions, and some, although they may have been nervous to start LEGO^® based therapy after the training, discovered that this potential problem reduced after the first session:

I would say, you know, maybe in the training, go over a little bit more the issues you may encounter and how to deal with those.

L/33/I1

Nearly all the facilitators agreed that TAs should have the right set of skills to be able to deliver LEGO^® based therapy in practice (i.e. outside the trial, if adopted by schools). Most suggested that patience is the most important skill needed, and it is important to give CYP enough time to solve their own problems before intervening. One facilitator explained that class teachers may not be as well equipped to deliver LEGO^® based therapy as TAs and that facilitators, therefore, should be chosen carefully:

I think they need to be very patient and understanding. And I think they need to know their children and know what makes them tick.

L/46/I1

Related to this, the facilitators emphasised the importance of group dynamics to the success of the sessions. They explained that relationships between CYP were important to consider when assembling the group. Similarly, facilitators’ prior knowledge about, and positive relationships with, the CYP were helpful to ensure smooth and effective delivery of and participation in LEGO^® based therapy:

You’ve got to really think about the group and the dynamics of your group.

L/26/I1

Most facilitators in this substudy did not and would not choose to make adaptations to the delivery of LEGO^® based therapy; they felt that the intervention already worked well. Some made or suggested small changes, including running the sessions twice weekly (when there was a time constraint such as completion before half-term), giving out more certificates on a regular basis and using the LEGO^® based therapy techniques in other settings (e.g. cooking lessons):

I don’t think I would adapt anything, because for me, it flowed really well.

L/07/I1

Twice a week did work really well. It might have felt a long time between sessions if we’d only done it once a week.

Y/21/I1

Cost-effectiveness analysis

Quality of life, using both the EQ-5D-Y and the CHU-9D, was rated higher by those in the intervention arm than those in the control arm. These differences are relatively small, but it is notable that both the parent-completed and the CYP-completed questionnaires showed the same positive differences. In addition, in the sensitivity analysis, when plotting the incremental cost of LEGO^® based therapy against the incremental QALYs, the majority of bootstrapped incremental cost–effect pairs were in the fourth quadrant (i.e. the bottom right-hand quadrant). This indicates that, after taking uncertainty into consideration and adjusting for the imbalanced baseline utility, the differences in EQ-5D-Y and CHU-9D scores remain positive, and the costs of LEGO^® based therapy remain lower than for usual support.

The results of the health economics analysis show a marginal increase in QALYs and reduction in school-based intervention costs [such as Social Stories™ (Carol Gray) and visual schedules] when comparing the intervention arm with the control arm. The results also show a reduction in overall costs (albeit with wide CIs), particularly through attendance at CAMHS. The lowered CAMHS costs found in the intervention therapy arm appear to be related more to a small number of CYP receiving high levels of high-tariff CAMHS support in the control arm than to a general drop across the whole group. This is a relatively positive finding in that many of the autistic CYP had evidence of co-occurring emotional and behavioural problems at baseline and CYP in both trial arms seemed to be receiving approximately similar support from CAMHS. It was also found that LEGO^® based therapy resulted in a reduction in the mean frequency of CAMHS support received compared with baseline. Given the high threshold for receiving CAMHS support, it is likely that the CYP requiring support had more severe needs. Such findings should be treated with caution and need to be further investigated. The reduction in education costs may be related to the fact that CYP in a LEGO^® based therapy intervention might be less likely to be put forward for other interventions (e.g. the SULP). 41 There was evidence, however, that schools reported a range of other interventions being received by autistic CYP in both arms of the trial, including Social Stories, visual schedules and one-to-one mentoring. The reduction in wider costs, including CAMHS costs, is worthy of comment. It could be that the clinical improvement or prevention of clinical deterioration lead to less likelihood of referral to CAMHS or a discharge from CAMHS if a child or young person is already in their care. It is important, however, to consider alternative explanations. For example, if the intervention improves comorbid problems, this might lead to a reduced likelihood of referral to CAMHS (e.g. for behavioural problems)88 or the child or young person may stop being seen by CAMHS. 89 We explored this by looking at changes in the subscales of the SDQ and found no statistically significant difference between arms for any of the five subscales. If a school-based intervention is improving a child or young person’s social skills and is viewed positively by teachers and parents/guardians, then this may reduce the likelihood for referral to CAMHS90 because of reduced levels of teacher and parent/guardian concern. However, research to date suggests that higher parental anxiety tends to reduce referral rates, not increase them, in many circumstances. 91 Other possible reasons for reduced costs might include a belief by a parent/guardian that an active intervention is happening and so for this reason taking part in another intervention at the same time is not necessary.

Qualitative analysis

Overall, there was a high level of intervention fidelity for the delivery of LEGO^® based therapy.

The qualitative analysis suggested that LEGO^® based therapy was very acceptable to facilitators and was also viewed positively by CYP and the wider school staff.

Facilitators believed that the LEGO^® based therapy training prepared them well for delivery during the trial and did not identify any crucially necessary adaptations. Facilitators commonly reported observing improvements in both communication and social skills for CYP in intervention sessions. There were mixed findings about whether or not the CYP generalised these skills. Some described improvements being taken into other settings (e.g. into the playground), whereas others reported that some CYP appeared to struggle to generalise these skills. Previous authors have described problems with the generalisation of social skills learning in autistic CYP. 92 However, it is also clear from research that autistic CYP are not a homogeneous group and many autistic CYP can generalise learning in the right conditions, with some CYP finding this easier than others. 93,94 Identifying the components of effective interventions that create the conditions that both accommodate the differing needs of autistic CYP and promote skills development is an important area for future research.

For a new school-based intervention, training generally ran smoothly, with the intervention being delivered to a high level of fidelity and not too resource intensive for busy school environments in terms of training, supervision or delivery.

Subgroup and secondary analyses

Turning to the subgroup analysis to explore the moderating effects of ASD symptom severity (as measured using the SCQ), the results showed no clear pattern (see Table 15). There is no convincing evidence of a differential effect for CYP with different levels of ASD symptomatology. However, when an interaction term between subgroups and intervention was added to the primary outcome model for the ITT population it was borderline statistically significant (p = 0.08), and an additional unplanned sensitivity analysis looking at the subgroup effect for selected secondary outcomes was equivocal. How might we understand these findings? Taking an active role in LEGO^® based therapy is reported to provide a structured, play-based opportunity for autistic CYP to develop socioemotional learning skills at a developmentally appropriate pace. 37

Post hoc exploratory analyses were conducted to explore the potential for observational analysis of the trial data in a subsequent substudy. These analyses show that the effect size is potentially greater for certain subgroups and where there is a critical mass of autistic participants in therapy groups. These additional analyses are not intended to be confirmatory and are not used in drawing the overall conclusion of a modest borderline significant effect. In this post hoc unplanned analysis we were able to look into the effect of school size and found some limited evidence that schools with six or more autistic CYP had a larger difference (8.23 points, 95% CI 1.79 to 14.68 points) than schools with fewer than six autistic CYP (1.45 points, 95% CI –2.95 to 5.85 points). Interestingly, in these schools the improvement approached the MCID. Our findings suggest a greater intervention effect when delivery adhered more closely to the protocol and there were two or more CYP with a diagnosis of ASD in each therapy session, as shown by the unplanned per-protocol analysis, which shows an effect of 4.56 points (95% CI 0.24 to 8.87 points; p = 0.038). These results should be read with caution but may provide direction for future research planning. It is instructive to consider in more detail the issue of numbers of autistic CYP in the school. There are a number of possible mechanisms. One of these is that schools with a large number of autistic CYP may have more staff training opportunities and larger numbers of skilled and trained teachers and TAs with knowledge and experience of ASD. This may mean that, when we were training teachers and TAs to deliver LEGO^® based therapy, some may have already had considerable knowledge of ASD and practical experience of working with autistic CYP and that this aided their ability to deliver the intervention. Another possibility is that, because of the larger numbers of autistic CYP in the school, it was more likely that more than one autistic CYP would be in the group. There is some evidence in adult ASD research that peer-to-peer communication95 and more relaxed social interaction96,97 may be facilitated by autistic people being together.

The capacity to gain new social skills and integrate social and emotional learning in everyday social interactions are tasks integral to successful social functioning and relationships. 98,99 There is evidence that in autistic individuals these skills can continue to develop through childhood and into adulthood. 100 This skills development over time was perhaps also supported by our secondary post hoc analysis finding that the beneficial effect was higher for secondary school CYP, at 7.45 points (95% CI –0.93 to 15.83 points). However, we found no meaningful differences in loneliness scores between groups from baseline to either 20 or 52 weeks. These subgroup analyses were not pre-planned and the results should therefore be treated with caution. Although these findings are modest, they are interesting given that, prior to the study, during our development and consultation work, peer reviewers questioned whether or not secondary school-aged CYP would enjoy or engage in LEGO^® based therapy. There are common misconceptions that LEGO is a toy for younger children. In fact, LEGO bricks appeal to a very wide developmental and chronological age range, including CYP and adults, as evidenced by very active adult fans of LEGO (AFOL) communities [e.g. there are AFOL groups with over 20,000 members on Facebook (Metaverse Platforms, Inc., Menlo Park, CA, USA) at the time of writing]. In fact, attrition rates and qualitative interviewing showed strong engagement and acceptability from secondary school participants. This is in line with the fact that the intervention was originally designed for CYP without learning disability, with a good level of language and in line with previous unpublished research exploring LEGO^® based therapy that showed positive outcomes for children aged 9–17 years. 101

It is common practice in the UK for autistic CYP to be placed in mainstream schools, particularly if they do not have a learning disability. As a result, the population of autistic CYP is widely spread rather than concentrated in SEN schools. During recruitment we found that a number of primary schools had only one child or young person in the school who was eligible for the trial. This is reflected in the 51% of sessions in primary schools that were run with just one autistic child or young person. Simple infographic data collected from the additional non-autistic CYP showed that, although some were neurotypically developing CYP, others were being assessed for ASD or had other social communication difficulties. We were able to carry out an unplanned compliance analysis using participants in groups with an average of two or more autistic CYP in their sessions as the definition of compliance, and this provided borderline significant evidence (p = 0.06, 95% CI –0.2 to 11.5 points) that greater numbers of autistic CYP in sessions were associated with a higher estimated effect on the main outcome measure (i.e. 5.7 points). Indeed, the CI for effect for compliers had an upper bound above the MCID (at 11.5 points). These signals of possible positive effect may be interpreted in a number of ways. There is a suggestion that LEGO^® based therapy is more effective where there are more autistic CYP in a group and in secondary schools in general. There is also a signal that this may also apply where there are more autistic pupils in the school. It may be that there is some confounding taking place, with, for example, larger schools (e.g. secondary schools with ASD units) contributing to all these factors. Larger schools may have larger numbers of staff trained in ASD or be better resourced to support autistic children. One approach to help disentangle this and get maximum use of the data collected is a subsequent in-depth causal impact analysis.

Long-term follow-up: main outcome measure at 52 weeks

Because the long-term outcome point was 52 weeks post randomisation, many of the CYP had moved into a new school year and therefore had a new teacher. Associated teachers were required to know the child or young person well, and this may be particularly relevant in this population owing to the importance of familiarity for autistic CYP; however, this was often difficult where CYP had moved schools and teachers were approached for follow-up after knowing the child or young person for only a short period. Having a more detailed knowledge about a child or young person could influence the rating of the outcome measures in a number of ways. On the one hand, it could be that teachers with longer-term knowledge of the child or young person are able to note subtle changes in the child or young person’s behaviours including, for example, social skills and interactions in the classroom setting. On the other hand, it could be that existing teachers were influenced by a bias in the knowledge that the child or young person had been in an intervention, introducing performance bias. However, it was also common for new teachers to be aware of the child or young person’s previous participation in the intervention, meaning that they may have been susceptible to the same bias. In addition, asking new teachers to complete the questionnaires about their associated child or young person, although necessary for long-term follow-up in a pragmatic school-based study, may have introduced some differences in perspective and styles of questionnaire completion to those of the initial teacher at previous time points. This raises the wider question of what might be the ‘best’ and/or most accurate way to record change in behaviours and skills over time in an education setting. For example, blinded outcome measures from RAs who observe social skills for short periods of time may be reliable in an efficacy evaluation research study but may not reflect overall everyday social competence, especially given the wide variety and variability of behaviours often seen in autistic CYP in different settings, on different days and in different social contexts. 102–104 Blinded observations usually take the form of 40-minute classroom or playground observations and thus may not reflect overall, everyday functioning. Teachers or parents/guardians who know the CYP well provide a different perspective; each may be able to rate CYP on complex social skills, but they are also likely to be influenced by a range of factors (e.g. a desire to see improvement, experience in home or education settings), which again may introduce bias. This is an ongoing concern for methodologists in all complex intervention evaluation efficacy and effectiveness studies of this nature.

It appears that no significant harm was caused by the intervention, based on the limited number of reports of AEs and SAEs throughout the trial. Minimal AEs were recorded and included instances of emotional distress experienced by some CYP participating in groups. Each instance fell under the category detailed in the study’s protocol as possible expected AEs based on the trial population and was reviewed by the study’s DMEC. It is important for those intending to train in and deliver LEGO^® based therapy to be aware of and constructively manage these types of expected AEs. Three SAEs were reported but these were deemed unrelated to the intervention.

Strengths of the trial

To the best of our knowledge, the I-SOCIALISE trial is the first large-scale RCT evaluating the clinical effectiveness and cost-effectiveness of LEGO^® based therapy in a UK education context. It was a pragmatic trial run in the real world, with teachers and TAs delivering the intervention. In our opinion, this study has successfully remedied a number of methodological limitations of previous trials of this intervention.

Recruitment to the trial was smooth and to time and target, with a very low attrition rate of 12% for the primary outcome at 20 weeks post randomisation (15% lost to follow-up across all outcome measures), which is in contrast to the drop-out rates of 28–75% reported in a meta-analytic review of child mental health studies. 122 Because strong predictors of drop-out rates found in the meta-analytic review122 were treatment and therapist factors, this was a positive finding.

The trial design incorporated a number of features to reduce bias. Cluster randomisation was used to reduce the possibility of contamination, with staff who were supporting either the intervention or control arm located in different schools to avoid them influencing each other. All trial participants were consented prior to informing the schools which arm the school had been allocated to in order to avoid selection bias. Frequent communication was maintained with both the intervention and control arm schools throughout the duration of the trial (i.e. the recruitment, intervention and follow-up phases) to reduce attrition bias. Control schools were offered the opportunity to receive training in LEGO^® based therapy after the end of their involvement in the trial to reduce both demotivation in the control schools, which could impact on data completion, and any risk of contamination if school staff sought out intervention training in LEGO^® based therapy from alternative sources during their involvement. Schools were made aware that the study had not yet concluded and results on the intervention’s efficacy and any potential harms were not yet available, but that this information would be shared once the study had completed and the findings had been approved.

Previous research undertaken in a school setting yielded less generalisable results owing to the low sample size. 42 The present study, however, was able to assess LEGO^® based therapy with a relatively large sample (for studies of ASD) using well-recognised measures that assess social competence. The nested feasibility study included in this trial allowed us to first assess the feasibility of carrying out the trial and to discover any hinderances to implementation. The trial was specifically designed around the mainstream school environment with an awareness of the needs of education staff, timetabling and other logistical constraints and the needs of CYP (including those with and without ASD). PPI work aided the careful design of the study for this setting. Recruitment and intervention delivery were undertaken with school term times in mind, meaning that recruitment and follow-up periods were built around the long school summer holidays and a shorter amount of time for schools to deliver their 12 sessions of LEGO^® based therapy when applicable. The trial team also anticipated that some mainstream schools (especially primary schools) were likely to have a relatively small number of eligible CYP and included in the trial protocol alternative arrangements for ‘additional CYP’ to join intervention groups where necessary.

The trial assessed both the clinical effectiveness and the cost-effectiveness of delivering LEGO^® based therapy to autistic CYP in mainstream school settings and included longer-term follow-up. The findings will be of interest to NHS health and social care providers, local authorities, families and community professionals including school staff members.

The fidelity and acceptability work show good completion rates for the self-report measures (facilitators, 93%; parents, 77%) and the fidelity checklists (overall, 99%). The use of Sekhon et al. ’s55 TFA to structure the data collection and analysis in the different aspects of the acceptability substudy was helpful in unifying understanding of a broad concept of acceptability. As the use of the TFA increases in the future, this data set will provide valuable information and enable direct comparisons with the acceptability of other, similar, interventions.

A further strength of this research was the continuous involvement of PPI representatives and experts in LEGO^® based therapy from the initial design of the research and the study set-up through to the end of the trial. The Young Dynamos and the NAS contributed to the initial design of the trial alongside a range of school and local authority staff, and provided the study team with invaluable feedback about implementation that allowed the trial to run smoothly throughout recruitment and follow-up stages, ultimately leading to over-recruitment and positive qualitative feedback from participants. Co-applicant Gina Gomez de la Cuesta, an expert in the field, was able to help create an intervention delivery manual (see Report Supplementary Material 1) and training package for all school staff members who were to run LEGO^® based therapy sessions (i.e. the facilitators). She also delivered the training package to local authorities in each region and to the study team members who went on to successfully train the school staff, who in turn achieved high rates of treatment fidelity.

The training model used for the trial allowed us to train > 100 schools in the delivery of LEGO^® based therapy. Our findings suggest that this model is appropriate for roll-out by local authorities. The study has also improved networking across local authorities, schools and parent groups, with the development of parent support groups and teacher-/school-based networks. This consolidated the expertise of the COMIC research team, which specialises in trialling CYP-friendly and CYP-oriented interventions for CYP with mental health difficulties.

Limitations of the trial

Although the recruited sample is representative of the gender balance in ASD123 and representative of the regions from which we recruited in terms of participant ethnicities,124 it is not representative of the cultural diversity of the UK as a whole. Recruitment was slightly above target and, owing to missing primary outcome data at baseline or at 20 weeks, the overall effective attrition rate was 13%. This was lower than allowed for in the sample size calculations (which was 16%). However, at 20 weeks, the attrition rate was slightly higher in the control arm (at 18%) than in the intervention arm (at 9%). There was a tendency for the control arm to have slightly lower return rates for all measures, although overall return rates were high, at 85%.