Impact of Social Stories on social and emotional health of autism spectrum primary school children: the ASSSIST2 RCT with economic evaluation

Autism spectrum condition

Around 1 in 57 (1.76%) children in England are on the autism spectrum. 2 Autism spectrum condition (ASC) is a lifelong neurodevelopmental condition characterised by differences in social communication and interaction and restricted, repetitive patterns of behaviour and interests. 3 Autism is also associated with strengths, such as some individuals having excellent attention and memory for detail and a strong drive to detect patterns. 4 Throughout this report, we will refer to ‘autistic children’ or ‘children on the autism spectrum’ to reflect the general preference for identity-first language rather than person-first language in the autistic community5 and amongst our public involvement representatives.

Children on the autism spectrum experience a higher prevalence of mental health problems than typically developing children, including anxiety and low mood. 6 Many children on the autism spectrum experience anxiety in social settings and feelings of frustration. 7 The International Society for Autism Research (INSAR) has recently highlighted that more research evaluating early interventions for autistic children is needed to ensure practitioners and policy-makers have robust data on intervention effectiveness and implementation to secure optimal outcomes for children on the autism spectrum. 8

Supporting children on the autism spectrum in primary school

Children on the autism spectrum often struggle socially and academically in educational settings, which may be due to differences in their experiences of social situations, differences in social communication and experiences of anxiety. 9 The educational environment can present many challenges for a child on the autism spectrum,10 such as busy classroom environments, frequent changes in school routines and difficulties navigating peer relationships. Given these challenges, the school environment can be difficult for autistic children to follow and engage with. 9 Autistic children can also have difficulty understanding the point of view, thoughts or feelings of someone else and have difficulties with the social use of language which can make forming and maintaining peer relationships challenging. 11 Children on the autism spectrum may struggle to manage their compulsions, sensory sensitivities and preoccupations within the school routine if reasonable adjustments are not made to account for these differences. Educational professionals face many demands on their time and may not be able to focus enough on the child’s needs for them to achieve their full educational potential.

Little is known about the best ways to support children on the autism spectrum12 in primary school, and more information is needed about what training is required for staff to feel confident supporting these children. Current interventions designed to support children on the autism spectrum can be time-consuming, needing involvement of outside experts. For example, specialist social skills support is delivered within Child and Adolescent Mental Health Services (CAMHS). It is important that educational professionals recognise the social challenges of children with autism,11 and there is a need for low-cost, child-friendly and evidence-based interventions that can be delivered in community settings to support children on the autism spectrum with the aim of potentially reducing the need for specialist referral elsewhere. A systematic review of school-based social skills interventions for autistic children suggested that these can be effective in improving social outcomes for students on the autism spectrum. 11 However, further randomised controlled trials (RCTs) are needed to further understand which interventions are most clinically effective and cost-effective. 12

Social Stories™ intervention

One intervention that attempts to alleviate these social difficulties while not being intrusive, time-consuming or requiring extensive involvement of outside experts is Carol Gray’s Social Stories. 9 Social Stories are a highly personalised intervention aiming to share accurate, meaningful information about a particular goal or topic that the child needs help with in a positive and reassuring way. 9 Social Stories can be written and delivered by both parents and professionals in a range of settings and represent a less time-consuming and intrusive intervention than other more intensive interventions. They are typically better suited to primary-aged children (age 4–11 years),13 though they are also sometimes used with older children.

Social Stories are short stories which describe a social situation or skill to help children on the autism spectrum to understand a situation applicable to the child more easily. They write a child into their own personal story about themselves to help them learn new social information. They are commonly used to enable children to understand socially expected behaviours and norms. Social Stories are defined by 10 criteria which guide story development as detailed below. 14

1. The Social Story Goal. Authors follow a defined process to share accurate information using content, format and voice that is descriptive, meaningful and physically, socially and emotionally safe for the Audience.

2. Two-Step Discovery. Authors gather information to (1) improve their understanding of the Audience in relation to a situation, skill or concept and (2) identify the topic and focus of each Story/Article. At least 50% of all Social Stories applaud achievements.

3. Three Parts and a Title. A Social Story/Article has a title and introduction that clearly identifies the topic, a body that adds detail and a conclusion that reinforces and summarises the information.

4. FOURmat. The Social Story format is tailored to the individual abilities, attention span, learning style and – whenever possible – talents and/or interests of its Audience.

5. Five Factors Define Voice and Vocabulary. A Social Story/Article has a patient and supportive ‘voice’ and vocabulary that is defined by five factors. These factors are: (1) First- or Third-Person Perspective; (2) Past, Present and/or Future Tense; (3) Positive and Patient Tone; (4) Literal Accuracy and (5) Accurate Meaning.

6. Six Questions Guide Story Development. A Social Story answers relevant ‘wh’ questions that describe the context, including place (WHERE), time-related information (WHEN), relevant people (WHO), important cues (WHAT), basic activities, behaviours or statements (HOW) and the reasons or rationale behind them (WHY).

7. Seven is About Sentences. A Social Story is comprised of Descriptive Sentences, as well as optional Coaching Sentences. Descriptive Sentences accurately describe relevant aspects of context, including external and internal factors, while adhering to all applicable Social Story Criteria.

8. A GR-EIGHT Formula. One Formula ensures that every Social Story describes more than directs.

9. Nine to Refine. A story draft is always reviewed and revised if necessary to ensure that it meets all defining Social Story criteria.

10. Ten Guides to Implementation. The Ten Guides to Implementation ensure that the Goal that guides Story/Article development is also evident in its use. They are: (1) Plan for Comprehension; (2) Plan Story Support; (3) Plan Story Review; (4) Plan a Positive Introduction; (5) Monitor; (6) Organise the Stories; (7) Mix and Match to Build Concepts; (8) Story Reruns and Sequels to Tie Past, Present and Future; (9) Recycle Instruction into Applause; and (10) Stay Current on Social Story Research and Updates.

Their original designer, Carol Gray, believes these criteria are the hallmark of their success. The criteria guide story development to ensure an overall patient supportive quality and relevant content that is descriptive, meaningful and safe for the audience. These stories are written in a specific way using a variety of defined sentence types and a formula for the ratio of sentences in a social story. Using this formula and framework suggested by Carol Gray, Social Stories can be a flexible intervention that can be individualised to different social situations.

The imperative to tailor these stories is particularly important for helping a child on the autism spectrum, as every autistic child has different strengths and weaknesses. 4 When deciding on the goal for a Social Story, it is important to consider: ‘What is the challenge or problem facing the child?’, ‘What do we know about why this is happening from the child’s perspective?’, ‘What does the child and young person need help with?’ and ‘What positive outcome would we like to see for the child?’. Individuals are asked to consider the child’s individual needs, including their current known empathy/socio-emotional skills, their ability to ‘get the gist’ of social situations, their communication and imagination skills and how they engage with their sensory environment. Understanding the child’s preoccupations, routines and how stress, anxiety or frustration emerge in various situations is also important so that the child’s needs and perspectives can be considered within a Social Story. The goal that is set for a Social Story should be positive for the child and specific, measurable, achievable, realistic and time-limited (also known as a SMART goal). Social Stories can also be used to coach adults around the child about how to respond to an autistic child in certain situations and can help adults understand the child’s perspective so that they are better able to support the child.

Previous case studies have suggested that following exposure to tailored Social Stories, children on the autism spectrum have shown improvements in mealtime skills,15 making independent choices and playing appropriately,16 reducing anxiety,17 supporting improved communication,18 managing and reducing frustrations18 and reducing behaviours that challenge. 19–21 Three systematic reviews have examined the effectiveness of Social Stories for supporting children on the autism spectrum. 22–24 As reported in Marshall et al.,13 these reviews indicate an overall positive effect of Social Stories on a number of social and behavioural outcomes in individual case studies. Wright et al.’s24 systematic review included any study that used a standardised, numerical measure of outcomes but also included non-standardised, numerical measures. For single-case designs, studies had to report repeated measurements of the target behaviour to be eligible for inclusion. Outcomes explored in the reviewed literature included social abilities and awareness, communication, restricted behaviours, life skills, emotional development and sustained attention. In addition to case studies, Wright et al.’s24 systematic review included seven between-group studies, four of which were RCTs. 25–28 The interventions included in these randomised studies were often delivered over a much shorter time frame than was typically used in single-case designs, and the interventions were standard ones, rather than individually tailored to each child, meaning they did not fulfil Carol Gray’s criteria. 24 The authors highlighted that the included literature was vulnerable to selection and reporting bias and was largely US-based. 24

Based on the recommendation from the systematic reviews22–24 that further well-designed large-scale RCTs were needed in this area, Wright et al. 13 conducted a feasibility RCT to assess the feasibility of recruiting children, parents and educational professionals to a trial of Social Stories and assess retention, the appropriateness of outcome measures and any barriers to delivering training and the intervention in mainstream primary and secondary schools. This trial focused on writing and delivering individualised Social Stories that followed the Carol Gray criteria within the school setting. 13 The results suggested that a future trial would be feasible to conduct and could inform the policy and practice of using Social Stories in primary schools. It was not recommended to include secondary schools in the full trial, as qualitative work revealed that the intervention was not viewed as appropriate for secondary-aged children due to being too simplistic and more difficult to implement. The feasibility results were in line with previous research that suggested that it is possible to train tier-one professionals, for example, teachers and teaching assistants (TAs), to develop and use Social Stories tailored to a child. 29 As previously described, studies examining the use of Social Stories have yielded mostly positive results but have largely been single-case studies. Despite the lack of rigorous evidence, numerous schools and families are already accessing Social Stories training and delivering the intervention with children, and therefore a fully powered RCT is timely. Schools have limited resources and limited access to specialist practitioner interventions, and therefore it is important that interventions such as Social Stories undergo robust evaluation. If they are found to be clinically effective and cost-effective, they can be delivered within schools on a day-to-day basis.

Rationale

Individuals on the autism spectrum require varying levels of support from different services (such as the NHS and charitable organisations), as the condition can have a widespread and persistent impact on quality of life (QoL), relationships, employment and standards of living. 30 An evaluation of the economic cost of autism in the UK30 revealed that the annual costs of supporting children on the autism spectrum were estimated to be £2.7 billion each year, and for adults, these costs were £25 billion each year. Given this significant cost and limited funding available for specialist support in schools31 and for other services, there is a need for low-cost, child-friendly and evidence-based interventions that can be delivered in community settings such as schools to support children on the autism spectrum, with the aim of potentially reducing the need for specialist referral elsewhere.

The current work aims to address this gap by exploring whether one promising intervention (Social Stories) could be effective in supporting children on the autism spectrum in primary school. This RCT follows a feasibility study,13 which explored the acceptability of running a trial examining Social Story use with 50 children across 37 primary and secondary mainstream schools. This demonstrated a high degree of acceptability with young people, families and schools. This main trial now seeks to assess the clinical and cost effectiveness of Social Stories, addressing the lack of fully powered RCTs in this area.

Study aims

As outlined in the study protocol,1 the aim of this trial was to assess whether Social Stories alongside care as usual is clinically effective and cost-effective in improving child social impairment, reducing behaviours that challenge and improving social and emotional health in children on the autism spectrum in primary schools when compared with care as usual alone.

Trial design

This trial was a multisite pragmatic cluster RCT comparing Social Stories and care as usual with a control group receiving care as usual alone. Care as usual being defined as the existing support routinely provided for a child on the autism spectrum from educational and health services such as specialist autism teacher teams, mental health teams or other associated professionals. The trial included an internal pilot, an economic evaluation and a nested process evaluation. The full trial protocol is published online via the National Institute for Health and Care Research (NIHR) Journals Library (https://fundingawards.nihr.ac.uk/award/16/111/91).

Patient and public involvement

The development of the Autism Spectrum Social Stories In Schools Trial 2 (ASSSIST-2) trial was supported by three core patient and public involvement (PPI) members, a parent of an autistic child, a specialist teacher and a member of the National Autistic Society. Two of these representatives were named co-applicants for the funding awarded from the NIHR to deliver the trial, and all were involved in the trial from its conception. In addition to these core members, a local youth council, which also worked with members of the trial team on a previous feasibility study of Social Stories,24 reviewed and contributed to the design of the present trial. This included shaping information leaflets and giving advice on young people’s preferences regarding the research study.

Furthermore, the study team utilised findings from the qualitative interviews completed during the previous feasibility study which included young people with autism, teachers and parents/carers. 24 The findings from these interviews informed the use of questionnaires, the presentation and content of materials such as information sheets and consent forms, the logicising of intervention delivery and the best means of contacting participants. These interviews, in conjunction with our PPI representatives, also informed the training manual used during this study. 9 Advice was also provided from parents, teachers and educational psychologists regarding more general aspects of trial delivery within education settings. Both the Trial Steering Committee (TSC) and Trial Management Group (TMG) included PPI representatives who were able to contribute to discussions around the trial throughout its implementation.

A PPI event was held at the end of the trial to disseminate the key findings to representatives from key sectors, specifically educational settings, parents, charities, mental health services and the local authority. Advice and guidance were sought with regard to dissemination activities, including the appropriateness of language and terminology around autism as well as key considerations for any roll-out and future research avenues.

Regulatory approvals and research governance

Initially, the trial opened as a non-NHS trial, recruiting schools and families directly with no NHS involvement. Ethics approval for the trial was obtained from the Health Sciences Research Governance Committee embedded within the Department of Health Sciences at the University of York on 2 July 2018. At a later stage of the research, ethical approval was sought and awarded from the Health Research Authority (HRA) on 24 July 2019 and the North East – York Research Ethics Committee (REC) on 23 July 2019 (REC reference number 19/NE/0237) to allow opening of an additional recruitment stream via the NHS. Substantial and non-substantial amendments to approve changes to the protocol and study documentation were submitted to the REC, HRA, the Department of Health Sciences Research Governance Committee and each NHS site’s research and development office as required during this trial. The trial sponsor was Leeds and York Partnership NHS Foundation Trust.

Trial registration

The trial was assigned the International Standard Randomised Controlled Trial Number (ISRCTN) ISRCTN11634810 on 23 April 2019.

Setting

The Social Stories intervention was delivered within educational settings by educational professionals. Educational settings included both mainstream primary schools and Special Educational Needs (SEN) schools. Parents/caregivers were invited to receive Social Stories training and were encouraged to utilise Social Stories within the home environment, though this was not considered part of the intervention. Parents were specifically advised not to deliver the story written for the purposes of the trial at home as the trial sought to evaluate the intervention through a school-based delivery model. No data were collected on the use of Social Stories at home.

Recruitment

Recruitment was on a rolling basis through the recruitment period, with schools being randomised as and when all pre-randomisation criteria had been fulfilled, that is, all relevant parties had provided consent and all baseline data had been collected. No additional children could be recruited from a school once it had been randomised. Participants were recruited via the following recruitment pathways.

Consenting participants

Participation in the ASSSIST-2 trial was entirely voluntary. There were three types of participants involved in the trial, namely, educational professionals, parents/caregivers and the children themselves. The consent process varied slightly depending on whether participants joined the trial prior to or during the coronavirus disease 2019 (COVID-19) pandemic.

Prior to the COVID-19 pandemic, consent was typically obtained in person during face-to-face visits to the child’s school by a trained research assistant (RA). Information sheets were always distributed to the relevant parties at least 24 hours prior to consent being taken. During the COVID-19 pandemic, consent forms were posted to participants and an appointment was scheduled for a RA to go through the form with the participants either over the telephone or via videoconferencing.

Educational professionals and parents/caregivers provided their own consent. Due to the age of the children, parents/caregivers provided consent on behalf of the children, though verbal assent was gained if possible. During the consent process, participants could opt in to being contacted to participate in the qualitative interviews for the process evaluation. They could also opt in to be sent details about future relevant research.

Sample size

The primary end point was that the teacher completed the Social Responsiveness Scale, Second Edition (SRS-2) T-score at 6 months. Within the pilot data, outcomes were measured at 6 and 16 weeks. The correlation between baseline and 6 weeks [r = 0.67, 95% confidence interval (CI) 0.44 to 0.80] was lower than that at 16 weeks (r = 0.83, 95% CI 0.68 to 0.91) for the pilot data. To be conservative, the lower 95% confidence limit was chosen for the lowest correlation between baseline and follow-up that we observed within our pilot data (r = 0.44). Assuming a difference of 3 points, standard deviation (SD) = 7, 5% alpha, 90% power, average cluster size 1.35, ICC = 0.34, correlation = 0.44 and 25% attrition, a total sample size of 278 was required.

Given the COVID-19 pandemic and subsequent closure of schools during this period, recruitment fell behind target. Recruitment began in mid-November 2018, and at the time of the first UK lockdown on 23 March 2020, 194 children (across 67 schools) had been screened, of which 134 (51 schools) had been randomised (53.8% of the final sample). The trial status was appraised by the TMG in October 2020, and it was decided that recruitment should end on 31 May 2021 with a total of 249 participants recruited. This was informed by the data collected to date and the impact on power was modelled. Keeping all sample size assumptions the same as the original outlined above, 249 participants would reduce the power from 90% to 86%. We also knew that the original assumption of the average cluster size of 1.35 was too low and this should be increased to 2.86 (the average school size), which would lead to a further reduction in power. As we had recent data to provide more accurate estimates of correlation between baseline and 6-month SRS-2 T-scores and attrition, we updated the sample size scenarios as outlined in Table 1. The estimate for correlation increased from 0.44 to 0.54, and attrition decreased from 25% to 23%. Unfortunately, we did not have enough data to get reliable estimates of the ICC; however, we had some information which may inform our choice of ICC further than that observed in the pilot trial. The ICC of the teacher SRS-2 T-scores at baseline was 0.09 (95% CI 0.00 to 0.24). Scenario 1 in Table 1 shows that 249 participants would provide 89% power, assuming the reduced estimate of the ICC (0.09) and other updated estimates of correlation (0.54), attrition (23%) and average cluster size (2.87). As the updated estimate of the ICC was not based on 6-month follow-up data, it is possible that our true ICC may be greater than this. For further assurance, we looked at the highest value of ICC we could have while still providing 80% power, and this would still be achieved with an ICC of 0.3, which is over three times > 0.09 and is greater than the upper confidence limit of the baseline ICC of 0.24. Evidence suggests that an ICC of 0.2 would be plausible in an educational setting,32 which gives us some confidence that a value of 0.3 would be conservative. Both updated scenarios provide at least 80% power, which gave us some assurance that ending recruitment with 249 participants would mean we had an adequate level of power even if the ICC were greater than expected. This change to the sample size was agreed by the trial oversight committees and funder.

Randomisation

Participants were entered on to a bespoke trial management system developed at YTU if they fulfilled the eligibility criteria, and their parent/caregiver and school staff provided written consent to take part. This system was used to record key identifiable data required to facilitate trial participation and data collection, as well as the various dates (e.g. baseline completion, randomisation, etc.) used to schedule and manage follow-up data collection. Randomisation was clustered at the school level; hence, all pupils within a school were allocated the same trial arm, Social Stories or usual care. Randomisation occurred after consent/assent and valid baseline data had been obtained from all participating families and educational professionals within a school. The randomisation process was completed by unblinded members of the trial team via the trial management system. Stratified blocked randomisation was used to allocate school clusters, with randomly varying block sizes (4, 6 and 8), and stratification by school type (SEN school or mainstream school) and the number of participating children within a school (≤ 5 or > 5 participating children). There were no stipulations regarding the minimum number of participants per school cluster (i.e. any school with ≥ 1 eligible/consented child with available baseline data was eligible for randomisation). Prior to COVID-19, schools were notified of their study allocation via telephone, e-mail and postal letter with parents/guardians receiving only a letter. Post COVID-19, all correspondence regarding allocation was via e-mail or telephone.

Blinding

Owing to the nature of the intervention, all participant types were unblinded to allocation. Research assistants collecting outcome data and the main trial statistician were blinded to study allocation until final data analysis. The Data Monitoring and Ethics Committee (DMEC) had access to unblinded data throughout the study. Any instances of unblinding were recorded using a bespoke form. In such instances, a substitute blinded RA collected participant data wherever possible.

Group allocation

Participant follow-up

All educational professionals (teachers and interventionists) and parents were followed up approximately 6 weeks and 6 months post randomisation. Interventionists were also asked to return session logs documenting their intervention (story-reading) sessions with the children. Follow-up data were collected in several ways.

Outcomes

Adverse event data

Although unlikely due to the nature of the intervention, possible harm as a result of the study was monitored according to YTU standard operation procedures (SOPs). Adverse events reported by individuals participating in the study were recorded using a bespoke Adverse Events Recording Form and assessed for seriousness. It was outlined in the trial protocol1 that any AE will be recorded as a serious adverse event (SAE) if it results in death, is life-threatening, prolongs or requires hospitalisation or results in disability or incapacity. All AEs were reported to the Chief Investigator, Professor Barry Wright, a child psychiatrist. The protocol specified that any SAEs related to the study should be reported to the study Sponsor, DMEC and TSC. The TSC and DMEC committees reviewed AE data at approximately 6-monthly intervals throughout the trial.

Participant withdrawal

Parents/carers and/or educational professionals could withdraw from the trial at any point during the course of the study. Parents/carers also had the option of fully withdrawing their child from the trial. Unless the parent/carer opted to fully withdraw the child from the trial, follow-up of the other participants associated with the child continued as usual where possible. If a participant indicated that they wanted to withdraw from the study, a member of the research team completed a change of circumstance form, and the new participant status was recorded on the trial management system.

Trial completion and exit

Participants completed the trial once they had completed the 6-month follow-up period post randomisation. At this stage, the educational professionals and parents/caregivers allocated to the usual care arm were offered Social Stories training. No further follow-up was required post training; however, support in story writing was available on demand. Participants (of any level) were considered to have exited the trial if they had withdrawn or lost to follow-up at the 6-month follow-up point. Parents/carers were offered a Love2Shop voucher to the value of £20 upon completion of the trial. These were delivered via post. Schools were offered £50 per participating child, paid in cash.

Data management

All information collected during the study was kept strictly confidential and stored on a secure password-protected server located at the University of York for the purposes of assisting in follow-ups during the study. All paper documents were stored securely, initially at the sponsor’s office (Leeds and York Partnership NHS Foundation Trust (LYPFT), throughout data collection prior to transferring the documents to the University of York, which occurred on a monthly basis.

Case Report Forms (CRFs) were initially checked for errors by the research team, and any queries were raised immediately with participants. CRFs were then logged on the YTU’s bespoke data management system and scanned using Cardiff Teleform. Original data sheets were securely stored at YTU. All data were collected and retained in accordance with the Data Protection Act 2018, the General Data Protection Regulation 2018 and YTU SOPs. All data will be archived for 10 years following the end of the study and then securely destroyed.

Data analysis

All outcomes were analysed after the trial had ended according to a pre-specified statistical analysis plan (SAP). Analyses were conducted using Stata version 17,38 following the principles of intention to treat (ITT), with available outcome data analysed according to the participants’ allocated group regardless of protocol deviations or non-compliance (unless otherwise stated). All statistical tests are two-sided tests of a point null hypothesis, and reported interval estimates are based on two-sided 95% CIs. The trial was designed and reported according to the Consolidated Standards of Reporting Trials (CONSORT) guidelines. 39 Participant withdrawals (number, type and timing) and follow-up response rates are presented overall and by allocation. Compliance data, as measured through session logs, are also summarised. All participant baseline data are summarised by trial arm and overall, with no formal statistical comparisons being undertaken. All continuous/quantitative data are reported in terms of their mean, SD, median, interquartile range, minimum and maximum, with categorical data reported in terms of frequencies and percentages.

Data collection schedule

To examine the possible impact of mistimed data collection (e.g. due to delays resulting from school holidays), analyses were conducted using only teacher-reported SRS-2 data that were collected within the data collection windows specified in the SAP, namely 2/+ 4 weeks for the 6-week follow-up and −4/+ 8 weeks for the 6-month follow-up.

Coronavirus pandemic impact analyses

Due to the impact of the COVID-19 pandemic (e.g. nationwide lockdowns and school closures), certain aspects of the trial were not delivered as intended. In particular, numerous participants had their follow-up data collection schedule disrupted (e.g. some month 6 follow-ups scheduled during summer 2020 could not be completed until September due to school closures) or received Social Stories from interventionists who were trained remotely (online) rather than in person. A number of pre-specified analyses were undertaken to investigate the possible impact of these disruptions on the effectiveness of the intervention.

To assess whether delays in the completion of the primary end point were associated with variation in treatment effect, we refitted the primary analysis model with an extra term denoting delayed outcome completion (binary – defined as the month 6 follow-up being completed more than 8 weeks after the planned follow-up date) and additional terms for all of the two- and three-way interactions between this variable, allocation and time point.

To investigate more broadly the extent to which disruption of follow-up was associated with variation in treatment effects, we refitted the primary analysis model with an additional term denoting whether or not the participant was due their month 6 follow-up during the school closures in the first UK lockdown (23 March 2020 to 3 September 2020) and additional terms for all of the two- and three-way interactions between this variable, allocation and time point.

Finally, to investigate the extent to which disruption of follow-up and online intervention delivery training were associated with variation in treatment effects, we refitted the primary analysis model, including an additional term denoting whether all available follow-up data for the primary outcome (at 6 weeks and 6 months) were provided/completed before/after the date of the start of the first UK lockdown (23 March 2020) and additional terms for all of the two- and three-way interactions between this variable, allocation and time point.

Subgroup analysis

Subgroup analyses were performed to explore the potential modifying effect of the following subgroups:

• The teacher’s preferred randomised group collected at baseline. Treatment preference was elicited using a VAS with a range (0, 100), where 0 indicates strong preference for usual care/support, 100 indicates strong preference for Social Stories and 50 indicates indifference/no preference. The participants were assigned to three preference subgroups based on the preference score indicated: (0, 50) = prefers usual care, (50) = no preference and (50, 100) = prefers Social Stories.

• Whether the child has been diagnosed with cognitive/intellectual problems or a learning disability as collected on the comorbidities CRF at baseline (Binary – Yes/No).

• Whether the child has been diagnosed with mental health or psychological problems as collected on the comorbidities CRF at baseline (Binary – Yes/No).

For each of the three baseline variables considered, the primary analysis model was augmented with additional terms for the main effect of the relevant subgroup, the two-way interactions between subgroup and allocation and subgroup and time point and the three-way interaction between subgroup, allocation and time point. The fitted models were used to obtain treatment effect estimates at 6 weeks and 6 months by subgroup, together with appropriate 95% CIs and p-values. As recommended by the literature, the subgroup analyses were restricted to the primary analysis, and subgroups were defined by baseline data, that is, data that are not dependent on the intervention. 43

Secondary analyses

The goal-based outcome score (6 weeks and 6 months), SRS-2 total score completed by parents (6 weeks and 6 months), RCADS total score (6 weeks and 6 months), EQ-5D-Y proxy VAS (6 weeks and 6 months) and Parental Stress Index (PSI) total stress score (6 weeks and 6 months) were analysed similarly to the primary outcome (i.e. using linear mixed-effect models with the same fixed and random effects and unstructured residual error covariance structure), but with the baseline score for the relevant secondary outcome included in the linear predictor in place of the baseline score for the primary outcome. The fitted models were used to estimate the between-group difference in expected score (for the relevant outcome) at 6 weeks and 6 months post randomisation together with appropriate 95% CIs and p-values.

Trial oversight

The conduct of the study was governed by three oversight committees:

• a TSC

• an independent DMEC

• a TMG.

These committees functioned in accordance with YTU SOPs. The DMEC and TSC were both independent from the sponsor. The TSC consisted of an independent chair, an independent subject specialist, an independent clinical academic, an independent statistician and a PPI representative. The DMEC consisted of an independent chair, an independent statistician and another independent member experienced in research with children and families. The TSC and DMEC met approximately every 6 months from the start of the trial. The TMG comprised co-applicants, members of the trial team (including the data manager), PPI representatives and the trial managers. Co-applicants and trial team members were invited as required, depending on their roles.

Internal pilot

The first 10 months of the trial following the start of recruitment served as the internal pilot. Clear stop/go criteria were established at the outset of the trial with the view of reviewing progress against these criteria before proceeding to the full trial. The stop/go criteria centred on the feasibility of recruitment (target 110 children), retention (target 44 final follow-ups complete) and safety outcomes (see Appendix 1, Table 45).

At the end of the 10-month period, the trial team reported to the oversight committees, the TSC and DMEC, who then critically reviewed the feasibility of continuing recruitment. The progress against the stop/go criteria was as follows.

Research questions

The research questions for the process evaluation were:

1. What does baseline practice in participating schools look like (in the control and intervention arms) in terms of interventions to support children on the autism spectrum in school?

2. To what extent do the schools and interventionists implementing the Social Stories intervention adhere to the intended delivery model (e.g. sessions delivered six times over 4 weeks)?

   1. What variability in implementation exists? Are there any barriers or adaptations?

   2. How well have the Social Stories been delivered, and how well have children on the autism spectrum and school staff engaged with it?

3. What are the views of specified stakeholders (interventionists, teachers and parents) about the implementation and effectiveness of the intervention during the trial period?

4. To explore the views of RAs and trainers about the goal-setting process as part of the Social Stories intervention and to describe any challenges faced.

5. To what extent do schools continue to deliver the intervention after the trial period?

   1. What are the barriers and facilitators to continuing to deliver the intervention?

Process evaluation methods

A full overview in relation to each of the process evaluation objectives is presented in Table 2.

Qualitative data

The Social Stories intervention is complex, as not only does it involve several interacting components (training, goal-setting, writing and delivering Social Stories), but it also requires tailoring to meet the needs of each child, setting and behaviour. The extent to which interventions are delivered as intended is a particular challenge when designing and evaluating pragmatic RCTs of complex interventions. 45–47 There is a wealth of evidence that has aimed to define the key components of intervention fidelity. 48 The Hasson framework44 is an updated version of a conceptual framework, which was originally developed by Carroll et al. 44 as a result of a critical review that synthesised existing research on intervention fidelity. Hasson proposes nine elements, as a way of conceptualising intervention fidelity: adherence, exposure or dose, quality of delivery, participant responsiveness, programme differentiation, intervention complexity, facilitation strategies, recruitment and context (see Appendix 1, Table 46). The Hasson framework44 has since been used to explore how qualitative findings can be used to understand intervention fidelity within a NIHR-funded pragmatic RCT that evaluated a multifaceted podiatry intervention trial. 46,49 In this process evaluation, we have adopted a similar approach, and by using the Hasson framework44 to underpin our analysis, we have addressed the aims of our qualitative study and provided an in-depth exploration of the key factors that affected the fidelity of the Social Stories intervention during the ASSSIST-2 trial. This approach has also enabled us to identify the key areas to consider prior to any potential future rollouts of the intervention.

The decision to use Hasson’s components of intervention fidelity as an analytical framework was based on early familiarisation with our qualitative data set during the early stages of data collection and analysis, where it became clear that many of the initial themes we had identified and indeed our research questions related to fidelity. This decision was also a pragmatic one due to the number of researchers involved in analysis, and we needed a framework which could be used consistently, broadly and flexibly (i.e. allowing new themes and/or subthemes to be identified) by the group, while at the same time allowing for an in-depth, deductive analysis of our qualitative data based on the overarching constructs proposed by Hasson to be undertaken.

Design

The qualitative study consisted of (1) interviews with interventionists who had received Social Stories training and delivered the intervention, (2) interviews with teachers who may have attended the training and were involved in completing outcome measures about the participating children, (3) interviews with parents who attended training and (4) focus groups with trainers to explore their experiences of delivering training and with the study RAs to explore their experiences of setting goals for the Social Stories with teachers and parents.

Sampling and participants

In the protocol,1 we specified that we would undertake semistructured interviews with a minimum total of 20 participants and would purposively select individuals from the cohort of trainers, interventionists and associated teachers who have been involved in the trial, as well as parents who attended training. To achieve maximum variation, we proposed selecting participants according to age, gender, time in profession and experience with ASC. The COVID-19 pandemic meant that we had to move from an in-person to a virtual training model. We therefore adapted our sampling frame and recruitment strategy to ensure that we selected participants according to whether they had received training online or in person, at both early and late stages of the transition to the online training model. To ensure further variation in our sampling, we also selected participants that represented schools and areas that were diverse geographically and in terms of the student population (e.g. school size and type) and setting (rural, urban). Data collection continued until a varied sample of teachers, interventionists and parents had been obtained50 (see Appendix 1, Table 47). This was assessed through monthly qualitative team meetings held between the RAs, the trial coordinator and lead qualitative researcher.

In total, 30 participants took part in the qualitative study, with 21 interviews (five teachers, seven interventionists and nine parents) and three focus groups (nine research team members) conducted. The 21 participants represented 15 schools throughout Yorkshire and the Humber. Schools were diverse in size (9 schools > 200 pupils, 6 schools < 200 pupils) and rurality (seven urban, eight rural). Three focus groups were conducted: one with trainers who delivered training (n = 4), one with RAs who were involved in goal-setting with parents and teachers when they could visit participants in person (prior to the COVID-19 pandemic) (n = 2) and one with RAs who were involved in goal-setting with parents and teachers during online meetings (during the COVID-19 pandemic) (n = 4). One RA participated in two focus groups as they had become unblinded to help deliver the Social Stories training.

Recruitment and consent

All ASSSIST-2 trial participants (interventionists, teachers, parents/carers) who expressed an interest in participating in a qualitative interview on the consent form and who had been involved in the intervention were eligible for participation in the qualitative study. Up until April 2021, eligible participants were approached to take part in the qualitative study after they had returned all 6-month follow-up data. This was to avoid the RAs who were conducting the interviews becoming unblinded. After April 2021, one RA became unblinded and was able to invite participants to take part in the qualitative study 3 months after randomisation (between the 6-week and 6-month follow-up points). Trial participants were recruited via e-mail which contained a participant information sheet (PIS) (see Report Supplementary Material 1) and consent form (see Report Supplementary Material 2). Recruitment e-mails also informed trial participants that a RA would contact them via telephone to find out if they would be willing to take part in the qualitative study. Once consent forms had been completed by participants and received by the research team, a date and time for the interview were arranged. If the interview took place via videoconference, the participants could choose whether or not they appeared on video and whether the video was recorded. For focus groups, RAs and trainers were contacted directly by the lead qualitative researcher via e-mail and were asked if they were willing to take part in a focus group. Prior to all interviews and focus groups, informed consent was obtained, and participants were assured anonymity and confidentiality and were given the opportunity to ask questions.

Data collection

All focus groups and interviews were semistructured and were conducted via the videoconferencing software Zoom or via telephone (depending on the participants’ preference) between August 2020 and December 2021. Interviews and focus groups ranged from 10 minutes to 1 hour 20 minutes in length. Four separate topic guides were developed for the interviews, and two separate topic guides were developed for the focus groups. Topic guides were used flexibly to ensure key topics were discussed, while allowing for probing by interviewers and for participants to voice issues they considered important (see Report Supplementary Material 5). Areas explored during interviews and focus groups broadly covered participants’ views and experiences of goal-setting, training, trial participation and the intervention. Interviews with interventionists also explored participants’ experiences of delivering the Social Stories intervention with the children. Interviews were conducted by six trained RAs. All had previous experience of working with children or individuals on the autism spectrum and had received Social Stories training from the study team. Given that the RAs were instrumental to the delivery of the trial and were involved in setting goals for participants, focus groups with RAs and trainers were conducted by a qualitative researcher who was not involved in the design of the ASSSIST-2 trial or its delivery (AS). Arabella Scantlebury is an experienced qualitative methodologist with expertise in using qualitative research alongside RCTs. She has no prior experience of Social Stories and joined the ASSSIST-2 team in October 2020. Analysis was primarily undertaken by the RAs and trial co-ordinator and overseen by AS. Arabella Scantlebury led the analysis of focus group data. No key differences were identified in the data collected or analysed by any of the researchers.

Process evaluation analysis

All interviews and focus groups were audio-recorded and transcribed verbatim prior to analysis. Initially, we developed a coding framework, which was informed by our topic guides and Hasson's conceptual framework of intervention fidelity. 44 This involved generating a ‘list’ of themes and subthemes that were mapped onto the nine components of Hasson’s conceptual framework. For instance, participant responsiveness (how participants respond to or engage with the intervention) included data relating to previous experiences of Social Stories, motivations to take part in the trial and views of the intervention. Data were then analysed according to this framework for thematic content, with our coding framework adapted as data collection and analysis progressed, reflecting our growing familiarity and understanding of our data. Lastly, to aid readability and to give a comprehensive overview of intervention fidelity during the ASSSIST-2 trial, quantitative data pertaining to intervention fidelity that was collected through questionnaires and screening logs were integrated and presented alongside qualitative data under the components of intervention fidelity as described by Hasson in the process evaluation results chapter (see Chapter 5).

Data analysis was undertaken alongside data collection, with monthly qualitative team meetings used to discuss and refine our coding framework and facilitate the interpretation of our data. All participants were allocated unique personal IDs to protect anonymity and confidentiality. Due to the number of researchers involved in data analysis who worked across different institutions, data were managed using Microsoft Word and Excel.

Effectiveness

Effectiveness of the health economics analysis was measured using EQ-5D-Y to calculate quality-adjusted life-years (QALYs). The EQ-5D-Y (3L proxy version)35 is a five-item generic preference-based measure of health-related QoL that allows a proxy person to complete the measure for the participant. The EQ-5D-Y was chosen for the following reasons. Firstly, the EQ-5D-Y has been shown to be a reliable and valid health-related QoL instrument for use in children and young people (CYP)51 and with children on the autism spectrum or with related conditions, such as attention deficit hyperactivity disorder (ADHD), speech/language disorder52 and functional disability. 53 Secondly, the EQ-5D-Y is commonly used in CYP with psychosocial problems, including our previously successfully delivered Social Stories feasibility study – ASSSIST. 12 Finally, the EQ-5D is the instrument suggested by the National Institute for Health and Care Excellence (NICE) for cost-utility analysis. 54 The measure allows for the comparison between groups across a wide range of comparable dimensions and, thus, can be used to calculate ICERs compared with the national willingness-to-pay (WTP) threshold of £20,000–30,000 per QALY gained.

In EQ-5D-Y, health of CYP can be classified on a 3-point scale (1, ‘no problems’; 2, ‘some problems’ and 3, ‘a lot of problems’) over five dimensions: mobility (walking about), looking after myself, doing unusual activities, having pain or discomfort and feeling worried, sad or unhappy. Additionally, the EQ-5D-Y includes a VAS where participants can indicate their overall health status from 0 (worst imaginable state) to 100 (best imaginable state). All questions refer to the health state ‘today’. The measured utilities at baseline and follow-up were further joined through time, and the area under the curve approach was used to calculate QALYs for further cost-utility analyses. 55

Cost of the Social Stories intervention

Costs of the Social Stories intervention included the cost of training and the cost of delivering the intervention. Training costs were measured by the time spent by the trainers. Costs per minute of time, based on salary costs, were allocated, and any travel costs, costs of preparation time and the cost of materials used for the training were also recorded, whereas costs associated with delivering the Social Stories intervention were collected using a bottom-up estimation of the time spent by professionals to plan and conduct sessions and to undertake any additional work. Tailored questionnaires for collecting the costs of training and delivering Social Stories were developed by the research team based on our LEGO^®-based therapy trial – I-SOCIALISE56 to capture the resource use in both Social Stories (trial) and usual care (control) arms.

Costs of service use

Service use information was collected using the tailored questionnaires (completed by the parent/guardian) to capture data on the use of the following services:

• community health services, including appointments with general practitioner (GP), nurse, child development centre, walk-in centre, social worker and family support worker

• mental health services, including appointments with psychiatrist, psychologist, CAMHS therapist and mental health nurse

• hospital services, including outpatient visits, inpatient admissions, accidents, emergency visits and urgent care centre visits

• school-based interventions/support provided by teachers, school nurses or school staff

• other, including medication and informal parental/guardian care (productivity cost).

Service use was multiplied by unit costs to arrive at a total cost in each arm. Unit costs of health and social service use were obtained from two public sources: the National Cost Collection 2019–2057 and the Unit Costs of Health and Social Care report produced by the Personal Social Services Research Unit (PSSRU). 58 The costs of medication were based on the most recent version of the Prescription Cost Analysis – England,59 and the costs of teachers were estimated based on the data reported by the Department for Education. 60 Privately paid services were estimated based on market prices, and the productivity costs were valued according to national average wage rates. Appendix 2, Tables 60–67, presents the summary of key unit costs used in the study.

Descriptive analysis

Total costs, including the intervention cost and service use costs, and QALYs were compared between the intervention and control groups at baseline and 6-month follow-up. Unadjusted costs and QALYs were presented using appropriate descriptive analyses.

Missing data

Based on the completed service use questionnaires, data were deemed as missing when all sections under a particular item were left blank. For instance, where one question had subquestions, all were deemed as missing when all were left blank. If one of these questions is answered, then the others will be assumed zero. For EQ-5D-Y, due to the structure of the questionnaire, the whole section is considered missing if any of the five questions is not answered. The missing data in service use and EQ-5D-Y were imputed using Rubin’s multiple imputation method. 61

Primary economic analysis

All the outcomes were analysed after the trial ended according to a pre-specified health economic analysis plan (HEAP). The primary analysis was a within-trial cost-utility analysis conducted from a societal perspective. Combining costs and QALYs, an incremental ICER of cost per QALY was calculated and against the WTP threshold of £20,000–30,000 per QALY gained to assess the cost-effectiveness of Social Stories compared to usual care in children with ASC. Discounting was not applied due to the short-term nature of the trial.

To adjust any imbalanced characteristics at baseline, regression models on an ITT basis were used to compare mean costs and QALYs. The regression analyses were controlled for age, gender, stratification factors (SEN status stratum: non-SEN or SEN and number of participants stratum: ≤ 5 or > 5), parent-completed SRS-2 score, baseline utility and baseline costs from the societal perspective. The specification of the model followed the approach recommended by Glick et al. ,55 which considers the distribution of the dependent variable as well as any correlation between the cost and QALY outcomes. The regression coefficients on treatment then represented the difference in mean cost and mean QALYs between groups.

To take uncertainty into consideration, a non-parametric bootstrap resampling method on the basis of 5000 iterations was used to produce CIs around the cost and QALY differences and ICER. The number of iterations was chosen because it was considered to be sufficient to generate robust estimates of standard errors62 and is widely used in trial-based cost-effectiveness analyses for mental health studies. 63,64

The outcomes of bootstrapped iterations were presented in the conventional form of a cost-effectiveness plane (CE plane) and a cost-effectiveness acceptability curve (CEAC). The CEAC presents the probability of the intervention being cost-effective over a range of WTP thresholds per QALY. 65 The higher the probability, the more likely it is that the treatment is cost-effective at the particular WTP threshold.

Sensitivity analyses

The following sensitivity analyses were conducted to test assumptions made in the primary analysis:

• To assess the impact of the missing data, a sensitivity analysis was carried out using the complete case analysis (CCA), whereby results are analysed only for those participants who had both the completed cost and outcome data at each time point.

• A sensitivity analysis was conducted based on the costs from the UK NHS and personal social services (PSS) perspective.

• A sensitivity analysis was conducted based on the costs from the NHS/PSS and education perspective.

Inclusion criteria

In the original trial design, we planned to only include primary schools in the trial, given that the evidence indicated that the intervention is best suited to this population. However, this study involved both mainstream and SEN schools. As SEN schools often have a different structure than mainstream schools, we identified that there may be occasions where a SEN school participant moved from year 6 to year 7 without there being a change to any practical or environmental circumstances likely to influence intervention delivery (i.e. they will be in the same school, same class and have the same teachers). As part of a substantial amendment approved by North East – York REC on 2 January 2020, we specified that if a participant in this situation wishes to be part of the trial, we would consent them to the trial as if they were to remain a primary school student throughout their participation in the research.

Baseline data collection

Due to the structure of the academic year and school holidays, as well as other unexpected delays, we found that there was sometimes a time lag between data being collected and randomisation. As part of a substantial amendment approved by North East – York REC on 1 July 2020, we amended the baseline data collection procedure to allow for baseline data to be recollected if significant time delays were observed between the date of data collection and randomisation so that no data were collected more than 3 months prior to the date of randomisation.

Recruitment

As part of a substantial amendment approved by North East – York REC on 1 July 2020, we were granted approval to expand our recruitment strategy to include social media. Following this approval, information about the study, including eligibility criteria, could be posted on the research teams’ social media accounts, specifically including The Child Oriented Mental Health Intervention Centre Twitter account (@COMICResearchUK), Facebook (COMIC Research) and the website (www.comic.org.uk). A further change to recruitment was later made via a non-substantial submitted on 17 August 2020. The purpose of this was to extend the pre-specified recruitment window to reflect an extension to the trial granted by the funder in light of the COVID-19 pandemic.

Follow-up procedures

Two changes were made to the follow-up procedures during the course of the trial. Firstly, clarifications were made to the follow-up procedure to formally document that where CRFs were not returned within 2 weeks of their due date, researchers would contact participants by e-mail, telephone and/or text (depending on participant contact preferences) a maximum of four times. For the 6-week follow-up, if no response was received after four attempted contacts, no further action was taken. For the 6-month follow-up, participants were considered lost to follow-up after four attempted contacts. These clarifications were accepted by North East – York REC on 1 July 2020.

Secondly, as part of the same amendment, due to the COVID-19 pandemic and associated social distancing guidelines, we received approval to amend the follow-up procedures from researcher-based data collection to allow for postal data collection. This meant that parents could choose whether to receive a postal questionnaire or to continue with researcher-supported data collection (via their preferred contact method of telephone or videoconferencing).

Consenting procedures

As a consequence of the COVID-19 pandemic and associated social distancing guidance, we were required to update our consenting procedures which were previously all undertaken in person. We amended our processes to allow for researchers to take consent remotely via telephone or videoconferencing. Participants were sent consent forms via post to complete during telephone/video calls, and these were returned to the trial team. This change was approved as part of a substantial amendment by North East – York REC on 1 July 2020.

Qualitative component

Revisions were made to the approved interview and focus group schedules, and a survey was introduced as part of the qualitative enquiry. These reflected only minor changes and were approved by North East – York REC on 1 July 2020. Final version copies of all associated paperwork from the qualitative work can be found in Report Supplementary Materials 3–6.

Randomisation procedures

Prior to the COVID-19 pandemic, the trial employed very strict timelines surrounding the randomisation process. Specifically, interventionists needed to be trained within 2 weeks post randomisation and following this, interventionists had a 4-week period to deliver the intervention before the 6-week CRF was due. This timeline meant that we had to book provisional training dates for all schools and then cancel those for schools allocated to usual care. Following the school closures associated with the pandemic and the limited time with which to randomise schools without needing to extend the trial into another school year, we proposed amending the randomisation procedure so that follow-ups were triggered by the date that schools were trained and could start delivering the intervention rather than by date of randomisation. This allowed us to randomise all schools and then schedule training only for those schools allocated to the intervention arm. This change was reviewed by all the trial oversight committees (TSC, DMEC, TMG) as well as the funder and was submitted as a non-substantial amendment on 22 February 2021.

Recruitment

Between November 2018 and May 2021, 295 children across 98 schools were screened for trial eligibility. Of these, 268 (90.8%) were found eligible according to the protocol-specified eligibility criteria, 26 (8.8%) were ineligible and 1 (0.3%) had unclear/unknown eligibility status (due to missing data on age). Of the 295 children screened, 292 (99.0%) had screening data available for analysis. A summary of the available eligibility data is given in Table 3. The other three children were consented and randomised and are therefore assumed to have been eligible. These participants are included in all subsequent analyses. Of the 26 children that were deemed ineligible, the reasons for ineligibility were (possibly more than one per participant): not attending a school in Yorkshire or the Humber (n = 1), the parent/carer reporting that they did not feel the child had behavioural problems at school (n = 16), took part in the ASSSIST feasibility study (n = 1) and use of Social Stories by the school in the current or preceding school term (n = 15).

Initial consent to trial participation was obtained for all 268 children who were eligible. Of these, 45 (16.8%) subsequently withdrew (or were withdrawn) from trial participation prior to being randomised. Hence, 223 children were eligible, consented and were not withdrawn prior to randomisation. Brief details for the pre-randomisation withdrawals are given in Table 4.

Between February 2019 and May 2021, 87 schools were randomised (44 to the intervention group and 43 to the control group), with a total of 249 children being randomised (129 to the intervention group and 120 to the control group). This figure includes the 223 eligible/consented children referred to above, but also 26 children that were ineligible according to the protocol-specified eligibility criteria (see above). Details of the school cluster sizes (at randomisation) by allocation and overall are given in Table 5. Plots of the cumulative number of clusters and participants randomised over the recruitment period are given in Figures 1 and 2 in the Report Supplementary Material 7. Overviews of the screening/recruitment activity (including baseline data collection) are given in Figures 1 and 2.

FIGURE 1.

Recruitment and teacher completed data collection. a, Eleven schools where all participants withdrew prior to randomisation; b, two participants with no teacher-completed baseline data.

FIGURE 2.

Recruitment and parent/carer completed data collection. a Eleven schools where all participants withdrew prior to randomisation.

Retention

Following randomisation, five (2.0%) participants were formally withdrawn from either certain aspects of the planned follow-up data collection (e.g. just parent-completed data collection) or all follow-up data collection (i.e. full withdrawal). A summary of post-randomisation withdrawals by allocation is given in Table 6. The completeness of both teacher- and parent-reported data collection (randomised participants only) is given in Table 7. The flow of the participants through the trial is illustrated in Figures 1 and 2 (teacher-completed data collection in Figure 1 and parent/carer-completed data collection in Figure 2).

From Table 7 and Figure 1, it is apparent that the proportion of participants retained for the purposes of teacher-completed follow-up was reasonably stable across the two time points (approximately 85% at both time points). However, it is clear from the figures in Table 7 that it was generally not the same group of participants missing teacher-completed data at each time point. For example, 238 (95.6%) participants had at least one available teacher-completed SRS-2 score (the primary outcome), meaning only 4.4% of participants had no follow-up data for the primary outcome, although 15.3% were missing primary end-point data (i.e. were missing primary outcome data at 6 months). There were some differences in the completeness of teacher-completed follow-up between the randomised groups. For example, at both week 6 and month 6, approximately 80% of participants allocated to the intervention group had teacher-completed SRS-2 data, compared with approximately 90% of participants allocated to the control. This difference is driven primarily by cluster-level loss to follow-up/non-response. For example, at month 6, 10 (22.7%) whole school clusters were lost to follow-up in the intervention group, compared with 5 (11.6%) in the control group.

For the parent/carer-completed data (see Table 7 and Figure 2), the proportion of participants retained diminished slightly between the week 6 and month 6 follow-ups (approximately 78% at week 6 vs. 74% at month 6). However, loss to follow-up was again not monotonic, meaning a larger proportion (approximately 86%) had parent-completed data for at least one follow-up time point. The completeness of the parent-completed follow-up data differed somewhat between randomised groups at week 6, with the proportion providing data at this time point being 5–10% lower in the intervention group than in the control group, although at month 6 there was little difference between groups. In addition, the proportion of participants with a baseline measurement and at least one follow-up measurement was similar for each of the parent-reported outcomes.

School-level baseline data

Of the 98 schools that provided screening and/or baseline data for at least one participant (or potential participant), 87 (88.8%) were randomised (44 schools to intervention and 43 to control). Randomisation was stratified by school SEN status (whether the school was mainstream or whether the school was a specific specialist provision setting, that is, non-SEN or SEN) and number of participants (≤ 5 or > 5). Summaries of the distribution of these stratification factors within randomised groups are given in Table 8, together with details regarding the distribution of school cluster sizes. It is evident from Table 8 that the stratification (and blocking) of the randomisation sequence achieved its intended goal of ensuring that the cluster allocations were (approximately) orthogonal with respect to the chosen stratification factors. The very slightly higher mean cluster size and additional allocation in the intervention group resulted in a slight imbalance in the number of participants in each group (129 in the intervention group and 120 in the control group); however, this imbalance would not be expected to have any material consequences for the analyses undertaken.

Participant-level baseline data

There were 266 children with available teacher-reported baseline data and 276 children with available parent-reported baseline data. However, 19 (7.1%) of the 266 children with available teacher-reported baseline data and 27 (9.8%) of the 276 children with available parent-reported baseline data were withdrawn prior to randomisation. Hence, 247 (99.2%) of the 249 randomised participants had available teacher-reported baseline data and 249 (100%) had available parent-reported baseline data. Participant-level summaries of the randomisation stratification factors are given (by allocation and overall) in Table 9. Parent-reported demographics of the randomised participants are given (by allocation and overall) in Table 10. Participant-level summaries of the baseline scores for the teacher- and parent-completed outcome measures are given (by allocation and overall) in Table 11. Plots illustrating the distribution of the observed baseline teacher-reported SRS-2 total raw and T-scores are given in Figures 3 and 4, respectively, in the Report Supplementary Material 7.

FIGURE 3.

Base-case cost-effectiveness plan of Social Stories compared with usual care (from the societal perspective).

FIGURE 4.

Cost-effectiveness acceptability curve of Social Stories compared with usual care.

The figures in Table 9 show the distribution of participants across the four randomisation strata was reasonably similar across the two groups, although naturally there are some departures from perfect balance at the individual level owing to the cluster randomised design. Again, this would not be expected to have any material influence on the analyses undertaken. The figures reported in Table 10 show that the participant-level demographic factors collected were well completed and that the distributions of these were similar across the groups as randomised. From Table 11, it is evident that the raw scores for the teacher-reported SRS-2 were slightly higher (‘worse’) in the intervention group than in the control group at baseline, and that this translated into a small difference in the T-scores (the primary outcome). However, in both cases, the discrepancies are well within what would be expected due to random error alone. This, coupled with the fact that randomisation was only completed after all baseline data were collected, suggests the observed differences are not the result of any systematic (i.e. non-random) processes or biases. Likewise, minor differences between groups are evident for the other teacher- and parent-reported secondary outcomes, but none are any more extreme than would be expected from purely random variation.

Parent, teacher and interventionist baseline demographics

Demographic data were also collected from participant’s parents/carers, teachers and interventionists associated with the participating children at baseline. Demographic data for the parents/carers of the 249 randomised participants are given in Table 12. Demographic data were available for 226 teachers associated with children that went on to be randomised and 184 interventionists associated with schools that went on to be randomised. These demographic data are summarised in Tables 13 and 14, respectively.

Descriptive analyses

The SRS-2 was collected from participating children’s teachers at baseline (prior to randomisation) and 6 weeks and 6 months post randomisation. The responses to the SRS-2 are used to derive a total score that is then converted to a T-score lying between 38 and 90 (inclusive). Of the 249 children randomised, 247 (99.2%) had available teacher-completed SRS-2 data, and for all but one of these participants, a valid total score could be derived. Hence, 246 (98.8%) participants had observed primary outcome data at baseline. Of the 249 randomised participants, 214 (85.9%) had observed primary outcome data at 6 weeks post randomisation, and 211 (84.7%) had observed primary outcome data at 6 months post randomisation (the primary end point). In total, 235 (94.4%) participants had observed primary outcome data at baseline and for at least one post-randomisation time point. Descriptive summaries of the primary outcome data at the three time points are given by allocation and overall in Table 17. Plots of the distribution of the primary outcome at each time point are given in Figures 5–7 in Report Supplementary Material 7, stratified by randomised group.

Primary analysis

Of the 87 schools (249 children) randomised, 83 (95.4%) schools [238 (95.6%) children] had at least one available follow-up measurement. However, three of these participants were missing a baseline score (due to there being ≥ 8 missing items for their baseline teacher-reported SRS-2 assessment, meaning a baseline T-score could not be derived). In the absence of pre-specified procedures for addressing missing baseline covariate data, these three participants were excluded from the primary analysis model. Hence, the primary analysis model included 235 (94.4%) participants across 83 school clusters. Point and 95% CI estimates for the differences in expected teacher-reported SRS-2 total T-score at 6 weeks and 6 months from the fitted model are given in Table 18 and are illustrated in Figure 8 in Report Supplementary Material 7. Basic diagnostic plots for the fitted primary analysis model are given in Figures 9 and 10 in Report Supplementary Material 7.

Conditional on all of the assumptions used to calculate them, the point estimates in Table 18 suggest the data are most compatible with the hypotheses positing that allocation to Social Stories causes small (around half the target difference used for planning) reductions in teacher-reported SRS-2 scores at 6 weeks and 6 months following randomisation. However, the p-values in Table 18 for the tests of H₀ : δ = 0 at each time point suggest that the observed differences would not be particularly surprising if allocation truly has no effect on these outcomes. Hence, the observed data are quite compatible with allocation to Social Stories, not causing any reduction in scores at 6 weeks and 6 months compared with control (again contingent on the manifold assumptions that the various estimates and test statistics are based on). The residual plots in Figures 12 and 13 in Report Supplementary Material 7 suggest departures from the key distributional assumptions of the fitted model are relatively limited and are unlikely to have any material effect on the substantive conclusions of the primary analysis. For example, removal of the extreme and potentially influential participant identified in Figure 9 results in fairly modest reductions of the estimated treatment effects at 6 weeks and 6 months [6 weeks −0.89 (95% CI −3.00 to 1.21); 6 months −1.59 (95% CI −4.15 to 0.96)], but the substantive conclusions remain relatively unaffected.

Teacher treatment preference

Teachers reported their treatment preference at baseline using a 0–100 VAS, where 0 indicates strong preference for usual care (for the relevant child) and 100 indicates strong preference for Social Stories (for the relevant child). For the purposes of the subgroup analyses reported here, the preference score was used to classify children into three categories: (1) Prefers usual care if < 50; (2) No preference if = 50 and (3) Prefers Social Stories if > 50. Summaries of the raw scores and the derived categories are given in Table 19. The distribution of observed preference scores is illustrated in Figure 11 in Report Supplementary Material 7.

Of the 249 participants that were randomised, 230 (92.4%) had observed primary outcome data (week 6 and/or month 6), complete baseline covariate data and complete baseline treatment preference data. To estimate treatment effects at 6 weeks and 6 months by treatment preference category (Prefers usual care, No preference, Prefers Social Stories), the primary analysis model was augmented with additional terms for the main effect of preference subgroup, the two-way interactions between preference and allocation and preference and time point and the three-way interaction between preference, allocation and time point.

The treatment effect estimates, 95% CIs and p-values for the 6 time points by subgroup strata are given in Table 20. While the point estimates do show some variation across these strata, this apparent variation is not any more than might be expected, even if there truly is no variation in treatment effect across these strata. The p-value for the likelihood ratio test of the full model (main effects and two/three-way interactions for allocation, time point and treatment preference) and the constrained model (main effects for allocation, time point and treatment preference and two-way interactions between allocation and time point and treatment preference and time point) is 0.75. Hence, overall, there is little evidence for any interaction between treatment preference and allocation. However, given the sparsity of the data, the absence of evidence for any interaction provides relatively little information concerning the presence/absence of treatment effect heterogeneity across treatment preference strata.

Diagnosed mental health or psychological problems

Parents of participating children reported whether their child had any diagnosed mental health and/or psychological problems at the time of the baseline data collection. A summary of the binary (Yes/No) responses is given in Table 19. The majority of participants (73.9%) had no diagnosed mental health problem, but a substantial minority (12.4%) had a diagnosis reported. Furthermore, 34 participants (13.7%) were missing information on mental health and/or psychological problem diagnoses. Of the 249 children randomised, 238 (95.6%) had observed primary outcome data at week 6 and/or month 6. Of these 238 children, 33 (13.9%) were missing information on mental health and/or psychological diagnoses, and three (1.3%) were missing other baseline covariate data. Hence, 202 participants had observed primary outcome data (week 6 and/or month 6), complete baseline covariate data and complete baseline mental health/psychological diagnoses data. The treatment effect estimates, 95% CIs and p-values are given by subgroup and time point in Table 20. These appear to be relatively stable across subgroups and time points, with a p-value for the relevant likelihood ratio test of interaction of 0.82. However, the 95% CIs contain a reasonably large range of values due to relatively sparse data in some strata. Hence, the observed data are not entirely at odds with hypotheses positing moderate treatment effect heterogeneity by mental health diagnosis subgroup.

Diagnosed cognitive problems and/or learning difficulties

Parents of participating children reported whether their child had any diagnosed cognitive problems and/or learning difficulties at the time of Table 19. The majority of participants (69.5%) had no diagnosed cognitive problem/learning difficulty, but a substantial minority (16.9%) had a diagnosis reported. Furthermore, 34 participants (13.7%) were missing information on diagnosed cognitive problems/learning difficulties.

The treatment effect estimates, 95% CIs and p-values are given in Table 20. There is some evidence that treatment effect at 6 weeks is larger in the cognitive problem/learning difficulty subgroup than in the subgroup without any cognitive problem/learning difficulty diagnoses. However, this apparent pattern disappears and is even reversed at 6 months, and the p-value for the relevant test of interaction is 0.27, suggesting this apparent variation in treatment effect would be relatively unsurprising even if there were truly no treatment effect heterogeneity present. However, the 95% CIs do contain a reasonably large range of values due to relatively sparse data in some strata. Hence, potentially important variations by subgroup cannot be completely ruled out.

Complier-average causal effect estimation

Of the 129 participants allocated to the intervention, 95 (73.6%) had available session log data, meaning 34 (26.4%) had no available session log data. Due to the clustered randomisation, participants allocated to the control group were assumed to have not received any Social Stories sessions. Social Stories delivery/attendance is reported by allocation in Table 21.

Of the 249 participants randomised, 214 (85.9%) had available primary outcome data at 6 weeks post randomisation and 211 (84.7%) had available primary outcome data at 6 months post randomisation. Of the 214 participants with available 6-week data, 23 (10.7%) were missing compliance status (allocated to the intervention but no session log data available) and 2 (0.9%) had complete compliance data but were missing a baseline score. Similarly, of the 211 participants with available 6-month outcome data, 24 (11.4%) were missing compliance status (allocated to the intervention but no session log data available) and 3 (1.4%) had complete compliance data but missing baseline score. Hence, 189 participants were included in the analysis to estimate the CACE at 6 weeks (79 intervention, 110 control) and 184 in the analysis to estimate the CACE at 6 months (76 intervention, 108 control). Point estimates and 95% CIs for the CACE at 6 weeks and 6 months are reported in Table 22, together with p-values for a two-sided test of the null hypothesis that treatment receipt has no effect on outcomes.

The estimates of the CACE in Table 22 suggest the data are quite compatible with the hypothesis that Social Stories cause a modest reduction in SRS-2 score at 6 weeks and 6 months for participants in the complier principal stratum (i.e. the latent subgroup of participants that would receive ≥ 6 Social Stories sessions if they were offered it). For example, the interval estimate and p-value for the CACE at 6 months suggest that the observed data would be reasonably surprising if the true effect of treatment was zero for participants in the complier principal stratum.

However, some caution is required in interpreting these estimates because a substantial number of intervention group participants have been excluded from the estimation due to having missing compliance data. It is quite plausible that those in the intervention group who are missing compliance data (and therefore excluded from the estimation of the CACE estimands) are not a random subset of the intervention group participants. In particular, they may be a subset of participants with poorer prognosis (treated or untreated) compared with those that have available compliance data. The summaries in Table 23 provide some evidence that this is the case. Hence, exclusion of these intervention participants is likely to have resulted in the intervention and control groups that are no longer exchangeable with respect to their potential outcomes, with the control group having poorer prognosis (on average) than the intervention group.

To assess the possible influence of the missing compliance data, two additional post hoc estimates of the CACE (at both 6 weeks and 6 months) were obtained. The first was undertaken assuming participants that were missing compliance data received < 6 Social Stories sessions (i.e. were not compliers), and a second assuming these participants received ≥ 6 sessions (i.e. were compliers). The results of these analyses are reported in Table 24. Clearly, these estimates have their limitations (e.g. uncertainty about the missing imputed values not adequately accounted for) and should also be interpreted with caution. However, they do suggest that the magnitudes of the CACE estimates presented in Table 22 are at least partly explained by selection bias arising from exclusion of intervention participants with missing compliance data. That said, the results in Table 24 again suggest the data are most compatible with hypotheses positing modest positive effects of treatment in the complier principal stratum (although hypotheses positing no effect or small negative effects are also reasonably compatible with the observed data).

Missing outcome data

The number of available/complete responses to the various teacher- and parent-reported outcomes is given in Table 25. For both the teacher- and parent-reported outcomes, the response rates were generally somewhat lower in the intervention group than in the control group.

The different patterns of primary outcome data missingness are reported by allocation in Appendix 1, Table 48. From this, it is evident that a considerably higher proportion of the control group had complete baseline and follow-up data for the primary outcome (65.9% of the intervention group had scores for the primary outcome at all three time points vs. 83.3% of the control group). Similarly, the proportion of participants included in the primary analysis (i.e. all those with primary outcome data available for at least one time point and complete baseline covariate data) was around 8% higher in the control group than in the intervention group (90.7% in the intervention group vs. 98.3% in the control group). Key baseline characteristics are reported by inclusion in/exclusion from the primary analysis in Appendix 1, Table 49. Given the relatively small number of participants excluded from the primary analysis, it is difficult to discern the extent to which these participants differ from those that were included. However, there are no immediately obvious patterns suggesting that those excluded differed greatly at baseline from those that were included.

The planned primary analysis assumes that the missing outcome data are MAR conditional on the observed baseline and outcome data included in the model. We believe this assumption to be reasonably plausible in this context given that missingness of primary outcome data is likely to be driven by teacher and/or school-level factors rather than being driven by the outcomes for these patients being better/worse than would be expected given their observed baseline/outcome data. However, to assess the robustness of the estimates obtained for the primary analysis to departures from MAR, we consider treatment effect estimates obtained assuming the missing outcome data exhibit various systematic departures from MAR. To do this, we performed a delta-based sensitivity analysis using a pattern mixture model, where the departures from MAR occur in each group individually (with MAR assumed for the other group) and both arms together. Treatment effect estimates for δ = −5 [i.e. expected value of the unobserved scores is 5 points less (‘better’) than the expected value of the observed scores conditional on the covariates in the substantive model] to δ = 5 [i.e. expected value of the unobserved scores is 5 points higher (‘worse’) than the expected value of the observed scores conditional on the covariates in the substantive model] in increments of 2.5 are provided in Table 26. The estimates for 6 weeks and 6 months are also illustrated in Figures 12 and 13 in Report Supplementary Material 7, respectively. The estimates for the 6-month outcome data in Table 26 show that the estimated treatment effect is reasonably stable across the range of departures from MAR considered. For instance, the point estimate of the treatment effect has the same sign across all of the missing not at random scenarios considered. Similar comments apply to the 6-week estimates.

Data collection timelines

Descriptive summaries of the timing of follow-up data collection are given by allocation in Tables 27 and 28 for the 6-week and 6-month outcome data, respectively. The distribution of the observed follow-up times is illustrated in Figures 14 and 15 in Report Supplementary Material 7 for the 6-week and 6-month follow-ups, respectively.

From Table 27 it is apparent that while half of the participants with available week 6 outcome data had these data collected between 6 and 10 weeks post randomisation, approximately 25% were collected at more than 10 weeks post randomisation, and approximately 10% at more than 20 weeks post randomisation. Delays in follow-up data collection at 6 weeks were slightly greater on average in the intervention arm, with a considerably higher proportion of week 6 follow-ups in the intervention arm being collected outside of the pre-specified window for completion (31.4% in the intervention group and 19.1% in the control group). Similarly, approximately half of the teacher follow-ups at 6 months took place between 26 and 34 weeks (in line with the 6 months + 8 weeks window specified), with around 25% being more than 34 weeks after randomisation and around 10% being more than 40 weeks after randomisation. In contrast with the week 6 follow-ups, the timing of the month 6 follow-ups was generally more similar between groups, with similar mean and median follow-up times and similar proportions taking place within the protocol-specified windows (24.0% in the intervention group and 26.9% in the control group). For both time points, there are seemingly no dramatic differences in outcome scores between those followed up within window and those followed up outside.

To examine the possible impact of mistimed data collection (e.g. due to delays resulting from school holidays), analyses were conducted using only primary outcome data that were collected within the data collection windows specified in the SAP, namely −2/+ 4 weeks for the 6-week follow-up and −4/+ 8 weeks for the 6-month follow-up. Treatment effect estimates at 6 weeks and 6 months from this analysis are reported in Table 29. The estimates in Table 29 are broadly similar to those obtained for the primary analysis (i.e. small point estimate in favour of the intervention with a range of small-to-moderate positive treatment effects or small negative treatment effects being compatible with the data), and the substantive conclusions are relatively unaffected. However, the point estimate at 6 weeks moved slightly further from zero, and the point estimate at 6 months moved slightly closer to zero, suggesting that any treatment effect wanes somewhat over time (in contrast to the primary analysis, which suggested a slight increase in treatment effect over time).

Coronavirus disease impact analyses

A crude overview of the timing of teacher-reported SRS-2 data collection is given in Figure 16 in Report Supplementary Material 7. From this, it is evident that numerous participants had follow-ups at 6 weeks delayed by 6 weeks or more, with some participants (particularly those randomised just prior to the pandemic) having week 6 follow-ups completed at around 6 months post randomisation. Similar delays are also evident for the 6-month follow-ups. Details regarding the timing for teacher follow-up data collection are provided by allocation in Table 30. This table clearly shows that while most participants had their week 6 data provided within −2/+ 4 weeks of the planned follow-up date and month 6 data provided within −4/+ 8 weeks of the planned follow-up date, there is a substantial minority whose follow-ups were severely delayed (approximately 12% and 14% at week 6 and month 6, respectively).

To assess whether delays in the completion of the primary outcome (at month 6) were associated with variation in treatment effect, we refitted the primary analysis model with an extra term denoting delayed outcome completion (binary – defined as the month 6 follow-up being completed more than 8 weeks after the planned follow-up date) and additional terms for all of the two- and three-way interactions between this variable, allocation and time point. The estimates from this augmented model are given in Table 31. From this table, there is some weak evidence that delayed primary outcome ascertainment/collection was associated with larger treatment effects at both 6 weeks and 6 months, particularly at the later time point.

To investigate the extent to which disruption of follow-up due to the pandemic was associated with variation in treatment effects, we refitted the primary analysis model with an additional term denoting whether or not the participant was due for their month 6 follow-up during the school closures in the first UK lockdown (23 March 2020 to 3 September 2020) and additional terms for all of the two- and three-way interactions between this variable, allocation and time point. The estimates from this augmented model are given in Table 31. From this table, there is some weak evidence that disruption to follow-up due to school closures is associated with variation in treatment effect at both 6 weeks and 6 months, particularly at the earlier time point. However, substantial uncertainty is evident (due to the relatively small effective sample size for the analyses undertaken/reported), and the data are also reasonably compatible with hypotheses that posit essentially zero effect at 6 weeks and 6 months in both strata.

To investigate the extent to which disruption of follow-up and online intervention delivery training were associated with variation in treatment effects, we refitted the primary analysis model, including an additional term denoting whether all available follow-up data for the primary outcome were provided/completed before/after the date of the start of the first UK lockdown (23 March 2020) and additional terms for all of the two- and three-way interactions between this variable, allocation and time point. The estimates from this augmented model are given in Table 31. From this table, there is some weak evidence that the effectiveness of Social Stories was greater at the month 6 time point among those who were followed up prior to the beginning of lockdown (little variation in treatment effect was apparent at week 6). However, the estimates for all between-group contrasts are quite imprecise (particularly for the pre-lockdown follow-up group), and a range of treatment effects across the different strata are plausible given the observed data (including the absence of any important treatment effects).

Teacher-reported outcomes

Goal-based outcome scores

To provide some context for the scores and analyses reported below, details of the types of goals set in consultation with the participants’ teachers at baseline are given in Table 33. These figures show that across both groups, the most common types of goals set related to prosocial behaviour and communication and understanding/processing emotions, as would be expected given the characteristics of the sample. Goals relating to understanding routines and dealing with transitions or changes in routines were also reasonably common and fairly evenly distributed between groups. Descriptive summaries of the teacher-reported goal-based outcome measures at each time point and the distribution of these responses are given by allocation in Table 51 in Appendix 1.

The teacher-reported goal-based outcome scores were analysed using a similar approach to the analysis of the primary outcome, except that the baseline scores for this outcome were conditioned on in place of the baseline score for the primary outcome. The estimated treatment effects at 6 weeks and 6 months are given in Table 32. Model assumptions were checked following similar approaches as used for the primary analysis. Despite the discrete nature of the outcome, these diagnostics did not identify severe departures from the distributional assumptions of the fitted model. However, the analysis of this outcome as a continuous outcome assumes that the gaps between adjacent values are all equal (i.e. that a shift between 0 and 1 reflects the same improvement in the underlying construct being measured as a shift between 9 and 10). This assumption is certainly doubtful, placing some limitations on the interpretation of the treatment effects estimated from this model. We therefore also conducted post hoc ordinal analyses of these data using a mixed-effect proportional odds model, with each time point modelled separately using the same fixed and random effects as before. For both models, there was little evidence that the effects of allocation deviated substantially from the proportional odds assumption. The estimated odds ratios for allocation at 6 weeks and 6 months were 1.97 (95% CI 1.05 to 3.70) and 2.08 (95% CI 1.13 to 3.84), respectively. To aid interpretation, we used the fitted model to derive absolute and relative differences in the probability of having a score strictly > 5 (goal met more than half of the time) at both the 6-week and 6-month time points, conditional on various covariate patterns X. These are reported in Tables 52 and 53 in Appendix 1.

The results in Table 32 suggest allocation to the intervention causes a small-to-moderate increase in the frequency that teachers perceived the goal set at baseline to have been met. The p-values indicate considerable discrepancy between the data and the test hypotheses, suggesting the observed data would be reasonably surprising if the entire set of assumptions used to compute the reported p-values were true (including the assumption that allocation to Social Stories had no effect on this outcome). Similarly, the estimates in Tables 52 and 53 in Appendix 1 suggest an overall shift towards higher scores (i.e. goals being met more of the time) in the intervention group, although some of the lower confidence limits suggest the data are somewhat compatible with hypotheses positing relatively small effects of allocation or none. Furthermore, the apparent treatment effects could be at least partly explained by the non-blinded nature of the outcome assessment. Overall, these results provide some weak evidence that allocation to Social Stories increases the frequency that children meet specific behavioural goals at 6 weeks and 6 months post randomisation.

Parent/carer-reported outcomes

Revised Children’s Anxiety and Depression Scale total score

Descriptive summaries of the RCADS total scores (parent-reported) at each time point are reported by allocation in Table 55 in Appendix 1. These scores were analysed using a similar approach to the analysis of the primary outcome, except that the baseline RCADS scores were conditioned on in place of the baseline measurement of the primary outcome. The estimated treatment effects at 6 weeks and 6 months are given in Table 32. Model assumptions were checked following similar approaches as used for the primary analysis. These diagnostics showed no severe departures from the distributional assumptions of the fitted model.

The estimates in Table 32 suggest allocation to Social Stories has a relatively limited impact on overall symptoms of anxiety and/or depression. The point estimates suggest the data are most compatible with hypotheses positing small negative effects of allocation to Social Stories (i.e. that allocation of Social Stories causes a slight increase in expected levels of anxiety/depression compared with allocation to care as usual). However, the interval estimates and p-values suggest that the observed data would not be hugely unexpected if allocation to Social Stories truly had no impact on this outcome at all or even reduced levels of anxiety and depression.

Parental Stress Index total stress score

Descriptive summaries of the PSI total stress scores (parent-reported) at each time point are reported by allocation in Table 56 in Appendix 1. These scores were analysed using a similar approach to the analysis of the primary outcome, except that the baseline PSI total stress scores were conditioned on instead of the baseline measurements of the primary outcome. The estimated treatment effects at 6 weeks and 6 months are given in Table 32. Model assumptions were checked following similar approaches as used for the primary analysis. These diagnostics showed no severe departures from the distributional assumptions of the fitted model.

The estimates in Table 32 suggest allocation to Social Stories has a relatively limited impact on overall stress due to problematic parent–child interaction and behaviours. The point estimates suggest the data are most compatible with hypotheses positing small positive effects of allocation to Social Stories (i.e. that allocation of Social Stories causes a slight decrease in expected levels of parental stress compared with allocation to care as usual). However, the interval estimates and p-values suggest that the observed differences at both time points would not be hugely unexpected if allocation to Social Stories truly had no impact on this outcome at all or even increased levels of parental stress.

European Quality of Life-5 Dimension Youth Questionnaire general health visual analogue scale

Descriptive summaries of the EQ-5D-Y general health VAS scores (parent-reported) at each time point are reported by allocation in Table 57 in Appendix 1. These scores were analysed using a similar approach to the analysis of the primary outcome, except that the baseline EQ-5D-Y VAS scores were conditioned instead of the baseline measurements of the primary outcome. The estimated treatment effects at 6 weeks and 6 months from the planned analysis model are given in Table 32. Model assumptions were checked following similar approaches as used for the primary analysis. These diagnostics revealed some departures from the distributional assumptions of the planned analysis model. In particular, ceiling effects were evident, resulting in heteroscedasticity and substantial departures from conditional normality. Hence, further semiparametric analyses that avoid some of the more restrictive parametric assumptions of the planned analysis model were conducted.

Mixed-effect ordinal logistic regression was used to model the EQ-5D-Y VAS outcome scores at week 6 and month 6 separately, with the same fixed and random effects as previously. The fitted models were used to estimate the difference in expected EQ-5D-Y VAS score between groups at week 6 and month 6, conditional on attendance at a non-SEN school with fewer than six participants, male gender and age and baseline score equal to the relevant sample means. The results of this post hoc analysis are reported in Table 58 in Appendix 1.

The estimates suggest allocation to Social Stories has a limited effect on general health (as proxy reported by participants’ parents/carers) at 6 weeks and 6 months post randomisation. The estimates from the planned analysis model suggest that the data are most compatible with hypotheses positing the intervention causes a very slight reduction in expected score compared with the control. However, the data are also seemingly quite compatible with hypotheses positing no treatment effect and even small positive treatment effects in favour of Social Stories. Despite the reversal of direction at 6 weeks, the estimates from the post hoc analysis also suggest allocation to Social Stories has a limited effect on general health at 6 weeks and 6 months post randomisation. Overall, there is limited evidence of any clinically relevant effects of allocation to Social Stories on general health (as perceived by parents/carers).

Intervention costs

Intervention costs include both training and intervention delivery costs. To reflect the cost of intervention when it rolls out in practice, only the costs that are associated with the intervention were considered. The costs that occur for research purposes (such as researcher time for collecting, recording and analysing data) were not included. The training cost information was obtained from the study team, and the estimated training costs were allocated to each session and each child who received the intervention. In relation to the intervention delivery costs, the information was obtained directly from the interventionists using the self-reported questionnaires. Intervention sessions were costed on the basis of the salary of the professional involved. Table 38 shows the total and average costs of each component that were used in order to deliver the Social Stories intervention. As shown, the main cost driver of training costs was the trainer fee (71.5%), while the main cost drivers of intervention delivery costs were the costs of interventionists’ time for preparation and delivery of the intervention (58.2%). On average, the estimated intervention cost per session per child was £15.22 (£12.52 for training and £2.70 for intervention delivery).

Service use and costs

Table 39 shows the summarised service use per child in each trial arm based on the complete cases only. The difference in service use between Social Stories and usual care arms were small at baseline and 6-month follow-up time points. At baseline, children in the Social Stories arm seemed to have used more NHS and school-based health services than those in the usual care arm. After receiving the intervention, resource use in both arms decreased, and such decreases were more prominent in the Social Stories arm than in the usual care arm. Also, children in the Social Stories arm were more likely to use fewer services at follow-up compared to baseline. Caution is needed in interpreting these results, as these summarised service uses include various items with different unit costs.

The total costs broken down by perspective, type of service, trial arm and before and after imputation are included in Table 40. Discounting was not applied due to the short-term nature of the trial. As shown, the costs substantially vary from one child to the other as illustrated by the wide 95% CIs. In detail, the total service costs to the NHS (before imputation) were £250.4 (95% CI £153.7 to £347.2) for the Social Stories arm compared with £379.0 (95% CI £157.9 to £600.0) for the usual care arm. The average total costs for the use of CAMHS-related community-based services, hospital-based services (both mental and non-mental health-related) and non-mental health-related medication were higher for the children in the usual care arm. The average total costs for the use of education services (before imputation) were £635.3 (95% CI 269.3 to £1001.2) for the Social Stories arm compared with £701.2 (95% CI £387.1 to £1015.3) for the usual care arm. It is observed that children in the Social Stories arm incurred less costs in school-based health services (such as educational psychologist and school nurse visits) and in general support from teachers compared to those in the usual care arm. However, higher costs for school-based intervention services (i.e. 1 : 1 mentoring/individual work, Social Communication groups and Social skills work group) were observed in the Social Stories arm. It is worth noting that some of the cost differences were likely to have been driven by the high-cost cases. For instance, the higher average cost of school-based intervention services in the trial arm was driven by two children, the first of which had a total of 540 and the other 624 hours of 1 : 1 mentoring sessions over 6 months. Given that those values were entirely plausible, we have decided to keep them in the data set without any adjustment.

The Social Stories arm also incurred fewer costs in parental productivity losses compared to those in the usual care arm. However, the Social Stories arm incurred slightly higher costs in private expenses. Such difference was likely to have been driven by two high-cost cases in the Social Stories arm. The first of which had 240 and the other 250 sessions (half day = 1 session) of paid child care over the past 6 months. Again, we have decided to keep those high-cost cases in the analysis, as the scenarios are plausible. Overall, the Social Stories arm incurred less costs across all the perspectives. This is observed in both the complete case and the base case. However, owing to the high-cost cases, the cost differences need to be interpreted with caution.

What were the main objectives of the process evaluation?

The process evaluation was cross-sectional and longitudinal, encompassing all aspects of the Social Stories intervention. The purpose of the process evaluation was to assess the fidelity of the programme (in terms of delivering the intervention as per the Carol Gray methodology outlined in Social Stories™ intervention), consider the views of various stakeholders and identify barriers and facilitators to successful implementation. We aimed to achieve this through a combination of qualitative and quantitative data collection techniques, including interviews, focus groups, questionnaires, surveys and diaries (session logs). A full overview in relation to each of the process evaluation objectives is presented in Table 2 (see Chapter 2). We interviewed five teachers, seven interventionists and nine parents/carers. We conducted three focus groups, one including the trainers and two including the trial RAs. We surveyed teachers, interventionists and parents yielding 32 respondents.

In this chapter, we have applied the components of intervention fidelity as described by Hasson44 as a framework for reporting the findings of our qualitative study: participant responsiveness, adherence and exposure/dose, recruitment, quality of delivery, comprehensiveness of the intervention and programme differentiation. Given that the relevance of each component of fidelity varies for a given trial,46 our application of the Hasson framework has been broad and flexible. As a result, some of the fidelity components have been interwoven throughout the results section (e.g. context), and some have been presented together (e.g. adherence and exposure/dose). The chapter concludes with an overall discussion of intervention fidelity during the ASSSIST-2 trial and a series of key areas to consider when implementing Social Stories in the future (programme differentiation).

What is usual care for autistic children in schools in Yorkshire and the Humber?

Given the high level of variation in the support offered to autistic children, it was important to fully understand what constituted ‘usual care’ within our sample. Data regarding the usual practices of schools were collected through teacher questionnaires completed at baseline and 6 months post randomisation (see Appendix 2, Table 59). As it was a criterion for randomisation, all teachers completed the questionnaire at baseline, providing data on what classroom-based support and specific interventions each child had received within the 6 months prior to the start of the trial.

At baseline, most children within our sample (76.3%) received some form of support within the classroom from someone other than the class teacher. In 96.8% of these cases, support was provided by a TA. Teachers also reported that 16.1% of children were receiving support from someone else during lesson time. Such individuals included behavioural mentors or behavioural leads, learning support, higher-level teaching assistants (HLTAs) and SENCO or SEN support or speech and language therapists. This support was well balanced between the two trial arms. A similar profile of support was also reported in the 6-month questionnaire, which was returned by 212 teachers (85.1% of the total sample), indicating that there was very little change over time.

To capture the interventional support being received by the children, teachers were provided with a pre-specified list of commonly used interventions from which they could select those implemented within the last 6 months. There was also the option for teachers to detail any other interventions delivered that were not in the specified list.

At baseline, 74.7% of children had received at least one intervention within the last 6 months, with 49.4% receiving two or more interventions. The maximum number of interventions received was nine; however, this applied to only one child, and all other children received six or fewer interventions. As per classroom-based support, the number of interventions received within the previous 6 months at baseline was well balanced between the two trial arms.

The interventions received both 6-month pre-randomisation and during the 6-month trial period are presented in Appendix 2, Table 59. The most frequently used intervention at both time points was 1 : 1 mentoring, which was received by 37.8% of children at baseline and 32.5% of children at 6-month follow-up. Other frequently used interventions include LEGO-based therapy, Social Communication Groups or Social Skills Groups and Visual schedules. The proportions of children receiving each of the pre-specified interventions were relatively similar across trial arms. At baseline, 24.9% of children were reported to have received an intervention other than those on the pre-specified list, falling to 15.6% of children at 6 months. The proportion of children receiving these other interventions was again well balanced by the trial arm. Examples of other interventions used include Thrive, Sensory Circuits, Sports Therapy and Games Group.

What were the profiles of the interventionists delivering Social Stories?

The full profile of interventionist demographics is presented by the trial arm in Table 14 (see Chapter 3). Most interventionists who were allocated to deliver Social Stories described themselves as TAs (83.3%). Other educational professionals involved in intervention delivery included a mixture of teachers, higher-level TAs, learning mentors, SENCOs, Emotional Literacy Support Assistants, Specialist autism teachers for the Local Authority (LA) or other individuals such as behaviour mentors, nursery nurses and SEN Specialists. The majority of interventionists were female (94.4%) with a mean age of 42.1 years (minimum 20; maximum 64). On average, interventionists had substantial experience working with CYP, with a mean duration of 12.7 years, though some had as little as 1 year of experience. Very few interventionists considered themselves to have in-depth knowledge and experience of working with children, specifically with ASC (7.4%), with almost a quarter describing themselves as having limited knowledge (22.2%).

How do participants respond to or engage with the Social Stories intervention?

The majority of parents, education professionals and members of the research team in our sample had some prior knowledge of Social Stories before taking part in the ASSSIST-2 trial, including general awareness of Social Stories as a methodology for supporting children on the autism spectrum (e.g. through training events, workshops) and prior experience of using non-personalised (i.e. not Carol Gray) version of Social Stories. For many, this experience was a key motivation for taking part in the trial, with participants keen to expand on their existing knowledge and obtain a more thorough understanding of how Social Stories could be used to support children to cope with a range of social and emotional situations both during the trial and in the future. For instance, parents spoke of wanting to learn how to apply Social Stories to manage challenges encountered at home, while education professionals wanted to further their understanding for their own professional development and to facilitate the more routine use of Carol Gray’s Social Stories at their school. Less positively, and speaking to the idea that Social Stories were viewed as one, of a range of tools available to help children on the autism spectrum, Social Stories were seen as a ‘last resort’ for one parent.

I have worked with children with autism before and had social stories done but it’s never been something that I have been particularly part of. It’s been something that maybe the well-being team have done or the child’s one to one and it’s always been quite like, if they were going to the next class there’s been a social story or we’re going on a trip and there’s been a social story. It’s never been for a particular goal. It’s always because they’ve been going to something different or out of routine and I think it was good for us to see that social stories can be used for actual goal setting, not necessarily because there has been a change to routine. Actually, it can be used in a way to develop them and achieve new objectives. So I found it as a teacher, I’ve been in schools 16 years and have been teaching for 10. I think for me it was good to see it used differently and how actually I can use it as a teacher.

Teacher 3

Irrespective of whether participants had prior knowledge or experience of Social Stories, the intervention exceeded the majority of participants’ expectations and was perceived to have positively impacted the child. Across all participant groups, the fact that goals and Social Stories were personalised, delivered one to one and tailored to address the needs of each child was consistently highlighted as a major strength and was deemed pivotal in ensuring children and those delivering the intervention were engaged with and accepting of it. Particularly for school staff, having the opportunity and dedicated time to focus on an individual child and produce tailored stories and goals was a major benefit, while also facilitating the delivery of the intervention.

I think (my child) had an opportunity to be part of a story that is about her and I think that makes it a little bit more special and she has a lot of 1 : 1 time with various people throughout her week at school but I think it’s just given her that little bit more purpose to that time, so she’s got something to look forward.

Parent 9

One of the aims of Social Stories was to improve school staff’s understanding of potential underlying reasons for the behaviour of the children (e.g. mindblindness, difficulties getting the gist, language/communication difficulties) and to use this knowledge to help them better support the child in the school setting. Reflecting their motivations for taking part in the trial, school staff felt that through training they had a far more detailed and improved knowledge of ASC. Individuals who had previous experience using more generic versions of Social Stories (that do not meet the Carol Gray criteria) and/or had attended previous training about Social Stories praised the ASSSIST-2 trial training for being more detailed and improving their knowledge and understanding of ASC and Social Stories. More specifically, school staff described how, by increasing their knowledge of ASC, they were able to ‘think differently’ about the child and often spoke of having a greater understanding of the child’s behaviour and the reasons for it, which improved how they responded to the child in situations they struggled with. Even those with limited involvement in the delivery of the intervention (e.g. SENCOs or Headteachers) described the clear observable benefits of using Social Stories and taking part in the trial for their own professional development and, in general, they felt that through increasing their knowledge of ASC, their ability to apply and understand Social Stories had been improved.

It’s been so beneficial to the child; I think when you take part in things like this you think oh, will it actually impact the child or are we doing it for us and actually it was definitely for the child and it’s impacted us as teachers, us as educators because we have developed our own CPD and our knowledge but also it’s had a huge impact on the child … I know with our child it’s had a huge impact, positive impact … I would recommend anyone else to do it because I think it was really, really useful.

Teacher 3

However, support for the intervention was not universal, with some participants describing how they had not observed any impact on their child’s behaviour. Criticisms of the intervention were largely related to Social Stories being too repetitive, lengthy or overly simplistic/childish for that specific child. Participants acknowledged that because each child and situation being addressed is different, engagement and responses to the intervention may vary. The importance of not dismissing Social Stories because they were not well received by one child in one specific context was therefore emphasised.

Even for those who did not perceive the intervention to have been beneficial on this occasion, support for Social Stories was strong and is exemplified by school staff and parents describing how they have either already used or plan to continue to use the intervention outside of the trial. For instance, parents explained how they have used Social Stories with siblings of trial participants, and school staff described having used Social Stories to support children who were not in the trial during transitions (e.g. new starters, children returning to school after absences due to COVID-19 school closures and/or sickness). Additionally, some school staff described how they plan to adapt and continue using the stories developed in the trial in the new academic year and/or use the knowledge they have gained to create new Social Stories, highlighting their beliefs that the intervention was of benefit.

I’ve saved them and put them in a pack to move them up to the next teacher … I’ve spoken to mum that it should continue and especially coming back from the break I think he’d really benefit from it. So it is something I’m hoping he will continue but me personally, as a teacher, it is something that I will continue to do with future classes and pupils … Another child in my class who at the minute really struggles with feeling like she’s bad if she gets something wrong … She’ll be really hard on herself and that sort of building resilience, I feel that would really help her as well and I have mentioned it to the next class teacher.

Teacher 4

The questionnaire data completed by interventionists at 6 months post randomisation (n = 72) revealed that 44.4% of the interventionists who were trained to write and deliver Social Stories in the trial were still using the intervention. The majority of these interventionists reported using Social Stories weekly. Some interventionists reported only using them ‘when needed’ and ‘for as long as needed’. Other interventionists reported that they had supported colleagues to write Social Stories for other children, either by working on them together or by providing information about the child they work with. The majority of the goals for these additional Social Stories were school-based and focused on managing transitions, classroom behaviour (e.g. asking for help when needed, putting hands up, making mistakes, listening) and managing emotions. When interventionists were asked if they had made any adaptations to the original training they received, 86.11% said ‘no’. For the 12.5% of interventionists that responded ‘yes’, the adaptations included simplifying the text, updating previous stories for a new year group and adding more information/pictures. These adaptations do not deviate from the Carol Gray methodology, which encourages refining the story during implementation if needed as part of her criteria 9. Twenty-five per cent of interventionists reported teaching Social Stories to someone else and this typically involved sharing the training video, PowerPoint slides and materials they had received with other members of staff and guiding other staff to write the stories using the training they received. When asked if they felt the school had supported them in using Social Stories, 88.9% said ‘yes’. These results reflect the findings of the qualitative interviews that some staff had planned to continue using the intervention within the school setting. However, as 55.6% of interventionists said they were no longer using Social Stories 6 months post randomisation, it is important to consider the reasons for this, which are explored under ‘quality of delivery’.

It is also important to reflect on how experiences of being involved in the ASSSIST-2 trial may have influenced an individual’s willingness to engage with and deliver the intervention. Mirroring the largely positive views of stakeholders towards the intervention, participants enjoyed partaking in the trial and the opportunity to gain detailed knowledge of Social Stories and ASC. Negatives of trial participation related mainly to the additional time associated with completing questionnaires and difficulties understanding how to answer some of the questions that were posed within them. For instance, one parent raised an objection to rating themselves as a parent. However, for many, these concerns, particularly those relating to time, were outweighed by the perceived benefits of the intervention and/or mitigated through the support, communication and availability of the research team to support participants throughout the trial period.

I’ve enjoyed taking part in the study. I think it has been interesting meeting people who are knowledgeable about these things because sometimes I think in school you kind of get oh we’re doing social stories and you’re like ‘and that is?’ and you’re expected to do something with it and you’re like I don’t really know how it works. So I feel that has been really nice. I know it is a commitment on our time and things from our point of view but I feel like we’ve had the training and feel more confident and have it as another strategy in the toolkit to kind of help support children who might be facing some challenging behaviours and don’t really know why. So yeah, no I’ve enjoyed taking part. I think my TA has as well and I think it has really benefited the child.

Teacher 5

Did the children on the autism spectrum receive the intervention as intended?

As described in Social Stories™ intervention, there were very few specific criteria to which delivery needed to adhere to. Stories needed to be generated according to the 10 Carol Gray criteria; however, because the trial team worked collaboratively with schools to generate stories, we were able to ensure that all stories had maximum fidelity to the criteria. During the ASSSIST-2 trial, participants were asked to deliver the intervention a minimum of six times across the 4-week intervention period. Further highlighting the importance of considering the unique needs and preferences of each child when creating and delivering social stories, the pre-specified frequency for delivering social stories was met with a mixed response from participants. For example, while some interventionists reported that children enjoyed reading the same story, for others, the novelty of stories was reported to have ‘worn off’ after the first time reading them. Repeating the same Social Story was deemed to have made it difficult for school staff and parents to make the intervention ‘exciting’ and to have had a detrimental impact on some of the children’s engagement and enjoyment. As a result, some children were reported to have not seen the value in social stories and questioned why they were being asked to engage with them. Other children really enjoyed the repetitive nature of the intervention. The divided opinion surrounding the frequency of the intervention meant that whether participants delivered Social Stories to the prescribed frequency varied during the intervention period; indeed, over one-tenth of the sample is known to have received less than the prescribed dosage, with a further quarter missing the compliance data. Some altered the number of sessions according to the perceived engagement and enjoyment of each child and, in rare cases, stopped using stories altogether.

It was quite interesting because the young man concerned when we started it I was reading to him and by the end of it because he knew it so well because we’d done it so many times, he was reading it to me quite happily and he’d say to me, again? Again you want to do it again? I was like aha, I want to do it again.

Interventionist 3

He didn’t really respond other than the first time, because it just keeps repeating itself, to do it six times, he just doesn’t want to read the same book six times. Whether it was about him or not, he just wasn’t interested in reading it that many times It’s very difficult to make it exciting to say, we’re going to go and read this book again, do you know what I mean, which is what I said at the beginning, you know, maybe we need to get around it, you, if he can keep reading it.

Interventionist 2

Social Stories are written in a specific way using a defined formula and sentence types. However, the appropriateness of this approach was perceived to vary depending on the preferences, age and receptive and expressive skills of each child. For example, the length and amount of language used within stories could mean they are pitched beyond a child’s level of understanding, and some children were perceived to respond and engage better with shorter, concise stories. Equally, others criticised stories for ‘over-explaining’ key messages, which was considered either ‘patronising’ or to mean key messages were lost and not understood depending on the individual child. The importance of tailoring Social Stories to the preferences and needs of each child was emphasised in line with the Carol Gray recommendations.

There was so much of that [preamble] and kind of qualifying things and explaining things that when I read it the message was totally lost.

Parent 2

Despite these challenges, participants were keen to ensure that Social Stories were not dismissed as a method due to an individual child not engaging in the intervention. Instead, participants suggested raising awareness of the fact that there are certain children for whom the intervention is less likely to be effective, for example, children who struggle with repetition or older children who may recognise the intent of the story or find it overly simplistic. Similarly, some participants suggested speaking to teachers and parents before developing the goals and Social Stories to ensure that they are not being developed for children who are unlikely to engage with them. Adapting the Social Stories training to include role-play scenarios of delivering stories to a child who is not engaged in the intervention was also suggested.

To have identified children that would be sort of receptive to it because, you know, I can think of a couple of cases where a child refused to read the story more than once and the whole point of it is you read the thing over and over again. But this child didn’t like repetition, so in that case a social story is going to be completely useless to them because the whole idea is you read this thing so many times that the information goes in. But this child didn’t like repetition, so the second time that they tried to read through the story the child was like, nope. I can think of another child who received the social story who thought it was patronising because they understood it, you know, they had a higher level of understanding and said it was really patronising.

Trainer focus group

The COVID-19 pandemic also had an impact on the perceived benefits of the intervention and the frequency with which it was delivered. Periods of school closures naturally meant that, in some cases, the intervention could not be delivered or was stopped. Some parents/carers perceived there to be a reduced need for the intervention during this time, as children were considered to have reduced stress levels when being home-schooled. In fact, this may have indicated there is even more of a need for the intervention if removing the school environment is reducing the stress levels for the child. Additionally, the usefulness of goals that were set during the COVID-19 lockdown was considered to have been reduced post lockdown, at which point, if the story had not been written, the goal needed to be discussed again with the teacher and a new one was set. However, parents acknowledged that their ability to assess the impact of Social Stories on their child’s behaviour due to the significant changes to their lifestyle that happened at this time made it difficult to attribute and isolate any perceived benefits or dis-benefits to Social Stories.

To be honest we’ve not used them at home because he’s not been interested in it and with lockdown he’s not actually needed it because all the stresses of school have not been there and we’ve not had any issues at all …

Parent 3

Given the importance that was placed on considering the context (e.g. impact of COVID-19) and varying needs, preferences and abilities of children on the autism spectrum, participants considered it difficult to prescribe a universal frequency and duration for delivering Social Stories in the future – suggestions varied between daily, weekly and more infrequent use of Social Stories. Indeed, some participants provided examples of how they either are or plan to continue to use Social Stories, albeit less frequently, according to the needs of each child and the resources they have available.

it is on a different basis but I would say at least 2 or 3 times a week on average. So they see it every day because obviously it’s behaviour that you would want them to see and they’re young children and they forget.

Teacher 2

Which goals are most appropriate for Social Stories?

Participants felt that Social Stories had the potential to support children with a wide range of social and emotional developmental needs and that their use should not be restricted to children on the autism spectrum. Participants were specifically asked which behaviours and support needs they felt Social Stories could be used for and, in response, they listed a wide range of appropriate behaviours relating to areas such as reassurance, emotional well-being, social skills, understanding others’ perspectives, personal care information (puberty, toileting), transitions (e.g. new school year, returning following periods of absence), support seeking, behavioural regulation and understanding sensory issues. One group, which participants felt may experience particular benefits from Social Stories, was children with anxiety. More specifically to the school environment was the potential use of Social Stories in supporting children around transitioning to new schools and/or classes/year groups or returning to school after a period of prolonged absence (e.g. due to sickness or COVID-19 school closures), topics that some teachers commented they had seen Social Stories used for the most prior to the study. Additionally, participants stressed that Social Stories should not be viewed as a ‘one-stop-shop’ and that it was important to consider creating new stories with the same child to address different and/or new challenges that they may face.

I think I would say nearly every child, I think it would be quite easy to set a goal for them of some sort because I think there’s always something in schools that children need a little bit of help with.

Interventionist 1

There were, however, some situations or types of behaviours for which Social Stories were considered less appropriate by participants. For example, Social Stories were not deemed appropriate by educational professionals for use in ‘crisis situations’ and/or for resolving complex issues such as those pertaining to child protection or a child’s sexuality where it was felt that strategies and/or interventions designed specifically for these situations should be used. Carol Gray’s Social Stories are designed to be a flexible intervention, and although some educational professionals felt more complex topics could not be addressed within the stories, this may not be the case if the individual writing them has taken time to gather sufficient information about the situation and the child so that the story is personalised and tailored to the child’s receptive understanding.

I suppose personal to them as in things that would need to be discussed further. Your child protection type of things wouldn’t be suitable. Anything sexually related definitely not … I think a story is okay if it’s … the simple one step problem, I suspect that any more complex wouldn’t work.

Interventionist 3

Despite the focus of ASSSIST-2 being on the use of Social Stories in a school setting, parents were invited to the training in order to help shape the development of the story and to gain the skills necessary to develop Social Stories at home in the future. To maintain consistency, educational professionals and teachers were asked to keep the Social Story developed for the trial specific to the school environment. By the time parents were invited to the interview, some had tried using Social Stories in the home setting too. As a result, during interviews, parents and school staff described their views on the use of Social Stories at home, both during the trial and in the future, giving us some insight into their perceived applicability in this setting. In some cases, the uniqueness of the home environment precluded the use of the intervention at home, and this was largely due to concerns of ‘singling out’ a child from their siblings by delivering the Social Stories intervention with them and finding the time to write and deliver the stories at home. Other parents described making adaptations to facilitate the use of Social Stories at home and so deviated from delivering them strictly according to the Carol Gray method. Many of these changes involved making pragmatic changes so that stories were more informal and less intensive, and this was considered to be more realistic as to how they would be used if developed for the home environment in future, particularly if different stories were being used at home and school. For example, participants described reducing the frequency with which stories were delivered, reducing the content or creating stories using only pictures, and, in some cases, involving older siblings in creating and delivering stories.

When I did write one I was careful to write it so that it would work for both kids because I’m quite aware of not trying to single [child] out as having problems. He knows that he has autism but … I don’t want him to feel embarrassed about it or that there’s anything wrong with that … How to introduce them in a way that doesn’t make a child feel like you’ve done something wrong and I’m going to read this to correct you. So at school they know that they have lots of different situations where they go with the teacher to do something … so it’s not word if a teacher just bring it out … we’re going to read through this together. Whereas at home it is a bit weird. When and how do you use this in your day if you’re not having a session with a professional?

Parent 2

Which factors of delivering Social Stories were particularly important?

To support the delivery of the intervention, many school staff described how they had developed a routine, which ensured that the Social Story was delivered at the same time each week. Reasons for adopting this routine-based approach to delivery were largely pragmatic and were devised as a way to overcome difficulties in finding the staff and time needed to deliver Social Stories regularly within a busy school environment. Having regular, dedicated times for delivering the intervention therefore allowed staff to be allocated and time to deliver Social Stories to be protected.

As a teacher obviously we are directing the teaching assistant, so we can say, look 10 minutes on a Monday morning needs to be this social story, every Monday morning. Normally we’d do a timetable as well, so that can literally go on a timetable so it’s a regular thing. That just makes it a bit easier.

Teacher 4

Despite advocating for protected time slots for delivering Social Stories, school staff also highlighted the importance of ensuring that the intervention is delivered at a time that is right for the child. Interventionists in particular emphasised the importance of ensuring that any planned times for delivering the intervention do not coincide with overly busy, stimulating and/or pressurised periods in the classroom. To ensure children are willing to engage with the story and maximise the potential benefits of the intervention, participants also described the need to adjust when the intervention is delivered in response to a child’s specific needs, preferences and behaviour. For example, one participant described how they delivered the intervention while a child was doing a craft-based activity, as this increased the likelihood that that child would sit and listen to the story for its duration.

The only problem was sometimes if I’d timed it and there was something going on in the classroom he was more wanting to be engaged in that was tricky because he’d want to rush through the story to get back out to whatever was going on. So I just adjusted the timings when I went it to make sure that wasn’t happening.

Interventionist 6

Interventionists also highlighted the importance of ensuring that Social Stories are delivered in a quiet, private, neutral, comfortable and familiar environment to the child without interruptions to promote discussion and for children to feel as though they are not in a formal lesson. This helped children’s concentration and engagement with the story while having the added benefit of ensuring that children were provided with dedicated 1 : 1 time which was protected from distractions and interruptions from other children who may want to be involved and avoided others overhearing personal and, in some cases, sensitive story content. While such an environment was considered ideal, the challenges of finding such a space regularly within a busy school environment were acknowledged.

I think a more ideal space would be somewhere you can kind of be focussed on and feel kind of comfortable and able to ask questions because really you want to promote that discussion and get to the bottom of what is causing some of these because often it is something underlying isn’t it that you’re struggling to get to the bottom of and I just don’t know if they’d be willing to kind of come out with that or have that discussion when there are people toing and froing all the time.

Teacher 5

When reflecting on the way the intervention was delivered, it is also important to consider the role of ‘who’ is responsible for delivery. While some participants stressed the importance of ensuring that those involved in writing and delivering social stories are experienced in dealing with children with varying emotional and social support needs and had experience and training knowledge of ASC and Carol Gray Social Stories, others felt that this knowledge was secondary to knowledge of the child. Indeed, across participant groups, knowledge of the child was described as the most crucial prerequisite for maximising the benefits of the intervention. This was largely as close relationships with the child were seen to facilitate exploration of the underlying reasons behind a chosen behaviour and mean that Social Stories are focused on an area that is of benefit to that specific child. More specifically, the process of creating stories and identifying and narrowing down goals was considered easier when undertaken with someone who understands the child, their behaviour and what motivates them. Indeed, trainers described how during training those without knowledge of the child struggled to ‘put pen to paper’ and made it difficult for trainers to ascertain whether the story and goal that had been created was right for that child and, in these cases, participants were advised to gather further information from other teaching staff and the child’s parents. Ensuring that the Social Stories were created to address the specific needs of each child was also considered paramount to the intervention’s success, with children described as more accepting and willing to engage with an intervention that was tailored to their specific needs and delivered by someone they know and trust. Perhaps exemplifying this point, a couple of interventionists described how the goal-setting and story development was more challenging at the start of the academic year when school staff had less familiarity with the child and is something to consider when planning to use the intervention in future.

I always found it much easier when the person setting the goal really knew the child. Because typically we’d have the teacher setting the goal and then their teaching assistant would be the one reading the social story and it could be different ways around. Some schools were a bit more flexible with that but especially like September, that was a really difficult time to set goals because even though we tried to ask the teacher from the year before, it wasn’t always possible to facilitate kind of getting them out of the class to do. But when a person really knew that kid and really understood what motivated them, what they enjoyed and also like maybe taking a bit of time to think through what they were experiencing, was a much easier process to set that goal. So I definitely feel like it needs to be someone that knows that child and maybe September is not a good time for a teacher to be setting the goal.

Focus group with research assistants

In a school environment, teachers, TAs and SENCOs (if trained appropriately) were considered to have the potential to deliver the intervention and develop goals and Social Stories. However, TAs were considered best placed to create and deliver a personalised intervention that was tailored to the needs of that specific child. By working with the child directly on a daily basis, TAs were deemed to have the closest relationship with the child and may have experience delivering other interventions with the child, allowing a sense of normality for the child. RAs and trainers highlighted the importance of ensuring that those involved in setting the goals and creating the Social Stories also deliver them, not only for continuity and engagement but also to ensure that those delivering the intervention understand the purpose of the intervention is to support the child.

I think it was really worthwhile for the people who actually write the social stories who work with the children … They [TA] are the right person because obviously they’re the people who work with the children day in and day out and they are the one that has the relationship.

Teacher 2

For school-based interventions, some participants also felt that there needed to be greater involvement of teachers to work alongside TAs when identifying goals and creating the Social Stories™. The generally higher levels of computer literacy amongst teachers were considered particularly important, as this was described as a barrier to some individuals being able to add pictures and personalise social stories. Additionally, given that teachers have overriding responsibility and are the main point of contact for a child, they often receive key information (e.g. child focus plans) that they pass onto TAs, their involvement is important.

The teachers really enjoyed the sessions. They were really competent in writing the sentences and they were more competent with technology. But then you often need the sort of personalised input that the TA might have because they work so closely with the child. So it sort of needs to be a collaborative process. The teachers really can’t do it without the TA but equally most TAs can’t do it without the teacher. So it does need to be a sort of combined effort to get these stories together and that’s often what was lacking.

Focus group with trainers

There was some uncertainty amongst participants as to whether, as a school-based intervention, parents should be involved in creating the Social Stories. Some participants felt that parents should be involved irrespective of whether the behaviour and/or situation being addressed applied to school, home or both environments. This was largely due to beliefs that there was a need for transparency and to acknowledge that most parents would want to be aware of any interventions/strategies that were being used with their child, even if they were not involved in the process directly. For instance, even in situations where a goal was only applicable at school, some felt it was important to keep an open dialogue with parents about the purpose and delivery of the intervention and to offer them training and support in designing their own stories at home. A number of parents reinforced this and felt that to help their child’s social and emotional development, they needed to be aware of any support their child was receiving at school. Some teachers and interventionists echoed this and felt they would not feel comfortable delivering interventions if the child’s parent was not aware of it and in support. Parent involvement was also considered beneficial for facilitating the writing of the Social Story, with the unique knowledge that a parent brings of their child considered crucial to providing insight and context to a child’s behaviour. Similarly, involving parents in the development of the Social Stories was considered beneficial for improving relationships and communication between parents and school staff.

So I spoke to the parent about the goal and then gone to the school and set the goal which was to do with hiding under the school table during lesson time and then when I went back to the teacher. So the story would have ended up going down the road of, try not to hide under the table da, da, da, and then when I went back to the parent she was saying, actually that’s a behaviour that the child is probably doing because she’s feeling anxious about something. So it was very similar. It was like explaining that behaviour and it would have, again, gone down the wrong route had the parent not been involved.

Research assistant focus group

While many advocated the inclusion of parents, others felt they had a limited role in creating and delivering a Social Story in a school environment. For some, involving parents during the goal-setting process had led to some discussions and disagreements about the focus of the goal for the child. In these situations, the fact that the trial was focused on addressing school-based behaviours was perceived to have potentially exacerbated tension between parents and school staff, particularly if a child’s needs differed across the two environments and parents viewed the research project as their ‘one chance’ to get help with a behaviour. Participants also acknowledged that there may be less relevance of including parents for children on the autism spectrum for which it is important to separate school and home life, with home for these children considered a ‘safe zone’ where the pressures of school are not addressed. While researchers and trainers found that pre-empting these disagreements and explaining to parents in advance of the goal-setting meeting, the school-based nature of the research helps in relieving this tension. Researchers/trainers also described themselves as having a mediation role when disagreements occurred, and it is important to note that it may be beneficial to have a third party, such as a teacher or TA not involved in the intervention to provide this role, although in practice this may be difficult.

I think it probably depends on the child with what you’re wanting to tackle really. I think for school-based issues, I think it’s just best that school do it because they know what he’s like in that situation but then I wouldn’t expect school to write one for a home based situation because they wouldn’t know really. I think whatever situation you’re trying to tackle you want to choose the best person to write the story.

Parent 7

Mirroring the need to tailor the intervention and consider the needs of each individual child, participants had varying views on the appropriateness of involving the child in the creation of goals and Social Stories and recommended that this should be considered on a case-by-case basis. Interventionists (typically TAs) and the research team who advocated involving the child felt adopting this approach may be of particular benefit for children where it is possible to anticipate issues with engaging them (i.e. older children, those who feel ‘babied’ by the intervention, and/or those who do not like repetition) as it may give them a sense of ownership and, where appropriate, an understanding of the purpose of Social Stories. Some participants felt it was difficult for a child to understand the ‘intricate’ process of goal-setting and the need to create Social Stories within the parameters specified by Carol Gray. There were also certain discussions around goals that were considered to be inappropriate with the child present (e.g. the impact of the behaviours on parental stress, the educational professionals or the wider classroom).

I don’t know whether it’s an age thing but certainly it’s nice isn’t it to brainstorm with them and get them … It’s their goals, it’s not my goal … It should be something that he wants to do and I think if you’ve got more buy-in at that stage rather than just something arriving, oh we’ve written a story for you because we think you need help … It’s like a performance review without the person who’s actually doing the work.

Parent 6

Ultimately, when considering who should be involved in goal-setting and Social Story development and delivery in the future, it is important to ensure that the person/people with the closest relationship to the child are involved. Despite the various pros and cons of involving different stakeholder groups, the majority of participants advocated adopting a collaborative approach (e.g. TAs, classroom teachers, parents) during goal-setting and story development to enhance the overall quality of the story and to encourage reflection and different experiences of the child and their behaviour to be discussed and incorporated into Social Stories.

Unique features of Carol Gray Social Stories as facilitators to intervention delivery

The ability to personalise Social Stories to the needs of each child was consistently identified as a main benefit of Social Stories. For example, incorporating pictures and photographs was perceived to facilitate learning and enhance children’s engagement with and enjoyment of the intervention. This ability to tailor the intervention was also perceived to accommodate the individual learning needs of each child. For instance, one parent described how using pictures was particularly helpful for their child, who they considered to be more visual in their learning and who it was felt would not have engaged with or understood an intervention that was solely word-based. Equally, the ability to make adaptations or adjustments to the content of Social Stories to reflect changes to the environment or an individual child’s needs was considered a huge benefit, particularly for children on the autism spectrum, who tend to prefer information that is literal and accurate. For example, changing the names of teachers throughout stories if substitutes were in place.

The fact it was personalised he really enjoyed that. He saw his pictures and his friends and she made it have a train aspect to it as well and he loves trains, so he was wanting to flick to the next page and find out more about himself.

Parent 8

The structure of the Social Stories also facilitated the development and delivery of the intervention. Breaking stories down into ‘manageable chunks’ (e.g. clear title, positive introduction, different sentence types to include) gave school staff an understanding of the importance of including each story component. Participants felt that the structure of the story and how it was tailored to the individual needs of the child were beneficial for helping the child to understand complex social situations. Many stories focused on gently guiding children to communicate their emotions to an adult. Participants reported that they had observed clear changes to a child’s behaviour following the story, and they felt that this was due to the information provided in the Social Story which had helped the child to better understand why it is important to tell an adult how they feel and potential methods for doing so.

Navigating the goal-setting process

Despite the goal-setting process being considered pivotal for understanding what underlies a behaviour or situation and praised for providing individuals with the chance to talk about and understand a child and their behaviour, specific challenges with goal-setting were discussed. One of the main difficulties participants faced during the goal-setting process was identifying a goal that was specific, beneficial to the child and realistic to achieve within the 6-week intervention period in a school setting. Goals based around routines within the school day and playtime, as well as those around understanding and communicating emotions and making mistakes, were all considered to work well and be child-oriented goals. The ‘goal-setting sheet’ and SMART framework (used to ensure the goals were Specific, Measurable, Achievable, Realistic and Time-specific) were considered crucial in overcoming many of the challenges identified with the goal-setting process, particularly in terms of ‘solidifying’ goals and for understanding how to break goals down into achievable steps from umbrella goals. The goal sheet was also deemed useful for ensuring goals were suitable, of high quality and designed in line with Carol Gray methodology. Additionally, while considered time-consuming to use, as many school staff were already familiar with the ‘SMART’ framework, the research team felt that this helped participants to identify and refine goals and ensure they were positively framed. Conflicting priorities between parents and school staff and difficulties redirecting goals away from something negative and/or vague were commonly reported issues. The goal sheet and SMART framework helped the research team to navigate difficult conversations and differences of opinion and ensure goals were focused on addressing the individual needs of each child and were not too vague. For example, if a goal around ‘managing playtime better’ was proposed, these resources were used to help determine if this is something that is in the child’s interest, and if so, the goal was converted into something more specific, for example, ‘to try to talk to some of the other children during playtime’.

It [the SMART framework] was good and it makes you think sometimes a little bit outside the box to make sure that it was achievable with the sort of parameters in mind I think. What we maybe wanted to go with perhaps sometimes didn’t exactly fit right … I think that was good.

Teacher 2

However, there were differences within and across stakeholder groups in terms of their confidence in creating goals, reflecting the different degrees of familiarity with autism interventions and, more specifically, Social Stories, goal-setting and resources such as the SMART framework. As such, the extent that the SMART framework and goal sheet were considered useful varied, with some trainers describing how they felt participants would not have used these resources unless encouraged to do so. While some found these resources ideal prompts to facilitate goal-setting as per the Carol Gray method, others found them time-consuming. Participants made a number of suggestions for the future to facilitate the creation of achievable, child-oriented, positively framed goals. For instance, including a section on ‘general information about the child’ on the goal sheet to ensure that, at the start of the goal-setting process, individuals are given the opportunity to discuss the behaviours and needs of that child, increasing the likelihood of creating a tailored and child-orientated goal. Additionally, providing examples of goals that have been broken down and emphasising the need for positively framed goals in resources was proposed. In the current trial, the goal for each individual child was set before randomisation and in cases where there had been a significant gap between baseline measures and receiving Social Stories training on occasion educational professionals asked for the goal to be updated when they met the trainers. Outside of a trial, it would be possible to set the goal as part of receiving Social Stories training, and therefore this issue could be avoided.

I think in terms of making it a bit easier, I think for the sheets it might be good to have a section that is just like, generally like about information about the child. So, when we do the kind of training sessions to write the stories, we often say okay can you tell me a bit about the child and then we’ll start writing the story. I think that’s just quite a nice opportunity for the teacher to just, or the intention is to just like chat about the child because in the training session that’s where we get their kind of likes and dislikes and all that from. But I think for the goal setting it would be nice to do that as well and then you can just … because you end up scribbling loads of stuff down on the sheet and then you’re like trying to set the goal and you look at it and it’s like all over the place. So, I think if there was a section that was just like kind of just general information and then moving onto, okay what is the problem. Because sometimes you’ll write it all down after you’ve said, do you know, what’s the problem at the moment and they’ll say loads of stuff and then they’ll say, oh actually so what we’d like to focus on is this or you have to like pick something from what they’ve said. So, I guess that’s more of a admin thing for us that might make it a bit easier but I think it would also make it easier for the teacher as well just to give them that time to talk about the child a bit more.

RA focus group

Profile of trainers

Four academic staff were trained to deliver Social Stories training to school staff (all female, mean age 31). All trainers had an undergraduate or master’s degree in psychology and three had completed a PhD in the field. Through their qualifications and work experience, the trainers had prior knowledge of autism and had some experience delivering workshops/training/presentations to staff and higher education students.

The training model

A cascade model of training (train-the-trainers approach) was implemented. The former Chief Investigator (Professor Barry Wright) who is a consultant child and adolescent psychiatrist and has extensive experience of autism and Social Stories trained the first trainer by delivering mock Social Stories training sessions, role-play scenarios, answering any queries and then observing the trainer deliver the session themselves. The trainer was only ‘signed off’ to deliver training independently when the Chief Investigator was satisfied with the performance. Due to staff changes, two further trainers were trained by the first trainer. This involved shadowing Social Stories training sessions in schools, becoming familiar with all training materials and resources and then being observed delivering the training themselves before being ‘signed off’ to deliver training alone. The second set of trainers were then able to train a fourth trainer following the method described above. The quotes below describe some of the trainers’ reflections on the training process.

My training will probably have been a little bit different from everybody else’s because it was [Chief Investigator] who delivered to me … [Chief Investigator] delivered the training with sort of his slides that have since been slightly amended but none of the content really has been amended … So it was like imagine a child that you know and, you know, what might you put in these different sections [worksheets] to get us a little bit more familiar with what it was that we were looking for when it came to like what do we mean by mind blindness, how would this, so on and so on. It could be a real child that we knew that had a diagnosis or it could be one that we just kind of made up and try and fill in to understand the sections. I didn’t have any training ‘in practice’, if that makes sense. So after that training it was just go out to schools and deliver it and then for a long time it was just me delivering it until [name] came on the scene and sort of did training.

Trainer focus group

Just before I started as a Trial Coordinator on the study … I shadowed three times and I think [name] you came long to maybe a couple of those as well. So we saw the presentation and the training getting delivered in practice, which was really useful, and it was good to be able to see it at a larger school because I think we went to one and it was very well attended by additional people who didn’t know anything about the trial. So it was good to see that it changes in every school that you go to. But, yeah it was basically shadowing three times and then having a go yourself. I think, I also did some mock training with [name] and I sat with [name] who was a research assistant on the study at the time and we’ve had a look through the slides and just made sure that I understood the content before going out to deliver it for the first time.

Trainer focus group

All trainers reported feeling slightly anxious or nervous before delivering the training to school staff, but this did not stem from lack of understanding of autism or Social Stories but more from the uncertainty about how the session would go and the needs of the attendees. Worries discussed included the unpredictable nature of whether the goals for the children may need to be changed during the session, whether the stories may address difficult topics that are more challenging to write and the level of support required by the attendees within the short time frame of the training session.

As to how I felt about the training. I think for me what I felt uncertain about was less to do with the autism specific parts because the team that we sat within COMIC, like [Chief Investigator] is very familiar with autism, it’s one of his sort of competencies. So, from that point of view I felt like I already knew quite a lot from how we were talking about that sort of stuff in the team, you know, we had other trials running around autism and adapting assessment instruments for autism and so on. From that point of view I can’t say that I felt unprepared. I did feel unprepared in the sense that I didn’t know what to expect, so it was kind of like straight out into school alone and there was no checks and balances there. So like, am I doing it right, am I not doing it right and so on. Then after a while I wouldn’t even say it took that long to sort of feel like, actually I feel okay with this, you know, in the sense that kind of similar to when I first started teaching, you know, when you first start going into it. Am I going to be able to answer any questions? Wonder about your own competency and so on and then you realise that the sort of questions that they tend to ask and the sort of guidance that they tended to need, I realised I was quite able to give that. But, yeah for sure, I was a bit nervous in the beginning that was a bit into the deep end but then again that’s also how I like to learn.

Trainer focus group

Trainers felt that it is important that anyone who is trained to become a Social Stories trainer in the future has a background in autism and/or psychology in order to be able to deliver high-quality psychoeducation about autism that can help school staff to understand the possible underlying reasons behind a child’s behaviour and challenges.

I personally don’t think someone that didn’t have a psychology background could just pick this up and run with it, you know. I think it would need to be like an educational psychologist or something like that that was delivering the training that would have that background level of autism because the training in autism and going through what is different about a child with autism and getting the school staff to really sort of understand it from a psychological perspective is almost more beneficial than the sort of story-writing process because you saw so many moments where we were explaining, you know, when a child behaves like this it’s not because they’re naughty, it’s not because they’re selfish and you could sort of see the cogs turning for them thinking ah. One of the sort of indirect ways that I feel like this trial has impacted is their learning about autism. So if you get someone who didn’t have that background about autism that knew how to write a social story and be able to tick the Carol Gray criteria then the school staff wouldn’t be getting the same sort of knowledge that’s really going to benefit the children.

Trainer focus group

The trainers felt that the package of training for school staff needed strengthening so that school staff could confidently continue to use the intervention and train others following the trial. In order to do this, trainers proposed simplifying some of the resources provided to schools (e.g. worksheets and the terminology used within them) and ensuring that the school staff attending the training had sufficient time to process the psychoeducation about autism and think about how it relates to the children they are working with.

You’ve got large numbers of children with autism, so it wouldn’t be practical for a practitioner to be coming in and writing social stories for all of them. So we need to move away from this really high level of handholding that we’ve had in this trial, so really intensely training, you know, one or two people within a school to really roll with this and not need that post support that we’ve given. So I think we really need to strengthen the training package for the practitioners, so that they can go forward and train the school staff.

Trainer focus group

I really liked the training on ASSSIST-2 because it emphasises not just positivity around the story and so on but also genuine understanding of the child, the child’s needs and how we have to adapt and respond to the child’s needs as well. I think that’s quite important and not always recognised particularly as [name] says, if they don’t have that much experience with autism and they kind of get seen as a bit difficult and like, you know, they’re being naughty and like they’re just really rigid and it’s like, well for a reason.

Trainer focus group

Participants’ views on training

As previously discussed in the section about participant responsiveness, irrespective of whether a participant had prior knowledge of social stories, training was regarded positively and considered to have increased participants’ confidence in writing and delivering Social Stories and setting goals. Increased knowledge of ASC and the underlying philosophies of Social Stories was considered a key benefit of taking part in ASSSIST-2 training. Despite this, a small number of participants criticised the in-person training for being rushed and including too much content, particularly the volume of written text in the PowerPoint presentation about autism and the Carol Gray criteria for writing Social Stories.

The training was really thorough and because of the thorough training it just meant that the writing of the social stories was much easier to do.

Teacher 1

Participants were particularly supportive of the practical and interactive components of training, which were thought to greatly facilitate their ability to write and deliver the intervention with confidence. A number of participants emphasised that the reality of producing Social Stories and identifying goals was very different to hearing about them, even if you understood the process, and that it was only through the interactive elements of training that they realised how challenging the process was and identified gaps in their knowledge. In particular, having the opportunity to create a Social Story during training was considered crucial to intervention delivery and meant participants felt less daunted about delivering the intervention. Creating the Social Stories during training also provided a personal, supportive space to learn and for those that received training in person, doing so in small groups, through exposure to combined expertise (e.g. trainers, TAs, teachers and parents) and different perspectives gave individuals the confidence and knowledge to create a personalised Social Story that was tailored to the child as well as an understanding of the language and structure that is used within the Carol Gray methodology.

I think what was also great it was just three of use literally, so we could ask the stupid questions and we could kind of, you know, you could get quite individual about what it was about and kind of what was on your mind really which was nice as well. It didn’t really feel like training. It’s just sort of felt somebody guiding you through a process really which was good.

Parent 4

Of those who were critical of training, the majority were parents. This may reflect the fact that parents were not directly involved in delivering the intervention and that training was focused on preparing educational professionals to deliver the intervention in an educational information. The perceived complexity of creating a Carol Gray Social Story also caused participants to feel daunted at the prospect of writing their own story and stemmed, particularly for parents, from a lack of familiarity and prior experience of using Social Stories or similar interventions in comparison to school staff. For example, participants were nervous at the idea of remembering how to create and deliver a Social Story as per the Carol Gray method (i.e. the sentence types to use and style) and felt that while they may be able to ‘give it a go’ that ‘it might not pass the hundred percent authenticity check’. Access to example Social Stories that were created both during training and in the manual mitigated these concerns to some extent. However, some parents still felt that to use Social Stories at home, they would need more examples and/or additional support and training before feeling confident to write a social story using the Carol Gray methodology.

I suppose it’s given me the opportunity to build a social story in the future about other things. I haven’t actually done that but I think it’s knowledge and training and I’ve kept all the information that would enable me to do that … If I was to create a social story at home … I’d kind of feel a bit alone doing it because I was doing it in partnership with someone who knew how my son is at school. I’d kind of feel like I’d want that again I suppose, a couple of hours of someone’s time just to work on it and I don’t know if it would be feasible … yes I would be interested, now and in the future, in being involved in some kind of, you know, once a year or whenever a meeting with school to try and keep that training and knowledge up and maybe work together.

Parent 6

During the ASSSIST-2 trial, training was originally designed to be delivered in person in a 3-hour session which took place either in the morning or afternoon. Due to restrictions put in place in response to the COVID-19 pandemic, from March 2020 the Social Stories training was delivered online. Changing the mode of delivery naturally had implications for how training was structured and delivered. During focus groups, trainers described how the adaptations that were made to training when transitioning to the online model appeared to result in less difficulty supporting individuals to write their own Social Stories. For instance, in comparison to the in-person model where trainers were supporting multiple groups of participants to develop Social Stories, the online model provided each interventionist with a dedicated 1-hour 30-minute time slot to create a Social Story for the child they were working with alongside a trainer. In advance of these sessions, trainers pre-populated standardised templates (e.g. a positive introduction), ensuring that these individual sessions were spent expanding and tailoring the Social Story with specific content relevant to each child. Educational professionals and parents attending the training were sent a 30-minute presentation to watch in advance of the 1 : 1 story-writing session. This presentation covered everything that was delivered in the presentation, which took place in person prior to the COVID-19 pandemic (e.g. psychoeducation about autism and the Carol Gray methodology). Additionally, at the start of the ‘live’ online session, participants were provided with a summary presentation, which included a brief explanation to recap Social Stories, an example Social Story. While it was acknowledged that the online model meant trainers had to allocate sufficient time to pre-populate a story template, this was considered preferable to the amount of post-training support trainers provided during the in-person model. Those who delivered training in person described regularly having to provide up to an additional half a day of support to ensure that Social Stories were completed and/or reported having to rewrite some of the Social Stories that participants had produced. In the future, regardless of whether training is delivered online, in person or using a hybrid approach, trainers emphasised that significant support is required to ensure that people are creating and delivering Social Stories as per the Carol Gray methodology and to ensure that the amount of pre- and post-training support required to produce social stories is not underrated.

Once we moved to the on-line where we’ve tended to sort of have one story at a time, so you know, we maybe [during the in-person model] had three interventionists in the room writing three stories and it was quite tricky to sort of manage that. Whereas, when we moved to the on-line model, we’d block out one hour slots with each of the interventionists which made things a lot easier, and because we sort of put together this template, which had a lot of the standard sentences. So you know, all the introductory pages were the same for each child so we sort of pre-populated that so we could just type in the details on the day. So I think once we moved to that model we had less sort of going over time, if you know what I mean, you know the story writing fit more into the session.

Focus group with trainers

Despite participants generally being accepting of online delivery models and acknowledging that due to COVID-19, training packages are likely to be delivered remotely in the future, for some their preference would always be for face-to-face delivery. Equally, concerns were raised about the potential for ‘zoom fatigue’ with such a comprehensive training package, and a very small number of participants experienced technical difficulties, which meant that due to poor audio quality, they may have missed some of the content during training sessions. Additionally, while the online model included interactive components, in-person events were considered more conducive to engaging and interactive discussions.

It’s going to be the norm anyway isn’t it from now on. Yeah, absolutely fine. I personally prefer face to face training but, yeah I thought it was absolutely fine. I was not left in anyway thinking, oh I didn’t get that or I didn’t understand that.

Interventionist 2

Irrespective of the model chosen to deliver training in the future, the importance of ensuring participants engage and most crucially that the interactive elements of training remain and are not removed for the sake of time and resource was highlighted. Participants stressed the importance of ensuring participants are able to ask questions and allocating sufficient time to facilitate discussions. Keeping the ratio of participants to trainers as small as possible was also suggested as important for ensuring a positive and supportive learning environment and for facilitating discussion. Trainers highlighted the importance of not underestimating the amount of time that is needed for them to prepare to deliver Social Stories training (online model) or to provide post-training support (in-person model).

In addition to training, education professionals were provided with a number of additional resources (a Social Stories manual, example Social Stories and worksheets focused on understanding autism, setting goals for the stories, sentence types and intervention delivery) which were described as crucial in supporting the creation and delivery of Social Stories.

Making a series of example Social Stories available, in addition to those created during training, was considered particularly beneficial not only for consolidating knowledge, but also for providing a reference point and reassurance to participants that they were creating Social Stories as per the Carol Gray method. For those who described themselves as less confident in creating an entirely new social story or for those with limited time, example stories provided a source of inspiration and/or useful templates that could be tailored to the needs of an individual child. As such, educational professionals and parents advocated developing a booklet, with a large number of Social Stories to support the continued and wider use of the intervention in the future but stressed the importance of ensuring this does not prevent future stories from being tailored to the needs of each child. Exemplars and other resources (e.g. goal-setting worksheets) were therefore not only seen as a resource for supporting individuals who had taken part in the trial to continue to use social stories but also to act as a mechanism for encouraging others to be involved in the development and delivery of Social Stories.

Although individuals were provided with training, support and resources (e.g. the Social Stories manual, exemplar stories), there was some uncertainty within the group of trainers as to whether individuals were likely to be sufficiently prepared to write and deliver Social Stories outside of the trial. This uncertainty stemmed from the fact that each goal and story is unique to the needs of a given child, and some interventionists did not have a strong knowledge base in autism and lacked confidence during writing sessions. RAs and trainers received a slightly different model of training from participants, which involved Cascade, or a train the trainer model, involving a combination of formal training, that is, shadowing Social Stories training sessions and spending time with an experienced trainer to review all materials in detail and gain feedback on their performance. This experience of shadowing and delivering training was perceived to have given them an ability to describe, understand and apply how to set a goal and create a Social Story. Observing and shadowing the goal-setting process in particular was perceived to have exposed RAs/trainers to the practical aspects of goal-setting, such as using information about a child provided by an individual to tailor goals. Additionally, shadowing more experienced individuals was thought to have given them exposure to different stories and insight into how to challenge teachers and parents about their choice of goals and have confidence in discussing the underlying reasons behind a child’s behaviour. Ultimately, they viewed the process as unpredictable due to the uniqueness of each goal and story. As a result, they placed significant importance on ‘learning on the job’ to accumulate the experience, but most importantly confidence needed to be able to translate skills and experiences of previous goals and stories. To mimic this experience in the future, participants highlighted the importance of including role plays, particularly those of challenging situations (e.g. narrowing down broad goals, dealing with conflicting priorities), to help trainers feel more confident.

I think getting that decision, you know, getting that end goal, the find, it’s like a skill that you have to keep … so the shadowing definitely helped and then just going in and actually doing the sessions helped because every teacher comes in with different things and different behaviours and when you have all these guidelines on how it has to be, like a smart guide it has to be specific, it has to be positively framed. Sometimes it’s really easy and sometimes it’s quite difficult. Yeah a lot of practice was definitely helpful.

Focus group with research assistants

What were the challenges faced in delivering Social Stories?

A number of barriers to delivering the intervention within a school environment were identified. Even for those who had allocated dedicated timeslots for delivering the stories, ensuring that there was sufficient time within the school day and a TA available to deliver the intervention was challenging. Particular difficulties surrounded trying to balance the need to ensure the intervention was delivered 1 : 1 in a quiet, private environment against not removing the child from an important lesson and ensuring Social Stories were delivered at the right time for the child. Some teachers described having engaged with wider professional groups during the trial, such as autism teams, to highlight the benefits of Social Stories. School staff suggested that to deliver Social Stories outside of the trial, they would need to allocate one member of staff per child to facilitate the delivery, and that dedicated time would need to be given from the headteacher to school staff (TA, teacher) for them to write an effective and personalised story together for each child and for staff to attend training.

From my point of view as a class teacher, I’ll probably have a bit more confidence in saying, well is there any chance we could have a bit of time in the day to do [the intervention] because actually if we give it a bit of time to it, it is more meaningful and effective and actually more likely to get you the outcome that you’re looking to achieve. So I think sometimes that is the issue and maybe doing the social story trial has made me and the SENCO think that a little bit.

Teacher 5

Given the amount of additional support (e.g. through the research team) and time that had been made available to support the delivery of the intervention during the trial, interventionists raised concerns about the amount of resources required to support the continued and wider use of Carol Gray Social Stories outside the trial, particularly in a school setting. As such, participants were keen to avoid being reliant on a small number of school staff to create and deliver stories at each school, with this approach deemed particularly problematic in larger schools where it is likely that multiple children would need support despite limited resources. Additional and/or refresher training was proposed to ensure that the knowledge and skills participants had obtained during the trial were maintained and passed on to other school staff. However, a popular suggestion, for facilitating the wider use of Social Stories within schools, was through ‘Social Stories Champions’. While there was some speculation as to who could take on these roles (e.g. SENCOs), having someone who is embedded within the school to co-ordinate the facilitation of Social Stories was considered important. This was largely because the amount of time required to support staff create stories was considered too great for an ‘outsider’ (e.g. local authority personnel). Participants also felt that champions would require specific, intensive training to ensure that after training, champions felt confident and were able to support staff to create and deliver Carol Gray Social Stories without requiring additional support. Participants also saw a key element of the champion role to be to encourage best practice and a supportive environment, underpinned by peer support. As such, signposting staff to resources and training and encouraging sharing of experiences, within and between schools, for instance, through the creation of Social Stories networks, was proposed.

… social stories champions within schools because I don’t think it would be feasible that educational psychologists or local authority practitioners or whoever be dealing by case-by-case basis, especially like [name] says when you’ve got these big schools. I mean in that school it was eleven participants, that wasn’t eleven children with autism, you know …. You’ve got large numbers of children with autism, so it wouldn’t be practical for a practitioner to be coming in and writing social stories for all of them. So we need to move away from this really high level of handholding that we’ve had in this trial, so really intensely training, you know, one or two people within a school to really roll with this and not need that post support that we’ve given. So I think we really need to strengthen the training package for the practitioners, so that they can go forward and train the school staff.

Additionally, school staff described the challenges of delivering such a resource-intensive intervention amidst a high workload and staff shortages and likened doing so to a ‘juggling programme’. Pressure around staffing was particularly heightened during the COVID-19 pandemic and resulted in some staff describing how they had had to rush the delivery of the intervention, or in some cases, to stop or miss sessions entirely due to child and/or staffing absences or school closures.

I think mainly it is the workload as in there is a lot of with COVID there’s been a lot of staff absences so there’s been, yes this is your plan and it fits in, but then on a daily basis a lot of things change. Either the child is absent or there is a staff absence so you’re filling in a slot. Same with most things and you’ve just got to prioritise what can be dropped and what can stay. So I think it became a bit of a juggling programme.

Interventionist 5

Poor communication between staff surrounding the logistics of delivering the intervention, Wi-Fi issues in schools and low levels of computer literacy were also described as barriers to implementing and delivering the intervention. Low levels of computer literacy among TAs and an unfamiliarity with PowerPoint created unanticipated difficulties in personalising Social Stories, particularly in regard to adding pictures. The implications of this for future roll-outs on resourcing warrant consideration. These barriers were not unique to school staff, as parents also found time and capacity, particularly during the COVID-19 pandemic, and access to materials and equipment (printers, laminators) a barrier during the trial.

You need to be relatively IT savvy in order to be able to use them. I think particularly for some children. The one that I did was just word but I know the other two we did in communicate and print as well because that is what was needed for our children to access it so that requires a certain level of IT skills to be able to communicate and print and then be able to personalise it and put pictures in as well.

Interventionist 6

As part of the interventionist 6-month follow-up questionnaire, interventionists were asked to report whether they had encountered any problems/barriers to using Social Stories. Of the 73 respondents, 68.1% said ‘no’, and of the 29.2% that said ‘yes’ the barriers generally fell within the following categories: the time to write and deliver Social Stories, COVID-19 disruption and school closures, staff changes (e.g. changing classrooms or the children they are working with), school holidays disrupting the intervention period and the child refusing to engage (e.g. not interested after the first read, inability to focus on a story). When asked if they had stopped using Social Stories completely, 56.9% said ‘yes’, and the reasons were very similar to the barriers described above, for example, COVID-19 disruption, stories no longer needed as the child had moved classroom/school and time constraints within the school.

For the use of Carol Gray Social Stories to become more widespread, some RAs/trainers felt that some schools may need convincing as to why they should invest their time and resources into delivering the Carol Gray method, particularly when considering that many of their staff will already have some familiarity and experience of delivering more generic versions of social stories that do not meet the Carol Gray criteria. Some participants described that since the trial rather than using PowerPoint and the structured format of Carol Gray’s Social Stories, they had condensed stories to one page, asked children to create their own stories or used non-personalised stories with a group highlighting that not all participants have continued following the Carol Gray criteria. To avoid adoption of the Carol Gray methodology being ‘hit and miss’, trainers/RAs proposed clearly communicating the benefits of Carol Gray social stories over alternatives. Accredited training and/or certification, delivered through established networks and/or professional groups, for example (SENCO, educational psychologists, academies, educational mental health practitioners), was considered one way to achieve this, alongside limits on the time that training is valid for to ensure that quality of social stories remains high and as per the Carol Gray methodology.

The other thing I think that is really confusing and I found that as a teacher, and I find that in my role now, is that actually most educational professionals don’t know that there’s like this Carol Gray social stories and there’s a lot of information on the internet on things like Twinkle. That isn’t a proper social story as we would say now that we’ve been trained in it but it’s still called a social story and it’s almost like too common a word now. So if you say, oh a social story, everyone is like, yeah I know what that is, we’ve used one of them. People haven’t and I know it’s got like the TM on but no one notices that. It almost needs a bit of rebranding or be called ‘a something social story’, so people know that it is something that you’ve got to be specifically trained in and there’s one way of doing it and you have to do it that way. You can just Google social stories and download them off the internet and not personalise them at all and most staff still think they’re using a proper social story.

Research assistant focus group

Some individuals reported that the main benefit of the intervention was a greater understanding of autism and how to improve their discussions about social situations with their child/the child they were working with. For example, some participants described how they are applying what they have learnt, specifically around the use of simple, declarative sentence types and are using this to break down social information in a number of situations in daily life, not necessarily in the form of a Social Story.

I think the process and the construction of social story could be broken down in verbal ways, in more formal ways I suppose that I’ve tended to do, that’s what I’ve taken from it in daily life … sometimes with children you just sort of tell them what to do, you don’t really say why you’re doing it, what the benefits are, what might happen if you do this. You sort of skip all of that presuming they’ve got the knowledge and then just go in with what you need them to do. The message was to try and ask for help in class, have the confidence to do that and there was sort of so many levels of why that would be good and what would happen … I think it just increased awareness around why that was a good thing to do and it helped me with other things really. Kind of communicating other messages to my son rather than just skipping to the purpose of the request and kind of actually taking the time to explain and maybe verbalise a social story sometimes when doing things.

Parent 6

The 6-month follow-up questionnaire asked interventionists to rate how useful they felt Social Stories have been (‘0’ not useful at all, ‘5’ somewhat useful and ‘10’ extremely useful). The most frequent ratings fell between 5 and 8, with 23.6% of interventionists responding ‘8’. No interventionists rated the usefulness of Social Stories between 0 and 3, reflecting the mostly positive views of the intervention held by the interventionists.

What are the considerations to make when implementing Social Stories in the future?

Lastly, we use our qualitative and quantitative findings to propose a series of areas to consider when informing how Social Stories could be used and implemented outside of the ASSSIST-2 trial. Table 44 outlines areas for consideration when using and implementing Social Stories in the future and is based on the qualitative study results and informed by our public involvement representatives, who met to review the results on 27 April 2022.

Trial population

This trial aimed to evaluate whether Social Stories is a clinically effective and cost-effective method of improving social responsiveness in children on the autistic spectrum. A total of 87 schools were randomised, representing 249 autistic children. This was lower than the initial target sample size of 278 pupils, with the target sample size being revised following the impact of the COVID-19 pandemic. Participants were recruited via multiple pathways, although direct recruitment of schools was the most successful. Despite all children being under the care of the NHS, this pathway yielded only three pupils from three schools, as most schools were already in discussion with the trial team through the direct school recruitment pathway by the time NHS referrals were received.

The average age of the participating children was 8.6 years, and three-quarters of children were male, as is typical for the population of individuals with ASC. 67 Most children were described by their parents as having a white ethnic background (88.3%). People delivering the intervention were mostly female (93.5%) and TAs (83.7%), with an average age of 42.6 years. Most considered themselves to have moderate or greater levels of knowledge and experience of working with autistic children (83.8%). Exhibiting behavioural challenges in school was criteria of participation in the trial. The challenges faced by autistic children that were supported through the Social Stories broadly fell into five categories: social interactions; emotional development; routines and change; classroom behaviours; and more general life skills.

Primary outcome

At baseline, children scored 72 points on average on the SRS-2, indicating a moderate level of difficulty. The average SRS-2 scores decreased in both groups at 6 weeks and 6 months. At 6 weeks, children allocated to Social Stories scored on average 1.14 points lower than those who received usual care (95% CI −3.35 to 1.06; p = 0.310). This difference increased slightly at 6 months to 1.61 points (95% CI −4.18 to 0.96; p = 0.220). However, at both time points, the observed difference was much lower than the 3 points postulated as clinically important, and neither difference was considered statistically significant.

After accounting for compliance with the intervention, defined as children receiving the intended six or more story sessions, a CACE analysis found that Social Stories led to a −3.37-point reduction in social responsiveness at 6 months for participants in the complier principal stratum (95% CI −6.65 to −0.10). However, some caution is required in interpreting these estimates because a substantial number of intervention group participants were excluded from the estimation due to having missing compliance data. Two post hoc sensitivity analyses to assess the possible influence of the missing compliance data suggested that the magnitudes of the CACE estimates were partly explained by selection bias arising from exclusion of intervention participants with missing compliance data. However, the results suggest the data were most compatible with hypotheses positing moderate effects of the intervention when Social Stories is received as per protocol.

Given that much of the compliance issues were thought to be due to the impact of COVID-19, rather than feasibility or systemic issues delivering the intervention, two sensitivity analyses were undertaken to specifically explore the magnitude of this impact. In the first, we investigated the extent to which disruption of follow-up due to the pandemic was associated with variation in treatment effects. The estimates generated through the statistical model provided some evidence that disruption to follow-up due to school closures was associated with variation in treatment effects at both 6 weeks and 6 months. This finding was somewhat replicated in the second analysis which explored the extent to which disruption of follow-up and the shifting of school training to an online model were associated with variations in treatment effects. Again, estimates generated from this model provided some evidence that the impact of Social Stories was greater at the 6-month time point amongst those who were randomised and followed up prior to the beginning of lockdown.

Taken together, the CACE and COVID-19 sensitivity analyses do indicate that poor compliance may have reduced the effectiveness of the intervention. This is further supported through the process evaluation, where teachers and interventionists frequently reported COVID-19, staff shortages and time issues to be key barriers for successful intervention implementation. Of the interventionists that returned 6-month questionnaires, a large number (29.2%) reported encountering problems during story sessions, with many reporting difficulties keeping the children engaged and issues with the repetitive nature of the intervention.

Secondary outcomes

There were no differences in any of the measures assessing children’s mental health and well-being or parental stress. However, in terms of the specific goals against which behavioural outcomes were measured, children who received Social Stories typically met their behavioural goal more frequently after 6 months than children who received care as usual (p = 0.012).

Cost-effectiveness

Compared with usual care, Social Stories slightly decreased the mean service use costs while maintaining similar QALY improvement. This was shown in both the primary and sensitivity analyses but subject to the given data with the uncertainty related to short study time frame, amount of missing data and the disruption of COVID-19. The results show a small reduction in overall costs across different perspectives, particularly through the cost reduction from the education sector, such as the costs of school-based health services and the costs of general support from teachers and/or TAs at school. Although school-based intervention costs were found to be higher in the Social Stories arm, it appeared to be more related to a small number of children receiving high levels of high-tariff 1 : 1 mentoring sessions in the Social Stories arm than a general cost increase across the whole group (see Chapter 4). This is a relatively positive finding in that Social Stories seems to be able to reduce the care needed in schools without affecting children on the autistic spectrum receiving school-based interventions. Overall, within a 6-month time period, compared with receiving usual care, there is around £200 saving per child (from the societal perspective) for receiving Social Stories. The cost saving is relatively small, but this cost saving may be relevant for commissioners and education sectors when considering resource utilisation if multiplied by the number of children on the autism spectrum in the UK. Owing to the fact that there are currently 166,041 autistic CYPs in schools in England and more than 70% of autistic CYPs are educated in mainstream schools,68 the potential cost savings to the NHS and education system could reach around £28 million for the CYP population as a whole. It is worth noting that although the cost reduction, especially in the education sector, is a rather positive finding, it needs to be interpreted with caution. This is because the cost reductions were relatively small no matter which perspective was taken, and the cost differences were not statistically significant, showing the uncertainty of the cost results.

On the other hand, the difference in QALY improvements between the two trial arms was very small (see Table 41). After taking uncertainty into consideration and adjusting for the imbalanced utility, cost and other characteristics at baseline, the difference in QALYs remains small (< 0.001 QALYs), suggesting that the Social Stories do not appear to improve the QoL of children on the autism spectrum based on the dimensions measured by the EQ-5D-Y.

Process evaluation

The intervention was facilitated by way of a cascade or ‘train-the-trainers’ training model whereby a child psychiatrist (Professor Barry Wright) trained one member of the trial team, who in turn trained other members of the trial team. These members of the team then became the ‘trainers’ responsible for training the educational professionals within schools. Training and support appeared to be well received at each level. Trainers felt sufficiently prepared to deliver training to educational professionals, who, in turn, felt well-prepared to deliver the intervention to pupils. Schools generally felt supported by trainers. Owing to the COVID-19 pandemic, the training model had to be adapted part-way through the trial to move the training online. There were mixed feelings around this, with some educational professionals and parents valuing the shorter sessions and improvements in access; for example, some parents were able to join videoconferences while at work or while caring for other children at home. Others valued the face-to-face interaction.

The process evaluation provided data surrounding the implementation of the Social Stories intervention. The majority of educational professionals appeared to grasp fully the core principles of Social Stories and, in most cases, were able to deliver them with fidelity. Around 30% of interventionists reported that they encountered some barriers in delivering Social Stories. These barriers were largely reported during the latter half of the trial, with professionals citing the COVID-19 pandemic and associated difficulties families, schools and the research team encountered. Other logistical issues, such as staff changes and school holidays, were also cited.

Comparison with current evidence base

Several systematic reviews have been conducted to synthesise the existing evidence base around Social Stories. The first review published in this area concluded that the effects of Social Stories in general (i.e. not specifically Carol Gray’s Social Stories) are highly variable. Twelve of the 16 studies examined in the review used a single-subject design, and the authors highlighted that it is difficult to interpret the results due to inadequate participant descriptions and because Social Stories were often used in combination with other interventions. 23 Another systematic review concluded that Social Stories in general (i.e. not specifically Carol Gray Social Stories) may be substantially more effective when used to target behaviour reduction than to teach appropriate social skills and appeared more effective when the story focused on a single behaviour rather than more complex social situations. The authors suggested that this may be because social behaviours are more abstract and complex, and thus more difficult for children on the autism spectrum to understand. 22

In the most recent systematic review conducted by Wright et al.,24 any study of Social Stories that used a standardised, numerical measure of outcomes or a non-standardised, numerical outcome measure was included. Single-case designs reporting repeated measurement of the target behaviour were included alongside seven between-group studies, four of which were RCTs. 25–28 Outcomes included social abilities and awareness, communication, restricted behaviours, life skills, emotional development and sustained attention. This review highlighted that although randomised studies appear to provide broadly supportive evidence of an effect of the Social Stories interventions, the findings from the studies are compromised as there were gaps in the reporting of results, which made it difficult to quantify the level of effect of the intervention and to assess the risk of bias in the studies. The Social Stories interventions examined in the RCTs included in the review were not individually tailored to the child or young person and therefore did not meet the Carol Gray criteria and were delivered under experimental conditions. It is therefore difficult to compare the current findings with those of the previous RCTs, with the exception of our feasibility RCT,25–28 which demonstrated a high degree of acceptability and some positive findings that were not subjected to statistical analysis because of the feasibility nature of the study.

Taken together, these reviews demonstrate that the effect of Social Stories is highly variable. These studies are likely to have included stories of varying quality, as none specifically evaluated Carol Gray’s Social Stories. This study examined Social Stories that met the Carol Gray criteria and were delivered within a school setting that was familiar to the child, with the intervention delivered six times over 4 weeks, whereas in the previous RCTs, they were delivered over a much shorter time frame (e.g. exposure to the story five times over the course of 1 day). 28 Another important factor to consider is that the intervention is complex. The Social Story is tailored to the child and may target a range of different goals that are difficult to combine in measuring outcomes. Our goal-based outcomes showed more promising results than our validated measure of social responsiveness. Not all of our goals were directed at social responsiveness, as many teachers or parents may not have chosen this as important for the child at this time. Goal-based outcome measures are more flexible in this regard but are less respected in some areas of the scientific community. 69,70 The intervention also involves other potential sources of benefit for the child, including the fact that the intervention usually requires the teacher and parent to converse about the child’s needs and to involve the child where possible, all of which may carry benefits that are difficult to disentangle from the reading of the story itself. The quality of these latter processes was greatly impacted by the COVID-19 pandemic. This study was a pragmatic trial in the real world involving educational professionals, parents/carers and children from a diverse set of mainstream and SEN schools across Yorkshire and the Humber and therefore the results may be more representative of the effectiveness of the Social Stories intervention for supporting children on the autism spectrum compared to previous results. Overall, this full-scale RCT has added to the current literature which includes mainly single-case studies and a small number of RCTs (with methodological limitations) conducted primarily in the USA.