The Social and Emotional Education and Development intervention to address wellbeing in primary school age children: the SEED cluster RCT

The primary school setting

There is evidence that programmes addressing the underlying determinants of non-communicable health conditions and risk behaviours need to be introduced in the early years of primary school and sustained over time to include key transition periods. 7,10,27 Improved SEW during primary school has been shown to have an impact on physical health and to be a protective factor against a range of risk behaviours in later years, including tobacco use, illicit drug use and alcohol misuse; violence and crime; and teenage pregnancy. 7,10,28–32 A 2011 review and meta-analysis including > 200 controlled studies of school-based interventions designed to enhance the social and emotional skills of children aged 5–18 years27 found positive benefits on a range of outcomes, including significant improvements in social and emotional skills, attitudes and positive social behaviours, and academic performance. Maximum benefits were observed when the programmes were evidence based and well implemented by school staff. In addition, there is strong evidence that interventions with a longer duration (i.e. multiyear interventions) are likely to have a greater effect than shorter-term programmes. 7,10,33

Other systematic reviews of the effectiveness of universal interventions to improve the mental well-being (encompassing emotional, psychological and social well-being) of children in primary education found that curriculum-only interventions appear to be effective only in the short term. In contrast, there is good evidence to support the use of programmes that combine a social and emotional development curriculum with components that focus on behaviour management and improvement of child–teacher relationships. 7,10,28–32,34 Furthermore, programmes that include a significant teacher training component show considerable promise, as do those that include a parenting support component. 7,10,28–32,34 The multicomponent intervention approach appears to be particularly effective in improving mental health, as well as reducing bullying and violence. 7,10,28–32 One of the key studies in this area is the Seattle Social Development Project (now known as Raising Healthy Children), which was implemented in year 1 of elementary school for 6 years, with follow-up into adolescence and young adulthood. The programme sought to promote connectedness to school and family and to strengthen children’s social competencies. It consisted of three components: teacher training, child social and emotional skill development, and parent training. Follow-up at age 21 years revealed significant reductions in the intervention group, compared with the control group, in health risk behaviours (including alcohol, tobacco and illicit drug use and sexual risk behaviour), violence and crime, and emotional and mental health issues, and increases in positive functioning in university or work. 7,10,35–38 Some of these effects remained significant when the study population was followed up at age 30 years. 39

When devising the SEED intervention, there was little robust evaluation of interventions aimed at improving SEW in UK primary schools. One exception is a recent cluster randomised controlled trial (RCT) of the Roots of Empathy (ROE) intervention, carried out in 74 primary schools in Northern Ireland. 40 ROE is a Canadian programme designed to promote empathy in children through demonstration of the attachment relationship. 41 It consists of annual visits from a mother and infant to the school over the course of a year and is designed to facilitate the labelling of feelings and the exploration of the relationship between feelings and behaviour. The investigators found that, among the intervention group, children (year 5, aged 8–9 years) were rated by their teacher as being more prosocial and exhibiting less difficult behaviour post intervention. However, only the latter was sustained over the 36-month follow-up period. No difference in effect by gender or socioeconomic status was observed. An economic analysis indicated that the ROE intervention was cost-effective. These findings are in accordance with those of the only other cluster RCT of ROE,42 which reported improvements in prosocial behaviour (although this was not maintained over time) and reduction in aggressive behaviour in the short and longer terms.

Whole-school intervention approaches

Whole-school interventions aim to adjust the school environment in an effort to improve health and have been shown to improve social competence and reduce aggression and health risk behaviours. 43–50 A comprehensive review of the impact of school environment interventions in both the primary and secondary school settings on health and well-being51 identified 10 experimental or quasi-experimental studies with a variety of outcomes. Interventions fell into one of three categories: promotion of sense of community and better interpersonal relationships to reduce aggression and other risk behaviours; encouragement of advocacy by staff and pupils for healthier eating and physical activities; and improvement of school playgrounds. Results were mixed, with many studies having important methodological limitations. The authors51 concluded that school environment interventions have the potential to reduce violence and aggression, but highlighted the need for more robust intervention studies. A parallel Cochrane review synthesised the evidence from cluster RCTs of multicomponent interventions aiming to improve health in schools that included input to the curriculum, changes to the school ethos, and family or community engagement. 52 Although the authors found some evidence for effectiveness, such as reduced bullying and improvement in some health behaviours, including smoking, physical activity and dietary outcomes, there was little evidence of effect on other outcomes, such as alcohol use and mental health. Few studies reported on these last two outcomes, and the methodological quality of included studies was generally considered to be low to moderate.

Some of the best evidence for the effectiveness of the whole-school approach within the secondary school setting comes from the Australian Gatehouse Project, which was designed to improve SEW through the promotion of a sense of social inclusion and connection in secondary schools. 53–56 Importantly, the strategies used to achieve this varied between schools, according to pupils’ perceptions of need, with the conceptual framework focusing on three ways of strengthening connectedness to school: (1) building a sense of security and trust, (2) enhancing communication and social connectedness and (3) building a sense of positive regard through valued participation in aspects of school life. 57,58 The effectiveness of the Gatehouse Project was evaluated using a RCT design. 57 Although the Gatehouse Project did not affect depressive symptoms or social and school relationships,53–56 it reduced substance use for a cohort of pupils in the intervention schools followed up longitudinally 2 and 3 years after the intervention began. 53–56 Subsequent year-8 pupils (aged 13–14 years) in intervention schools, surveyed 5 years after the trial began, also reported lower rates of substance use than those surveyed in the control schools. 57,58 Both the Seattle Social Development Project8,11,36–39 and the Gatehouse Project included changes to the school curriculum that focused on developing or improving SEW.

There is also some evidence that targeted programmes can have an impact on SEW. A review of targeted programmes in primary schools reported modest improvement, particularly in social problem-solving and development of positive peer relations, with lengthy, multicomponent programmes. 59 The evidence therefore supports an integrated approach that provides targeted support for those experiencing particular difficulties within a supportive whole-school approach to promote mental well-being. 26,43 Furthermore, a review of family-based programmes to promote mental well-being10,60 supports the need for generic programmes to promote mental well-being at a population level and more intensive programmes for more serious problems among individuals. National Institute for Health and Care Excellence (NICE) guidance on promoting children’s SEW in primary education strongly supports the adoption of universal approaches, which have the capacity to address emotional well-being in a connected way that reduces potential for stigmatisation, but which also include provision of targeted approaches and early identification of children at risk. 61

Component 1: data collection – pupils’ social and emotional well-being and school culture

Pupils’ SEW was assessed using the SDQ. Data were gathered as part of the trial at baseline [i.e. time 0 (T0)] and at follow-ups 1 and 2 [i.e. time point 1 (T1) and time point 2 (T2), 2 years and 3 years post T0, respectively], which allowed three intervention cycles over the course of the study. Additional questions captured data on pupils’ friendships with peers, relationships with staff and expectations of transition to secondary school. Staff perspectives of school culture were assessed through a survey that included the HTs, teaching staff, classroom assistants and support staff. The survey questionnaire included questions on staff stress and distress, whole-school relationships, perceptions of pupil behaviour and well-being, and school ethos. Parents’/carers’ perceptions of the school were captured in a parent survey that included questions on school ethos and environment. For further details of the questionnaires and survey methods, see Chapter 3.

Component 2: feedback reports – analysis and feedback to schools

The data from component 1 were analysed and compiled into individualised reports for each school. To facilitate comparison and reflection, the data were aggregated by cohort (i.e. P1, P5, staff and parents). For T1 and T2, the reports depicted the changes in scores from T0 to T1 and T2. In addition to the individual school reports, an ‘all schools’ report was produced depicting the same data collated across all intervention schools involved in the SEED RCT. Each school was positioned relative to the others in either graphic or tabular form to facilitate comparison (two examples relating to T0 and T2 are available in Report Supplementary Material 1 and 2, respectively).

A SEED resource guide (see Report Supplementary Material 3) was also provided to staff that contained an up-to-date and comprehensive review of programmes to help pupils improve their social and emotional education and well-being. All known programmes delivered to pupils, staff or parents in Scotland that were evidence based or had a strong theoretical underpinning were included. The content of the resource guide was identified through our systematic search and a review of relevant websites, and further information was sought from programme providers if possible. The resource guide was greatly assisted by two existing reviews: the American website Collaborative for Academic, Social and Emotional Learning (CASEL)33,73 and the Australian website KidsMatter. 74 All resources included in the guide were rated in terms of the strength of evidence of effectiveness and were ideally endorsed by the Scottish Government as aligning with the central ethos of the CfE. 75 Summary information was provided for each programme, including the target population, the aim, key components, effectiveness and links to further information. The guide was updated after 2 years.

Component 3: presentation and reflective discussion

Staff from each school chosen to implement the SEED intervention were invited to a presentation and reflective discussion (RD) session in school. The data outlined in component 1 were presented, followed by a discussion, aimed at providing a safe space for open discourse and identifying and prioritising key actions. EPs were included because of their pivotal role in providing additional support to pupils and staff. The teaching staff were encouraged to co-present the data (with the research team); when possible, the EPs co-led the RDs with teaching staff.

The RD sessions aimed at developing a commitment to positive change, co-producing tailored school-wide initiatives and establishing broad support for those initiatives. An assets-based approach was adopted whereby feedback was provided using positive language with reference to best hopes and preferred futures. When considering how to respond to feedback, staff were encouraged to be aware of their goals, considering their preferred options and existing teaching commitments. Reflection was encouraged as part of a continual process of change and development and not an annual event, nor an end in itself.

It was recognised that reflective sessions could be considered sensitive; therefore, the senior management team of each school was given the option of having a separate presentation and RDs. It was impractical to invite parents to the feedback sessions, but schools were encouraged to share results of the needs assessment with parents in other ways, for example through newsletters or electronic communications.

Component 4: selection of actions

There were several co-produced ground rules when selecting actions. Agreed actions should be practical and not add excessively to existing workloads, and preferred options should be aligned with the ethos and strategic goals of the school. The RDs should lead to the formulation of an AP (see Appendix 5), either by all staff or through an ‘action team’ subgroup, which was sometimes the senior management team. Staff were encouraged to use the SEED resource guide to help in their selection of appropriate actions and activities with short-, medium- and long-term time frames. It was recommended that the key priorities, and how these would be implemented, be set out in the APs and focus on:

• the classroom, for example use of existing interventions such as Creating Confident Kids and Cool in School

• whole-school initiatives, including training for staff and parents, supporting the implementation of restorative practices aimed at promoting proactive classroom management and interactional instruction, the understanding of the importance of the SEW of children and the opportunity of being positive role models. 18

When possible, the APs were incorporated into the school improvement plan, which schools are obliged to submit to their local authority (LA), typically in a 1- to 3-year cycle.

Component 5: implementation

After the development of APs, schools implemented their activities, supported by their EP and, when appropriate, QIO. The SEED RCT researchers maintained contact with schools during this period, but the intention was that schools should use their usual support systems to implement and maintain initiatives to replicate how the intervention would work in a real-life setting.

Meetings were held with EPs from LAs to discuss the delivery of the intervention and the role of the EP in that delivery. The dominant role of the EP was that of facilitation based on appreciative enquiry, principles that would be familiar to EPs and therefore require minimal training. EPs working in Scottish schools are commonly accredited as chartered psychologists by the British Psychological Society and have membership of the Scottish Division of Educational Psychology. QIOs were also invited to these meetings, although they were less involved. Qualifications for QIOs are unclear, but this role would be likely to require a formal teaching qualification and extensive experience in LA educational settings, including leadership roles and meeting national education priorities.

Socioecological framework

The socioecological framework76–78 distinguishes different levels of social influence that operate simultaneously and interact with each other. The levels are commonly described as intrapersonal (e.g. genetics, personality), individual (biographical factors, attitudes and beliefs), interpersonal, community, organisational and macro. The SEED intervention operates at four of these levels:

1. The individual, for example changing pupils’ and staff cognition and attitudes. These include raising staff members’ awareness of SEW as a result of participating in the presentations and RDs, and improving pupils’ core competencies such as self-awareness and social awareness through tailoring the classroom curricula.

2. The interpersonal, such as improving relationships between staff and pupils through greater focus on SEW and between staff by engaging in RD.

3. Organisational, for example changing the culture of a school through a collective commitment to address SEW and develop trust between staff and pupils.

4. The SEED intervention also interacts with macro-level influences, in particular the policy environment as set by the Scottish Government’s CfE17 and education authorities’ translation of that policy at local level.

Co-production

The term ‘co-production’ describes an equitable approach to intervention innovation, development and evaluation. The fundamental principle is that users design and deliver services in equal partnership with professionals. 67 Greenhalgh et al. 79 set out three key principles required for effective co-production in community-based research:

1. A systems perspective comprising an iterative and flexible design that adapts an intervention to local need. This was of central importance to the SEED intervention, including the choice of activities through RD.

2. The framing of research as a creative enterprise with human experience at its core. Research based on pupils’ SEW was the starting point of each of the intervention cycles in the SEED intervention.

3. An emphasis on process, including the framing of the programme, the nature of relationships, and governance and facilitation arrangements, especially the style of leadership. HTs and EPs were identified as local leaders and facilitated the SEED intervention through appreciative enquiry.

Social learning theory

Social learning theory posits that new behaviours can be learnt, and existing behaviours changed, through modelling behaviour, either by demonstration or being taught. 80,81 The concept of ‘modelling’ involves the development of self-efficacy, intentions and planning, and modification through social approval. 80,81 People observe credible role models, with whom they can identify, engaging in particular behaviours. They see the benefits of these behaviours and are motivated to adopt similar behaviours. This learnt behaviour is then positively reinforced by significant others. For example, it was important that staff took ownership of the SEED intervention, and, as part of their collective commitment, they modelled SEW by demonstrating their emotional competency to each other and their pupils.

School connectedness

Connectedness refers to diverse aspects of people’s social experiences, ranging across systemic dyadic relationships, perceptions of relationships, satisfaction with institutions and feelings of belonging. It is associated with a host of positive health outcomes for children and young people. 82 The concept of connectedness is very similar to Markham and Aveyard’s,83 Bonell et al. ’s,47,48,51 Jamal et al. ’s49 and Naghieh et al. ’s50 notion of pupils being ‘committed’ to their school, in that they are able to meet the challenges of learning and accept the school’s norms of behaviour and ethos.

It was decided that dedicated time for staff to better understand the school’s SEW needs, to reflect on these and to discuss how to respond would have a favourable impact. First, the SEED intervention was designed to enhance staff awareness and understanding of both pupils’ and staff members’ SEW problems. Second, the discussion of SEW was expected to lead staff to express and share problems, which would, in turn, improve collegiate relationships. These two processes would enhance motivation and collective commitment for positive change to tackle and prevent SEW problems.

Delivering classroom and environmental SEW interventions should lead both staff and pupils to become more aware of, and practise, their SEW skills and lead staff to reward pupils who exercise these skills. If successful, this should develop pupils’ core competencies such as self-awareness, self-management, social awareness, relationship skills and responsible decision-making, thus contributing to the ‘instructional order’. 83 Another pathway to these outcomes is through the ‘regulatory order’,83 whereby rewarding SEW skills contributes to pupils’ connectedness and commitment to their school. 47–51,82 Theoretically, improvement in school connectedness should lead to reductions in disruptive behaviour, poor well-being and absenteeism among pupils. 84

The next chapter will present the methods for the outcome, process and economic evaluations.

Trial setting

The trial recruited 38 primary schools from 18 learning communities across three Scottish LAs, anonymised as LA1, LA2 and LA3 in this report. There were four waves of data collection in the main study [baseline (T0) and three follow-ups: T1, T2 and time point 3 (T3)] and a further fifth wave [time point 4 (T4)] following the prespecified primary outcome at T3. The baseline (T0) and the first follow-up (T1) data collection and intervention delivery took place in primary schools. Data collection for the OC of pupils at the second (T2), third (T3) and fourth (T4) follow-ups took place in secondary schools.

Changes to trial design

The SEED intervention was registered with an International Standard Randomised Controlled Trial Number (ISRCTN) in April 2013 (see https://fundingawards.nihr.ac.uk/award/10/3006/13 and https://doi.org/10.1186/ISRCTN51707384). No changes were made to the trial design, aims or primary and secondary outcomes after registration. However, we did secure additional funding, also from the National Institute for Health and Care Research (NIHR), to conduct longer-term follow-up (T4) with the OC just before they reached the end of compulsory education (aged 16 years).

Eligibility criteria for local authorities

Any LA within a reasonable geographical location accessible to the main study site was eligible for participation. Any LA with involvement in the SEED pilot was excluded, as were LAs that required opt-in consent for parents. LAs known to be taking part in studies similar to the SEED RCT were also excluded. Three eligible Scottish LAs were selected based on the spread of urban and rural populations, socioeconomic diversity and reasonable proximity to the research unit base.

Eligibility criteria for schools

Within each of the three selected LAs, all state primary schools and associated secondary schools were eligible. Special schools (offering specialist education independently of mainstream school base), independent schools and home-educated children were excluded (accounting for ≈ 4.3% of primary school children in Scotland85–88).

Eligibility criteria for participants

Eligible staff included all teaching and non-teaching staff within study primary schools. At T0 all primary school pupils in P1 (aged 4–6 years; YC) or primary school year 5 (P5) (aged 8–10 years; OC) in the 2012–13 academic year were eligible to participate. Baseline (T0) measures were collected in academic year 2012/13, and follow-up 1 (T1) measures were collected in academic year 2014/15. Because teachers or classroom support assistants completed the SDQ on behalf of their pupils, individual pupils were not excluded on the basis of additional support needs or language difficulties unless it was decided, in liaison with the class teacher and HT, that the SDQ would not be a valid measure for a particular child. Pupils completing the self-completed questionnaire were given additional support when needed, unless it was decided, in liaison with the class teacher and HT, that the pupil’s level of understanding of the questions would not allow them to complete the questionnaire accurately (e.g. not understanding spoken English or severe learning difficulties) or when participation would be detrimental to the young person’s well-being.

Local authorities level

The directors of education in the selected LAs were approached by e-mail and telephone. This was followed by meetings with nominated EPs and QIOs. Three LA directors of education granted permission, and we were instructed to contact primary schools directly, inviting them to take part. One LA refused because it was in the middle of a restructure.

School level

Purposive sampling was used to recruit 38 schools across the three LAs between November 2012 and February 2013. Demographic information for the participating LAs was collated for all potential primary schools, and clustered learning communities were identified. We prioritised inviting schools that enabled us to represent LA levels of rurality, denomination, school size and proportion of pupils eligible for free school meals (as a proxy for deprivation).

We aimed to recruit up to three primary schools belonging to the same learning community to limit the number of secondary schools needed for data collection following the OC’s transition from primary to secondary school, and owing to limits on the number of learning community clusters available in each LA. Invitation letters and participant information sheets (see Report Supplementary Material 4) were sent to primary school HTs by post, with follow-up telephone contact and visits in person by the research team if requested by the school. Two further rounds of selection and invitation were necessary owing to lack of response or schools declining to participate. In total, 91 schools were approached, and recruitment stopped (in February 2013) when 38 primary schools from 18 learning communities had agreed to participate: 15 schools in LA1, 13 in LA2 and 10 in LA3. Once a school agreed to participate, it was sent an information pack with a checklist of next steps and asked to nominate a key contact to liaise with the trial team.

At the end of T1 (early summer 2015), link secondary schools in participating primary schools’ learning communities were contacted to give notice of the trial and to invite them to support the ongoing data collection for the participating pupils joining their schools in August 2015.

Individual level: school staff, pupils and parents

As the SEED intervention operated at the whole-school level, once a school had agreed to participate, it was not possible for school staff or parents/carers of individual pupils in intervention schools to opt themselves or their child out of the intervention itself, only the data collection for evaluation. For the data collection, individual-level recruitment was required for two cohorts of pupils in each school, their parents/carers and all primary school staff; informed consent was sought for all.

Retention

To optimise retention among trial schools, each participating school was given £500 on completion of baseline data collection and a further £500 following the final follow-up (T3). In addition, a payment at each wave was made to compensate for the time taken by teachers to complete SDQs for their pupils at a rate of £4.38 per participating pupil, calculated on the basis of the cost of 10 minutes of primary teacher cover.

Throughout the life of the trial, the research team maintained communication with schools, staff and parents, providing periodic updates on the status of the trial and responding to any requests for information in a timely manner. Christmas cards were sent to all participating schools each year. During the trial, several primary schools changed HTs or key trial contacts. The new HTs were contacted by e-mail in the first instance, and this was followed up by telephone calls and/or meetings in person to explain the trial and the school’s involvement in it. In all these instances, these schools were retained in the trial.

For staff and parents, reminder letters were sent with the questionnaires in each wave. Schools were encouraged to let parents of the participating cohorts know of the trial through school newsletters to raise awareness and encourage participation. When possible, schools were asked to set aside collegiate time to complete the SEED survey to optimise participation. Regardless of how it was administered, staff were told that they could return their questionnaires in a prepaid envelope directly to the research unit to reassure about confidentiality of responses.

Over the course of the four waves of data collection, a small number of pupils who had previously been opted out were opted back in by their parents/carers, and a small number of pupils were opted out. Some pupils moved between trial schools, and some pupils left trial schools and were not located in other trial schools.

Withdrawals

The right to withdraw from data collection was emphasised to all individual pupils, parents/carers and staff at each wave. Pupil and parental opt-outs, withdrawals and pupil refusals (pupils who had been consented by parents/carers, but who opted themselves out of the self-completed questionnaire) were recorded with reasons for withdrawal, when known, and a check to ensure that this was actioned on the trial participant database.

Confidentiality and anonymity

All data were anonymised. Pupil and staff participants were allocated unique identification numbers (IDNOs) and barcodes were used on all questionnaires and detachable pages to account for and track returned data. Identifying information linking participants to IDNOs was stored securely in a database accessible only by main research and Population Health Research Facility (PHRF) staff at the MRC/CSO SPHSU, University of Glasgow, and held separately from participant questionnaire responses. Questionnaires for parents/carers of participating pupils were allocated the IDNOs for the pupil they were being asked to complete questionnaires about. All questionnaires were designed with detachable pages so that names could be removed from blank questionnaires before completion and identifying data (e.g. postcode or job title) would be returned, processed and stored separately from other non-identifying questionnaire responses.

Results forming part of the intervention were fed back to schools randomised to the intervention in aggregate form, broken down by year group. Neither staff nor parents were informed of individual pupils’ SDQ scores. If parents contacted the trial team requesting to know their child’s SDQ scores, we reminded them of our commitment to confidentiality regarding individual scores. Responses from free-text fields in the staff and parent questionnaires were fed back to schools in a way that removed any identifying information.

Care was taken to conduct interviews ethically and sensitively. Complete confidentiality of interview content and recordings was ensured, and best efforts were made to ensure that quotations and comments could not be traced back to the interviewee. Participating schools will not be identified in any publication and pseudonyms will be used when necessary.

Classroom-based fieldwork

Pupil self-completed written questionnaires (including, but not limited to, the SDQ) were administered in a classroom session lasting approximately 1 hour, supported by a team of trained fieldworkers from the PHRF.

Classroom sessions were scheduled in liaison with the school for a suitable time during school hours and a detailed overview of the session was given to schools in advance. For primary school sessions, schools were asked to block off 1 hour. For secondary school sessions, two periods (40–50 minutes each) were usually allocated, with pupils returning to class either at the end of the first period or whenever they completed the questionnaire during the second period.

In both primary and secondary settings, at the beginning of the session, fieldworkers introduced their team to the pupil group; explained what the session would entail; emphasised that consent had been given by their parents/carers, but that pupils could opt not to complete the questionnaire if they wished; and answered questions. Throughout the session, pupils completed their individual questionnaires supported by fieldworkers who answered questions as they arose or offered one-to-one support if pupils required it (e.g. reading the questions aloud). During questionnaire completion, care was taken to ensure that pupils had as much privacy as possible. Pupils were requested and encouraged to complete questionnaires quietly on their own and not to confer with neighbours. At the end of the session, pupils and staff were thanked for their participation and support and pupils were given a debrief leaflet (see Report Supplementary Material 9) that included a telephone number for ChildLine, and a verbal explanation of the leaflet.

Returned questionnaires

Teacher-completed SDQs (for YC pupils in all waves and OC pupils at T0 only) were delivered to the school along with instructions on how these should be completed, deidentified and returned. The majority of these were then collected in bulk from the school by fieldworkers, but small numbers were returned in Freepost envelopes to the research base.

Staff and parent/carer questionnaires were packed in individual questionnaire packs containing a participant information sheet (see Report Supplementary Material 4) and Freepost return envelopes. Staff and parents/carers were given the option of returning these individually by post in the envelopes provided or by leaving them with the school office, where a SEED fieldworker would collect them. Instructions were given on how to detach pages that contained identifying data, and participants were asked to return these in Freepost envelopes separate to the main questionnaire.

Data processing, entry and management

A Microsoft Access^® (Microsoft Corporation, Redmond, WA, USA) database was used to generate IDNOs, to securely store participants’ names (and associated school details) and to log and track questionnaires via barcodes attached to each questionnaire/detachable page. Anonymised questionnaires were passed to the Robertson Centre for Biostatistics (RCB) Clinical Trials Unit at the University of Glasgow for data entry, then returned to the SPHSU for further data entry and later archived. The detached identifying data and all free-text data were entered in a blind two-pass system by two different PHRF staff members onto a Microsoft Access database. Any disagreement between these passes resulted in the entries being flagged for a third pass. If there was any further disagreement about what had been written by participants or uncertainty about what should be entered, the research team was consulted.

Pupil self-completed questionnaires that were completed electronically at T4 were transferred directly to a secure PHRF database, removing the need for manual entry. Anonymised data were then passed via secure file transfer to the RCB.

On return from the RCB, anonymised paper questionnaire data were stored in a locked, secure data room in the PHRF. Electronic questionnaire data entered by the PHRF were stored confidentially on password-protected servers. All data collected were stored securely in compliance with the General Data Protection Regulations for the European Union and UK (URL: https://gdpr-info.eu). Data were kept in either locked filing cabinets or password-protected databases accessible only by main researchers and PHRF staff. The final data set was accessed only by approved members of staff from the research team and the RCB. All data will be kept for 10 years, in line with University of Glasgow research governance framework regulations for clinical research.

Primary outcome measures

The primary outcome measure for pupils was the SDQ-TDS. 32,69–71 This is an internationally validated, publicly accessible 25-item behavioural screening questionnaire designed to assess levels of emotional and behavioural problems comprising four ‘difficulties’ subscales and one ‘strengths’ subscale. The SDQ-TDS is calculated by summing the scores on the four difficulties subscales.

The pupil self-completed and parent/carer SDQs were administered as part of the larger pupil/parent questionnaires (see the T0 pupil questionnaire in Report Supplementary Material 7, the T0 parent/carer questionnaire in Report Supplementary Material 8 and all changes to questionnaires from T0 in Report Supplementary Material 6). The teacher-completed SDQ was administered as a stand-alone assessment measure given to teachers to complete in class time (see Report Supplementary Material 10). For the OC, the SDQ was self-completed at all five time points (including T4, when only the OC were surveyed), was completed by teachers at T0 and was completed by parents at T0, T1 and T3. The YC SDQs were teacher-completed at four time points (T0 to T3), self-completed at T3 only and parent-completed at T0, T1 and T3. See Appendix 2 for details of the SDQ scoring.

Secondary outcome measures

For pupils, the secondary outcome measures assessed in the pupil questionnaire were as follows:

• The five validated SDQ subscales: four difficulties subscales (emotional symptoms, conduct problems, hyperactivity/inattention and peer relationship problems) and one strengths subscale (prosocial behaviour). Higher scores represent greater issues in all the difficulties subscales, whereas lower scores represent greater challenges in the prosocial ‘strengths’ scale.

• Pupils’ social and emotional skills: self-awareness, social awareness/empathy, self-management, responsible decision-making, emotional regulation and self-esteem.

• Pupils’ well-being and relationships at school – school relationships; whole and class level, liking school, happy friendships, school climate, experience of antisocial behaviour and participation in antisocial behaviour.

• Pupils’ well-being and relationships at home – family relationships, materialism and daily quality of life.

These measures were collected through pupil self-completed questionnaires. The questionnaires were designed (some for previous studies such as Healthy Respect Phase 2,90 which involved some of the SEED RCT co-investigators) and refined through piloting. 68,91

The secondary outcome collected from the OC at T2 and T3 was the impact of the intervention on health-related behaviours (smoking, drinking and drug use).

For staff, the secondary outcomes were staff attitudes towards pupil behaviour, pupil confidence, pupil engagement and pupil relationships.

Additional quantitative staff measures were collected on staff–staff relationships, school ethos, self-efficacy, staff mental health [Warwick–Edinburgh Mental Well-being Scale (WEMWBS),9,10,92 used, with permission, from T1 onwards], perceptions of management, staff–pupil relationships, staff support, training opportunities, staff who played a key role in developing SEW in the school and school support for SEW. For a full list of measures, including sources (when applicable), see the T0 staff questionnaire in Report Supplementary Material 5 and all changes to questionnaires from T0 in Report Supplementary Material 6.

Changes to outcomes

The pupil domain scores were refined between baseline and the first follow-up to validate the groupings of questionnaire items fed back to schools as part of the intervention. Confirmatory factor analysis was carried out, which did not alter the items included in questionnaires, but changed the membership of items within some of the secondary outcomes (see Report Supplementary Material 5 for the T0 staff questionnaire and Report Supplementary Material 6 for all changes to questionnaires from T0).

Blinding

The trial researchers were not blind to intervention arm. PHRF fieldworkers were kept blind to which schools were intervention or control, but it was occasionally necessary for unblinded trial researchers to accompany and support fieldwork. EPs supporting the intervention delivery were not blinded.

Additional analyses

To test the robustness of the analysis, sensitivity analyses were performed by looking at alternative assumptions regarding missing outcome measures. Assumptions included last observation carried forward and return to baseline. This analysis was carried out on the primary analysis results only.

The secondary outcome repeated measures analysis models were extended to estimate the intervention effects within the gender, cohort and deprivation level subgroups by including interaction terms for each group in the model to provide separate intervention effects.

For the staff domain scores that were not included as secondary outcomes (staff–staff relationships, school ethos, self-efficacy, perceptions of management, staff–pupil relationships, staff support, training opportunities, valued member of staff and school support for staff emotional well-being), repeated measures analyses were carried out at all time points, as for the primary and secondary outcomes.

The SDQ-TDS was split into two groups using cut-off points (normal: 0–15; not normal: ≥ 16); using these subgroups at baseline, interaction terms were added to the SDQ subscale repeated measures models to determine the intervention effect within these subgroups. In addition, the responses to staff questionnaires were examined for the subgroups of teaching and non-teaching staff.

The OC had the additional T4 visit when the pupils were in fourth year of secondary school. Data collected at T4 were used in the analyses only if the pupil had data recorded for at least one of the first three follow-up visits. The main analyses conducted for the primary outcome were repeated to include these additional T4 data. This model was extended to estimate the intervention effects within subgroups of the population defined by gender (male/female) and deprivation level (deprived/not deprived). Repeated measures analyses were carried out at all time points (T0 to T4) to determine the effect of the intervention over the duration of the trial.

The analyses conducted for the secondary outcomes were repeated to include the data collected at T4, and the health-related behaviours were summarised for this follow-up visit, with Fisher’s exact tests carried out to compare the trial arms.

Ethics and consent

For the HT interviews, staff members were given information sheets and sufficient time to ask questions before deciding if they wished to take part. Written consent was obtained prior to conducting the interviews if held face to face. Verbal consent was obtained for telephone interviews and written consent forms returned by post following completion of interviews. Prior to interviews, informed consent was gained from the school HT and interviewee (if different). Parental consent was gained for participation in pupil focus groups and informed written consent was also sought from pupils in person (see the consent forms in Report Supplementary Material 4 and process evaluation consent forms in Report Supplementary Material 11). Information on storage of data can be found in Data processing, entry and management.

Analysis

A coding framework was developed through initial exploration of the data, pilot coding and by discussion between Sarah Blair, Daniel Wight, Marion Henderson, Susie Smillie, Carrie Parcell and Craig Macdonald, and was informed by MRC guidance on process evaluation in RCTs of complex interventions. 93 The coding framework was structured into six sections: (1) the context for SEW in schools (pre-existing situation prior to involvement in the SEED trial); (2) methodological issues of participation in the SEED trial; (3) implementation processes for the SEED intervention; (4) results of the SEED intervention (immediate effects of the SEED intervention or other SEW programmes that are observable/visible, are attributable to the SEED intervention/SEW programmes and can be directly influenced by school); (5) implementation processes of SEW programmes in control schools; and (6) results of SEW programmes in control schools. For more detail on the full coding framework, see Appendix 5, Proposed simplified Social and Emotional Education and Development coding framework.

A first stage of analysis was carried out prior to knowing the outcomes of the trial, to avoid bias in the analysis. 93 Case study data were coded to the agreed framework by Sarah Blair and Carrie Parcell using NVivo 10 (QSR International, Warrington, UK). The coding framework was mapped to the principal research questions from the protocol and to a set of enhanced research questions. These additional research questions explored the differences between the arms of the trial in SEW activities implemented, the extent to which the SEED intervention worked as intended, the degree to which the SEED intervention was embedded in intervention schools and the contextual factors that may have facilitated or obstructed the implementation of the SEED intervention (see Appendix 5, Table 55). The relevant codes were then extracted, and analysis of the case study schools was conducted independently by Sarah Blair, Daniel Wight, Sally Haw and Lawrie Elliott. An interim report was produced following discussion and consensus.

Following the main trial outcomes, a further stage of analysis was conducted to help interpret the findings. This stage focused on the SEED theory of change (see Figure 2) to assess the strength of existing data and the evidence available to explore further. Each proposed pathway within the theory of change was mapped to the existing data; if there were sufficient data from the case studies to provide evidence for the proposed pathways in the theory of change, no further analysis was undertaken. If evidence was moderate or weak, additional coding and analysis were performed on the data available from all schools using the existing coding framework. At this stage, selected data were also explored thematically to gain a more nuanced understanding of the mechanisms of change underpinning the intervention and to allow the generation of themes that could explain pathways beyond those proposed in the theory of change.

Further details on the methods employed in the analysis of the mixed-methods and quantitative data can be found in Chapter 6.

Overview

As the effects of the SEED intervention are expected to affect costs and outcomes outside health services, in line with NICE’s recommendation for public health interventions, a public sector perspective [including NHS, Personal Social Services (PSS) and local government perspectives] was adopted for the analysis. 94 Data on the costs include those costs associated with the provision of the intervention, including staff time, travel and consumables at each stage of the intervention, as well as the resource use cost collected at T1 and T3 in the public sector, such as general practitioner (GP) visits and police visits. The Child Health Utility-9 Dimensions (CHU-9D)95–98 was used in the trial to collect the utility data, among the OC only, for the cost–utility analysis via the self-completed questionnaires. As the SDQ-TDS was the primary outcome for both cohorts, this enabled a cost-effectiveness analysis, per se, of all the pupils. The costs and outcomes were both discounted at 1.5% per annum in the base-case analysis, as recommended in the NICE methods of technology appraisal and public health guidance. 94 All analyses were undertaken according to the principle of intention to treat and in Stata^®/SE 14.0 (StataCorp LP, College Station, TX, USA).

Intervention costs

Data on the cost of implementing the SEED intervention were collected by the research team. The intervention cost was concerned with three components of the intervention (as stated in Chapter 2, Key components of the Social and Emotional Education and Development intervention): (1) a needs assessment of pupils using the SDQ screening tool collected during baseline data collection, (2) analysis of SDQ data by the research team and (3) feedback provided to schools based on the needs assessment through reflective feedback sessions and the development and implementation of APs.

The economic evaluation focused on the ‘incremental’ (or ‘additional’) cost of the intervention, compared with the control group. The control group received the same needs assessment data at the end of the trial; however, it did not receive analysis and reflective feedback sessions from the research team and EPs. Given this, the cost of SDQ data collection for each participant was offset as it was conducted in both arms. Any additional costs for the resources purchased by the school to carry out the activities were not included in the intervention cost, as the resources required varied because of the specific activities recommended and the existing resources in the school. Despite this, we recorded the extra resources required in each school (see Appendix 4, Table 49). Most of the activities were carried out with no additional cost (i.e. based on existing resources), and the per-pupil additional cost was minimal. Compared with the control group, the intervention is viewed as a reflective and co-produced process during which the bespoke activities were selected based on the data collated by the research team. Any ‘research-related’ cost was removed from the intervention cost. The intervention cost is collected specific to each school; thus, the school-specific per-pupil intervention cost was added to each pupil’s resource use cost in the intervention arm.

Healthcare, social care service and other public service resource use

Resource use data were collected from the pupils’ parents/carers via questionnaires (see Report Supplementary Material 12) that were distributed via pupil school bags at T1 and T3. Public sector service use data collected included visits to the NHS services, social care services and other public sector services. The NHS services included GP visits; practice nurse visits; accident and emergency attendance; visits to speech therapists, occupational therapists, school nurses, physiotherapists, psychiatrists and child and adolescent mental health services; outpatient appointments; inpatient stays; NHS dentist visits; and NHS optician visits. The use of social care and public sector services was collected in terms of social worker visits, EP visits and police attendance. Unit costs in the financial year 2016/17 published by the Personal Social Services Research Unit (PSSRU)99 or NHS reference costs100 were attached to each item of resource use. Other published sources were used if the unit cost was unavailable from the above two sources (see Appendix 4, Table 50).

Utility and quality-adjusted life-years

The CHU-9D95–98 was collected from the OC at T0, T1, T2 and T3. Utility values were generated from the CHU-9D index values for the OC and expressed as quality-adjusted life-years (QALYs). The area under the curve method was used to estimate QALYs over a 12-month period, following the trapezium rule, assuming a linear change in utility between each assessment time point. 101 The CHU-9D is a generic preference-based health-related quality-of-life instrument suitable for use with children aged 7–17 years. It has demonstrated itself to be a practical and valid measure for use in economic evaluations of child and adolescent healthcare programmes. 95–98 Valuation of the CHU-9D was directly elicited from an adult and adolescent population. Preference weights were derived from 300 members of a UK adult population using a standard gamble technique for use in children aged 7–11 years. 95,96,98 Subsequently, preference weights were derived from best–worst scaling discrete choice experiment interviews of 590 Australian adolescents aged 11–17 years. 95,96,98 The Australian adolescent weights were used for valuation of the CHU-9D responses in this trial. The utility values range from 0.3261 [if the respondent answers level 5 (worst state) for all dimensions] to 1 [if the respondent answers level 1 (perfect state) for all dimensions].

Discounting

The final evaluation costs were valued in the year 2016/17, which was the year of the latest version of the published unit cost in PSSRU and NHS reference costs. As the SEED intervention accrues costs each year over the 4 years, discounting was required to bring future costs and benefits to present value. Therefore, in line with NICE’s recommendations for economic evaluations of public health interventions, a discount rate of 1.5% was applied to both costs and health effects in the base-case analysis, and a discount rate of 3.5% was used in sensitivity analyses. 100

Missing data

The resource use data were collected for the 12-month periods prior to T1 and T3 only, because parents were not surveyed at T2 (partly to reduce burden on schools), which means that the resource use data between T1 and T2 are missing. Given that costs and outcomes should cover the entire trial period, it was decided that available resource use data between T2 and T3 be used as a proxy for the missing resource use data between T1 and T2. For the missing individual responses in the questionnaires, or the missingness of the whole questionnaire, removing incomplete cases is usually not recommended as this may bias estimates or result in a large number of data being discarded, leading to severely reduced power. 102,103 To maximise the use of the returned resource use questionnaires, the incomplete response of any single item was assumed to indicate that no resource was used for that item during the assessment period. This was followed by a complete-case analysis because of the very high number of missing data (see Chapter 1, Whole-school intervention approaches). According to Jakobsen et al. ,104 if the missingness is considerable (the authors give an example of > 40%), a complete-case analysis is recommended, but the results may be considered only as hypothesis-generating. 104 Therefore, a complete-case analysis was adopted. In addition, multiple imputation with chained equations was also initially conducted for log-transformed utility values and log-transformed total costs at each data collection wave at aggregate level (compared with individual dimension and resource use item). However, the two-tier clustered estimation model could not proceed, which may be because of the large number of missing data; therefore, the complete-case analysis was determined to be the base-case analysis.

Cost–utility analysis

Like the primary outcome analysis, multilevel mixed-effects models were used to estimate the difference in mean costs and QALYs, which adjusted the standard errors and 95% estimates for the two-tier nested clusters, that is primary schools linked to each secondary school (learning communities). This cluster effect assumed a fixed effect within each cluster, but a random effect between the schools at each tier of the cluster. This non-independence is measured by the ICC, which represents the proportion of variance due to between-cluster variation. 105

The distributions of the costs and the QALYs were visualised using histograms. The mixed-effects generalised linear models were originally performed with gamma family and log-link, which took into account the skewed cost distribution and QALY distribution, followed by a recycled prediction to estimate mean cost and QALYs for each arm. However, owing to the large number of missing data, the number of pupils within each cluster significantly reduced. This led to the failure to converge of the generalised linear model, and the subsequent recycle prediction could not proceed. To address this, the right-skewed cost data (i.e. majority of the participants have small cost) were log transformed and a linear mixed-effects model was applied. In contrast to the distribution of the cost data, the QALY data were left skewed, with a large proportion of observations lying at the higher end of the QALY scale. Therefore, the QALY data were transformed by taking the log transform of the decrements of QALYs, which were calculated as the difference between the maximum number of discounted QALYs that could possibly be accrued within the 4-year time horizon (i.e. 3.92) and the actual QALYs gained. 106 This decrement method ensures the distribution of log-transformed QALYs to be closely fitted to a normal distribution.

Covariates for the mixed-effects model were baseline CHU-9D values as recommended37 and all the demographics covariates used in the primary outcome analysis, including baseline SDQ-TDS (continuous, 0–40), gender (categorical: male, female), proportion of pupils receiving free school meals (categorical: < 20%, ≥ 20%), number of pupils in school (categorical: < 250, ≥ 250), religious denomination (categorical: non-denominational, Roman Catholic), location (categorical: large urban, other urban, non-urban), LA (categorical: LA1, LA2, LA3), number of P1 pupils in school (categorical: < 30, ≥ 30) and proportion of pupils from minority ethnic groups (categorical: 0% to < 10%, ≥ 10%, not available or blank).

The incremental cost-effectiveness ratio (ICER) was estimated from the adjusted difference in cost and QALYs from the mixed-effects regression. A 1000-iteration bootstrapping procedure was performed to investigate the uncertainty surrounding the ICER estimate and the probability that the SEED intervention was cost-effective under a wide range of hypothetical thresholds (£0–100,000). Standard errors and 95% CIs for differences in the adjusted cost and QALYs were estimated via the bootstrap, adjusting for clusters. These results were visualised in cost-effectiveness planes and cost-effectiveness acceptability curves (CEACs).

Sensitivity analyses

Five sensitivity analyses varying the cost, and one varying the discount rate, were undertaken (see Appendix 4, Table 50) to assess the robustness of the base-case results of the cost–utility analysis to the alternative cost assumptions. All the scenarios were followed by bootstrap procedures to estimate the uncertainty surrounding the ICER in each. The cost of the intervention is assumed to vary by ± 30%, which takes into account the variations when carrying out this intervention in different settings, for example the analysis and preparation time for the feedback sessions, the staff cost involved, the number of pupils in a school and the location of a school. Owing to the large number of missing cost data, the sample size of the base-case cost analysis was only around 10% of the total population; therefore, the sensitivity analysis varied the assumptions regarding the methods for dealing with the missing data. It was assumed that no resources were used when the data from the whole resource use questionnaire were missing, but CHU-9D was not missing; consequently, the sample size for the economic evaluation was restored to a level similar to that of the primary outcome analysis. With this step, both complete-case analysis and multiple imputation (for the cases that had both CHU-9D and cost missing) were both conducted. The next sensitivity analysis increased the discount rate from the base case of 1.5% to 3.5% for both costs and benefits, as recommended by NICE’s methods for the development of public health guidance. 100 The base-case analysis employed a public sector perspective assuming that the intervention was paid for using public funds. However, given that it is unclear if the intervention will be paid for using public funds or by the individual school when it is rolled out, the last sensitivity analysis employed an educational sector perspective by limiting the cost to the intervention cost only, to reflect the cost-effectiveness when the intervention is paid for by the schools. The difference between the public sector perspective and the educational sector perspective is that the latter excluded the NHS, social care and other public sector resource use costs from the total cost by keeping only the confirmed intervention costs. Therefore, this scenario also addresses the sensitivity of the results to the measurement accuracy of the resource use questionnaire and any potential random NHS, social care and other public sector routine resource use that is unrelated to the intervention.

Cost-effectiveness analysis

In addition to the primary cost–utility analysis, a cost-effectiveness analysis of the whole population (i.e. both the YC and the OC) was conducted on the primary outcome, the SDQ-TDS and the SDQ ‘strengths’ (prosocial) subscale. As with the analysis of QALYs in the cost–utility analysis, the difference in the SDQ scores was estimated using a multilevel mixed model regression adjusting for the school characteristics and then combined with the difference in cost to generate cost per improved SDQ score (both SDQ-TDS and ‘strengths’). A 1000-iteration bootstrap was conducted to explore the uncertainty of these estimates.

Assessment of harms

The intervention and its evaluation was considered to be of low risk to participants. A protocol was in place in the case of any unexpected adverse events. Fieldworkers were encouraged to report negative outcomes or experiences to the study team.

Schools

Table 2 presents summary information about the 38 schools that participated in the trial, namely their LA, their deprivation level [Scottish Index of Multiple Deprivation (SIMD) decile], the percentage of pupils who were eligible to receive free school meals, their urban/rural classification, the percentage of pupils from a minority ethnic group, and whether or not the schools were denominational (religious). There were no significant differences between randomised groups, as expected, given the process of balancing the randomisation scheme (see Chapter 3).

Pupils

Table 3 presents summaries of the gender, age (cohort) and deprivation level (SIMD decile) of pupils at baseline. No significant differences were observed between intervention and control pupils’ gender and age distributions. However, despite being balanced at school level, there were significant differences between pupils attending intervention schools and those attending control schools in the levels of deprivation of their area of residence. The statistical analysis plan specified that analyses would adjust for variables used to stratify the randomisation; however, given the differences in pupil profiles between the control and intervention schools, it was decided to adjust for individual-level SIMD decile, based on home postcode, rather than school-level deprivation (although the school-level SIMD decile was used when individual SIMD decile was unknown).

Baseline

Table 4 presents summaries of the baseline (T0) teacher- and parent-completed SDQ-TDS and the five SDQ subscales. The primary outcome is the teacher-completed SDQ-TDS; the subscale scores and the parent-completed scores are relevant to additional analysis presented later in this chapter. Teacher-completed SDQ-TDS and subscale scores showed significantly greater difficulties in the intervention group at baseline. The statistical analysis plan prespecified adjustment for baseline scores in all SDQ-related outcome analyses.

The parent-completed SDQs (see Table 4) did not show any statistically significant differences between the two trial groups. Although all parents were invited to complete questionnaires, only about one-third complied. Therefore, the parents’ questionnaires may not be representative of the whole sample of pupils, but similar proportions of parents responded in each arm of the trial.

Primary outcome at follow-up 3 (prespecified main outcome point)

Table 5 presents summaries of the primary outcome (i.e. SDQ-TDS) at baseline (T0) and follow-up 3 (T3) for those with data at both time points, and for all available data. For those with data at both time points, summaries of the change over time are also provided. Estimates of the between-group differences are provided based on the primary analysis (baseline-adjusted mixed-effects regression), and from a repeated measures analysis of all available data from all time points. For each effect estimate, a standardised effect size is reported, derived by dividing the estimated intervention effect and CI by the residual SD from the model. This statistical analysis and presentation of results is repeated for all the other follow-up time points later in the report.

The primary analysis did not show a statistically significant difference between intervention and control schools (p = 0.078), although the trend was towards lower SDQ-TDSs in intervention schools [estimated difference –0.84 (95% CI −1.77 to 0.09), standardised effect size −0.15 (95% CI −0.31 to 0.02)]. The repeated measures analysis, using all available data at all four time points, did show a statistically significant difference (p < 0.001), with the SDQ-TDSs being lower in intervention schools (estimated difference –1.30, 95% CI −1.87 to −0.73). The standardised effect size was small, at −0.27 (95% CI −0.39 to −0.15).

Table 6 provides summaries of changes in SDQ-TDS between T0 and T3 within subgroups defined by age (OC or YC), gender (male or female), deprivation (deprived or not deprived, based on SIMDs), and the combination of age and gender (four subgroups). The baseline-adjusted regression model of SDQ-TDSs was extended to allow for an interaction between each subgrouping variable and the intervention effect, to estimate and test for differential intervention effects between subgroups of children.

There was a significant interaction between the intervention and the combination of age and gender (p < 0.001), with the strongest intervention effect seen in the older male cohort [−2.36 (95% CI −3.62 to −1.11; p < 0.001), standardised effect size −0.42 (95% CI −0.64 to −0.20)] and, to a lesser extent, in the older female cohort [−1.35 (95% CI −2.66 to −0.04; p = 0.043), standardised effect size –0.24 (95% CI −0.47 to −0.01)], but with no significant intervention effect observed in either YC. This was mirrored in subgroup analyses by age and gender separately, with a significant intervention effect seen among the older, but not younger, pupils (interaction p < 0.001), and among males, but not females, although the interaction with gender was not statistically significant. There was no evidence of different intervention effects according to deprivation.

Primary outcome at follow-up 1

Table 7 shows the analysis of the SDQ-TDS at T1. This is when both cohorts were still in primary school and the intervention process was ongoing. At T1 there was a significant effect of the intervention from both the primary analysis (p = 0.004) and from the repeated measures analysis using all available data (p < 0.001), with lower SDQ-TDS at T1 in the intervention group.

Intervention effect estimates were in favour of the intervention group within all subgroups (Table 8), and were statistically significant for all but the older male and older female groups, although there was no evidence that the intervention effect differed between any subgroups.

The Strengths and Difficulties Questionnaire-Total Difficulties Score at follow-up 2

Table 9 shows the analysis of the SDQ-TDS at T2. It should be noted that T2 is when three schools temporarily did not participate in the trial, and so, along with T4, is the least representative of our outcome time points. When including only pupils with data at baseline and T2 (baseline-adjusted mixed-effects regression), there was no evidence of an intervention effect. However, within a repeated measures analysis, using all available data from all four time points, the intervention effect estimate at T2 was statistically significant (p = 0.012), with lower SDQ-TDSs in the intervention group.

Extending the primary analysis model to look at subgroups (Table 10) found no evidence of intervention effects for any particular groups of pupils.

Primary outcome at longer-term follow-up 4 (6 years post baseline)

Table 11 shows the analysis of the SDQ-TDS at T4. It should be noted that, during T4, three schools did not participate in the study; therefore, along with T2, T4 is the least representative of our outcome time points. T4, our longer-term follow-up (6 years post baseline), involved the OC only. When including only pupils with data at baseline and T4 (baseline-adjusted mixed-effects regression), there was no evidence of an intervention effect. However, within a repeated measures analysis, using all available data from all four time points, the intervention effect estimate at T4 was statistically significant (p = 0.044), with lower SDQ-TDSs in the intervention group.

Extending the primary analysis model to look at subgroups (Table 12) found no evidence of intervention effects for any particular groups of pupils.

Primary outcome over time

Figure 5 provides an illustration of the primary outcome for each arm of the trial, showing the cohort by gender subgroups, at each applicable time point. The data shown, zero-centred for T0, are predicted values from repeated measures models fitted within each subgroup, with 95% CIs. In all subgroups, the SDQ-TDS increased between T0 and T1, but more so in the control arm, so that intervention arm scores were lower at T2 for both cohorts. In the YC, scores changed much less over time, with intervention and control arm scores being more similar. The change between T0 and T1 in the OC may be due to the change from teacher-completed to pupil-completed SDQs. The largest difference between the intervention and control arm for the OC was at T3. Boys started with higher SDQ-TDSs than girls from both cohorts, and this difference persisted over time; even at T4, 6 years post T0, the older boys showed a significant difference between the intervention and control arms.

FIGURE 5.

Primary outcome at each time point by age and gender subgroups. (a) Older male; (b) younger male; (c) older female; and (d) younger female. (continued)

Covariate effects on the Strengths and Difficulties Questionnaire-Total Difficulties Score

Table 13 shows the covariate effects from the baseline-adjusted mixed-effects regression models at each time point. Baseline SDQ-TDSs were strongly predictive of scores at each follow-up point. Older pupils had significantly higher scores and female pupils had significantly lower scores throughout the trial. Otherwise, none of the stratification variables used when assigning schools to trial arms was found to be significantly associated with the SDQ-TDS.

Parent-reported Strengths and Difficulties Questionnaire total difficulties

Parental questionnaires were completed by approximately one-third of parents, at T0, T1 and T2. These are summarised in Table 15. There were no significant intervention effects at either follow-up for the SDQ-TDS.

Strengths and Difficulties Questionnaire subscales

Table 16 reports the intervention effect estimates and 95% CIs for each of the five SDQ subscales, for all pupils, and in subgroups defined by age (cohort), gender, deprivation and baseline SDQ-TDS (see Appendix 3, Tables 26–31 for more detail).

Overall, and in most subgroups, the mean emotional symptoms scores were lower in the intervention group at all follow-up time points. The between-group difference failed to reach statistical significance at T2 for older pupils, female pupils, those living in more deprived areas and those with a SDQ-TDS in the normal range. For those with borderline or abnormal SDQ-TDSs, intervention effect estimates were large, and favoured the intervention group, but were not statistically significant. However, the only subgroup analysis showing differential intervention effects was deprivation, with generally larger effects seen among those living in less deprived areas.

Conduct problem scores were significantly lower for the intervention arm at T1 and T3. Peer problem scores were significantly lower for most of the intervention subgroups. Hyperactivity subscale results showed a significant reduction only for the OC at T2.

The only strength subscale of the SDQ is prosocial behaviour. At T1 the intervention group showed significantly better prosocial behaviour scale scores; at other time points a range of subgroups benefited (see Appendix 3, Tables 26–31).

The analyses included 135 between-group comparisons (see Table 16). We would have expected 20% to be significant by chance alone; we found 63 (47%, more than double what was expected by chance) of these comparisons to be significant at a 5% significance level, all in favour of the intervention arm of the trial. This adds some weight to these results.

Pupils’ social and emotional well-being domain scores

In the OC, pupils self-completed questionnaires at each wave of data collection, from which several measures of SEW were derived. Intervention effect estimates for these SEW domain scores are presented in Table 17 (see also Appendix 3, Tables 33–37 and Table 45). Between-group differences were observed in favour of the intervention group for the self-management and responsible decision-making domains. For self-management, these differences were observed consistently for boys and girls, and for pupils living in more deprived areas and pupils living in less deprived areas; for responsible decision-making, however, benefits in the intervention group appeared to be stronger for boys than for girls, and appeared to be specific to those living in less deprived areas.

Overall, statistically significant improvements were found for the intervention group at T3 in terms of the emotional regulation, social awareness and materialism domain scores. No between-group differences were observed in the overall analysis of the self-esteem and self-awareness domain scores.

Of the 105 statistical tests represented in Table 17, we found 24 (23%) to be statistically significant at a 5% significance level in favour of the intervention group, with none in favour of the control group.

Domains relating to pupils’ social contexts

Pupil-completed questionnaires also included questions relating to their social context, from which a range of domain scores were derived. Intervention effect estimates for these scores are reported in Table 18 (see also Appendix 3, Tables 38–44). In overall analyses, significant differences in favour of the intervention group were observed at T3 for family relationships, school relationships (relating both to the whole school and to within classes), school climate and participation in antisocial behaviour. A benefit at T2 was also seen for the school relationships (class) domain. No other between-group differences were observed in overall analyses. Within subgroups defined by gender and deprivation, these results were generally consistent, although there was some evidence of greater intervention effects for pupils living in less deprived areas, in particular for school relationships (class) and liking school. There were statistically significant differences in favour of the control group for experience of antisocial behaviour for female pupils at T1 and T2, and for participation in antisocial behaviour at T2 for those living in more deprived areas; however, there were no differences between randomised groups in overall analyses, and no evidence of subgroup-by-intervention interactions for these measures.

Of the 120 between-group comparisons reported (see Table 18), 22 (18%) were statistically significant at a 5% significance level in favour of the intervention group, and three (2.5%) were significant in favour of the control group.

Pupil-reported health risk behaviours at follow-ups 3 and 4

We undertook additional exploratory analysis around pupils’ health risk behaviours when the OC reached 12–13 years (T3), and at age 15–16 years (T4). Please note that, at T3, the pupils were still relatively young regarding alcohol, tobacco, e-cigarettes and cannabis use. A further limitation is that, at T4, three secondary schools dropped out, which weakens our ability to interpret the data. For these results, see Appendix 3, Tables 46 and 47. At T3 we found significantly less use of e-cigarettes among the intervention pupils; at T4 this outcome was not significant. At T4 we found significantly less reporting of alcohol use. For all outcomes, the health risk behaviours reported were lower for the intervention arm.

School staff-reported domains

Table 19 reports the intervention effect estimates for a range of domain scores derived from teacher-completed questionnaires at each time point (see Appendix 3, Table 48, for more detail). The analyses are cross-sectional, given the relatively high turnover of staff in trial schools. Furthermore, although HTs supported participation, staff questionnaire completion was voluntary; participation rates were 53% at T1, 46% at T2 and 44% at T3.

No statistically significant differences between the arms of the trial were found at T1. However, at T2 and T3, a number of between-group differences emerged in favour of the intervention arm. At T2, staff in intervention schools reported significantly better perceptions of management, staff support, school ethos, school support for SEW, pupil engagement and staff–pupil relationships. At T3, these benefits were retained, except for school support for SEW, and an improvement in staff–staff relationships became apparent. Caution is required when interpreting these results given the relatively low response rates.

Intervention cost

The SEED intervention costs are detailed in Table 20. Intervention costs included the staff costs of analysing SDQ responses, recommending activities and preparing for the feedback, and the cost of providing the feedback sessions to the schools (see Chapter 3, Intervention costs). The total average cost per pupil for the 4-year intervention across schools is estimated to be £165.19; the research staff costs of analysing SDQ responses, recommending tailored activities based on these results and preparing for the feedback account for around 90% of the total. For the details of intervention cost calculation for each component, see Appendix 4, Table 54. The total cost for each school ranged between £153.80 and £225.03, varying primarily based on the total number of pupils in a school. For example, the largest intervention cost, £225.03, came from a school cluster with very few pupils (approximately 17 pupils for the 3 intervention years), whereas the smallest intervention cost (£153.80) came from a school that had around 76 pupils for the intervention. The other factors affecting the intervention cost include feedback session duration, number of feedback sessions and school location (time and distance to travel).

Missing data

For the number and percentage of missing data for each collected resource use item and CHU-9D index score, see Appendix 4, Tables 52 and 53. A similar number of missing data were observed for both arms, although, for all items, the intervention group has a slightly higher percentage of missing data than the control group. For the resource use, approximately 75% of data were missing for the T0 (baseline) to T1 (follow-up 1) period and 85% of data were missing for the T1 to T3 period. For the CHU-9D index, the proportions of missing data were approximately 18%, 15%, 43% and 32% for T0, T1, T2 and T3, respectively.

Resource use and costs

Appendix 4, Table 51, shows the use of each health and societal resource item (mean number of visits, SD, median, minimum and maximum) accrued among the OC over the first 2 years (2013–2015), assessed at T1, and over the last year (2016–2017), assessed at T3. Overall, there were no statistically significant differences in the use of each resource item between the intervention and control groups, except for the GP visits between T2 and T3.

The observed and estimated mean costs of service use in both arms were compared by using ordinary least squares regression without adjusting for any covariates, and by using a multilevel mixed model of the log form of the cost of resources plus adjusting for covariates. In Figure 6, the distribution of the cost of resource use in the OC over the 4 years in its natural unit (see Figure 6a) and log form (see Figure 6b) is presented. Overall, the cost of the intervention arm was distributed more towards the lower end in Figure 6a, compared with the control group, with a few extreme high-cost values from the intervention arm. The cost of resource use is right-skewed; therefore, the adjusted model predicted the cost in its log form, which was distributed closer to a normal distribution (see Figure 6b). For both models, there were no statistically significant differences at the 0.05 level in resource use between arms (Table 21). Despite this, the adjusted model suggested that the intervention arm was estimated to have a reduced resource use of £279.20 (p = 0.063), compared with the control arm.

FIGURE 6.

Distribution of cost of resource use in its natural unit and in its log form. (a) Natural unit; and (b) log form. Note that the number of observations is 145.

Child Health Utility-9 Dimensions index and quality-adjusted life-years

The CHU-9D index values for each arm of the trial at T0 and at each of the follow-up assessments, and the QALYs accrued over the 4 years, are shown in Table 22. The utility values for both groups declined (suggesting decreased quality of life) over the 4 years from 0.85 to 0.83 in the intervention group and from 0.84 to 0.82 in the control group, with the 0.01 difference at T3. However, this was not statistically significant. There were no statistically significant differences between groups at any of the assessment points, although the number of QALYs gained in the intervention group was 0.05 larger than that of the control group.

Cost–utility base-case analysis

The cost–utility results for the SEED intervention in the OC are presented in Table 23. Overall, the intervention arm consumed approximately £75.58 less in resources (i.e. a total of intervention cost plus NHS, PSS and public sector resource use) per pupil and gained 0.038 more QALYs than the control group.

The average cost per participant was £1315 (95% CI £1131 to £1529) in the intervention group and £1391 (95% CI £1207 to £1603) in the control group. The mean number of QALYs accrued over the 4-year trial period was 3.50 (95% CI 3.45 to 3.54) for the intervention group and 3.46 (95% CI 3.42 to 3.50) for the control group, leading to a 0.038 (95% CI –0.023 to 0.107) additional QALY gain in the intervention group, compared with the control group.

The 1000 iterations of non-parametric bootstrapped pairs of mean costs and QALYs are shown in the cost-effectiveness plane (Figure 7). The x-axis represents the incremental QALYs and the y-axis represents the incremental costs (£). It shows that the incremental cost is almost evenly distributed across the x-axis, which suggests that the costs across the intervention and control groups are similar. Most of the bootstrapped pairs are to the east (see Figure 7), which corresponds to the positive point estimate of the incremental QALY. The precise percentage of the bootstrapped pairs of mean cost and QALYs distributed in each of the quadrants is outlined below. The probability that the intervention achieved an incremental QALY was approximately 90% (north-east + south-east quadrants), and the probability that the intervention was cost-saving was approximately 56% (south-west + south-east quadrants).

FIGURE 7.

Cost-effectiveness plane representing 1000 bootstrapped cost difference and QALY difference pairs.

The distribution of the bootstrapped pairs of cost difference and QALY difference in the four quadrants are as follows:

1. north-east – intervention is more costly and more beneficial (40.2%)

2. north-west – intervention is more costly and less beneficial (4.1%)

3. south-west – intervention is less costly and less effective (6.5%)

4. south-east – intervention is less costly and more effective (49.1%).

The CEAC for the base-case analysis is shown in Figure 8. The probability that the SEED intervention was cost-effective at the standard threshold of £20,000 per QALY gained was 88.5%. This corresponds to the proportion of pairs under the £20,000 per QALY threshold slope in the north-east, south-west and south-east quadrants of the cost-effectiveness plane (see Figure 7). When the threshold equals 0, the probability of the intervention being cost-effective was 55.6%, which corresponds to the proportion of cost and QALY difference pairs under the x-axis in the cost-effectiveness plane (i.e. the south-west + south-east quadrants in Figure 7). The probability increases with the increase of the cost-effectiveness threshold and remains at around 89% after the threshold is > £40,000 per QALY gained.

FIGURE 8.

Cost-effectiveness acceptability curve.

Sensitivity analyses

The results of the sensitivity analyses are presented in Table 24. When varying the intervention cost up and down by 30%, the incremental cost varied; however, the conclusion did not change: the intervention was still associated with a reduced incremental cost and improved QALY, with > 80% probability that the ICER was lower than the £20,000 threshold. When the missing whole-wave cost questionnaire (see Chapter 3, Economic evaluation) was assumed to be no resources used as long as the CHU-9D index at the same wave was completed, the number of observations available for cost estimation increased from 124 to 511. This scenario also generates a similar conclusion, although the cost saving became more obvious and the probability for the intervention to be under the £20,000 per QALY threshold increased to 100%. Both complete-case analysis and multiple imputation (for the cases that had both CHU-9D index and cost missing) were originally planned; however, the estimation failed to proceed for the cluster analysis after the multiple imputation procedures. Therefore, only the complete-case analysis result is presented here. When the discount rate increased from 1.5% to 3.5%, neither the incremental cost (–£69) nor QALY (0.036) varied much from the base-case analysis, and the bootstrapped probability decreased to 84.8%.

Cost-effectiveness analysis

Table 25 summarises the estimated additional cost per 1-point decrease in SDQ-TDS among the whole sample and the YC and OC separately over the 4 years. For the whole sample, the cost-effectiveness analysis resulted in an additional cost of £82 per 1-unit decrease in the SDQ-TDS, and an additional cost of £396 per 1-unit decrease in the SDQ prosocial score (i.e. the strengths subscale). Among the YC, the intervention was associated with a £166 increase in the cost and a 0.19 decrease in SDQ-TDS, leading to an £856 per 1-unit SDQ-TDS decrease. Costs were lower in the OC because of the smaller incremental cost and much larger treatment effect, with an ICER of £1.22 per 1-unit decrease in SDQ-TDS. The ‘strengths’ (prosocial) score did not differ between the intervention and control groups in either of the separate cohort analyses.

Perceptions of the Social and Emotional Education and Development intervention

This section summarises perceptions of the intervention from the perspectives of staff, pupils and parents. The following section examines the implementation of each of its core elements and considers who engaged most with each.

Implementation of the Social and Emotional Education and Development intervention

Mechanisms for change

This section explores the possible mechanisms by which the SEED programme achieved its impacts.

Study effects: the experiences of control schools

Examining the experiences of control schools can highlight potential effects of participating in the research that may contaminate and dilute the effects of the intervention. Around half of the HTs in control schools that took part in an interview said that participation in the SEED RCT had an impact on their approach to SEW. This was supported by a small number of qualitative responses from the staff questionnaires.

Contextual factors

Raising awareness of social and emotional well-being and fostering a collective commitment towards addressing needs (quadrant 1: inputs)

Identifying suitable activities and implementing initiatives (quadrant 2: activities and outputs)

Improving pupil social and emotional well-being through school connectedness, relationships and embedding social and emotional well-being activities into the curriculum (quadrant 3: short-term outcomes for children)

The Social and Emotional Education and Development intervention as a cycle

The SEED intervention was intended to function as an iterative process with yearly cycles of data collection, feedback and action planning. Following feedback on baseline data, no feedback sessions were planned for year 2 of the trial as this was intended to be when schools would focus on implementing their APs. It was the original intention that schools would have further RD sessions on T1 and T2 data, but in practice this was not always possible. Of the 18 intervention schools, only two had three cycles of data feedback including RD sessions. Four schools had only one cycle and the rest (n = 14) had two. When RD sessions were not possible, schools were offered electronic versions of their feedback to use in their own time. The main reasons as to why schools did not follow the expected cycle were practical limitations in finding time to schedule feedback sessions within a school calendar that has very limited staff collegiate time and a loss of commitment to the process due to staff changes and school dynamics. Evidence from EPs also suggests that more regular contact and meetings between EPs and schools would strengthen the ability to maintain the SEED intervention in schools for the longer term.

Intervention

The SEED intervention and its evaluation were timely in their strong fit with the CfE,16,121 which was a rationale for testing the concept in Scotland. The co-production and adaptability of the reflective process was seen as a strength by school staff. The ability of the SEED intervention to adapt to local needs suggests a likely fit with other policy contexts. The SEED intervention was seen as integral to the cyclical process of school improvement planning and as a useful way to structure SEW practices efficiently, suggesting intensiveness of integration into routine practice.

The process and outcome evaluations point to the importance of the quality of staff–pupil and staff–staff relationships and trust; however, we experienced a relatively high turnover of school staff, which may have reduced the efficacy of the intervention. Furthermore, resource and staffing cuts, and LAs’ competing demands on schools, inhibited implementation of SEW activities. Within LAs, the need to avoid ‘contamination’ of the control schools, and the temporary research status of the programme, prevented the SEED intervention from becoming embedded in LA structures.

The weakest element of the SEED intervention was the creation of co-produced APs. First, the APs did not always reflect the priorities identified in the RD sessions. Second, the plans made little use of the SEED resource guide to select evidence-based SEW interventions. Although some schools implemented those packages identified as having strong evidence, few schools used the resource guide to shape their AP. Third, parents or pupils were not involved in the production of APs, as had originally been intended. In some schools these were produced solely by senior staff. We had also hoped for greater pupil involvement in the action planning process; however, this was difficult as most schools chose staff in-service days for their RD sessions. Pupils are not present during in-service days.

On the other hand, engagement with the pupil needs assessment data motivated all staff to improve SEW in the schools. Specifically, the reflective element of the intervention supported existing improvement planning processes and contributed to a culture of evaluating practice. The SEED RCT gave space for professional dialogue in which whole-school activities were prioritised over individual-level programmes. The intervention schools offered a similar amount of SEW-related activities to the control schools, but the activities may have been selected and implemented with greater consensus and fidelity in the intervention arm.

Given that the SEED intervention encourages whole-school initiatives, it would have been helpful to involve all year groups of pupils, and this is supported by process evaluation findings concerning the representativeness of the data. However, that would have been costly in terms of time and money. We opted instead to recruit two cohorts of pupils starting at P1 (aged 4–6 years) and P5 (aged 8–10 years). The relatively small proportion of pupils surveyed could be addressed in future roll-out of the SEED intervention (see Sustainability and implications for roll-out).

Existing relationships between staff, and with the EP, shaped the adoption of the SEED intervention. Although all EPs supported the objectives of the SEED RCT, their intended role as a ‘critical friend’ worked best when this fitted with their current working relationship with the school, and when this did not conflict with existing individual caseload priorities. The contexts that facilitated the SEED intervention were closely related to relationships, for example strong leadership, readiness for change and good relationships with the school EP. Conversely, there were some school contexts in which the SEED intervention did not work as well, for example if the SEW needs of pupils were too high, if there were fractious staff relationships to start with or if there was a poor relationship with the EP. Thus, relationships both have predictive value in determining the success of the intervention and are an important mechanism for change.

Evaluation

The SEED questionnaires contain a rich range of reliable, validated scales and some novel items selected to ensure suitability for the ages of pupils included in the trial. These questionnaires served the dual purpose of a needs assessment for intervention schools and of providing outcome, process and economic measures for the evaluation, as recommended by MRC guidelines for the evaluation of complex interventions. 122

The three participating LAs had heterogeneous characteristics and demographics, strengthening the generalisability of our findings (see Generalisability). Overall, we had strong pupil retention and, despite inevitable attrition, the trial had sufficient statistical power. We anticipated attrition particularly around the transition from primary to secondary education, and so we over-recruited by one school in each arm to compensate for this, in addition to our expected 25% dropout. We allowed for school clustering within the statistical analysis. A further strength was that contamination was minimised by clustering at the school learning community level (the secondary school together with its primary feeder schools). EPs are allocated to learning communities, so, in general, each EP was associated with either the intervention or control arm only. Furthermore, in general, intervention and control primary school pupils graduated to their linked secondary schools.

The randomisation was conducted by a registered clinical trials unit, and demonstrated good balance between the arms of the trial (see Table 2) in terms of school demographics. The pupil-level (based on self-reported postcodes), not school-level, SIMD was the only measure by which arms differed. The intervention arm contained more pupils (self-reported) from deprived areas, which we would expect would decrease the chance of showing an effect, given that the literature suggests that interventions are less effective for deprived pupils. 123 In line with this, our intervention arm pupils started out with statistically significant higher SDQ-TDSs. Thus, there was no bias in favour of the SEED intervention and adjustment was made for deprivation in the outcome analyses. The trial was registered with the ISRCTN registry, and a protocol was available in the public domain before data were collected and analysed. There were no significant deviations from our protocol, although the process evaluation’s data collection was less comprehensive than intended.

There were, however, limitations to the evaluation. For instance, as mentioned previously, for time and cost reasons we involved only two cohorts of pupils, starting at P1 (aged 4–6 years) and P5 (aged 8–10 years), in the intervention and its evaluation. Given that the intervention focused on a whole-school approach, not involving pupils from every school year may have weakened the intervention and thus its evaluation. Furthermore, interpretation of T2 and T4 data is problematic because three schools temporarily withdrew from data collection; we believe that this was due to the burden of data collection, particularly at T4, when our pupils and schools were focusing on national external examinations. In addition, as mentioned previously, although we invited all staff to complete questionnaires at all time points, the high turnover of staff meant that we had to use those data cross-sectionally (rather than repeated measures longitudinally).

We anticipated a challenge in collecting data from parents, both for the intervention and its evaluation, as has been found in other studies,124 but the level of parental involvement was disappointing. This had two impacts. First, parent data were not sufficiently representative to give confidence in the analysis for parent-reported outcomes. Second, it meant that the parent data fed back at the RD sessions were unlikely to be a fair basis for influencing decision-making (we communicated that to schools). Ideally, as described previously, we would have also welcomed pupil involvement in the feedback and planning process.

It was not possible to conduct stakeholder interviews at all planned time points. Therefore, the experiences of some SEED RCT schools, particularly intervention schools, were poorly represented in the process evaluation. It is also possible that those who were more positive about the SEED RCT were more likely to agree to be interviewed. Furthermore, key staff (SB and SS) had the combined roles of programme designers, implementers and data collectors. This was likely to create two kinds of bias. First, they were highly invested in the success of the programme, and therefore might have sought favourable data and/or interpreted them positively. Second, their interviewees were aware that they had helped design, and had implemented, the programme, and so might have been inclined to report positive aspects of it, and withhold negative aspects, through courtesy.

The funding for the SEED trial was allocated during an economic recession. We had hoped to offer schools core staff training in SEW, in collaboration with Hawkins et al. ,125 based on the Seattle Social Development Project, but the necessary additional funding was not available. NIHR does not fund interventions and, given the then-impending national referendum on Scottish independence, the Scottish Government was unable to commit to additional intervention costs. The Scottish Government did offer in-kind support for training. The strength of this is that the SEED intervention was able to demonstrate its effectiveness, while being very cost-effective and fitting with the current funding model for schools.

The older cohort effect

The SEED intervention effects were stronger for the OC (those that started in P5, aged 8–10 years) than for the YC (those that started in P1, aged 4–6 years). Although this may be counterintuitive, as the older pupils were least exposed to the intervention, this could be explained by the relative richness of data collected for the older pupils. Throughout the trial, data were collected for older pupils by self-completed questionnaires, as well as teacher-completed surveys at baseline, which meant that RDs and APs were more focused on needs identified by older pupils. It is also possible that the OC’s self-reported data may have been more sensitive to change than the YC’s teacher-completed SDQs. However, as the SDQ is not validated for self-completion by the YC, we had no option but to use the teacher-completed SDQs. SDQ-TDSs increase with age, so older pupils had more need for action and intervention for their SEW, plus more capacity for change.

This stronger effect size for the OC might be linked to the greater number of intervention options available for older pupils. More packages have been developed and evaluated for that age group, partly because the YC cannot yet read and write. This is reflected in the curricular packages contained in the resource guide being skewed towards the OC.

Gender effect

Among the OC, the effect was significant for both boys and girls, but strongest for boys. This resonates with findings from other studies that boys can benefit most from school-based health promotion interventions. 90,128,129 Bonell et al. 127 reported a reduction in SDQ scores for boys, but not for girls, resonating with our finding of a stronger effect for boys. This warrants further study.

Longer-term outcomes

Our prespecified primary outcome, SDQ-TDS at T3, demonstrated encouraging results, particularly for the older boys (effect size of 0.42), at 4 years post baseline. This is important as very few studies have conducted long-term outcome assessments of environmental interventions that span the transition from primary to secondary school. A notable example is the Seattle Social Development Project (see Chapter 1). 36 The Gatehouse Project, which also influenced this work, had some longer-term outcomes, but did not commence in primary school. 53

Although the primary outcome was the SDQ-TDS, when pupils reached secondary school we also explored health risk behaviours. Descriptive statistics show that intervention arm pupils exhibited lower levels of risk taking for all health risk behaviours explored. Statistically significant reductions were found for e-cigarettes (T2 and T3) and for alcohol (T4). The intervention was a process to improve SEW and not directly aimed at health risk behaviours, but an association with improved SEW reducing health risk behaviour is expected. 26 These findings provide further evidence of the potential benefits of investing in young people’s SEW.

Unfortunately, by T4, 6 years post baseline, the intervention effect had been diminished. Although it was disappointing not to retain the level of effect reported at T3, on reflection, pupils with higher SDQ-TDSs may have been more likely to drop out the longer we ran the trial, as they are likely to be the pupils least engaged with school, leaving fewer pupils with capacity for reduction in their SDQ-TDS. 130 The T4 data collection was possible only because of additional funding; there was uncertainty and delay in securing this funding, necessitating an extension to our ethics permission. Unfortunately, by the time we were able to collect the T4 data, some pupils in some schools in the OC may have left school, particularly those disengaged with education, who may also have higher SDQ-TDSs. Three out of 18 secondary schools declined to take part in T4, reducing our statistical power. This also highlights the challenges that schools face in engaging with research among school years facing national examinations. It may be worth noting that secondary schools were not part of our intervention, rather they were co-operating to help us collect longer-term data; perhaps this affected their motivation to stay in the trial during a challenging school year.

Domain items from pupil self-completed and staff questionnaires, follow-up 2 (2016)