Notes
Article history
The research reported in this issue of the journal was funded by the PHR programme as project number 10/3006/01. The contractual start date was in January 2012. The final report began editorial review in July 2017 and was accepted for publication in October 2017. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The PHR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.
Declared competing interests of authors
none
Permissions
Copyright statement
© Queen’s Printer and Controller of HMSO 2018. This work was produced by Humphrey et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.
2018 Queen’s Printer and Controller of HMSO
Chapter 1 Introduction
Social and emotional learning
Universal, school-based social and emotional learning (SEL) interventions aim to develop the social and emotional skills of children and young people through explicit instruction in the context of learning environments that are safe, caring, well-managed and participatory. 1 This is typically achieved through the delivery of a taught curriculum, activities and strategies to promote a more positive school environment or ethos, and/or work with parents and the wider community. 2 Three recent meta-analyses have demonstrated that high-quality, well-implemented SEL interventions can produce practically significant improvements in a range of outcomes, including children’s social and emotional skills, mental health and academic attainment. 3–5
Social and emotional learning therefore has significant potential utility as a vehicle for both promotion (of skills) and prevention (of problems). With regard to promotion, social and emotional skills can be thought of as comprising emotional competence and relational/pro-social domains. 6 Emotional competence includes self-awareness, self-regulation and social awareness. Relational/pro-social competence includes relationship skills and responsible decision-making. The emergence of these skills follows a developmental trajectory that is influenced by both implicit and explicit socialisation practices (e.g. ‘caught’ and ‘taught’) in primary developmental contexts such as the home and school. 7 Notwithstanding the importance of emotional and relational/pro-social competence for adaptive functioning within childhood and adolescence, longitudinal research consistently notes their utility in predicting a range of salient later-life outcomes. For example, when compared with cognitive ability, childhood social and emotional skills are (1) more strongly associated with adult general mental well-being (e.g. life satisfaction, mental health); (2) similarly associated with adult health and health-related outcomes (e.g. obesity, smoking); and (3) similarly associated with some labour market outcomes (e.g. employment status). 8 They are thus increasingly referred to as ‘skills for life and work’. 8
In relation to prevention, the extant evidence base also highlights the efficacy of SEL in stemming the maintenance or progression of nascent mental health difficulties. The onset of such difficulties is characterised by changes in thought, mood and/or behaviour that impair functioning. 9 The range of mental health difficulties are typically dichotomised in terms of their directionality, with a distinction made between those that are primarily internalising in nature (e.g. depression, anxiety) and those that are externalising in nature (e.g. conduct problems). 10 Prevalence estimates suggest that 1 in 10 children and adolescents experience clinically significant mental health difficulties. 11 Furthermore, up to half of all lifetime cases of mental ill health have their first onset by the age of 14 years. 12 The consequences of such difficulties include reduced quality of life, lost economic productivity, destabilisation of communities and higher rates of health, education and social care use. 13 Indeed, by 2030, depression alone is projected to yield the highest disease burden in high-income countries, accounting for nearly 10% of disability-adjusted-life-years. 14 In England, mental health care costs approximately £105B per annum. 15
Thus, there is a compelling rationale for school-based, universal SEL interventions. Developmental theory and research suggest that early life experiences influence later life outcomes: ‘prevention pays’ from both economic and quality-of-life perspectives. 16 In addition, schools have a central role in the lives of children and their families, with unparalleled reach. 17 This means that children with emergent difficulties can access much-needed support that is unavailable through standard care pathways. By definition, universal intervention is also more inclusive and there is less chance of harmful stigmatisation than in targeted/indicated approaches. 18 Finally, health economic analyses of the effects of early mental health problems on labour market outcomes suggest clear indirect effects that operate through education. 19
The aforementioned meta-analyses, in addition to related syntheses of the literature such as those published by the National Institute for Health and Care Excellence (NICE)20 and the Early Intervention Foundation,21 highlight the distinct lack of a rigorous evidence base for universal, school-based SEL interventions in the UK. In the most recently published analysis, for example, only 4 of 89 studies reviewed originated here. 5 Thus, in terms of a robust evidence base, universal SEL is still very much in its infancy in the UK.
More broadly, a number of important gaps in the literature are evident. First, a relative lack of follow-up data to assess the maintenance of intervention effects has been noted. For example, only 33 of the 213 studies reviewed by Durlak et al. 3 collected follow-up data at least 6 months after the intervention had ceased. Second, although most SEL trials now provide accompanying descriptive data on the implementation of the intervention being evaluated (e.g. fidelity, dosage),5 relatively few formally assess the relationship between levels of implementation and outcome variability. Third, in spite of the noted health economic component of the rationale, a major review commissioned by NICE identified no published studies of the cost-effectiveness of SEL interventions. 22 Finally, the temporal relationships between intervention outcomes have been neglected,3 limiting the extent to which underpinning SEL theory can be verified. These gaps are evident in the international SEL evidence base, but are brought into even sharper focus in the UK, which is limited to a much smaller number of trials from the outset. Thus, of the four UK-based studies included in the aforementioned meta-analysis,5 none was a randomised controlled trial (RCT), none assessed the maintenance of intervention effects or the relationship between levels of implementation and intervention outcomes, none addressed cost-effectiveness, and none examined temporal relations between outcomes.
The Promoting Alternative THinking Strategies curriculum
The Promoting Alternative THinking Strategies (PATHS) curriculum aims to promote self-control, emotional understanding, positive self-esteem, relationships and interpersonal problem-solving skills among children aged 4–11 years. This is primarily achieved through the implementation of a taught curriculum by the class teacher. Lessons on topics such as identifying and labelling feelings, controlling impulses and understanding the perspectives of others are delivered approximately twice a week throughout the year. The curriculum is supplemented by generalisation activities and techniques that support the application and consolidation of new skills throughout the day. Finally, supplementary parental materials are provided, the aim of which is to extend learning to the home environment. Teachers in PATHS schools receive 1 initial day and 1 half-day follow-up of training, and are aided by trained external coaches, who offer ongoing technical support and assistance (e.g. lesson modelling, observation and feedback) throughout the school year as a means to optimise delivery. A full description of the PATHS curriculum using an adapted version of the Template for Intervention Description and Replication (TIDieR)23 can be found in Chapter 2, Intervention.
The PATHS curriculum was originally developed in the USA by Mark Greenberg and Carol Kusche as an intervention to promote social and emotional competence among children with hearing impairments. It was subsequently adapted for use in other special education settings before being revised and applied as a universal intervention in mainstream schools. PATHS has a strong international evidence base. It is one of only 14 programmes from a review of > 1400 youth promotion approaches to be awarded ‘model program’ status by the Centre for the Study and Prevention of Violence [see www.blueprintsprograms.com/factsheet/promoting-alternative-thinking-strategies-paths (accessed 21 June 2017)]. Not including the current study, there have been 10 RCTs of PATHS to date. We note that there are also many quasi-experimental and other studies of PATHS. However, we restrict our discussion to RCTs here in the interests of prioritising studies that make use of more robust evaluation designs and comparing like with like. Six of these RCTs were conducted in the USA. 24–29 With one exception,27 these US trials have consistently found small intervention effects across the range of outcomes noted at the outset of this chapter. Elsewhere in the world, RCTs of PATHS have been conducted in Switzerland,30 Croatia31 and the UK,32,33 but have generally failed to replicate the positive effects noted above, with null intention-to-treat (ITT) results reported in all but one of the trials. The exception was a trial of PATHS conducted in Northern Ireland, which found small, positive effects for a range of outcomes, including pro-social and aggressive behaviour. 32 Of the trials reporting null findings, there was evidence of small subgroup effects in two, although these were not consistent. Thus, although a recent trial of PATHS in Birmingham, England (involving children aged 5–7 years) found some limited evidence of differential gains among children with elevated emotional symptoms at baseline,33 a Croatian evaluation found positive effects only among children without mental health difficulties. Finally, a Swiss trial found evidence of a ‘sleeper’ effect on children’s externalising problems at 24-month post-intervention follow-up, having found no intervention effects immediately post intervention. 30
There are a number of reasons why PATHS may not be as successful when evaluated outside its country of origin, including issues of cultural transferability and/or the independence of such evaluations from the developer. One of the aforementioned meta-analyses offers some useful insights in this regard. Wigelsworth et al. 5 found that the transposition of SEL interventions from one country or culture to another led to significantly diluted impact in relation to outcomes such as social and emotional skills, pro-social behaviour and emotional distress. This ‘voltage drop’, which in the case of social and emotional skills equated to an average effect size difference of 0.5 between interventions trialled in their country of origin and those trialled elsewhere, could be the consequence of differences in education systems and policies, cultural beliefs and/or expectations of the roles of teachers and students. Each of these factors may act as a barrier to successful implementation and thus the achievement of intended intervention outcomes. 34 By contrast, the influence of developer involvement is less clear; in theory, this could lead to inflated intervention effects as a result of higher quality implementation and/or bias. 35 However, the same meta-analysis found no significant differences in any outcomes between SEL trials that were independent, involved the developer or were led by the developer. In relation to PATHS specifically, all developer-led or involved trials have yielded positive effects, compared with only around half of independent trials. However, it is difficult to draw firm conclusions about this pattern of findings because several of these independent studies were also conducted outside the USA, meaning that developer involvement and intercultural transferability issues are conflated.
The current study
In the light of this, there is genuine uncertainty in relation to the likely effects of PATHS when implemented outside its country of origin and evaluated independently. In the year prior to the beginning of the trial reported herein, it had been recommended for widespread adoption in an influential report to the government on early intervention. 36 More broadly, interest in and implementation of SEL had been growing in the English education system for a number of years, driven primarily by governmental promotion of the Social and Emotional Aspects of Learning (SEAL) programme from 2005 onwards, in spite of the ultimately rather limited evidence of its effectiveness. 37 More recently, there has been a resurgence of interest in the role of schools in establishing and protecting the mental health of children and young people (as evidenced by the launch of a Commons Select Committee inquiry in late 2016). Alongside this, the ‘what works’ agenda has well and truly gathered pace and has a strong foothold in education, with > 140 school-based RCTs registered on the International Standard Randomised Controlled Trial Number (ISRCTN) registry (of which around two-thirds were registered from 2010 onwards). 38
The reporting of the current study is therefore particularly timely and significant. Each of the objectives outlined in Aims and objectives was selected based on identified gaps and inconsistencies in the literature noted earlier in the chapter. Thus, we sought to contribute to the extant literature by developing the UK (and by extension, international) evidence base for PATHS through further independent evaluation of its efficacy in promoting a broader range of outcomes than had been assessed previously, among a sample of children in Key Stage 2 (KS2) of primary education. The provision of analyses to determine the presence of maintenance and/or sleeper effects at longer-term (i.e. 12- and 24-month) follow-up, associations between levels of implementation and intervention outcomes, temporal relations between outcomes, and cost-effectiveness all map directly on to identified areas for development in relation to advancing the scientific base for SEL. 2,3
Aims and objectives
The primary aim of the trial was to examine the impact of the PATHS curriculum on the social and emotional well-being of children in primary schools in England. The version of PATHS used in the trial was subjected to a process of cultural adaptation by Barnardo’s [(London, UK) the children’s charity that owns the UK licence to distribute PATHS] in order to ‘Anglicise’ it. This entailed primarily surface-level changes (e.g. modified vocabulary, photographs and names, changes to cultural references) and did not substantively change the structure, delivery model or content of the intervention. The objectives of the trial were as follows.
-
To determine the impact of PATHS on a variety of outcomes for children
Hypothesis 1: children in primary schools that are implementing PATHS over a 2-year period will demonstrate significant improvements in social skills (1a) and pro-social behaviour (1b), mental health difficulties [specifically internalising symptoms (1c) and externalising problems (1d)] and health-related quality of life (HRQoL) [specifically psychological well-being (1e), social support and peers (1f) and school environment (1g), exclusions (1h), attendance (1i) and academic attainment (1j)], compared with those of children attending control schools.
-
To determine whether or not the impact of PATHS is sustainable
Hypothesis 2: the effects of PATHS on social skills (2a) and pro-social behaviour (2b), mental health difficulties [specifically internalising symptoms (2c); and externalising problems (2d)] and HRQoL [specifically psychological well-being (2e), social support and peers (2f) and school environment (2g)] will be sustained at the 24-month post-trial follow-up.
-
To determine the impact of PATHS on children’s psychosocial adjustment to secondary school
Hypothesis 3: children in primary schools implementing PATHS over a 2-year period will demonstrate significantly better psychosocial adjustment [specifically, psychological well-being (3a), social support and peers (3b) and school environment (3c)] upon transfer to secondary school (12-month post-trial follow-up) than those attending control schools.
-
To assess the role of implementation variability in moderating the impact of PATHS on outcomes for children
Hypothesis 4: variability in the implementation of PATHS [specifically, fidelity (4a), dosage (4b), quality (4c), participant responsiveness (4d) and reach (4e)] will be significantly associated with variability in intervention outcomes.
-
To assess the validity of the logic model for SEL programmes
Hypothesis 5: social skills and perceptions of peer and social support will be associated with later perceptions of the school environment and mental health difficulties, which in turn will predict later academic attainment.
-
To examine the cost-effectiveness of PATHS
Hypothesis 6: the PATHS curriculum will demonstrate cost-effectiveness.
Chapter 2 Methods
Design
The study utilised a two-group parallel cluster RCT,39 with schools as the unit of randomisation. The Consolidated Standards of Reporting Trials (CONSORT) diagram depicting the flow of schools and children through the main trial can be seen in Figure 1. Schools allocated to the intervention arm of the trial implemented PATHS throughout the school years 2012/13 and 2013/14. Those allocated to the usual provision arm of the trial continued as normal during this period.
FIGURE 1.
Flow of schools and children through the PATHS trial. a, Thirteen schools failed to complete baseline measures because of lack of time (n = 7), lack of information technology facilities (n = 2), other priorities [e.g. Office for Standards in Education, Children’s Services and Skills (Ofsted)] (n = 1) and lack of response (n = 3). b, A total of 818 pupil surveys were not completed because of absence, changing school, non-completion or parental opt-out. c, Of the 4400 pupils who provided baseline data, five schools were lost (n = 420 pupils) and n = 623 individual pupils post intervention when surveys were not completed because of absence, changing school, non-completion or parental opt-out. However, additional pupils were gained (n = 531) who provided post-intervention data only. Numbers refer to data for primary outcome measure [Social Skills Improvement System (SSIS) score] in the main trial period (T1–T3). See Figure 8 for the full CONSORT diagram.
Ethics approval, consent and trial monitoring
The study was approved by the University Research Ethics Committee at the University of Manchester (reference number 11470).
Consent/assent involved three stages. First, eligible schools signed a memorandum of agreement (MoA) indicating their willingness to participate. The MoA contained detailed information about what participation entailed (e.g. data collection procedures and requirements), the nature of the RCT (e.g. that only half of participating schools would receive PATHS, and that this would be determined by a random allocation procedure) and what schools could expect in return (e.g. nominal payment for survey completion, aggregated survey feedback). Second, participating schools sent information and consent sheets to parents of all eligible children. Parents who wished to opt out their children from the study were able to do so by returning the opt-out form on the consent sheet to the authors via a Freepost address at the University of Manchester. In total, 133 parents (2.5%) exercised their right to opt out their children from the trial. Finally, children were provided with information about the study (including their guarantee of anonymity and right to withdraw) and were asked to give their assent to participate. No children declined assent or exercised their right to withdraw from the study. The parental opt-out procedure was repeated ahead of the onset of the follow-up (T4) phase of the study, at which point a further 24 parents (0.5%) declined to consent to their child’s continued participation, leading to a total of 157 (3%) opted out of the study.
An additional consent process was followed for any children selected by their schools to participate in the pupil focus groups that were conducted in the implementation and process evaluation (IPE) strand of the trial. However, this followed an opt-in, as opposed opt-out, parental consent procedure (i.e. explicit, written consent was sought and received for each pupil). Standard protocols were followed in respect to confidentiality and disclosure during the conduct of the focus groups; thus, children were assured that their responses would remain anonymous and confidential except in the event of the disclosure of information indicative of a child protection issue, at which point the school’s designated safeguarding lead would be informed. No such disclosures took place.
Anonymity and confidentiality were ensured through data management procedures as follows: security for online surveys was ensured using hypertext-transfer-protocol-secure data encryption. Data matching (e.g. across time, T1 to T2, and across respondent, child self-report to teacher informant-report) was achieved through the use of a unique pupil number. All qualitative data were anonymised during the transcription process, with pseudonyms given to any personally identifying information. All data were held safely on secure drives, with the University of Manchester and Microsoft Best Practice guidelines followed. Data were held behind both internal and external firewalls, and physical transportation (e.g. on flash drives) was prohibited.
Participants, recruitment and randomisation
Sample size and power
A total of 58 schools were recruited, 45 of which met the eligibility criteria for randomisation, which included signing the MoA and completing the baseline measures. There were n = 5218 eligible children (those in Years 3, 4 and 5 at the start of the 2012/13 school year) attending these schools. Based on this sample size, with an intracluster correlation coefficient (ICC) of 0.04 for our primary outcome measure, an estimated pre–post-test correlation of 0.3, power at 0.8 and Alpha at 0.05, the minimum detectable effect size for our primary outcome (social skills – see hypothesis 1a) was determined to be 0.20 standard deviations (SDs) at the randomisation stage of the trial. A standardised intervention effect size of 0.20 equates to a difference of approximately 4 points on the Social Skills Improvement System (SSIS) total social skills scale based on a SD of 20. 40
Fulfilment of objectives 2, 3 and 5 made use of a subsample (n = 1631) of the main trial cohort. These were children in Year 5 (aged 9–10) at the outset of the trial who completed their end of KS2 examinations and transferred to secondary school at its conclusion. They are hereafter referred to as the ‘transition subsample’.
Fulfilment of objective 4 made use of data from PATHS schools (n = 23) and children (n = 2676) only.
Recruitment of schools
A list of primary schools in the Greater Manchester region was compiled (n = 808). All schools were sent basic project information and invited to a recruitment event held in March 2012, and/or offered a 1 : 1 meeting with a representative from the research team. In both cases, the trial and PATHS curriculum were outlined in detail, and schools were free to ask any questions that they had about participation. As noted already, this process yielded an initial recruitment pool of 58 schools, of which 45 eventually met the criteria for randomisation.
Inclusion/exclusion criteria
All children in Years 3, 4 and 5 (aged 7–9 years) at the start of the 2012/13 school year were eligible to participate.
Randomisation
Random allocation of schools was conducted independently of the authors by the Clinical Trials Unit at the Manchester Academic Health Science Centre and was balanced by proportions of children eligible for free school meals (FSMs) and speaking English as an additional language (EAL) via adaptive stratification (minimisation). This is considered to be the ‘platinum standard’ for trials, conferring the benefits of randomisation in terms of rigour and causal inference while also guaranteeing similarity of groups on key observables. 41
Intervention
In the interests of clarity and transparency, we utilise a version of the TIDieR framework23 that has been specifically adapted for school-based interventions38 to describe PATHS.
Brief name
Promoting Alternative THinking Strategies (PATHS).
Why (rationale/theory)
Promoting Alternative THinking Strategies is based on the Affective-Behavioural-Cognitive-Developmental model of development, which emphasises the developmental integration of affect, emotion language, behaviour and cognitive understanding to promote social and emotional competence. 42 Core programme components are a taught curriculum, generalisation activities and techniques, and a set of supplementary parent materials. A version of the PATHS programme logic model adapted by the authors of the current study to fit the Evidence Based Practice Unit template43 can be seen in Figure 2.
FIGURE 2.
Adapted PATHS logic model. 43 Reproduced with permission. © Evidence Based Practice Unit (EBPU).
Who (recipients)
All children in a given class. The PATHS curriculum contains lessons for children throughout the primary phase of education.
What (materials)
Promoting Alternative THinking Strategies curriculum packs are distributed in the UK by Barnardo’s. Curriculum packs are provided for each class containing lessons (and occasional send-home activities) that cover topics such as identifying and labelling feelings, controlling impulses, reducing stress, and understanding other people’s perspectives, in addition to associated physical resources and artefacts. These include Feelings Face cards, Feelings Dictionaries and posters relating to PATHS concepts and strategies (e.g. the ‘Control Signals’ poster outlines a series of social problem-solving steps taught in PATHS lessons). In the current trial, class teachers were also given an implementation guidance manual developed by the research team, which emphasised the PATHS programme theory and the importance of effective implementation. In addition, the ‘Fingers Linked, Time to Think’ self-control strategy was developed by the research team in conjunction with the programme developer to mirror the ‘Doing Turtle’ technique used with younger pupils in PATHS. ‘Fingers Linked, Time to Think’ posters were produced and distributed to schools. Table 1 provides an overview of the lessons included in the Years 3, 4, 5 and 6 curriculum packs.
| Unit | Year group | |||
|---|---|---|---|---|
| 3 | 4 | 5 | 6 | |
| 1 | Establishing a positive classroom environment and enhancing self-esteem (2 lessons) |
Getting started (5 lessons) Jump Starta in 2012/13 only (12 lessons) |
Getting started (8 lessons) Jump Starta in 2012/13 only (12 lessons) |
Getting back in to PATHS (7 lessons) |
| 2 | Basic emotions (7 lessons) | Feelings and relationships (15 lessons) | Problem-solving (8 lessons) | Study and organisational skills (7 lessons) |
| 3 | Improving self-control, self-awareness and anger management (4 lessons) | Making good decisions (5 lessons) | Goals and identity (4 lessons) | Conflict resolution (6 lessons) |
| 4 | Using our thinking skills (4 lessons) | Being responsible and caring for others (3 lessons) | Making and keeping friends (9 lessons) | Number the stars (7 lessons) |
| 5 | Getting along with others 1 (8 lessons) | Problem-solving (14 lessons) | Being responsible and caring for others (12 lessons) | Respect (7 lessons) |
| 6 | Feelings in relationships (6 lessons) | + 3 supplementary feelings intensity lessons | + 2 supplementary control signals lessons | Endings and transitioning (4 lessons) |
| 7 | Getting along with others 2 (3 lessons) | + 4 supplementary control signals and problem-solving lessons | ||
| 8 | Feelings and expectations (3 lessons) | |||
| 9 | Feelings about school (3 lessons) | |||
| 10 | Feelings in relationships 2 (4 lessons) | |||
| 11 | Endings and transitions (2 lessons) | |||
What (procedures)
The PATHS lessons follow a common format that includes an introduction from the teacher (in which the lesson topic and objectives are introduced), a main activity (often built around a group activity or story) and a brief plenary/closure (in which learning is reviewed). Frequent prompts to elicit pupil responses and clarify learning are included throughout. The programme utilises a ‘spiral’ curriculum model, whereby (1) topics and concepts are revisited, (2) units and lessons are developmentally sequenced, (3) new learning is linked to previous learning and (4) the competence of learners increases with each successive visit to a topic or concept. In addition to this, a daily procedure of compliment-giving is encouraged using the ‘Pupil of the Day’ system, in which children are randomly selected and wear a badge or identifier to be recognisable to other pupils and staff around the school. The Pupil of the Day may be assigned special roles and responsibilities, and other pupils and staff complete a compliment sheet for the assigned pupil.
Who (implementers)
Promoting Alternative THinking Strategies is designed to be delivered by class teachers.
How (mode of delivery)
Delivery of teacher-led PATHS lessons to all children is undertaken as part of the normal class timetable. Generalisation activities and strategies are implemented routinely throughout the school day. An example of a generalisation strategy is teachers encouraging children to apply the social problem-solving steps (‘control signals’) to resolve disputes in the playground.
Where (setting)
Regular classrooms in participating schools.
When and how much (dosage)
Promoting Alternative THinking Strategies lessons last for approximately 30–40 minutes and are designed to be delivered twice-weekly throughout the school year. Curriculum packs contain an average of 40 lessons per year group.
Tailoring
Promoting Alternative THinking Strategies is a ‘manualised’ intervention and optimal fidelity is emphasised by the developer. Nonetheless, implementers are encouraged to make surface adaptations (e.g. changes of names in stories) in order to facilitate a sense of ownership and to fit to local contexts.
How well (planned)
Strategies to maximise implementation effectiveness include 1 full day of initial training for teachers with 1 half-day follow-up approximately 4 months later. Training was led by experienced PATHS trainers from Pennsylvania State University (PSU). This is supplemented by ongoing technical support and assistance (e.g. lesson modelling, observation and feedback) from trained PATHS coaches (CJ, KP and ES), who were all qualified and experienced teachers with master’s degrees in psychology and/or education.
Usual school provision
Schools allocated to the control arm of the trial continued their usual provision during the main trial period. As primary schools in England routinely engage in a range of activities that would be expected to influence the outcomes of interest in the current trial, all participating schools were surveyed at baseline in relation to their use of SEL and SEL-related interventions at universal and targeted levels in KS2. A total of 42 of the 45 schools responded, of which 21 were subsequently allocated to the usual provision arm of the trial. Survey responses in these schools were provided by headteachers (n = 7), deputy headteachers (n = 4), special educational needs co-ordinators (SENCos) (n = 3) and others [e.g. personal, social and health education (PSHE) lead, Key Stage lead, classroom teacher, n = 7]. In addition to implementing PSHE as part of the standard school curriculum, schools in the usual provision arm of the trial reported that the following universal initiatives were ‘well underway’ or ‘fully embedded’ in KS2: (1) SEAL whole-school resources (e.g. termly themes, assemblies; n = 17, 81%); (2) SEAL whole-class lessons (n = 13, 62%); (3) National Healthy Schools programme (n = 17, 81%); (4) Circle Time (n = 12, 57%); (5) the Inclusion Development Programme (Behaviour, Emotional and Social Development strand, n = 2, 10%); (6) Behaviour for Learning (n = 1, 5%); and (7) other, ‘imported’ SEL curricula (e.g. Second Step, n = 1, 5%). At the targeted level, the following initiatives were reported to be ‘well underway’ or ‘fully embedded’ in KS2: (1) SEAL small group work (n = 5, 24%); (2) Family SEAL (n = 2, 10%); (3) Targeted Mental Health in Schools (n = 4, 19%); (4) Circle of Friends (n = 4, 19%); (5) Nurture Groups (n = 6; 29%); (6) Achievement for All (n = 1, 5%); (7) Place2Be (n = 1, 5%); and (8) restorative justice (n = 5, 24%).
Assessment of outcomes
Annual assessments were completed at baseline (T1), interim (T2) and post intervention (T3) for the main trial sample in relation to hypothesis 1. In relation to hypotheses 2 and 3, assessments were completed at 12-month (T4) and 24-month (T5) follow-up for the n = 1631 subsample that had transferred to secondary school following T3. Assessments involved the completion of surveys through a secure, password-protected online platform [World App Key Survey (World App, Braintree, MA, USA), version 8.25]. Table 2 provides an overview of the outcome assessment battery.
| Hypothesis | Domain | Measure | Subscale(s) | Data source |
|---|---|---|---|---|
| 1a | Social skills | SSIS40 | Total social skills | Child |
| 1b | Pro-social behaviour | SDQ44 | Pro-social behaviour | Teacher |
| 1c | Mental health difficulties | SDQ44 | Internalising symptoms | Teacher |
| 1d | SDQ44 | Externalising problems | Teacher | |
| 1e | HRQoL | KS2745 | Psychological well-being | Child |
| 1f | KS2745 | Social support and peers | Child | |
| 1g | KS2745 | School environment | Child | |
| 1h | Exclusions | School census | Total number of sessions excluded from school | NPD |
| 1i | Attendance | School census | Proportion of days absent from school | NPD |
| 1j | Attainmenta | KS2 standardised assessment tests | English KS2 points score; Maths KS2 points score | NPD |
| 6 | QALYs | CHU-9D46 | N/A | Child |
Our instrumentation was informed by practical (e.g. data burden) and psychometric (e.g. reliability and validity) considerations, in addition to published guidance on the assessment of social, emotional and behavioural outcomes. 47,48
Social Skills Improvement System
The SSIS40 was the primary outcome measure. Specifically, the 46-item social skills domain of the self-report version of this instrument was utilised, given that an improvement in social skills is likely to be the most direct, proximal outcome of an intervention such as PATHS. The scale follows a Likert response format in which the child reads a statement (e.g. ‘I stay calm when I am teased’) and indicates their level of agreement on a four-point scale (never, sometimes, often, always). The SSIS is psychometrically sound, with good reliability (internal: α = 0.94; test–retest: r = 0.81) and strong validity [factorial: established through confirmatory factor analysis (CFA); convergent: correlates with a range of similar instruments; discriminative: discriminates between clinical and non-clinical samples]. Furthermore, the development and subsequent refinement of the SSIS utilised Item Response Theory. 40,49 Internal consistency of the total social skills subscale in the current study at baseline was α = 0.93.
Strengths and Difficulties Questionnaire
The teacher-informant-report version of the Strengths and Difficulties Questionnaire (SDQ)44 provides a measure of children’s internalising symptoms, externalising problems and pro-social behaviour. 10 Historically, the SDQ has been scored according to a five-factor structure (emotional symptoms, conduct problems, inattention/hyperactivity, peer problems and pro-social behaviour). However, research has indicated that a three-factor structure (internalising symptoms, externalising problems, pro-social behaviour) can offer an improved data fit. 10 Respondents read a statement (e.g. ‘[This child] often lies or cheats’) and indicate their level of agreement on a three-point scale (not true, somewhat true, certainly true). The SDQ has robust psychometric properties, with evidence of both reliability (internal Cronbach’s alpha of up to 0.87; test–retest r of up to 0.8) and validity (factorial: established through CFA; convergent: correlates with a range of similar instruments; predictive: strongly predictive of independently diagnosed psychiatric disorders). 10,50 Internal consistency in the current study at baseline was α = 0.79 for internalising symptoms, α = 0.89 for externalising problems and α = 0.85 for pro-social behaviour.
Kidscreen-27
The psychological well-being (seven items), peer and social support (four items) and school environment (four items) subscales of the self-report version of the Kidscreen-27 (KS27)45 were utilised. The respondent reads a statement (e.g. ‘In the last week my friends have been around when I needed them’) and indicates their level of agreement on a five-point scale (e.g. never, one day, some days, most days, every day). The KS27 is psychometrically robust, with high internal consistency (α > 0.8 for each of the above subscales; clear factor structure established through CFA), good reproducibility (test–retest ICCs for each of the above subscales are > 0.6) and criterion validity (correlates with a range of similar instruments). 45,51 Internal consistency in the current study at baseline was α = 0.77 for psychological well-being, α = 0.78 for peer and social support and α = 0.71 for school environment.
Child Health Utility 9 Dimensions
The Child Health Utility 9 Dimensions (CHU-9D)46 was utilised specifically to produce quality-adjusted life-year (QALY) data for the cost-effectiveness strand of the trial. It is a generic, multiattribute, preference-based measure, constructed specifically for use with children aged 7–11 years. In-depth qualitative analysis was used to identify each of its nine attributes – worried, sad, pain, tired, annoyed, schoolwork/homework, sleep, daily routine and activities – with each attribute containing five levels (e.g. ‘I don’t feel sad today’, ‘I feel a little bit sad today’, ‘I feel a bit sad today’, ‘I feel quite sad today’, I feel very sad today’). 46 It is psychometrically sound, with demonstrable discriminative, construct and convergent validity. The development of the CHU-9D included standard gamble modelling of preference weights for health states defined by the instrument, enabling the calculation of QALYs. 52,53
National Pupil Database
In addition to outcome data pertaining to exclusions, attendance and attainment noted in Table 2, data were extracted from the National Pupil Database (NPD) to be used as covariates in our analyses and/or to assess sample representativeness via comparison to national norms. The following child-level data were provided by the NPD: sex, FSM eligibility, EAL status, special educational needs and disability (SEND) status, index of deprivation affecting children score, ethnicity and Key Stage 1 (KS1) academic data (reading/writing and Maths). At the school level, data were gathered on size, the proportion of pupils eligible for FSMs, the proportion of pupils speaking EAL, the proportion of pupils identified as having SEND, and attendance and attainment from school-level NPD files, Edubase (the government’s online school directory) and the Department for Education (DfE) performance tables database.
Implementation and process evaluation
The IPE strand of the trial served three main purposes. First, the trial aimed to provide a clear description of ‘usual provision’ in order to establish a well-defined counter-factual. Second, the trial sought to document the implementation of PATHS in schools allocated to the intervention arm and to establish the extent to which variability in implementation moderated intervention outcomes. Third, the trial strived to explore the processes underpinning PATHS implementation, including issues pertaining to social validity (e.g. acceptability, feasibility and utility) and the factors affecting delivery at different levels (e.g. implementer factors and the implementation environment).
In relation to the documentation of usual provision, a school-level survey was developed. The research team generated a comprehensive list of SEL and SEL-related interventions that were known to be available in the UK at the time of the trial: 10 that were universal in nature (e.g. primary SEAL whole-class lessons) and nine that were targeted (e.g. nurture groups). To determine response validity, a fictional intervention name generated by the research team (‘Friends Forever’) was also included in the survey. For each intervention, respondents were required to endorse the level of implementation in their school on a scale from 0 to 3 (not implementing, just getting started, well under way, fully embedded). Descriptive data generated through this survey can be found in Usual school provision.
In order to document the implementation of PATHS, a structured observation schedule was developed, drawing upon the theoretical framework for implementation of Berkel et al. ,54 existing instruments used in other studies of PATHS,55 advice from the developer and colleagues at PSU, and existing literature on the assessment of implementation of school-based interventions. 56 Two objective indicators – one each for dosage and reach – were generated and supplemented by 10 observer-rated items designed to assess fidelity, quality and participant responsiveness. The schedule and an accompanying explanatory rubric were explained in detail to the data collection team (CJ, KP, ES) ahead of piloting and refinement. The trial used video footage of PATHS lessons being implemented in the Birmingham trial noted earlier. 33 The initial emphasis was on developing a shared understanding of the various implementation indicators and their application in the context of a PATHS lesson. Additional video footage of PATHS lessons was subsequently used to generate inter-rater reliability data. Given the multiple raters and the ordinal response format of the coding schedule, the ICC was used. The overall ICC was determined to be 0.91, which is considered to be ‘excellent’. 57 During the live trial observations, each teacher was observed implementing a single PATHS lesson at a mutually agreed date and time. A senior member of the research team (AL) moderated a randomly selected 10% of these observations in order to guard against drift over time.
Finally, in order to explore the processes underpinning PATHS implementation, the trial undertook interviews and focus groups as outlined in Table 3. The focus of this qualitative data generation process was informed by key sources of theory58 and research59 in IPE; however, it also evolved over time based on emergent issues and schools’ progress in implementation through the course of the trial. Thus, at the broadest level, early qualitative data generation focused on schools’ existing foundations for PATHS, perceptions of need (and benefit) and early installation of the curriculum. Subsequently, this shifted to implementation and the factors affecting delivery of PATHS. Issues of perceived impact (including differential gains) and sustainability were addressed towards the end of the trial.
| Method and source | Year group(s) | Number of participants | Dates | Notes |
|---|---|---|---|---|
| Class teacher interviews | 3 | 38 | November/December 2012 | 37 teachers (two a class share) and one teaching assistant from 22 of the 23 PATHS schools |
| 4 | 29 | March/April 2013 | 29 of the 32 Year 4 teachers from all 23 PATHS schools. Two teachers withdrew from interview, one teacher on sick leave | |
| 5 | 19 | March/April 2014 | 19 of the 35 teachers from 19 of the 23 PATHS schools. To reduce burden in the second year of the trial it focused on one teacher per school. Three classes no longer implementing, and two teachers on sick leave | |
| 6 | 20 | November/December 2013 | 20 of the 36 Year 6 teachers from 20 of the 23 PATHs schools. To reduce burden in second year, the trial focused on one teacher per school. Two classes no longer implementing, and one teacher left school | |
| PATHS co-ordinator interviews | N/A | 11 | May/June 2013 | PATHS co-ordinators were members of the school leadership team (and, therefore, it was not a classroom teacher implementing the PATHS curriculum) |
| Parent interviews | 3–6 | 9 | December 13–May 14 | School-nominated sample: parents of pupils from a range of year groups, both sexes and from differing socioeconomic backgrounds |
| Pupil focus groups | 3–6 | 11 | July 13–April 14 | School-nominated sample: pupils from a range of year groups, both sexes and from differing socioeconomic backgrounds |
| Field notes from lesson observations | 3 | 37 | November/December 2012 | 37 of the 38 classes from 22 of the 23 PATHS schools. One teacher left school |
| 4 | 32 | March/April 2013 | All 32 classes were observed | |
| 5 | 27 | March/April 2014 | 27 of the 32 classes were observed from 19 of the 23 PATHS schools. Inability to arrange due to two teachers being on sick leave, one teacher on maternity leave, two teachers no longer implementing PATHS and one teacher cancelling the lesson, and one teacher teaching two classes | |
| 6 | 32 | November/December 2013 | 32 of the 35 classes were observed from 21 of the 23 PATHS schools. Two classes not implementing in Year 6, and one teacher left the school |
Statistical analysis
Statistical analyses were conducted using MLWin (Centre for Multilevel Modelling, University of Bristol, Bristol, UK) version 2.36, SPSS (IBM, Armonk, NY, USA) version 22, MPlus (Muthen & Muthen, Los Angeles, CA, USA) version 8, and Stata® (Timberlake, New York, NY, USA) version 14.0. Analysis and presentation of data follow CONSORT guidelines in relation to RCTs where applicable. 60
For hypothesis 1, ITT analyses were conducted using hierarchical linear modelling (HLM) with fixed effects and random intercepts. Two-level (school, pupil) hierarchical models (controlling for T1 at the pupil level) were fitted to account for the nested nature of the data set, with T3 score as the response variable. Outcome data were standardised prior to analysis to improve ease of interpretation within and across models. 61 First, empty (‘unconditional’) models were fitted, entering only the levels and no explanatory variables. This allowed approximations of the proportion of unexplained variance attributable to each level of the model. Second, full (‘conditional’) models were fitted, including minimisation variables (percentage of FSM and percentage of pupils with EAL) and trial group (PATHS vs. control) at the school level, and sex, FSM eligibility (as covariates) and T1 score at the pupil level. For a given outcome, an intervention effect was noted if the coefficient associated with the trial group variable was statistically significant and in the expected direction (although, of course, the trial was also prepared to note negative, iatrogenic effects if the coefficient was not in the expected direction). The same approach was adopted in relation to hypotheses 2 and 3, with the exception that T4 (hypothesis 3) and T5 (hypothesis 2) scores were the response variables. Exact α values are presented in our model tables (e.g. see Table 7A) as opposed to applying corrections to account for the increased family-wise error rate (e.g. Type I error) associated with multiple comparisons, given the wide variety of correction methods available (e.g. Sidak, Bonferroni). Thus, the reader may apply a more conservative reading of the α values reported as desired (e.g. for objective 1, assuming a family of m = 10 hypotheses, α can be adjusted from 0.05 to 0.05/10 = 0.005 if applying the Bonferroni correction method).
For hypothesis 4, data reduction processes were undertaken in order to streamline subsequent analyses, avoid ‘model overfitting’62 and collinearity, and establish clear differentiation between implementation dimensions. The observer-rated implementation data on fidelity, quality and participant responsiveness were subjected to exploratory factor analysis (EFA) in SPSS using the Principal Axis Factoring extraction method (common factor analysis) with Varimax rotation (oblique rotation method) (see Table 12). Following factor extraction, factor scores were generated using the least squares regression approach63 and standardised as per our other analyses. These data were subsequently used as explanatory variables alongside single-item, factual indicators of dosage and reach in two-level (class, child) HLMs. Each class/teacher was classified as either ‘low’, ‘moderate’ or ‘high’ for each aspect of implementation using a distributional cut-point method (low, < –1 SD; moderate, –1 to +1 SD; and high, > +1 SD; in subsequent dummy coding, low implementation was the designated reference group). This approach was taken in view of the notion of ‘markers of sufficiency’ in implementation64 and to avoid an arbitrary threshold of ‘good’ implementation. 59 An exception to this approach method was reached: this was coded as high (100%), moderate (90–99%) or low (≤ 89%) according to the proportion of students present during the PATHS lesson being observed. As per the aforementioned outcome analyses, sex and FSMs were fitted at the child level alongside T1 score. Separate analyses were conducted for the first (T2 as the response variable) and second (T3 as the response variable) years of the trial.
For hypothesis 5, the study used multilevel structural equation modelling (SEM) with weighted least squares with means and variance adjustment. In brief, factor analytic approaches [e.g. multilevel confirmatory factor analysis (MCFA)] were applied to assess the structure and validity of the measurement models before proceeding to evaluate the full SEM model that represented the hypothesised temporal relations between study outcomes. Our analysis assessed both direct (e.g. initial social skills predicting later academic attainment) and indirect (e.g. initial social skills predicting later academic attainment via interim mental health difficulties) pathways in the resultant model. Model fit was assessed using multiple measures as generally recommended, specifically: chi-squared goodness of fit, the Tucker–Lewis Index (TLI), Comparative Fit Index (CFI) and root-mean-square error of approximation (RMSEA) [including 90% confidence intervals (CIs), where available].
Given the longitudinal nature of the project, missing data were considered to be inevitable. Analysis of primary outcome (SSIS) data for the main trial period (T1–T3) revealed that 35.7% (n = 1861) of cases were partially observed, with the remaining 64.3% (n = 3357) being complete cases. Logistic regression identified a range of variables that predicted partially observed status (e.g. missingness), namely being younger (e.g. pupils in Year 3), having higher absence rates in 2012/13 and higher SDQ internalising symptoms, and attending a usual provision school. Although no single approach to dealing with missing data is considered to be completely satisfactory, multiple imputation (MI) is an approach that is generally considered to be acceptable among statisticians and is increasingly used in educational research. 65 Accordingly, MI procedures were carried out in REALCOM-Impute (Centre for Multilevel Modelling, University of Bristol, Bristol, UK), using the missing at random (MAR) assumption. 66 This enabled us to include both partially and completely observed cases in our analyses, thereby reducing the bias associated with attrition. The constant trial group, partially observed outcome scores and a range of demographic and other data (e.g. sex, FSM eligibility, ethnicity, EAL, SEND provision, KS1 English and Maths scores, attendance) were entered as auxiliary variables. REALCOM-Impute default settings of 1000 iterations and a burn-in of 100 and a refresh of 10 were used, following guidance for multilevel imputation with mixed response types. 66 These processes were undertaken for the analyses pertaining to hypotheses 1–4 above where missing data exceeded 5%. 67 The exception to this was for the analysis associated with hypothesis 5, in which the full information approach was adopted as the recommended procedure for handling missing data,68 thereby enabling the inclusion of observed or estimated information for all cases.
Qualitative analysis
Qualitative data were thematically analysed following the six-stage procedure outlined by Braun and Clarke,69 as follows: (1) familiarisation, (2) initial coding, (3) searching for themes, (4) reviewing themes, (5) defining and naming themes and (6) report production. A hybrid approach was taken, involving both inductive (e.g. ‘bottom up’) and deductive (e.g. ‘top-down’) processes of thematic identification and interpretation. This enabled our analysis to be informed equally by the emergent data and our a priori theoretical and analytical interests. As is standard in thematic analysis, negative ‘disconfirming’ examples were coded alongside those supporting a given theme in order to provide a fuller and more nuanced understanding of the processes underpinning PATHS implementation. The deductive aspect of the analysis was driven by key sources of knowledge in implementation science (e.g. Durlak and DuPre’s59 seminal review of implementation; see Chapter 4).
Economic analysis
For hypothesis 6, a microcosting approach was undertaken in line with previous efforts to cost school-based SEL interventions. 22,33 This included the recurrent and non-recurrent costs of the intervention in the trial setting reported herein, and those that would be borne by policy-makers in the event of a national roll-out. QALYs were calculated from CHU-9D utility values, using the area under the curve method. Incremental cost and QALY estimates were used to calculate incremental net benefits (INBs) using cost-effectiveness thresholds of between £0 and £150,000 (increasing in £1000 increments), thereby including the NICE threshold of £20,000–30,000 per QALY. 70 Cost-effectiveness acceptability curves (CEACs) were subsequently produced to provide an estimate of the probability of PATHS being cost-effective in different scenarios via a series of sensitivity analyses. As per hypotheses 1–4, the economic analyses utilised HLM to account for data clustering and MI to deal with missing data.
Patient and public involvement
Target group and stakeholder involvement in the project was via the trial steering group, which consisted of a primary school teacher (from a non-trial school that had previously delivered PATHS), an educational psychologist from one of the local authorities in which trial recruitment took place, representatives from Barnardo’s, Young Minds (London, UK), the Education Endowment Foundation [EEF (London, UK)] and the Manchester Academic Health Science Centre (Manchester, UK), in addition to an independent academic chair. The steering group met periodically throughout the project and served a governance function, which included the provision of advice on the content and format of public reports produced at the beginning of the trial and feedback reports for participating schools, as well as reviewing project progress against agreed milestones. A number of non-academic dissemination activities also took place during the project, including presentations of emergent findings at Barnardo’s inaugural PATHS Education UK conference (London, 2015) and Place2Be’s Research into Practice seminar (London, 2016).
Chapter 3 Outcome analyses
This chapter presents descriptive data on the study sample, followed by analyses pertaining to trial outcomes (hypotheses 1, 2, 3 and 5).
School characteristics
Table 4 provides summary data pertaining to trial school characteristics, alongside national averages derived from DfE statistical releases in the baseline year of the trial. The composition of the trial school sample mirrored that of primary schools in England in respect of attendance, attainment and the proportion of pupils speaking EAL, but our schools were significantly larger, with higher proportions of pupils of pupils eligible for FSMs and lower proportions identified as having SENDs. There were no significant differences between trial arms for any of these variables (F6,36 = 1.02; p = 0.427).
| Characteristic | School sample, mean (SD) | National average (2011/12)a | ||
|---|---|---|---|---|
| PATHS | Usual provision | Overall trial sample | ||
| Size (number of full-time pupils on roll) | 313.26 (111.15) | 287.36 (96.47) | 300.60 (103.87) | 242.46b |
| FSMs (percentage of pupils) | 30.13 (20.12) | 30.86 (19.29) | 30.49 (19.50) | 18.2b |
| EAL (percentage of pupils) | 20.63 (24.65) | 23.55 (24.51) | 22.06 (24.34) | 17.5 |
| SEND (percentage of pupils) | 16.59 (5.99) | 17.28 (5.96) | 16.93 (5.92) | 19.8b |
| Attendance (percentage of half days absent) | 5.0 (0.94) | 5.60 (1.24) | 5.32 (1.12) | 5.2 |
| Attainment (percentage of pupils achieving ≥ level 4 in English and Maths) | 80.73 (12.09) | 75.38 (11.88) | 78.12 (12.15) | 79 |
Balance between trial arms
Sample characteristics at baseline by trial arm are presented in Table 5. Balance between trial arms at baseline was considered to be good, with differences between outcome measures all d = 0.16 or less. The trial arms were also well balanced in terms of sociodemographic characteristics.
| Characteristica | Trial arm | |
|---|---|---|
| PATHS (N = 2676) | Usual provision (N = 2542) | |
| Sex, n (%) | ||
| Male | 1335 (49.9) | 1346 (53.0) |
| Female | 1341 (50.1) | 1196 (47.0) |
| Year group, n (%) | ||
| Year 3 | 974 (36.4) | 908 (35.7) |
| Year 4 | 855 (32.0) | 850 (33.4) |
| Year 5 | 847 (31.7) | 784 (30.8) |
| FSM eligibility, n (%) | ||
| Yes | 837 (31.3) | 696 (27.4) |
| No | 1748 (65.3) | 1744 (78.6) |
| Unclassified | 91 (3.4) | 102 (4.0) |
| Ethnicity, n (%) | ||
| White | 1856 (69.4) | 1682 (66.2) |
| Black | 243 (9.1) | 125 (4.9) |
| Asian | 234 (8.7) | 351 (13.8) |
| Chinese | 19 (0.7) | 14 (0.6) |
| Mixed | 115 (4.3) | 174 (6.8) |
| Any other ethnicity | 66 (2.5) | 60 (2.4) |
| Unclassified | 133 (5.4) | 136 (5.8) |
| First language, n (%) | ||
| English | 2017 (75.4) | 1872 (73.6) |
| Other | 562 (21.0) | 568 (22.3) |
| Unclassified | 67 (3.6) | 102 (4.0) |
| SEND provision, n (%) | ||
| None | 2102 (78.6) | 1931 (76.0) |
| School Action | 328 (12.3) | 318 (12.5) |
| School Action Plus | 136 (5.1) | 161 (6.3) |
| Statement | 19 (0.7) | 30 (1.2) |
| Unclassified | 91 (3.4) | 102 (4.0) |
| Teacher informant report data, mean (SD) | ||
| Internalising symptoms | 2.77 (3.25) | 2.93 (3.13) |
| Externalising problems | 4.15 (4.43) | 4.22 (4.38) |
| Pro-social behaviour | 7.72 (2.46) | 7.43 (2.42) |
| Pupil self-report data, mean (SD) | ||
| Social skills | 106.16 (19.94) | 103.47 (20.40) |
| Psychological well-being | 29.76 (4.80) | 28.97 (5.04) |
| Social support and peers | 16.90 (3.34) | 16.37 (3.54) |
| School environment | 17.75 (2.63) | 17.35 (2.88) |
| Academic attainment, mean (SD) | ||
| KS1 Reading/Writing | 14.99 (3.67) | 14.81 (3.98) |
| KS1 Maths | 15.49 (3.59) | 15.43 (3.73) |
Objective 1: to determine the impact of Promoting Alternative THinking Strategies on a variety of outcomes for children
Hypothesis 1: children in primary schools that are implementing PATHS over a 2-year period will demonstrate significant improvements in social skills (1a) and pro-social behaviour (1b); mental health difficulties [specifically internalising symptoms (1c) and externalising problems (1d)]; HRQoL [specifically psychological well-being (1e), social support and peers (1f) and school environment (1g)], exclusions (1h), attendance (1i) and academic attainment (1j) compared with those of children attending control schools.
Tables 6 and 7A–C present descriptive and inferential statistics in relation to hypothesis 1a–j. In these, and the analyses that follow, the coefficient β pertaining to trial group allocation [‘trial group (if PATHS)’ in the tables] can be read as the equivalent of Cohen’s d, as outcome data were standardised prior to analysis (see Chapter 2, Statistical analysis). Thus, in the text following, β as d are reported with accompanying 95% CIs.
| Outcome | Intervention, mean (SD) | |||
|---|---|---|---|---|
| PATHS | Usual provision | |||
| Baseline (T1) | Post intervention (T3) | Baseline (T1) | Post intervention (T3) | |
| Social skills | 106.16 (19.94) | 103.75 (19.39) | 103.47 (20.40) | 102.00 (20.03) |
| Internalising symptoms | 2.77 (3.25) | 2.53 (3.03) | 2.93 (3.13) | 2.37 (3.12) |
| Externalising problems | 4.15 (4.43) | 3.44 (3.90) | 4.21 (4.38) | 3.38 (3.69) |
| Pro-social behaviour | 7.72 (2.46) | 7.86 (2.30) | 7.15 (6.19) | 7.63 (2.37) |
| Psychological well-being | 29.76 (4.80) | 30.16 (4.35) | 28.97 (5.04) | 29.63 (4.97) |
| Social support and peers | 16.90 (3.34) | 17.32 (3.04) | 16.37 (3.54) | 16.94 (3.32) |
| School environment | 17.75 (2.63) | 17.81 (2.45) | 17.35 (2.88) | 17.66 (2.62) |
| Exclusions (total number of sessions) | 0.05 (0.82) | 0.06 (1.16) | 0.06 (0.86) | 0.10 (1.53) |
| Attendance (percentage of half days absent) | 4.24 (5.19) | 3.60 (4.80) | 4.48 (5.17) | 3.57 (4.35) |
| Reading/writinga | 14.86 (4.06) | 28.73 (4.09) | 14.39 (4.09) | 28.52 (4.33) |
| Mathsa | 15.32 (4.09) | 29.11 (5.36) | 14.98 (3.89) | 28.77 (5.25) |
| Level | Outcome | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Social skills | Internalising symptoms | Externalising problems | Pro-social behaviour | |||||||||
| β0ij = –0.239 (0.051) | β0ij = –0.034 (0.065) | β0ij = 0.095 (0.077) | β0ij = 0.225 (0.087) | |||||||||
| Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | |
| School | 0.020 | 0.008 | 0.008 | 0.060 | 0.023 | 0.006 | 0.064 | 0.022 | 0.003 | 0.053 | 0.020 | 0.006 |
| Trial group (if PATHS) | 0.087 | 0.060 | 0.077 | 0.031 | 0.095 | 0.372 | 0.023 | 0.095 | 0.406 | 0.065 | 0.096 | 0.253 |
| % FSM | –0.031 | 0.029 | 0.145 | –0.096 | 0.053 | 0.039 | –0.043 | 0.050 | 0.197 | 0.082 | 0.042 | 0.029 |
| % EAL | –0.012 | 0.032 | 0.353 | 0.056 | 0.063 | 0.189 | 0.060 | 0.063 | 0.174 | –0.087 | 0.049 | 0.041 |
| Pupil | 0.825 | 0.018 | < 0.001 | 0.857 | 0.022 | < 0.001 | 0.586 | 0.018 | < 0.001 | 0.763 | 0.020 | < 0.001 |
| Sex (if female) | 0.384 | 0.030 | < 0.001 | –0.032 | 0.037 | 0.194 | –0.268 | 0.026 | < 0.001 | 0.429 | 0.033 | < 0.001 |
| FSM (if yes) | –0.111 | 0.033 | < 0.001 | 0.188 | 0.038 | < 0.001 | 0.165 | 0.034 | < 0.001 | –0.159 | 0.038 | < 0.001 |
| Baseline (T1) score | 0.311 | 0.018 | < 0.001 | 0.335 | 0.017 | < 0.001 | 0.579 | 0.015 | < 0.001 | 0.333 | 0.017 | < 0.001 |
| Level | Outcome | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Psychological well-being | Social support and peers | School environment | |||||||
| β0ij = –0.111 (0.053) | β0ij = –0.111 (0.053) | β0ij = –0.144 (0.057) | |||||||
| Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | |
| School | 0.022 | 0.014 | 0.062 | 0.020 | 0.010 | 0.026 | 0.019 | 0.007 | 0.005 |
| Trial group (if PATHS) | 0.115 | 0.067 | 0.046 | 0.107 | 0.068 | 0.062 | 0.028 | 0.061 | 0.232 |
| % FSM | 0.017 | 0.031 | 0.293 | 0.074 | 0.031 | 0.011 | 0.037 | 0.032 | 0.126 |
| % EAL | –0.028 | 0.030 | 0.179 | –0.034 | 0.029 | 0.124 | –0.015 | 0.031 | 0.317 |
| Pupil | 0.940 | 0.023 | < 0.001 | 0.953 | 0.023 | < 0.001 | 0.899 | 0.022 | < 0.001 |
| Sex (if female) | 0.045 | 0.031 | 0.074 | 0.143 | 0.031 | < 0.001 | 0.310 | 0.033 | < 0.001 |
| FSM (if yes) | –0.083 | 0.049 | 0.046 | –0.127 | 0.045 | < 0.001 | –0.169 | 0.042 | < 0.001 |
| Baseline (T1) score | 0.212 | 0.017 | < 0.001 | 0.199 | 0.019 | < 0.001 | 0.223 | 0.018 | < 0.001 |
| Level | Outcome | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Exclusions | Attendance | Englisha | Mathsa | |||||||||
| β0ij = –0.027 (0.027) | β0ij = –0.013 (0.035) | β0ij = –0.048 (0.050) | β0ij = –0.070 (0.051) | |||||||||
| Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | |
| School | 0.000 | 0.000 | 0.500 | 0.014 | 0.005 | 0.004 | 0.031 | 0.009 | < .001 | 0.029 | 0.009 | < 0.001 |
| Trial group (if PATHS) | –0.040 | 0.029 | 0.088 | 0.016 | 0.045 | 0.360 | –0.020 | 0.063 | 0.375 | 0.038 | 0.063 | 0.276 |
| % FSM | 0.009 | 0.017 | 0.300 | –0.024 | 0.028 | 0.197 | –0.047 | 0.035 | 0.094 | –0.025 | 0.035 | 0.241 |
| % EAL | –0.031 | 0.015 | 0.022 | –0.044 | 0.026 | 0.007 | 0.065 | 0.035 | 0.035 | 0.077 | 0.035 | 0.017 |
| Pupil | 1.008 | 0.020 | < 0.001 | 0.747 | 0.018 | < 0.001 | 0.387 | 0.014 | < 0.001 | 0.472 | 0.017 | < 0.001 |
| Sex (if female) | –0.093 | 0.028 | < 0.001 | –0.043 | 0.025 | 0.043 | –0.032 | 0.033 | 0.167 | –0.116 | 0.036 | < 0.001 |
| FSM (if yes) | 0.131 | 0.035 | < 0.001 | 0.162 | 0.034 | < .001 | –0.121 | 0.040 | 0.001 | –0.133 | 0.044 | 0.001 |
| Baseline (T1) score | 0.131 | 0.016 | < 0.001 | 0.455 | 0.016 | < .001 | 0.743 | 0.017 | < 0.001 | 0.679 | 0.019 | < .001 |
Our analyses revealed marginal, non-significant trends (e.g. a p-value of < 0.10) indicative of intervention effects of PATHS in relation to children’s social skills (1a) (d = 0.09, 95% CI –0.03 to 0.20; p = 0.08), their perceptions of social support and peers (1f) (d = 0.11, 95% CI –0.03 to 0.24; p = 0.06), and the total number of sessions for which they were excluded from school (1h) (d = –0.04, 95% CI –0.10 to 0.02; p = 0.09). In addition, a statistically significant intervention effect of PATHS on children’s psychological well-being (1e) (d = 0.12, 95% CI –0.02 to 0.25; p < 0.05) was found. No intervention effects were found in relation to children’s pro-social behaviour (1b) (d = 0.07, 95% CI –0.12 to 0.25; p > 0.05), internalising symptoms (1c) (d = 0.031, 95% CI –0.16 to 0.22; p > 0.05), externalising problems (1d) (d = 0.02, 95% CI –0.16 to 0.21; p > 0.05), perceptions of the school environment (1g) (d = 0.03, 95% CI –0.09 to 0.15; p > 0.05), attendance (1i) (d = 0.02, 95% CI –0.07 to 0.10; p > 0.05), or academic attainment (1j) in English (d = –0.02, 95% CI –0.14 to 0.10; p > 0.05) or Maths (d = 0.04, 95% CI –0.09 to 0.16; p > 0.05).
Objective 2: to determine whether the impact of Promoting Alternative THinking Strategies is sustainable
Hypothesis 2: the effects of PATHS on social skills (2a) and pro-social behaviour (2b); mental health difficulties [specifically internalising symptoms (2c) and externalising problems (2d)]; HRQoL [specifically psychological well-being (2e), social support and peers (2f) and school environment (2g)] will be sustained at 24-month post-trial follow-up.
In view of the findings reported pertaining to objective 1, the interpretation of analyses for objective 2 changes slightly, such that the distinction is made between establishing whether or not intervention effects are sustained [psychological well-being (2e) and, marginally, social skills (2a) and social support and peers (2f)], and determining the presence of sleeper effects (all other variables noted earlier, where immediate post-intervention outcomes yielded null results). The reader is also reminded that analyses pertaining to objective 2 utilise outcome data from the subgroup of young people (n = 1631) who transferred to secondary school at the end of the main trial.
Tables 8 and 9A–B present descriptive and inferential statistics in relation to hypotheses 2a–g. Our analyses revealed no statistically significant differences between the PATHS and usual provision arms of the trial in respect of children’s social skills (2a) (d = –0.01, 95% CI –0.38 to 0.37; p > 0.05), pro-social behaviour (2b) (d = 0.15, 95% CI –0.23 to 0.52; p > 0.05), internalising symptoms (2c) (d = –0.05, 95% CI –0.45 to 0.35; p > 0.05), externalising problems (2d) (d = –0.04, 95% CI –0.40 to 0.33; p > 0.05), psychological well-being (2e) (d = –0.09, CI –0.56 to 0.39; p > 0.05), perceptions of social support and peers (2f) (d = –0.09, CI –0.48 to 0.30; p > 0.05), or perceptions of their school environment (2 g) (d = –0.12, 95% CI –0.51 to 030; p > 0.05) at the 24-month follow-up.
| Outcome | Intervention, mean (SD) | |||
|---|---|---|---|---|
| PATHS | Usual provision | |||
| Baseline (T1) | Follow-up (T5) | Baseline (T1) | Follow-up (T5) | |
| Social skills | 105.54 (20.44) | 97.81 (19.45) | 99.70 (19.94) | 95.94 (20.34) |
| Internalising symptoms | 3.08 (3.51) | 2.80 (3.44) | 3.26 (3.29) | 3.04 (3.36) |
| Externalising problems | 4.23 (4.39) | 3.47 (4.11) | 4.48 (4.50) | 3.69 (3.61) |
| Pro-social behaviour | 7.59 (2.49) | 7.18 (2.54) | 7.21 (2.53) | 6.50 (2.58) |
| Psychological well-being | 30.28 (4.51) | 29.51 (4.25) | 28.84 (5.10) | 29.17 (4.98) |
| Social support and peers | 17.24 (3.15) | 16.71 (3.34) | 16.19 (3.71) | 16.83 (3.05) |
| School environment | 17.79 (2.60) | 16.70 (2.84) | 17.28 (2.81) | 16.73 (2.87) |
| Level | Outcome | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Social skills | Internalising symptoms | Externalising problems | Pro-social behaviour | |||||||||
| β0ij = –0.143 (0.170) | β0ij = –0.030 (0.189) | β0ij = –0.058 (0.144) | β0ij = –0.257 (0.161) | |||||||||
| Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | |
| School | 0.214 | 0.102 | 0.021 | 0.188 | 0.078 | 0.010 | 0.117 | 0.053 | 0.016 | 0.178 | 0.090 | 0.027 |
| Trial group (if PATHS) | –0.007 | 0.192 | 0.484 | –0.049 | 0.204 | 0.406 | –0.036 | 0.186 | 0.425 | 0.149 | 0.191 | 0.220 |
| % FSM | –0.058 | 0.105 | 0.293 | 0.091 | 0.106 | 0.197 | 0.062 | 0.089 | 0.244 | 0.008 | 0.119 | 0.472 |
| % EAL | –0.002 | 0.102 | 0.492 | 0.010 | 0.112 | 0.464 | 0.052 | 0.086 | 0.276 | –0.037 | 0.086 | 0.335 |
| Pupil | 0.941 | 0.089 | < 0.001 | 0.906 | 0.058 | < 0.001 | 0.859 | 0.069 | < 0.001 | 0.912 | 0.089 | < 0.001 |
| Sex (if female) | 0.184 | 0.085 | 0.016 | –0.019 | 0.113 | 0.433 | –0.148 | 0.080 | 0.032 | 0.314 | 0.085 | < 0.001 |
| FSM (if yes) | –0.049 | 0.132 | 0.356 | 0.226 | 0.101 | 0.013 | 0.251 | 0.084 | 0.001 | –0.215 | 0.125 | 0.043 |
| Baseline score | 0.226 | 0.046 | < 0.001 | 0.306 | 0.043 | < 0.001 | 0.359 | 0.041 | < 0.001 | 0.209 | 0.050 | < 0.001 |
| Level | Outcome | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Psychological well-being | Social support and peers | School environment | |||||||
| β0ij = –0.127 (0.174) | β0ij = –0.033 (0.117) | β0ij = –0.128 (0.138) | |||||||
| Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | |
| School | 0.159 | 0.064 | 0.008 | 0.166 | 0.072 | 0.013 | 0.206 | 0.087 | 0.011 |
| Trial group (if PATHS) | –0.086 | 0.245 | 0.364 | –0.089 | 0.197 | 0.327 | –0.117 | 0.198 | 0.279 |
| % FSM | –0.003 | 0.085 | 0.484 | 0.077 | 0.095 | 0.221 | 0.188 | 0.094 | 0.026 |
| % EAL | 0.000 | 0.098 | 0.500 | –0.038 | 0.102 | 0.357 | –0.092 | 0.126 | 0.235 |
| Pupil | 0.984 | 0.070 | < 0.001 | 0.970 | 0.049 | < 0.001 | 0.958 | 0.089 | < 0.001 |
| Sex (if female) | –0.098 | 0.084 | 0.121 | 0.262 | 0.093 | 0.002 | 0.007 | 0.087 | 0.468 |
| FSM (if yes) | –0.076 | 0.114 | 0.251 | –0.020 | 0.101 | 0.421 | –0.102 | 0.151 | 0.248 |
| Baseline score | 0.219 | 0.048 | < 0.001 | 0.167 | 0.042 | < 0.001 | 0.193 | 0.045 | < 0.001 |
Objective 3: to determine the impact of Promoting Alternative THinking Strategies on children’s psychosocial adjustment to secondary school
Hypothesis 3: children in primary schools implementing PATHS over a 2-year period will demonstrate significantly better psychosocial adjustment [specifically, psychological well-being (3a), social support and peers (3b) and school environment (3c)] upon transfer to secondary school (12-month post-trial follow-up) than those attending control schools.
Tables 10 and 11 present descriptive and inferential statistics in relation to hypotheses 3a–c. Our analyses revealed no statistically significant differences between the PATHS and usual provision arms of the trial in respect of children’s psychological well-being (3a) (d = 0.05, 95% CI –0.17 to 0.27; p > 0.05), perceptions of their social support and peers (3b) (d = 0.02, 95% CI –0.22 to 0.27; p > 0.05) or perceptions of their school environment (3c) (d = 0.04, 95% CI –0.20 to 0.27; p > 0.05) at 1-year post-intervention follow-up.
| Outcome | Intervention, mean (SD) | |||
|---|---|---|---|---|
| PATHS | Usual provision | |||
| Baseline (T1) | Follow-up (T4) | Baseline (T1) | Follow-up (T4) | |
| Psychological well-being | 30.28 (4.51) | 29.64 (4.38) | 28.84 (5.10) | 29.18 (4.58) |
| Social support and peers | 17.24 (3.15) | 17.12 (3.13) | 16.19 (3.71) | 16.90 (3.28) |
| School environment | 17.79 (2.60) | 17.10 (2.82) | 17.28 (2.81) | 16.94 (3.05) |
| Level | Outcome | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Psychological well-being | Social support and peers | School environment | |||||||
| β0ij = –0.047 (0.095) | β0ij = –0.105 (0.091) | β0ij = –0.006 (0.089) | |||||||
| Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | |
| School | 0.078 | 0.044 | 0.042 | 0.094 | 0.040 | 0.012 | 0.069 | 0.031 | 0.016 |
| Trial group (if PATHS) | 0.049 | 0.113 | 0.335 | 0.021 | 0.125 | 0.433 | 0.035 | 0.118 | 0.383 |
| % FSM | –0.043 | 0.070 | 0.272 | –0.036 | 0.063 | 0.232 | –0.065 | 0.062 | 0.150 |
| % EAL | 0.022 | 0.069 | 0.375 | 0.055 | 0.074 | 0.232 | 0.076 | 0.060 | 0.106 |
| Pupil | 0.945 | 0.043 | < 0.001 | 0.928 | 0.039 | < 0.001 | 0.936 | 0.045 | < 0.001 |
| Sex (if female) | –0.110 | 0.065 | 0.046 | 0.189 | 0.073 | 0.005 | 0.080 | 0.084 | 0.171 |
| FSM (if yes) | 0.009 | 0.098 | 0.464 | –0.003 | 0.077 | 0.484 | –0.160 | 0.088 | 0.034 |
| Baseline score | 0.245 | 0.037 | < 0.001 | 0.234 | 0.031 | < 0.001 | 0.207 | 0.038 | < 0.001 |
Objective 5: to assess the validity of the logic model for social and emotional learning programmes
Hypothesis 5: social skills and peer and social support will be associated with later school environment and mental health difficulties, which in turn will predict later academic attainment.
Analyses pertaining to objective 5 utilised outcome data for the aforementioned transition subsample (n = 1631), as this enabled us to make use of pupils’ KS2 academic attainment data as the distal outcome in our model. The conceptual model tested is shown in Figure 3 and represents our approximation of the generic SEL logic model,75 based on the outcomes assessed in the main trial.
FIGURE 3.
Conceptual model adapted from the SEL logic model.
In line with the temporal nature of the conceptual model, baseline data on children’s social skills (SSIS) and perceptions of peer and social support (KS2745) were extracted at T1, proximal data on perceptions of the school environment (KS2745) and mental health difficulties (SDQ – total difficulties score utilised) were extracted at T2, and distal data on academic attainment (KS2) were extracted at T3.
Table 12 presents descriptive statistics for these variables. Overall, small to medium correlations were observed between the study variables. Directionality was as expected (e.g. mental health difficulties correlated negatively with academic attainment, perceptions of peer and social support, perceptions of the school environment and social skills).
| Variable number | Outcome | Variable number | ||||
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||
| 1 | Academic attainment (T3) | – | ||||
| 2 | Perceptions of peer and social support (T1) | 0.07 | – | |||
| 3 | Social skills (T1) | 0.10 | 0.46 | – | ||
| 4 | Perceptions of school environment (T2) | 0.19 | 0.22 | 0.38 | – | |
| 5 | Mental health difficulties (T2) | –0.42 | –0.18 | –0.22 | –0.40 | – |
| Mean | 28.82 | 16.75 | 102.79 | 17.54 | 6.13 | |
| SD | 4.29 | 3.46 | 20.41 | 2.65 | 6.03 | |
| Cronbach’s alpha | N/A | 0.81 | 0.94 | 0.74 | 0.88 | |
Measurement models
Given the nested nature of the data set, the ICC was estimated for all variables. The design effect (Deff), which indicates the extent to which standard errors (SEs) are underestimated in a clustered sample compared with a random one,76 was also calculated, using a threshold of Deff > 2 to indicate the need to utilise multilevel models. 77 Given, however, that the Deff formula is merely an approximation, the SEs of multilevel models were also compared against those of single level models. As such, multilevel procedures were not applied to those models with similar SEs.
The ICC values of the categorical observed variables mirrored those from existing studies assessing similar outcomes. 78 For the majority of the items, the Deff value was found to be > 2, and differences in SEs were observed between the single- and multi-level models. Given this, MCFA models were used to assess factor structures. The MCFA model fit and factor loadings were examined for all measures and are presented in Table 13. High-factor loadings (λ > 0.40)79 were observed for all items and were thus considered to represent meaningful constructs. In addition, model fit was found to be adequate for all measures, confirming their use as latent factors in the model.
| Measure | Model | χ 2 | DF | RMSEA | CFI | TLI | ICC range | λ rangea |
|---|---|---|---|---|---|---|---|---|
| Perceptions of peer and social support | Univariate | 14.633* | 7 | 0.027 | 0.999 | 0.998 | 0.021–0.043 | 0.65–0.83 |
| Social skills | 7-factor second order | 2821.928*** | 2017 | 0.016 | 0.955 | 0.954 | 0.017–0.086 | 0.52–0.78 |
| Perceptions of school environment | Univariate | 14.582 | 8 | 0.025 | 0.996 | 0.994 | 0.012–0.064 | 0.68–0.76 |
| Mental health difficulties | 4-factor second order | 1018.242*** | 356 | 0.039 | 0.961 | 0.959 | 0.073–0.249 | 0.35–0.94 |
Logic model
The multilevel SEM model (Figure 4) was constructed in line with the conceptual model shown in Figure 3. The cross-domain relationships of the model were assessed while accounting for the data clustering and within-time covariance. Measures taken at the same time point were allowed to covary, and baseline (T1) variables were regressed onto academic attainment (T3) in order to control for the unique contribution of the proximal variables (T2). Four possible indirect relationships between the two base variables and academic attainment were identified and statistically assessed. With the exception of academic attainment, domains were modelled as latent variables, as they estimate and remove item-level measurement error, thereby providing more accurate estimates of autoregressive estimates. 80
FIGURE 4.
Multilevel SEM of the SEL logic model for the transition subsample. ***, p < 0.001; †, p < 0.10.
In accordance with existing research,81,82 the ICC of academic attainment was substantial (ICC = 12.4%, Deff = 5.36), providing further justification for the use of multilevel modelling techniques. In line with recommended thresholds, model fit was considered to be acceptable if TLI and CFI were > 0.95, and RMSEA was < 0.05. 83 The chi-squared goodness of fit statistic was calculated but not used to assess model fit given its sensitivity to sample size. The model provided an acceptable fit to the data: χ2(5455) = 6816.604, RMSEA = 0.012, CFI = 0.945, TLI = 0.944.
Within-time correlations were statistically significant and in the expected direction. Overall, 32% of the variance in academic attainment was explained by the overall model. However, only social skills and mental health difficulties were found to be statistically significant predictors of academic attainment. In addition, although earlier analysis (see Table 12) indicated a statistically significant relationship between peer and social support and subsequent perceptions of the school environment, this relationship was not evident in the SEM analysis. Consistent with the SEL logic model, greater social skills at T1 were associated with more positive perceptions of the school environment at T2. However, such positive perceptions were not subsequently associated with academic attainment at T3. Mental health difficulties were found to have the strongest explanatory power in the model, such that more difficulties at T2 predicted lower academic attainment at T3.
Contrary to expectations, the direct effect of social skills on academic attainment was found to be negative (although small in magnitude and only marginally significant). It is believed that this was the result of inconsistent mediation. Inconsistent mediation is observed when an indirect effect has a different sign from a direct relationship, and it is known to occur in multiple mediator models. 84 In this case, the direct effect from social skills to academic attainment (β = –0.07; p = 0.05) was shown to have a different sign from that of its corresponding indirect effect through mental health difficulties (β = 0.24, b = 1.17; p < 0.001). In other words, the impact of social skills on academic attainment becomes negative when adjusting for the effects of mental health difficulties. Indeed, when tested alone, social skills at T1 were found to be positively associated with academic attainment at T3 (β = 0.67; p < 0.001), as hypothesised. Notably, this effect was the only statistically significant indirect relationship observed in the model.
Chapter 4 Implementation and process evaluation
This chapter provides descriptive data on PATHS implementation, analyses pertaining to the associations between levels of implementation and outcomes and an analysis of qualitative data. Through these the study explores the processes underpinning PATHS implementation, the factors affecting delivery at different levels and perceptions of impact among key stakeholders. This exploration was intended to provide a richer and more nuanced picture of implementation than would be possible through the structured observations alone, and, critically, to identify why implementation was variable.
Objective 4: to assess the role of implementation variability in moderating the impact of Promoting Alternative THinking Strategies on outcomes for children
Hypothesis 4: variability in the implementation of PATHS [specifically, fidelity (4a), dosage (4b), quality (4c), participant responsiveness (4d) and reach (4e)] will be significantly associated with variability in intervention outcomes.
In the light of our factor analysis of the structured observation data set (see Chapter 2, Statistical analysis), it is noted that hypotheses 4c and 4d are combined and thus reference is made to ‘hypothesis 4c/d’ from this point onwards.
Table 14 presents descriptive data on PATHS implementation in each year of the trial. Overall, a general trend can be seen in which PATHS was delivered once a week, with most children in a given class present, teachers adhering to most procedural elements outlined in lesson materials and delivering them well, and with children responding appropriately. However, there is a clear downwards shift in dosage between the first and second years of the trial, such that in the former, teachers were estimated to deliver 65% of lessons, whereas in the latter, this dropped to 39% (overall average of 53%). The reasons for this trend are explored in Qualitative analysis and also in Chapter 6.
| Item | Scoring | 2012/13, mean (SD)a | 2013/14, mean (SD)b | Overall, mean (SD) | Initial designation | Factor | |
|---|---|---|---|---|---|---|---|
| Quality and responsiveness | Procedural fidelity | ||||||
| Projected dosage (percentage of lessons complete by end of the school year) based on progress against the implementation schedule? | 0–100 | 65.18 (17.43) | 39.09 (13.92) | 53.06 (20.53) | Dosage | N/A | N/A |
| Proportion of the class present during the lesson? | 0–100 | 91.86 (1.10) | 90.85 (1.00) | 91.34 (1.06) | Reach | N/A | N/A |
| To what extent does the teacher cover the general and specific objectives of the lesson? | 0–10 | 8.74 (1.46) | 8.41 (1.96) | 8.58 (1.71) | Fidelity | 0.57 | 0.048 |
| To what extent does the teacher follow the structure and sequence of activities outlined in the lesson guidance? | 0–10 | 8.04 (2.33) | 8.37 (2.04) | 8.20 (2.20) | Fidelity | 0.09 | 0.96 |
| How closely does the teacher adhere to the guidance when teaching the core activities of the lesson? | 0–10 | 7.37 (2.22) | 7.81 (1.98) | 7.57 (2.11) | Fidelity | 0.08 | 0.87 |
| How well prepared is the teacher for the lesson? | 0–10 | 8.68 (1.29) | 8.69 (1.58) | 8.69 (1.42) | Quality | 0.74 | 0.30 |
| Rate the teacher’s interest and enthusiasm in his/her delivery of the lesson | 0–10 | 8.99 (1.11) | 8.88 (1.39) | 8.94 (1.25) | Quality | 0.72 | 0.18 |
| How clearly does the teacher explain key concepts and activities in the lesson? | 0–10 | 8.41 (1.28) | 8.22 (1.74) | 8.32 (1.51) | Quality | 0.81 | 0.21 |
| How well does the teacher respond to pupil queries/meet the needs of all of the class if it is required? | 0–10 | 8.47 (1.35) | 8.20 (1.57) | 8.35 (1.45) | Quality | 0.82 | 0.04 |
| Rate the extent to which children in the class actively participate in the lesson activities | 0–10 | 7.46 (1.45) | 7.39 (1.63) | 7.43 (1.53) | Responsiveness | 0.77 | –0.01 |
| Rate the level of sustained interest and attentiveness among children in the class during the lesson | 0–10 | 6.82 (1.85) | 7.22 (1.70) | 7.01 (1.79) | Responsiveness | 0.84 | –0.01 |
| Rate the extent to which the learning objectives have been met | 0–10 | 7.56 (1.51) | 7.75 (1.88) | 7.65 (1.69) | Responsiveness | 0.87 | 0.21 |
Tables 15 and 16 depict the corresponding descriptive statistics for pupil outcome data in PATHS schools. Finally, Tables 17A and 17B and 18A and 18B provide inferential statistics in relation to hypotheses 4a–e.
| Outcome | PATHS, mean (SD) | |
|---|---|---|
| Baseline (T1) | Interim (T2) | |
| Social skills | 106.58 (19.59) | 104.33 (19.75) |
| Internalising symptoms | 2.57 (3.06) | 2.51 (2.97) |
| Externalising problems | 4.09 (4.42) | 3.72 (4.03) |
| Pro-social behaviour | 7.80 (2.43) | 7.69 (2.34) |
| Psychological well-being | 29.55 (4.86) | 29.71 (4.67) |
| Social support and peers | 16.76 (3.40) | 16.86 (3.32) |
| School environment | 17.74 (2.65) | 17.63 (2.69) |
| Outcome | PATHS, mean (SD) | |
|---|---|---|
| Baseline (T1) | Post-intervention (T3) | |
| Social skills | 105.50 (20.13) | 103.08 (19.19) |
| Internalising symptoms | 2.85 (3.30) | 2.30 (2.96) |
| Externalising problems | 4.11 (4.41) | 3.18 (3.78) |
| Pro-social behaviour | 7.69 (2.42) | 8.09 (2.22) |
| Psychological well-being | 29.98 (4.66) | 30.44 (4.13) |
| Social support and peers | 16.93 (3.37) | 17.50 (2.98) |
| School environment | 17.72 (2.66) | 17.84 (2.34) |
| Level | Outcome | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Social skills | Internalising symptoms | Externalising problems | Pro-social behaviour | |||||||||
| β0ij = –0.052 (0.168) | β0ij = –0.196 (0.30) | β0ij = –0.190 (0.206) | β0ij = –0.054 (0.266) | |||||||||
| Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | |
| Class | 0.031 | 0.012 | 0.006 | 0.213 | 0.045 | < 0.001 | 0.089 | 0.022 | < 0.001 | 0.174 | 0.035 | < 0.001 |
| Dosage |
–0.230 (if moderate) –0.253 (if high) |
0.113 0.150 |
0.023 0.048 |
–0.056 (if moderate) 0.082 (if high) |
0.195 0.264 |
00.386 0.379 |
–0.113 (if moderate) –0.110 (if high) |
0.137 0.177 |
0.208 0.269 |
0.106 (if moderate) –0.432 (if high) |
0.175 0.246 |
0.272 0.042 |
| Reach |
–0.042 (if moderate) 0.089 (if high) |
0.107 0.104 |
0.349 0.197 |
–0.053 (if moderate) –0.157 (if high) |
0.207 0.199 |
0.398 0.216 |
0.037 (if moderate) 0.096 (if high) |
0.154 0.140 |
0.406 0.247 |
–0.249 (if moderate) –0.277 (if high) |
0.178 0.169 |
0.084 0.053 |
| Quality and responsiveness |
0.021 (if moderate) –0.132 (if high) |
0.102 0.127 |
0.417 0.151 |
–0.086 (if moderate) –0.139 (if high) |
0.182 0.237 |
0.320 0.279 |
–0.110 (if moderate) –0.256 (if high) |
0.135 0.172 |
0.211 0.071 |
0.129 (if moderate) 0.212 (if high) |
0.167 0.217 |
0.222 0.165 |
| Fidelity |
–0.066 (if moderate) 0.090 (if high) |
0.133 0.161 |
0.313 0.289 |
–0.048 (if moderate) 0.275 (if high) |
0.234 0.307 |
0.417 0.186 |
0.039 (if moderate) 0.234 (if high) |
0.162 0.211 |
0.406 0.136 |
–0.137 (if moderate) –0.184 (if high) |
0.211 0.269 |
0.259 0.243 |
| Pupil | 0.748 | 0.028 | < 0.001 | 0.651 | 0.024 | < 0.001 | 0.423 | 0.016 | < 0.001 | 0.562 | 0.035 | < 0.001 |
| Sex (if female) | 0.294 | 0.046 | < 0.001 | 0.026 | 0.041 | 0.264 | –0.163 | 0.034 | < 0.001 | 0.365 | 0.041 | < 0.001 |
| FSM (if eligible) | –0.101 | 0.057 | 0.038 | 0.101 | 0.049 | 0.020 | 0.121 | 0.040 | 0.001 | –0.173 | 0.045 | < 0.001 |
| Baseline score | 0.413 | 0.026 | < 0.001 | 0.381 | 0.023 | < 0.001 | 0.693 | 0.018 | < 0.001 | 0.389 | 0.023 | < 0.001 |
| Level | Outcome | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Psychological well-being | Social support and peers | School environment | |||||||
| β0ij = –0.053 (0.178) | β0ij = –0.138 (0.186) | β0ij = –0.187 (0.179) | |||||||
| Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | |
| Class | 0.035 | 0.013 | 0.005 | 0.051 | 0.017 | 0.002 | 0.042 | 0.014 | 0.002 |
| Dosage |
–0.047 (if moderate) –0.301 (if high) |
0.113 0.154 |
0.338 0.028 |
–0.006 (if moderate) –0.083 (if high) |
0.128 0.176 |
0.480 0.320 |
–0.066 (if moderate) 0.176 (if high) |
0.114 0.149 |
0.282 0.121 |
| Reach |
–0.074 (if moderate) –0.010 (if high) |
0.108 0.111 |
0.247 0.464 |
0.053 (if moderate) 0.067 (if high) |
0.128 0.125 |
0.341 0.296 |
0.098 (if moderate) 0.089 (if high) |
0.118 0.114 |
0.205 0.219 |
| Quality and responsiveness |
0.245 (if moderate) 0.194 (if high) |
0.109 0.130 |
0.014 0.071 |
–0.095 (if moderate) –0.048 (if high) |
0.115 0.146 |
0.205 0.371 |
0.085 (if moderate) 0.028 (if high) |
0.109 0.143 |
0.219 0.421 |
| Fidelity |
–0.139 (if moderate) –0.168 (if high) |
0.138 0.176 |
0.161 0.173 |
0.102 (if moderate) –0.047 (if high) |
0.144 0.187 |
0.240 0.402 |
–0.035 (if moderate) 0.066 (if high) |
0.131 0.164 |
0.394 0.345 |
| Pupil | 0.905 | 0.036 | < 0.001 | 0.903 | 0.033 | < 0.001 | 0.873 | 0.031 | < 0.001 |
| Sex (if female) | 0.157 | 0.049 | < 0.001 | 0.188 | 0.047 | < 0.001 | 0.302 | 0.048 | < 0.001 |
| FSM (if eligible) | –0.019 | 0.057 | 0.371 | 0.025 | 0.058 | 0.334 | –0.086 | 0.057 | 0.066 |
| Baseline score | 0.219 | 0.027 | < 0.001 | 0.211 | 0.025 | < 0.001 | 0.215 | 0.025 | < 0.001 |
| Level | Outcome | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Social skills | Internalising symptoms | Externalising problems | Pro-social behaviour | |||||||||
| β0ij = –0.050 (0.139) | β0ij = –0.074 (0.222) | β0ij = –0.111 (0.156) | β0ij = –0.152 (0.222) | |||||||||
| Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | |
| Class | 0.038 | 0.014 | 0.004 | 0.151 | 0.037 | < 0.001 | 0.068 | 0.017 | < 0.001 | 0.162 | 0.036 | < 0.001 |
| Dosage |
–0.039 (if moderate) –0.042 (if high) |
0.096 0.132 |
0.342 0.375 |
0.261 (if moderate) 0.527 (if high) |
0.166 0.258 |
0.061 0.023 |
0.074 (if moderate) 0.083 (if high) |
0.124 0.166 |
0.275 0.310 |
0.071 (if moderate) –0.025 (if high) |
0.172 0.234 |
0.342 0.456 |
| Reach |
–0.027 (if moderate) 0.011 (if high) |
0.091 0.092 |
0.383 0.452 |
–0.187 (if moderate) –0.343 (if high) |
0.158 0.167 |
0.122 0.023 |
–0.151 (if moderate) –0.288 (if high) |
0.102 0.102 |
0.072 0.003 |
–0.063 (if moderate) –0.038 (if high) |
0.161 0.156 |
0.349 0.406 |
| Quality and responsiveness |
–0.049 (if moderate) 0.220 (if high) |
0.116 0.190 |
0.338 0.126 |
–0.021 (if moderate) 0.031 (if high) |
0.189 0.293 |
0.456 0.456 |
0.059 (if moderate) 0.164 (if high) |
0.121 0.191 |
0.313 0.197 |
0.056 (if moderate) 0.044 (if high) |
0.177 0.275 |
0.375 0.437 |
| Fidelity |
–0.031 (if moderate) 0.000 (if high) |
0.110 0.000 |
0.390 0.500 |
0.044 (if moderate) 0.000 (if high) |
0.175 0.000 |
0.401 0.500 |
0.046 (if moderate) 0.000 (if high) |
0.117 0.000 |
0.349 0.500 |
–0.081 (if moderate) 0.000 (if high) |
0.188 0.000 |
0.334 0.500 |
| Pupil | 0.754 | 0.032 | < 0.001 | 0.719 | 0.028 | < 0.001 | 0.460 | 0.019 | < 0.001 | 0.594 | 0.023 | < 0.001 |
| Sex (if female) | 0.338 | 0.052 | < 0.001 | –0.018 | 0.045 | 0.345 | –0.257 | 0.038 | < 0.001 | 0.435 | 0.046 | < 0.001 |
| FSM (if eligible) | –0.132 | 0.055 | 0.008 | 0.171 | 0.056 | 0.001 | 0.141 | 0.045 | < 0.001 | –0.107 | 0.051 | 0.018 |
| Baseline score | 0.403 | 0.026 | < 0.001 | 0.397 | 0.024 | < 0.001 | 0.626 | 0.022 | < 0.001 | 0.368 | 0.024 | < 0.001 |
| Level | Outcome | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Psychological well-being | Social support and peers | School environment | |||||||
| β0ij = –0.056 (0.137) | β0ij = –0.207 (0.126) | β0ij = –0.149 (0.157) | |||||||
| Coefficient β | SE | p-value | Coefficient β | SE | p-value | Coefficient β | SE | p-value | |
| Class | 0.029 | 0.013 | 0.015 | 0.009 | 0.009 | 0.161 | 0.055 | 0.017 | 0.001 |
| Dosage |
0.004 (if moderate) –0.168 (if high) |
0.103 0.142 |
0.484 0.122 |
0.168 (if moderate) 0.103 (if high) |
0.084 0.119 |
0.025 0.194 |
–0.023 (if moderate) –0.146 (if high) |
0.113 0.146 |
0.421 0.161 |
| Reach |
0.042 (if moderate) 0.019 (if high) |
0.097 0.098 |
0.334 0.425 |
–0.011 (if moderate) –0.111 (if high) |
0.083 0.080 |
0.449 0.085 |
0.007 (if moderate) 0.095 (if high) |
0.105 0.102 |
0.472 0.178 |
| Quality and responsiveness |
–0.023 (if moderate) 0.123 (if high) |
0.117 0.176 |
0.421 0.243 |
0.113 (if moderate) 0.061 (if high) |
0.096 0.148 |
0.124 0.342 |
–0.027 (if moderate) 0.083 (if high) |
0.123 0.198 |
0.413 0.338 |
| Fidelity |
0.022 (if moderate) 0.000 (if high) |
0.108 0.000 |
0.421 0.500 |
–0.047 (if moderate) 0.000 (if high) |
0.100 0.000 |
0.320 0.500 |
0.026 (if moderate) 0.000 (if high) |
0.127 0.000 |
0.421 0.500 |
| Pupil | 0.913 | 0.037 | < 0.001 | 0.917 | 0.036 | < 0.001 | 0.804 | 0.031 | < .001 |
| Sex (if female) | 0.047 | 0.050 | 0.174 | 0.181 | 0.056 | < 0.001 | 0.327 | 0.050 | < .001 |
| FSM (if eligible) | 0.035 | 0.064 | 0.291 | –0.031 | 0.060 | 0.302 | –0.151 | 0.060 | 0.006 |
| Baseline score | 0.258 | 0.027 | < 0.001 | 0.257 | 0.026 | < 0.001 | 0.316 | 0.028 | < .001 |
Using data from the first year of the trial (classes in Years 3 and 4 observed, T2 as the response variable, controlling for T1 at the child level) our analyses demonstrated that, compared with low levels, moderate (and, marginally, high) levels of implementation quality and participant responsiveness (hypothesis 4c/d) were associated with significant improvements in children’s psychological well-being (moderate: d = 0.25, 95% CI 0.03 to 0.46; p < 0.05; high: d = 0.19, 95% CI –0.06 to.45; p = 0.07). A marginal, non-significant trend was also identified, whereby compared with low levels, high levels of implementation quality and participant responsiveness were associated with reductions in externalising problems (d = –0.26, 95% CI –0.59 to 0.08; p = 0.07). Contrary to expectations, compared with low levels, moderate and high levels of dosage (hypothesis 4b) were associated with significant reductions in children’s social skills (moderate: d = –0.23, 95% CI –0.45 to –0.01; p < 0.05; high: d = –0.25, 95% CI –0.55 to 0.04; p < 0.05). Similarly, compared with low levels, high levels of dosage were also associated with significant reductions in pro-social behaviour (d = –0.43, 95% CI –0.91 to 0.05; p < 0.05) and psychological well-being (d = –0.3, 95% CI –0.60 to 0.00; p < 0.05). Finally, and again contrary to expectations, a marginal non-significant trend was identified whereby, compared with low levels, high levels of reach (hypothesis 4e) were associated with reductions in pro-social behaviour (d = –0.28, 95% CI –0.61 to 0.05; p = 0.05).
In the second year of the trial (classes in Years 5 and 6 observed, T3 as the response variable, controlling for T1 at the child level), our analyses demonstrated that, compared with low levels, moderate levels of dosage (hypothesis 4b) were associated with significant improvements in children’s perceptions of social support and peers (d = 0.17, 95% CI 0.00 to 0.33; p < 0.05). However, high levels of dosage were associated with significant increases in internalising symptoms (d = 0.53, 95% CI 0.02 to 1.03; p < 0.05). Compared with low levels, high levels of reach (hypothesis 4e) were associated with significant reductions in both internalising symptoms (d = –0.34, 95% CI –0.67 to –0.02; p < 0.05) and externalising problems (d = –0.29, 95% CI –0.49 to –0.09; p < 0.05). A marginal effect was found in relation to moderate reach for externalising problems (d = –0.15, 95% CI –0.35 to 0.05; p = 0.07). However, compared with low levels, moderate reach also (marginally) predicted reductions in children’s perceptions of social support and peers (d = –0.11, 95% CI –0.31 to 0.09; p = .09). Throughout both years of the trial, our analyses revealed no associations between levels of implementation fidelity (hypothesis 4a) and intervention outcomes.
Qualitative analysis
The following analysis reflects our hybrid inductive–deductive processes of thematic identification. 69 The deductive aspect draws primarily on key sources of knowledge about implementation processes, including (but not limited to):
-
Durlak and DuPre’s59 review and model of implementation dimensions and factors affecting implementation
-
the implementation quality model of Domitrovich et al. 58
-
Nilsen’s85 taxonomy of theories, models and frameworks in implementation science
-
the integrated model of programme implementation of Berkel et al. 54
-
the taxonomy of Moore et al. 86 for understanding adaptations of evidence-based interventions in natural contexts.
Each of the above informed the development of our thematic framework and/or interpretation of qualitative data. In addition, it is also important to acknowledge the influence of existing studies that have focused on the implementation of PATHS using qualitative approaches,87,88 particularly those in the UK educational context,88–90 in helping us to contextualise our findings within the broader literature. Table 19 provides an overview of the final thematic framework for the qualitative strand of our IPE. This framework is used to give structure and order to the reporting that follows, with one important exception. It begins with the subordinate theme ‘Foundations for PATHS’, as this seemed to be the most logical place to start in terms of offering a coherent narrative. Example major codes are provided when these are used to structure the text because of the substantive nature of the content (e.g. curriculum structure, and lesson structure as major codes in the subordinate theme of fidelity).
| Theme | Example major codes (where applicable) | |
|---|---|---|
| Superordinate | Subordinate | |
| Implementation | Dosage | |
| Fidelity | Curriculum structure | |
| Lesson structure | ||
| Adaptations | ||
| Quality | ||
| Pupil responsiveness | ||
| Reach | ||
| Factors affecting implementation | Foundations for PATHS | A need for change |
| A more structured approach | ||
| Perception of pupil need | ||
| Shared decision-making (or lack thereof) | ||
| Programme characteristics | Resources and materials | |
| PATHS content, concepts and strategies | ||
| Teacher attitudes | An explicit approach to SEL | |
| Perceptions of need and benefit | ||
| Fitting PATHS | Time | |
| Competing priorities | ||
| Curriculum demands | ||
| Leadership support | ||
| External support | Training | |
| Coaching model | ||
| Perceptions of impact | Emotional literacy | |
| Generalisation of learning | ||
| Behaviour problems | ||
| Universal delivery, differential gains | ||
| Wider impact | Language and literacy | |
| Well-being, confidence and self-esteem | ||
| Beyond the pupil | Improved learning environment | |
| Perceived impact on teachers | ||
| Sustainability | Embedding and extending PATHS | |
| Adapting to facilitate sustainability | ||
Finally, although presented as ‘distinct’ in the interests of clarity and readability, from the outset the overlap and associations between the various superordinate and subordinate themes and major codes that follow are acknowledged. Three examples are provided to illustrate this proposed interconnectivity. First, the extent to which class teachers were involved in a school’s decision to take part in the project [see Shared decision-making (or lack thereof)] is theorised to have influenced their initial attitudes to PATHS (see Teacher attitudes) and associated support systems (see Training). This in turn may have had an impact on their assessment of the extent to which PATHS could be successfully integrated into their practice (see Fitting Promoting Alternative THinking Strategies in), ultimately influencing how they implemented the programme (see Implementation variability), with concurrent implications for their views on whether or not it made a difference (see Perceptions of impact) and the likelihood that they would continue with implementation once the main trial was concluded (see Sustainability).
Second, the legacy of other SEL initiatives in the school and, in particular, the view that the SEAL programme had run its course (see The need for change) created a strong driver for change, especially given the context of continuing high levels of pupil need (see Perception of pupil need). The fact that PATHS was seen as aligning with the principles of SEAL but offering a (potentially) more attractive delivery model (see A more structured approach) made it an attractive proposition. However, in practice, this was difficult to implement consistently as per the developer’s expectations (particularly in relation to dosage and fidelity) in the context of continuing (and even increasing) internal and external pressures (see Curriculum demands) and a history of SEL initiatives being sidelined (see Foundations for Promoting Alternative THinking Strategies). Somewhat ironically, the structured nature of PATHS, which was initially an attractive feature of the programme (see A more structured approach), seemed to amplify difficulties in its integration (see Fitting Promoting Alternative THinking Strategies in) (indeed, this apparent paradox has been reported elsewhere87), leading to adaptations during the main trial (see Adaptations), alongside planned future adaptations as teachers – who continued to value what PATHS offered – sought to keep it going, even if in diluted form (see Adaptation to facilitate sustainability).
Third, it is hypothesised that high levels of pupil need (see Perception of pupil need and Perception of need and benefit) in a given class may be an initial driver of implementation, insofar as teachers in schools in which there is a higher level of perceived need for PATHS would be likely to deliver lessons more frequently (see Dosage). It is theorised that the extent to which this is maintained over time is at least in part connected to teachers’ views on whether or not the programme is having a positive impact in the classroom (see Perceptions of impact). Where teachers note such impact, they may downshift their implementation of PATHS as their perception of need reduces, and they revert back to other priorities (see Competing priorities).
The data excerpts from teachers, school PATHS co-ordinators, parents and pupils are attributed to year group (where appropriate) and school number (all participating schools in the trial were assigned a number from 1 to 45 in order to protect their anonymity) in order to evidence differentiation in data usage. Contextual information (e.g. size, percentage of FSM eligibility) is not provided about each individual school in order to protect the anonymity of respondents. This is particularly pertinent given that 12 were single-form entry schools (and as such, there was only one teacher per year group, meaning that school-level contextual information could potentially be used to identify them). Instead, the reader is referred to Table 3, which summarises the characteristics of PATHS schools.
Foundations for Promoting Alternative THinking Strategies
In broad alignment with existing studies of PATHS implementation in the UK,90 there was a sense among respondents that the principles of the programme were embedded in the ‘whole school ethos’, and ‘interlinked’ to the religious ethos of the school (in Catholic and Church of England primary schools). However, in the taught curriculum, SEL ideas and values were felt to be embodied primarily in PSHE and religious education (RE) lessons: ‘generally it was just your PSHE slot once a week, because we are a religious school, we do RE quite frequently and a lot of PSHE does kind of overlap with that, so that is quite good for getting that side of things as well’ (Year 4 teacher, School 1). There was also a sense that these types of lessons cover ‘the concepts and the ideas [that] would teach social skills and how people teach each other how they treat others’ (Year 5 teacher, School 1).
In terms of named or ‘proprietary’ initiatives, the overwhelming majority of schools had been implementing universal interventions such as SEAL and Circle Time. Other named initiatives that were in place prior to the introduction of PATHS included anti-bullying week/bullying awareness, Rights Respecting Schools and Philosophy for Children. In addition, bespoke and/or targeted work relating to SEL was commonplace, but this was specifically for the ‘pockets of children who have been difficult’ (Year 3 teacher, School 17) and children with SEN. For example, schools reporting using proprietary targeted initiatives such as Place2Talk and Place2Be, employed school counsellors, and used Social Stories with pupils identified as having autism spectrum conditions (one school). Alongside this, various ‘behaviour groups’ were used to support children with identified behavioural, emotional and social difficulties, such as ‘Coral Cove’ (one-to-one or small group work) and ‘BLIS’ (Behaviour for Learning and Inclusion Service). Thus, in terms of existing practice, our qualitative data strongly supported the findings of the baseline usual provision survey, indicating a high level of SEL and SEL-related activity prior to the introduction of PATHS. This high level of existing activity reinforces the importance of ‘programme differentiation’59 in understanding the implementation and impact of PATHS. In terms of implementation, schools were arguably well primed, having already grappled with numerous similar interventions over a number of years. However, in terms of impact, the relatively low level of programme differentiation between PATHS and existing practice (in both arms of the trial) arguably makes demonstrating the added value of the former more challenging.
In spite of this, there was a concern that SEL was not a priority and teachers commented that this was an aspect of their provision, ‘that would have ended up getting left’ (Year 6 teacher, School 17) and was done ‘if you’ve got the time’ (Year 3 teacher, School 9), because ‘it wasn’t really monitored so nobody knew’ (Year 3 teacher, School 5). As a result, ‘[SEL] tended to sort of be pushed to the back really. It wasn’t very prominent’ (Year 4 teacher, School 5). Thus, rather than being delivered consistently over time, existing SEL interventions were instead ‘brought out’ when teachers felt that they ‘needed’ it. There was also a clear sense of drift over time when new interventions were brought in, and, indeed, the issue of SEL programmes ‘coming and going’ was highlighted in Honess and Hunter’s88 recent study of PATHS implementation. These issues are summarised well in the following extract:
Circle Time games . . . people don’t stick to it as rigidly as they used to. I know everyone used it, but now people just dip into it here and there to fit in . . . Sometimes we would have Circle Time outside of a session just to deal with any issues that were really prominent at the time.
Year 3 teacher, School 18
The frequent ‘turnover’ of SEL and SEL-related initiatives in participating schools may provide a partial explanation for the reduction in dosage observed from the first to the second year of the trial (see Table 12). That is, PATHS may have fallen victim to the initiative overload observed across the school system,91 particularly if initial expectations about impact (e.g. amount of change and when this would occur) were not met. Indeed, there is some evidence to support this assertion, as seen in the following sub-sections, in that schools clearly viewed PATHS as being superior to existing provision, heightening expectations.
The need for change
There was a clear perception of need echoed across schools and a multitude of reasons that they had decided to join the trial (hoping, of course, to be allocated to the PATHS group). Schools wanted a new and ‘fresh’ approach, or something to complement and develop existing provision. Some felt the need for a structured, more prescribed programme with a consistent and shared language in order to meet the specific needs of their current cohort, and PATHS seemed to ‘fit the bill’ (Year 3 teacher, School 26).
There was a growing feeling that existing initiatives, in particular SEAL and Circle Time, were now ‘stale’ and ‘outdated’. It had ‘got to the point where SEAL was dying a death’ and ‘people were just grabbing books on Circle Time’ (PATHS co-ordinator, School 10). Schools were thus looking for something to ‘replace’ these. It was ‘time for a fresh approach’ and PATHS was seen as ‘something that was a little bit different, a little bit newer than what’s gone previously’ (Year 5 teacher, School 17):
They did do SEAL before this, they’re very big on . . . the emotions of the children and stuff like that, but I think it was just a lot of work for them to plan SEAL, so I think it was very wishy washy, they didn’t know really where to go and what was happening.
Year 3 teacher, School 2
It was really hit and miss. Some teachers, some of the more experienced teachers were still looking at the SEAL and using that documentation and we found that the NQTs [newly qualified teachers] were doing nothing other than a bit of Circle Time and if something happened on the playground they would then do a Circle Time activity to try and look at how that could have been prevented. But there was no structure in place. We didn’t have curriculum in any way, shape or form. So it was just sort of going on what you’d done in previous years and stuff so it was really, really ad hoc.
Year 6 teacher, School 18
The strong international evidence base for PATHS attracted a number of schools to being part of the trial: ‘the research that’s gone into it [PATHS] and the research wasn’t there with SEAL, that’s really what made us go with it’ (PATHS co-ordinator, School 10). It was therefore anticipated that PATHS would have more of an impact than existing SEL provision:
. . . SEAL is slightly different, in a way I prefer PATHS because it’s looking at psychology directly, it’s engaging with things that are happening in children and it’s the emotional literacy thing where you can make distinctions between how you’re feeling and word it and name it I think that’s really important that that wasn’t evidently in SEAL.
Year 4 teacher, School 5
Promoting Alternative THinking Strategies was seen to fill an important void and was ‘offering something a little bit different’ (PATHS co-ordinator, School 24):
So there is this sort of missing link and it’s nice that we can sort of fill it with PATHS, educate them through the PATHS programme rather than the generic things that have gone before like SEAL and things related like that.
Year 6 teacher, School 9
Well we’ve been using SEAL for a number of years but . . . the behaviour in school and some of the relationships and the personal and social aspects of classroom management seems to be slipping a little bit. So last year we spent a lot of time doing a lot of time on the behaviour management and policies and strategies we were using in school but it just seemed that SEAL was a little bit dated and didn’t fit into what we were doing.
Year 4 teacher, School 10
A more structured approach
Thus, schools felt a need to ‘look perhaps at other avenues see if other more structured things might help’ (Year 5 teacher, School 6). The PATHS programme, with its highly prescribed, ‘manualised’ delivery model (e.g. a spiral curriculum, two lessons per week and provision of necessary resources, including curriculum packs, implementation manuals, worksheets, pictures and a set of posters) was initially viewed as a ‘really strong, really strong set of lessons . . . with lots of nice resources, really easy to be adapted into the classroom’ (Year 3 teacher, School 25). PATHS is designed so that teachers can pick up the curriculum pack and deliver the lessons ‘off the shelf’ with high fidelity: ‘[It is a] package . . . you just follow it and it does make life a little bit easier’ (Year 4 teacher, School 1). This degree of prescription, cohesion and structure was exactly what some schools were looking for in relation to their SEL provision, and, indeed, the attraction of these features has been noted in other qualitative studies of PATHS implementation. 88
It was also thought that PATHS could enable a ‘more consistent . . . theme running through the school’ (Year 5 teacher, School 38) and would provide the teachers and pupils with a shared ‘language to explain your emotions [which was] was lacking’ (Year 4 teacher, School 2):
We wanted uniform[ity] across the school really but especially across the key stages so that everyone was using the same language and using the same strategies.
PATHS co-ordinator, School 5
So that everyone was using the same language and everyone was teaching . . . the PSHE curriculum in the same way and also to deal with . . . we have a lot of . . . we have children that have a lot of emotions that they don’t know how to deal with and it was the cause of a lot of incidents on the playground and in classes. So we wanted something where we could teach the children about how to deal with behaviour and control anger.
Year 4 teacher, School 5
Perception of pupil need
Promoting Alternative THinking Strategies was also viewed as an intervention that ‘children would react well to’ (Year 4 teacher, School 2) and would be ‘more fun’ in view of the range of activities, resources and approaches to teaching SEL inherent within the programme:
The generations are changing every year and I think PATHS is something that children do need. We’re living in a world where everything moves so quickly and maybe stories, things like that, don’t always appeal to children. So something like PATHS, it has lots of different activities. So it has got stories and it’s got drama and it’s got writing. So it’s got lots of different things that children can respond to rather than just sit and listen to a story and learn something from that.
Year 5 teacher, School 17
The content of PATHS (e.g. identifying, labelling, expressing and managing feelings, understanding the difference between feelings and behaviours, delaying gratification, controlling impulses, reducing stress, understanding others’ perspectives, social problem-solving, and decision-making) connected directly to perceived pupil need in participating schools. One school stated that these were issues that the Office for Standards in Education, Children’s Services and Skills (Ofsted) specifically wanted them to address. Teachers commented that social and emotional skills needed to be addressed explicitly in the classroom because ‘relationships between the children can be an issue . . . having ways to deal with their relationships and things and trying to stop problems before they arise I think we see as a potential benefit’ (Year 3 teacher, School 6) and that children need to be ‘think[ing] about their actions’ (Year 6 teacher, School 5). This perceived need was often linked directly to schools’ local contexts (e.g. schools in the trial sample were significantly larger, with higher proportions of pupils of pupils eligible for FSMs than the national average; see Table 3). The suitability of the PATHS programme for children in areas of high social deprivation noted in earlier studies88 was borne out:
There is a need particularly because of our demographic. That maybe children aren’t as sort of educated in understanding the way that they feel, particularly boys I suppose. Maybe the common thing is for them to suppress their feelings and not necessarily look into them and understand. So there is this sort of missing link and it’s nice that we can sort of fill it with PATHS, educate them through the PATHS programme rather than the generic things that have gone before like SEAL and things related like that.
Year 6 teacher, School 9
Consistent with hypothesis 3 of the current study, the social and emotional skills that are fundamental to PATHS were also viewed as being important in the context of transition to secondary school, enabling children to be ‘secondary school ready’, as some schools ‘were getting stories back that they weren’t really equipped with the tools to deal with problems’ (Year 6 teacher, School 17). Finally, there were schools in the trial that felt as though they did not really have a ‘problem’ to address, but wanted to ‘promote the advancement of better emotional and social awareness and social skills’ (PATHS co-ordinator, School 18).
Shared decision-making (or lack thereof)
Each school’s decision to join the trial was ultimately made by the headteacher, although this was often done in consultation with the senior leadership team and/or other pertinent stakeholders (e.g. a SENCo):
Myself and another person came on the initial day and we brought the information back to the school and discussed it with the SENCo . . . So it was an agreement within the school from the information we brought back from the training/information day.
PATHS co-ordinator, School 6
In this quotation, the training/information day refers to a recruitment event in March 2012 for prospective trial schools held by the research team, involving a presentation from Barnardo’s, testimonials from teachers and coaches with experience of PATHS implementation in Birmingham, and the details of the trial (e.g. data collection requirements and timeline).
Teaching staff were notably less involved in the decision-making process, and the majority of teachers felt quite ‘removed’, stating that it was all done at ‘management level’ and that they knew little about it until they were asked to attend the initial training:
it was a bit of a surprise when all of a sudden we were told we had to start doing it and we didn’t know what it was or anything . . . it was thrown on us very, very quickly without any consultation and without us even knowing what it was.
Year 3 teacher, School 5
In contrast, a minority of schools did engage in consultation with teaching staff, stating, ‘unless people were prepared to buy in I think you just get compliance which wouldn’t actually help in terms of testing PATHS as a project’ (PATHS co-ordinator, School 17):
The Head did bring all the junior teachers in and we talked about it . . . I don’t know whether he had already made up his mind or whether he was testing, but I think if one of us had been like ‘Oh no, I don’t want to do this’, maybe we wouldn’t, I don’t know, I think it was the Head’s decision overall but I think we all had a bit of an input.
Year 4 teacher, School 1
Educational psychologists in the Greater Manchester city-region were made aware of the trial and, in some cases, their advice contributed to schools’ decision-making:
. . . our educational psychologist said it was a good project to be involved in . . . the teacher that was involved did speak to the headteacher and they made the agreement . . . there wasn’t a big discussion of the whole staff, it was a quick conversation, ‘will you do it?’.
Year 5 teacher, School 6
Implementation
Dosage
The recommended delivery model of PATHS is two discrete lessons or a ‘double’ lesson each week. Almost all teachers initially aspired to achieve this, and lessons were timetabled as such. Most opted for two discrete lessons a week, with a minority running the ‘double’ lesson variation. However, despite the aspiration to follow the recommended implementation model, staff reported that, as in other studies of PATHS implementation,87 ‘it does not always work out’ (Year 3 teacher, School 30). As the school year progressed, staff struggled to maintain the frequency with which PATHS lessons were intended to be delivered, particularly in the light of competing priorities (discussed in more detail in Fitting Promoting Alternative THinking Strategies in):
At the beginning somehow I found the time to do two lessons a week, I seemed to be doing quite well with it, but then over this last half term there has been, even this last term, there has been quite a lot going on.
Year 4 teacher, School 1
In the terms of Moore et al. ,86 this represents the notion of ‘logistical fit’ as an explanation for adaptations to dosage. It was noted that there is some flexibility within the timetable for the PATHS curriculum, such that sometimes teachers could either ‘condense’ or ‘double-up’ and ‘put a couple of the sessions together so that they [pupils] don’t miss out’ (Year 6 teacher, School 2). The length of the school year relative to the number of PATHS lessons created some additional flexibility (e.g. if teachers did consistently implement two lessons per week, the curriculum for a given year group would typically be covered by around March or April), meaning that there was ‘a bit of leeway throughout the year with the number of lessons . . . it is not like one for every week of the year so we catch up’ (Year 3 teacher, School 6). Despite this, competing priorities and the perception of PATHS as not being part of the central academic curriculum meant that it could easily ‘fall by the wayside’ (Year 3 teacher, School 30), and that ‘certain people would maybe drop PATHS from the timetable in a very hectic week’ (PATHS co-ordinator, School 5).
In terms of duration, teachers reported that PATHS lessons lasted on average 30 minutes, with a range of 20–45 minutes, but this was largely dependent on the specific lesson content and the level of need and engagement among pupils. Some teachers commented that they found it ‘difficult’ to cover the lesson content in the allocated time, ‘especially when you’re going into discussions and if the children want to keep going you don’t want to stop them’ (Year 5 teacher, School 10). In other cases, it was felt that children needed ‘a bit longer on it and going through it and actually understanding it’ (Year 3 teacher, School 13). This was particularly the case for children whose learning needs meant that they struggled to access the curriculum:
Sometimes it takes a bit longer just because of some of the children’s lack of ability to follow instructions, it depends how hyper they are, it depends on the way I’m doing the lesson. If they’re doing a bit of role play we can get through it quicker, but if they’re sitting and listening a lot first and then maybe do some recording, then that takes longer.
Year 3 teacher, School 1
This too was noted in a lesson observation, where extra discussion was needed given the ability level of the class, as pupils took longer for the concepts to ‘sink in’. Consequently, the lesson ran over, taking almost 50 minutes. However, teachers also extended lessons in cases in which ‘the children are engaging with it, some of them they do enjoy a lot more so you can extend it a little bit’ (Year 3 teacher, School 6). This meant that they felt able to ‘spend a bit more than what I should because of the debates and the discussions . . . as long as they are engaged’ (Year 5 teacher, School 13).
Fidelity
Curriculum structure
As noted in the TIDieR programme description, PATHS uses a spiral curriculum model, in which topics and concepts are revisited, units and lessons are developmentally sequenced, new learning is linked to previous learning, and the competence of learners increases with each successive visit to a topic or concept. Teachers recognised this, stating that the lesson order seem to ‘build on each other quite nicely’ (Year 3 teacher, School 6) and ‘you can see the progression of skills’ (Year 5 teacher, School 13); they also appreciated that ‘it’s put together by somebody with experience’ (Year 3 teacher, School 19). They did not feel a need to deviate from the proposed sequence of the lessons, and in any case were not ‘brave enough’ to change this aspect of the programme: ‘PATHS is more important in some ways because it’s a structured scheme and I want to keep to the scheme’ (Year 6 teacher, School 7). Furthermore, as reported in other qualitative evaluations,90 the PATHS curriculum structure was viewed as enabling teachers to minimise preparation time and planning, with a subsequent tendency to focus only on the coming lesson:
I tend to just look at the lesson that I’m on and then . . . you know, I haven’t looked very much further on, but I mean yeah, I think I’d still just follow it as it is because I’m sure there’s kind of all theory behind the way it’s been put together, so yeah, I’d just keep going with it.
Year 3 teacher, School 38
There was a feeling at the stage of delivery that the lessons and concepts covered felt a little repetitive, but teachers acknowledged that this too may be a good thing in order to help the children ‘consolidate’ the information.
There was relatively little reporting of omission (or ‘skipping’) of lessons in the early stages of implementation, and where there was, this occurred mainly because of miscommunication with a supply teacher in cases in which the class teacher was absent (e.g. through ill health). In a minority of cases, specific lessons were moved or omitted so that they could be tied into another aspect of the curriculum, or if they were felt to be inappropriate to the local context and needs of the class. For example, one teacher commented that a lesson was ‘maybe a bit too Americanised’ and ‘it just wouldn’t have applied to them’ (Year 6 teacher, School 19). At another school, there was a large Muslim pupil population and one PATHS lesson covered being offered alcohol, so ‘we just didn’t think it was it was appropriate for our intake of children, so I think it would open a bit of a can of worms’ (Year 6 teacher, School 17). Finally, one teacher reported missing out the ‘Golden Rule’ (e.g. ‘treat others as you would expect to be treated’) lesson as it was felt that this particular imperative was already well established in the classroom.
One aspect of the curriculum structure in which there was evidence of deviation was the provision of ‘Jump Start’ lessons (see Table 1), which are designed to precede the main curriculum and introduce key PATHS concepts to children who have not previously been exposed to the programme (e.g. in preceding school years, as was the case in the current trial). Teachers reported that they did not deliver these lessons in their entirety and tended to ‘condense’ them, as they felt that they were quite ‘similar’ and/or the concepts were already familiar to their children (e.g. because they had been covered in SEAL):
I did think they were they were very useful [Jump Start lessons], I think just to make sure they knew the basic concepts, but then we did miss a couple of the first lessons of the unit . . . because they’d done it so recently, there were a lot of similarities, so we just made sure they didn’t repeat those because the children then would have been would have turned off thought we’ve just done this why are we doing it again?
Year 5 teacher, School 18
In a handful of cases, teachers repeated certain lessons when they felt that this was necessary, either to consolidate learning [e.g. ‘I repeated one about the Golden Rule just to really sort of force the point home’ (Year 4 teacher, School 18) or in response to a particular need that arose (e.g. ‘I’ve done the stop and calm down lessons, I’ve done that one, and the strategies for calming down and dealing with that, I did that a few times actually because there were issues’ (Year 4 teacher, School 10)]. However, consistent with the generalisation element of the programme, teachers were more likely to simply ‘recap’ certain points or ‘refer back’ to lessons (including making reference to particular skills when needed, such as the steps to calming down) as a means to help children to consolidate and apply the skills they have developed in PATHS lessons in other contexts:
Sometimes it just comes from opportunities. So if there’s a bad lunchtime, a group of children have had a bad experience, we’ll stop what we’re doing and go back . . . It might be like we just revisit possibly the breathing technique . . . So lessons aren’t disrupted because we’ll find a lot of the issues are over football or something that could have been dealt with. So they’ve got the, I’ve got them photocopied, so they will go to a quiet area, write down what they did and then reflect what could they have done, what their plans could be. So I keep that as a record then so we go back to that if that happens again. So they reflect.
Year 6 teacher, School 13
Lesson structure
The core elements, learning objectives and sequence of activities in PATHS lessons tended to be adhered to by the majority. Indeed the ‘prescriptiveness’ of the lesson guide and clear layout of the curriculum pack meant that ‘we have the clear objectives’ (Year 3 teacher, School 18) and this particularly aided newly qualified teachers (NQTs):
. . . having them [lesson plans] there that is incredibly useful especially as I’ve never really taught properly before, because I’m an NQT. So having that resource there as a base is good.
Year 3 teacher, School 43
It was recognised that there is some flexibility within a lesson and that teachers were able to ‘slant them a bit sometimes’ (Year 5 teacher, School 5) and deliver them in their ‘own style’. Thus, PATHS lesson plans were used more as an overarching structure or framework, meaning that teachers could ‘deviate away from the script’ but not the ‘actual teaching side of it’ (Year 3 teacher, School 18). However, conversely, the perceived prescriptive nature of the lessons was also deemed restrictive by some: ‘people felt a little bit constricted by the pack and having to follow the script’ (PATHS co-ordinator, School 5). Ultimately, achieving the appropriate balance was seen as important:
Sticking to the plan is quite important, as you say it’s that fine balance between the two because if you stick to the plan too rigidly you may potentially lose them. And if you stray then you’ve lost what PATHS is about.
Year 5 teacher, School 38
However, our lesson observation notes highlighted that there was a much greater deviation from the PATHS lesson plans than was perhaps revealed through the above excerpt, owing mainly to timing issues (e.g. being able to fit lesson content within a prespecified space in the class timetable). In line with this, some teachers commented that they did need to ‘chop down’ lessons, as some were ‘really, really long’ and some found themselves ‘tweaking and cutting’ (Year 4 teacher, School 7) and having to ‘condense the main themes’ (Year 6 teacher, School 13):
I mean the lessons, because they are so wordy, it does lend itself to be, because it is so detailed you can pick and choose which bits to use.
Year 4 teacher, School 10
As concepts are revisited throughout the curriculum, an element of repetitiveness was highlighted and, again, in these cases, this meant that lesson adherence could be affected:
They do adapt. I know the Year 4 teacher said she feels there are some cohorts of children you would have to keep hammering the same message, but with her cohort at the moment she feels some of it is very repetitive that she just skips over. She said ‘I don’t miss it out I just kind of like briefly “do you remember blah blah” and then move on’. Because she doesn’t feel that they need it quite so repetitive.
PATHS co-ordinator, School 10
Finally, the occasional changes made to the curriculum structure in response to local context and needs were mirrored at the lesson structure level, such that teachers, ‘might digress a little bit if it’s pertinent to other things that we’re doing or something’s happened’ (Year 4 teacher, School 5) or they could ‘take longer with you know what your class needs and take shorter with [sic.] but not necessarily the order’ (Year 4 teacher, School 25). Teachers stressed that they did try to ‘keep the underlying principles of each lesson, but be adaptable maybe to the stories that they use and if you know a better story that you could use . . . or an interactive story or one that the children are aware of that would be characters know maybe from a reading scheme that they use . . . then be open to that, but keep the underlying principles there’ (Year 3 teacher, School 1).
Adaptations
In line with the preceding discussion, most explicit adaptations of the PATHS curriculum were proactive and tended to be superficial changes to make the resources more visual and the content ‘suitable’ and ‘relevant’ for pupils, as teachers recognised the need to keep to the ‘objectives’ and ‘ethos’ of the lessons/programme. The most common adaptation was to enhance delivery through the production of additional materials such as Microsoft PowerPoint® (Microsoft Corporation, Redmond, WA, USA) slides, which were used as both a ‘visual aid’ and as ‘prompts’ for the teacher so that they could avoid reading directly from the lesson scripts. This was supported by additional photocopying of worksheets and images from the PATHS curriculum packs. Evidence of such minor adaptations were also found in the field notes from the lesson observations conducted by the research team.
Despite the fact that PATHS was ‘Anglicised’ by Barnardo’s for use in the UK, teachers still felt that the materials and content were very ‘Americanised’ and felt the need to make further minor adaptations to some of the terminology/words (e.g. ‘principal’ to ‘headteacher’, ‘soccer’ to ‘football’). Some also reported finding the approach to delivering the lessons, ‘a bit cheesy . . . so I will change it into my own words that I know would fit more with my children’ (Year 3 teacher, School 30) and ‘some of the things in the booklet are very American and sometimes you think “oh” and you might cut a few bits out and do it in a different way’ (Year 4 teacher, School 13). Thus, some adaptations were seen as a process of cultural translation. However, others were made to suit class needs and a given teacher’s preferred approach to teaching and learning, as ‘you’ve got to get your own, your own teaching style round it really and adapt it to you to suit your class’ (Year 5 teacher, School 1). Teachers were concerned about all pupils ‘accessing’ PATHS and about making sure that they were ‘engaged’, particularly those with additional needs (e.g. poor literacy levels and/or emotional and behavioural problems). For these reasons, teachers would adapt lesson content by taking ‘some words out because they [pupils] wouldn’t understand them or we would really have to go through things with them’ (Year 4 teacher, School 1). Similarly, if there were particular known issues and content that was especially relevant to the class or the pupils within the class, this could receive greater attention and focus, and/or be extended:
I can’t remember which lesson number it was, but it was the one about the chap – was it Baxter, with the balloon? Because there were a few pupils in my class, who are likely to do that sort of thing just to explode and we actually extended that lesson. We went for the full hour on it rather than the half an hour session, just to make sure that we went through it all and they really understood when they have got that feeling inside and what they need to do to not let it get that big. It seemed to work quite well actually.
Year 3 teacher, School 30
Efforts were also made to make the lesson content more ‘relevant’ by teachers replacing the examples and stories provided with their own, so that they could ‘talk about our own experiences rather than the person in the story’ (Year 5 teacher, School 17):
I kind of did my own story rather than the one that was in the book because I like to personalise it more . . . it has more of a connection I think if you can put your twist on it but still keep into the main ethos.
Year 5 teacher, School 13
In some cases, the lesson plans were felt to be too ‘wordy’ and ‘long’ and could lead to the delivery being ‘dry’ and ‘dull’, and, indeed, this has been noted in previous evaluations of PATHS, both in the UK89 and the USA. 92 In such cases, teachers proactively added in a little more interactive work, such as role plays and small group or paired discussions, to ‘break it up a bit’ and ‘to try spice it up a little bit just, so there’s a bit more action . . . it can then have that maybe a bit of movement and a more of an active part in the lesson as well’ (Year 5 teacher, School 2):
I would probably instead of saying, ‘this is Peter’ I would get one of the children to come up to the front, get them to be the person. I try and involve a bit more drama to get them acting out the story more.
Year 5 teacher, School 17
I think some of the content of the lessons is a bit dry in its approach to the children. For instance today we did about ‘we are unique’, and whilst each element was fine I felt there was too much there, so I didn’t read the biography bit at the back. There was too much content.
Year 4 teacher, School 32
The overlap between PATHS and the variety of other SEL and SEL-related initiatives being implemented in participating schools (see Foundations for Promoting Alternative THinking Strategies) was also noted, and in some cases this created opportunities to ‘mingle’ the programme with Circle Time or other approaches:
I mean we do Philosophy for Children as well so sometimes it changes and it becomes one of those sort of lessons and . . . so it just depends on where it goes.
Year 4 teacher, School 27
I’ve tweaked it a little bit because obviously we were using the Kagan structures which lends itself exceptionally well to the PATHS things . . . it’s just built in it links so well . . . so I’ve just tweaked it like today I’ll hopefully look at using one of the Kagan structures to build up the idea of sharing and the concepts and things.
Year 4 teacher, School 24
Although most adaptations reported by teachers were proactive, there was also some evidence of reactive adaptations, primarily ‘responding to how the children are getting on at that point in time’ (Year 4 teacher, School 9) and reacting to the needs of the pupils so that they did not ‘lose concentration’:
[if you’re] teaching something new and an emotion that they’re not sure of, you know they’re worried, they couldn’t explain that, it needs a little bit longer. You need more examples and that will take longer.
Year 3 teacher, School 18
Sometimes I do make adaptations just generally how the children respond to it, like I don’t go out of my way to meet the changes, but obviously if you’re interested in in what a child’s got to say or you want to take it further you feel like you need to and that’s when I adapt it really, I don’t adapt the story or anything like that.
Year 5 teacher, School 9
Obviously, the nature of these adaptations reaffirms the preceding discussion pertaining to dosage (e.g. the extension of lessons in order to ensure that lesson objectives had been met).
Quality
Distinct from the concept of fidelity, implementation quality is understood as referring to how effectively an intervention has been delivered for the achievement of intended outcomes. In the context of PATHS, particular emphasis is placed on the extent to which teachers are able to extend and connect aspects of the programme to the broader context of the class and school (in particular the standard academic curriculum), in addition to children’s lives outside school. 55,56 The foundations for this were strong. Key PATHS concepts and curriculum content (e.g. recognising and understanding emotions, values, respecting others, empathy, co-operation and conflict resolution) were seen as being inherently intertwined with other aspects of the curriculum, namely English and RE. PATHS connected with themes of ‘decision-making’ and ‘making choices’ and the Golden Rule (e.g. ‘treat others how you would expect to be treated’; Year 4 teacher, School 1) that were already being discussed in RE. More general skills such as ‘working in small groups, turn taking, listening’ could be applied ‘across the board’ (Year 4 teacher, School 2). PATHS concepts were also applied in response to incidents and work with the children to resolve problems (e.g. ‘You can actually use your PATHS time for when something has happened at dinnertime and then say “what could we have done here?”’; Year 3 teacher, School 10) and in some cases were built into school assemblies to promote them across the school:
Our Head asked me about some of the things from PATHS and he did an assembly linked with some of the problem solving. So he did ‘Fingers Linked’ but then we talked about breathing, we talked some of the things we talked about in Year 6 that the children come up with. You know I like to have a bath if I’m stressed, like to go and sit on my own or I like scribble on a bit of paper. So we talked about those as a whole school assembly so that’s quite nice that we’re using that not just with the children that are doing PATHS but it’s sort of dripping it into other parts of the school.
Year 6 teacher, School 17
Understanding emotions was often linked directly with literacy and story work such as ‘describing how the character is feeling’ (Year 4 teacher, School 1), using the ‘PATHS terminology’ and resources, such as the Feeling Dictionaries and Feelings Face cards, to facilitate such work. For example, one classroom had ‘the Feelings Faces pinned up and the children often use it when we are talking about characters in stories or their own written work in literacy’ (Year 4 teacher, School 32).
The PATHS curriculum contains a host of strategies to aid the management, regulation and communication of emotions. It was clear that children were encouraged to use these approaches outside the immediate lesson context, and, indeed, there were numerous examples of children spontaneously making use of these strategies in response to incidents that occurred throughout the school day. Schools reinforced this by making efforts to implement the strategies ‘school-wide’ by training lunchtime organisers and teaching assistants, and making posters and other resources available and accessible (e.g. having copies of the ‘Control Signals’ posters placed strategically around the school). Indeed, one school had planned to paint the control signal traffic lights on the playground:
We’ve tried to link a little bit to the dinnertime . . . the signals posters we’ve enlarged one and put one outside at dinner time so we’re trying I think they’re trying to work with the dinnertime supervisors again ‘cos that’s often where issues with fallings out comes up at dinnertime so I think they tried to have the dinnertime supervisor spoke to one of your colleagues I think when she was in and just about the idea of what the process of what they do if there’s a something arises so they stop.
Year 3 teacher, School 6
Pupils made use of calming strategies, including the ‘Fingers Linked, Time to Think’ approach, which encourages children to take time out to calm down and reflect before they react, and the ‘traffic light’ control signals calming down process. Staff commented on their use of these strategies in playground, ‘drawing their hands or taking deep breaths or . . . getting a bit . . . they seem very grown up now that they’re talking about it, they’re like “I’m a bit stressed now” and I’m like “OK”, like “calm down” ’ (Year 3 teacher, School 2):
They come in and say that they do it they come in very proud of themselves they’ll say you know ‘somebody tried to have a fight and I did my fingers link [sic.] time to think and walked away’ so they are they are using the language when they come back in, we do want to get it out in the playground, we want to get some big playground posters.
PATHS co-ordinator, School 5
This was further corroborated by the pupils and parents:
My friend said I’d done something that I didn’t do but I used my PATHS . . . I felt a little bit angry that she said I’d done something when I didn’t so I just used Fingers Linked, Time to Think.
Pupil focus group, School 5
My son and this little boy clashed in class quite a lot and he said ‘it should help him deal with his emotions . . .’ he gets quite cross with my son at times and I think it’s been directed some of the techniques that they’ve learnt in the past and they’ve said to this child ‘why don’t you try doing this?’ and it might control his outburst so he said ‘hopefully it will help him’. I think what has happened in the past is he’s got frustrated and he’s took it out on [son, Year 4 pupil], like he’s screwed his work up and threw it in the bin. So I don’t think it ever gets to that stage anymore. Think it’s the strategies in place within the classroom to stop that ever getting to escalate it and he puts that down to PATHS.
Parent, School 6
However, it was noted that lower-ability and/or younger pupils sometimes needed a reminder:
. . . they can’t translate the skills, they know and they can tell you the perfect answers every time and they will say ‘Fingers Linked’ and ‘Time to Think’, but when it comes to a situation, they seem unable to do it. You always have to sort of step in before it goes beyond, they can’t yet do it themselves, but that will come as they go through the programme.
PATHS co-ordinator, School 6
Use of the strategies outside schools was also actively encouraged. For example, parents reported that children used ‘Fingers Linked, Time to Think’ and ‘Control Signals’ techniques at home and in one particular case a teacher remarked that this had been successful with a pupil who ‘finds it very hard to control his emotions’ and ‘was suspended from his previous school’ as he tended to get very angry and: ‘hit out and kick people . . . it’s been very useful to him because occasionally when he forgets and he hurts someone . . . I’ll say to him “well what about the traffic light system” and he’ll say “oh yeah, I should have done that, but I can’t, I need to remember quickly”. And sometimes he has remembered before he’s retaliated.’ (Year 3 teacher, School 1).
Pupil responsiveness
Unsurprisingly, children responded positively to the more interactive aspects of the PATHS curriculum; they particularly enjoyed the discussion elements and role-plays and ‘love the compliments sheets and “Pupil of the Day”’ (Year 4 teacher, School 1). They enjoyed time away from reading and writing and the more formal academic curriculum to talk to each other about themselves and their feelings, and ‘they love interacting with each other and chatting, so yeah, they do seem to enjoy it . . . there’s no rights or wrongs is there or . . . writing, it’s just about how you feel and having a bit of a chat’ (Year 4 teacher, School 5). PATHS provided ‘time out to think about themselves’ (Year 6 teacher) and ‘time to just reflect’ (Year 6 teacher, School 19): ‘they know they can relate to it and they like to tell you about their own experiences’ (Year 6 teacher, School 7).
In addition to lessons, the PATHS curriculum involves a process called ‘Pupil of the Day’. Children are randomly selected and wear a badge or identifier to be recognisable to other pupils and staff around the school. The Pupil of the Day may be assigned special roles and responsibilities for the day, and other pupils and staff complete a compliment sheet for the assigned pupil. This aspect of PATHS was particularly popular, with pupils in focus groups reporting that it made them feel ‘happy’, ‘proud’, ‘pleased’ and ‘confident’. These findings mirror other qualitative evaluations of the programme,87,89,92 in which the Pupil of the Day initiative was highlighted for praise:
They’re very good at giving each other compliments, they absolutely love it. At the end of the day usually and they take them [compliment sheets] home and they go out waving them at their parents. They’re getting very good at giving each other compliments. We did have to do a lot of demonstrating before they got it but, no it really does work for them. And the child kind of usually sits there looking quite pleased and happy . . . they’re normally smiling. So I mean the children have grown in confidence over the year that they’ve been with me anyway. I think at the time the compliments definitely help and I imagine it does have some impact on the child’s confidence.’
Year 3 teacher, School 24
That said, Pupil of the Day was not universally popular. In particular, there were concerns about its appropriateness for older pupils: ‘they hate being on a pedestal and don’t like all the jobs, it’s quite a young thing to be Pupil of the Day’ (Year 6 teacher, School 19). Thus, as in earlier research,92 questions are raised regarding the developmental appropriateness of aspects of PATHS for children at the upper end of the intended age range of the programme. There were also mixed reactions to the actual PATHS lessons, with some pupil responses echoing teachers’ comments concerns regarding length:
He said ‘I don’t mind doing it but it can be a bit boring sat on the carpet’.
Parent, School 6
Sometimes it’s a bit long.
Pupil focus group, School 5
Less interactive lessons I think are not as enjoyable as the interactive ones.
Year 4 teacher, School 6
Teachers felt that in some cases, the lesson content was ‘sheets just full of words’ (Year 3 teacher, School 13) and this was negatively affecting pupils’ engagement:
. . . the stories would be better with pictures and books because you’re reading it and they [pupils] just switch off because it’s just really long and they just get fed up and even though you’re asking questions and things it’s difficult like . . . it just didn’t work for them.
Year 3 teacher, School 13
Thus, there was a clear need to break up the lessons with more interactive elements, such as discussion and role play, and this may explain some of the adaptations reported earlier:
It depends on the lesson. I think more active lessons, especially when they’re doing role play, they’ve absolutely loved. Which I’ve been trying to do more with them recently, getting them up and experiencing it and expressing it. Because I think the couple of times when I’ve picked it up and dipped into it it’s been more me reading off a sheet, whereas when I’ve prepared it more and I’ve kind of brought in role play a bit more with them they’re more engaged, they’ve got a lot more involved.
Year 3 teacher, School 24
Teachers also commented that the age and general ability level of their pupils affected their response and engagement with the materials and lessons. The Jump Start lessons were also thought to be a little too simple for the older pupils: ‘I just didn’t think the maturity level and the stories that were being used were the right level for the children so they weren’t engaging as positively as I thought would have been beneficial for them’ (Year 5 teacher, School 6). In a similar vein, Year 6 teachers commented that they felt that some of the ideas were more applicable to the younger cohorts, and that, ‘Year 6 because of SATS [National Curriculum assessments] . . . it’s not been adapted as well’ (PATHS co-ordinator, School 5). This may, in part, explain the lower dosage observed in the second year of the trial (see Table 12).
Teachers were required to differentiate lessons for pupils perceived as being lower ability:
[children] who aren’t very articulate find it very difficult to express themselves and tend to just sit, and the shy children don’t want to be involved . . . some of them don’t or some of them just copy what everybody else is saying . . . if I just ask them a question and ask them to volunteer, they won’t volunteer.
Year 3 teacher, School 30
In these cases certain PATHS materials, such as the Feelings Face cards, were viewed as useful to aid communication and provide a visual scaffold for learning, but overall it was felt that more visual aids were required: ‘[I don’t think there are] enough visual things’ (Year 5 teacher, School 17). Unsurprisingly, pupils with behavioural problems and ‘anger issues’ were felt to ‘need extra support’ beyond that which could be provided through PATHS, whereas for pupils ‘who go out for social interventions sometimes maybe some of the strategies are a bit too advanced for them, so they just focus on the ones that they’re being taught in their one to ones’ (Year 5 teacher, School 10). Finally, there were also some notable sex differences in how pupils reacted and engaged in lessons:
. . . your girly girls really want to please and really want to give you examples . . . sometimes some of the boys are a little bit more reluctant but then some of them will surprise you. But I would say it’s more your girls, your pleasers I think.
Year 4 teacher, School 7
Reach
Consistent with the intended universal delivery model, class teachers and lesson observations concurred that ‘mainly all are present’ for PATHS lessons, and pupils reported that ‘we do it altogether’. Teachers understood the importance of the universal model, with some decreeing that ‘there’s no interventions when PATHS is on’ (Year 3 teacher, School 9) and ‘nobody’s removed because . . . it wouldn’t work if they were not all involved’ (Year 3 teacher, School 19). However, despite such intentions, this was often out of the teacher’s control and ‘sometimes that can’t be helped’ (PATHS co-ordinator, School 5), ‘they just kind of crop up all the time’ (Year 4 teacher, School 9), and ‘we can’t guarantee 100 per cent but we try to [have all pupils present] as far as possible’ (PATHS co-ordinator, School 17):
Mostly yes, I do try and make sure that they’re all in because that’s part of the programme that they’re all in, but there are certain times when some children have to be out for interventions. For example, I’ve got a child who’s just learning to read now and that child needs to be in, that needs to take precedence at the moment but I do try my very best to have all the children in PATHS lessons.
Year 4 teacher, School 18
If pupils were withdrawn during a PATHS lesson, this tended to be for ‘small groups’ or ‘one-to-ones’ for specific targeted interventions relating to literacy and/or social and behavioural needs, with teachers noting that ‘that’s perhaps a downside with the way schools are working at the moment’ (Year 4 teacher, School 24). Teachers did, however, comment that this could be the case for ‘any lesson’ (e.g. it wasn’t PATHS-specific), that it ‘varies’ and was not ‘deliberate’:
. . . [there are] so many different things going on you see in a school . . . you’ve got your booster groups where a TA [teaching assistant] might take some out, you’ve got Coral Cove, which is for different issues, it can be behavioural issues, it can be rewards, it can be worries and concerns, and there’s never a time when all the children really are in the class.
Year 3 teacher, School 5
It was noted that ‘the sad thing, sometimes they do miss out on that and it is those children who need it’ (Year 6 teacher, School 43). However, efforts were made to enable children who missed PATHS lessons to be brought ‘up to speed’: ‘we do try and catch up, I do give the children an update if they weren’t there on what we have been doing’ (Year 6 teacher, School 43), and ‘we have a little, like class, what we were doing while you were out, just so they caught up’ (Year 3 teacher, School 2).
Those most likely to be withdrawn were children with statements of SEN, which are now referred to as Education, Care and Health Plans, who could ‘come and go’ and be could be ‘in and out’ depending the nature of the support provided for them and how they were felt to be coping with a given lesson:
I’ve got one boy who’s got autism, so it depends on his need that day whether . . . because sometimes he can’t cope in the classroom situation, so I try to have him in every time, but if his need is such that you know, he can’t be in at the time, then he might miss it.
Year 3 teacher, School 38
Factors affecting implementation
A wide range of factors were identified that affected the patterns of implementation noted above. In alignment with the key knowledge sources informing our analysis (see Qualitative analysis), factors are presented ‘from the inside out’, beginning with programme-level issues and radiating outwards through the social ecology of the classroom and school to encapsulate the attitudes of teachers, the implementation support system and, finally, broader influences nested within and beyond the implementation environment.
Programme characteristics
Resources and materials
The PATHS resources are comprehensive and each curriculum pack comes with a ring-bound file containing each lesson plan, the relevant activities, worksheets and resources required, meaning that ‘you’ve got everything’ (Year 4 teacher, School 13). As highlighted in previous qualitative research on PATHS,90 it was viewed as ‘easy to use’ and follow, ‘manageable’ and ‘low maintenance’: ‘you can just pick up the file and teach it’ (Year 4 teacher, School 7). This ‘makes it easy for you to understand as a teacher’ (Year 5 teacher, School 17). This made teachers feel ‘quite comfortable’ in delivering the PATHS curriculum. Materials were considered to be accessible and ‘very structured, it’s very clear what the objectives are and how you’re going to get to those objectives’ (Year 5 teacher, School 10). This was viewed as facilitating delivery because it minimised planning and preparation time:
I just think it’s all fairly straight forward. It sort of explains itself doesn’t it? There’s nothing that I’ve thought ‘oh, I don’t understand that’.
Year 4 teacher, School 5
The lesson plans were succinct, easy to follow, very directive, that’s great for class teachers these days because they have so much planning to do for so many areas and so much of it doesn’t give you enough guidance you have to go and suss things out for yourself.
PATHS co-ordinator, School 17
It is nice that they are kind of set out for you, you don’t have to do the thinking kind of thing.
Year 4 teacher, School 1
You have the folder, you can just pick it up and everything, posters . . . there’s not many programmes where you just get a folder and everything you need is in it so that was brilliant as a teacher.
Year 4 teacher, School 2
However, one teacher noted that the comprehensiveness of the materials could prove to be deceptive in terms of the amount of preparation time required:
. . . you definitely need to familiarise yourself with it . . . we were just given this file and it’s like ‘don’t worry everything’s there for you’ . . . I then realised very quickly you do need to be prepared, you need to familiarise yourself with stuff.
Year 3 teacher, School 18
Furthermore, a significant number of teachers were averse to the scripted nature of PATHS, feeling that it was too prescriptive, and ultimately feeling restricted by it: ‘now that it’s been delivered I think it was that people felt a little bit constricted by the pack and having to follow the script’ (PATHS co-ordinator, School 5). Among such teachers, there was a desire for flexibility within the lessons: ‘I’d love to be more . . . because I’m very flexible in my teaching anyway, so to be able to adapt it’ (Year 3 teacher, School 1). In line with the preceding sections on fidelity and adaptations, the need to adapt PATHS to suit class needs and increase the interactivity of lessons was of paramount concern:
Some activities that I’ve done, just took them straight off the lesson plan, have got a bit dry . . . I found this out last year more, whereas this year I’ve done a lot more adapting, I know what would work and how could I adapt them.
Year 5 teacher, School 1
There was an overwhelming sense that the nature of the PATHS materials required more visual forms of presentation, with teachers now commonly using interactive whiteboards as part of their daily teaching:
I think the majority of people now would be using white boards and presentations. It does take time you’ve got to type up you know a lot of the [lesson materials], making it more child friendly but I think no more than you would prepare for any other lesson.
PATHS co-ordinator, School 5
As curriculum packs are paper based, a significant number of teachers reported using Microsoft PowerPoint and whiteboards to display the materials to the class and make the content more accessible, interactive and visual, because:
. . . the sheet’s just full of words and it’s reading it and the stories, it would be better with pictures and books because you’re reading it and they just switch off because it’s just really long and gets really . . . they just get fed up and even though you’re asking questions and things, it’s difficult to read.
Year 3 teacher, School 13
Ironically, in contrast to the aforementioned ‘off the shelf’ nature of the materials, the desire to present PATHS through visual media led to significantly increased preparation time for some teachers. For example, in one observed lesson, the teacher reproduced the entire lesson plan on PowerPoint slides in order to provide a visual anchor for pupils (and to avoid reading directly from the lesson plan script in the curriculum pack).
Promoting Alternative THinking Strategies content, concepts and strategies
The central content, concepts and strategies embodied in PATHS were familiar for most teachers. They reported having experienced ‘versions’ of them through their work in delivering SEL and SEL-related provision prior to the introduction of the programme (see Foundations for Promoting Alternative THinking Strategies):
I’d say they are kind of messages I’ve had before. You know through different programmes and just generally life. I mean it’s strategies, things that you need to do. I’ve met similar kind of strategies through SEAL and other programmes before as well.
Year 6 teacher, School 10
Although some teachers perceived the concepts taught through PATHS to be developmentally appropriate (e.g. ‘I don’t think there has been anything that I thought was too young for them or too old for them, it seems to be about the right pitch for their age’; Year 3 teacher, School 30), others have felt that ‘perhaps there could be even more and sort of cater for a range of children, backgrounds and abilities’ (PATHS co-ordinator, School 24). This did, however, seem to be class dependent:
I think the scope of abilities in my classroom are massive and I think that the very high abilities are fine with it and the ones who are emotionally intelligent are fine with it but the ones who are very low with self-esteem issues and the ones like I’ve mentioned, EAL children or SEN children, I think they do struggle.
Year 5 teacher, School 13
Aligning with issues raised in Adaptations, there were also concerns pertaining to cultural transferability of some of the ideas, concepts and language were, ‘very Americanised, it’s hard to get your head around what they’re trying to say . . . because it’s worded differently than we word things and everything’s worded way, way over the top so we have to try and bring it down to a level that our children can understand so it’s hard’ (Year 3 teacher, School 30) and ‘they are very American in their approach I think and quite didactic and preachy sometimes’ (Year 6 teacher, School 17):
Oh let’s put our happy face on, let’s put our cross face on, that’s what all the Year 3 seems to be about and the kids don’t like it, teachers don’t like it. It’s the language and it’s the concept, it’s everything about it. It’s typical how they speak in America and they do things over the top in America, whereas we don’t do that here.
Year 3 teacher, School 30
Some of the themes addressed were also deemed to be inappropriate, for example the study skills content in Year 6, ‘just wouldn’t have applied to them . . . it was a bit too mature’ (Year 6 teacher, School 19), whereas some of the story content was not suitable for the cohort as the ‘children just can’t relate to some of the stories’ (PATHS co-ordinator, School 10):
Quite a lot of the stories aren’t relevant to our particular children because of their background. There are stories about going to shops and choosing new clothes and that doesn’t happen very often for the kids in our school. Going on holidays, and that doesn’t really happen. So the stories, I’d say about 80% of them we tweak to make them relevant . . . the ones that are just a generic story that matches the objective, those tend to all be, like there was a child stealing in a sweet shop and all of the kids said ‘but it’s against our religion to steal so why would they be stealing?’ and that was the end of it.
Year 3 teacher, School 39
Teacher attitudes
Teacher attitudes towards SEL were positive. They saw the need for it and valued its importance in education in a general sense, but also connected this more specifically to the needs of their class and the potential benefits of implementing universal interventions.
An explicit approach to social and emotional learning
Social and emotional learning was perceived to be a ‘crucial part of their [the children’s] education’ (Year 3 teacher, School 18), imparting ‘more skills that they need now’ that would be ‘useful for them’ (Year 3 teacher, School 27). Such skills were viewed as a ‘beneficial tool for equipping children with how to deal with emotions and problems which a lot of children face more and more now’ (Year 4 teacher, School 5). Teachers valued the fact that PATHS uses an explicit instruction model (e.g. a taught curriculum), with a structured approach, thorough resources, strategies and varied activities, feeling that this was something children would respond well to, as ‘it’s what they need, the pace is good, a nice mixture of listening, talking, deciding, reporting back, it keeps them engaged’ (Year 4 teacher, School 2):
There was never an actual lesson with SEAL, it was just taught, you know, through other lessons, to actually concentrate it on a programme for an hour per week, looking at the friendship, looking at various ways to solve problems and things in schools, so I suppose in that way been good thinking about that for one hour and week and then they can come in as well to other areas of the curriculum.
PATHS co-ordinator, School 19
Perception of need and benefit
It was widely believed that an intervention like PATHS was needed and would benefit pupils in participating schools (indeed, this was one of the main reasons for schools joining the trial – see Foundations for Promoting Alternative THinking Strategies). Particular needs were identified to help address ‘the behaviour stuff, calming down stuff’ (Year 4 teacher, School 13) and provide children with a ‘toolkit’ to manage and regulate their behaviour and work well with others. Pupils’ social and emotional skills were recognised to be ‘poor’ or ‘lacking’, and the concepts and strategies imparted through PATHS were seen as means to address this:
We have a few pupils that go from 0 to 100 in a millisecond on the anger scale. And it’s that kind of repetition and the calmness of PATHS really and the steps and the logic that they’re missing. Because we do PATHS so regularly, hopefully it will be a drip, drip effect and start to affect their behaviour and they will soon hopefully start, even one time, to take a second and take that deep breath. Because they’re constantly hearing it now.
Year 4 teacher, School 2
There was also a perceived need to support pupils with ‘the relationship stuff’, with the feeling that PATHS would be ‘helpful for them to have strategies to stop and think before they kind of react badly to something that somebody said off the cuff to them’ (Year 6 teacher, School 17). It was believed that if these sorts of issues were dealt with, it would have a ‘knock on effect of improving the learning and the quality of teaching that goes on’ (Year 3 teacher, School 9). As noted earlier, PATHS was also viewed as having potential utility in relation to the transition to secondary school, such that, ‘when they go off on their own they’ve got those tools and that’s what we’re hoping PATHS will help them with’ (Year 4 teacher, School 17). Thus, despite some initial pockets of resistance manifested by ‘grumbling’ and ‘here we go, another new thing’ (PATHS co-ordinator, School 5), most teachers identified a need for and potential benefit of PATHS and were therefore open to change and willing to be ‘flexible’ and to ‘adapt’ their teaching approach to accommodate the curriculum. Ultimately, they were ‘willing to try anything if it’s something that we think will benefit the children’ (Year 6 teacher, School 2).
Fitting Promoting Alternative THinking Strategies in
Despite the initial attraction of the structured, prescribed approach to SEL embodied by PATHS (see A more structured approach), there were clear difficulties experienced by teachers in their attempts to integrate the programme in line with the developer’s expectations (an issue that has been noted in other qualitative evaluations87,92). Ironically, given the perceived constraints and pressures that teachers reported (see Time, Competing priorities and Curriculum demands), a more flexible implementation model – such as that used in the oft-maligned SEAL programme – would probably have been more feasible.
Time
Teachers reported that the most significant barrier to implementation as planned was finding the time to fit the PATHS lessons and associated processes (e.g. Pupil of the Day) into an already packed timetable, particularly given the push and pull of competing priorities and curriculum demands (see Competing priorities and Curriculum demands). Despite PATHS lessons being timetabled, such factors often affected the regularity, frequency and duration of actual delivery, with teachers reporting that they ‘squeezed’ lessons together to make them ‘shorter’ or to ‘be a bit more snappy with it’ (Year 3 teacher, School 10). Typically, then, teachers noted that ‘the intention was twice a week but the reality of it was very different’ (Year 6 teacher, School 17) and that ‘it’s down for twice a week but realistically it’s not feasible’ (Year 4 teacher, School 17):
It’s really tricky in our timetable.
Year 6 teacher, School 2
There hasn’t been time to do . . . most of the supplementary activities.
Year 3 teacher, School 10
Competing priorities
Despite the fact that teachers valued PATHS and saw it as ‘important’, the principal pressure on time was the fact that ‘there’s so many other things going on’ (Year 5 teacher, School 1) and ‘so many other things to be thinking about’ (Year 3 teacher, School 5), especially when ‘little things crop up . . . the school is involved in lots and lots of different things as well’ (Year 5 teacher, School 2). There was a need to be ‘flexible’ where PATHS was concerned, as it could ‘slip’ when things got ‘chaotic’ at particular points in the school year (e.g. Christmas and Easter productions, holy weeks, SATS week): ‘different times of year are obviously a lot harder to fit things into your timetable than other times’ (Year 3 teacher, School 7). At such times, PATHS lessons and procedures were the most likely to be discontinued in order to free up space:
We tend to do a lot of activities in this school, we tend to do . . . we do . . . we’re learning violin at the moment. Being a Christian school we’re quite big on Christmas productions, we’re just about to embark on an Easter production, it tends to eat into the curriculum, so normally when we’re doing all these extra activities . . . in a way something has to drop off and it tends to be [PATHS].
Year 4 teacher, School 6
Thus, despite clear perceptions of need and potential benefit, in practice PATHS was rather easily displaced, particularly when considered in the context of the central academic curriculum (see Curriculum demands).
Curriculum demands
In a packed timetable, PATHS was not seen as a ‘priority’, as it was not one of the ‘core’ academic lessons, with teachers noting that ‘it’s obviously not as important as literacy, numeracy, things like that’ (Year 6 teacher, School 14) and ‘you can’t afford to lose that from your literacy, your numeracy’ (Year 4 teacher, School 17). As such, it was ‘easier to push aside’ (Year 3 teacher, School 2). Mirroring findings elsewhere, external (e.g. Ofsted) and internal (e.g. senior leadership) pressure to focus on English and Maths was high, as ‘we don’t get judged on the assessment of PATHS but we do on literacy and numeracy’ (Year 5 teacher, School 13). 87 This was particularly notable in higher year groups with the ‘SATS push’:
And in the pecking order, you know you want to put that equally at the top but there are other targets that are going on. You can’t say to Ofsted and your grades online but I got a green flag for my PATHS. So that’s the issue that you are under as a Year 6.
Year 6 teacher, School 7
The pressures relating to the core academic curriculum produced constraints on when PATHS could be timetabled, with English and Maths taking priority in morning lessons when the children are regarded as being more attentive and focused:
Mornings of our school were timetabled in rigidly with Maths, English and reading. Like every morning is completely out. So it’s just the constraints of the school timetable in some cases that doesn’t make it easy to fit it in.
Year 6 teacher, School 7
Thus, finding the time to accommodate PATHS was a notable challenge that meant schools that were ‘struggling’ to fit it in, leading to the perception that, ‘this was just another thing on top of everything . . . it was just extra workload on an already pressing timetable’ (PATHS co-ordinator, School 6):
The amount of subjects they expect us to teach, like we have to do an hour of Maths every day, we have to do an hour of literacy every day, we have to do two hours of physical education a week now, we’re a church school so we have to do two and a half hours of religious education . . . it’s a massive chunk out of your day. So by the time you add those up, you’ve got to do two hours of science, we get an hour of history or geography a week and an hour of art or design and technology a week and that’s your whole week gone.
Year 3 teacher, School 41
I think there was a real wariness over curriculum time, that’s been an ongoing issue because we have a very, very crowded curriculum. We’ve had to devote a lot of extra time to literacy in recent years and we’re now realising that numeracy suffered and much as we try and keep the curriculum broad it is really hard to curriculum time to devote.
PATHS co-ordinator, School 17
Leadership support
The PATHS curriculum includes specific guidance for headteachers and the senior leadership team (SLT), with a focus on their role in supporting the embedding of practice within school. There was a clear recognition that the programme needed to be ‘led’ and supported from senior management: ‘when you do start something new there is . . . that support network within school that people will help you with things’ (Year 3 teacher, School 13). In order to be afforded the level of priority required, PATHS needed to be ‘in the school development plan so you have key priorities and background priorities’, with ‘everyone trying to focus on it’ and it being ‘discussed at senior leadership meetings’ (Year 4 teacher, School 9).
Schools were asked to assign a PATHS co-ordinator, and this was suggested to be someone from the SLT. Their role, although it did vary from school to school, generally was as a ‘point of reference to PATHS’ (Year 6 teacher, School 10) and encompassed ‘overseeing it, organising training, sorting out cover for staff to go on training, making sure everyone’s got the packs, making sure everyone’s got the resources to go with the pack’ (PATHS co-ordinator, School 5). PATHS co-ordinators would ‘talk . . . to teachers regularly about what’s going on, how they’re feeling about things’ to get ‘a good overview of how the thing has developed and how people are feeling about what they’ve been doing’ (PATHS co-ordinator, School 17). The co-ordinator role, particularly in the early stages of implementation, was needed to create some ‘enthusiasm . . . chipping away at the negative attitude that we’ve got in some of the other classes’ (Year 3 teacher, School 5), sustaining focus by ‘making sure it’s in the forefront of everybody’s mind, which is good’ (Year 6 teacher, School 5), and in order to ‘make sure it remains high profile’ (PATHS co-ordinator, School 5). Class teachers noted that having someone in this role at the school was ‘helpful’, as it gave them a ‘first port of call’ when needed.
Staff at one school specifically highlighted the importance of the role of the SLT in promoting PATHS and embedding it within everyday whole-school practice:
We also recognise if senior leadership do not wholeheartedly promote it, the class teachers can’t drive it alone. It needs to come from us. It needs to make reference to it, I make reference to it in my weekly assemblies, I make reference to it in corridors when I speak to children and it’s that raising of the profile of it which I think we do very well.
PATHS co-ordinator, School 18
They went on the note the functional and leading role of the SLT in supporting and guiding class teachers at the ‘chalk-face’. Class teachers themselves highlighted the utility of this support as, ‘if it’s not coming down from the top . . . they need to put it on a higher profile for everyone else to realise how important it is’ (Year 6 Teacher, School 18):
It would be unfair and it will not work for it just to fall on class teachers because the class teachers have got enough initiatives that are thrown at them. It’s our job as strategic leaders to make sure we decide on the initiatives and with the input but that we sift through almost what is going to be beneficial and what’s useful for the whole school.
PATHS co-ordinator, School 18
At a more practical level, schools’ SLTs were important in enabling teacher time to be freed up to ensure that ‘we all had our training’ (PATHS co-ordinator, School 5). They also had a role in embedding PATHS across the school, beyond just ensuring that the class teachers delivered the programme as part of the trial: ‘they support it, make sure that it’s run consistently [and] ensure that it’s run throughout the school, going to lunch time organisers’ (Year 5 teacher, School 13). Ensuring that all staff were familiar with the PATHS concepts, strategies and language was seen as facilitative of generalisation of learning. In line with this, there were many observed examples of headteachers recognising the Pupil of the Day and complimenting them. Headteachers were also kept updated on progress, and observed some of the lessons being delivered as part of the monitoring process. Such actions were perceived as ‘good because it also raised the profile in our monitoring process across the school that we’re not just doing literacy, numeracy and other curriculum subjects we’re actually including PATHS in that’ (PATHS co-ordinator, School 18).
Promoting Alternative THinking Strategies was also extended to whole-school assemblies and there were a number of examples of headteachers using PATHS approaches and strategies in these, ‘so it is really well known around the school’ (Year 5 teacher, School 17). It was also noted that it was a particular advantage when the class teachers involved in implementing PATHS were also members of the SLT. This enabled the programme to gain traction, as it ‘makes it sort of easier in a way because you know the things that are happening at classroom level are then fed back because she is a member of SLT’ (Year 6 teacher, School 9). There were, however, also instances at a minority of other schools in which SLT involvement was more of a lighter touch and ‘more from a distance, so she [the SLT member] knows that we implement it and she expects us to do that. But really it’s more from a distance I think’ (Year 4 teacher, School 7).
External support
Given the status of the project as an efficacy trial, the training and support package for teachers and school staff followed the model recommended by the programme developer and team at PSU. As noted in the TIDieR programme description, this comprised a whole-day training event at the start of the school year, with a half-day follow-up event 4 months later [delivered by experienced PATHS trainers from PSU, alongside the project team PATHS coaches, who received continued supervision from one of the trainers via Skype™ (Microsoft Corporation, Redmond, WA, USA) throughout the trial]. In addition, each school was allocated a coach (referred to as a ‘PATHS psychologist’ because of confusion among some schools over the term ‘coach’). PATHS psychologists conducted regular school visits to support teachers’ implementation. This typically involved lesson modelling, observation and feedback, training additional staff members (e.g. lunchtime organisers and new staff) as required, and dealing with queries relating to implementation.
Training
Teachers found the training to be ‘very useful’, reasoning that ‘attending the training session . . . is pretty crucial, I think otherwise you don’t feel like you’ve bought into the programme’ (Year 4 teacher, School 6). It gave teachers ‘confidence’ and ‘motivated’ them. The initial coverage of PATHS theory and evidence was considered vital, so that staff could understand ‘why it is beneficial scientifically and emotionally’ (Year 5 teacher, School 17), as ‘people want to know background and research and figures attached to it don’t they as to why we’re doing it’ (PATHS co-ordinator, School 10). It also gave ‘meaning and a context for what you’re doing in the classroom’ (Year 6 teacher, School 9). The subsequent coverage of the more practical aspects of PATHS, including the introduction of the lesson packs and other resources, helped teachers to visualise delivery, with one stating that ‘it was good to be able to go through the process of how we were going to deliver it, go through the resources and generally just highlight things . . . that I’ve actually thought about it before we had some practice sessions’ (Year 3 teacher, School 9).
The training also provided ‘discussions about how to timetable and how could we adapt things’ (Year 6 teacher, School 7) and ‘it was really useful, some of the ideas I could take back as well into school’ (PATHS co-ordinator, School 2). Overall, the teachers found the training to be very beneficial in demystifying PATHS:
It was brilliant seeing the examples of PATHS lessons and how other teachers had used the resources. I thought they were absolutely brilliant. I think I wouldn’t be able to teach PATHS like I do now if I hadn’t gone on those training days because I would still be a little bit like ‘oh my goodness what’s this big book?’
Year 5 teacher, School 17
Despite the overwhelmingly positive reception to the training, the length was queried [e.g. ‘I don’t think we need the whole day’( Year 4 teacher, School 1)]. Some teachers also reported a preference for focusing purely on the logistics of implementation rather than the context and background [e.g. ‘I am not sure really that we need to know the ins and outs’ (Year 4 teacher, School 1)]. This practical emphasis was noteworthy in other teachers’ responses, particularly in their recommendations to ‘maybe show a video of the lesson’ (Year 3 teacher, School 14):
I think we definitely needed more on the specific lessons and the structure . . . some of the things we were a bit confused about, when we started to introduce the Golden Rule, when to introduce Fingers Linked, Time to Think.
Year 4 teacher, School 27
The follow-up training received much more mixed responses. It was recognised as being a good ‘refresher’ and as being useful for ‘hearing what other schools did’ (Year 5 teacher, School 2) and it was noted that it ‘reminds you about some of the things that you might have neglected to do’ (PATHS co-ordinator, School 5) and provides opportunity for ‘shared practice’ and ‘discussion’, which ‘was beneficial in that I could speak to other Year 4 teachers and see how they were doing with it’ (Year 4 teacher, School 5). However, in some cases, this opportunity for discussion was also used to air grievances (e.g. ‘the people who I was sat with had a bit of a negative view towards PATHS and I felt like they were ramming this negative view down my neck’; Year 4 teacher, School 1). There was also a focus on barriers to implementation, but some teachers ‘felt the half day was spent listening to that rather than teaching them anything new’ (PATHS co-ordinator, School 10) and noted that ‘there wasn’t really any solutions to the problems that we had . . . it wasn’t massively useful’ (Year 4 teacher, School 17). Other teachers did not find the follow-up training necessary as they had ongoing support from the school’s PATHS psychologist: ‘Yourself and [PATHS psychologist] are always keeping in touch and can help us out at the drop of an e-mail or anything’ (Year 4 teacher, School 2).
Coaching model
As in previous research that has focused on support for PATHS implementation,92 PATHS psychologists/coaches were viewed as ‘really supportive, really helpful’ (Year 4 teacher, School 9) and provided a ‘back-up’ throughout the course of implementation: ‘it’s nice to have, knowing that somebody is there’ (Year 5 teacher, School 13), it is ‘good to have people to know you can ask’ (Year 6 teacher, School 5), and ‘if there’s any problems we know that he’s [PATHS Psychologist] approachable and that we can we can contact him for support there’ (Year 3 teacher, School 9). Such support ‘can be a lifeline for you’ (Year 6 teacher, School 7). Thus, the coaching model provided reassurance and helped to inspire confidence in delivery: ‘it is good to have somewhere to go if you are a bit worried and good to have someone to come in to make sure you are not doing it wrong’ (Year 4 teacher, School 1). This was seen as especially important for NQTs: ‘it’s quite vital if you’ve got maybe NQTs or teachers who are not very confident . . . for them being able to call the PATHS psychologist to say “please help me sort these problems out”’ (Year 4 teacher, School 36).
The fact that the PATHS coaches were also qualified teachers aided them in their role and in their interactions with schools and teachers, giving them more credibility and authority: ‘the fact that she’s a practitioner first and foremost has been essential. She comes in and she understands the context in which teachers and they really appreciate that’ (PATHS co-ordinator, School 17). Crucially, their support ‘keeps it [PATHS] at the forefront . . . it keeps that impetus going’ (Year 4 teacher, School 10):
I’ve been guilty – and I know everybody has – of starting off with the best intentions and then it kind of goes by the wayside. So it’s definitely kept me on track.
Year 4 teacher, School 2
One of the key duties of the PATHS coaches was the modelling of lessons, and this was seen as particularly beneficial for class teachers, as ‘we could see how it is supposed to be done’ (Year 6 teacher, School 18). It also allowed teachers to ‘check I was doing it OK myself, so it was quite reassuring’ (Year 3 teacher, School 24). Lesson modelling also gave the teachers a rare opportunity to focus on children’s responsiveness:
When they come in to maybe do a PATHS lesson, not necessarily to see how it’s done, but to have chance to see how all of them react or focus on certain children, because you don’t always get to do that when you’re at the front and you’re teaching . . . it’s hard to hone in on just a few of them, so it gives you chance to sort of stand . . . it’s really useful.
Year 6 teacher, School 2
The additional support functions (e.g. training additional staff members, briefing the SLT and school governors, talking at parents evenings) aided the ‘knitting’ of PATHS into the wider school community, so ‘it’s more like a whole school thing’ (Year 3 teacher, School 6):
She [PATHS Psychologist] even came into senior leadership one week, I wanted her go through all the main points, you know with the traffic light signals and that kind of thing, we got leaflets so that anyone like caretakers and other people that are in school or supply teachers had leaflets to look at so that they knew what was going on even if they were only here for a short time.
PATHS co-ordinator, School 5
Despite the perception of the need for coaching support, there were also a significant number of teachers who did not access, or feel like they needed, this additional support. One reason was that the training and resources were thorough and ‘self-explanatory’, meaning that ‘we have felt that the resources are so good and so easy to follow that she has offered to come in and teach classes, so we can observe, but I have never felt that we have really needed that’ (PATHS co-ordinator, School 24) and, ‘it is so prescriptive it is quite clear what the expectations are how to use it . . . if you’re following the script you really shouldn’t be able to go far wrong’ (Year 3 teacher, School 9). Consequently, teachers felt that they ‘haven’t needed it [coaching support] that much because I feel I understand’ (Year 5 teacher, School 18).
Additional barriers to accessing coaching support were the issues of time and competing priorities. One teacher remarked that ‘understanding PATHS isn’t our biggest priority’ (Year 4 teacher, School 43), ‘with all the pressure that we’re under to have children achieve in English and Maths and Science and reading and writing and grammar . . . they’re the people we need a helpline for sometimes . . . I’m never going to use the PATHS e-mail’ (Year 4 teacher, School 36). Teachers struggled to find time to make full use of the support on offer: ‘[with a] limited amount of time, even 5 minutes is a big chunk of your time at lunch time’ (Year 5 teacher, School 9):
I’ve rarely the time to ask . . . you’ve got a million other lessons to plan so you’re not going to use that support as far as I’m concerned . . . time doesn’t permit it and the curriculum doesn’t.
Year 5 teacher, School 1
Perceptions of impact
Promoting Alternative THinking Strategies is intended bring about improvements in a variety of areas (see Figure 2 and also hypotheses 1a–j). As in previous qualitative evaluations,88–90 impact was perceived across these areas, which we discuss in more detail in this subsection, but in particular the direct proximal impact on social and emotional skills was noted, with frequent references to pupils becoming more ‘emotionally literate’ and ‘emotionally intelligent’. They were also observed to make regular use of the strategies taught within PATHS to regulate their emotions and more effectively manage their relationships with others. Indirect, distal effects on academic skills and learning and behaviour were also identified. Of particular note is that teachers theorised impact in line with SEL theory,75 that is, they saw improved social and emotional skills as laying the foundation for more effective learning in the classroom.
Emotional literacy
Pupils were generally observed to be ‘more expressive, being more emotionally intelligent’ (Year 3 teacher, School 25), following participation in PATHS. More specifically, their emotional vocabulary was seen to have improved, as pupils were able to ‘label their emotions a bit more’ (Year 3 teacher, School 27). Teachers were ‘impressed with the words they came up with’ (Year 6 teacher, School 17) to articulate their feelings, and their understanding of the subtle differences between related emotions:
I would say it’s made them more understanding of different feelings they might not have heard of before. Because to define between scared and afraid even though they’re quite similar. I just think it’s given them an understanding of maybe verbalising what that feeling was. Because they’ve understood what that feeling actually means when they might not have heard it before or maybe they’ve said it before and not known what it meant. So they now understand the name of the feeling, putting a label to it.
Year 3 teacher, School 7
Pupils were regarded as being more comfortable discussing their emotions, even early on in the implementation process: ‘[they] seem to enjoy and they like talking about it’ (Year 3 teacher, School 1). Alongside this was a growing recognition that feeling ‘negative’ emotions was acceptable and that pupils were better equipped to discuss why they feel a certain way, because they were ‘find[ing] it much easier to talk about emotions’ (Year 4 teacher, School 9):
I think it’s done quite a few of them the world of good because they feel that they can talk about things that maybe they couldn’t talk about before . . . and sort of realising that you know, it is OK to feel angry or to feel upset, but it’s how you deal with it and it’s a learning curve really at this age for them to you know . . . so they’ve gone from screaming and crying and being really upset to being able to explain really what’s caused them to feel that way.
Year 3 teacher, School 13
The shared language, concepts and strategies embedded in PATHS were seen as aiding communication and relationships between pupils and teachers. The Feelings Face cards were particularly useful in this regard:
And it’s good for me because I noticed when I’ve walked round the class sometimes . . . I’ve noticed a feelings card. And I think, ‘oh I must speak to that child then about that’. And particularly if it’s one of the uncomfortable feelings, if they feel scared, worried, you then want to delve a bit deeper and find out. And you’re finding that now that the children are sharing more and more with you. Things that you probably wouldn’t have discussed before and wouldn’t have known about the child. So you’re getting to know that, you feel you’re getting to know them as a person a lot more.
Year 3 teacher, School 18
Generalisation of learning
Pupils were observed to have successfully generalised the concepts, strategies and approaches to managing friendships and resolving conflict taught in and through PATHS. In line with the proximal impact on emotional literacy, teachers observed greater consideration of others and other people’s feelings, stating that pupils were ‘more aware of each other’s emotions’ (Year 3 teacher, School 2), were ‘sympathising and empathising with each other’, and that they ‘look out for each other a lot more’ (Year 6 teacher, School 19). Pupils were able to ‘solve problems more readily’ (Year 6 teacher, School 26) and, importantly, ‘they’ve related parts of the PATHS lesson when they’re trying to sort out conflicts between themselves’ (Year 3 teacher, School 24):
I think there’s been an improvement in children’s ability to talk about, have the skills to talk about the way they feel about something that has happened, and then talk about reaching a resolve on that. I think they have particularly with this class I’ve got now, their ability to sit and talk through a problem is really good. They’re really, really particularly good at it.
Year 6 teacher, School 9
Promoting Alternative THinking Strategies procedures such as Fingers Linked, Time to Think gave pupils a ‘little toolbox of strategies’ (Year 3 teacher, School 18) ‘for resolving problems and calming themselves down’ (PATHS co-ordinator, School 24). Teachers commented that they observed a reduction in ‘lunchtime incidents, our big monitoring thing because that’s where the un-structure [sic.] is, so we’ve seen a big reduction of serious problems’ (Year 6 teacher, School 13) as well as a ‘lot less falling out in the class’ (Year 4 teacher) as a result of PATHS:
I do think that the problem-solving skills have helped them, instead of just falling out they’ve been able to go “well actually” and they come to me and said “they think, this we think that” and we’ve managed to sort the problem out before that would never have happened.
Year 4 teacher, School 41
The pupils, too, recognised that they were also using the PATHS strategies to calm themselves and manage potential problems and conflicts with their peers, as evidenced in the following excerpt, which was noted earlier in the section on implementation quality:
My friend said I’d done something that I didn’t do but I used my PATHS . . . I felt a little bit angry that she said I’d done something when I didn’t so I just used Fingers Linked, Time to Think.
Pupil focus group, School 5
Moreover, children in our focus groups reported that PATHS ‘tells us how to calm down’, ‘can help you with your friends’, ‘helps you when you when you have a problem’ (Pupil focus group, School 6) and ‘helps you learn about your feelings’ (Pupil focus group, School 27). The evidence supports previous studies in this area, in which generalisation of learning was highlighted. 89
Behaviour problems
Staff reported notable improvements in behaviour and conduct, stating that ‘incidents in the playground seem to have lessened’ (Year 5 teacher, School 18) and that there were ‘a lot less arguments and altercations’ (PATHS co-ordinator, School 25). Pupils were less likely to be sent out of the classroom or reported to the headteacher. As noted in other qualitative research relating to PATHS implementation,92 this led to a perception that the school environment was calmer, as ‘certain children that we think they would have just kicked off and let rip are now using the Control Signals as kind of “stop, make a plan”, and Fingers Linked . . . they’re going through all those steps’ (PATHS co-ordinator, School 5):
It does feel a lot calmer, you’ve still got your pockets of trouble and I think you’re always going to have that, but generally speaking it does feel like there’s a bit of a calmer atmosphere, you know, all the children are walking round sensibly and calmly and no-one’s storming out of class or slamming doors which it did used to, it used to be like that a lot.
Year 4 teacher, School 5
Promoting Alternative THinking Strategies was seen as helping reduce pupils’ behaviour problems by ‘making them think a bit more now before they act’ (Year 6 teacher, School 26). It also provided a means to talk about such issues through the development of children’s emotional literacy:
. . . the children who have behaviour issues have benefited from it more, talking about why they have conflict, why they have difficulties with calming down and what makes them angry – they’re very happy to talk about all that I think – it’s them that [PATHS] has the most impact on.
Year 6 teacher, School 19
The programme was perceived to be particularly impactful for boys, who tended to ‘a little bit louder, little bit rowdier’ (Year 5 teacher, School 17) and to ‘flare up really quickly – they lose their temper really quickly . . . it has really benefited them’ (Year 5 teacher, School 26):
They have conversations with each other and especially quite a few boys who get angry very, very quickly and you can see them actually taking themselves away into a corner and thinking to themselves, ‘I am angry, I am angry, but it is alright, what shall I do about it’? In the past they would just smack, so they have got a lot less actually blowing up, you can see I need a strategy.
Year 5 teacher, School 26
Universal delivery, differential gains
Consistent with the view that children do not respond to universal interventions in a uniform manner,93 and the findings of earlier research,92 the quotations in Behaviour problems highlight that respondents believed that PATHS was helpful for some children more than others:
. . . for particular children in the class I think it’s really useful for them and because we do it weekly it reminds them more it’s giving them new ways to calm down or new ways to have a think about their actions.
Year 6 teacher, School 2
In particular, children identified as having SEN were felt to benefit from the programme. For example, one pupil was ‘starting to identify how he feels and what’s going to happen, so if he starts to get upset he will come to me and say “I’m getting upset” and at that point he just goes for a walk and he knows that you know he needs to go for a walk or else he’s going to end up in trouble, that took a long time for him to get to that point but I think PATHS has definitely encouraged that’ (Year 5 teacher, School 17). In addition, teaching an emotional vocabulary was viewed as being particularly useful for pupils speaking EAL: ‘it’s been quite good to look at the emotion side for him and to teach him some of those words as part of his vocabulary so he actually used them’ (Year 3 teacher, School 24).
Pupils with behavioural problems were also felt to have experienced particular benefit from PATHS. One Year 3 teacher (School 1) discussed the case of a pupil with severe social, emotional and behavioural difficulties who had been suspended from his previous school and ‘gets very angry . . . and he’ll just hit out and kick people’. The teacher was able to use the traffic light system with him to calm down, and had taught his mother to use it in the home environment, and ‘it’s been very useful to him . . . occasionally when he forgets and he hurts someone . . . I’ll say to him “well what about the traffic light system”’ and he’ll say “oh yeah, I should have done that, but I can’t, I need to remember quickly”’. A PATHS co-ordinator (School 5) recounted a similar example of a pupil with Attention Deficit Hyperactivity Disorder who was ‘really nasty to children’ but ‘now he’s just taken the time to think’. However, it is worth noting that in these examples, teachers typically referred to specific cases of individual children in their classroom. Despite being extremely helpful in terms of providing illustrative examples, it is noted that these teachers were not necessarily generalising these perceived benefits to all pupils with similar needs.
Finally, PATHS was perceived by several teachers as helping withdrawn, shy pupils, who had ‘developed in confidence’ so that ‘they contribute more’ (Year 4 teacher, School 5). This, in turn, helped them to build relationships and participation within the class: ‘it’s really brought them out of their shell and they answer questions freely they put their hands up, it’s been quite entertaining to watch how well they’re meshing in with the rest of the class’ (Year 5 teacher, School 41).
Wider impact
Our analysis also revealed perceptions of impact beyond the obvious, proximal domains such as social and emotional skills. Thus, many teachers observed increased confidence and self-esteem in their pupils, whereas several other teachers noted improvements in literacy and writing, and general student well-being. In addition, many teachers commented on the impact on their classroom management through the provision of a shared language as well as tools and strategies to manage the classroom and create a calming learning environment.
Language and literacy
The perceived improvements in emotional literacy noted earlier ‘made our story writing a lot better’ (Year 3 teacher, School 26), as ‘you could write how the character’s feeling, which I think before they started PATHS they would have just said happy or sad whereas now they would have used a lot more feeling’ (Year 3 teacher, School 36). Pupils were seen as moving beyond simply describing events to consider how characters were feeling and thinking, and had ‘more of a broad vocabulary’ (Year 3 teacher, School 32):
I think the work on feelings was another aspect that’s been quite good because we’ve used a lot of that in literacy as well in developing writing and one of the threads of learning within literacy has been writing about characters’ feelings in stories. So that’s coming through. The vocabulary has improved so when they talk about how they’re feeling they don’t always say sad or happy, they’ll look for other words, which I think has been really positive about it.
Year 4 teacher, School 10
A boy who was low ability in his literacy . . . was coming into contact with all the different vocabulary, he was using it in PATHS to describe how he was feeling was then thinking any stories how the characters would feel so that worked in his literacy.
Year 5 teacher, School 18
Well-being, confidence and self-esteem
The introduction of PATHS and, in particular, Pupil of the Day (and the associated compliments) was widely well-received by teachers and pupils and was perceived to help pupils become ‘more caring towards each other’ (Year 3 teacher, School 2) and ‘more well-rounded’ (Year 4 teacher, School 25). There were numerous instances of reported increases in ‘confidence’ and ‘self-esteem’ from both teachers (e.g. ‘the compliment Pupil of the Day has made them feel much better about themselves’; Year 6 teacher, School 10) and parents (e.g. ‘they love being Pupil of the Day . . . it really boosts their confidence’; Parent, School 6). Pupils appeared really ‘pleased’ and ‘proud’ to be Pupil of the Day and enjoyed the extra responsibilities and privileges associated with it:
He has grown more confident because the other day when I saw him in the assembly, yes he was speaking out more. He was confident about standing in front of a mic and saying things . . . so that way maybe PATHS is a thing that’s helped him, maybe the school, maybe the teacher, maybe it’s only his age. I don’t know really but yes, he has been growing from Year 3 to Year 4, he has grown more confident. That I can say really. And it’s not only about how he speaks and everything it’s how he writes and all other things, his studies too he’s grown. He can express his feelings more in writing and all other things.
Parent, School 19
Beyond the pupil
Improved learning environment
The implementation of PATHS was associated with improvements to the classroom environment that were considered to be more conducive to learning; teachers reported that it led a more positive ‘atmosphere’ characterised by ‘a calmness’ (Year 6 teacher, School 2) in which ‘[pupils] will listen more . . . they will be more focused’ (Year 3 teacher, School 27). The aforementioned increased consideration and respect for other people’s feelings was seen as contributing to the promotion of ‘inclusivity’, as ‘nobody’s afraid to make a mistake’ (Year 3 teacher, School 24). Specific PATHS lessons that focus on working collaboratively (and in which pupils are assigned different roles that contribute to the achievement of a goal or completion of a task) ‘definitely helped with co-operative learning, when they go off to do groups like that, that’s really helped’ (Year 4 teacher, School 9):
The co-operative group strategy, we use that quite a bit when we do other areas of learning. So it could be in science, getting yourself into a group of four and when they’re doing the investigating making sure that somebody’s taking one of the roles needed. But it’s not something I have to say any more because they will do it. The children when they’re working in a group, say ‘why don’t you be the recorder and why don’t you be this’. So they sort of delegate those roles themselves now so I think that’s been really effective.
Year 4 teacher, School 10
Perceived impact on teachers
Promoting Alternative THInking Strategies was viewed by many teachers as providing a shared language and a set of common tools and strategies to manage the classroom. The structure, degree of prescription and ‘very clear direction’ (Year 4 teacher, School 32) has meant ‘it’s teaching them about their emotions in a different way than I’d ever have done before’ (Year 3 teacher, School 1) and is ‘facilitating the teaching of social emotional skills definitely’ (PATHS co-ordinator, School 17). It has aided the relationship between teaching staff and pupils by supporting the communication between them. Several teachers noted that they are finding much more out about their pupils through the discussions generated in the PATHS lessons, ‘it’s helped me, you know with my relationship with them . . . knowing more about them’ (Year 3 teacher, School 25) and pupils are more willing to disclose information to teachers.
I think they learn to trust you more as well and open up to you and feel like you’re not just a teacher that they can just ask a Maths question to. If they’re feeling a certain way they know they can come and talk to you about it.
Year 3 teacher, School 18
A big asset of the PATHS curriculum is having a set of strategies and approaches to deal with problems and conflicts, and it is something ‘practical’, to ‘refer back’ to or ‘reference’ that can be used outside the PATHS lessons. It has provided teachers with strategies such as the Control Signals to deal with situations and ‘intervene with things a little bit quicker . . . to help them calm down’ (PATHS co-ordinator, School 14) and ‘you can spend half an hour trying to deal with something, and what I’m finding now is if something does happen staff now choose the PATHS stuff and they nip it in the bud really quick and they move on’ (Year 6 teacher, School 18). These strategies have also been beneficial for lunchtime organisers to use at break times outside the classroom too. It has helped teachers and staff talk through issues and discuss what they should do or could have done to resolve a problem:
If we’ve had to deal with something we can often say ‘well what have you done in PATHS about that? You’ve talked about how you can calm down, you’ve talked about if you’re getting angry what you could do. You maybe need to try and adopt one of those strategies’. So it’s just good to have something to refer to instead of children sometimes that are misbehaving find it difficult to talk about why they’ve done it or what they could do instead. Your having like a list of things they could do from PATHS is quite helpful really.
Year 6 teacher, School 17
Sustainability
As the main trial period reached its conclusion, a key focus of our qualitative inquiry was the extent to which schools intended to continue their implementation of PATHS. The overwhelming majority of schools reported that they would ‘stick with it’ (PATHS co-ordinator, School 5) as it was ‘better than what they had in place’ (Year 3 teacher, School 5) and that they ‘can’t see any alternative to it’ (Year 6 teacher, School 41). The importance of SEL was recognised and teachers noted that PATHS mapped across to aspects of the PSHE curriculum. The perceived benefits and impact of PATHS were also noted, with staff reasoning that ‘they’ve [pupils] got a lot out of it’ (Year 3 teacher, School 13) and ‘it’s working [so] there will be no reason why it won’t be carrying on’ (PATHS co-ordinator, School 2). Despite these positive overtures, they also acknowledged that they would need to review their provision and discuss with the headteacher and/or SLT to determine how PATHS could be timetabled and embedded across the whole school:
It will not work if it’s not led from whole school . . . it’s our job as strategic leaders to make sure we decide on the initiatives and with the input but that we sift through almost what is going to be beneficial and what’s useful for the whole school.
PATHS co-ordinator, School 18
In contrast, a small number of schools stated that they would not be continuing implementation beyond the main trial period, citing other priorities within the school and that there were ‘longer term aims and aspirations for next year’ (PATHS co-ordinator, School 18). The continuation of implementation was also seen to be largely dependent on the arrival of new initiatives: ‘there’ll probably be something coming in. They’re talking about changing around the curriculum, there’s quite a lot going on I think at the moment and it’s whether PATHS would get pushed out for something else’ (Year 3 teacher, School 17). This, of course, can be seen as being associated with the issue of shared decision-making [see Shared decision-making (or lack thereof)], in terms of the extent to which teachers felt that they had a voice in strategic decisions made about future provision.
Embedding and extending Promoting Alternative THinking Strategies
Several schools noted that over the 2 years of the main trial, PATHS was gradually becoming embedded within the school, stating that ‘[its] becoming a part of the fabric of the school’ (Year 4 teacher, School 36), as ‘it’s what they [pupils] understand, it’s a common language that they all can express’ (Year 6 teacher, School 7). Given this, the natural next step was to continue and develop PATHS to run throughout the school (bearing in mind that in the main trial it was implemented in classes in KS2 only):
I would certainly push for it and I want it in Key Stage 1 as well now, I want to have the uniformity from Key Stage 1 up to Key Stage 2, so that they’re coming up to you know Key Stage 2 with it.
Year 4 teacher, School 5
As noted previously, PATHS uses a spiral curriculum model, in which the concepts and strategies developed in lessons are revisited and extended in subsequent years. Teachers wanted to capitalise on this more fully through school-wide delivery: ‘we like the fact it can go from Key Stage 1 all the way through and it follows on’ (PATHS co-ordinator, School 10). Indeed, some schools had already taken aspects of PATHS for use in Key Stage 1, in particular Pupil of the Day, and one school was ‘actually in process of ordering through Barnardo’s for Key Stage 1’ (PATHS co-ordinator, School 5). It was also recognised that PATHS principles could become embedded into aspects of wider school policies and practices. In some cases it had ‘already [been] incorporated into the behaviour policy’ (PATHS co-ordinator, School 14) and meant that they were ‘doing a lot of the things from PATHS now as part of our behaviour management’ (PATHS co-ordinator, School 24).
Adaptation to facilitate sustainability
Despite overwhelmingly positive perceptions of impact among teachers, most schools recognised that sustaining PATHS implementation would be challenging, as it was connected intimately to some of the key barriers to delivery noted earlier. Particularly pressing concerns were having the time to fit it into the timetable and making the programme appropriate to local context and needs. This led to staff suggesting a number of future adaptations, effectively reducing the dosage significantly and deviating from the structural fidelity of the curriculum:
I would say not necessarily have to follow lesson by lesson by lesson but if we’ve got the resources then we can definitely, we know if we are covering something in particular or we have got a certain sort of children who are angry and wanting to control it we’ve got the lessons there to help us and support other things that we do.
Year 6 teacher, School 14
However, one school did acknowledge the inherent (potential) dangers of this kind of approach: ‘I think if you’re going to do it you’ve got to try and do it all haven’t you? Otherwise it will become like SEAL where it’s a bit hit and miss’ (PATHS co-ordinator, School 10).
Many schools and teachers remarked that PATHS was ‘good to have there as a framework’ (Year 4 teacher, School 7), noting that ‘if the time wasn’t an issue I don’t really think I would change much, I think it’s think it’s a good structure’ (Year 3 teacher, School 14). With at least 1 year of teaching the PATHS curriculum, they noted feeling more ‘confident’ in how to make best use of the lessons and resources, which largely entailed being, ‘more creative and just putting our stamp on it’ (Year 5 teacher, School 18) or ‘put[ting] a different spin on some of it’ (Year 5 teacher, School 26). Others spoke of plans to ‘make small adaptations and add our bits to it’ (PATHS co-ordinator, School 24), noting how they might ‘pick and choose . . . specific lessons’ (Year 4 teacher, School 18) and ‘may just dip into it for certain things’ (PATHS co-ordinator, School 6). This planned divergence from the core PATHS delivery model was seen as being driven by the increased ‘flexibility’ once the main trial was concluded:
If I’ve got a particular class at the time that have difficulties with certain things I’d think, ‘right this PATHS goes well with that’ . . . and do a lesson so that’s what I might start doing now. And choosing ones which I think that might work really, well. I’m not sure if the structure of PATHS will allow me to do that but I can always change it.
Year 4 teacher, School 18
The above quotations highlight how PATHS might be expected to be implemented in an effectiveness (e.g. ‘real-world’) context as opposed to the efficacy conditions of the current trial, in which significant support and direction was available to optimise implementation. 94 Of particular note is the sense that teachers would ‘pick and choose’ when and how to deliver the programme, implying further deviations in terms of dosage and fidelity.
As noted earlier, implementing PATHS with the recommend frequency was a particularly troublesome issue as schools struggled to find the time – in the context of many competing priorities – to ‘religiously’ deliver it. There were suggestions that ‘maybe you wouldn’t do it every week’ (Year 4 teacher, School 5) and ‘maybe shorten it slightly and do one lesson a week and try and combine lessons that way’ (Year 4 teacher, School 2). Another suggestion was to ‘cut your lesson down by about 50%, in terms of things like PATHS Pupil of the Day, you probably want to do switch it on and switch it off at certain times of the year . . . in terms of supply and demand’ (PATHS co-ordinator, School 17).
Most teachers acknowledged that the strategies and resources (e.g. Pupil of the Day, the Golden Rule, Fingers Linked, Time to Think) rather than the actual lessons had been the most ‘useful’, noting that ‘I don’t think you need to be focused so much on delivering lessons’ (Year 3 teacher, School 5) and that they would ‘use the resources but don’t feel constricted by them’ (Year 3 teacher, School 7):
I’d keep it as a resource and not sort of wade through it step by step, lesson by lesson, which might be the main idea . . . but to me that didn’t work . . . some of it is useful and the main ideas for Pupil of the Day and the Golden Rule to keep but I don’t think the structure of it to go lesson by lesson would survive.
Year 5 teacher, School 1
If I was redesigning it I would be looking for one 20 minute [session] a week instead of the amount there is. I would cut down by half, I would look at the real essentials and I would repetitively hammer them home and I would look at the whole school implementation that maybe needs a much, much more focused and simplified but repetitive message for those members of staff who are dealing with situations when they go wrong. And that usually is lunchtime staff. That’s our Achilles heel as a school anyway.
PATHS co-ordinator, School 17
Chapter 5 Economic analysis
In this chapter, the question of whether or not PATHS provides value for money is turned to.
Objective 6: to examine the cost-effectiveness of Promoting Alternative THinking Strategies
Hypothesis 6: The PATHS curriculum will demonstrate cost-effectiveness.
Costs
Cost analysis is based on only the costs of the PATHS intervention itself, as resource use data (e.g. inpatient hospital stays) were not collected. Two approaches were taken. First, a microcosting approach mirroring previous studies22,33 that examined the costs of SEL interventions was used to cost PATHS in a trial setting. Second, as a sensitivity analysis, the likely costs that would be borne by policy-makers in the event of a roll-out were generated. Costs are reported in GBP and were inflated to 2014/15 prices as required. Costs incurred beyond the first year of the trial were discounted at a rate of 3.5%, in line with the NICE reference case. 70
Microcosting (trial setting)
Intervention costs were covered by a top-up grant from the EEF. 95 Costs of PATHS included those pertaining to (1) materials, (2) training and (3) ongoing support. Materials costs included (1) curriculum packs, (2) the implementation manual and (3) the costs for worksheets and other class materials. Curriculum packs were bought for a fixed fee from Barnardo’s (which holds the UK licence for PATHS) at a discounted bulk purchase unit price. Implementation manual costs were derived from University of Manchester reprographics price lists. Teacher training costs included (1) training materials, (2) teaching and preparation time for PSU staff, (3) travel and accommodation for PSU staff, (4) venue hire and catering, (5) teaching time for PATHS coaches and (6) supply cover costs to release teachers to attend training. Costs pertaining to the training of PATHS coaches included (1) training materials, (2) costs of teaching and preparation time for PSU staff, (3) travel and accommodation for PATHS coaches and (4) PATHS coach time during training.
In the first year of the trial, initial full-day teacher training for teachers of Years 3–5 took place over 2 days at the start of the school year, with a half-day follow-up training taking place over 1 day, 4 months later. All training was conducted by PSU staff alongside PATHS coaches. In the second year, initial full-day teacher training for Year 6 teachers took place over 1 day, with a half-day of follow-up training taking place 4 months later, conducted solely by PATHS coaches. Training took place using the authors’ host university facilities, which were costed at rates for conference hire. Catering was provided and this was costed using expenditure records from the EEF research account.
Approximately 20% of teachers were not able to attend their assigned training sessions and were instead trained solely by PATHS coaches individually within schools. In addition, there was considerable teacher turnover throughout the trial. A total of 25 of the initial 99 teachers left their schools at the end of the first year and replacements for Year 4 and 5 teachers represented ‘new starters’ in year 2 of the trial. The initial and follow-up training for these replacements was conducted alongside that for Year 6 teachers. Furthermore, 12 of the initial 99 teachers left mid-way through the school year. Training for their replacements was carried out by PATHS coaches individually at the schools. The same applies to the 10 of the 99 teachers who left during the second year of the trial. No data are available on the number of teachers leaving at the end of the second year, but this was assumed to be equal to numbers leaving in the first year. PATHS coaches were three doctoral students in the research team (CJ, KP and ES). An apportionment of 2014/15 annual stipends (inclusive of 25% employers on costs) were used to cost teacher training, assuming a total of 1820 working hours per year, consistent with how hourly pay is calculated for University of Manchester research staff.
Supply teacher cover enabled classroom teachers to attend PATHS training. Attempts were made to schedule the training events to coincide with mandated IN-SErvice Training (INSET) days. It was estimated that this was achieved for 50% of schools. The costs of supply teacher cover for half of the teachers trained were therefore apportioned equally across all schools. It was assumed that training for all ‘replacement teachers’ (see previous paragraph) happened outside INSET days and, therefore, that schools needed additional supply teacher cover for these. For the half-day follow-up training conducted outside INSET days, teachers were assumed to be able to return to work for teaching in the other half of the day. Supply teachers were costed identically to classroom teachers, with 1 day of work set at 1/195 of salary, consistent with the standard teaching contract. 96 Teachers’ annual costs were set at the 2014/15 national average salary for a mid-point qualified teacher (M4 on the main pay scale). 97
In the first year of the trial, two PSU staff members conducted training for the first training session and one conducted the follow-up session. Travel and accommodation costs included the costs of return flights and two nights in a hotel for each PSU staff member. These costs were derived from expenditure records. Training for PATHS coaches involved 2 full days of training at PSU. The cost of PATHS coach time was calculated identically to the time spent training teachers. Travel and accommodation costs for the three PATHS coaches consisted of return flights from Manchester to Pennsylvania and three nights of hotel stays, and were derived from EEF account expenditure.
Costs for PSU staff relating to PATHS coach training and teacher training in year 1 of the trial were covered by a fixed fee paid in 2012. The fee was apportioned proportionally based on the time spent on each element of training. Ongoing support costs comprised solely the costs associated with ongoing supervision from PATHS coaches. Overall, one-third of the PATHS coaches’ annual stipend was allocated to their support role. Therefore, after subtracting costs associated with teacher training and PATHS coach training, this was assigned to the costs of ongoing support.
Salary costs of teachers implementing PATHS (e.g. hours spent preparing and delivering lessons) were not treated as incremental, because the programme inevitably displaces other parts of the school curriculum, and teachers’ core working hours are necessarily constant across both arms of the trial. Teaching time costs are therefore incremental only if, in the event of roll-out, the decision-maker funds a section of the curriculum. However, this would not be the case, given that the government (e.g. the DfE) would still fund the whole curriculum. The same is true of preparation and training time, as these hours are also included in the standard teaching contract, and the prescriptive nature of the intervention meant that lesson preparation and planning time required was limited and likely to be similar to that in usual provision. 95 Therefore, teacher salary costs were not included in the base case.
However, the effect of the inclusion of teacher salary costs on results was examined in a sensitivity analysis and costed in line with supply teacher time, but with the standard 25% uplift for employers’ on-costs applied. It was assumed that, in both years of the trial, lessons lasted 30 minutes on average and were delivered twice per week, over a period of 40 weeks. Forty minutes of preparation time (per teacher, per week) was estimated. 95 The robustness of results to assuming 45 minutes of preparation time, consistent with a recently published microcosting of PATHS,33 was examined in a further sensitivity analysis.
The total cost of PATHS was generated by summing across all cost elements. Per-child costs were then calculated by dividing the total cost by the number of children receiving the intervention (n = 2676). As no data on resource use were collected, costs for the usual provision arm were set equal to zero.
Designation of recurrent and non-recurrent costs
Materials costs relating to the PATHS curriculum packs and implementation manual were classed as non-recurrent. Costs relating to the training of PATHS coaches were also classed as non-recurrent, under the assumption that no additional top-up training is required beyond that provided initially. Initial training for the original classroom teachers was also classed as non-recurrent, but teachers’ half-day top-up training was classed as a recurrent cost, with an assumption that top-up training is required every 2 years. The costs of both initial and follow-up training for replacement teachers (for teachers leaving both at the end and mid-way through the school-year) were classed as recurrent, as teacher turnover occurs in all years across all schools. However, this creates difficulty when costing, as training for replacements for Year 4 and 5 teachers who left at the end of year 1 of the trial was combined with training for Year 6 teachers in the group training sessions. Therefore, costs were apportioned equally across teachers attending the group training sessions in the second year of the trial. Fifty teachers were assumed to attend, which included 33 Year 6 teachers and 17 replacements for Year 4 and 5 teachers. The value for the 17 replacements was calculated by apportioning the total of 25 teachers leaving at the end of year 1, under the assumption that the proportion of teachers leaving at the end of year 1 of the trial was equal across year groups.
As non-recurrent costs of classroom materials, PATHS coach training and initial full-day classroom teacher training were likely to provide benefits beyond the trial period (e.g. they could be used after the trial concluded), costs were annuitised over the expected life of each component at a discount rate of 3.5%. 98 The expected life of classroom materials and PATHS coach training was set equal to the expected life of the intervention, assuming an expected intervention life of 5 years, consistent with previous studies. 33 A sensitivity analysis was conducted to assess the robustness of results, assuming a 10-year intervention life. For initial full-day classroom teacher training, the expected life was set equal to the minimum of either the 5-year expected life of the intervention or expected length of service for a teacher. Assuming a constant rate of attrition from teaching, the latter can be estimated as 1/θ, where θ is the rate of teacher attrition. θ was estimated as the average teacher turnover rate across both years of the trial. Although initial full-day training for replacement teachers was classed as a recurrent cost, these costs were also annuitised because of the benefits provided beyond the completion of the trial. These costs were annuitised identically to initial full-day teacher training for original teachers.
Coaching support for PATHS teachers was classed as recurrent, as were the costs of teachers’ time spent preparing and delivering PATHS lessons, and attending follow-up training. The costs of the original teachers’ time spent attending initial full-day training was classed as non-recurrent and annuitised identically to other costs associated with initial teacher training. The cost of the replacement teachers’ time attending training was assumed to be recurrent, but costs of initial training were also annuitised.
Costs in the event of a roll-out
The costs incurred in this trial were not reflective of the costs of PATHS in the event of a nationwide roll-out, as follows: (1) the research team bought intervention materials in bulk and, therefore, received a discount; (2) a large proportion of training was carried out in bulk, with teachers of all schools being trained simultaneously, rather than individually or at school-level, meaning that costs of training sessions could be spread over a greater number of teachers; (3) external trainers for both teachers and PATHS coaches were US based rather than UK based, meaning that costs relating to travel and accommodation were incurred; (4) not all training could be incorporated into INSET days, meaning that supply cover was required; (5) ongoing support was provided by PATHS coaches who were members of the research team; (6) some teacher training was carried out by PATHS coaches within the research team rather than by external trainers; (7) there was poorer than average teacher retention, resulting in higher levels of training for replacement teachers, in addition to the occurrence of mid-year teacher drop-outs; and (8) only a subset of year groups was initially involved in PATHS, such that teachers needed to be initially trained in the second year of the trial.
To address points (1), (2) and (3), unit costs were derived from a menu of prices charged by Birmingham City Council to English schools wanting to implement PATHS. 95 This provides a bundle cost for a single-form entry school with seven year groups (Reception to Year 6) and seven teachers (one per year group). This cost includes (1) a curriculum pack and supplementary materials for each class, (2) initial training for teachers, (3) training of a PATHS coach/co-ordinator (usually a headteacher) and (4) 2 days of PATHS coaching support, assumed to occur in the first year and to be non-recurrent. Unbundled costs for curriculum packs and supplementary material, and full-day and half-day teacher training for seven teachers are also provided by the council. Unbundled costs were subtracted from the cost of the bundle to ascertain the unbundled cost of PATHS coach training and coaching support. It was assumed that curriculum packs and supplementary materials provided by Birmingham City Council were sufficient for delivering PATHS (meaning that an implementation guidance document – as used in the trial – would not be necessary).
To address point (4), it was assumed that in the event of a roll-out, all schools would be able to incorporate PATHS teacher training into their INSET days, such that the costs of supply teacher cover would not be incurred. In relation to point (5), consistent with the Birmingham City Council bundle, it was assumed that headteachers or other members of school staff would be trained as PATHS coaches, rather than external coaches. This training was also assumed to be incorporated into INSET days. As the PATHS coach/co-ordinator is a member of the school staff, there was assumed to be no cost of ongoing support to the school. This is equivalent to an assumption that PATHS coaching forms part of the standard teaching contract. To overcome point (6), it was assumed that in the event of roll-out, follow-up training for original teachers and initial and follow-up training for replacement teachers would be conducted by external trainers and not by in-school PATHS coaches.
In relation to point (7), turnover rates for teachers were assumed to equal the 2015 ‘wastage rate’ of 10.6% (equivalent to the number of teachers leaving as a proportion of the total number of teachers in service). 99 Teachers were assumed to leave at the end of the school year, such that their replacements would be training together at the start of the school year, and that follow-up training would be conducted alongside existing teachers. It was also assumed that all teachers would be able to attend group training sessions, such that no teachers would be trained individually. Finally, in relation to point (8), it was assumed that all year groups in the school would be involved in the implementation of PATHS from the outset, such that all initial training takes place in year 1 of implementation. All other assumptions remained identical to those in the base-case microcosting scenario.
Total costs were divided by seven to get a total cost per class and then multiplied by the number of classes in the PATHS arm of the trial to get a cost equivalent to the size of the PATHS school sample. This cost was then divided by the number of children receiving the PATHS intervention to get an equivalent cost per pupil. The implicit assumption in using these unit costs from Birmingham City Council is that the cost per class for larger, multiform entry schools is equivalent to that for single-form entry schools.
Missing data
No data were missing on the cost data. A total of 51% of children in the trial had partial missing data on CHU-9D utility values (e.g. at least one time point missing). Methods for handling this missing data were adopted from recommendations for handling missing data in within-trial cost–utility analyses (CUAs). 100 Descriptive analysis identified substantial differentials across trial arms, driven by increased missingness in follow-up data for the usual provision group. This was primarily because five of these schools were lost to follow-up. Logistic regression models identified a range of baseline covariates at child (ethnicity, FSM eligibility, SEND) and school (school size) levels as predictors of CHU-9D missingness (thus, data could not be considered completely MAR).
Accordingly, MI methods were employed to handle missing CHU-9D data, under the assumption that data were MAR. Given the low levels of domain-specific missingness for the CHU-9D, QALYs were imputed at the utility value level. Consistent with recommendations, 50 imputations were chosen; this was approximately equal to the percentage of cases with missing data. 101 The ‘ice’ command in Stata 14.0 was used to generate each multiply imputed dataset. Predictors included in the model to impute the CHU-9D at each time point include the CHU-9D utility values at all other time points, and child-level covariates (sex, year group, ethnicity, FSM eligibility, SEND, EAL). A range of school-level covariates (size, percentage FSM, percentage EAL) were included in the prediction model to ensure consistency with the covariates used in the other analyses (see Chapter 3).
Findings
The aim of the analysis undertaken was to estimate the incremental costs and QALYs associated with PATHS compared with those associated with usual provision. Published recommendations102 were followed, including the need to account for both data clustering, and the correlation between costs and outcomes.
Incremental cost-effectiveness ratios, calculated by dividing incremental costs by incremental QALYs, are typically used to present the results of CUAs, but suffer a serious drawback if either element of the ratio is close to zero. As the main impact analyses (hypothesis 1) found that PATHS led to, at best, small improvements in outcomes, small incremental QALY estimates are possible. As a result, incremental cost and QALY estimates were used to calculate INBs as follows:
The p and u subscripts refer to the PATHS and usual provision arms, respectively, and λ denotes the cost-effectiveness threshold. This represents policy-makers’ willingness-to-pay for an additional QALY, and broadly represents the value of the health forgone by displacing existing interventions. PATHS is deemed cost-effective at a given threshold if the INB statistic is positive. In the UK, the threshold is set by NICE at between £20,000 and £30,000. 70 For this analysis, the INBs for a range of thresholds ranging were estimated from £0 to £150,000, increasing in £1000 increments.
Uncertainty associated with INB statistics was examined via non-parametric bootstrapping techniques using 10,000 iterations, combined with the MI methods noted earlier. For each bootstrap replication, a sample of 1000 was randomly drawn with replacements from each of the 50 imputed datasets. Incremental QALYs and costs were re-estimated within each sample and Rubin’s rules were used to combine incremental statistics across samples in each imputed dataset to form average statistics for the bootstrap replication. INBs were then calculated for each value of the threshold within each bootstrap replication, and standard bootstrap methods were used to construct CIs. 103 These were subsequently used to construct CEACs to determine the probability that PATHS is cost-effective at the threshold values noted above.
As noted throughout, a number of sensitivity analyses were conducted to assess the robustness of findings to changes in key assumptions. To summarise, these were (1) using alternative preference weights for constructing CHU-9D utility values, (2) changing the expected life of the intervention over which training and non-recurrent materials costs were annuitised, (3) including costs relating to teachers’ time in delivering the intervention (teaching and preparation) and (4) costing PATHS in a hypothetical scenario similar to that in the event of roll-out.
Tables 20–22 summarise the microcosting for the base-case scenario. Incremental non-recurrent costs for the PATHS materials and training for PATHS coaches and original classroom teachers totalled £35,997 (Table 20). Recurrent costs totalled £44,102 and were primarily associated with the costs of ongoing support from PATHS coaches and follow-up teacher training (Table 21). Overall, the total cost of the intervention was calculated as £80,099, corresponding to a cost per child of £29.93 (Table 22).
| Non-recurrent costs | Total costs (2014/15 £)a |
|---|---|
| Materials costs | |
| Curriculum packs and supplementary materials | 34,431 |
| Implementation guidance | 94 |
| Total materials costs | 34,525 |
| Annuitised for the 2-year trial period | 15,293 |
| PATHS coach training | |
| Training materials and teaching and preparation time for PSU staff | 4519 |
| PATHS coach time while attending training | 462 |
| Travel and accommodation costs | 3120 |
| Total PATHS coach training costs | 8101 |
| Annuitised for the 2-year trial period | 3588 |
| Initial full-day training for original teachers | |
| Training materials and teaching and preparation time for PSU staff for training teachers in year 1 of the trial | 4519 |
| Travel and accommodation costs for PSU staff for the first year of teacher training | 2112 |
| PATHS coach teaching time for initial group training conducted alongside PSU staff in year 1 of the trial | 462 |
| PATHS coach teaching time for initial group training for Year 6 teachers in year 2 of the trial | 147 |
| Venue hire for group training sessions (both years of the trial) | 1583 |
| Catering for group training sessions (both years of the trial) | 843 |
| PATHS coach teaching time for training teachers who could not attend group training sessions (both years of the trial) | 2136 |
| Cost of supply teacher cover (both years of the trial) | 10,269 |
| Total initial full-day training cost | 22,071 |
| Annuitised for the 2-year trial period | 17,116 |
| Total non-recurrent costs | 35,997 |
| Recurrent costs | Total costs (2014/15 £)a |
|---|---|
| Curriculum and teacher support costs | 23,560 |
| Follow-up training for original teachers | |
| Training materials and teaching and preparation time for PSU staff for training teachers in year 1 of the trial | 2259 |
| Travel and accommodation costs for PSU staff for the first year of teacher training | 813 |
| PATHS coach teaching time for follow-up group training conducted alongside PSU staff in year 1 of the trial | 231 |
| PATHS coach teaching time for follow-up group training for Year 6 teachers in year 2 of the trial | 147 |
| Venue hire for group training sessions (both years of the trial) | 983 |
| Catering for group training sessions (both years of the trial) | 541 |
| PATHS coach teaching time for training teachers who could not attend follow-up group training sessions | 1220 |
| Cost of supply teacher cover (both years of the trial) | 5135 |
| Total follow-up training costs for original teachers | 11,329 |
| Training replacements for Years 4 and 5 teachers leaving at the end of the school year (training alongside Year 6 teachers in second year of trial) | |
| Initial full-day training | |
| PATHS coach teaching time for group training session | 76 |
| Venue hire for initial group training session in year 2 of the trial | 197 |
| Catering for initial group training session in year 2 of the trial | 112 |
| PATHS coach teaching time for training teachers who could not attend the initial group training session | 223 |
| Cost of supply teacher cover | 1344 |
| Total initial full-day training cost | 1952 |
| Annuitised for the 2-year trial period | 1514 |
| Follow-up half-day training | |
| PATHS coach teaching time for group training session | 76 |
| Venue hire for initial group training session in year 2 of the trial | 197 |
| Catering for initial group training session in year 2 of the trial | 112 |
| PATHS coach teaching time for training teachers who could not attend the initial group training session | 127 |
| Cost of supply teacher cover | 672 |
| Total initial half-day training cost | 1184 |
| Training replacements of teachers leaving part-way through the school year (both years of trial) | |
| Initial full-day training | |
| PATHS coach teaching time | 1666 |
| Cost of supply teacher cover | 3349 |
| Total initial full-day training cost | 5015 |
| Annuitised for the 2-year trial period | 3889 |
| Follow-up half-day training | |
| PATHS coach teaching time | 952 |
| Cost of supply teacher cover | 1674 |
| Total follow-up half-day training cost | 2626 |
| Total recurrent costs | 44,102 |
| Total and per-child costs | Total costs (2014/15 £)a |
|---|---|
| Total non-recurrent costs | 35,997 |
| Total recurrent costs | 44,102 |
| Total intervention costs | 80,099 |
| Cost per child | 29.93 |
Incremental quality-adjusted life-years
Table 23 presents the average CHU-9D HRQoL scores at baseline and follow-up for children in PATHS and for children in usual provision schools, as well as QALYs gained over the 2-year trial period, after accounting for missing data. In both groups, HRQoL increases over the trial period. On average, children in PATHS schools had higher HRQoL at baseline (T1) than those in usual provision schools. HRQoL values converge at the end of the first year (T2), before diverging at the end of year 2 of the trial (T3). Overall, the PATHS group accumulated more QALYs than the usual provision group over the 2 years of the trial. Adjusting for baseline HRQoL, and child- and school-level covariates in a random intercept model, PATHS was found to lead to a statistically significant improvement in QALYs. Adjusted mean incremental QALYs, derived from 10,000 bootstrap replicates, was 0.0019 (95% CI 0.0009 to 0.0029; p < 0.05).
| INB calculation information | Trial arm | |
|---|---|---|
| PATHS (n = 2676) | Usual provision (n = 2542) | |
| HRQoL | ||
| Baseline (T1) | 0.839 ± 0.003 | 0.827 ± 0.003 |
| 12-month follow-up (T2) | 0.855 ± 0.002 | 0.853 ± 0.003 |
| 24-month follow-up (T3) | 0.872 ± 0.002 | 0.863 ± 0.003 |
| QALYs | 1.711 ± 0.004 | 1.698 ± 0.004 |
| Incremental QALYs | ||
| Unadjusted | 0.013 ± 0.005 | |
| Adjusted | 0.0019 (95% CI 0.0009 to 0.0029) | |
| Costs (£) | 29.93 ± 0 | 0 ± 0 |
| Incremental costs | 29.93 ± 0 | |
| INB (£)(λ = £20,000) | 7.644 (95% CI –12.60 to 27.89) | |
Cost-effectiveness
Evidence regarding the cost-effectiveness of PATHS is provided in Table 24 and in Figures 5–7. Figure 5 presents the cost-effectiveness plan of the PATHS curriculum compared with that of usual provision, and highlights that, although PATHS is more costly, it generates greater increases in QALYs (0.0019, 95% CI 0.0009 to 0.0029). Assuming the minimum cost-effectiveness threshold used by NICE of £20,000 per QALY, the expected INB of introducing the PATHS curriculum is £7.64 (95% CI – £12.60 to £27.89). Using this same threshold, the CEAC presented in Figure 6 indicates that the probability of PATHS resulting in a positive INB and, therefore, being cost-effective, is approximately 88%. At the higher NICE threshold of £30,000 per QALY, the probability that the PATHS curriculum is cost-effective compared with usual provision is approximately 99%.
| Incremental costs | Total costs (2014/15 £) |
|---|---|
| Materials costs | |
| Total materials costs | 34,525 |
| Annuitised for the 2-year trial period: 5-year intervention life | 15,293 |
| Annuitised for the 2-year trial period: 10-year intervention life | 8303 |
| PATHS coach training costs | |
| Total PATHS coach training costs | 8101 |
| Annuitised for the 2-year trial period: 5-year intervention life | 3588 |
| Annuitised for the 2-year trial period: 10-year intervention life | 1948 |
| Other intervention costs | 61,217 |
| Total intervention costs (5-year intervention life) | 80,099 |
| Total intervention costs (10-year intervention life) | 71,468 |
| Cost per child (5-year intervention life) | 29.93 |
| Cost per child (10-year intervention life) | 26.71 |
FIGURE 5.
Cost-effectiveness plane of PATHS vs. usual provision.
FIGURE 6.
Cost-effectiveness acceptability curves of the PATHS curriculum vs. usual provision: base-case and HRQoL-related sensitivity analysis.
FIGURE 7.
Cost-effectiveness acceptability curves of the PATHS curriculum vs. usual provision: base-case and cost-related sensitivity analysis.
Sensitivity analyses
Figures 6 and 7 also examine the robustness of results to changing key assumptions in the analysis. Figure 6 shows the CEACs generated when changing the algorithm using to construct HRQoL utility scores and, therefore, QALYs. Compared with the algorithm derived from a sample of UK adults using standard gamble methodology, the strength of the effect of PATHS on QALYs was found to be stronger when the algorithm derived using best–worst scaling and an Australian adolescent sample104 was used, with mean incremental QALYs increasing from 0.0019 (95% CI 0.0009 to 0.0029) to 0.0028 (95% CI 0.0015 to 0.00040). This resulted in a higher probability of cost-effectiveness at all threshold values, and an approximate 99.6% probability of PATHS producing a positive INB at the £20,000 per QALY threshold. This is unsurprising given that the Australian algorithm places higher weights on mental health domains when constructing the overall utility index.
Figure 7 demonstrates the sensitivity of cost-effectiveness conclusions to the assumptions made when costing the intervention. Assuming a 10-year rather than a 5-year expected intervention life when annuitising non-recurrent costs led to a small reduction in incremental costs of £3.22 per child (see Table 24), increasing the probability of PATHS being cost-effective at the £20,000 per QALY threshold to 93.5% from 88% in the base case.
However, the inclusion of teacher salary costs, irrespective of the assumption made regarding the required lesson preparation time for PATHS lessons, had a major impact on both incremental costs and INB statistics. Incremental costs increased to more than £185 per child (Table 25) and resulted in a 0% probability of PATHS being cost-effective at either NICE QALY threshold.
| Incremental costs | Costs (2014/15 £) |
|---|---|
| Non-recurrent costs | |
| Original classroom teacher time during initial full-day training | |
| Teachers’ time | 25,287 |
| Annuitised for the 2-year trial period | 19,609 |
| Recurrent costs | |
| Original classroom teacher time during follow-up full-day training | |
| Teachers’ time | 15,601 |
| Annuitised for the 2-year trial period | 12,099 |
| Replacement classroom teacher time during initial full-day training and follow-up half-day training | |
| Teachers’ time | 8880 |
| Annuitised for the 2-year trial period | 6886 |
| Classroom teacher time during lesson delivery | |
| Teachers’ time | 232,025 |
| Classroom teacher time during lesson preparation | |
| Teachers’ time: 40 minutes/week | 154,684 |
| Teachers’ time: 45 minutes/week | 174,019 |
| 40 minutes/week lesson preparation | 425,303 |
| 45 minutes/week lesson preparation | 444,639 |
| Total other intervention costs | 80,099 |
| 40 minutes/week lesson preparation | 505,402 |
| 45 minutes/week lesson preparation | 524,738 |
| Cost per child (40 minutes/week lesson preparation) | 188.86 |
| Cost per child (40 minutes/week lesson preparation) | 196.09 |
Finally, given evidence that the costs in the trial were unlikely to represent accurately those borne if PATHS was to be introduced to schools nationwide, an attempt was made to cost a roll-out scenario. In this costing, the per-child cost of the intervention was estimated to reduce from the base case cost of £29.93 to £17.78 (Table 26). Thus, the additional costs attributable to an inability to purchase training and curriculum materials in bulk were outweighed by cost savings made from avoiding supply teacher costs and internalising ongoing support. At this cost, the probability of PATHS being cost-effective was > 99% at both conventional thresholds.
| Incremental costs | Total cost (2014/15 £)a |
|---|---|
| Costs for a single-form entry, seven year group school | |
| Non-recurrent costs | |
| Initial full-day training for original classroom teachers | 500 |
| Curriculum packs and supplementary materials | 2100 |
| PATHS coach training and initial support | 3932 |
| Total non-recurrent costs | 6532 |
| Annuitised for the 2-year trial period | 2893 |
| Recurrent costs | |
| Half-day follow-up training | 250 |
| Initial full-day training for replacement teachers, annuitised for the 2-year trial period | 221 |
| Total recurrent costs | 471 |
| Total cost for single-form entry, seven year group, school | 3365 |
| Price per class | 480.70 |
| Total intervention cost for PATHS sample (99 classes) | 47,589 |
| Cost per child | 17.78 |
Chapter 6 Discussion
Principal findings
The primary aim of this study was to examine the impact of the PATHS curriculum on the social and emotional well-being of children in primary schools in England, with the following objectives: (1) to determine the impact of PATHS on a variety of outcomes for children, (2) to determine whether or not the impact of PATHS is sustainable, (3) to determine the impact of PATHS on children’s psychosocial adjustment to secondary school, (4) to assess the role of implementation variability in moderating the impact of PATHS on outcomes for children, (5) to assess the validity of the logic model for SEL programmes, and (6) to examine the cost-effectiveness of PATHS.
In relation to objective 1, limited and very modest evidence of the impact of PATHS in our ITT analyses was found. HLM of outcome data indicated that the intervention produced marginal, non-significant improvements in children’s social skills (hypothesis 1a: primary outcome) and perceptions of peer and social support (hypothesis 1f), in addition to reducing the total number of sessions from which they were excluded from school (hypothesis 1h). In addition, PATHS led to a statistically significant improvement in children’s psychological well-being (hypothesis 1e). Associated intervention effect sizes were very small (d < 0.2 in all cases) and were, therefore, likely to be deemed low in terms of their ‘practical significance’ (e.g. U3 < 6% in all cases). 105 There were no significant differences between children in PATHS and usual provision schools in respect of their pro-social behaviour (hypothesis 1b), internalising symptoms (hypothesis 1c), externalising problems (hypothesis 1d), school environment (hypothesis 1g), exclusions (hypothesis 1h), attendance (hypothesis 1i) or academic attainment (hypothesis 1j) at the end of the main trial period.
Our analyses pertaining to objective 2 focused on determining the extent to which the limited impacts noted above were sustained at 24-month post-intervention follow-up, and also whether or not there was any evidence of so-called ‘sleeper’ effects on other trial outcomes. Using outcome data from the subsample (n = 1631) of children who transitioned to secondary school at the end of the main trial, these analyses revealed no significant differences between children from PATHS or usual provision schools for any outcome specified in hypotheses 2a–g. Fulfilment of objective 3 also made use of the abovementioned subsample, with analyses of their psychosocial adjustment (psychological well-being, hypothesis 3a; perceptions of peer and social support, hypothesis 3b; school environment, hypothesis 3c) to secondary school at 12-month post-intervention follow-up, revealing no significant differences between children from PATHS or usual provision schools for any outcome.
With regard to objective 4, the study sought to assess the role of implementation variability in moderating the impact of PATHS on outcomes for children. Findings were mixed. In the first year of the trial, our analyses using data from observations of classes in Years 3 and 4 demonstrated that, compared with low levels, moderate (and, marginally, high) levels of implementation quality and participant responsiveness (hypothesis 4c/d) were associated with significant improvements in children’s psychological well-being. A marginal, non-significant trend was also identified, whereby compared with low levels, high levels of implementation quality and participant responsiveness were associated with reductions in externalising problems. Contrary to expectations, compared with low levels, moderate and high levels of dosage (hypothesis 4b) were associated with significant reductions in children’s social skills. Similarly, compared with low levels, high levels of dosage were also associated with significant reductions in pro-social behaviour and psychological well-being. Finally, and again contrary to expectations, a marginal non-significant trend was identified whereby compared with low levels, high levels of reach (hypothesis 4e) were associated with reductions in pro-social behaviour.
In the second year of the trial, our analyses using observational data from classes in Years 5 and 6 demonstrated that, compared with low levels, moderate levels of dosage (hypothesis 4b) were associated with significant improvements in children’s perceptions of social support and peers. However, high levels of dosage were also associated with significant increases in internalising symptoms. Compared with low levels, high levels of reach (hypothesis 4e) were associated with significant reductions in both internalising symptoms and externalising problems (with a marginal effect found in relation to moderate dosage for the latter outcome), but also (marginally) with reductions in children’s perceptions of social support and peers. Throughout both years of the trial, our analyses revealed no associations between levels of implementation fidelity (hypothesis 4a) and any intervention outcomes.
Our analysis of the temporal relations between outcomes (objective 5) using data from the transition subsample indicated that our adapted SEL logic model provided an acceptable fit to the data. Baseline and proximal variables explained 32% of the variance in academic attainment at the end of the main trial. However, only social skills and mental health difficulties were found to be statistically significant predictors of academic attainment. Of particular interest was the identification of a significant mediation pathway, by which baseline social skills predicted academic attainment indirectly via their protective effects on proximal mental health difficulties.
In relation to objective 6, the mean incremental cost of PATHS compared with usual provision was £29.93 per child. Mean incremental QALYs were positive and statistically significant, but very small in magnitude. Assuming the minimum threshold recommended by NICE of £20,000 per QALY, the INB of introducing PATHS was determined to be £7.64; the probability of the programme being cost-effective in this base-case scenario is approximately 88%. These findings were found to be sensitive to changing key assumptions used the analysis (e.g. using different preference weights for the CHU-9D, costing the intervention in a hypothetical roll-out scenario); however, in most cases, the probability of PATHS being cost-effective increased. The only scenario in which PATHS was determined to be unlikely to be cost-effective was that in which the costs of PATHS lesson teaching and preparation time were included.
Limited and modest impact of Promoting Alternative THinking Strategies
The study found a marginal, non-significant trend relating to the impact of PATHS on our primary outcome of social skills. In addition, the study found a significant effect on our secondary outcome of psychological well-being and further marginal, non-significant trends on our secondary outcomes of perceptions of peer and social support, and exclusions. The effects identified were very small in all cases. No significant effects of PATHS were identified for any other outcome. Despite this limited impact, the intervention effects found were somewhat consistent with the PATHS logic model (see Figure 2) and the principal foci of the programme materials. PATHS is designed to promote SEL and help children to work well with others. Improved psychological well-being can be seen as a consequence of improvements in these domains when viewed through the lens of resilience theory. Social skills and peer and social support can be considered key protective factors that support psychological well-being,106 and, indeed, this proposition is borne out to a certain degree in other analyses (e.g. the study found a protective effect of social skills on later mental health difficulties in our analysis pertaining to objective 5).
It should be noted that the effects identified above were almost exclusive to our child self-report surveys; there was no evidence of the impact of PATHS from our teacher informant-report surveys and very little in relation to the educational outcome data drawn from the NPD (with the marginal, non-significant effect on exclusions being the only exception). One possibility that this raises is that the effects identified pertain primarily to perception of self (i.e. PATHS increased children’s awareness of their own intra- and inter-personal qualities, and this was then reflected in their self-ratings) rather than actual changes in behaviour that might be directly observed by a teacher (or, indeed, educational outcomes that are then formally recorded). This does not denigrate the validity or utility of self-report methods of assessment but rather reinforces the need for a multi-informant, multioutcome approach in trials of this kind so that different frames of reference can be considered in tandem. 47
There are a number of potential reasons for the limited impact of PATHS in the current study. One possibility is that the ITT approach to analysis reported herein masked gains made by particular subgroups, and, indeed, some evidence of this emerged in the qualitative IPE analysis reported in Chapter 4. This raises an important question that is at the heart of the current debate around the ‘what works’ agenda in education, namely what works for whom?93 Although ITT analyses provide the most unbiased estimate of impact, it should not be expected that individuals participating in interventions will respond in a uniform manner. 107 The ITT approach may, therefore, underestimate impact by failing to appreciate the natural heterogeneity in universal populations. 108 As noted in Chapter 1, evidence of differential gains in PATHS has been observed in some other trials, albeit with little consistency in terms of which subgroups of children and young people benefit more than others. An important avenue for further analysis of the study data set is therefore to examine whether or not the potential differential gains noted in the qualitative IPE are evidenced in the trial outcome data, although with the obvious caveat that because these were not specified a priori, there is a risk of perceived bias (e.g. ‘fishing’ for positive findings).
Another potential explanation for the limited and modest impact of PATHS observed in our trial relates to usual school provision for SEL. Recall that all schools reported that they were already implementing a wide range of universal (e.g. SEAL) and targeted (e.g. Nurture Groups) SEL and SEL-related interventions, in addition to delivering PSHE as part of the standard school curriculum, when they were surveyed at baseline. In the schools allocated to the intervention arm of the trial, it is therefore possible that PATHS displaced other, equally efficacious, approaches, thus limiting the measurable impact of the programme. If delivered at the developer’s recommended frequency, PATHS lessons and associated activities could take up to 100 minutes per week (e.g. two 30- to 40-minute lessons, plus procedures such as PATHS Pupil of the Day, and any ad hoc generalisation activities instigated by the class teacher) of the school timetable. Of course, the actual implementation observed in the trial took significantly less time than this (see Chapter 4). Nonetheless, the possibility remains that what schools stopped doing in order to start doing PATHS had a bearing on our findings. Data from our IPE indicated that the usual school provision most likely to be displaced by PATHS was either PSHE and/or SEAL. Unfortunately, there is currently limited evidence regarding the extent to which either of these initiatives influence the outcomes assessed in the current trial. Thus, although a recent government review noted that PSHE can improve the ‘psychosocial well-being’ of pupils,109 the evidence underpinning this conclusion is, ironically, drawn from the SEL literature (e.g. the Sklad et al. 4 meta-analysis) rather than from a formal evaluation of the PSHE curriculum. A complicating issue here is that PSHE provision remains extremely varied, at least in part because it remains non-statutory. By contrast, the various components of the SEAL programme have been subjected to formal evaluation, but methodological limitations of these evaluations, and the mixed findings they produced, limit the conclusions that can be drawn regarding the programme’s overall efficacy. 37
One additional possibility is that PATHS, as implemented in the context of the current trial, lacked the intensity to alter developmental pathways of those children who stood to benefit the most from intervention. 17 Put another way, it is perhaps unsurprising that a relatively limited amount of teaching and associated activities delivered as part of an overstretched curriculum (as indicated in our IPE) did not produce improvements of a more noteworthy magnitude. To produce meaningful change in children’s social and emotional skills and mental health, particularly in the context of an education system that does not explicitly prioritise such outcomes, the relationship between teachers and their pupils is crucial. 110 Such relationships cannot be expected to flourish through a primarily curricular approach, particularly when, despite the initial training and ongoing support provided in PATHS, teachers may have limited understanding of emotional development and the change process. The increasing interest in the role of teacher characteristics (e.g. professional, psychological) in the implementation process58,111 is therefore to be welcomed. Finally, it should also be noted that the school context is just one of a number of intersecting microsystems (e.g. family, peer group, community) that has an influence on children’s developmental trajectories. 112 Although some intervention theory reflects this (e.g. the Domtrovich et al. 113 integrated model of school-based prevention), intervention design and research arguably lags behind. One obvious example from the current trial is the rather peripheral nature of the parent materials in PATHS.
The limited impact of PATHS observed in this trial aligns more closely with the main findings of the only other RCT of the programme conducted in England to date than with the broader international evidence base. Berry et al. 33 found null results in their ITT analyses of PATHS as implemented in KS1 (ages 4–6 years) in Birmingham schools. Interestingly, this study also reported similar findings to the current study vis-à-vis average levels of implementation; thus, as reported here, teachers in the Birmingham trial delivered only around half of the possible PATHS lessons each year, but did so with high levels of procedural fidelity. Although the argument could be made that the levels of implementation achieved in both trials were insufficient to trigger measurable change in our study outcomes (e.g. implementation failure), our findings are nonetheless likely to represent the ‘best case scenario’ in terms of both implementation and outcomes in the event that PATHS is disseminated more widely in English schools. In support of this, this trial contained a number of features designed to optimise implementation that would not necessarily be present if PATHS were to be delivered in ‘real-world’ conditions. First, as the intervention costs were covered by a supplementary grant from the EEF, schools were not required to pay for the PATHS materials or coaching support/visits. Second, initial training for schools in the intervention arm was led by experienced staff from the developer’s team at PSU. Third, the research team provided a written guide to support implementation and also undertook a range of activities designed to reduce teachers’ preparation burden. For example, in the PATHS packs provided by Barnardo’s, only one set of ‘Feelings Face cards’ is provided, despite the need for one per child in each participating class. Thus, the research team prepared and distributed sufficient copies to each intervention school prior to the start of implementation. With these supportive aspects not likely to be in place under typical circumstances, it might be reasonable to expect the level of implementation to drop and the magnitude of intervention effects to reduce accordingly. In support of this proposition, the recent SEL meta-analysis by Wigelsworth et al. 5 noted significantly lower effect sizes for several outcome domains in effectiveness trials than for those found when analysis was carried out under efficacy conditions.
Lack of maintenance or sleeper effects at the 24-month follow-up
Using data from the subsample of young people who transferred to secondary school at the end of the main trial, the study was able to test whether or not the limited impacts identified in the analyses discussed were still evident at the 24-month follow-up. In addition, the study was able to examine whether or not there were sleeper effects for any of the outcomes that yielded null results immediately post intervention. In both cases, our analyses revealed no significant differences between children from PATHS or usual provision schools on any outcome measure pertaining to hypotheses 2 (a–f).
As noted in Chapter 1, there has been a general paucity of analyses of follow-up effects in SEL trials. However, of the studies that collected follow-up data at least 6 months after a given intervention ended, the Durlak et al. 3 meta-analysis demonstrated that intervention effects remained evident across the outcomes studied, albeit with significant attenuation in effect sizes compared with those observed immediately post intervention. An updated meta-analysis by the same group,114 published shortly before this report was originally submitted, focused specifically on follow-up effects of SEL interventions and affirmed this pattern of findings. In terms of PATHS specifically, a follow-up study115 of children with special educational needs who formed part of the original sample of the first RCT of the programme24 indicated that it reduced the growth rate of teacher-reported internalising symptoms and externalising problems at 24-month post-intervention follow-up and also produced a sustained reduction in children’s self-reported depressive symptoms. More recently, the Swiss trial of PATHS identified sleeper effects for children’s externalising problems after the same post-intervention follow-up period. 30
Thus, the findings of the current study with regard to follow-up effects appear to be somewhat discordant with both the general SEL literature and the available evidence pertaining directly to PATHS. There are, however, a number of plausible explanations. First and foremost, caution is required in relation to the interpretation of our follow-up analyses. The design of the trial meant that these would be based on only a subgroup of the original trial sample (see previous paragraph) and, as such, may produce biased estimates, which are in turn compounded by the significant attrition experienced at T5 (at which point data were available only for 28% (N = 463) of the subgroup (n = 1631). Although MI procedures were employed as a matter of course in order to address the missing data problem, the study was conscious that the scale of the loss to follow-up for these particular analyses may have affected their security. Thus, although MI has been found to produce robust estimates (comparable to those produced when little or no data are missing) even with relatively large numbers of missing data (e.g. 60%),65 our findings in relation to this hypothesis need to be treated with caution.
This analytical issue notwithstanding, the lack of evidence for maintenance of effects may simply reflect the fact that of those identified immediately post intervention, all were very small in effect size magnitude and, indeed, only the impact on psychological well-being was actually statistically significant at the p < 0.05 level. Thus, one can easily envisage how these rather marginal gains could attenuate to the point at which they are no longer detectable 2 years later, particularly when one considers the changes experienced by the transition subsample during this period and the academic and social challenges associated with moving from primary to secondary school,116 an issue that is now turned to.
No impact on children’s psychosocial adjustment to secondary school
To the best of our knowledge, this is the first RCT to explicitly test whether or not a SEL programme can have an impact on children’s psychosocial adjustment (psychological well-being, perceptions of peer and social support, and perceptions of the school environment) to secondary school. The closest reference points in the existing literature are Rosenblatt and Elias’117 evaluation of the impact of the ‘Talking with TJ’ intervention on academic achievement in the transition from elementary to middle school, and the Qualter et al. 118 study of the effects of an emotional intelligence intervention in the transition from primary to secondary school, although the exploratory nature of both of these projects (e.g. small scale, no comparison group in the Rosenblatt and Elias study117) precluded firm conclusions from being drawn. However, in spite of the lack of direct empirical evidence, the proposition that SEL interventions could support children’s adjustment to secondary school is theoretically plausible. Previous research has shown that successful navigation of the key challenges associated with this important transition (e.g. moving from the protected top of the social hierarchy in primary school to the bottom of a more complex social hierarchy in secondary school) has been shown to vary as a function of children’s social and emotional competence. 118 Given this, the lack of evidence for the impact of PATHS on children’s psychosocial adjustment to secondary school may appear to be incongruous. However, recall that the immediate effect of the programme on children’s social skills was meagre (d = 0.09) and indicative of only a marginal, non-significant trend (p = 0.08). Thus, it is speculated that the null results found in relation to adjustment to secondary school were perhaps inevitable, given the very modest evidence of impact at the end of the main trial. Furthermore, the study was mindful of the potential impact of attrition on the security of our analyses, although the scale of this issue was considerably less in relation to 12-month post-intervention follow-up [e.g. data available for 49% (N = 793) of the subgroup (n = 1631)].
Relationship between levels of implementation and intervention outcomes
Our starting point in considering the relationship between implementation and outcomes is to assess the average levels of implementation recorded in the study. Recall that a general trend was seen in which PATHS was delivered once a week, with most children in a given class present, teachers adhering to most procedural elements outlined in lesson materials and delivering them well, and with children responding appropriately. However, there was a clear downward shift in dosage between the first and second years of the trial, such that in the former, teachers were estimated to deliver 65% of lessons, but in the latter were estimated to deliver 39% of lessons. This contrasts somewhat with other studies of PATHS, in which dosage was found to be stable over time. 25,56,92 Our qualitative analysis is extremely helpful in explaining this trend and why it diverges from previous findings. In brief, implementing PATHS at the recommended dosage became increasingly challenging for teachers in the second year of the trial, particularly those with Year 6 classes, in which pupils were being prepared for their end of KS2 examinations (e.g. ‘Mornings of our school were timetabled in rigidly with Maths, English and reading. Like every morning is completely out. So it’s just the constraints of the school timetable in some cases that doesn’t make it easy to fit it in’, Year 6 teacher, School 7). There is also pertinent evidence from the broader literature. For example, the Moore et al. 86 study of adaptations of evidence-based programmes in natural contexts found that more than 40% of implementers adapted dosage, and that nearly two-thirds of all adaptations were made for logistical reasons such as those noted previously.
Thus, there is a convergence of evidence that suggests that the expectation of delivering PATHS (or, indeed, similar universal SEL interventions) twice a week is incongruent with what is realistically possible given the issues noted here. This begs the question of whether or not a future, more ‘streamlined’ iteration of the curriculum containing only those lessons considered to be absolutely core to the achievement of the intended outcomes would be beneficial in terms of fit to the English school context. However, such revisions are perhaps incompatible with the spiral curriculum model used in PATHS (whereby topics and content are revisited to embed learning), and, in any case, would need to be underpinned by the kind of critical component analysis that is not yet evident in implementation science. 38 Furthermore, an important counterpoint is that the challenges experienced in delivering PATHS lessons at the recommended frequency perhaps reflect a lack of prioritisation of SEL in participating schools. After all, some teachers (e.g. those in the ‘high dosage’ group in our implementation–outcomes analyses) were able to implement lessons at or near the recommended frequency, but faced no less pressure in terms of ensuring that the core academic curriculum was also delivered. Thus, in our further analyses of the qualitative IPE data, it may be fruitful to explore the enabling factors specifically for this group.
Our analyses of the relationship between levels of implementation and intervention outcomes produced mixed findings. To a certain extent, this is not surprising; as Durlak119 notes, ‘we should not assume that each component [of implementation] is equally important for all possible outcomes’. The study was also mindful that research on implementation–outcomes relationships has been rather narrowly focused on fidelity and dosage to date. Much less is known about the potential moderating effects of dimensions such as quality and participant responsiveness59 and so this is very much new territory.
Starting with fidelity, the lack of association with any intervention outcomes across both years of the trial is significant both in terms of its implications for implementation science generally and PATHS more specifically. The ‘zero-sum-game’ view of implementation has predominated in recent years: higher fidelity results in better outcomes, and any deviation from the intended intervention model must therefore be detrimental. 120 Building on conceptual and theoretical objections to this perspective,121 the analyses demonstrated empirically that variability in procedural fidelity appears to be unrelated to intervention outcomes in the PATHS curriculum (findings that align with those of Berry et al. 33 and the Social and Character Development Research Consortium27 in their trials of PATHS, although it is important to note that these earlier studies did not take account of the distinction between fidelity and quality as was made here). Furthermore, our qualitative IPE analysis demonstrated that teachers called upon their knowledge of their students’ characteristics and needs, and general pedagogical knowledge, to make a range of adaptations and modifications to PATHS. Taken as a whole, these findings suggest that a shift in emphasis is perhaps needed in future evaluations, whereby instead of focusing simply on programme requirements (e.g. fidelity to a prescribed delivery model), greater attention is paid to how, why and when teachers (inevitably) make modifications to interventions. Following on from this, examination of the associations between different type of adaptations and intervention outcomes would be a logical next step.
In relation to dosage, contrary to initial expectations, higher levels were associated with significantly worse intervention outcomes in several domains (e.g. social skills, pro-social behaviour, psychological well-being, internalising symptoms). Indeed, only one dose–outcome association (in relation to perceptions of peer and social support) was found in the expected direction. These findings contrast with the only other studies of PATHS that have assessed the relationship between dosage and outcomes,28,92 both of which found evidence of a positive association. The dosage levels reported here (see Table 14) are comparable with those of these previous studies, so the divergent findings are unlikely to be the result of failure to reach the threshold of a ‘minimum effective dose’122 in the current study. Instead, methodological and other differences may account for the differences in our findings. For example, Schonfeld et al. 28 found that the probability of achieving academic proficiency status in reading increased 1.37 times for each additional PATHS lesson taught, but used different methods to assess implementation (teacher self-report), covered a much longer period of implementation (up to 4 years) and assessed different outcomes (academic proficiency status modelled as a nominal outcome) from the current trial.
However, the question still remains as to why higher levels of dosage were found to be associated with significantly worse intervention outcomes for so many outcome variables. One possibility is that the teachers who implemented PATHS more frequently did so because they had a lower functioning class; thus, higher dosage could reflect higher perceived need. In support of this proposition, our study data indicated that children in the moderate- and high-dosage classrooms in the first year of the trial demonstrated marginally higher levels of need in respect to certain outcomes (specifically internalising symptoms, externalising problems and lower pro-social behaviour) at baseline. An alternative view is that high dosage was at the expense of quality (e.g. a ‘race to the finish line’ mentality may have led to some teachers to focus on delivering the complete PATHS curriculum, but in so doing, neglect to engage and enthuse their pupils fully during lessons). Finally, it may simply be the case that more frequent delivery of PATHS lessons meant that other efficacious activities (including, potentially, targeted interventions) were displaced.
Higher levels of implementation quality and participant responsiveness were found to be associated with increased psychological well-being and (marginally) reduced externalising problems. These analyses support Durlak’s119 claim that ‘in some circumstances, quality of delivery . . . may be more strongly related than other implementation components to some program benefits’ and also add to a small but growing evidence base on the importance of this dimension of implementation as a moderator of intervention outcomes. In particular, our findings resonate with those of Pettrigrew et al. ,123 whose equivalent analyses in their study of the keepin’ it REAL Program revealed that implementation quality and participant responsiveness predicted intervention outcomes more reliably than fidelity. This emergent pattern of findings suggests that the focus of implementation support processes could be broadened to incorporate more explicitly quality and responsiveness processes. This may, however, prove to be challenging for manualised interventions that perhaps lend themselves to a more procedural emphasis. PATHS lessons are presented in a scripted format, and this may be taken to mean that they should be delivered ‘word for word’. Our analysis suggests that this need not be the case.
Understanding Promoting Alternative THinking Strategies implementation processes
Our qualitative IPE data provided a rich explanatory narrative that enabled us to illuminate and build upon our trial findings. In particular, these data highlighted the importance of context in understanding variability in implementation, including perceptions of pupil need, each school’s history of SEL provision and the extent to which school leaders actively involved staff in the decision to join the trial. Internal and external drivers pertaining to competing priorities and curriculum demands significantly influenced the extent to which PATHS was delivered as planned, most notably with regard to dosage.
Our qualitative data largely converged with our quantitative findings with regard to how PATHS was implemented. Thus, it was seen that PATHS was delivered to most children in a given class, with teachers adhering to key procedural elements outlined in curriculum materials and delivering lessons well, and with children responding appropriately. Explicit adaptations were made by staff based on their knowledge of local context and need, but these were generally rather minor surface changes that did not affect the substantive content of the intervention (i.e. teachers strived to ‘keep the underlying principles of each lesson’, Year 3 teacher, School 1). The drop in dosage noted in our structured observational data was clearly reflected in the qualitative analysis, which demonstrated that despite clear initial intentions among teachers to deliver two lessons per week, this became increasingly unfeasible over the course of the 2-year implementation period.
One area of clear divergence between the qualitative and quantitative findings was in relation to the question of impact. Recall that our qualitative analysis highlighted perceived improvements across a number of domains. This of course contrasts with the rather narrow and modest gains evidenced in our quantitative outcome analyses. In moving beyond the obvious explanations relating to social desirability and the anecdotal nature of the qualitative evidence proffered by school staff, Matthews et al. 124 are referred to, who argue that the benefits of attempts to improve SEL ‘appear to reside mainly in raising awareness of emotional issues and motivating educators and managers to take emotional issues seriously’. Thus, the effusive school staff testimonials regarding the impact of PATHS provide evidence of staff members’ increased attunement to and consideration for children’s emotional states.
Longitudinal relations between social skills, mental health difficulties and academic attainment
The data collected in fulfilment of the other objectives in the current trial afforded us the opportunity to perform a secondary analysis in which the veracity of the general SEL logic model was assessed. 75 As a secondary analysis, there were immediate compromises that had to be made. Chief among these were the fact that that our conceptual model represented an approximation based on the data available through the trial and that, in terms of the temporal aspect of the analysis, we were similarly limited by the overall study design. Finally, the requirement of having academic attainment as the distal variable in the model restricted us to using our ‘transition subsample’ as opposed to the whole trial sample (given that end of KS2 academic attainment data were available only for the former). With those caveats in mind, our adapted SEL logic model provided an acceptable fit to the data. Baseline and proximal variables explained 32% of the variance in academic attainment at the end of the main trial. However, only social skills and mental health difficulties were found to be statistically significant predictors of academic attainment. Of particular interest was the identification of a significant mediation pathway, by which baseline social skills predicted academic attainment indirectly via their protective effects on mental health difficulties. This finding aligns with previous work in this area, which has established a clear inverse association between children’s social skills and their mental health difficulties125 and has also demonstrated the erosive effect of mental health difficulties on later academic attainment. 126 However, it is noteworthy that other pathways in our longitudinal model did not provide support for the SEL logic model. For example, there was no evidence that children’s perception of the school environment predicted their later academic attainment. However, given the necessarily exploratory nature of this analysis and the range of caveats noted above, it is suggested that this indicates the need for further analyses of this type, rather than an outright rejection of the SEL logic model, and indeed, this is highlighted as a priority area for future research in the field. 2,3
Promoting Alternative THinking Strategies is likely to provide value for money
Although much has been written about the potential economic value of SEL,127 our economic analysis represents the first trial-based cost-effectiveness evaluation of a universal SEL intervention. This is a critical step change for the field, which has previously relied on estimates and simulations as opposed to direct measurement of QALYs. 22,128 Work to date has established a microcosting methodology for SEL interventions129 and this has been applied to PATHS. 33 Building on this, our microcosting demonstrated that the mean incremental cost of the programme compared with usual provision was £29.93 per child. This contrasts sharply with the £127 cost per child in the previously reported microcosting,33 although it is important to note that this included the costs associated with teacher time to prepare and deliver PATHS (e.g. as in our sensitivity analysis, see Table 25). Recall, however, that our base-case scenario omitted these costs for a variety of reasons (see Chapter 5), and so a direct comparison is perhaps inadvisable.
Our analysis demonstrated that the mean incremental QALYs associated with delivering PATHS as opposed to usual provision were positive and statistically significant, but very small in magnitude. Assuming the minimum threshold recommended by NICE of £20,000 per QALY, the INB of introducing PATHS was determined to be £7.64. The probability of the programme being cost-effective in this base-case scenario was determined to be approximately 88%. In two sensitivity analyses, one of which used different preference weights for the CHU-9D and the other of which costed for a hypothetical roll-out scenario, this probability increased to ≥ 99%. The only scenario in which PATHS was determined to be unlikely to be cost-effective was that in which the costs of lesson teaching and preparation time were included, as per the only other microcosting of the programme. 33
These findings are significant because they suggest that, despite the very modest impact of PATHS (both in terms of QALYs and, by extension, the other outcomes assessed in the trial) identified here, it can still be considered to be likely to offer value for money. This serves as a salient reminder that estimates of effect size, when divorced from context (e.g. detailed evidence regarding the ‘input’ required to achieve a given effect), do not provide the full information required to determine whether or not an intervention is worthy of adoption. 2 In this case, although PATHS was shown to produce very small increases in QALYs relative to usual provision, the cost per child is sufficiently low to make this likely to be a worthwhile endeavour, at least from the perspective of NICE’s willingness-to-pay threshold. 70
Strengths and limitations
Our study had many strengths. Using what is considered to be a ‘gold standard’ research design (at least in terms of determining the impact of a prescribed intervention on a pre-specified set of outcomes)130,131 and a large and appropriately powered sample, a comprehensive analysis of the PATHS curriculum, examining its impact on a range of pertinent outcomes, in addition to the maintenance of effects at the 24-month follow-up, the impact on psychosocial adjustment to secondary school after 12 months, the moderating role of different dimensions of implementation on study outcomes, the temporal relations between outcomes, and cost-effectiveness was not undertaken. Our measurement protocol used well-established instruments with demonstrably sound psychometric properties and drew from multiple informants (e.g. children, teachers) and methods (e.g. surveys, standardised tests). Our analyses used recommended techniques to take account of data clustering and missingness. Alongside this quantitative work, a detailed and comprehensive qualitative inquiry was undertaken that shed light on the implementation of PATHS and the processes and factors underpinning this in participating schools, in addition to highlighting issues pertaining to perceived impact and sustainability. Our qualitative analysis incorporated both inductive and deductive elements, enabling insights to emerge directly from the data while also being shaped and contextualised by the existing body of knowledge in implementation science.
However, as with any study, there were also a number of limitations that need to be borne in mind when considering the implications of our findings. First, our sample of schools was significantly larger (in terms of number of pupils on roll) than is seen nationally, with higher proportions of pupils of pupils eligible for FSMs and lower proportions identified as having SENDs. As such, it cannot be considered to be fully representative of primary schools in England. Second, although retention at the end point of the main trial was within acceptable limits, the loss-to-follow-up rates experienced at the 12- and 24-month follow-up assessments were very high. This was of course compounded by the fact that, by design, only one-third of the main trial sample was to be involved in these assessments to begin with. Similarly disappointing attrition rates are also noted in another recent school-based trial132 that spanned the primary to secondary transition. Of the 124 secondary schools that our transition subsample (n = 1631) went on to attend, 93 (representing n = 1244) agreed to participate in the follow-up phase of the trial, with only 31 (n = 387) declining or not responding. Thus, the numbers missing (n = 781) at T5 (n = 1244 possible, minus the 463 responses) is particularly disappointing, but may simply be a reflection of the (perhaps understandable) lack of engagement of secondary schools in a project that offered no direct benefit to them. Although MI was made up of to account for missing data throughout our analyses, the sheer scale of partially observed cases suggests that the findings pertaining to hypotheses 2 and 3 should be treated with caution. More broadly, further work is needed to examine how best to optimise retention rates in trials that traverse more than one phase of schooling. The current project would perhaps have benefited, for example, from the inclusion of a secondary school representative in the trial steering group, such that practical advice could be sought on how best to engage secondary schools and incentivise their participation.
Third, although procedural fidelity, quality and responsiveness, and reach were all determined to be high in our IPE, our data indicated that, on average, PATHS lessons were being delivered only once per week rather than twice. One perspective is that this therefore represents suboptimal dosage, although it is noted that it is broadly in line with several other studies of PATHS, including the only other RCT33 to be conducted in England,33,92 and given the considerable support made available in order to optimise delivery in the trial, it is likely to represent a ‘best-case scenario’ for the implementation of PATHS in typical circumstances. At the very least, however, our findings need to be considered in the context of the dosage received as opposed to the dosage as planned. Put another way, PATHS was found to improve significantly children’s psychological well-being and QALYs (and, marginally, to reduce exclusions and improve social skills and perceptions of peer and social support) even when only one lesson per week was being delivered on average across the 2 years of the trial.
Fourth, despite being more comprehensive than much previous work on the implementation of PATHS, our IPE was also limited in a number of respects. Each teacher/classroom were observed only once, thereby providing only a ‘snapshot’ of implementation. Generally, it is recommended that implementation be captured over multiple occasions to improve reliability, such that temporal patterns can be identified and taken into account. 38 However, this was simply not possible given the resource implications and burden that would have been placed on participating schools. Furthermore, it was also noted that a previous study56 of PATHS found evidence of considerable stability in implementation when multiple measures were taken over the course of a school year. A more pressing problem is that the number of cases at the second level of our implementation–outcome models reported in Chapter 4, in combination with the distributional cut-point method used to classify teachers and classes as low, moderate or high on different dimensions of implementation, meant that the statistical power of these analyses were somewhat compromised. This is, of course, a difficult issue to circumvent given that the power and sample size calculations for the trial as a whole had to be driven by the requirements of an ITT analysis for our primary outcome variable. Finally, although the study used a robust, multi-informant methodology for the qualitative aspect of our IPE, the bulk of the data came from school staff (given that direct access was had to them), and it is important to consider that the responses received may be subject to social desirability bias (particularly in relation to perceptions of impact), even in spite of our clear positioning as independent evaluators. As a counterpoint to this, however, it is noted that staff in participating schools gave frank and balanced views of the PATHS curriculum (including clearly noting aspects that they found to be in need of modification or improvement), and so this is not a major concern.
Chapter 7 Conclusions
Main findings
-
Our ITT analyses demonstrated that PATHS led to marginal, non-significant gains in children’s social skills and perceptions of peer and social support, in addition to reducing the number of sessions from which they were excluded from school. The intervention also impacted significantly on children’s psychological well-being. Effect sizes were very small in all cases.
-
There was no evidence that any of the above effects were maintained at the 24-month follow-up. Furthermore, there was no evidence of sleeper effects for other outcomes that initially yielded null results.
-
There was no evidence that PATHS impacted on children’s psychosocial adjustment to secondary school at the 12-month follow-up.
-
With respect to the implementation of PATHS:
-
Our data indicated that PATHS was delivered with high levels of procedural fidelity, quality and responsiveness, and reach, but that lessons were delivered only once per week on average as opposed to twice (as recommended by the programme developer).
-
Significant associations were found between levels of implementation and outcomes, although not always in the direction expected. Higher levels of quality and responsiveness were associated with improvements in some outcomes, most notably psychological well-being. Findings in relation to dosage and reach were very mixed (e.g. higher levels of dosage predicted improved perceptions of peer and social support, but reductions in pro-social behaviour). There was no evidence of a relationship between levels of procedural fidelity and any intervention outcomes.
-
Our qualitative IPE data indicated that PATHS was initially received very positively by participating schools, and was seen as an ideal replacement for the SEAL programme that could address the needs of pupils in social, emotional and behavioural domains. However, teachers reported experiencing difficulties in implementing it at the recommended frequency, particularly in the context of competing priorities. The manualised, prescriptive nature of the programme divided opinion. Teachers reported making a series of adaptations to improve goodness of fit to their local context. Reflecting on the success of the programme, a number of areas of perceived impact were noted, spanning SEL, academic and other domains. Finally, the post-trial sustainability of the programme was examined, with staff reporting an intention to continue delivering PATHS, albeit with further modifications.
-
-
Our analysis of temporal relations between outcomes provided partial support for the SEL logic model. Specifically, evidence was found that children’s social skills indirectly predicted their later academic attainment via their protective effect on mental health difficulties.
-
Our economic analysis indicated that the mean incremental cost of PATHS compared with usual provision was £29.93 per child. Mean incremental QALYs were positive and statistically significant, but very small in magnitude. Assuming the minimum threshold recommended by NICE of £20,000 per QALY, the INB of introducing PATHS was determined to be £7.64; the probability of the programme being cost-effective in this base-case scenario is approximately 88%. All but one of our alternative scenarios increased this probability to ≥ 99%.
Implications for practice
Taken together, our findings suggest that the impact of PATHS is limited and modest. However, the observed effects of the intervention on children’s psychological well-being, in addition to the positive findings of the cost-effectiveness analysis, suggest that it may still be somewhat beneficial to schools in terms of primary prevention. Our other analyses suggest that efforts to optimise the effects of PATHS should focus on ensuring that implementation quality is sufficiently high. Thus, initial training and subsequent implementation support processes should emphasise this, perhaps at the expense of procedural fidelity (which appeared to be unrelated to outcomes).
However, the lack of maintenance or sleeper effects at follow-up highlights the need for a systematic and comprehensive approach to SEL throughout the compulsory stages of schooling. It is perhaps unrealistic to expect 2 years of exposure to a low-intensity intervention such as PATHS to have long-lasting effects, particularly across the transition from primary to secondary school. However, implementing a systematic and comprehensive approach to SEL may prove to be difficult to achieve in secondary education, where there are fewer evidence-based interventions available,3 and in which the barriers posed by external pressures and competing priorities are arguably amplified by a more rationalist ethos.
A further implication of the current study pertains to the external support that is provided to optimise implementation. Our IPE data suggested that the initial and follow-up training, alongside the continued support and assistance provided by the PATHS coaches, was considered by most teachers to be critical in terms of developing their understanding of the intervention and helping them to maintain a consistent approach to implementation (even though, ultimately, this drifted somewhat, particularly in terms of dosage, in the second year of the trial). However, these data also indicated the need for greater flexibility in the delivery of the training and coaching for PATHS. Thus, in addition to an increased focus on implementation quality, revisions to the support model for the intervention could also be undertaken such that it is tailored more to individual need and circumstances. For example, given the mixed reception for the follow-up training session, this could usefully be revised so that it is optional and/or the aims and objectives are more explicitly driven by the experiences of teachers in their early implementation efforts.
In sum, our findings indicate that PATHS is a potential candidate for inclusion in strategies to improve the psychological well-being of children and young people. The positive cost-effectiveness findings reported here may make it particularly attractive to local commissioners, given the increasing emphasis to demonstrate the value for money of investments made to improve mental health and well-being outcomes. The findings reported here are also particularly timely given the recent devolution of responsibilities for health and social care in the Greater Manchester Combined Authority, included in which is a strategic priority133 to improve mental health and well-being by ‘shifting the focus of care to prevention, early intervention and resilience’. Key principles to be implemented in order to meet this strategic priority include targeting children through schools and increasing low-level intervention spend. PATHS may thus have a part to play as one aspect of a broader, multicomponent strategy to improve children and young people’s mental health and well-being in the city region.
Research recommendations
Our study raises a number of questions that future research might usefully address:
-
Future trials of universal school-based interventions such as PATHS should include provision for hypotheses and accompanying analyses relating to differential subgroup gains. These should be seen as complementary to ITT analyses and should always be clearly specified a priori, on the basis of robust theory and evidence, in order to avoid the bias associated with post hoc data mining.
-
Given that the presumed optimal dosage of PATHS was not achievable in the majority of cases, consideration of a ‘streamlined’ iteration of the curriculum containing only core lessons and activities would be beneficial in terms of fit to the English school context. This would require research to determine the so-called ‘kernels’ (also sometimes referred to as active ingredients) of intervention content that are critical to the achievement of intended outcomes.
-
In the light of our IPE findings, researchers may seek to:
-
explicitly model the association between differing levels of existing provision and subsequent implementation variability as a means to better understand the influence of programme differentiation (i.e. distinctiveness of an intervention relative to existing provision)2
-
determine the supporting conditions for optimal levels of implementation (e.g. given the same external pressures and competing priorities, what enables some teachers to integrate fully the recommended PATHS delivery model into their existing practice but not others?)
-
systematically document the frequency and types of adaptations made by teachers in the course of implementation, such that their influence on intervention outcomes can be examined
-
examine the extent to which the clear distinction between implementation fidelity and quality evidenced here (both in terms of the factor analysis of observational data and the subsequent association with intervention outcomes) can be replicated.
-
-
Further research is required to determine the optimal methods of data generation and analysis to increase our understanding of the association between levels of implementation and outcome variability. For example, the potential utility of capturing temporal patterns in implementation, requiring multiple observations over the course of a school year, should be explored. In addition, the use of person-centred (as opposed to variable-focused, as in the current study) approaches to the analysis of implementation data may be fruitful. 134
-
The processes underpinning the SEL logic model need to be explored in more detail:
-
The indirect contribution of children’s social skills to their later academic attainment (via their protective influence on mental health difficulties) established in the current study should be examined in more detail to determine the mechanisms supporting this relationship (e.g. is the protective function observed attributable to improved self-regulatory capacity and/or increased ability to engage support from others?).
-
The extent to which temporal relationships between outcomes in the model vary as a function of school and pupil characteristics should be examined.
-
Acknowledgements
The authors would like to thank the schools and pupils who participated in the project, in addition to the trial steering group, led by Professor Paul Stallard (University of Bath). We are grateful to the codeveloper of PATHS, Professor Mark Greenberg, and the PATHS trainers at PSU. Finally, we would like to thank the reviewers of this report for their feedback during the drafting process.
Contributions of authors
All authors participated in the interpretation of the findings, contributed ideas and were involved in critically revising this report for important intellectual content. All authors read and agreed the final report.
Neil Humphrey (Professor of Psychology of Education) was the principal investigator on the study and took overall responsibility for the trial and the writing of this report.
Alexandra Hennessey (née Barlow) (Research Associate) was the trial manager and led the quantitative and qualitative analyses.
Ann Lendrum (Senior Lecturer in Psychology of Education) provided expert input into the design of the IPE and contributed to the qualitative analyses.
Michael Wigelsworth (Senior Lecturer in Psychology of Education) provided expert input into the outcome assessment protocol and quantitative analyses.
Alexander Turner (Research Associate) performed the cost-effectiveness analysis.
Margarita Panayiotou (Research Associate) performed the logic model analysis.
Craig Joyce (Doctoral Student) supported data generation and worked as a PATHS coach.
Kirsty Pert (Doctoral Student) supported data generation and worked as a PATHS coach.
Emma Stephens (Doctoral Student) supported data generation and worked as a PATHS coach.
Lawrence Wo (Research Associate) managed the online data collection infrastructure for the trial.
Garry Squires (Senior Lecturer in Educational Psychology) contributed to the design of the study and advised on intervention delivery.
Kevin Woods (Professor of Educational Psychology) contributed to the design of the study and advised on intervention delivery.
Mark Harrison (Honorary Senior Research Fellow in Health Economics) contributed to the design of the study and provided expert input on the cost-effectiveness analysis.
Rachel Calam (Professor of Child and Family Psychology) contributed to the design of the study.
Publications
Humphrey N, Barlow A, Lendrum A. Quality matters: implementation moderates student outcomes in the PATHS curriculum. Prev Sci 2017;19:197–208.
Humphrey N, Barlow A, Wigelsworth M, Lendrum A, Pert K, Joyce C, et al. A cluster randomized controlled trial of the Promoting Alternative THinking Strategies (PATHS) curriculum. J Sch Psychol 2016;58:73–89.
Humphrey N, Barlow A, Wigelsworth M, Lendrum A, Pert K, Joyce C, et al. Promoting Alternative THinking Strategies (PATHS): Evaluation Report. London: Education Endowment Foundation; 2015.
Data-sharing statement
All data requests should be submitted to the corresponding author for consideration. Access to available anonymised data may be granted.
Disclaimers
This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the PHR programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the PHR programme or the Department of Health and Social Care.
References
- Weissberg RP, Durlak JA, Domitrovich CE, Gullotta TP, Durlak JA, Domitrovich CE, et al. Handbook of Social and Emotional Learning. New York, NY: Guilford Press; 2015.
- Humphrey N. Social and Emotional Learning: A Critical Appraisal. London: Sage Publications; 2013.
- Durlak JA, Weissberg RP, Dymnicki AB, Taylor RD, Schellinger KB. The impact of enhancing students’ social and emotional learning: a meta-analysis of school-based universal interventions. Child Dev 2011;82:405-32. https://doi.org/10.1111/j.1467-8624.2010.01564.x.
- Sklad M, Diekstra R, De Ritter M, Ben J, Gravesteijn C. Effectiveness of school-based universal social, emotional, and behavioral programs: do they enhance students’ development in the area of skills, behavior and adjustment?. Psychol Schs 2012;49:892-909. https://doi.org/10.1002/pits.21641.
- Wigelsworth M, Lendrum A, Oldfield J, Scott A, Ten-Bokkel I, Tate K, et al. The influence of trial stage, developer involvement and international transferability on the outcomes of universal social and emotional learning outcomes: a meta-analysis. Cambridge J Educ 2016;46:347-76. https://doi.org/10.1080/0305764X.2016.1195791.
- Denham SA, Ferrier DE, Howarth GZ, Herndon KJ, Bassett HH. Key considerations in assessing young children’s emotional competence. Cambridge J Educ 2016;46:299-317. https://doi.org/10.1080/0305764X.2016.1146659.
- Saarni C. The Development of Emotional Competence. New York, NY: Guilford Press; 1999.
- Goodman A, Joshi H, Nasim B, Tyler C. Social and Emotional Skills in Childhood and their Long-Term Effects on Adult Life. London: Early Intervention Foundation; 2015.
- Murphey D, Barry M, Vaughn B. Mental health disorders. Child Trends 2013:1-10.
- Goodman A, Lamping DL, Ploubidis GB. When to use broader internalising and externalising subscales instead of the hypothesised five subscales on the Strengths and Difficulties Questionnaire (SDQ): data from British parents, teachers and children. J Abnorm Child Psychol 2010;38:1179-91. https://doi.org/10.1007/s10802-010-9434-x.
- Green H, McGinnity A, Meltzer H, Ford T, Goodman R. Mental Health of Children and Young People in Great Britain. Newport: Office for National Statistics; 2005.
- Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry 2005;62:593-602. https://doi.org/10.1001/archpsyc.62.6.593.
- Belfer ML. Child and adolescent mental disorders: the magnitude of the problem across the globe. J Child Psychol Psychiatry 2008;49:226-36. https://doi.org/10.1111/j.1469-7610.2007.01855.x.
- Mathers CD, Loncar D. Projections of global mortality and burden of disease from 2002 to 2030. PLOS Med 2006;3. https://doi.org/10.1371/journal.pmed.0030442.
- The Economic and Social Costs of Mental Health Problems in 2009/10. London: Centre for Mental Health; 2010.
- Davies SC. Our Children Deserve Better: Prevention Pays (Annual Report of the Chief Medical Officer 2012). London: Department of Health and Social Care; 2012.
- Greenberg MT. School-based prevention: current status and future challenges. Eff Educ 2010;2:27-52. https://doi.org/10.1080/19415531003616862.
- Stallard P, Sayal K, Phillips R, Taylor JA, Spears M, Anderson R, et al. Classroom based cognitive behavioural therapy in reducing symptoms of depression in high risk adolescents: pragmatic cluster randomised controlled trial. BMJ 2012;345. https://doi.org/10.1136/bmj.e6058.
- Johar M, Truong J. Direct and indirect effect of depression in adolescence on adult wages. Appl Econ 2014;46:4431-44. https://doi.org/10.1080/00036846.2014.962227.
- Adi Y, Kiloran A, Janmohamed K, Stewart-Brown S. Systematic Review of the Effectiveness of Interventions to Promote Mental Well-Being in Children in Primary Education. Warwick: University of Warwick; 2007.
- Clarke AM, Morreale S, Field CA, Hussein Y, Barry MM. What Works in Enhancing Social and Emotional Skills Development During Childhood and Adolescence?. London: Early Intervention Foundation; 2015.
- McCabe C. Estimating the Short Term Cost Effectiveness of a Mental Health Promotion Intervention in Primary Schools. London: National Institute for Health and Care Excellence; 2008.
- Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014;348. https://doi.org/10.1136/bmj.g1687.
- Greenberg MT, Kusche CA, Cook ET, Quamma JP. Promoting emotional competence in school-aged children: The effects of the PATHS curriculum. Dev Psychopathol 1995;7:117-36. https://doi.org/10.1017/S0954579400006374.
- Conduct Problems Prevention Research Group . Initial impact of the Fast Track prevention trial for conduct problems: II. Classroom effects. J Consult Clin Psychol 1999;67:648-57. https://doi.org/10.1037/0022-006X.67.5.648.
- Domitrovich CE, Cortes RC, Greenberg MT. Improving young children’s social and emotional competence: a randomized trial of the preschool ‘PATHS’ curriculum. J Prim Prev 2007;28:67-91. https://doi.org/10.1007/s10935-007-0081-0.
- Efficacy of School-Wide Programs to Promote Social and Character Development and Reduce Problem Behavior in Elementary School Children. Washington, DC: Social and Character Development Research Consortium; 2010.
- Schonfeld DJ, Adams RE, Fredstrom BK, Weissberg RP, Gilman R, Voyce C, et al. Cluster-randomized trial demonstrating impact on academic achievement of elementary social-emotional learning. Sch Psychol Q 2015;30:406-20. https://doi.org/10.1037/spq0000099.
- Morris P, Mattera SK, Castell N, Bangser M, Bierman KL, Raver C. Impact Findings from the Head Start CARES Demonstration. Washington, DC: Manpower Demonstration Research Corporation; 2014.
- Malti T, Ribeaud D, Eisner MP. The effectiveness of two universal preventive interventions in reducing children’s externalizing behavior: a cluster randomized controlled trial. J Clin Child Adolesc Psychol 2011;40:677-92. https://doi.org/10.1080/15374416.2011.597084.
- Novak M, Mihić J, Bašić J, Nix RL. PATHS in Croatia: A school-based randomised-controlled trial of a social and emotional learning curriculum. Int J Psychol 2017;52:87-95. https://doi.org/10.1002/ijop.12262.
- Ross SM, Sheard MK, Cheung A, Elliott L, Slavin R. Promoting primary pupils’ social-emotional learning and pro-social behaviour: longitudinal evaluation of the Together 4 All Programme in Northern Ireland. Eff Educ 2011;3:61-8. https://doi.org/10.1080/19415532.2012.665773.
- Berry V, Axford N, Blower S, Taylor RS, Edwards RT, Tobin K, et al. The effectiveness and micro-costing analysis of a universal, school-based, social–emotional learning programme in the UK: a cluster-randomised controlled trial. School Ment Health 2016;8:238-56. https://doi.org/10.1007/s12310-015-9160-1.
- Lendrum A, Wigelsworth M. The evaluation of school-based social and emotional learning interventions: current issues and future directions. Psychol Educ Rev 2013;37:70-4.
- Eisner M. No effects in independent prevention trials: can we reject the cynical view?. J Exp Criminol 2009;5:163-83. https://doi.org/10.1007/s11292-009-9071-y.
- Allen G. Early Intervention: the Next Steps. London: Cabinet Office and Department for Work and Pensions; 2011.
- Humphrey N, Lendrum A, Wigelsworth M. Making the most out of school-based prevention: lessons from the social and emotional aspects of learning (SEAL) programme. Emot Behav Difficulties 2013;18:248-60. https://doi.org/10.1080/13632752.2013.819251.
- Humphrey N, Lendrum A, Ashworth E, Frearson K, Buck R, Kerr K. Implementation and Process Evaluation (IPE) for Interventions in Educational Settings: An Introductory Handbook. London: Education Endowment Foundation; 2016.
- Puffer S, Torgerson DJ, Watson J. Cluster randomized controlled trials. J Eval Clin Pract 2005;11:479-83. https://doi.org/10.1111/j.1365-2753.2005.00568.x.
- Gresham FM, Elliot SN. Social Skills Improvement System: Rating Scales Manual. Minneapolis, MN: Pearson; 2008.
- Treasure T, MacRae KD. Minimisation: the platinum standard for trials?. Randomisation doesn’t guarantee similarity of groups; minimisation does. BMJ 1998;317:362-3. https://doi.org/10.1136/bmj.317.7155.362.
- Greenberg MT, Kusche CA. Promoting Social and Emotional Development in Deaf Children: The PATHS Project. Seattle, WA: University of Washington Press; 1993.
- Wolpert M, Sharpe H, Humphrey N, Patalay P, Deighton J. EBPU Logic Model. London: CAMHS Press; 2016.
- Goodman R. The Strengths and Difficulties Questionnaire: a research note. J Child Psychol Psychiatry 1997;38:581-6. https://doi.org/10.1111/j.1469-7610.1997.tb01545.x.
- Ravens-Sieberer U, Auquier P, Erhart M, Gosch A, Rajmil L, Bruil J, et al. The KIDSCREEN-27 quality of life measure for children and adolescents: psychometric results from a cross-cultural survey in 13 European countries. Qual Life Res 2007;16:1347-56. https://doi.org/10.1007/s11136-007-9240-2.
- Stevens KJ. Working with children to develop dimensions for a preference-based, generic, pediatric, health-related quality-of-life measure. Qual Health Res 2010;20:340-51. https://doi.org/10.1177/1049732309358328.
- Wigelsworth M, Humphrey N, Kalambouka A, Lendrum A. A review of key issues in the measurement of children’s social and emotional skills. Educ Psychol Pract 2010;26:173-86. https://doi.org/10.1080/02667361003768526.
- Merrell KW. Behavioural, Social, and Emotional Assessment of Children and Adolescents. Oxford: Routledge; 2008.
- Humphrey N, Kalambouka A, Wigelsworth M, Lendrum A, Deighton J, Wolpert M. Measures of social and emotional skills for children and young people: a systematic review. Educ Psychol Meas 2011;71:617-37. https://doi.org/10.1177/0013164410382896.
- Goodman R. Psychometric properties of the strengths and difficulties questionnaire. J Am Acad Child Adolesc Psychiatry 2001;40:1337-45. https://doi.org/10.1097/00004583-200111000-00015.
- Robitail S, Ravens-Sieberer U, Simeoni MC, Rajmil L, Bruil J, Power M, et al. Testing the structural and cross-cultural validity of the KIDSCREEN-27 quality of life questionnaire. Qual Life Res 2007;16:1335-45. https://doi.org/10.1007/s11136-007-9241-1.
- Stevens K. Valuation of the Child Health Utility 9D Index. PharmacoEconomics 2012;30:729-47. https://doi.org/10.2165/11599120-000000000-00000.
- Stevens K. Assessing the performance of a new generic measure of health-related quality of life for children and refining it for use in health state valuation. Appl Health Econ Health Policy 2011;9:157-69. https://doi.org/10.2165/11587350-000000000-00000.
- Berkel C, Mauricio AM, Schoenfelder E, Sandler IN. Putting the pieces together: an integrated model of program implementation. Prev Sci 2011;12:23-3. https://doi.org/10.1007/s11121-010-0186-1.
- Kam CM, Greenberg MT, Walls CT. Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prev Sci 2003;4:55-63. https://doi.org/10.1023/A:1021786811186.
- Domitrovich CE, Gest SD, Jones D, Gill S, Sanford Derousie RM. Implementation quality: lessons learned in the context of the Head Start REDI trial. Early Child Res Q 2010;25:284-98. https://doi.org/10.1016/j.ecresq.2010.04.001.
- Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol 2012;8:23-34. https://doi.org/10.20982/tqmp.08.1.p023.
- Domitrovich CE, Bradshaw CP, Poduska JM, Hoagwood K, Buckley JA, Olin S, et al. Maximizing the implementation quality of evidence-based preventive interventions in schools: a conceptual framework. Adv Sch Ment Health Promot 2008;1:6-28. https://doi.org/10.1080/1754730X.2008.9715730.
- Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol 2008;41:327-50. https://doi.org/10.1007/s10464-008-9165-0.
- Campbell MK, Elbourne DR, Altman DG. CONSORT CLUSTER group . The CONSORT statement for cluster randomised trials. Med Clin 2005;125:28-31. https://doi.org/10.1016/S0025-7753(05)72206-5.
- Bierman KL, Nix RL, Heinrichs BS, Domitrovich CE, Gest SD, Welsh JA, et al. Effects of Head Start REDI on children’s outcomes 1 year later in different kindergarten contexts. Child Dev 2014;85:140-59. https://doi.org/10.1111/cdev.12117.
- Myung IJ. The importance of complexity in model selection. J Math Psychol 2000;44:190-204. https://doi.org/10.1006/jmps.1999.1283.
- DiStefano C, Zhu M, Mindrila D. Understanding and using factor scores: considerations for the applied researcher. Pract Assess Res Eval 2009;14:1-11.
- Hansen W, Sloboda Z, Petras H. Defining Prevention Science. New York, NY: Springer; 2014.
- Pampaka M, Hutcheson G, Williams J. Handling missing data: analysis of a challenging data set using multiple imputation. Int J Res Method Educ 2016;39:19-37. https://doi.org/10.1080/1743727X.2014.979146.
- Carpenter JJR, Goldstein H, Kenward MGM. REALCOM-IMPUTE software for multilevel multiple imputation with mixed response types. J Stat Softw 2011;45:1-12. https://doi.org/10.18637/jss.v045.i05.
- Schafer JL. Multiple imputation: a primer. Stat Methods Med Res 1999;8:3-15. https://doi.org/10.1177/096228029900800102.
- Schafer JL, Graham JW. Missing data: our view of the state of the art. Psychol Methods 2002;7:147-77. https://doi.org/10.1037/1082-989X.7.2.147.
- Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006;3:77-101. https://doi.org/10.1191/1478088706qp063oa.
- Guide to the Methods of Technology Appraisal 2013. London: NICE; 2013.
- Schools, Pupils and their Characteristics. London: DfE; 2012.
- Pupil Absence in Schools in England: 2011 to 2012. London: DfE; 2013.
- Children with Special Educational Needs 2013: An Analysis. London: DfE; 2013.
- National Curriculum Assessments at Key Stage 2: 2011 to 2012. London: DfE; 2013.
- How Evidence-Based SEL Programs Work to Produce Greater Student Success in School and Life. Chicago, IL: CASEL, LSS; 2007.
- Maas CJ, Hox J. Sufficient sample sizes for multilevel modeling. J Res Methods Behav Soc Sci 2005;1:86-92.
- Muthen BO, Satorra A. Complex sample data in structural equation modeling. Sociol Methodol Am Soc Assoc 1995;25. https://doi.org/10.2307/271070.
- Krishnakumar A, Narine L, Roopnarine JL, Logie C. Multilevel and cross-level effects of neighborhood and family influences on children’s behavioral outcomes in Trinidad and Tobago: the intervening role of parental control. J Abnorm Child Psychol 2014;42:1057-68. https://doi.org/10.1007/s10802-014-9852-2.
- Matsunaga M. How to factor-analyze your data right: do’s, don’ts, and how-to’s. Int J Psychol Res 2010;3:97-110. https://doi.org/10.21500/20112084.854.
- Newsom JT. Longitudinal Structural Equation Modeling: A Comprehensive Introduction. New York, NY: Routledge; 2015.
- Sellström E, Bremberg S. Is there a ‘school effect’ on pupil outcomes? A review of multilevel studies. J Epidemiol Community Health 2006;60:149-55. https://doi.org/10.1136/jech.2005.036707.
- Hedges LV, Hedberg EC. Intraclass Correlation values for planning group-randomized trials in education. Educ Eval Policy Anal 2007;29:60-87. https://doi.org/10.3102/0162373707299706.
- Bollen KA, Curran PJ. Latent Curve Models: A Structural Equation Approach. Hoboken, NJ: John Wiley & Sons, Inc.; 2006.
- MacKinnon DP, Fairchild AJ, Fritz MS. Mediation analysis. Annu Rev Psychol 2007;58:593-614. https://doi.org/10.1146/annurev.psych.58.110405.085542.
- Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci 2015;10. https://doi.org/10.1186/s13012-015-0242-0.
- Moore JE, Bumbarger BK, Cooper BR. Examining adaptations of evidence-based programs in natural contexts. J Prim Prev 2013;34:147-61. https://doi.org/10.1007/s10935-013-0303-6.
- Seifer R, Gouley K, Miller A, Zakriski A. Implementation of the PATHS curriculum in an urban elementary school. Early Educ Dev 2004;15:471-86. https://doi.org/10.1207/s15566935eed1504_6.
- Honess A, Hunter D. Teacher perspectives on the implementation of the PATHS curriculum. Educ Psychol Pract 2014;30:51-62. https://doi.org/10.1080/02667363.2013.869490.
- Curtis C, Norgate R. An evaluation of the promoting alternative thinking strategies curriculum at Key Stage 1. Educ Psychol Pract 2007;23:33-44. https://doi.org/10.1080/02667360601154717.
- Kelly B, Longbottom J, Potts F, Williamson J. Applying emotional intelligence: exploring the Promoting Alternative Thinking Strategies curriculum. Educ Psychol Pract 2004;20:221-40. https://doi.org/10.1080/0266736042000251808.
- The National Strategies: A Review of Impact. London: Ofsted; 2010.
- Faria AM, Kendziora K, Brown L, O’Brien B, Osher D. PATHS Implementation and Outcome Study in the Cleveland Metropolitan School District: Final Report. Washington, DC: American Institutes for Research; 2013.
- Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med 2012;75:2299-306. https://doi.org/10.1016/j.socscimed.2012.08.032.
- Gottfredson DC, Cook TD, Gardner FE, Gorman-Smith D, Howe GW, Sandler IN, et al. Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: next generation. Prev Sci 2015;16:893-926. https://doi.org/10.1007/s11121-015-0555-x.
- Humphrey N, Barlow A, Wigelsworth M, Lendrum A, Pert K, Joyce C, et al. Promoting Alternative Thinking Strategies (PATHS): Evaluation Report. London: Education Endowment Foundation; 2015.
- NUT Guidance on the Teachers’ Pay Structure for 2014/15. London: NUT; 2014.
- School Teachers’ Pay: NUT Guidance on Pay Appeals September 2014. London: NUT; 2014.
- Netten A, Knight J. Annuitizing the human capital investment costs of health service professionals. Health Econ 1999;8:245-55. https://doi.org/10.1002/(SICI)1099-1050(199905)8:3<245::AID-HEC430>3.0.CO;2-4.
- School Workforce in England: November 2015. London: DfE; 2015.
- Faria R, Gomes M, Epstein D, White IR. A guide to handling missing data in cost-effectiveness analysis conducted within randomised controlled trials. PharmacoEconomics 2014;32:1157-70. https://doi.org/10.1007/s40273-014-0193-3.
- White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and guidance for practice. Stat Med 2011;30:377-99. https://doi.org/10.1002/sim.4067.
- Gomes M, Grieve R, Nixon R, Edmunds WJ. Statistical methods for cost-effectiveness analyses that use data from cluster randomized trials: a systematic review and checklist for critical appraisal. Med Decis Making 2012;32:209-20. https://doi.org/10.1177/0272989X11407341.
- Nixon RM, Wonderling D, Grieve RD. Non-parametric methods for cost-effectiveness analysis: the central limit theorem and the bootstrap compared. Health Econ 2010;19:316-33. https://doi.org/10.1002/hec.1477.
- Ratcliffe J, Stevens K, Flynn T, Brazier J, Sawyer M. An assessment of the construct validity of the CHU9D in the Australian adolescent general population. Qual Life Res 2012;21:717-25. https://doi.org/10.1007/s11136-011-9971-y.
- Durlak JA. How to select, calculate, and interpret effect sizes. J Pediatr Psychol 2009;34:917-28. https://doi.org/10.1093/jpepsy/jsp004.
- Wright M, Masten AS, Narayan A, Goldstein S, Brooks RB. Handbook of Resilience in Children. Boston, MA: Springer; 2013.
- Farrell AD, Henry DB, Bettencourt A. Methodological challenges examining subgroup differences: examples from universal school-based youth violence prevention trials. Prev Sci 2013;14:121-33. https://doi.org/10.1007/s11121-011-0200-2.
- Greenberg MT, Abenavoli R. Universal Interventions: Fully exploring their impacts and potential to produce population-level impacts. J Res Educ Eff 2017;10:40-67. https://doi.org/10.1080/19345747.2016.1246632.
- Personal, Social, Health and Economic (PSHE) Education: A Review of Impact and Effective Practice. London: DfE; 2015.
- Jennings P, Greenberg MT. The prosocial classroom: teacher social and emotional competence in relation to student and classroom outcomes. Rev Educ Res 2009;79:491-525. https://doi.org/10.3102/0034654308325693.
- Ransford C, Greenberg M, Domitrovich CE, Small M, Jacobson L. The role of teachers’ psychological experiences and perceptions of curriculum supports on the implementation of a social and emotional learning curriculum. School Psych Rev 2009;38:510-32.
- Bronfenbrenner U. Making Human Beings Human: Bioecological Perspectives on Human Development. London: Sage Publications, Inc.; 2005.
- Domitrovich CE, Bradshaw CP, Greenberg MT, Embry D, Poduska JM, Ialongo NS. Integrated models of school-based prevention: logic and theory. Psychol Sch 2010;47:71-88. https://doi.org/10.1002/pits.20452.
- Taylor RD, Oberle E, Durlak JA, Weissberg RP. Promoting positive youth development through school-based social and emotional learning interventions: a meta-analysis of follow-up effects. Child Dev 2017;88:1156-71. https://doi.org/10.1111/cdev.12864.
- Kam C-M, Greenberg MT, Kusche CA. Sustained effects of the PATHS curriculum on the social and psychological adjustment of children in special education. J Emot Behav Disord 2004;12:66-78. https://doi.org/10.1177/10634266040120020101.
- Mclaughlin C, Clarke B. Relational matters: a review of the impact of school experience on mental health in early adolescence. Inquiry 2010;27:91-103.
- Rosenblatt JL, Elias MJ. Dosage effects of a preventive social-emotional learning intervention on achievement loss associated with middle school transition. J Prim Prev 2008;29:535-55. https://doi.org/10.1007/s10935-008-0153-9.
- Qualter P, Whiteley HE, Hutchinson JM, Pope DJ. Supporting the development of emotional intelligence competencies to ease the transition from primary to high school. Educ Psychol Pract 2007;23:79-95. https://doi.org/10.1080/02667360601154584.
- Durlak JA. Studying program implementation is not easy but it is essential. Prev Sci 2015;16:1123-7. https://doi.org/10.1007/s11121-015-0606-3.
- Elliott DS, Mihalic S. Issues in disseminating and replicating effective prevention programs. Prev Sci 2004;5:47-53. https://doi.org/10.1023/B:PREV.0000013981.28071.52.
- Lendrum A, Humphrey N, Greenberg MT, Shute RH, Slee PT. Mental Health and Well-Being Through Schools: The Way Forward. London: Taylor and Francis; 2016.
- Liu J. Encyclopedia of Biopharmaceutical Statistics. London: Informa; 2010.
- Pettigrew J, Graham JW, Miller-Day M, Hecht ML, Krieger JL, Shin YJ. Erratum to: Adherence and delivery: implementation quality and program outcomes for the seventh-grade keepin’ it REAL program. Prev Sci 2015;16. https://doi.org/10.1007/s11121-014-0506-y.
- Matthews G, Roberts RD, Zeidner M. Seven myths about emotional intelligence. Psychol Inq 2004;15:179-96. https://doi.org/10.1207/s15327965pli1503_01.
- Humphrey N, Wigelsworth M. Modelling the factors associated with children’s mental health difficulties in primary school: A multilevel study. School Psych Rev 2012;41:326-41.
- Masten AS, Cicchetti D. Developmental cascades. Dev Psychopathol 2010;22:491-5. https://doi.org/10.1017/S0954579410000222.
- Belfield C, Bowden B, Klapp A, Levin H, Shand R, Zander S. The Economic Value of Social and Emotional Learning. New York, NY: Center for Benefit-Cost Studies in Education; 2015.
- Hummel S, Naylor P, Chilcott J, Guillaume L, Wilkinson A, Blank L, et al. Cost-Effectiveness of Universal Interventions which Aim to Promote Emotional and Social Well-Being in Secondary Schools. London: National Institute for Health and Care Excellence; 2009.
- Long K, Brown JL, Jones SM, Aber JL, Yates BT, Long K, et al. Cost analysis of a school-based social and emotional learning and literacy intervention. J Benefit-Cost Anal 2015;6:545-71. https://doi.org/10.1017/bca.2015.6.
- Torgerson C, Torgerson D. The need for randomised controlled trials in educational research. Br J Educ Stud 2001;49:24-6. https://doi.org/10.1111/1467-8527.t01-1-00178.
- Tymms P, Merrell C, Coe R. Educational policies and randomised controlled trials. Psychol Educ Rev 2008;32:3-7.
- Stallard P, Skryabina E, Taylor G, Anderson R, Ukoumunne OC, Daniels H, et al. A cluster randomised controlled trial comparing the effectiveness and cost-effectiveness of a school-based cognitive–behavioural therapy programme (FRIENDS) in the reduction of anxiety and improvement in mood in children aged 9/10 years. Public Heal Res 2015;3:1-88. https://doi.org/10.3310/phr03140.
- Greater Manchester Mental Health and Wellbeing Strategy. Manchester: Greater Manchester Combined Authority; 2016.
- Low S, Smolkowski K, Cook C. What constitutes high-quality implementation of SEL programs? A latent class analysis of Second Step Implementation. Prev Sci 2016;17:981-9.
Appendix 1 Participant flow
FIGURE 8.
Full CONSORT diagram depicting flow of participants through the study (T1–T5). a, Thirteen schools failed to complete baseline measures because of lack of time (n = 7), lack of IT facilities (n = 2), other priorities (e.g. Ofsted) (n = 1) and lack of response (n = 3). b, Pupil surveys were not completed because of absence, changing school, non-completion or parental opt-out (n = 818). c, Of the 4400 pupils who provided baseline data, the study lost post intervention pupils from five schools (n = 420), as well as individual pupils (n = 623), by whom surveys were not completed because of absence, changing school or parental opt-out. However, the study gained additional pupils (n = 531) who provided post-intervention data only. d, Pupils moved to 124 secondary schools, of which 93 (n = 1244 pupils) agreed to participate, 6 declined to participate and 25 did not respond (n = 387 total). Pupil surveys (n = 451) were not completed because of absence, changing schools, etc. e, Of the 793 pupils who provided baseline data, the study lost pupils to follow-up owing to non-completion, absence, changing schools, etc. (n = 407). However, the study gained pupils who provided post-trial follow-up data only (n = 75).
Appendix 2 Full tables (with explanatory notes) detailing costs of the Promoting Alternative THinking Strategies intervention
| Costs | Total costs (2014/15 £) | Notes |
|---|---|---|
| Materials costs | ||
| Curriculum packs and supplementary materials | 34,431 | Fixed fee of £33,000 paid by research team. Inflated from 2012/13 prices using the RPI. 2012 index of 246.8, 2014 index of 257.5 |
| Implementation guidance | 94 | One manual for each of the 99 classes. Cost of printing and binding 99 26-side documents costed at £90 derived from University of Manchester reprographics quote. Inflated from 2012/13 prices using the RPI |
| Total materials costs | 34,525 | |
| Annuitised for the 2-year trial period | 15,293 | A 5-year intervention life assumed. Annuitised with a discount rate of 3.5%. Multiplied by two to account for the 2 years of the trial |
| PATHS coach training | ||
| Training materials and teaching and preparation time for PSU staff | 4519 | Apportionment of fixed fee of £10,828 paid to PSU, based on 2 out of the total of 5 training days spent on PATHS coach training. Inflated from 2012/13 prices using the RPI |
| PATHS coach time while attending training | 462 | For each of the three coaches: 14 hours of training time (7 hours on each of 2 days). Apportionment of £16,000 annual stipend (excluding 25% employer on-costs) calculated assuming 1820 working hours per year (consistent with standard University of Manchester researcher contract). Implies hourly rate (including on-costs) of £10.99 |
| Travel and accommodation costs | 3120 | For each of the three coaches: £658 return flight, £361 for 3 nights’ accommodation. Inflated from 2012/13 prices using the RPI |
| Total PATHS coach training costs | 8101 | |
| Annuitised for the 2-year trial period | 3588 | A 5-year intervention life assumed. Annuitised with a discount rate of 3.5%. Multiplied by two to account for the 2 years of the trial |
| Initial full-day training for original teachers | ||
| Training materials and teaching and preparation time for PSU staff for training teachers in year 1 of the trial | 4519 | Apportionment of fixed fee of £10,828 paid to PSU, based on 2 out of the total of 5 training days spent on initial teacher training. Inflated from 2012/13 prices using the RPI |
| Travel and accommodation costs for PSU staff for year 1 teacher training | 2112 | For two PSU trainers: flights – £1827.28; accommodation – £196.66. Inflated from 2012/13 prices using the RPI |
| PATHS coach teaching time for initial group training conducted alongside PSU staff in year 1 of the trial | 462 | For each of the three coaches: 14 hours of teaching time (7 hours in each of the 2 days). Costed identically to other uses of PATHS coach time |
| PATHS coach teaching time for initial group training for Year 6 teachers in year 2 of the trial | 147 | For each of the three coaches: 1 day for full-day training (7 hours) of 33 Year 6 and 17 new-starting Years 4 and 5 teachers, apportioned to include costs only for the Year 6 teachers. Costed identically to other uses of PATHS coach time. Costs discounted at a rate of 3.5% |
| Venue hire for group training sessions (both years of the trial) | 1583 | Year 1 of the trial: 2 days for full-day training of original 99 Years 3–5 teachers. Year 2 of the trial: 1 day for full-day training of 33 Year 6 and 17 new-starting Years 4 and 5 teachers, apportioned to include costs only for the Year 6 teachers. 2014/15 unit price of £600 (including VAT) per day taken from University of Manchester conference prices. Year 2 costs discounted at 3.5% |
| Catering for group training sessions (both years of the trial) | 843 | Catering provided by Le Mange Tout Ltd (Manchester, UK). Year 1 of the trial: cost of catering for 2 full-day training sessions of £600. Inflated from 2012/13 prices using the RPI. Year 2 of the trial: cost of catering for 1 full-day training session of £330. Apportioned to include costs only for the Year 6 teachers. Inflated from 2013/14 prices using the RPI, and discounted at a rate of 3.5% |
| PATHS coach teaching time for training teachers who could not attend group training sessions (both years of the trial) | 2136 | Catering provided by Le Mange Tout Ltd. Year 1 of the trial: cost of catering for 2 full-day training sessions of £600. Inflated from 2012/13 prices using the RPI. Year 2 of the trial: cost of catering for 1 full-day training session of £330. Apportioned to include costs only for the Year 6 teachers. Inflated from 2013/14 prices using the RPI, and discounted at a rate of 3.5% |
| Cost of supply teacher cover (both years of the trial) | 10,269 | In both years: group training sessions, as well as one-on-one training sessions for those who could not attend group training sessions, were estimated to have occurred outside INSET days for approximately half of the schools, and so it was assumed that supply teacher cover was required for approximately half of the teachers. Each teacher was trained for 1 full day. Supply cover was costed using a daily rate of £154.6, assuming the 2014/15 national average salary of £30,138 for a mid-point (M4) qualified teacher and a 195-day yearly contract, consistent with the standard teaching contract. Year 1 of the trial: 99 teachers were initially trained, with 50 assumed to require supply teacher cover. Year 2 of the trial: 33 Year 6 teachers were initially trained; 17 were assumed to have required supply teacher cover. Year 2 costs were discounted at a rate of 3.5% |
| Total initial full-day training cost | 22,071 | |
| Annuitised for the 2-year trial period | 17,116 | Given an average teacher turnover rate of 36.36% across the trial, the expected length of service for a teacher was 2.75 years. Annuitised with a discount rate of 3.5%. Multiplied by two to account for the 2 years of the trial |
| Total non-recurrent costs | 35,997 | |
| Costs | Total costs (2014/15 £) | Notes |
|---|---|---|
| Curriculum and teacher support costs | 23,560 | In each year of the trial, one-third of £16,000 annual PhD stipend (£5333) for each of the three PATHS coaches assigned to support the trial = £16,000 per year. After netting off costs of PATHS coach time during PATHS coach training, and PATHS coach time during teacher training during PSU-led teacher training the remainder was assigned to ongoing PATHS support. Year 2 costs discounted at 3.5% |
| Follow-up teacher training for original teachers | ||
| Training materials and teaching and preparation time for PSU staff for training teachers in year 1 of the trial | 2259 | Apportionment of fixed fee of £10,828 paid to PSU, based on 1 out of the total of 5 training days spent on initial teacher training. Inflated from 2012/13 prices using the RPI |
| Travel and accommodation costs for PSU staff for the first year of teacher training | 813 | For one PSU trainer: flight – £682.39; accommodation: £96.67. Inflated from 2012/13 prices using the RPI |
| PATHS coach teaching time for follow-up group training conducted alongside PSU staff in year 1 of the trial | 231 | For each of the three coaches: half-day training took place over 1 day (7 hours). Costed identically to other uses of PATHS coach time |
| PATHS coach teaching time for follow-up group training for Year 6 teachers in year 2 of the trial | 147 | For each of the three coaches: 1 day for half-day training (7 hours) of 33 Year 6 and 17 new-starting Years 4 and 5 teachers, apportioned to include costs only for the Year 6 teachers. Costed identically to other uses of PATHS coach time. Costs discounted at a rate of 3.5% |
| Venue hire for group training sessions (both years of the trial) | 983 | Year 1 of the trial: 1 full day for half-day training for initial Years 3–5 teachers. Year 2: 1 full day for half-day training for Year 6 teachers and new-starting Years 4 and 5 teachers, apportioned to include costs only for the Year 6 teachers. 2014/15 unit price of £600 (including VAT) per day taken from University of Manchester conference prices. Year 2 costs discounted at 3.5% |
| Catering for group training sessions (both years of the trial) | 541 | Catering provided by Le Mange Tout Ltd. Year 1 of the trial: cost of catering for 1 half-day training session of £310. Inflated from 2012/13 prices using the RPI. Year 2 of the trial: cost of catering for one half-day training session of £330. Apportioned to include costs only for the Year 6 teachers. Inflated from 2013/14 prices using the RPI, and discounted at a rate of 3.5% |
| PATHS coach teaching time for training teachers who could not attend follow-up group training sessions | 1220 | Year 1 of the trial: 21 out of 99 teachers received initial half-day training by a PATHS coach outside PSU-led group training sessions (4 hours per teacher). Year 2 of the trial: 7 out of 33 Year 6 teachers received half-day training by a PATHS coach outside PATHS coach-led group training sessions (4 hours per teacher). Costed identically to other uses of PATHS coach time. Year 2 costs were discounted at a rate of 3.5% |
| Cost of supply teacher cover (both years of the trial) | 5135 | In both years: group training sessions, as well as one-on-one training sessions for those who could not attend group training sessions, were estimated to have occurred outside INSET days for approximately half of the schools, and so it was assumed that supply teacher cover was required for approximately half of the teachers. Each teacher was trained for half a day, and teachers were assumed to be able to return to work for teaching in the other half of the day. Supply cover was costed as above. Year 1 of the trial: 99 teachers were initially trained, with 50 assumed to require supply teacher cover. Year 2 of the trial: 33 Year 6 teachers were initially trained 1; 17 were assumed to have required supply teacher cover. Year 2 costs were discounted at a rate of 3.5% |
| Total follow-up training costs for original teachers | 11,329 | |
| Training replacements of Years 4 and 5 teachers leaving at the end of the school year (training alongside Year 6 teachers in year 2 of the trial) | ||
| Initial full-day training | ||
| PATHS coach teaching time for group training session | 76 | For each of the three coaches: 1 day for full-day training (7 hours) of 33 Year 6 and 17 new-starting Years 4 and 5 teachers, apportioned to include costs only for the Years 4 and 5 replacement teachers. Costed identically to other uses of PATHS coach time. Costs discounted at a rate of 3.5% |
| Venue hire for initial group training session in year 2 of the trial | 197 | 1 day for half-day training of 33 Year 6 and 17 new-starting Years 4 and 5 teachers, apportioned to include costs only for the new-starting Years 4 and 5 teachers. 2014/15 unit price of £600 (including VAT) per day taken from University of Manchester conference prices. Year 2 costs discounted at 3.5% |
| Catering for initial group training session in year 2 of the trial | 112 | Catering provided by Le Mange Tout Ltd. Cost of catering for 1 full-day training session of £330. Apportioned to include costs only for the new-starting Years 4 and 5 teachers. Inflated from 2013/14 prices using the RPI, and discounted at a rate of 3.5% |
| PATHS coach teaching time for training teachers who could not attend the initial group training session | 223 | 3 out of 17 Year 6 teachers received initial full-day training by a PATHS coach outside PSU-led group training session for 1 day (7 hours). Costed identically to other uses of PATHS coach time. Year 2 costs were discounted at a rate of 3.5% |
| Cost of supply teacher cover | 1344 | Group training sessions, as well as one-on-one training sessions for those who could not attend group training sessions, were estimated to have occurred outside INSET days for approximately half of the schools, and so it was assumed that supply teacher cover was required for approximately half of the teachers. A total of 25 teachers left at the end of year 1 of the trial, with 17 of these assumed to be from Years 4 and 5. Nine of these were assumed to be trained outside INSET days. Each teacher was trained for 1 full day. Supply teaching cover was costed as earlier. Year 2 costs were discounted at a rate of 3.5% |
| Total initial full-day training cost | 1952 | |
| Annuitised for the 2-year trial period | 1514 | Given an average teacher turnover rate of 36.36% across the trial, the expected length of service for a teacher was 2.75 years. Annuitised with a discount rate of 3.5%. Multiplied by two to account for the 2 years of the trial |
| Follow-up half-day training | ||
| PATHS coach teaching time for group training session | 76 | For each of the three coaches: 1 day for half-day training (7 hours) of 33 Year 6 and 17 new-starting Years 4 and 5 teachers, apportioned to include costs only for the Year 4 and 5 replacement teachers. Costed identically to other uses of PATHS coach time. Costs discounted at a rate of 3.5% |
| Venue hire for initial group training session in year 2 of the trial | 197 | 1 day for full-day training of 33 Year 6 and 17 new-starting Years 4 and 5 teachers, apportioned to include costs only for the new-starting Years 4 and 5 teachers. 2014/15 unit price of £600 (including VAT) per day taken from University of Manchester conference prices. Year 2 costs discounted at 3.5% |
| Catering for initial group training session in year 2 of the trial | 112 | Catering provided by Le Mange Tout Ltd. Cost of catering for one full-day training session: £330. Apportioned to include costs only for the new-starting Years 4 and 5 teachers. Inflated from 2013/14 prices using the RPI, and discounted at a rate of 3.5% |
| PATHS coach teaching time for training teachers who could not attend the initial group training session | 127 | A total of 3 out of 17 Year 6 teachers received initial full-day training by a PATHS coach outside PSU-led group training session for half a day (4 hours per teacher). Costed identically to other uses of PATHS coach time. Year 2 costs discounted at a rate of 3.5% |
| Cost of supply teacher cover | 672 | Group training sessions, as well as one-on-one training sessions for those who could not attend group training sessions, were estimated to have occurred outside INSET days for approximately half of schools, and so it was assumed that supply teacher cover was required for approximately half of the teachers. A total of 25 teachers left at the end of year 1 of the trial, with 17 of these assumed to be from Years 4 and 5. Nine of these were assumed to be trained outside INSET days. Each teacher was trained for half a day, and teachers were assumed to be able to return to work for teaching in the other half of the day. Supply teaching cover was costed as earlier. Year 2 costs discounted at a rate of 3.5% |
| Total initial half-day training cost | 1184 | |
| Training replacements of teachers leaving part-way through the school year (both years of the trial) | ||
| Initial full-day training | ||
| PATHS coach teaching time | 1666 | Each teacher received a full-day (7 hours) individual training session led by a single PATHS coach. Costed identically to other uses of PATHS coach time. Year 1 of the trial: 12 teachers left part-way through the year. Year 2 of the trial: 10 teachers left part-way through the year. Year 2 costs discounted at a rate of 3.5% |
| Cost of supply teacher cover | 3349 | Training of teachers replacing original teachers who left part-way through the school year was all assumed to occur outside INSET days and so all required supply teacher cover. Each teacher received a full-day (7 hours) training session. Supply teaching cover was costed as earlier. Year 1 of the trial: 12 teachers left part-way through the year. Year 2 of the trial: 10 teachers left part-way through the year. Year 2 costs discounted at a rate of 3.5% |
| Total initial full-day training cost | 5015 | |
| Annuitised for the 2-year trial period | 3889 | Given an average teacher turnover rate of 36.36% across the trial, the expected length of service for a teacher was 2.75 years. Annuitised with a discount rate of 3.5%. Multiplied by two to account for the 2 years of the trial |
| Follow-up half-day training | ||
| PATHS coach teaching time | 952 | Each teacher received a half-day (4 hours) individual training session led by a single PATHS coach. Costed identically to other uses of PATHS coach time. Year 1 of the trial: 12 teachers left part-way through the year. Year 2 of the trial: 10 teachers left part-way through the year. Year 2 costs discounted at a rate of 3.5% |
| Cost of supply teacher cover | 1674 | Training of teachers replacing original teachers who left part-way through the school-year was all assumed to occur outside INSET days and so all required supply teacher cover. Each teacher received a half-day training session, and teachers were assumed to be able to return to work for teaching in the other half of the day. Supply teaching cover was costed as earlier. Year 1 of the trial: 12 teachers left part-way through the year. Year 2 of the trial: 10 teachers left part-way through the year. Year 2 costs discounted at a rate of 3.5% |
| Total follow-up half-day training cost | 2626 | |
| Total recurrent costs | 44,102 | |
| Costs | Total costs (2014/15 £) | Notes |
|---|---|---|
| Total non-recurrent costs | 35,997 | |
| Total recurrent costs | 44,102 | |
| Total intervention costs | 80,099 | |
| Cost per child | 29.93 | Total intervention cost divided by total number of pupils (n = 2676) in PATHS schools |
| Costs | Costs (2014/15 £) | Notes |
|---|---|---|
| Non-recurrent costs | ||
| Original classroom teacher time during initial full-day training | ||
| Teachers’ time | 25,287 | Each teacher was trained for 1 full day. Teacher time was costed using a daily rate of £193.20, assuming the 2014/15 national average salary of £30,138 (£37,672.50 inclusive of 25% employers’ on-costs) for a mid-point (M4) qualified teacher and a 195-day yearly contract. Year 1 of the trial: 99 Years 3–5 teachers were initially trained. Year 2 of the trial: 33 Year 6 teachers were initially trained. Year 2 costs were discounted at a rate of 3.5% |
| Annuitised for the 2-year trial period | 19,609 | Given an average teacher turnover rate of 36.36% across the trial, the expected length of service for a teacher was 2.75 years. Annuitised with a discount rate of 3.5%. Multiplied by two to account for the 2 years of the trial |
| Recurrent costs | ||
| Original classroom teacher time during follow-up half-day training | ||
| Teachers’ time | 15,601 | Each teacher was trained for a half day, which was approximated to be 4 hours. Teacher time was costed using an hourly rate of £29.80, assuming the 2014/15 national average salary of £30,138 (£37,672.5 inclusive of 25% employers’ on-costs) for a mid-point (M4) qualified teacher and a 1265-hour yearly contract. Year 1 of the trial: 99 original Years 3–5 teachers received follow-up training. Year 2 of the trial: 33 original Year 6 teachers received follow-up training. Year 2 costs were discounted at a rate of 3.5% |
| Annuitised for the 2-year trial period | 12,099 | Given an average teacher turnover rate of 36.36% across the trial, the expected length of service for a teacher was 2.75 years. Annuitised with a discount rate of 3.5%. Multiplied by two to account for the 2 years of the trial |
| Replacement classroom teacher time during initial full-day training and follow-up half-day training | ||
| Teachers’ time | 8880 | Each teacher was initially trained for 1 full day and trained for a half day (4 hours) as a follow-up. Teachers’ time was costed as above. Year 2 costs were discounted at a rate of 3.5%. Year 1 of the trial: 12 teachers left part-way through the year and were trained in year 1 of the trial. Seventeen Years 4 and 5 teacher left at the end of the year and were trained in year 2 of the trial. Year 2 of the trial: 10 teachers left part-way through the year and were trained in year 2 |
| Annuitised for the 2-year trial period | 6886 | Given an average teacher turnover rate of 36.36% across the trial, the expected length of service for a teacher was 2.75 years. Annuitised with a discount rate of 3.5%. Multiplied by two to account for the 2 years of the trial |
| Classroom teacher time during lesson delivery | ||
| Teachers’ time | 232,025 | 30-minute lessons, delivered twice per week, over a period of 40 weeks per year. Lessons delivered to 99 classes in both years of the trial. Teachers’ time was costed as earlier. Year 2 costs were discounted at a rate of 3.5% |
| Classroom teacher time during lesson preparation | ||
| Teachers’ time: 40 minutes/week | 154,684 | Lessons delivered to 99 classes in both years of the trial. Year 2 costs were discounted at a rate of 3.5% |
| Teachers’ time: 45 minutes/week | 174,019 | Lessons delivered to 99 classes in both years of the trial. Year 2 costs were discounted at a rate of 3.5% |
| Total cost of teachers’ time | ||
| 40 minutes/week lesson preparation | 425,303 | |
| 45 minutes/week lesson preparation | 444,639 | |
| Total other intervention costs | 80,099 | |
| Total intervention costs | ||
| 40 minutes/week lesson preparation | 505,402 | |
| 45 minutes/week lesson preparation | 524,738 | |
| Cost per child | ||
| 40 minutes/week lesson preparation | 188.86 | |
| 45 minutes/week lesson preparation | 196.09 | |
| Costs | Total cost (2014/15 £) | Notes |
|---|---|---|
| Costs for a single-form entry, seven year group school | ||
| Non-recurrent costs | ||
| Initial full-day training for original classroom teachers | 500 | Unbundled cost from Birmingham City Council |
| Curriculum packs and supplementary materials | 2100 | Unbundled cost from Birmingham City Council |
| PATHS coach training and initial support | 3932 | Bundled cost from Birmingham City Council of £6532, net of the unbundled costs of curriculum packs and supplementary materials and initial full-day teacher training |
| Total non-recurrent costs | 6532 | |
| Annuitised for the 2-year trial period | 2893 | Using the national wastage rate of 10.6%, the expected length of service for a teacher is 9.43 years. As this is less than the 5-year expected life of the intervention, all non-recurrent costs are annuitised assuming a 5-year intervention life and a discount rate of 3.5%. Multiplied by two to account for the 2-year length of the trial |
| Recurrent costs | ||
| Half-day follow-up training | 250 | Unbundled cost from Birmingham City Council. This included both original teachers and replacement teachers, as it is assumed that departing teachers leave at the end of the school year |
| Initial full-day training for replacement teachers, annuitised for the 2-year trial period | 221 | Unbundled cost from Birmingham City Council, annuitised as earlier |
| Total recurrent costs | 471 | |
| Total cost for single-form entry, seven year groups, school | 3365 | |
| Price per class | 480.70 | Annuitised costs divided by seven as a single-form entry school contains seven classes (Reception to Year 6) |
| Total intervention cost for PATHS sample (99 classes) | 47,589 | |
| Cost per child | 17.78 | Total intervention cost divided by total number of pupils (n = 2676) in PATHS schools |
List of abbreviations
- CEAC
- cost-effectiveness acceptability curve
- CFA
- confirmatory factor analysis
- CFI
- Comparative Fit Index
- CHU-9D
- Child Health Utility 9 Dimensions
- CI
- confidence interval
- CONSORT
- Consolidated Standards of Reporting Trials
- CUA
- cost–utility analysis
- Deff
- design effect
- DfE
- Department for Education
- EAL
- English as an additional language
- EEF
- Education Endowment Foundation
- EFA
- exploratory factor analysis
- FSM
- free school meal
- HLM
- hierarchical linear modelling
- HRQoL
- health-related quality of life
- ICC
- intracluster correlation coefficient
- INB
- incremental net benefit
- INSET
- IN-SErvice Training
- IPE
- implementation and process evaluation
- ITT
- intention to treat
- KS1
- Key Stage 1
- KS2
- Key Stage 2
- KS27
- Kidscreen-27
- MAR
- missing at random
- MCFA
- multilevel confirmatory factor analysis
- MI
- multiple imputation
- MoA
- memorandum of agreement
- NICE
- National Institute for Health and Care Excellence
- NPD
- National Pupil Database
- NQT
- newly qualified teacher
- Ofsted
- Office for Standards in Education, Children's Services and Skills
- PATHS
- Promoting Alternative THinking Strategies
- PSHE
- personal, social and health education
- PSU
- Pennsylvania State University
- QALY
- quality-adjusted life-year
- RCT
- randomised controlled trial
- RE
- religious education
- RMSEA
- root-mean-square error of approximation
- SATS
- National Curriculum assessments
- SD
- standard deviation
- SDQ
- Strengths and Difficulties Questionnaire
- SE
- standard error
- SEAL
- Social and Emotional Aspects of Learning
- SEL
- social and emotional learning
- SEM
- structural equation modelling
- SENCo
- special educational needs co-ordinator
- SEND
- special educational needs and disability
- SLT
- senior leadership team
- SSIS
- Social Skills Improvement System
- T
- time point
- TIDieR
- Template for Intervention Description and Replication
- TLI
- Tucker–Lewis Index