A cluster randomised controlled trial and evaluation and cost-effectiveness analysis of the Roots of Empathy schools-based programme for improving social and emotional well-being outcomes among 8- to 9-year-olds in Northern Ireland

Deviations of the evaluation from the original protocol

The protocol for this evaluation was published in August 2011, before ethics approval was secured and, thus, before the recruitment of schools and pre-testing. It has not been amended since and is available online. 30 The trial was also registered with an International Standard Randomised Controlled Trial Number (ISRCTN) in December 2011 (ISRCTN07540423). The original aims and objectives of the evaluation have not been altered and the overall approach to the research design in relation to the cluster randomised controlled trial, the process evaluation and the cost-effectiveness evaluation have also remained unchanged. The specified primary and secondary outcomes, with their accompanying measures, have also remained the same, as has the proposed analysis plan.

Within this, a small number of minor deviations to the original protocol, published in 2011, have been made, and these are detailed below.

Data collection

The research instruments used for data collection are provided in Report Supplementary Material 1. Initial pre-test data [time (T) 0] from the children, parents and teachers were collected in October 2011 across all participating schools before the first sessions of ROE were delivered in the intervention schools. The first (immediately) post-test data (T1) were collected in June 2012 and again, annually, at 12 (T2), 24 (T3) and 36 months (T4). The final sweep of data collection took place in June 2015 when children were 11–12 years of age and at the end of their first year in secondary school.

Teachers were asked to complete questionnaires, which included the SDQ and the CBS, for each participating child in their class at each time point. When children were attending primary school, their class teacher completed the questionnaire. At the final sweep of data collection, when children were completing their first year of secondary school, their form teacher (or the teacher who knew the child best) was asked to complete the questionnaire. Thus, a different teacher completed the questionnaire at each sweep of data collection.

Consenting parents were contacted by post and asked to complete a questionnaire, which included the SDQ and background information on family composition, parental education and employment, and return it to the research team in a freepost envelope. Parents were given the option of completing the questionnaire via telephone interview, but none chose this.

Experienced field workers visited each school and administered questionnaires to the children on a whole-class basis. Fieldworkers were fully trained and co-ordinated by the research team. The children’s questionnaires covered the secondary outcomes detailed in Table 3 (emotional regulation, empathy, recognition of emotions, understanding of infant crying, bullying and quality of life). Children were asked not to confer, and this was ensured by the field worker and the class teacher. Each question was read aloud to the class and any difficult words/phrases were explained. Depending on the ability level of the group, testing took between 30 and 40 minutes. If a child was absent on the day of testing, efforts were made to return to the school at a later date and test these children separately. The procedure and materials were pilot tested by the research team during an earlier, feasibility study of ROE implementation. Figure 2 presents a flow diagram of teacher, pupil and parent responses through the trial. As can be seen, barring the seven schools that withdrew before the start of the trial, retention rates were good overall, with 1278 pupils tested pre test (583 control and 695 intervention pupils) and 949 remaining in the study at the final 3-year follow-up (76.3%) and included in the analysis (424 control and 525 intervention pupils, i.e. 74.3%). However, parental engagement was less successful, with only 686 returning data pre test (53.7% of the sample of 1278 children tested), reducing to 506 at the end of the study that were included in the analysis (234 control and 272 intervention parents, i.e. 39.6%).

FIGURE 2.

Flow diagram of recruitment and testing of children.

Sensitivity analyses

Two sensitivity analyses were undertaken to assess the reliability of the findings arising from the main analysis outlined above. The first involved comparing the effects of the programme that were estimated with regard to the two primary outcomes using the teacher-rated SDQ with the effects estimated using the alternative parent- and child-rated forms of the SDQ and also the CBS child-rated measures for prosocial behaviour and aggressive behaviour. This analysis assessed the robustness of the findings for the two primary outcomes and, also, with the parent- and child-rated measures, whether or not there was any evidence of bias introduced as a result of the use of teachers’ ratings, given that the teachers were not blind to the children’s participation in the evaluation.

The second sensitivity analysis explored how the findings may have been affected by attrition and, thus, by missing data. This was undertaken by rerunning the main analyses, conducted using only observed data, using multiple imputation.

Selection of the sample

A sample of six of the intervention primary schools were selected purposively from across four of the five health and social care trust areas in Northern Ireland. One of the selected schools declined to take part and another school was substituted, matched as closely as possible to the characteristics of the original school. The six schools selected were all intervention schools. The six case study schools were selected purposively to represent:

• the main types of schools (controlled, Catholic maintained and integrated)

• the trust areas (Southern, South Eastern, Belfast and Western)

• a mix of urban/rural schools

• size (large versus small)

• different catchment areas in terms of socioeconomic background.

Ethics, consent and data analysis

Organisational consent was sought via a letter to the principal of each of the six case study schools, informing them of the purpose and nature of the research and outlining what taking part would mean for them and their school. This letter was followed by a telephone call to each principal to address any concerns or queries and to determine principals’ levels of interest in taking part. After principal consent had been obtained, parental and participant consent was sought.

To explain the study to the parents of the Year 5 children, the research team wrote a letter that was sent to parents via the school. The letter explained the study in non-technical terms and invited the parent and child to take part. It outlined the aims of the study, explained what was involved if the parent agreed that their child could take part and highlighted that fact that the participation of the child was voluntary.

Informed verbal consent was sought directly from the children. Children were provided with clear information on consent and confidentiality. Care was taken to make the information understandable, and the children had the opportunity to ask questions about the study. Following this, they were given the opportunity to consent to take part in or to withdraw from the study.

The data from the interviews were analysed using thematic analysis following the approach outlined by Braun and Clarke. 49 Thematic analysis is a flexible and descriptive method that allows the emergence of a narrative to formulate the important features relevant to the research questions. With prior consent, the qualitative interviews were tape-recorded and transcribed verbatim, and field notes were written up from the lesson observations. To perform the thematic analysis adequately, the coding programme MAXQDA 10 (VERBI GmbH, Berlin, Germany) was used. Six initial categories were created to reflect the research questions of this evaluation, namely:

1. benefits

2. programme content

3. mentoring and support

4. limitations

5. main challenges

6. parental involvement.

Themes that emerged were identified, and illustrative quotations have been used in the report. The observational data were analysed in a similar way to those from the interviews and focus groups, with a thorough process of reading, categorising, testing and refining that was repeated by the researcher until all emerging themes were compared against all of the observations.

Methods overview

The base-case analysis compared the ROE intervention group with the usual classroom activities control group in terms of (1) costs incurred over the 3.75-year period and (2) quality-adjusted life-years (QALYs) gained over the 3.75-year period. Data were collected at five time points:

1. pre test (baseline)

2. post test (after intervention completion)

3. 1-year follow-up post test

4. second-year follow-up post test

5. third-year follow-up post test.

The analysis had a time horizon of 3.75 years (45 months), which equates to 3 years’ follow-up after intervention completion, as described above. A cost–utility analysis was undertaken, with costs considered from a public sector perspective (2014 GBP) and health outcomes measured by QALYs. Health utilities were measured using the Child Health Utility – 9D (CHU9D),46 a health-related quality-of-life measure designed specifically for children. All analyses were performed on individual patient-level data, taking clustering into account, and collected from the ROE trial. Table 4 describes the data collected for the economic evaluation.

Costs and QALYs were discounted using NICE’s recommended public health economic evaluation discount rate of 1.5%. 52 Missing data on costs and QALYs were handled by multiple imputation with chained equations. 53 Regression methods were used to obtain incremental cost and effect estimates. Multiple regression methods that ignore clustering (e.g. the within-school clusters as in this trial) can lead to biased coefficients and, especially, biased standard errors. 54 Multilevel models have been proposed as a method to address issues surrounding clustering in economic evaluation,9 and their use was explored. On recognition of the model being a poor fit for costs in this particular data set, regression with robust standard errors was conducted to adjust standard errors by indicating that observations within schools may be correlated but are independent between schools.

Incremental cost-effectiveness ratios (ICERs) were estimated by dividing the difference in mean costs between groups by the difference in mean effects between groups:

The uncertainty surrounding the ICER was investigated by use of a non-parametric bootstrap of 1000 iterations. This uncertainty was then presented on the cost-effectiveness plane and summarised on the cost-effectiveness acceptability curve. ICER estimates were compared with the £20,000–30,000 per QALY threshold generally accepted by NICE. 55 To allow for uncertainty, a series of sensitivity analyses were performed. All analyses were conducted as intention-to-treat analyses and performed in Stata/SE version 14.1 (StataCorp, College Station, TX, USA).

Methods

Quality-adjusted life-years

The QALY is a generic measure of health that combines life expectancy with health-related quality of life and is defined as a year lived in full health. 60 QALYs are calculated by weighting length of life by health-related quality of life. In this trial health-related quality of life was measured using the CHU9D, which is a generic preference-based measure specifically designed for use with children to estimate QALYs for economic evaluation of programmes/interventions for young people. 61 It was originally developed for younger children (aged 7–11 years) and scored using weights based on UK adult general population values (n = 300). 46 Since then, an alternative value set has been developed for use with adolescents (aged 11–17) using weights based on Australian adolescent population values (n = 590). 62 For our base-case analysis, the CHU9D was scored using the original UK value set. In this context, QALYs should be interpreted in the same way as the outcome of any population health intervention. ROE QALYs reflect the quality-of-life gains achieved from the intervention’s aims to increase social and emotional understanding, empathy, promote prosocial behaviours and decrease aggressive behaviours.

Missing data

Health and resource use costs for children were measured using parental self-report. The health and resource use questionnaire (see Appendix 4) was posted home to parents, who were asked to fill it in and return it in a stamped, addressed envelope. Health and resource use data were available for the second- and third-year follow-ups. A descriptive analysis of missing data was first undertaken to identify an appropriate analysis method to deal with the missing data. The missing data analysis follows recommendations set out by Faria et al. 63 for handling missing data in a cost-effectiveness analysis.

Missing data mechanisms are often categorised using Rubin’s framework for missing data. 64 ‘Data missing completely at random’ assumes that missing data do not depend on the observed and unobserved data values, and thus that the missing data vary indepedently of these. If data are missing completely at random, a complete-case analysis is valid. In complete-case analysis, only individuals with complete data at each follow-up point are included in the analysis. This is an inefficient use of the data because any individuals with missing follow-up data are dropped from the analysis. 63 Available case analysis makes more efficient use of the data by calculating costs and QALYs by treatment group at each follow-up point. They are then summed by treatment group over the whole time horizon of the study. A limitation is that different samples of costs and QALYs may be used, which can lead to non-comparability and affect the covariance structure. 65 ‘Data missing at random’ is a less restrictive assumption, as missing data depend only on the observed data and not the unobserved missing data. Multiple imputation is an appropriate analysis strategy for dealing with missing at random data. Data are not missing at random when the probability that data are missing depends on unobserved values.

A descriptive analysis of the proportion of missing values by group and in total was undertaken, along with the range, mean and SD of the observed data. Patterns of missing data were explored using the Stata ‘misspattern’ command. Logistic regression was performed to explore if baseline covariates were associated with the probability of data being missing. A dummy variable indicating missing data was created for overall costs and QALYs. Logistic regression was conducted with baseline covariates including gender, year group, multiple deprivation and number of siblings. A significant association between a baseline covariate and missing data indicates that data are not missing completely at random. 63 Dummy variables were created for costs and QALYs at each time point to explore the association between missing data and observed outcomes. Each indicator variable was then regressed on all other costs and QALYs observed in each year (i.e. missing baseline QALYs were regressed on costs and QALYs in each subsequent follow-up). Data were then assumed to be missing at random, in which case multiple imputation is an appropriate method of analysis for missing at random data.

Analysis

Regression methods were used to estimate the incremental difference in cost and QALYs while simultaneously adjusting for baseline characteristics that were the same covariates used in the imputation model. Generalised linear models were selected because of their advantage over ordinal least squares and log models, in that they model both mean and variance functions on the original scale of cost. 66 They also take into account the typically skewed nature of cost and QALY data. 67 As cost data are typically right-skewed, a right-skewed gamma distribution is appropriate. As QALYs are typically left-skewed, the QALY decrement (described in Quality-adjusted life-years) was analysed with a gamma distribution. Thus, both costs and QALYs were analysed with a generalised linear model specifying a gamma family and identity link. Cost and QALY decrements were adjusted for the following covariates: gender, year in school, intervention allocation, number of siblings, school, trust and deprivation level. Baseline health-related quality of life was also included to adjust for any imbalance of health-related quality of life between groups. 68

The mean costs and QALYs for each group were presented using the method of recycled predictions. 66 Incremental costs and QALYs, along with their corresponding robust standard errors, were reported from results of the generalised linear model. The ICER was estimated, and uncertainty surrounding the estimate of incremental costs, QALYs and ICERs was investigated by use of a non-parametric bootstrap of the cost and effect pairs for 1000 iterations. This uncertainty was then presented on the cost-effectiveness plane and a 95% CI of the bootstrapped ICER was calculated. The results were summarised using a cost-effectiveness acceptability curve to reflect the probability of ROE being cost-effective at various willingness-to-pay thresholds. The thresholds varied from £0 to £50,000 per QALY, reflecting the range generally accepted to be considered cost-effective by NICE (£20,000–30,000 per QALY gained).

Partnership meetings

From the outset of the trial, the research team has attended and fully engaged with the partnership meetings with staff from the health trusts, staff from education library boards and principals from schools participating in the trial. Each meeting has been attended by the lead health programme co-ordinator from the health trust, the pupil personal development officer for the education and library board and usually 6–8 principals from intervention schools. This forum has maintained a schedule of twice-yearly meetings throughout the trial (5 years). The research staff established links at the start and then continued to have consultations with the forum attendees throughout the research period. The aim has been to help influence the research at an early stage of development, to raise awareness of the research and to support the schools’ involvement in the process.

Stakeholder members of the Trial Steering Committee

Alongside the above partnership meetings, key stakeholders, comprising staff from the health trusts and staff from education and library boards, as well as the Public Health Agency, have also contributed directly to the evaluation as members of the Trial Steering Committee. Critically, this has included contributing to the emerging interpretation of the findings and the development of the dissemination strategy.

Process evaluation

The research team has given particular prominence to the process evaluation element of the study. As described above, this has involved in-depth engagement with all of the key stakeholders to ascertain their experiences and perspectives of the programme. The rich qualitative insights gained from the teachers, ROE instructors, parents and pupils are demonstrated clearly in Chapter 4 of this report.

End-of-project consultation meetings

Towards the end of the trial (May–June 2016), discussion groups were conducted in three of the intervention schools to provide preliminary feedback on the findings of the study in order to ensure stakeholder engagement in the interpretation and dissemination of the core findings. The consultations involved nine interviews, and, in addition, a focus group was held with pupils. Overall, as part of this phase, the research team talked with three principals, two teachers, two ROE instructors, two parents and a group of eight pupils. A purposive quota sample of three schools was recruited, which, within the available budget and time frame, represented the different subsectors of the diverse primary school population in Northern Ireland. The sample was chosen to include schools of different sizes, schools from rural as well as urban locations and schools from three of the education and library boards. These key stakeholders continue to be consulted regarding the development of a regional dissemination strategy, including a national dissemination seminar in Belfast.

Dissemination events

A regional launch of the findings of this evaluation took place in September 2016. This attracted over 100 attendees representing participating schools and a range of voluntary and statutory organisations in the region. This event was planned with the Public Health Agency and in ongoing consultation with members from the Partnership Meetings and those recently engaged through the end of project consultation meetings.

A further regional event, aimed at schools, was held in November 2017, by the Centre for Evidence and Social Innovation at Queen’s University Belfast. This event was also co-organised with the Public Health Agency and the Department of Education and included presentations on the findings of this evaluation and another evaluation of an SEL programme undertaken by the Centre for Evidence and Social Innovation. Further regional events are planned for 2018.

In addition, the PHA has funded members of the present research team to undertake a broader systematic review of school-based universal SEL programmes for children aged 3–11 years who have been registered with the Campbell Collaboration. 74 The findings of this present evaluation, together with the forthcoming findings of the systematic review, will be used to inform future policy and practice with regard to the promotion of children’s social and emotional development in Northern Ireland.

Secondary outcomes

With regard to the secondary outcomes, after controlling for pre-test scores, it can be seen from Table 13 that children who participated in the ROE programme were able to report a greater number of reasons for why babies cry (effect size = +0.24; p = 0.01). It is important to note, however, that part of the intervention involves explicitly teaching children about how babies communicate and why they cry (it is one of the nine themes involving three lessons). It is conceivable, therefore, that this measure is treatment-inherent and biased in favour of the intervention group. Furthermore, the effect is small, given that the findings suggest that the children in the intervention group are recalling only less than one reason more than the children in the control group for why babies cry.

No evidence of any differences between the groups was found in relation to the other secondary outcomes. Full details of these models can be found in Appendix 2.

Exploratory subgroup analyses

Prespecified subgroup analyses were undertaken to explore whether or not the programme worked better according to:

• the child’s gender

• the socioeconomic background of the child’s family (measured using NIMDM ranking for the child’s home address)

• the number of siblings.

Given the large number of significance tests undertaken in relation to exploring interaction effects for each of the above in relation to all of the primary and secondary outcomes listed, these analyses need to be treated with caution and considered as essentially exploratory in helping to identify potential patterns that may be useful to consider in future research. To provide an overview of the findings of these additional analyses, Table 15 sets out the statistical significance for each of the interaction terms added to the original models (as described in Appendix 2) for each of the three subgroup variables. This provides an indication of whether or not there is any potential evidence of subgroup differences in the effects of the programme for each of the outcome variables.

As can be seen, only two interaction terms were found to fall below the threshold of p = 0.05. Given that there were 27 tests in total, these findings could have occurred by chance. Moreover, and as can also be seen, there is no clear or consistent pattern to these findings to provide at least some suggestion that these may represent underlying differential effects. Therefore, it was decided not to explore the nature of these isolated subgroup differences any further.

In addition, it was found that ROE was delivered with high fidelity in all intervention schools (see Chapter 4). As a result, there was insufficient variation in fidelity to assess whether or not fidelity was associated with differences in outcomes achieved.

Further exploratory analysis

In addition to the above prespecified exploratory analysis, the following further analysis was undertaken in response to a number of queries raised by key stakeholders. This further analysis was not prespecified in the original analysis plan and should, therefore, be regarded as simply providing contextual data that may help in the interpretation of the findings from the main analysis. The analysis focused on three queries raised:

1. Is the baseline social and emotional functioning of children in Northern Ireland comparable to that of children in the UK, in Ireland and internationally?

2. Was there a difference in the duration of Personal Development and Mutual Understanding (PDMU) lessons delivered between control and intervention classrooms?

3. Does the programme work better for children who have poor prosocial behaviour to start with?

Primary outcomes at 12-, 24- and 36-month follow-up

Tables 18–20 summarise the findings of the analysis of the effects of the ROE programme at 12, 24 and 36 months following the end of the intervention, respectively. As before, full details of each of the models fitted are provided in Appendix 2. As can be seen, after controlling for pre-test scores, the potentially positive effect found immediately post test for prosocial behaviour appears to have disappeared after 12 months, with no notable differences between the scores of those in the intervention and control groups at any of the subsequent follow-up time points (at 12, 24 or 36 months post intervention).

However, and in relation to total difficulties (as measured by the teacher-rated SDQ), the effect size immediately post test appears to have been maintained at the 12-month (g = –0.14), 24-month (g = –0.13) and 36-month (g = –0.14) follow-up time points. However, and because of the reduction in sample size due to attrition, this effect is not statistically significant and so needs to be treated with a degree of caution.

Secondary outcomes and exploratory analyses

As is also evident from these tables, no evidence of any differences between the intervention and control groups was found in relation to the other secondary outcomes at any of the follow-up time points (with the exception of quality of life at 12-month follow-up).

In addition, Tables 21–23 summarise the evidence of any subgroup differences in relation to the effects of ROE with regard to the three prespecified comparisons for child gender, family socioeconomic status and number of siblings. As can be seen, the picture is essentially similar to that immediately post test, with very few interaction terms proving to be statistically significant. Moreover, of the terms that are significant, there is no clear or convincing pattern. Therefore, it can be concluded that there remains insufficient evidence of any subgroup differences in the effectiveness of the ROE programme at each stage of follow-up.

Sensitivity analysis for missing data

As noted earlier and summarised in the flow diagram for the study (see Figure 2), the trial has experienced some attrition of pupils and schools since baseline pre-testing. To test whether or not this attrition has introduced any bias to the findings, the main statistical models used to estimate the effects of the programme for the primary and secondary outcomes, across all time points (immediately post test to 36-month follow-up), were rerun with imputed data.

Multiple imputation was employed using chained equations, using Stata version 14.1. Imputed data sets were created using all of the outcome variables at all five time points (pre test through to 36-month follow-up), multiple deprivation scores and the fully observed variables for gender and trust location. Twenty imputations (m = 20) were performed for the purposes of the analysis. Multiple imputation was also performed separately by allocation (intervention and control groups).

Comparisons of the effects reported above from the main analysis at each time point with those estimated using multiply imputed data are provided in Table 24. As can be seen, the findings using the imputed data sets are broadly similar to those using just the observed data.

Missing data

In total, 38% of resource use questionnaires were returned for the second-year follow-up and 29% were returned at the final third-year follow-up (Tables 25 and 26). Some variables did not provide any data as nearly 100% of the data were missing. Such variables were subsequently dropped from the analysis; these included days off work because a child was home from school, other resource use and medications.

Missing data followed a non-monotonic pattern (Figure 3) because data may be missing for an individual in one follow-up but then return in subsequent follow-ups. Here the missing completely at random assumption would be inefficient because data from subsequent follow-ups would not be utilised and all non-complete cases would be dropped.

FIGURE 3.

Pattern of missing data in (a) costs and (b) QALYs.

Clustering in economic evaluation

A simple multilevel model of cost was fit, but owing to issues with the design of the trial (i.e. resource use was only collected at second- and third-year follow-up), the data did not fit this type of model as there were only two time points for cost. The ICC was estimated for cost, and it was low, at 0.0055. The low ICC was deemed to have only a very small design effect for this outcome, so robust standard errors were reported within the generalised linear model regressions to account for clustering in the uncertainty estimates.

Costs and quality-adjusted life-years

The total cost of providing ROE over one academic year to 33 schools in four out of the five health and social care trusts throughout Northern Ireland was £133,866. The average cost was £4057 per school and £175 per pupil. Table 27 summarises the intervention costs.

In addition to information on the costs of the intervention, information on public sector service use was collected. The resource use unit costs are reported in Table 28. Most resource use costs were valued using unit costs from the Personal Social Services Research Unit’s Unit Costs of Health and Social Care 2014. 59 The mean cost (including intervention and resource use costs) for the ROE group was £1181 and the mean cost for the control group was £1028. The incremental cost was £153 (95% CI £14 to £292), significantly higher for ROE (p = 0.032). The additional cost of the intervention is the main cost driver in this incremental cost.

The mean QALY gain in the ROE group was 3.0908 versus 3.0748 for the control. The incremental QALY gain of 0.0160 (95% CI –0.0143 to 0.0462) was not statistically significant (p = 0.300). The results of the base-case cost–utility analysis, as well as those of the sensitivity analyses, are reported in Table 29.

Cost-effectiveness

The ICER was £9571 per QALY gained (95% CI –£87,776 to £106,676) (see Table 29). Uncertainty around this estimate was explored through bootstrapping. The cost-effectiveness plane is presented in Figure 4. At a cost-effectiveness threshold of £20,000, ROE had an 83.1% probability of being cost-effective. This probability rises to 90.1% at a threshold of £30,000. The cost-effectiveness acceptability curve is presented in Figure 5.

FIGURE 4.

Cost-effectiveness plane representing 1000 bootstrapped cost and QALY pairs.

FIGURE 5.

Cost-effectiveness acceptability curve showing probability of ROE being cost-effectiveness compared with the control.

Discussion

During the trial period, the base-case analysis indicated that the ROE intervention incurred a mean additional cost of £153 (95% CI £14 to £292) per pupil. Utility, as measured by the CHU9D instrument and combined with duration to calculate QALYs, showed no significant QALY difference between the groups at an incremental QALY gain of 0.0160 (95% CI –0.0143 to 0.0462). However, these analyses do not capture any spillover effects, such as QALY impacts on parents and siblings, and other children coming into contact with participants. Given the direction of QALY gain, these impacts are likely to be positive, if anything, and thus spillover effects would most likely strengthen the cost-effectiveness result. QALY gains in other areas of child health-related utility research are often small and insignificant; however, economic evaluation methods still use such estimates to explore the probability of cost-effectiveness when combined with the cost of achieving these gains. When applied across a population, even small QALY gains can be highly cost-effective. A recent study looking at a family-based childhood obesity treatment used the EuroQol-5 Dimensions youth version (EQ-5D-Y) to measure QALYs. 81 The authors reported a non-significant QALY gain of 0.03 (95% CI –0.04 to 0.10). Another recent study for an asthma intervention in children used adult EuroQol-5 Dimensions QALY estimates. 82 They found a difference in mean QALYs of –0.00017 (95% CI –0.00051 to 0.00018).

This research adds to the evidence from other studies, which have used other outcome measures including mental health, empathy, perspective taking and SDQ, showing that ROE is effective immediately post intervention. 20–26 However, most studies had no follow-up post test and the only published study that did follow up pupils (3 years post test) similarly found no significant differences in effect after 3 years of follow-up. 20 Although the QALY differences between the arms of this randomised controlled trial were not statistically significantly different, the majority of the incremental points lie in the north-east quadrant (see Figure 4), indicating a more costly yet more effective intervention. This leads to a high probability of ROE being cost-effective within the £20,000–30,000 per QALY threshold.

The cost-effectiveness analysis of the SDQ total difficulties score (sensitivity analysis 1) was the only effect that was statistically significantly different at the final follow-up between groups. This perhaps reflects the fact that the SDQ is the most sensitive for detecting changes in social-emotional well-being, the main outcome ROE intends to improve.

The CHU9D is appropriate for a QALY framework, but many of its dimensions would not have been affected by ROE (e.g. pain and daily routine). Therefore, its appropriateness for detecting change in social-emotional well-being is questioned. It does, however, capture a generic health improvement. Its nine dimensions – worried, sad, pain, tired, annoyed, school work/homework, sleep, daily routine and ability to join in activities – capture an overall improvement in functioning. One of the hypothesised health outcomes of ROE is that it decreases aggressive and bullying behaviour, so if fewer children are being bullied that may be evidenced in the worried, sad, pain, annoyed, sleep and ability to join in activities dimensions of the CHU9D. The CHU9D is the only health-related quality-of-life instrument designed for children and valued by adolescents, and, thus, it was the best choice for measuring QALYs in children. Other health-related quality-of-life measures for children exist; however, they are usually adapted from an existing adult measure (16D),83 are valued using adult values (the EQ-5D-Y84 and the Health Utilities Index Mark 285) or have not been valued at all but are mapped to an adult measure [Pediatric Quality of Life Inventory86 (PedsQL)]. This is because it has typically been very difficult to elicit children’s health preferences because of ethical and cognitive difficulties. Time trade-off would involve asking children about death, and the ethics of such an activity is questioned. It is also a cognitively challenging task that may not be appropriate for children. It should be noted that the use of different tariffs to score the CHU9D does result in different cost-effectiveness results, with the use of the alternative tariff62 resulting in an ICER of nearly £1200 lower than the original tariff. 46 The use of annuitisation and the assumptions around the useful life of the intervention do have an impact on the cost-effectiveness results. Sensitivity analysis 4, when there was no annuitisation, resulted in an ICER of £13,909 versus £10,309 and £9571 when costs are annuitised over 3 and 5 years, respectively. In this study, the choice between a 1.5% and 3.5% discount rate minimally affects the cost-effectiveness results.

The available case analysis demonstrated the most conservative estimate, with greater incremental costs and lower incremental QALYs resulting in the highest ICER estimate and lowest probability of cost-effectiveness for ROE. However, the ICER of £19,626 for the available case analysis is still well within the limits that NICE typically considers cost-effective. In fact, all reported results of the sensitivity analyses would typically be considered cost-effective by NICE; however, it is important to note that the threshold of £20,000–30,000 per QALY gained is from a NHS and Personal Social Services perspective. If ROE were to be rolled out to schools across Northern Ireland, it is likely that the cost of providing the programme would largely fall on schools or local education authorities, and their willingness to pay for the programme may be very different from that expressed by the current NICE-supported threshold.

The health and medical fields have long used cost–utility analyses to aid policy decision-making. Without such analyses, there is a risk that decisions will be made based on emotional appeal, absolute intervention cost and political pressure. 87 This cost–utility analysis (base-case analysis or sensitivity analysis 0) provides initial evidence that school-based population health interventions are feasible, are likely to be cost-effective according to current thresholds and can be employed to aid decision-making.

Limitations

Ideally, data on resource use would have been collected at each data collection time point. It was recognised that recall bias was likely as a result of the long recall periods for estimating resource use expenditure; however, the alternative was to completely forego collecting any resource use information for the trial. The lack of consistently collected resource use data was the main limitation of this cost–utility analysis, which also had a limiting effect on the choice of analytical methods employed.

The available resource use data were also limited by large numbers of missing data. Variables with the largest numbers of missing data may have been affected by a survey design effect as they were all questions that were self-reported using free-form text. Therefore, a detailed descriptive analysis was employed to determine the appropriate assumptions around the missing data, and missing data were subsequently handled using multiple imputation. Future evaluation work of school-based population health interventions should be mindful of potentially large numbers of missing data, particularly those that are collected from parents by post.

It would have been useful to explore the longer-term impacts of ROE by modelling potential impacts over the child’s lifetime. However, there is a paucity of longer-term evidence using the main outcomes of our analysis, the SDQ and CHU9D, especially the latter, which is a relatively new generic health-related quality-of-life measure. Additionally, the lack of statistically significant difference in effects at the third-year follow-up meant that any potential longer-term benefits would have significant assumptions and uncertainty attached.

Teachers’ perspectives of the limitations of Roots of Empathy

Although teachers were, on the whole, very positive about the ROE programme, they did report a number of perceived limitations, with three key issues reported in particular. First, some mentioned the fact that the programme would not continue after the current year:

It’s just that it’s only 1 year. I think that’s the limitation. I think that it needs to be picked up on again. I think actually the other staff here aren’t doing it to . . . like, the P6 teachers should have more of an awareness of what’s going on to be able to carry that on. Because I would worry that it would just be something that happened in P5.

Teacher

Second, teachers referred to the programme as being quite rigid and unimaginative at times, with too much repetition of certain activities, such as artwork:

It can be very prescribed and set in stone.

Teacher

I feel that maybe the baby end of things is very good and a lot of the lessons are very good, but some of the activities are very repetitive. They’re drawing a picture a lot and discussing their feelings and that.

Teacher

There’s one child in the class who actually said – he probably is on the autism spectrum and he said – ‘But I just did that’. You know, because he just says it as it is, ‘But I did that before’. And I could see where he was coming from, because a lot of the drawings or whatever they have to do, they could be repetitive.

Teacher

I know the person giving them would expect that maybe they would draw a different picture or discuss about it a different scenario but very often some of the children would use the same scenario so therefore they’re drawing the same thing. So I think it falls down a wee bit there.

Teacher

I don’t know what child care courses are like, right, but I do feel that a lot of it is sort of child caring, minding the baby and watching the baby, and I just feel that our curriculum addresses a lot of these areas. It sort of in a way is a wee bit repetitive and with a little bit of imagination it could be extended in a worthwhile way.

Teacher

Finally, the consensus from all of the teachers was that parents did not know much about the programme and yet were crucial to its success. This is reflected in the following comment:

It would be lovely for parents to actually see more or hear more; get more feedback, so they can see what their actual children are learning. Because, really, they’ve got their questionnaire, which gives them their information, and that’s really all they know about it.

Teacher

Another teacher commented that:

Some parents ask me about the baby because their child goes home and talks about it. There is no real direction on how we should involve parents, so that’s a bit ad hoc.

Teacher

School principals’ perspectives of the limitations of Roots of Empathy

When asked about the limitations of ROE, many principals had very strong concerns about the lack of resources in schools and insufficient future funding for sustaining the programme. Other principals supported this, as they felt under pressure to release teachers from their usual duties so that they could take on the instructor role, and also because they had to find funding for cover for those teachers’ classes. Principals also considered time constraints a drawback of the programme; for example, they noted the need for preparation time so that the lessons could be built into the timetable, as well as an opportunity for the instructor and teacher to discuss each lesson in advance and then reflect on it afterwards. Some of the principals felt that it might not always be possible to give adequate time to this, owing to the pressures of other workloads on the instructor and teacher:

I would say the limitations are that I can’t afford to have the likes of [instructor] to every class in the school. If I had my way and I had the money, I would have her deliver it in every class. I might not be able to get enough babies but I could see benefits of this right throughout the school. It would be getting the money to do this and then sustain the programme.

Principal

Resources, and funding, and release of teachers, and substitute cover, and back to that whole issue of resources and funding to facilitate that, without appearing to be negative, which I don’t want to be, because the programme is so positive.

Principal

Well the only negatives I would say here would be the funding, the possible funding. Well I’m thinking of the bigger picture, not our school, about overemphasis on data and league tables, etc., rather than this.

Principal

A limitation discussed by one principal was that, owing to the fidelity of the programme, the school felt restricted in being able to involve parents. Another principal stated:

Limitation as in it would be good to be able to communicate more with the parents and share activity but, at the minute, we can’t because everything has to be vetted, you send. . . if you want to do anything in the local school newsletter, it has to be sent away and that really is not very practical so I think if it was really shared a lot more in the class newsletters going out, it would really strengthen the whole process but because it’s just kept so insular, I would say that is a limitation of it.

Principal

Instructors’ perspectives of the limitations of Roots of Empathy

Most of the instructors had views on the programme’s limitations. Some of the issues discussed were time constraints, the prescriptive nature of the programme, school issues, age appropriateness of materials, and cultural adaptability of the programme.

That it’s very rigid.

Instructor

You have to stick to it.

Instructor

One limitation is when they do all start to talk, where do you cut it off? When you’ve only got a certain timeslot in your day, and you don’t like to say, ‘OK’. And I had to say it a few times, ‘OK, the last one, the last one.’ That’s the problem with it too, just time constraint.

Instructor

I think for the age group that I’m using, I think it’s a bit too childish. There were things about losing teeth and although the children have been through it, I think there is maybe room where you could use some lessons from the older age group for the P5 age group.

Instructor

The books, well that book today, because it was American, very American like starting off, I don’t what. And like when we were going through different things like diapers and pacifier, and there were words that they knew from TV. But I think if we were going to use it in Northern Ireland, we might need to put another slant on it.

Instructor

Yeah. The limitations are, I suppose, trying to make sure that it’s not just the 1 hour, and that it’s right through the rest of the week.

Instructor

Well I think one of the difficulties is the content, there’s so much content, and the lessons aren’t that long. I always feel I’m whizzing through things a lot. To me that’s the main drawback really.

Instructor

One of the instructors stated that she did not see any limitations of the programme:

No, none whatsoever.

Instructor

Many of the instructors noted that most of the limitations were related to the programme being in its first year of implementation. They saw this first year as a ‘learning curve’ and felt that implementation in the second year would be easier in terms of time and familiarity with the materials:

I think after the first year I will be more familiar with the lesson and will actually have all my lesson plans available. I will not have to spend hours preparing and writing.

Instructor

Next year I think I will be able to focus more on the interactions with the baby and the children. This year I am very comfortable with the programme but sometimes you are checking your notes because you don’t want to leave anything out.

Instructor

A fundamental issue raised by two of the instructors was the possibility that a child could make a disclosure and that very little guidance and instruction is given about this:

I know, and in some parts, when we’re talking about all this nurture and things, some children have said, ‘Oh, that wouldn’t happen in my house’, or, ‘We just leave the baby’. And sometimes there’s a thing with it, disclosures and things. If a child in the middle of that, I think there might need to be a bit more done. I know a teacher knows.

Instructor

And I suppose anybody who’s in the social work background would know that. But maybe there just needs to be a wee bit more awareness that there could be a disclosure, and how that’s dealt with by an outsider coming into it.

Instructor

Children’s perspectives of the benefits of Roots of Empathy

When asked what they had learned from the programme, the overwhelming majority of the children from all of the groups enthusiastically referred to the baby. The children spontaneously linked learning outcomes from the programme with concepts relating to child development, milestones and caring for the baby. As illustrated by the following representative sample of comments taken from across the focus groups, the children were keen to convey their knowledge of the key principles of looking after the baby, such as safety, crying, communicating, and meeting and greeting:

It’s about learning how a baby would grow up and seeing a baby grow up.

Child

Different changes from a baby and like its milestones and what it can achieve.

Child

It teaches you how to care for a baby and all that.

Child

How to make your home baby proof.

Child

That babies’ teeth comes 6 or 5 weeks after they are born.

Child

Don’t put a baby at the top of cot just if they have a blanket they could wriggle down and it could suffocate them.

Child

Never shake a baby because it can damage their brain and they can have problems like spinal injuries or something like that.

Child

Many children across the focus groups displayed understanding of a range of reasons why a baby might cry. In a few of the groups, children indicated that the programme had taught them about this and most children in all of the focus groups talked freely about why a baby might cry and gave solutions for helping the baby to stop:

And it tells you what the baby wants and what’s happened to him when he cries.

Child

If a baby cries it means that she needs something or something is wrong.

Child

It’s that just she’s sore or it’s like that she’s hungry or she needs the toilet or something else.

Child

Without prompting, some children in only a few of the groups related the learning from the ROE lessons to helping them to understand the feelings of others. This then led some to reiterate some responses and triggered other children to talk freely about how they had learned about feelings and emotions:

It’s like walking in someone else’s shoes.

Child

It’s trying to understand what other people feel like.

Child

I can understand my sister more when she has to do her homework and then I need her to help me I have to let her do what she needs to do first.

Child

Our different emotions and things like that.

Child

It’s helped me to understand my baby brother and sister better.

Child

Teachers’ perspectives of the benefits of Roots of Empathy

All of the teachers commented on their role in the ROE lessons as being unique, as they were observers rather than active participants. One teacher commented that she felt that she was in a privileged position as an observer in her classroom, as she was able to observe behaviours from the children that, as a class teacher, she would never have been able to because the school timetable was so busy. This teacher stated that:

I wasn’t really sure what my role was, but now I can see that just by being able to sit and listen and watch the children in my class is something I rarely get to do! One child in my class was talking about the content of her dreams during the lesson and I know there are difficulties going on in her home life. That really signalled to me to maybe just watch out and maybe even follow that up.

Teacher

All but one of the class teachers reported experiencing positive benefits and outcomes in the classroom and that there had been a change in some children’s behaviour as a result of the lessons. The following comments from teachers reflect this:

I think they show more empathy.

Teacher

I see a softer side come out in them.

Teacher

I can see progression with the children in my class.

Teacher

When the teachers were asked what they thought the main benefits of the programme were for the children, nearly all felt that the majority of children had benefited, with only one teacher stating that so far she could see few benefits for her class, although she was optimistic that these could become evident in the future. This important point is found in the following quotation:

They struggle to name their feelings and to generally express them and it comes out in other ways. I can see that they enjoy the ROE and having the baby in and we do talk about feelings more now. So there is little bit of improvement but it is still not great but I think there could be benefits further down the line.

Teacher

Some of the changes in behaviour suggested by the teachers were that children seemed to be more empathetic and were more able to identify and reflect on their own feelings and those of others, and that many of the children were showing a softer side to their personality. This is illustrated by the following comments:

The main thing would be that the children are more in touch with their feelings, and they are certainly more open to talk about their feelings than they were at the beginning of the programme and they do tend to see other people’s point of view more easily now than they did. I think there are less rows in the playground.

Teacher

They are a bit more considerate now of each other and really don’t want to end up in the principal’s office after calling someone a name on the playground.

Teacher

Some of these teachers made a link between the ‘experiential learning’ aspect of the programme and the children being more open in their discussions about their feelings. All of the teachers mentioned that the baby as the ‘teacher’ was a lever for the children to learn through observation and interaction with the baby and its mother, as reflected in the following comments:

The baby is the teacher in the class and when it comes in and it shows you its emotions, like say you and it like laughs and moves, it’s showing it’s like happy and its emotions to the children.

Teacher

This is real life. Very powerful.

Teacher

The baby is a 3D figure, she is real and the children really get this as opposed to learning these lessons from a book.

Teacher

The children just love the baby.

Teacher

As demonstrated by the following quotations, teachers also mentioned that children were more able to express their feelings, label their feelings with appropriate language and show more consideration to others:

I do feel they are showing slightly more empathy with each other. So if there was a child on the fringes, they know the right answers. Whether they actually do the right thing, you know, they’re still children, they still have to be prompted, so they know what they should be doing but I wouldn’t necessarily say that they’re automatically doing it any more than they would have done in previous years that didn’t have this programme.

Teacher

They’re definitely more open to their feelings to say, ‘I feel sad’, or, ‘I’m happy’, or why they’re happy.

Teacher

They do explore their feelings and in that end, yeah, their language would have changed.

Teacher

A key observation expressed by many the teachers was with regard to the perceived change in boys. The teachers were surprised at how well the boys engaged with the programme, particularly in being more open when talking about their feelings. As one teacher cogently put it:

It’s the boys that are showing their feelings a lot more. Girls will always have shown their feelings, and at that age group of girls, they tend to be very competitive and one of them wants to lead and be the leader and all that, so in many ways that hasn’t changed, which is a bit of a disappointment. But the boys are opening up more, that’s all the way I could put it.

Teacher

Similarly, other teachers commented on this important point:

Yes I think the boys have come round to the programme. I don’t think they have always been engaged.

Teacher

Some of the wee boys will just come out with one word like ‘sad’.

Teacher

At the beginning the boys wouldn’t hold the doll but now they just nurse it so naturally in the absence of the baby.

Teacher

Some teachers felt that younger children would have benefited from receiving the programme at an earlier age as it may have helped those who are now struggling to talk about their feelings and this quite often manifests itself in behavioural issues. One teacher said that she noticed only small positive changes in the classroom and that most of these benefits were seen in the boys. She continued to report that many of the children in the class, particularly the boys, had major behavioural problems and found it very difficult to express themselves; during the period of the fieldwork, one boy had been at risk of being expelled from the school for engaging in challenging behaviours that required disciplinary action. Many children had poor language skills and their emotional literacy levels were considered low or non-existent. The Primary 5 teacher expressed her exasperation by referring to the class as ‘a difficult class’:

I think actually ‘Seeds of Empathy’ would be something that I would be very interested in looking at, to get in younger, because we’re having kids in primary school who maybe don’t have the experiences of playing with other children than they might have done in the past as they are on PlayStations [Sony, London, UK], etc. So I think there can be difficulties for kids in school when it comes to turn-taking and co-operative play. I think it would be good for them to have the experience of that.

Teacher

Another teacher, who also thought that younger children would benefit from the programme, said:

I think the younger children in the school would benefit from this. The earlier these programmes are given the better the outcomes for them.

Teacher

The teachers were asked if they had noticed any changes in the classroom environment in general as a result of the programme. There were mixed reactions to this question, with some of the teachers commenting that, in general, the children were a little more sensitive and a lot of trivial arguments that used to take place between the children had stopped. As one teacher explained:

I don’t see as many wee silly arguments or they don’t keep going on and on, oh for example, ‘such and such a one fell out with me’. We used to try to work these problems through PDMU and we’ve related it now to ROE and they can identify with that.

Teacher

A few teachers who had said that some of the children in their class had behavioural problems commented that the programme had made a difference to these children. One teacher mentioned that there had been a serious issue with bullying in the class and commented on how the ROE lessons had helped the children to think more about their actions:

There were bullying issues going on, and I really feel that the programme highlighted those and it didn’t make anybody stand out, but it gave them a chance to reflect on what that was. And it worked as well for the person that was maybe doing the bullying or the teasing as it worked for the weaker person which I think is great.

Teacher

One teacher commented that the programme was a great tool to refer to if children were misbehaving in class. A few of the teachers felt that it was too early to see changes but again they were hopeful that there would be some before the end of the school year. This is reflected in the comment below:

I think it’s helped a bit but I think you’ll see more of it near the end, like already I’ve seen from February.

Teacher

Children learning emotional language was considered a major strength of the programme. One teacher shared that most of the children in her class did not have the language to express their feelings and instead often expressed them using anger and hyperactivity:

A lot of the children in my class don’t have the words to describe feelings even if they wanted to. They may know primary feelings like happy or sad, but as they often express sadness in anger, happiness in hyperactivity, and anger in temper, then translating their behaviour back to its possible underlying emotion can be helpful. It is difficult to communicate feelings with no basic emotional vocabulary.

Teacher

A few teachers commented on the fact that they did not really know what to look out for in terms of benefits to the children or how they should be building on the ROE lessons afterwards, and that this was definitely a weak aspect of the programme. Such findings raise key questions for this research study in terms of how teachers view their role. The following teachers’ comments reflect this:

I think at the beginning of the programme the P5 teacher has nil visibility on what’s coming, and from my understanding it’s deliberately kept that way as the teacher is not really supposed to engage other than scribe and assist the instructor. I do think that’s to the detriment of the programme, because if I had access to – and I understand that I can now but I couldn’t before – if I could have access to the handbook and even know, right, next week or for the next 4 weeks we’re going to be concentrating on keeping baby safe, I can link that into health and safety from a mathematical point of view, why does a measurement have to be exact if we’re going to be measuring doses of medicine for example.

Teacher

This is all new to me so I am not sure what the lesson is going to cover in advance. I could probably tie it in with other parts of my lessons with the children if I did know.

Teacher

I think we should have access to the manual, even a photocopy. We could help prepare the children for the lesson and certainly reflect with them afterwards if we had more information about the lesson. They are very strict about that. It is really only the instructor that has access to the manual.

Teacher

School principals’ perspectives of the benefits of Roots of Empathy

School principals were asked if they had observed any outcomes or benefits of the ROE programme in the children. All six principals from the case study schools said that they had observed positive benefits in terms of, for example, less aggressive behaviours, a more caring attitude, improved concentration and fewer bullying incidents. As one principal stated:

But I have definitely seen . . . because I would do dinner duty out there and, yeah, I’ve seen more of a caring . . . maybe a bit of caring coming. It can be a difficult for them, being able to express themselves.

Principal

Again, the gender difference was recognised by the principals, who made particular comment on improvement in behaviour in a few boys who were seen as ‘troublesome’. Those changes included children having fewer disputes in the playground and therefore being sent to the principal’s office less often. Some of the words principals used to describe the changes in the children’s behaviour and actions since the programme were ‘more caring’, ‘willingness’ and ‘softness’. This is reflected in the following comments:

There are one or two boys who’d be devilish and there’s nothing wrong with that but they always get involved in disputes. But I’ve definitely seen that there’s a willingness among them to be more considerate now. I must say on the playground now, I do the dinner duty every day so I’m out there and behaviour has improved.

Principal

We have found definitely that there’s softness has come, and it probably was there but boys are now not afraid to show it, if you know what I mean. Boys are maybe articulating their views more you know, on a playground or in a situation where both feel that they can say their piece and then it’s over and hopefully then the bullying will end or, you know, it’ll certainly help to stamp out the bullying.

Principal

Three of the school principals discussed how they felt the programmes fitted very well into the school curriculum, in particular how it could build and strengthen the pastoral care aspect:

Where a lot of time, as much as we do circle time, and you were talking through different situations and so on, it’s a much more sophisticated form of that.

Principal

The key building block of a good school is the pastoral experience. And thankfully in our school, ours was described as outstanding, so it’s something we’re very proud of. And we’re always trying to enrich that way of working in schools with ROE, or the newest programme.

Principal

Parts of ROE feed into PDMU and improve our pastoral experience.

Principal

Instructors’ perspectives of the benefits of Roots of Empathy

When asked about the benefits of the ROE programme for the children, all of the instructors gave very positive views. Their responses included comments on perceived improvements in a range of outcomes, including the children’s ability to show empathy, emotional literacy, behaviour in class and the playground, knowledge on child development and parenting skills:

I think there’s more tolerance in the original sense of the word tolerance not the condescending version of tolerance. I think they are a wee bit more tolerant.

Instructor

The playground, for example, the way they would normally behave. But then if I have to deal with an issue, they’re able to very quickly tell me how the other person would be feeling, and they can empathise, which some of them wouldn’t have done before. So it’s certainly a way in to dealing with issues.

Instructor

I feel they’re more receptive to talking about feelings too. They do seem to, through the course of the year, talk a lot more about feelings to me as an instructor, and I think the teachers would say that they see some of the children doing that. For some of those children they’re learning to speak up a bit more in the classroom learning, which you mightn’t see in other programs or other bits of the classroom delivery.

Instructor

The instructors said that it was difficult to identify improved language or behaviour at this stage:

But in terms of their development of empathy and their development of emotional literacy it’s hard to know really at this stage.

Instructor

The instructors talked about the family visit being a very special and positive experience for the children. The experiential learning gained from the family visit was recognised as this also encouraged the children to talk more about feelings:

Certainly the baby visits are the highlight, and that is where you can get more out of the children, and more language.

Instructor

In my opinion in our school the children, they just drink it all up. You know, so they can’t wait for it to come. And their retention of the information just amazes me, you know. Whether it’s because it’s something new or whether it’s because, you know, it’s parallel with the baby coming in – there’s a real live baby that they can demonstrate what they’ve learned with or what they’ve learned before they can then demonstrate it with the baby. It’s just that they think it’s brilliant.

Instructor

I think they now project their thoughts and feelings onto the baby, it gives a really nice way of understanding when it’s OK if the baby is feeling frightened because I sometimes feel frightened. And perhaps it’s going along the lines of the bully or the unhelpful or the nasty kind of person, that’s probably where it’s more targeted trying to understand, because they’re older and they see the baby younger and crying they might think, ‘Oh that’s . . . I don’t want to be that type of person. I want to be a nice caring kind of person’.

Instructor

One instructor said that even though the children in her class were poorly behaved most of the time in their normal class as reported by their teacher, they behaved reasonably well during the ROE class and, in particular, during the family visit. She also felt that this particular class had great difficulty describing or talking about their feelings and one reason for this was that they did not have the language to describe how they felt. This instructor felt that these children had progressed to using one word to describe how they thought the baby was feeling because of the learning from the ROE programme, and this was a very positive outcome for the class:

The class are a difficult class. Their behaviour is a lot better with me in the ROE and I really don’t have many negative situations. I don’t know whether there has been an improvement yet in their overall behaviour but certainly some children has more empathy for others. A lot of our children only use or answer questions with one word. We find that happens right throughout the school, they just say one word and then very often when we hear just the one-word answers we accept it. We are now trying to encourage more, like sentences from the children but it is hard. Some of the wee boys will just come out with one word like sad.

Instructor

Interestingly, all of the instructors commented that there were more notable changes in the boys in terms of being more open when talking about their feelings, and in the relationship that they developed with the baby:

The boys do engage with the baby so well even though they are all so macho.

Instructor

It’s the boys that – I just feel that the boys are showing their feelings a lot more. Girls will always have shown their feelings, and at that age group of girls, they tend to be very competitive and one of them wants to lead and be the leader and all that, so in many ways that hasn’t changed which is a bit of a disappointment. But the boys are opening up more, that’s all the way I could put it.

Instructor

Some of the bigger boys in P6 initially were saying, ‘We’re not singing’, but now they do.

Instructor

The boys love it.

Instructor

Parents’ perspective of the benefits of Roots of Empathy

There was a mixed response from parents when they were asked if they had noticed any changes in their child’s behaviour because of ROE. Among the parents interviewed, opinions were evenly split among those who noticed a change in their child’s behaviour, those who did not and those who were unsure. Parents who noticed positive benefits of the ROE lessons on their child’s behaviour, with the exception of one, had sons. Several parents said that they had not noticed any difference in their child’s behaviour because of the programme, and these tended to be mothers of daughters. They said they were not sure if this was because they did not really know much about the programme and therefore they were not really looking out for any changes in their child or if they had just not really thought about it. Parents had mixed views as below:

I think the main benefit of the programme, for my son, was that he has just become more thoughtful about the children around him. He would have been selfish at times but I noticed him starting to say sorry more to me for bad behaviour and becoming more affectionate.

Parent

Definitely it had a positive effect on him.

Mother referring to impact of the ROE programme on her son

My child is a very kind child and she would be very – I would say she would have quite a lot of empathy anyway. I’m sure it has benefited her to do it because I think the more things you do with them like this the better. I don’t know that I’ve seen massive differences, she would always be a very kind child; if someone was hurt or whatever she would always have taken the child in and looked after them and stuff. So she was already like that. That would kind of be her personality.

Parent

She loves babies. Her cousins . . . I mean, she’s wonderful with her younger cousins and, you know, all the uncles and aunts will always say that. In comparison to any other cousins her age she has always got an awful lot of time for them. Whether that’s something that’s been enhanced by the programme or not, I don’t know, but she’s definitely very good with younger children.

Parent

The benefits communicated by parents were related to increases in prosocial behaviours only. No parent suggested that his or her child was less aggressive because of their participation in the programme. Some of the prosocial behaviour changes included being more thoughtful, being more tolerant, showing increased affection and being apologetic:

Yes I think he is a bit more thoughtful to his siblings.

Parent

He is now wanting to nurse his baby cousin a lot more and is really quite affectionate. He is a very kind child anyway.

Parent

I think she maybe finds it a bit easier to say sorry now. Not sure if that is because she is getting more mature or whether its to do with what she is learning in school.

Parent

Well what I did notice from him was that when he was cheeky, he would come to me shortly after and apologise for what he had done and give me a hug, which he hadn’t done before. And that he would come and say, ‘I’m really sorry for shouting’ or ‘I’m sorry for doing that’ and give me a hug.

Parent

As illustrated below, one mother felt that her son had benefited greatly from the interaction with the baby, as he was an only child with autism who had a very low tolerance of babies, particularly when they cried:

I think it has been very good for my child because he is an only child and doesn’t get to see babies much. My son has autism and he doesn’t really like babies especially when they cry so I think this programme has helped him to be more tolerant.

Parent

Among the parents interviewed there did not appear to be a high awareness of the content of the programme or of what the outcomes or benefits may be for their children. However, all parents were aware that there was a baby visiting the class, because their children had gone home and talked about it or they had read about it in the school bulletin or on the school noticeboard. When asked what they know about the ROE programme, parents responded:

Ah I think it is lovely having the baby in the class. I’m not sure what it’s suppose to do.

Parent

Maybe indirectly. I’m not sure what I’d be looking for. I mean, she’s a very empathetic child anyway, she’s a very caring child anyway and I am not sure what the programme is suppose to be teaching them.

Parent

Is it about how to look after a baby? How to be a good parent?

Parent

Some of the parents reported that because their child had siblings they were used to being considerate towards other family members:

She’s the eldest. OK. So she has younger siblings. OK. Yeah. So whenever my daughter was in P1 we had a baby in the house so you know, she knows and remembers all about bathing the baby, changing the baby’s nappy, the baby’s crying, needs to be fed, you know, comforting the baby, all that sort of stuff she’s went through. So I wouldn’t say that it has made an awful lot of difference with her but I would think that maybe with my youngest child who has never had a baby in the house it may make more of a difference.

Parent

I didn’t really see any real changes with regards to my daughter. Usually she’ll come home and she’ll talk about the baby and, you know, what stage it’s at or it’s growing or it’s got bigger or whatever. I wouldn’t really say that it has made any difference in her manner or her view towards babies or anything like that, more so because of the fact that . . . Well, [she’s] 9 now and she has a smaller brother and a sister.

Parent

In contrast, other parents suggested that the programme had helped their child to be more considerate to siblings:

He didn’t. He wouldn’t be much of a talker but he would always have been quite an empathetic child anyway, but I suppose yeah it did help. So I suppose towards his other siblings he would be more.

Parent

Well I think the main benefits of the program, for my son anyway, was really . . . obviously when the baby came in. He has brothers and a sister younger than him so he would be used to some exposure to children and young kids. He got really into it in terms that he would talk about it and he would tell me everything that happened with the baby, I almost felt as if I knew the baby myself. So I think he did get very into it and was very excited about the programme and was looking forward to baby coming in to the classroom. So I definitely think that it did give him a bit more empathy for the children.

Parent

One parent expressed concern that her child had been bullied during the year that the ROE programme was running in her class. She was exasperated by the fact that the programme did not appear to be making any difference to the bullies’ behaviour or how the school was dealing with it:

I guess my issue with the ROE last year was that my child was still very badly bullied at the time by another child in the class, and we weren’t listened to at all by the school. It wasn’t a problem but there was a problem, a very big one, which they’ve now realised actually. They have now recognised it and they’re acting on it now. So I kind of was very disappointed because in my head I thought, ‘They’ve got this ROE thing. This is great; this is working with the children, it’s teaching them that when you’re nasty to someone else or whatever how will they feel? It’s kind of building that in’. And yet I felt that nothing was happening for my child. The reality was Yeah. ‘We’ve got this lovely programme, it’s great. When the baby comes in and isn’t that super?’. And it is, but why is nothing happening to stop this happening to my child.

Parent

In summary, the ROE programme was perceived by most key stakeholders in the case study schools to have positive benefits for the children; the children seemed to show more empathy, open up more about their feelings and, overall, have fewer disputes in the playground. Only one school felt that the children would have benefited more from the programme being implemented at an earlier age; this was because in that school, by the time the programme started, a lot of children had established very tough exteriors and did not have a wide emotional vocabulary. This meant that it was very difficult for these children, particularly the boys, to start talking about emotions and feelings. Teachers, principals, instructors and parents all commented that more benefits had been seen among the boys. The children mainly identified learning about babies and their development with the programme. Parents were generally unsure about the programme and the benefits it may have for their children.

School personnel’s perspectives on parental involvement

There was a disparity between the case study schools in terms of how, or if, they involved parents with the programme. However, all schools stated that they would have liked to involve parents more. When asked how they involved parents in the programme, school personnel and programme co-ordinators referred to a kind of ‘ad hoc’ involvement, as there was no overall strategy in the manual for engaging parents:

I don’t think we have done anything. There are posters around the school so the parents may see those but I don’t think there has been anything else.

Teacher

It took longer really for parents to find out what the programme was about, but I did have a call from one mum saying about feelings and her little girl had been involved in a very traumatic situation before that and she didn’t like this talk about feelings, she didn’t want to talk about feelings, and the mummy was saying, ‘What’s all this talking about feelings, and she’s not coping very well and it’s bringing stuff back for her’. So I went to see the mum and spoke to her and said to her, ‘OK, and this is hard but obviously she hasn’t dealt with the stuff that’s going on then’, you know. So she was still getting counselling for this particular situation at this time, so I said, ‘Look, can we stick through it and we’ll see?’. So the outcome on that was OK and the child did stick through it but initially the mummy wanted to take her out.

Instructor

I know that the schools would have written to all parents to tell them that their children were participating in the programme not asking for permission, but I do know that when it came then to the research some of the parents wouldn’t allow their children to participate in the research, am I right in saying that?

Programme co-ordinator

Well it’s very limited now I think. Certainly in the schools that I work in halfway through I do a wee news sheet and just take a photograph of the baby and say, ‘This is a letter to go home to your parents to say, “Look, this is our baby” ‘.

Instructor

The required strict fidelity to the programme and the media guidelines supplied by programme developers were considered ‘off-putting’ reasons for deciding against writing to parents or inviting them to take part in the programme during the year. Conversely, a barrier mentioned by school personnel was that parents might be too busy to be involved or that some could be struggling with literacy and so would have difficulty reading information sent home. Some schools identified opportunities during the year to inform parents of the programme’s progress, and these schools tended to align this communication with their own practice for engaging parents. For example, some of the case study schools held school assemblies and distributed school bulletins that parents had access to, and the ROE updates were included in these:

I remember in one of the newsletters, I think the principal put a little bit about the Roots of Empathy programme on that. The name of the baby and that he would be visiting the school, once a month and a little bit about the programme. This was at the start of the year and I don’t think there has been anything lately. I know that some schools have talked about having a little assembly.

Instructor

We would have open assemblies, whereby our parents are invited in to see the children just into their classrooms. They just come in and sit in the chairs, and the children talk about what they’ve been doing, or emphasis, celebration of the feast, or whatever it is that they’re doing and the teacher has included it in that. And the board up so it’s very visible. So parents can see it when they’re coming into the room. But I’m sure there’s other ways. But even if we have a meeting, you might only get a few parents that would come.

Principal

I think it was something like, if you’re going to send home something, it has to be approved by Canada, so that’s time-consuming.

Teacher

I am sure it would be an advantage but you would have some parents who would be illiterate and they wouldn’t really understand maybe what it’s all about. Certainly it would be good if they could talk to them about it. I would say that some of the parents would actually have forgotten that it is still going on. Perhaps that something the school could look at to try and promote Roots of Empathy and maybe the principal could put monthly updates in the newsletter as to what the themes are and what size baby is, etc., to encourage maybe that home–school link.

Principal

I haven’t had any feedback from any parents. But it’ll be interesting to see what the parents’ view is.

Instructor

A recurring theme on parental engagement was that parents would benefit from knowing more about the ROE programme. Key stakeholders considered it important for parents to know more about the programme because they felt that this would encourage parents to be more involved and improve outcomes for the children by reinforcing lessons learned from the programme at home:

Yeah, it would be nice to send home a newsletter to say look, this is what we’ve done, even just like a monthly one or like every 2 month one so at least parents then are a bit more involved, like if I’m sure if I did ask the parents they probably wouldn’t ever call you, do you know what you mean, but then I suppose you can’t really invite them in because it’s hard to get them in during the day.

Teacher

I think it’s always good if you’re embedding something that’s really forced home or that they can talk about it and I think that angle of it, at the minute, is somewhat lacking.

Instructor

I think it’s something you need to do because what you’re also saying to the parents is, this is a really important part of what we do, the whole pastoral social emotional dimension, because sometimes parents need to be educated to the fact that, I mean you’re saying that education is not just about the three Rs, it’s also about all this other stuff, and then some parent would reflect on challenges they’re thinking on as a parent, because actually, you teach your kids right and wrong, there’s a way of dealing with things in terms of how they manage behaviour at home.

Principal

I know that there have been a few issues round parental participation and I know one of the things that was pointed out at one of the original instructors’ meetings was that they felt that maybe going forward with the programme a good thing to do would be to bring the parents in advance of the programme and do some sort of a general presentation with them. Some of the schools did that, others of the schools didn’t, they just wrote out.

Instructor

Parents’ perspectives on parental involvement

There was a mixed response from parents when they were asked how much they knew about ROE. Many of those interviewed knew that their child was taking part in the programme but there did not appear to be a high level of awareness about the content of the programme or what the outcomes or benefits may be for their children. However, all parents were aware that there was a baby visiting the class, mostly as a result of their children going home and talking to them about it or from reading about it in the school bulletin or on the school notice board. Parents shared what they knew about the ROE programme:

The only reason I know this is taking place is because my child comes home and talks about the baby all the time.

Parent

I think the programme is about how to look after a baby, am I right?

Parent

Some of the parents mentioned that being part of the research programme meant that they had received written information about it. There was unanimous agreement from the parents, school personnel and programme co-ordinators that greater parental knowledge about the programme would perhaps increase children’s chances of positive outcomes:

I think I got information from Queen’s University about taking part in this research. The letter told us what Roots of Empathy was about.

Parent

I think the only time I got any written information about this was from Queen’s University. I thought they were running this in school.

Parent

Children’s perspectives on parental involvement

According to the children, their parents were really interested in the programme and the ROE baby was often discussed at home:

Well the first week of the ROE I took part I went home and told my dad. And my dad ever since has just known about and he’s asked me every time I come home on Friday, how was the baby?

Child

I always tell my parents about how the baby is doing, when he got his first tooth and things. They loved to hear about it.

Child

Although this was less common, a few children said that their parents had no interest in hearing about the programme or the baby:

My mum and dad aren’t very interested they don’t say much.

Child

Some children said that their parents’ knowledge about and interest in the programme were clear from the fact that they came home and talked about the baby or stemmed from the fact that they were familiar with the ROE family from their own community:

Yeah because my mum knows the baby from church.

Child

My mum knows the mum and baby because the mum taught my sister when her teacher was having a baby.

Child

Some parents approached school staff to talk about the ROE programme, which the staff commented was positive:

I had one parent who had been on a behavioural management programme with the board previously and she’s personally came and said she has seen a change in her child but she wanted to talk and I, you know, there’s only so much then you can share, read the notice board, talk about the noticeboard with your child, get him to talk about it so the few parents that I have had come to me about it have been positive but again, they would have liked a little bit more continual.

Principal

One teacher mentioned that many fathers had relayed that their sons had been talking a lot about the baby at home:

Yeah. I’ve had more feedback from dads about how their sons have responded, to me, because it’s the brotherhood all together. A lot of the mums have said, ‘Oh, my daughter’s really enjoying it’, but again, I think, and I may be stereotyping myself, but I think girls are already imbibed with that maternal thing.

Teacher

Effectiveness of Roots of Empathy

Immediately post test (T1), there is evidence that the programme is achieving a positive effect in relation to both of the primary outcomes. In particular, participation in the ROE programme is associated with an increase in prosocial behaviour (g = +0.20; p = 0.045) and a decrease in difficult behaviour (g = –0.16; p = 0.060); however, as this latter finding is only approaching statistical significance, it needs to be interpreted with a degree of caution. This being said, both findings are consistent with the small number of other evaluations undertaken of the ROE programme, for which the pooled effect sizes are 0.13 and –0.18, respectively.

One year after the end of the programme (T2), the findings from this present study indicate that the effects for prosocial behaviour have disappeared (g < 0.01), and this continues to be the case at the subsequent follow-up points (at 24 and 36 months). Interestingly, the size of the effect in relation to the decrease in difficult behaviour remains fairly stable across the 3 further years but, with the reduction in sample sizes because of attrition, these are no longer statistically significant (T2,: g = –0.14, p = 0.22; T3, g = –0.13, p = 0.25; and T4, g = –0.14, p = 0.20). The consistency of this difference over time would suggest that the programme may be having a sustainable effect in terms of reducing difficult behaviour. However, because these effects are no longer statistically significant, this finding must be treated with caution and requires further verification.

With regard to secondary outcomes, other than the programme-specific outcome of the understanding of infant feelings, no differences were found immediately post test (T1) or at any of the follow-up time points between the children who participated in the ROE programme and those in the control group. This finding also seems to be consistent with other existing evaluations of ROE. It is important to stress that this does not mean that the programme was not effective at improving these secondary outcomes. Rather, it simply demonstrates that the programme was no more effective than the existing curriculum, and especially PDMU, in relation to these outcomes.

The additional exploratory subgroup analyses did not provide any convincing evidence that the programme tended to have differential effects with regard to gender (boys or girls), socioeconomic background or number of siblings. As the programme was delivered consistently with high fidelity across intervention schools, it was not possible to assess whether or not variations in fidelity were associated with variations in outcomes achieved.

Cost-effectiveness of Roots of Empathy

Overall, it is estimated that the average cost of delivering ROE is £4057 per school and £175 per pupil. Against generally accepted national guidelines, the findings of this present study suggest that ROE is a cost-effective intervention. In particular, NICE suggests that interventions costing the NHS < £20,000 per one-unit increase in QALYs are cost-effective. It also suggests that those costing between £20,000 and £30,000 may be cost-effective. For the present evaluation, it was found that ROE had an 83.1% chance of being cost-effective at the £20,000 per QALY threshold and a 90.1% chance at the higher threshold of £30,000 per QALY.

Programme delivery and stakeholder perspectives

In relation to programme delivery, it is notable that the ROE programme was delivered with high fidelity, with all lessons being delivered in all of the intervention schools. This was seen as the result of the clearly defined structure of the programme and the strong training and ongoing support provided to ROE instructors in schools. The programme was also very well received overall and it was felt to include good resources and be linked in closely with the Northern Ireland curriculum, particularly the element on PDMU. Beyond this, five key issues emerged from the qualitative process evaluation.

1. Some believed that it would be beneficial if another teacher in the same school was the ROE instructor, which could allow stronger communication and planning between the instructor and the class teacher.

2. There was a perception that the delivery of the programme in the first year may have been a little more challenging, especially in those schools whose ROE instructor was not a teacher within that school. Relatedly, there was a belief among some that there would be enhanced opportunities in future years once the instructor and teacher had greater experience of and knowledge about the programme.

3. There was concern regarding the resources required to deliver the programme, especially if the instructor were to be one of the teachers in the school, and whether or not this would be sustainable in the longer term.

4. There was concern that the programme lasts for only 1 year and is not followed up in subsequent years. In addition, and relatedly, some thought that it would be worthwhile to build the key knowledge and skills among children at an earlier age and before the programme took place, with some mentioning the ‘Seeds of Empathy’ programme.

5. The relative lack of involvement of or engagement with parents in the programme and how this may have been arisen partly because of the emphasis on maintaining fidelity to the existing programme.

Inclusion and exclusion criteria

As there are relatively few evaluations of ROE, it was decided that the inclusion criteria would not be as stringent as those for typical meta-analyses. Studies were considered eligible if they employed an experimental or quasi-experimental design, quantitatively measured (at least) teacher-rated prosocial and aggressive behaviour, and collected outcome data both pre and post test. Studies were excluded if they were not an effectiveness evaluation or if they collected only qualitative data. The characteristics of included studies, including the current trial, are described in Table 30, and excluded studies are described in Characteristics of excluded studies.

Study design and location

Apart from the current trial, there is only one other randomised controlled trial of the programme36 and – similar to the current trial – this evaluation also followed up participants after the immediately post-test data collection for a further 3 years. The remaining six studies employed a quasi-experimental design with pre- and post-test data collection only (no follow-up). Four studies were conducted in Canada,36,45–47 two were conducted in Scotland48,49 and one was conducted in Australia. 50

Sample

In total, 4140 primary school-aged children from 145 primary schools took part in the seven studies. The sample sizes ranged between 132 and 785 children, with an average sample size of 591.

Outcomes

All of the evaluations to date – including the current trial – have measured teacher-rated prosocial and aggressive behaviour using valid and reliable instruments. A range of other teacher- and child-rated outcomes were also measured; however, this meta-analysis focuses only on synthesising the effects for the most commonly measured outcomes:

• teacher-rated prosocial behaviour immediately post test (all seven previous studies)

• teacher-rated aggressive behaviour immediately post test (all seven previous studies)

• child-reported empathy immediately post test (five studies)

• child-reported emotional regulation immediately post test (two studies).

In addition, the one study that measured outcomes for 3 years after the immediately post-test time point36 will be summarised to provide an initial insight into the potential longer-term impacts of the programme.

Analysis

All of the studies were conducted in the school and classroom setting; however, only two of the seven studies took account of the clustered nature of the data in their estimation of the effect size and associated standard error. Ignoring clustering in the analysis can lead to artificially small standard errors and, consequently, smaller p-values. The implications of this are twofold:

1. Studies with smaller standard errors are deemed to be more precise estimates of the effect and are given more weight in the analysis (even though they might be methodologically weak), thereby inflating their relative contribution to the calculation of the weighted SMD.

2. A small standard error is more likely to yield a statistically significant result and lead to the incorrect rejection of the null hypothesis [i.e. concluding there is an effect when in fact there is not (a false positive)].

Publication status

Only two of the studies included in the analysis have been published in peer-reviewed journals. 36,47 The remaining studies are unpublished reports and are considered grey literature.

Risk of bias

The risk of bias with respect to the methodological quality of each included study was assessed by the authors. It includes judgements about each of the following criteria:

• random sequence generation (selection bias)

• blinding of outcome assessment (detection bias)

• incomplete outcome data (attrition bias)

• other bias [defined in this instance as taking (or not) the clustered nature of the data into account at the analysis stage].

Table 31 shows the judgements about the risk of bias for each included study.

Prosocial behaviour

The results of the meta-analysis for teacher-rated prosocial behaviour are reported in Figure 6. All seven studies were included in the analysis and the overall SMD was 0.13 (95% CI 0.06 to 0.19) in favour of the intervention group. This means that ROE improves prosocial behaviour by, on average, 0.13 of a SD and this improvement is statistically significant (p < 0.001).

FIGURE 6.

Forest plot of SMDs between intervention and control for prosocial behaviour. SE, standard error.

The studies with the smallest standard errors are making the largest relative contribution to the calculation of the total SMD. It should be borne in mind, however, that some studies are rated as being at a high risk of bias despite having narrow CIs (e.g. Smith22 and, to a lesser extent, Kendall et al. 26) and ordinarily might not have been included in this type of analysis, simply because the methodology is not sufficiently robust to allow confidence that the results are accurate or trustworthy. As the evidence for ROE grows and more rigorous studies are conducted, it will become possible to restrict the analysis to only those most rigorous studies, which will ultimately yield a more precise, less biased and more trustworthy result. At the moment, however, these analyses should be interpreted with a degree of caution.

The forest plot depicted in Figure 6 provides an alternative representation of the table in Figure 6. Each green square represents the effect size associated with that study (the larger the green square, the larger the effect size) and the bars on either side of the square represent the 95% CI. The wider the CI, the less precise the estimated effect size. If the bars touch or cross the zero line, this means that the effect of the intervention was not statistically significant in that study. The black diamond depicts the overall effect size for all the studies combined and the tips of the diamond on either side represent the precision of the estimate. The wider the diamond, the less precise the estimate. If the diamond touches or crosses the zero line, this means that the overall effect is not statistically significant. τ² (0.00, χ² = 8.39, degrees of freedom = 6; p = 0.21) represents an estimate of the between-study variance in a random-effects meta-analysis. The square root of τ² is the estimated SD of the underlying effects across studies. If τ² is not statistically significant, this suggests that there is no statistical evidence for differences between studies and it is valid to pool the results from these studies into a single estimate. When τ² is statistically significant, this suggests that there are important differences between studies (statistical heterogeneity) and that it might be questionable or invalid to pool the results. τ² tells us whether or not the heterogeneity is significant, although it should be interpreted with caution as it typically has low power. It does not, however, indicate the magnitude of the heterogeneity, which is represented by I². Although the following rubric should be carefully applied, in general I² = 0% (no heterogeneity), I² = 25% (low heterogeneity), I² = 50% (moderate heterogeneity) and I² = 75% (high heterogeneity). 95 Thus, for the meta-analysis reported above, there is low statistical heterogeneity between studies.

Aggressive behaviour

The meta-analysis for teacher-rated aggressive behaviour also utilised all seven included studies (Figure 7). The overall SMD was –0.18 (95% CI –0.33 to –0.03) in favour of the intervention group. This means that ROE decreases aggressive behaviour for participating children by, on average, 0.18 of a SD and this improvement is statistically significant (p = 0.02).

FIGURE 7.

Forest plot of SMDs between intervention and control for aggressive behaviour. SE, standard error.

In relation to the first research question, therefore, this meta-analysis of the results from all seven previous evaluations of ROE provides evidence that the programme is effective in improving prosocial behaviour [effect size (ES) = 0.13; p < 0.001] and decreasing aggressive behaviour (ES = –0.18; p = 0.02). These effects are statistically significant but there is considerable heterogeneity between the studies that generated these data (τ² = 0.04; p < 0.001; I² = 98%) and so caution should be exercised when interpreting these results. The CI associated with the effect size for aggressive behaviour is particularly wide (95% CI –0.33 to –0.03), indicating that this is not a very precise estimate of the true effect.

Empathy

Five of the seven studies measured child-reported empathy. As shown in Figure 8, the estimated SMD was 0.1 (95% CI –0.05 to 0.25) in favour of the intervention group, but this was not statistically significant (p = 0.17). This means that there is no evidence from the previous evaluations to suggest that ROE is effective at improving child-reported empathy. As above, there is moderate heterogeneity between the combined studies (τ² = 0.02; p < 0.001; I² = 72%).

FIGURE 8.

Forest plot of SMDs between intervention and control for child-reported empathy. SE, standard error.

Emotional regulation

Even fewer studies measured child-reported emotional regulation. Of the two that did, the SMD was 0.03 in favour of the intervention group, but this is an extremely small effect size and – as the forest plot in Figure 9 depicts – is not statistically significant (p = 0.60).

FIGURE 9.

Forest plot of SMDs between intervention and control for child-reported emotional regulation. SE, standard error.

With respect to the second research question, it seems that there is no (or, at best, insufficient) evidence to indicate that ROE improves child-reported empathy or emotional regulation.

Long-term effects of Roots of Empathy on (teacher-rated) prosocial and aggressive behaviour

Only one evaluation20 studied the longer-term impact of the programme. This is the only other randomised controlled trial we have data for and it appears that, after 3 years, the intervention group had poorer prosocial behaviour than the control group (SMD –0.12, 95% CI –0.17 to –0.07). This is in contrast to the positive impact of ROE on prosocial behaviour reported by this study immediately post test (ES 0.21). With respect to aggressive behaviour 3 years post intervention, the intervention group was displaying only slightly less aggressive behaviour than the control group (SMD –0.06, 95% –0.09 to –0.03) and, although statistically significant, this effect was much reduced compared with the effect observed immediately post test (ES –0.25).