The Good Behaviour Game intervention to improve behavioural and other outcomes for children aged 7 8 years: a cluster RCT

Evidence base for the Good Behaviour Game

There have been 14 RCTs of the GBG to date: six in the USA,17,18,23,29,33,37 two each in the Netherlands31,32 and Canada;19,35 and one each in Belgium,34 Northern Ireland,36 Estonia21 and England. 1 These trials represent the standard context for implementation of the GBG (e.g. whole class delivery during a normal school day); a fifteenth RCT based in the USA reports on the impact of the intervention in the context of an after-school programme. 22 Table 1 provides a summary of the designs and findings of these studies.

To aid interpretation of Table 1, we note that (1) the Turkkan version of the GBG (manualised by Jaylan Turkkan at Johns Hopkins University), referenced in relation to two trials17,23 is a precursor to the AIR model that uses the same procedures and rules (i.e. four or fewer infractions to win the game); (2) that the two follow-up analyses for one trial23 have conflicting findings, with one reporting null effects25 and the other reporting significant intervention effects for the aggressive male subgroup;24 and (3) that a definitive sample size is not provided for one trial,18 but the authors report that 8–10 students were sampled randomly for each of 188 teachers.

Inspection of Table 1 reveals a number of trends in study design that are directly pertinent to the current study.

First, several RCTs have trialled the GBG in combination with other interventions, often using designs that mean that the effects of each cannot be properly isolated. 17,18,35,38

Second, intention-to-treat (ITT) findings, in which analyses include every participant according to their randomisation, irrespective of their characteristics (e.g. sex, baseline risk status) and/or post-randomisation events (e.g. non-compliance, withdrawal),39 have been somewhat variable. Some trials17,23,33 have failed to report true ITT findings, raising the risk of bias in these studies. Among those that have reported ITT findings, and in which the specific effects of the GBG can be isolated, most note significant intervention effects on behavioural and other outcomes,19,21,31,32,34,36 although there are a couple of notable exceptions,1,37 including the trial on which the current study builds (see The Good Behaviour Game in England).

Third, medium- (i.e. 12–24 months) and long-term (i.e. > 24 months) post-intervention follow-up is rare: only four trials17,23,29 – including the current study – have included any kind of follow-up beyond the immediate post test, precluding assessment of sleeper and/or maintenance effects in most cases.

Fourth, the reporting of implementation data is highly variable. In many cases, either it is not reported at all23,32,33 or the data are extremely sparse. 17,34,38 Furthermore, for an intervention explicitly premised on the frequency and duration of delivery, the reporting of dosage data is surprisingly absent from many trials, precluding rigorous analysis of the moderating effects of intervention compliance. Where dosage is documented, it is typically self-reported by teachers, a method that is known to be subject to bias and impression management. 40

Fifth, analysis of subgroup effects is commonplace and typically focuses on sex and/or risk status, with the latter usually defined by elevated behaviour problems at baseline.

Last, most RCTs of the GBG have been modestly sized (e.g. < 50 clusters and < 1000 participants), placing limits on statistical power of analyses.

A recent meta-analysis41 captured six of the 14 GBG trials. The two most recent US trials,18,37 the most recent Canadian trial,19 and the trials in Northern Ireland,36 Estonia21 and England1 were concluded or published after the closing census date for the meta-analysis; two early US-based trials were also excluded for unknown reasons. The meta-analysis indicated that the intervention significantly outperformed comparison conditions in main effect/ITT analyses for three out of the six outcomes that were examined: teacher-rated conduct problems, peer-rated conduct problems and peer-rated peer relations (the other three outcomes were teacher-rated inattention, reading performance and teacher-rated peer relations). 41 The size of the intervention effect for these three outcomes (g = 0.1–0.2) was broadly in line with the above-noted findings of a meta-analysis of universal approaches to behaviour management. 10 Among the eight GBG RCTs not included in the meta-analysis,1,18,19,21,29,33,36,37 three reported null ITT results,1,18,37 and one did not report ITT findings. 33 The four that reported a significant main effect of the intervention reported a range of effect sizes (ESs), from 0.11 to 0.42, on behavioural and related outcomes. 19,21,29,36 Taken as a whole, these findings indicate small or moderate overall effects of the GBG when conventional ES thresholds are applied. 42 This probably reflects the fact that children’s behaviour is typically very good in most schools,2 with very few children displaying the symptoms of conduct or other problems at the outset of any given trial. 4

Alongside the main effect estimates provided by ITT analysis, it is also important to consider moderated effects. Three treatment effect modifiers are particularly pertinent here: subgroups, implementation and timing of follow-up. First, it is widely recognised that children do not respond uniformly to exposure to universal interventions. Accordingly, subgroup analyses can be very informative, provided that they are specified in advance, are informed by theory and/or research, and include clear specification of the expected direction of effects and population subgroup(s) of interest (using features measured pre randomisation, e.g. demographic characteristics, individual differences at baseline and/or family factors). 43

The effects of the GBG appear to vary by baseline risk status (e.g. higher levels of difficulties) and/or sex. In relation to baseline risk status, it stands to reason that those whose behaviour is already a significant cause for concern would stand to benefit the most from the GBG, especially given its emphasis on adaptive socialisation processes (e.g. alerting children to and rewarding them for meeting social task demands in the classroom). It is perhaps unsurprising that several GBG trials found amplified intervention effects among children considered ‘at risk’ because of their elevated levels of problematic behaviour. 17,23,33,37 In relation to sex, the intervention procedures may particularly appeal to boys, given the gendered socialisation of competitiveness. 44 A recent trial of the GBG in Canada found evidence to support this, with significantly greater reduction in conduct problems among boys than among girls. 19 The intersection of these two factors – that is, boys at risk of developing conduct problems – is a specific focus in this trial. Given the aforementioned early adulthood outcomes for this particular stratum of the population,6,7 research that rigorously establishes efficacious, early, preventative strategies would be particularly welcome. 9 Promisingly, there is some existing evidence of amplified gains in this subgroup following exposure to the GBG. 24

The second potential treatment effect moderator of note is variability in implementation. Such variability is considered to be inevitable, particularly in the case of universal school-based interventions,45 and the accumulated evidence base suggests that it is associated with variability in the achievement of intended outcomes. 46 In the GBG, teachers may vary the frequency and/or duration of game sessions (i.e. dosage), their adherence to prescribed procedures (i.e. fidelity) and any associated changes to these (i.e. adaptations), and the extent to which they play the game in an enthusiastic and engaging manner (i.e. quality). Whether or not the game is played with all children in a given class (i.e. reach) and how they react when it is played (i.e. participant responsiveness) may also be important.

How similar or different the game is to existing behaviour management approaches in a given classroom (i.e. programme differentiation) and/or those against which it is being compared (i.e. control group activity) is also likely to contribute to its relative success. As noted above, there has been remarkably little empirical scrutiny of the extent to which variability in one or more of these implementation dimensions moderates treatment effects in RCTs of the GBG; instead, the norm has been to simply provide descriptive summaries. 41 A notable exception is Ialongo et al. ’s trial,17 in which the authors’ per-protocol analysis (i.e. intervention schools divided into ‘high’ and ‘low’ implementation groups on the basis of fidelity scores) indicated an association between implementation level and the magnitude of certain intervention outcomes. However, caution is required in interpreting such findings given that per-protocol analysis compromises the randomised design.

Here, the application of complier-average causal effect (CACE) estimation (see Chapter 2) and related instrumental variable techniques offer great promise, but, to date, to the best of our knowledge, there have been only two applications of CACE in GBG trials. 47,48 Leveraging data from a recent US-based RCT,37 Bradshaw et al. 47 found that the presence and magnitude of intervention effects for at-risk children in ‘PATHS to PAX’ (an integration of the PAX GBG and the Promoting Alternative Thinking Strategies curriculum) varied as a function of compliance. Thus, the initial effect on social competence grew from 0.01 to 0.28, and previously unidentified effects on academic engagement and emotion regulation emerged in CACE models that took account of variability in GBG dosage (e.g. total duration of exposure in minutes). However, in models focusing on the PAX GBG alone, there was no difference between (null) initial and CACE findings; in other words, the PAX GBG was found to be ineffective for at-risk children even after robustly accounting for implementation variability. The second application of CACE was reported by the authors of the current study using data from the English GBG trial1 that is the focus of this report (see The Good Behaviour Game in England). In contrast to null ITT findings, Ashworth et al. 48 revealed sleeper effects of the intervention on academic attainment at the 12-month post-intervention follow-up when compliance (i.e. dosage, as in the Bradshaw study47) was taken into account.

Last, the effects of the GBG may be moderated by the timing of follow-up. It is important to study intervention effects over time to establish whether effects detected immediately post intervention are sustained (i.e. maintenance effects) or effects only become apparent in the years that follow (i.e. sleeper effects). Although medium-term follow-up is generally lacking, there is promising evidence of longer-term maintenance effects in the GBG. For example, when Ialongo et al. 28 followed up the sample of one of the original US trials17 ≈ 5 years after the intervention was concluded, those who had received the GBG (in combination with curriculum enhancements and back-up strategies) were significantly less likely than those in the control condition to meet the diagnostic criteria for conduct disorder. Similarly, following up another US trial sample,37 Kellam et al. reported that male participants who were initially classified as aggressive and had participated in the GBG in first grade (i.e. aged 6–7 years) were significantly less likely to engage in high-risk sexual behaviours and drug abuse as young adults (i.e. aged 19–21 years). 49 Such findings are in line with the LCSFT that underpins the GBG, as they are demonstrative of effective socialisation of behaviour influencing social adaptational status in other social fields as these change throughout the life course. 15

Sample size and power

A total of 79 schools were recruited, of which 77 met the eligibility criteria for randomisation (i.e. signing the MoA and completing baseline measures). There were 3084 eligible children (i.e. those in Year 2 at T1) attending these schools. As this sample was initially recruited for the EEF-funded education-related trial,1 the numbers of schools and children required were based on prospective power calculations relating to the primary outcome for that project (i.e. children’s reading scores at T3). Accordingly, the power calculations presented here for the health-related analyses are necessarily post hoc.

A Monte Carlo simulation (see Report Supplementary Material 1), with robust maximum likelihood and 10,000 replications, was conducted in Mplus, version 8.2 (Muthén & Muthén, Los Angeles, CA, USA), to assess the power for the GBG intervention effect on the main outcome (i.e. conduct problems), with 3084 children across 77 clusters (mean cluster size, n = 40). Attrition was set to 0% and 15% for the intervention and outcome variables, respectively. The binary intervention variable (GBG vs. control) was set to have a variance of 1 and a mean of 0. The simulation was carried out for a logistic regression, with different seed values to ensure the stability of findings. For an intracluster correlation coefficient value of 0.06, the between-level variance was set to 0.019 using the following formula:

where S_w² is set to π²/3 for logistic regression. The probability of being at risk was set to 13.4% based on national averages. 59 Therefore, the threshold for the outcome variable and the slope of the trial were adjusted to reflect this using the following formula:

where log = a + b × x. Acceptable power levels were achieved when the trial effect was set to b = –0.38 and the outcome threshold to τ = 2.25. 60 Findings showed small population parameter (0.16%) and standard error (SE) bias (1.4%), and satisfactory coverage (0.94) and power (0.80) for an ES of b = –0.38, which corresponds to Δ = 0.38, based on a total variance of 1 and Δ = b/standard deviation (SD). This means that the trial was powered to detect intervention effects considered to be small to moderate when judged by conventional thresholds,42 aligning well with the overall trend reported across previous GBG trials (see Chapter 1, Evidence base for the Good Behaviour Game).

Recruitment of schools

Mentor UK recruited schools to the trial from three regions (i.e. Greater Manchester, West and South Yorkshire, and the East Midlands) using a number of strategies, including regional recruitment events, using contacts at local authorities and independent providers to identify prospective trial schools, and e-mailing project flyers to schools. Initial expressions of interest were sought using an online form, followed by direct contact from Mentor UK, before the MoA was signed. Recruited schools were then assessed for representativeness against all schools on key characteristics (e.g. size, FSMs) (see Table 3 and accompanying commentary).

Inclusion/exclusion criteria

All children in Year 2 (i.e. aged 6–7 years) at T1 (May–July 2015) were eligible to participate.

Randomisation

Participating schools were the unit of randomisation to minimise the risk of contamination that would have resulted from within-school (e.g. class) randomisation, and for practical reasons, given that the intervention model includes a GBG coach being assigned to each participating school in the intervention arm.

A total of 77 schools met the criteria for randomisation (i.e. signed MoA, > 90% of T1 measures complete) and were randomly allocated to either implement the GBG or continue usual practice following the completion of baseline measures at T1. The allocation procedure was conducted independently by the Manchester Academic Health Science Centre Clinical Trials Unit. A minimisation algorithm was applied to the randomisation to ensure balance across the arms of the trial in terms of the proportion of children eligible for free school meals (FSMs) and school size (data were provided from the school performance tables on the Department for Education website. 61 This approach is described as the ‘platinum standard’ for trials, conferring the benefits of randomisation in terms of rigour and causal inference, as well as guaranteeing similarity of groups on key observables. 59 Thirty-eight schools were allocated to implement GBG and 39 schools were allocated to continue with their usual practice.

Brief name

The Good Behaviour Game (GBG).

Why (rationale/theory)

The GBG is underpinned by three theories of human development and learning: behaviourism (specifically contingency management),13 social learning theory14 and LCSFT. 15 In terms of behaviourism, it is assumed that behaviour that is rewarded is more likely to be reproduced. Consequently, children receive positive reinforcement when they engage in desired behaviours (e.g. following the teacher’s instructions during an activity). However, the group-based orientation of the GBG means that it also draws on social learning theory. In particular, at-risk children can benefit from appropriate behaviour being modelled effectively by other team members. Last, a key tenet of LCSFT is that successful adaptation at different stages of life is contingent on our ability to meet particular social task demands. In school, these include being able to pay attention, work well with others and obey rules. Success in social adaptation is rated both formally and informally by other members of a given social field (e.g. teachers, peers). LCSFT predicts that improving the way in which children are socialised in the classroom (e.g. explicitly highlighting and promoting social task demands, then rewarding children for meeting them) will improve their social adaptation. It is also predicted that early improvements in social adaptation in the classroom will extend to positive adaptation in other social fields (e.g. peer group, family, work) throughout the lifespan. 15

Who (recipients)

The GBG is a universal intervention that is delivered to all children in a given class.

What (materials)

Schools receive GBG manuals that outline the programme theory, goals and procedures. Other materials include some tangible rewards (e.g. stickers), displays (e.g. scoreboard, rules posters) and data forms for recording and monitoring purposes. In the current study, two additional resources were developed by a member of the evaluation team (Wo) following a request from the delivery team (Mentor UK). First, an online GBG scoreboard was created. Each teacher was able to log in to a secure website to record game and probe data [see What (procedures)] in real time and retrospectively, and these data could then be downloaded to assess temporal trends and inform future implementation planning. In turn, each GBG coach [see How well (planned)] was able to access their assigned teachers’ data for use in later support sessions, and the research team was able to access all teachers’ data so that it could be used to monitor the length and frequency of games (used in this study to examine the extent to which intervention effects varied by levels of compliance; see Hypothesis 3: implementation effects). Second, we developed an electronic version of the fidelity checklist used by GBG coaches. This was identical to the paper version used by the licensing organisation (AIR) and was used for the same purpose (e.g. to facilitate feedback following an observation session).

What (procedures)

The teacher divides the class into mixed teams with up to seven members, with team membership typically varied several times in a school year (e.g. every half term). Where possible, each team should be balanced, with equal representation of salient factors such as behaviour, academic ability and sex. The teams then attempt to win the game as a means to access particular privileges/rewards. The game is played during a typical class activity. During the game period, the class teacher records the number of infractions of the following four rules among the teams:

1. We will work quietly.

2. We will be polite to others.

3. We will only get out of our seats with permission.

4. We will follow directions.

In relation to the first rule, adherence is defined as working at a noise level that is deemed to be appropriate for the classroom activity being undertaken while the GBG is being played. Prior to the commencement of the game, the teacher agrees one of the following noise levels with the class: level 0 (voices off, silence), level 1 (whisper, only the person sitting next to you can hear you), level 2 (inside voice, only people sitting at your table can hear you), level 3 (speaker, your classmates can hear you) and level 4 (outside, ‘playground’, voice). The game is ‘won’ by the team(s) with four or fewer infractions, which then access an agreed reward. 15,52 The procedures undertaken before, during and immediately after a game session are detailed in the aforementioned intervention manual [see What (materials)] and are as follows:

• Before the game –

  • The teacher explains the task/activity.

  • The teacher checks understanding of the task/activity.

  • The teacher reminds pupils that they cannot ask for help.

  • Pupils are placed in teams of between 3 and 7 (except in special circumstances, for example a situation in which a child is placed in a team on their own as a response to them deliberately and repeatedly sabotaging their team’s efforts to win the game).

  • The pupils are in clear teams.

  • The teams are sex balanced.

  • The rules are appropriately verbally reviewed with the class.

  • Exemplars are modelled/described by the teacher and/or pupils.

  • Infractions are modelled/described by the teacher.

  • Infractions are described, but not modelled, by students.

  • The voice level for the task/activity is given by the teacher.

  • The teacher states when the game begins.

  • The teacher states how long the game will be played for.

  • The teacher sets a timer.

  • The teacher states that they will monitor infractions.

  • The teacher states that four infractions are permitted per team.

  • The teacher reminds pupils that they are not competing against each other.

• During the game –

  • The teacher identifies infractions when they occur.

  • The teacher records infractions on the scoreboard.

  • The teacher identifies rule breaking team (e.g. ‘team 4 have broken rule 4: “we will follow directions” ’).

  • The teacher discreetly indicates the infraction to specific pupil.

  • The rest of the team and/or class are praised for adhering to rules (e.g. ‘well done everyone else for following rule 4’).

  • The teacher does not punish pupils/teams for infractions.

  • The teacher monitors behaviour.

  • The teacher does not interact with pupils.

  • The teacher adheres to the time limit that was set.

  • The teacher announces the end of the game.

• After the game –

  • The teacher repeats the criterion of four infractions or fewer.

  • The teacher announces the winning team(s) only.

  • Members of the winning team receive a stamp (or marker, etc.) in their individual booklets.

  • A star is placed on the wall chart (or equivalent).

Over the course of the implementation of the GBG, it is intended for there to be a natural progression in terms of the types of rewards used (from tangible rewards, e.g. stickers, to more abstract rewards, e.g. free time), how long the game is played for (from 10 minutes to a whole lesson), the frequency at which the game is played (from three times per week to every day) and when rewards are given (at the end of the game, end of the day or at the end of the week). 11,64 This progression is designed to maintain responsiveness, interest and challenge for students, as well as encouraging generalisation. Thus, good behaviour achieved during the relatively brief ‘game’ periods is increasingly generalised to other activities and parts of the school day. The intervention aims to build intrinsic reinforcement so that modified behaviour is retained even after external reinforcement is removed (i.e. maintenance) and will be exhibited in all settings (i.e. generalisation). These processes are documented through ‘game’ and ‘probe’ data collected by teachers during implementation. 52 Probe data, used to assess generalisation, are collected covertly during an ordinary task/activity, following the same procedures as those used in a game session (e.g. the teacher monitoring rule infractions among teams), but without explicitly setting up the rules and announcing infractions.

Who (provider)

The GBG is implemented by class teachers.

How

The GBG is implemented face to face during the normal school day. As it is a behaviour management strategy rather than a taught curriculum, the GBG does not require an explicit ‘space’ in the class timetable, thereby minimising the displacement of other activities. However, the pre- and post-game procedures undertaken by the teacher [e.g. reminding the class of the rules, announcing the winners and providing rewards; see What (procedures)] mean that some time is taken up before and after the game period/class activity.

Where

The GBG is implemented on site in participating schools.

When and how much

The GBG is played throughout the school year. As in What (procedures), dosage evolves throughout the period of implementation in terms of both the duration of the game (from 10 minutes to a whole lesson) and the frequency at which it is played (from three times per week to every day).

Tailoring

The GBG is a manualised intervention and participating teachers receive initial and follow-up training, in addition to technical support and assistance, as a means to optimise the fidelity of implementation. However, it is now widely accepted that some form of adaptation is inevitable and may be desirable to improve local ownership and fit to context. 65,66 A critical aspect of the GBG coach role, therefore, is to support teachers to make adaptations that are in keeping with the goals and theory of the intervention. 67

How well (planned)

Teachers receive 3 days of training (2 days of initial training and 1 day of follow-up training approximately 4 months later) from coaches (mostly former teachers), who are contracted by Mentor UK and trained by AIR. Day 1 of the initial training covers an introduction to the GBG theory and logic, such as understanding the core elements of the game (e.g. class rules, team membership, positive reinforcement and monitoring). Day 2 focuses on implementation procedures and practices (e.g. overview of successful GBG implementation, introduction to the implementation fidelity checklist and development of a plan for implementation for their class). The follow-up ‘booster’ training session revisits these ideas, with an opportunity for sharing of good practice and problem-solving.

Ongoing technical support and assistance is provided by the trained coaches, as noted in Tailoring. In the current study, participating schools were each allocated a GBG coach who visited approximately once per month to support implementation throughout the trial. These visits typically comprised modelling of game sessions, observation and feedback [including review of the game, and the probe and fidelity checklist data – see What (procedures)], ad hoc e-mail and telephone support, and provision of additional/booster training or information sessions, as required.

Primary outcome measure

Secondary outcome measures

Covariates

Background data on both schools (e.g. school size, proportion of children eligible for FSMs) and children (e.g. sex, FSM eligibility) were collected for use as covariates in our analyses. School-level data were taken from Department for Education performance tables, and child-level data were extracted from the NPD. The NPD also provides an anonymised child reference number that was used to ensure accurate data matching (e.g. across time and between informants).

Procedures for handling missing data

First, the proportion of missing data was determined for a given outcome variable. If, for a given analysis, < 5% of data were missing then a complete-case analysis was undertaken. Second, if > 5% of data were missing, differences between partially and completely observed cases were examined to establish any pattern to the missingness. Multilevel logistic regression was used to predict missingness, whereby each child was coded as having non-missing (0) or missing (1) outcome data, with other study data as explanatory variables (e.g. trial group, KS1_READPOINTS, conduct problems and TOCA-C behaviour scores at T1, sex, and FSMs). Third, if this analysis determined that data were likely to be missing at random (i.e. conditional on other observed variables), then full information maximum likelihood (FIML) estimation was used so that partially and completely observed cases of all 77 schools and 3084 children were included in our analyses, thereby reducing the bias associated with attrition.

Hypothesis 1: intention-to-treat effects

Hypothesis 2: subgroup effects

Hypothesis 3: implementation effects

Hypothesis 4: longer-term effects

Hypothesis 5: temporal associations between mental health and academic attainment

Hypothesis 6: value for money

Deviations from the statistical analysis plan

Deviations from the SAP were minimised wherever possible. However, some changes were, inevitably, necessary, primarily because of two factors. First, planned analyses involving NPD data had to be altered to reflect the software available in the virtual laboratory environment of the Office for National Statistics’ (ONS) Secure Research Service (SRS) and/or ONS reporting restrictions. 87 Second, for hypothesis 6 specifically, cost data for the GBG provided by Mentor UK were not as granular as had been planned (i.e. they were provided in aggregate form only), limiting the analyses that could be undertaken. This was caused by staff turnover and the subsequent collapse of Mentor UK.

Specific SAP deviations and an accompanying rationale for each are delineated below.

Hypothesis 1

Children in primary schools implementing the GBG over a 2-year period will demonstrate significantly better outcomes in mental health; conduct problems (hypothesis 1a), psychological well-being (hypothesis 1b) and emotional symptoms (hypothesis 1c); sources of resilience; peer and social support (hypothesis 1d) and school environment (hypothesis 1e); school absence (hypothesis 1f), and significantly lower rates of bullying (i.e. social acceptance; hypothesis 1g) and exclusion from school (hypothesis 1h) than children attending control schools.

Tables 6–8 present the findings of our ITT analyses. In relation to the primary trial outcome, conduct problems, there is no evidence of the impact of the GBG (standardised coefficient –0.039, SE 0.323; p = 0.903). There were also no intervention effects identified in relation to psychological well-being (standardised coefficient –0.251, SE = 0.354; p = 0.477) or emotional symptoms (standardised coefficient –0.265, SE = 0.318; p = 0.405); peer and social support (standardised coefficient –0.016, SE = 0.356; p = 0.965) or school environment (standardised coefficient 0.019, SE = 0.319; p = 0.952); school absence (coefficient –0.065, SE = 0.046; p = 0.154), bullying (i.e. social acceptance; standardised coefficient 0.191, SE = 0.393; p = 0.627) or exclusion from school (coefficient –0.991, SE = 0.586; p = 0.091). In sum, our ITT findings at immediate post-intervention follow-up (T3) were null for all outcomes.

Hypothesis 2

Boys at risk of developing conduct disorders (defined as scoring in the borderline or abnormal ranges of the conduct problems subscale of the teacher-rated SDQ at baseline) in primary schools implementing the GBG over a 2-year period will demonstrate significantly better outcomes in mental health; conduct problems (hypothesis 2a), psychological well-being (hypothesis 2b) and emotional symptoms (hypothesis 2c); sources of resilience; peer and social support (hypothesis 2d) and school environment (hypothesis 2e); school absence (hypothesis 2f), and significantly lower rates of bullying (i.e. social acceptance; hypothesis 2g) and exclusion from school (hypothesis 2h) than those at-risk boys attending control schools.

Tables 9–11 present the findings of our subgroup analyses. In relation to conduct problems, there is no evidence of the impact of the GBG for at-risk boys (standardised coefficient 0.012, SE = 0.081; p = 0.879). There were also no intervention effects identified in relation to psychological well-being (standardised coefficient –0.039, SE = 0.507; p = 0.507) or emotional symptoms (standardised coefficient –0.093, SE = 0.084; p = 0.264); peer and social support (standardised coefficient 0.019, SE = 0.060; p = 0.758) or school environment (standardised coefficient –0.012, SE = 0.060; p = 0.849); school absence (coefficient –0.275, SE = 0.187; p = 0.141) or exclusion from school (coefficient 1.259, SE = 1.874; p = 0.502) for this subgroup. The analysis did reveal an intervention effect on bullying (i.e. social acceptance) for at-risk boys (standardised coefficient –0.125, SE = 0.051; p = 0.014). However, the nature of the Kidscreen-27 scoring is such that this represents a significant increase in bullying for this subgroup in GBG schools. This is because the Kidscreen-27 technically labels this domain ‘social acceptance’, with higher scores indicative of increased social acceptance, and lower scores indicative of increased bullying. 70 In sum, our subgroup analysis findings at immediate post-intervention follow-up (T3) were null for all outcomes, with the exception of an unexpected negative effect on bullying.

Hypothesis 3

The magnitude of intervention effects noted in hypotheses 1a–h above will vary as a function of intervention compliance. Specifically, we predict larger ESs in schools defined as compliers than non-compliers in terms of dosage (hypotheses 3a–h).

Classroom-level dosage across the span of the intervention period (T1–T3) ranged from 0 to 3535 minutes (mean 1066, SD 719.50). In terms of frequency and duration, teachers played the game approximately twice per week between T1 and T2, and between once and twice per week between T2 and T3; the average game session length in both years was approximately 15 minutes. Nine schools formally ceased implementation prior to T3, although their dosage data are included in the above estimates. For the main analysis, moderate compliers (50th percentile) were identified as the classrooms in which the game was played for > 1030 minutes (n_student = 672; 43.1%). In a sensitivity analysis, classrooms that fell above the 75th percentile (> 1348 minutes) were deemed to be high compliers (n_student = 333; 21.3%). All CACE models were shown to have appropriate and easily distinguished classes103,104 with no less than 1% total count, high posterior probabilities (> 90%) and acceptable entropy values (0.68–0.86).

Compliance predictors

The compliance predictors for moderate- and high-compliance models can be found in Tables 12 and 13. Sex was shown to be a statistically significant predictor in all moderate-compliance models, except for bullying (i.e. social acceptance) and school absence. Similarly, both concentration problems and prosocial behaviour were shown to predict compliance in all moderate models, except for school absence. Thus, classrooms with more boys and those that had lower levels of concentration problems and prosocial behaviour were more likely to comply with the intervention. All school-level characteristics were shown to predict compliance to some extent, although this varied by model (see Table 12).

With respect to high-compliance models, concentration problems and prosocial behaviour were shown to predict compliance, except for school absence, for which concentration problems and baseline school absence were the only significant predictors. As with the moderate-compliance models, classrooms with students that had lower levels of concentration problems and prosocial behaviour were more likely to comply with the intervention; this was true for all variables except for school absence. Only school-level EAL was found to be a statistically significant school-level predictor, and only for the conduct problems model; thus, classrooms in schools with higher percentages of students with EAL were more likely to be high compliers (see Table 13).

Complier-average causal effect models

The moderate- and high-compliance intervention effects for trial outcomes are summarised in Table 14. For full details of the models, see Appendix 1, Tables 28–34.

In relation to the primary trial outcome, conduct problems (hypothesis 3a), there is no evidence of intervention effects in the context of either moderate (standardised coefficient 0.006, SE 0.248; p = 0.982) or high compliance (standardised coefficient 0.258, SE 0.539; p = 0.632). Similarly, there were no intervention effects in either compliance context for emotional symptoms (hypothesis 3c: moderate compliance, standardised coefficient –0.247, SE = 0.38; p = 0.515; high compliance, standardised coefficient –0.341, SE 0.428; p = 0.426); peer and social support (hypothesis 3d: moderate compliance: standardised coefficient 0.491, SE 0.341; p = 0.150; high compliance, standardised coefficient –0.246, SE 2.341; p = 0.916) or school environment (hypothesis 3e: moderate compliance, standardised coefficient –0.121, SE 0.671; p = 0.857; high compliance, standardised coefficient 0.044, SE 1.224; p = 0.972); or bullying (i.e. social acceptance; hypothesis 3g: moderate compliance, standardised coefficient 0.37, SE 2.989; p = 0.901; high compliance, standardised coefficient 0.235, SE 0.592; p = 0.692).

In the case of psychological well-being (hypothesis 3b), intervention effects were found for both moderate (standardised coefficient –1.239, SE 0.377; p = 0.001) and high (standardised coefficient –0.959, SE 0.380; p = 0.013) compliance. However, contrary to predictions, this indicates that increased intervention compliance led to reduced psychological well-being. In terms of ES, moderate compliance led to a reduction of approximately one-quarter of a SD at T3 (ES –0.241, 95% CI –0.303 to –0.179); the high compliance effect was smaller, being equivalent to a reduction of approximately one-twelfth of a SD at T3 (ES –0.084, 95% CI –0.155 to –0.013).

In the case of school absence (hypothesis 3f), significant intervention effects were found in the context of both moderate (coefficient –0.656, SE 0.072; p < 0.001) and high compliance (coefficient –0.674, SE 0.162, p < 0.001). In line with predictions, this indicates that increased intervention compliance led to reduced absence from school. In terms of ES, children in moderate-compliance GBG classrooms have an incidence rate of 51.9% of that of the would-be compliers in the control group (IRR 0.519, 95% CI 0.450 to 0.598); in the high-compliance classrooms, the ES was slightly larger, with an incidence rate of 51% (IRR 0.510, 95% CI 0.371 to 0.701).

In sum, our CACE analysis findings at the immediate post-intervention follow-up (T3) were null for all outcomes, with the exception of an unexpected intervention effect on psychological well-being and an expected intervention effect on school absence.

Hypothesis 4

The effects of the GBG on mental health; conduct problems (hypothesis 4a), psychological well-being (hypothesis 4b) and emotional symptoms (hypothesis 4c); sources of resilience; peer and social support (hypothesis 4d) and school environment (hypothesis 4e); school absence (hypothesis 4f), bullying (i.e. social acceptance; hypothesis 4g) and exclusion from school (hypothesis 4h), reading attainment (hypothesis 4i), prosocial behaviour (hypothesis 4j), concentration problems (hypothesis 4k) and disruptive behaviour (hypothesis 4l), will be maintained at 12- and 24-month post-intervention follow-ups.

Tables 15–18 present the findings of our longer-term follow-up analyses at 12 months post intervention (T4).

At the 12-month follow-up, there was no evidence of the impact of the GBG on conduct problems (hypothesis 4a: standardised coefficient –0.235, SE 0.261; p = 0.369). There were also no intervention effects identified in relation to psychological well-being (hypothesis 4b: standardised coefficient –0.259, SE 0.306; p = 0.396) or emotional symptoms (hypothesis 4c: standardised coefficient –0.228, SE 0.292; p = 0.435); peer and social support (hypothesis 4d: standardised coefficient –0.555, SE 0.363; p = 0.126) or school environment (hypothesis 4e: standardised coefficient –0.014, SE 0.291; p = 0.961); school absence (hypothesis 4f: coefficient 0.061, SE 0.043; p = 0.149), bullying (i.e. social acceptance; hypothesis 4g: standardised coefficient –0.226, SE 0.321; p = 0.481) or exclusion from school (hypothesis 4h: coefficient –0.913, SE 0.575; p = 0.112) at this time point.

Furthermore, there was no evidence of intervention effects at the 12-month follow-up in relation to reading attainment (hypothesis 4i: standardised coefficient 0.100, SE 0.229; p = 0.661), concentration problems (hypothesis 4k: standardised coefficient –0.221, SE 0.240; p = 0.356), disruptive behaviour (hypothesis 4l: standardised coefficient –0.190, SE 0.240; p = 0.428) or prosocial behaviour (hypothesis 4j: standardised coefficient 0.164, SE 0.242; p = 0.499). In sum, our ITT findings at 12-month post-intervention follow-up (T4) were null for all outcomes.

Tables 19–22 present the findings of our longer-term follow-up at 24 months post intervention (T5).

At the 24-month follow-up, there was no evidence of the impact of the GBG on conduct problems (hypothesis 4a: standardised coefficient –0.400, SE 0.273; p = 0.143). There were also no intervention effects identified in relation to psychological well-being (hypothesis 4b: standardised coefficient –0.389, SE 0.260; p = 0.135) or emotional symptoms (hypothesis 4c: standardised coefficient –0.244, SE 0.321; p = 0.449); school environment (hypothesis 4e: standardised coefficient –0.485, SE 0.262; p = 0.064); school absence (hypothesis 4f: standardised coefficient –0.014, SE 0.049; p = 0.770), bullying (i.e. social acceptance; hypothesis 4g: standardised coefficient 0.036, SE 0.304; p = 0.907) or exclusion from school (hypothesis 4h: standardised coefficient 0.703, SE 0.490; p = 0.151) at this time point.

Furthermore, there was no evidence of intervention effects at the 24-month follow-up in relation to reading attainment (hypothesis 4i: standardised coefficient –0.006, SE 0.059; p = 0.919), concentration problems (hypothesis 4k: standardised coefficient –0.171, SE 0.253; p = 0.500), disruptive behaviour (hypothesis 4l: standardised coefficient –0.270, SE 0.252; p = 0.285) or prosocial behaviour (hypothesis 4j: standardised coefficient 0.300, SE 0.245; p = 0.220).

In the case of peer and social support (hypothesis 4d), there was a significant intervention effect at the 24-month follow-up (standardised coefficient –0.743, SE 0.241; p = 0.002). Contrary to predictions, however, this was a negative effect; in other words, children in GBG schools reported significantly lower levels of peer and social support than their counterparts did in the control schools. The associated ES was roughly equivalent to one-fifth of a SD (ES –0.195). In sum, our ITT findings at the 24-month post-intervention follow-up (T4) were null for all outcomes, with the exception of an unexpected negative effect on peer and social support.

Hypothesis 5

Children’s educational and health-related outcomes will be related over time.

The chi-squared difference test indicated that the model in which time-invariant covariates (e.g. trial group, sex, shared risk) were freely estimated (hypothesis 1) provided a significantly better fit for our data than the model in which they were constrained to zero (hypothesis 0) (χ² difference 954.831, degrees of freedom 9; p < 0.001). Longitudinal measurement invariance tests were conducted for the measures of conduct problems and emotional symptoms over time. Change in comparative fit index (CFI) was used to determine significant changes in fit. The differences in CFI between the configural and the scalar invariance for conduct problems and emotional symptoms were < 0.01.

A preliminary model without time-invariant covariates displayed an acceptable overall fit, with a CFI of 0.95 (scaled CFI 0.90), Tucker–Lewis index (TLI) of 0.95 (scaled TLI 0.90) and root-mean-square error of approximation (RMSEA) of 0.062 (scaled RMSEA 0.055). This model is statistically equivalent to hypothesis 0. The nested modelling procedure required the specification of parameter constraints on time-invariant covariates, but this had the undesired effect of lowering the overall model fit, with a CFI of 0.77 (scaled CFI 0.59), TLI of 0.8 (scaled TLI 0.63) and RMSEA of 0.113 (scaled RMSEA 0.096). This is most likely to be due to the estimation of means and variance for the covariates, which have an effect on the estimated coefficients of both the measurement and structural parts when comparing hypothesis 0 with the preliminary model.

The overall model fit of hypothesis 1 was also poor, but, as shown by the chi-squared difference test, it was an improvement from hypothesis 0, with a CFI of 0.81 (scaled CFI 0.63), TLI of 0.83 (scaled TLI 0.66) and RMSEA of 0.087 (scaled RMSEA 0.092). Below, we report model hypothesis 1, depicted visually in Figure 2 (between-person effects) and Figure 3 (within-person effects). For full details of the models, see Appendix 2, Tables 35–42.

FIGURE 2.

Between-person effects for emotional symptoms, conduct problems and reading attainment, and the influence of trial group, sex and shared risk (number of participants = 2987). ***p < 0.001; standard arrow, statistically significant pathway; dashed arrow, non-significant pathway.

FIGURE 3.

Within-person lagged and cross-lagged effects for emotional symptoms, conduct problems and reading attainment (number of participants = 2987), using R. ***p < 0.001; standard arrow, statistically significant pathway; dashed arrow, non-significant pathway.

Inspection of Figure 2 reveals significant between-person associations in emotional symptoms, conduct problems, and reading attainment. Thus, individuals experiencing emotional symptoms tend to also experience conduct problems. The between-person associations for both of these mental health variables and reading attainment were negative; hence, individuals experiencing emotional symptoms and/or conduct problems tend to also have lower reading scores. Furthermore, there are differential effects of our time-invariant covariates. Shared risk is significantly positively associated with emotional symptoms and conduct problems, and significantly negatively associated with reading attainment. In each case, a 1-unit increase in shared risk equates to change of approximately one-quarter of a SD in a given outcome variable. Sex is significantly positively associated with conduct problems and significantly negatively associated with emotional symptoms and reading attainment. Thus, being male is associated with an increase of just over half a SD in conduct problems, just below a fifth of a SD decrease in emotional symptoms and just over a tenth of a SD decrease in reading scores. Trial group is not significantly associated with any of these outcomes.

Figure 3 provides evidence of several notable temporal trends in the time-varying within-person part of the model. First, we observe significant increases in conduct problems over time, contrasted with significant decreases in reading scores. Second, there is evidence of significant cross-lagged effects, wherein emotional symptoms are negatively associated with later conduct problems. Thus, an increase of one SD in emotional symptoms at T3 is associated with just less than a fifth of a SD decrease in conduct problems at T4. This effect seems to increase over time, with a one-SD increase in emotional symptoms at T4 being associated with approximately a quarter of a SD decrease in conduct problems at T5. However, in relation to hypothesis 5, there is no evidence of significant cross-lagged effects between either emotional symptoms or conduct problems and attainment.

Hypothesis 6

The GBG will represent an efficient use of resources when considered from a public-sector perspective.

The economic analysis comprised a CCA of the available GBG data. The analysis was conducted from a public-sector perspective, with the inclusion of indirect costs to parents and caregivers. Up to 79% of the total costs of conduct disorder are borne by the child’s family,105 and inclusion is in line with recent publications in this field. 96

A CCA provides a descriptive analysis of the costs and outcomes of competing alternatives, which is summarised in a CCA balance sheet. The approach allows the decision-maker to draw their own conclusions on the balance of costs and benefits across the alternatives. CCA is the preferred method for assessing the impact of public health interventions106 and is increasingly used in studies exploring behavioural interventions. 96,107 In this analysis, costs were limited to GBG implementation costs and an estimate of the indirect costs related to exclusion from school. GBG impact is reported based on the primary study outcome (conduct problems; hypothesis 1a) and seven secondary outcomes [psychological well-being, emotional symptoms, peer and social support, school environment, school absence, bullying (i.e. social acceptance) and exclusion from school; hypotheses 1b–h, respectively]. Results are presented as disaggregated costs and outcomes in a cost–consequences table.

Analysis inputs

Analysis outputs

Hypothesis 1: main intervention effects (intention to treat)

The lack of any intervention effect across eight different health- and education-related outcomes (hypothesis 1) appears to provide a robust indictment of the GBG, at least from the perspective of the most bias-free analytical framework: ITT. Our findings align with those of the EEF-funded trial on which the current study was built, which found no impact of the intervention on children’s reading attainment, disruptive behaviour, concentration problems or prosocial behaviour. 1 As this is the first trial of the GBG in England, cultural incompatibility cannot be ruled out. Aspects of the IPE reported in the EEF-funded trial support this assertion; for example, many teachers reported struggling with certain mandated intervention procedures, most notably not being able to directly interact or intervene with pupils during gameplay. 1

However, the initial pilot in Oxfordshire concluded that, on the whole, the GBG was acceptable and feasible in the English school context,52 and so an apparent lack of cultural transferability cannot be the only explanation. Furthermore, there are also parallels with two recent US-based trials,18,37 which also found no main effect of the GBG on a range of outcomes (note that although the Ialongo et al. 37 trial did report significant intervention effects, this was for the combined ‘PATHS to PAX’ condition only; in the ‘GBG only’ condition, no main effects were identified).

These two recent trials notwithstanding, the majority of studies outlined in Table 1 that undertook true ITT analyses (recalling that three trials17,23,33 reported subgroup moderator effects only) identified at least one significant intervention effect (albeit typically with small or moderate ESs). What then are we to make of this apparent divergence from a clear trend in the evidence base for the GBG? The discrepancy appears unrelated to the nature of the outcomes assessed, given that these other trials have assessed similar domains to those reported here, including sometimes using the same instruments (e.g. the trial by Jiang et al. 19 used the teacher-informant report version of the SDQ from which our primary outcome of conduct problems and secondary outcome of emotional symptoms are drawn). Similarly, it does not appear to be an artefact of the specific version of the GBG used here, as the two Dutch trials also implemented the AIR model and reported positive effects. 31,32 Despite the fact that it is often trialled in combination with other interventions,17,29,35 the fact that the GBG was implemented in isolation in the current study cannot account for our null findings, given the multiple cases of positive effects where this is also the case. 19,21,31,32,34,36 Furthermore, at 2 years, the overall period of implementation was clearly sufficient, given that effects of the GBG have been found after as little as 10–12 weeks. 36,38

One immediate possibility is that, as noted in Chapter 1, children’s behaviour is typically very good in most schools, with very few displaying the symptoms of conduct or other problems at the outset of any given trial. The lack of main intervention effect may, therefore, simply reflect a low base rate of conduct problems in our sample (≈ 16% with scores in the borderline or abnormal range at baseline; and ≈ 60% with scores of 0 at baseline). Unfortunately, there is insufficient information provided in the Jiang et al. 19 study to enable a direct comparison in terms of base rate and scores for conduct problems, and so this explanation cannot be ruled out. Beyond this, there are three substantive explanations (suboptimal implementation, insufficient programme differentiation and delayed effects) that we focus on in our discussion of findings relating to hypothesis 3 [see Hypothesis 3: implementation effects (dosage)] and hypothesis 4 [see Hypothesis 4: maintenance/sleeper effects (12- and 24-month post-intervention follow-ups)].

Hypothesis 2: subgroup effects (boys at risk of developing conduct problems)

In relation to hypothesis 2, the lack of evidence for subgroup moderator effects relating to our subsample of at-risk boys across the range of trial outcomes conflicts with the trial evidence on which our hypothesis was based. 23,25 However, that study was set in a single, socioeconomically deprived American city (i.e. Baltimore) at a time when it was beset by crime, substance abuse, antisocial behaviour and myriad other social problems. By contrast, the current trial spanned 23 local authorities across 3 regions of England. Although this encompassed a very diverse range of settings, few, if any, could be argued to parallel the challenges faced by children and families in 1980s inner-city Baltimore. Thus, the ‘at-risk boys’ subgroups in the two trials were probably qualitatively different in respect of presenting patterns of aggressive and antisocial behaviour and the factors underpinning them. In other words, our subsample of at-risk boys may not have been sufficiently at risk to reap the benefits observed in the Baltimore trial.

In the context of the current study, it is noteworthy that teachers’ views on who benefited more (or less) from the GBG, solicited and reported in the EEF trial IPE,1 were mixed. Although some provided accounts indicating that the intervention had been more beneficial for boys whose behaviour was a cause for concern (consistent with hypothesis 2), others instead highlighted differential gains for children with distinct needs (e.g. those with autism), and several reported that they felt that any benefits were not specific to any particular children in their classroom. These findings align broadly with the subgroup moderator analyses reported here, in that they indicate a lack of clear and distinct benefit for boys at risk of developing conduct problems across classrooms in the intervention arm of the trial, at least to the extent that implementing teachers would consistently identify this in their accounts of the impact of the intervention.

Nonetheless, our findings have important implications for the utility of the GBG as an efficacious means of preventing the maintenance or escalation of conduct problems from childhood to early adolescence, a current priority for both the Government8 and NICE. 9 On the basis of the evidence accrued in our hypothesis 1 and hypothesis 2 analyses, this universal intervention should not be recommended as an efficacious approach, either to prevent the development of conduct problems (hypothesis 1) or to address existing conduct problems (hypothesis 2). Indeed, the only noteworthy finding in our subgroup moderator analyses was a potential negative intervention effect: at-risk boys in GBG schools reported significantly increased experiences of bullying. Although unexpected, there is a plausible explanation for this potential iatrogenic effect of the GBG. As noted in our description of the intervention (see Chapter 2, Methods), the GBG uses an interdependent group contingency model; in other words, the provision of rewards/reinforcement is contingent on all members of a given group abiding by the rules. This approach may inadvertently evoke negative peer pressure towards those who frequently break the rules (e.g. our at-risk boys subsample) and thereby reduce the likelihood of their group winning the game. 112 This aspect is a hypothesis for a clearly described and testable causal mechanism that future research could explore (including whether or not the effect is sustained in the short to medium term).

In the interim, this unanticipated negative intervention effect suggests that caution is warranted among those who might consider using the GBG. Although the associated ES is very small when judged by conventional standards,42 it is important to consider what it means in less abstract terms.

First, by transforming our standardised ES (Hedges’ g) to Cohen’s U³ [using the formula U³ = Φ(δ), in which Φ is the cumulative distribution function of the standard normal distribution, and δ is the population parameter of Cohen’s d/Hedges’ g], we can conclude that the subgroup effect of the GBG approximates to a 5-percentile point increase in bullying. Put another way, the observed ES of –0.125 means that 55% of at-risk boys in GBG schools will have a bullying score that is below the mean of those in usual-practice schools (remembering, of course, that a lower score reflects more frequent bullying in the measure used, and that, with no intervention effect whatsoever, we would naturally expect 50% to have a lower score).

Second, by referring back to Table 4, we can see that the negative intervention effect identified reflects an approximate decrease of just 1.5 points (SD of 12 multiplied by standardised ES of –0.125) in the possible scoring range of 0–100. For reference, a qualitative shift in the average reported frequency of bullying (e.g. from ‘seldom’ to ‘quite often’, or ‘quite often’ to ‘very often’ at the item level) would be between 9 and 19 points on the Rasch normalised scores used in the Kidscreen survey. 70

Last, the average scores for both the trial sample overall and the at-risk boys subgroup at T3 equated to being bullied somewhere between ‘never’ and ‘seldom’. Given all of the above, we conclude that, although the negative intervention effect observed is clearly real and potentially uncomfortable for the affected participants, its magnitude is unlikely to reflect significant harm. Despite this, we are mindful of the fact that this effect, however small, was observed among a particularly vulnerable subgroup of the population who already probably experience low levels of social acceptance. As such, we repeat our note of caution about the future use of the GBG based on the results reported here, unless adaptations can be made to intervention procedures that prevent the negative peer pressure hypothesised above from being the source of this unexpected finding.

This finding aside, the lack of differential gains among our at-risk subgroup provides evidence in support of a more general concern about universal interventions such as the GBG, which is that they lack the intensity to produce meaningful change for children already at risk for psychopathology. 113 This may particularly be the case in the current trial, in which the average dosage was markedly lower than both that recommended by the developer53 and the dosage that has been reported in other trials (albeit using teacher self-report in most cases). 18,19,36,37 One might tentatively predict that optimal intervention exposure matters most for those who are already at risk. The social adaptation process through which the GBG is believed to impact on behaviour is cumulative in nature, and those at risk are arguably in greater need of the increased opportunities for reinforcement, consolidation and generalisation of learning associated with increased levels of exposure, as this will mitigate the lack of adaptive socialisation in other developmental contexts. This is a point to which we return when discussing directions for future research at the end of Chapter 5.

Hypothesis 3: implementation effects (dosage)

As noted above, intervention compliance in the trial, specifically dosage, was suboptimal. The GBG was played for an average of 1066 minutes in total in classrooms in the intervention arm of the trial during the 2 years of implementation (i.e. approximately 530 minutes in the first year and 524 in the second year). Teachers played the game between once and twice a week, for an average of approximately 25 minutes per week in total. Furthermore, nine schools formally ceased implementation prior to the end of this main trial period. These data contrast sharply with dosage estimates in other trials. For example, teachers in Ialongo et al. ’s recent US-based trial37 reported playing the game for between 1432 (GBG only) and 1583 (combined GBG and PATHS) minutes on average in a single year of implementation. Furthermore, most of the teachers in Streimann et al. ’s Estonian GBG trial21 reported playing the GBG every day; in Tolan et al. ’s study,18 they described playing the GBG more than twice a day on average, for a total of nearly 80 minutes a week. A key caveat with these higher dosage estimates is that they rely on teachers’ self-reported implementation behaviour, which is known to be subject to positive bias and impression management. 40 Indeed, a key reason that we developed the bespoke online ‘scoreboard’ tool was to provide more robust, objective and accurate dosage estimates. 114 However, even if one assumes some level of ‘uplift’ in the teacher-reported estimates, the total duration reported here is still likely to be significantly lower than the actual level of dosage in the other trials.

Given the above, implementation failure provides a potential explanation for the null results in hypothesis 1. However, our CACE analyses do not appear to support this proposition, as we found no moderate- or high-compliance effects for the overwhelming majority of outcomes (the exceptions being a negative effect on well-being and a positive effect on school absence). In other words, the intervention was found to be ineffective even after robustly accounting for implementation variability. These findings align with those of Bradshaw et al. ,47 whose ITT and CACE analyses focusing on an at-risk subsample also returned null results for the GBG when implemented in isolation (as opposed to in combination with PATHS, for which there were significant intervention effects in the ITT analysis that grew in magnitude once compliance was taken into account). However, before ruling out implementation failure, we must consider the possibility that a ‘minimum effective dose’115 was not reached in even our high-compliance classrooms. In these settings, children were exposed to the GBG for > 1348 minutes across the 2-year implementation period. This is still significantly lower than the average cumulative intervention intensity achieved in a much shorter period in the above trials. 18,21,37 Thus, in the absence of a minimum effective dose, generalisation of learning (and consequent changes in behaviour and other outcomes) beyond the immediate context of the game itself may not follow. 116

In accordance with this, it is worth briefly considering the findings from the EEF trial IPE,1 which highlighted a range of factors affecting implementation. Notably, these included pupil needs (e.g. teachers were more likely to play the game frequently if they believed that there were children in their classroom for whom it would be beneficial), teacher attitudes (e.g. teachers were less likely to play the game frequently if they felt that it was difficult to integrate with their lesson plans), and competing priorities (e.g. teachers were less likely to play the game frequently during busy periods of the school year in which there were scheduled school events and/or assessments). The first of these qualitative findings (pupil needs) is supported by the CACE models reported in the current study, in as much as individual- and school-level characteristics indicative of need (e.g. conduct problems, prosocial behaviour) were found to be predictors of compliance. Taken together, the findings indicate that, in some contexts, such as those where pupil need is considered to be low, achieving a minimum effective dose may not be a realistic objective; in other contexts, where need is high but attitudinal or logistic barriers are present, there are implications for the training and support model used in the GBG (e.g. additional input to ‘win hearts and minds’ and problem solve implementation challenges).

Despite the above, we remain resolute in our conclusions about the implications for the GBG as a means of preventing the maintenance or escalation of conduct problems and other maladaptive outcomes in the English school context. This is because the dosage reported in the current trial is likely a ‘best-case scenario’ for implementation here, as it was achieved in an efficacy trial context in which initial training and ongoing coaching support for teachers, subsidised intervention costs for schools, additional provision for data monitoring from our research team, and developer (AIR) support for the delivery team, Mentor UK, were all available. 1 In other words, although we may have seen more evidence of meaningful intervention effects with significantly higher levels of implementation than those that were observed here, it is very unlikely that such levels would ever realistically be achieved were the GBG to be implemented at scale in England, in which case such a comprehensive implementation support system would be absent.

Setting aside insufficient implementation, a further potential explanation for our overall pattern of findings relates to programme differentiation. Recall that our survey of teachers’ behaviour management strategies and approaches revealed that those in the control arm of the trial were enacting practices that mirrored some of the core components of the GBG (e.g. classroom rules, team membership, monitoring behaviour and positive reinforcement). Given this, it is possible that the null results observed were due to the fact that the intervention was insufficiently differentiated from the usual practice of schools. This proposition appears to be supported by a general trend in which null findings are more commonplace in recent GBG trials (see Table 1). This may be the result of the behaviour management practices that are central to the intervention reducing in novelty and becoming more endemic over time (although we note that this explanation is confounded by the increased rigour of trials over time; more frequent null results in recent years may also be the result of more robust testing of the intervention). It may also explain why the GBG is so often delivered in combination with other interventions. 17,18,35,37,38 That is, in isolation, the GBG is more likely to be ineffective because of limited programme differentiation, but when implemented in conjunction with other preventative interventions, multiplicative effects are observed as a consequence of the interaction of complementary active ingredients. 117

Hypothesis 4: maintenance/sleeper effects (12- and 24-month post-intervention follow-ups)

Notwithstanding the preceding explanations for the lack of intervention effects, a further possibility is that they simply had not yet emerged at the end of the main implementation period. Preventative effects may take time to emerge, especially when a relatively small proportion of the population has (or is at risk of developing) problems in the first place. 118 Thus, comprehensive evaluation requires outcome assessment to go well beyond the cessation of a given intervention for changes among intervention recipients to consolidate, for small but key changes to snowball and for the members of the control group to exhibit difficulties of the kind that are the focus of prevention efforts. 119 Although longer-term follow-up studies of the GBG are scarce, those that were conducted prior to the current trial each found intervention effects (albeit following improved outcomes in the short-term; hence, they all establish maintenance rather than sleeper effects; see Table 1). However, this was not the case here. Indeed, the only notable effect in our post-intervention follow-up analyses was an unexpected one, whereby children in the intervention arm reported significantly lower levels of peer and social support than their peers in the control arm 2 years after the end of the main trial. Thus, although it is technically possible that sleeper effects may yet emerge in our trial cohort, this seems highly unlikely. As a consequence, we feel confident in ruling out the timing of effects as a contributory factor in the results of this study.

Hypothesis 5: the temporal association between mental health and academic attainment

The design of the current study afforded an opportunity to contribute to the growing literature on the temporal associations between health and educational outcomes. Specifically, research in developmental cascades has documented, over varying lags, ranging from 12 months5 to many years,56 the apparently reciprocal longitudinal relations between domains of mental health and academic attainment. We found no evidence of this in our analyses pertaining to hypothesis 5. However, it is important to note that most prior work in this area has used cross-lag panel models that are unable to disaggregate between- and within-individual effects. This means that the apparent cross-domain within-individual temporal effects reported in these studies may be erroneous. 86 In the current study, we used the RI-CLPM approach, finding clear trait-like between–individual associations (e.g. children experiencing emotional symptoms and/or conduct problems tend to also have lower reading scores), but no evidence of genuine within-individual temporal associations between either domain of mental health and academic attainment (albeit with relatively poor model fit). Thus, after controlling for shared risk and between-individual effects, we do not find support for the hypothesis that mental health influences later attainment, or vice versa, at the end of middle childhood. Whether or not our analysis is an outlier remains to be seen; further research that uses RI-CLPM in this space is needed.

Hypothesis 6: costs and consequences of the Good Behaviour Game

Our health economic analysis found no evidence to support the argument that the GBG provides value for money. The total implementation costs of the intervention were calculated at £11,318 per school over the main trial period (a cost of £275.68 per pupil). This intervention cost was the driver in cost differences in the trial, with the only other captured cost, exclusion from school, showing little difference (£0.48 per pupil) over the same time period. Examining the outcomes at deeper levels of granularity (e.g. mapping costs to level of compliance and/or dose) was not possible because of the aggregated nature of the available data.

It should be noted that our reported outcomes are based on a single year group in each intervention school being exposed to the GBG, whereas, in a real-world setting, a roll-out across the whole school might be expected, resulting in a lower cost per pupil. Hence, the main report of the EEF-funded GBG trial estimated that the cost of implementing the GBG within primary schools was similar to that noted here, £11,000 per school over a 3-year period, comprising £4500 start-up costs in the first year and £3000 per annum in the subsequent years, but a per-child implementation cost of £37 that assumed roll-out across the whole school. 1 In addition, it should be noted that Mentor UK’s original plan for roll-out of the GBG programme was to include a discount for grouped school uptake, which may have resulted in lower costs; their costs were modelled on a 10-school cohort, and larger cohorts would probably incur lower costs owing to shared resources (e.g. coaches).

Previous studies of the GBG have indicated strong performance and good value for money, with the Washington State Institute for Public Policy (WSIPP) most recently reporting a 60-fold return on an investment, that is a return of $63 for every $1 spent implementing the intervention. 120 However, although targeted parent-level interventions for the prevention of conduct problems have shown good economic return in the UK,121 in general, strong economic arguments for UK-based classroom-level interventions remain limited. Recent studies have highlighted the challenge of capturing the benefits of such approaches within a standard economic framework,96,122–124 with the per-child cost estimates of trialled interventions ranging from £8124 to £153. 123 These findings are not directly comparable with the GBG cost estimates reported here because of the limited granularity of the data.

Owing to the reporting restrictions on exclusions data,87 it is not possible to elicit, in either arm of the trial, whether these exclusions represented one child being excluded many times or many children being excluded once; this is information that could be important when assessing the impact of the GBG on exclusion from school. Further exploration of findings could help to focus the intervention on schools that are most likely to benefit. It is also possible that there are other measurable costs that were not captured by the study that may be important to the health economic assessment of the GBG (e.g. health and social care utilisation); however, given the lack of available data, the decision on whether or not the intervention should be implemented further is likely to be made using the consequence outcomes (hypotheses 1a–g).

Non-hypothesised findings

In the course of our analyses, some notable findings emerged that did not directly relate to our prespecified trial hypotheses. In keeping with the intention to avoid mining/dredging, these have been compartmentalised from the main findings discussed above and are noted briefly here, with a very clear caveat that they should be considered as exploratory.

The first finding relates to subgroup moderator effects. Modelling our hypothesised subgroup effect (differential effects for at-risk male pupils) required us to fit a three-way interaction term (e.g. GBG*male*at risk); doing so meant that we also needed to fit three possible two-way interaction terms involving the trial group, sex and risk status variables (e.g. GBG*male, GBG*at risk, and male*at risk). In the course of doing so, we observed significant interactions between trial group and risk status that denote reductions in bullying and absence among at-risk students in GBG schools; we also observed a significant interaction between trial group and sex that was indicative of an increase in absence among male pupils in GBG schools (see Tables 10 and 11).

The second finding relates to within-individual temporal associations between emotional symptoms and conduct problems. As above, these were not specified in the study hypothesis and are an artefact of the modelling process. Here, we found a consistent pattern in which emotional symptoms were significantly inversely related to later conduct problems. From T3 to T4, an increase of one SD in emotional symptoms was associated with just less than one-fifth of a SD decrease in later conduct problems. This effect increased in magnitude from T4 to T5, with a one SD increase in emotional symptoms being associated with approximately one-quarter of a SD decrease in later conduct problems. This finding supports the proposition that the experience of emotional symptoms may serve a protective function in relation to later conduct problems by interrupting the trajectories from risk to disruptive behaviours, perhaps owing to the increased self-isolation and withdrawal associated with emotional symptoms. 125

Strengths and limitations

The current study has numerous strengths that increase the security of our findings. First, a cluster-randomised design with appropriate analyses that took account of the hierarchical and clustered nature of the data set was used. Second, the trial was very large and well powered; furthermore, the 77 trial schools spanned 23 local authorities across 3 regions of England, thereby providing a much greater diversity of settings than most other studies of the GBG. Third, attrition was within acceptable limits, being 0% at the school level and 18.5% at the child level at the point of our main ITT analyses, and missing data were accounted for using FIML, eliminating the bias associated with complete-case analysis. Fourth, the use of a randomised design (in which allocation was determined by an independent trials unit) meant that, in expectation, we would be free from confounders. Fifth, the use of a cluster-randomised design and the proprietary nature of the GBG minimised the possibility of contamination effects. Sixth, our assessment of primary and secondary outcomes used multiple methods (e.g. surveys, standardised tests) and informants (e.g. children, teachers), which was consistent with recommended practices. Last, our design enabled a very thorough and comprehensive assessment of the GBG, including the robust examination of three key effect modifiers: subgroups, implementation and timing of follow-up.

Nonetheless, there are also a number of limitations that need to be considered. First, children and teachers completing outcome measures were not blinded to trial allocation, which potentially introduced bias (although this seems unlikely given the null findings). Furthermore, given the lack of blinding, we cannot rule out compensatory rivalry as a partial explanation for our findings (e.g. increased efforts among usual-practice schools in response to not being allocated to the intervention arm of the trial). Second, independent observation of children’s behaviour would perhaps have been preferable to the use of surveys; however, this was not feasible within the resources available for the trial and would also have significantly increased the data burden on participating schools. Third, our analyses were reliant on point-in-time estimates, and it has been argued that these do not provide a fair test of the efficacy of preventative interventions, which are designed to alter children’s developmental trajectories. 119 Fourth, intervention compliance was suboptimal, and, although our CACE analyses indicated that outcomes mostly did not vary as a function of dosage, we cannot rule out the possibility that a minimum effective dose was not reached, even in our high-compliance settings. Fifth, our CACE analyses were specifically related to dosage, meaning that other potentially important dimensions, such as procedural fidelity, were neglected (although these appeared to be less variable and much closer to optimal levels than dosage). Sixth, the ‘augmented’ nature of the trial meant that, for several of our secondary outcomes, a post-test-only design was used rather than the generally preferred pre-test-post-test design that was used for our primary outcome. However, as a counterpoint, we note Gorard’s argument126 that the post-test-only design is ‘generally at least as safe as its alternatives’. Last, as for any study of this kind, attrition occurred and increased over time (e.g. 80% of participants with primary outcome data for both T1 and T3; 76% with data for both T1 and T5). However, appropriate techniques to account for missing data (e.g. FIML) were employed where possible, meaning that the majority of models were based on the full trial sample (n = 3084), thereby reducing the bias associated with attrition.