Initiating change locally in bullying and aggression through the school environment (INCLUSIVE): pilot randomised controlled trial

Consultation with young people on measures of bullying and aggression

We consulted with young people from the NCB YRG at a meeting held before the surveys were undertaken to explore young people’s views concerning how to define bullying and aggression within schools and assess whether the aggression items designed for INCLUSIVE were appropriate and acceptable (see Chapter 1, Involvement of young people). It was important to assess young people’s views on what constitutes aggressive behaviour both to inform the aims of the intervention and to inform our measures. The group identified a wide range of behaviours that could be perceived as bullying and aggression, including physical violence, verbal abuse, ‘jokes’, cyber bullying, pictures, drawings and websites, rumours, exclusion, peer pressure and theft. They concluded that bullying should be defined based on how often it occurs, arguing that repetitive and relentless behaviour is more likely to be experienced as aggressive.

We then asked the young researchers for their opinions on the scales we were testing as potential primary outcomes for a Phase III RCT. The group identified some concerns with both questionnaires. There were concerns about false reporting, which might be improved by ensuring confidentiality. Some items on the AAYP violence scale were thought of as possibly too ‘extreme’, leading to concerns that no British students would report these (e.g. stabbing someone). In contrast, other questions might assess experiences reported by almost everyone (e.g. using bad language). Some items were considered not to be well defined (e.g. verbal bullying), which may lead to variability in student responses. However, in general, the young researchers felt that the measures were likely to reflect students’ experiences of aggression and bullying and were appropriate to pilot.

Quality of life

This was measured using two instruments. First, the 30-item Paediatric Quality of Life Inventory (PedsQL) version 4.080 was used to assess all-round quality of life (QoL). The PedsQL has been shown to be a reliable and valid measure of QoL in normative adolescent populations. 80 It consists of 30 items representing five functional domains: physical, emotional, social, school and well-being. Items are rated on a series of five-point Likert scales ranging from 0, ‘never’, to 4, ‘almost always’. The PedsQL yields a total QoL score, two summary scores for ‘physical health’ and ‘psychosocial health’ and three subscale scores for ‘emotional’, ‘social’ and ‘school’ functioning. For the total QoL score, items are reverse-scored and linearly transformed to a scale of 0–100 (i.e. 0 = 100, 1 = 75, 2 = 50, 3 = 25 and 4 = 0); higher scores represented better QoL. Second, the CHU-9D measure81 was used to assess health-related QoL and piloted and examined prior to use in a subsequent economic analysis; the validation of the CHU-9D is described below in detail (see Validation of the Child Health Utility 9D questionnaire).

Psychological function

Psychological distress was measured using the child self-completion Strengths and Difficulties Questionnaire (SDQ), a validated measure of a range of behavioural and emotional problems in children and adolescents. 82 The SDQ consists of 25 items which assess difficulties in four domains (emotional symptoms, conduct problems, hyperactivity/inattention, peer relationship problems) and an additional measure of pro-social behaviour. Items are rated on a scale of 0 (not true) to 2 (certainly true). A ‘total difficulties’ score is calculated by the summation of all scales, apart from pro-social behaviour, a high score being indicative of a higher level of emotional and behavioural difficulty. In this study, the total SDQ score and the emotional, conduct, hyperactivity and peer problem scores were treated as continuous variables.

Psychological well-being

Psychological well-being was measured using the Short Warwick–Edinburgh Mental Well-Being Scale (SWEMWBS), which consists of seven items designed to capture a broad concept of positive mental well-being, including psychological functioning, cognitive-evaluative dimensions and affective-emotional aspects. 83 Items were rated on a five-point scale: none of the time (score = 1), rarely (2), some of the time (3), often (4), all of the time (5). The responses are scored and aggregated to form a ‘well-being index’ (total score), which can range from a minimum of 7 (those who answered ‘rarely’ on every statement) to a maximum of 35 (those who answered ‘all of the time’ to all statements). Higher scores represent improved mental well-being. The SWEMWBS was piloted in this study because it potentially provides a brief (seven-item) measure of well-being, although it has not yet been validated with year 8 students in English schools.

Health risk behaviours

Measures of two key adolescent health risk behaviours were piloted as secondary outcome variables. Self-reported use of tobacco and consumption of alcohol were examined using the HSE pilot trial measures. 69 Students were asked if they had smoked in the past month (30 days) and if they had drunk alcohol, defined as ‘more than a sip’, in the past month (30 days).

Note that drug use and contraceptive use would also be secondary ‘health risk behaviour’ outcomes in a Phase III trial in which students would be followed up until the end of year 10 (aged 14 or 15 years), but they were deemed not appropriate for the pilot trial surveys of year 8 students (aged 12 or 13 years).

NHS use

Self-reported use of NHS services (primary care, accident and emergency, other) in the past year was scored as a dichotomous variable (i.e. those who reported that they had used NHS services in the last year were given a value of 1).

Contact with the police

Contact with the police was assessed using the Young People’s Development Programme evaluation measure. 84 Young people were asked if they had been stopped, told off or picked up by the police in the last 12 months. This was a dichotomous response variable (ever/never had contact).

Truancy and school exclusion

Self-reported truancy and school exclusion were measured using items from the ESYTC survey. 78 Self-reported truancy was a dichotomous variable (‘ever bunked/skipped school? yes/no’), as was self-reported school exclusion, which was assessed according to whether they had ever been permanently or temporarily excluded from school.

Student perceptions of school climate

The Beyond Blue School Climate Questionnaire (BBSCQ) scale was used to measure students’ perceptions of the school climate. The scale was originally developed in Australia,85 using items selected from the Quality of School Life,86 Patterns of Adaptive Learning87 and Psychological Sense of School Membership questionnaires. 88 It consists of 28 items, which produce an overall score and also assess four key domains of school climate (subscale): supportive teacher relationships, sense of belonging, participative school environment and student commitment to academic values. Each item was coded 1–4 on a four-point scale. Responses ranged from ‘Yes, totally agree’ to ‘No, totally disagree’. A total BBSQ score was calculated by the summation of all scales, reversing scores when necessary. The total BBSQ score and subscale scores were treated as continuous variables; higher scores represent a positive report of school climate.

Antischool actions

Students’ reports of antischool actions were derived from the ESYTC self-reported delinquency (SRD) subscale. 79 The total SRD score was a summed frequency based on six items, with high scores indicating higher levels of self-reported school delinquency.

Student approval: ‘having a say’, ‘having respect’

Key cross-cutting themes in student and staff accounts were that the focus on students ‘having a say’ and greater ‘respect’ for students’ views were important sources of acceptability and might, as intended in our logic model, be a key component in terms of promoting student commitment to school and thereby reduce aggression and bullying. For example, year 8 students consistently reported that the needs assessment survey and the action group had given them a greater ‘voice’. Furthermore, the action group was seen as a new ‘experience’ to be more ‘involved’ in ‘having a say’ at school. This provided a strong motivation for joining action groups. For example, one student explained:

I’ve never been involved in things like that before, so I thought it would be a new, good experience.

Action group student, year 8 female, Goldstone Park

Specific practices arising from the intervention, such as circle time and restorative conferencing, were also seen as acceptable and ensured that students across the school felt they had more ‘voice’. Students frequently reported that they considered such practices appropriate because they addressed both the tendency of some school staff not to ‘listen’ to students and key problems in schools such as unhappiness and fighting:

A PE teacher, he did it [circle time] in PE, before the lesson we all sit in a circle on the benches and we’ll just speak, like how we’re feeling and what’s made us feel like that.

Male student, year 8, Whitehorse Road

There are fights, there are situations that like need restorative justice conferences.

Action group student, year 8 male, Williamson High School

Changing the school environment in order to make it ‘calmer’ and more ‘respectful’ was also considered a very appropriate aim and also motivated participation of students in the project:

Yeah, I thought that because [with restorative approaches] you’re saying things in a more calmer way they’d want to listen more and take in what you’re saying [ . . . ] It is needed in this school!

Action group student, year 8 male, Williamson High School

This perception of a ‘lack of respect’ from teachers and other staff at their school was an overarching theme among students’ accounts of their school life and a key source of frustration, disengagement and conflict. For example, one student explained:

We’re the students, yeah, it’s ‘you’re excluded’ – teachers make things sounds worse than they actually are [ . . . ] and then you have to respect them all the time.

Female student, year 8, Whitehorse Road

A female student, who reported poor relationships with some staff at Goldstone Park and had already become quite disengaged from school during year 7, reported that her motivation for joining the action group was partly a result of such frustration:

Teachers think we have to respect them, they don’t give us respect back.

Action group student, year 8 female, Goldstone Park

Students also often reported that there were ‘only one or two’ teachers or other staff members to whom they felt comfortable talking. Overall, students felt that the action group and restorative practices ‘could work’, as per our logic model, for improving student–teacher relationships in this context, which was a strong source of acceptability to them. A student member of the action group at Williamson High School explained that ‘slowing down’ the disciplinary process to listen to students was needed to improve relationships:

As teachers obviously you have to be strict but there are times, there are like limits like instead of [using] on-calls [internal exclusions so much]. If I’ve sworn, if I’ve been totally rude to the teacher, my behaviour’s bad, yeah, but for doing a little thing straight away, bang, on-call, that’s, I just think it’s ridiculous, that’s why I think it would work on the teachers, tell them to sort of you know, slow it down!

Action group student, year 8, Williamson High School

Staff approval: an ‘attractive’ locally led process of ‘positive’ behaviour change

School staff members were consistently supportive of a new whole-school restorative approach for addressing bullying and aggressive behaviours in English secondary schools, which provided a strong source of acceptability and had prompted interest in participating in this pilot trial. One head teacher explained:

I guess what attracted us was the idea that, you know, we could get something out of it [ . . . ] I mean, we’ve been trying to develop a kind of, more restorative approach to secure behaviour over a period of time [and] so this fitted in with the way we want to work.

Head teacher, Whitehorse Road School

School SMT members consistently reported that they were already aware of some restorative approaches being used either in their school or elsewhere, although, if they were being used at their school, they consistently reported that such approaches had been used inconsistently to date and not employed in a whole-school manner. The intervention was therefore highly attractive because it could provide a new framework, process and resources for ‘embedding’ restorative practices more coherently and consistently across the school:

The reason I wanted us to be a pilot was because I was very interested in the concept of the restorative justice and I had tried to incorporate that to some extent in our systems [ . . . ] I wanted it to be embedded across [this] school.

Head teacher, Railside High School

The intervention was also attractive to the new head teacher at Goldstone Park, who opted to participate in the pilot trial in his first year at the school because he felt it addressed what the school ‘needed to do’ most urgently in terms of students’ behaviour:

[I thought the intervention] sounded really interesting, and would be perfect, and it was just before I started here [I found out]. I knew some of the issues, because I had worked with the SMT on where we needed to go [prior to coming into post], and what we needed to do. [ . . . For example] staff not always dealing with students in a particularly positive way, so by that I mean, quite a lot of shouting, a lot of negative feeling about, when they’re dealing with behaviour.

Head teacher, Goldstone Park

The adaptability of the intervention, in contrast to overly prescriptive, ‘one-size-fits-all’ interventions, was also a strong motivating force and a source of acceptability to school managers. They felt that the process of implementation would meet local needs regarding behaviour change and that they could ‘make it work’ locally for their own specific institutional context. The new head teacher at Goldstone Park explained:

Were there any potential negatives going into it? I don’t think so because, you know, with these sort of things you think we’re going to make it work for us. So, if problems arise we’ll address them and if we need to adapt it for our context and our circumstances we’ll do that, so I didn’t really have any doubts.

Head teacher, Goldstone Park

The focus on adaption to the local school context avoided schools having to ‘reinvent the wheel’ and allowed them to build on their existing work. The assistant head teacher at Railside High School, who co-ordinated the intervention locally, emphasised how the restorative practices that staff were trained in during the intervention both complemented and added value to what they had done in the past, as well as supporting school managers’ long-term vision and philosophy:

The checking in, checking out, circle time, the idea of restorative language with staff and students, which again goes along with things we’ve done in the past in terms of how you all speak to each other. That is an area that we could develop as a school, you know, that every interaction we have, needs to be with that philosophy behind it.

Assistant head teacher, Railside High School

Other school staff were also supportive of this new structured process to formalise and extend restorative practices across the school, some of which had already been delivered, albeit inconsistently. A senior member of support staff explained:

Selling it to staff in a really positive way is the key: ‘I’m not bringing you something new, you’re doing this. What I’m bringing you is how we can make it more structured. What I’m bringing you are the skills to make you better at it’. Because there are key questions that you should and shouldn’t be asking when using RA [restorative practices] [ . . . ] That’s what the training and the formalisation does.

Pastoral support manager, Goldstone Park

Members of the SMTs and other staff also consistently reported how schools are ‘restless’ to improve staff–student relationships and address aggressive behaviour more effectively. The intervention was seen as a helpful ‘push’ and source of community mobilisation within this context. Staff members at intervention schools also reflected on the value of this ‘external push’, which was provided by the creation of action groups, the provision of an external facilitator and the needs assessment data, which identified clear priorities to inform the process of addressing behavioural problems. For example:

It’s the external push which I think is always really important, because if those meetings weren’t happening regularly and somebody from outside wasn’t coming in to make sure they were happening . . . I’m not saying it wouldn’t happen, but I think sometimes that external support, that external push, external pressure makes sure that those things do happen.

Head teacher, Goldstone Park

The education ‘market’: supporting adoption and implementation

School managers and the intervention team reported that a strength of this intervention was its fit with national education policies and schools’ ‘core business’ of teaching and discipline. In particular, a restorative approach to changing behaviour in schools was seen as highly compatible with institutional priorities and the criteria against which schools are assessed. This was therefore a further source of acceptability, and facilitated both interest in the trial and implementation of the intervention for those schools allocated to deliver it. For example, the desires to promote a safe environment and more supportive teacher–student relationships were consistently reported as institutional priorities in order to raise attainment. The head teacher at Railside High school explained why improving relationships was intrinsic to achieving their priorities:

Our key priorities are raising attainment through quality teaching and learning, and, within that, there are very specific priorities [ . . . ] [One specific priority was] behaviour, behaviour for learning in the classroom, but also behaviour around the school, that creates a safe environment. [ . . . So] if we don’t address those we’re never going to allow them to achieve their potential academically, so that’s why this fits in. For us it’s not an interesting project that we tinker in because everything else is running along finely, it’s intrinsic to what we’re trying to do in terms of raising attainment.

Head teacher, Railside High School

School managers also felt that the intervention was timely regarding the strong focus within the national inspection framework on managing conflict and aggressive behaviours, and this approach appears to be appropriate both for schools developing new school improvement plans, in light of previous Ofsted reports that identified this as a specific priority, and more generally for those schools that were planning and preparing for future inspections. A member of the SMT team at Whitehorse explained how this project would ‘hopefully come out’ positively for them during an Ofsted inspection:

We are due to have an Ofsted inspection next year [ . . . ] we would like to think that when they speak to students, for example, they will be able to talk about this, whether it’s ‘I’ve been involved’ or ‘I’ve noticed there’s been a change’, or ‘we studied these units in personal development’ so hopefully it will come out. But also more directly . . . I’ll probably have a conversation with them [the Ofsted inspectors] about behaviour for learning and one of the things I will talk about is that we have, you know, we spent this year looking at our behaviour policy in relation to this project and you know, we decided to go forward with more restorative practice.

Deputy head, Whitehorse Road

Goldstone Park was the subject of an Ofsted inspection towards the end of the 2011–12 academic year. Inspectors praised the school’s new approach to aggressive behaviours and its initial successes, which helped build further whole-school support for the intervention and a restorative approach, following this ‘glowing’ report:

I think we’ve had some success. [ . . . ] Do they all really believe it yet? I think that’s to do with results and outcomes isn’t it? I think we’ve started to get that because of the Ofsted report, because of all the positive feedback on behaviour, because of our survey that we did with students and parents, came out very positively [ . . . ] This really tells us, or gives us the evidence that things are changing. Compared to the previous Ofsted before, this says the students’ behaviour is excellent.

Head teacher, Goldstone Park

Attendance and exclusion rates are also key ‘metrics’ for head teachers, especially in schools with a high proportion of students from lower socioeconomic groups or higher rates of students with special educational needs, which was seen as a further incentive to adopt new restorative approaches to aggressive behaviour. One head teacher explained:

One of the measures [we] use is attendance and we’ve gone up from 92.8% to 93.4%. [and] we have challenging children in our school, and we have a high number of fixed term exclusions [ . . . ] You have to learn different strategies, and while we had lots of different strategies which we could offer, this seemed to be a new strategy which seemed to be much more organised which we could use.

Head teacher, Williamson High School

Anecdotal evidence from Williamson High School suggested that the approach may have contributed to improved relationships and behaviour at the school, as a reduction in the number of internal (‘on call’) and fixed-term (external) exclusions was observed:

There seems to be a notable reduction in on-calls [this year . . .] It’s very difficult to say but I [normally] get an on call sheet out on each Thursday, and there are peaks and troughs on that, but it does seem to be, I would say, this is an anecdotal evidence – I haven’t got the [final] evidence to back it [up but] I will have in the summer holidays when the year is complete – but it looks to me like a one third reduction in on-calls.

Head teacher, Williamson High School

Implementation challenges: co-ordination, communication and timing

Although school staff reported that the intervention was valued and appropriate, it was most acceptable once all school staff understood how the different components fitted together and how it ‘all tied in’ with their work and roles.

However, other staff reported ‘not knowing enough’ and that it was not always clearly communicated to them how it all ‘married up’. For example:

I can see how it would help relationships but at the same time I think we don’t know enough, like, I don’t know enough about what, like I said, the aim is for the school. So I don’t know how exactly it fits into the action group.

Action group member, female staff member, Goldstone Park

It was suggested that a clear, more engaging, launch to all school staff as part of the half-day training and information session at the very start of the year would support communication and facilitate a better understanding and interest across the whole school. Improving the intervention manual and resources to include more ‘real-life’ examples was also suggested as a means to improve communication and support co-ordination:

The one thing schools need is a model, of how it’s going to work in the school, in a real life school, so that they can almost touch it, taste it, feel it, and then start implementing it in their own schools.

Head teacher, Railside High School

It was also suggested that ‘front loading’ the enhanced three-day training at the very start of the project would further facilitate whole-school co-ordination and communication via ‘cross-fertilisation’ and the diffusion of information by local ‘champions’:

You always need cross-fertilisation [ . . . ] the ten of us [who did the enhanced three-day restorative approaches training] could be spread out so that, or we could have given feedback, [been] the champions of it to support it [ . . . ] She’s obviously got lots of materials, so you don’t want to keep re-inventing the wheel [but] we could have then actually helped lead it and spread the messages so it [had] more ownership to the school really.

Head of year, Railside High School

Members of the SMTs at intervention schools also reflected on the need for greater clarity regarding what time commitment was required of staff to implement the intervention:

It was very positive. I mean I haven’t got anything to say negative, just like I say it’s, you know, any school is going to [be] just busy really. I suppose all organisations are, but that’s just the way it is. And so the school needs to fit this project into how it works and that can be tricky at times [ . . . ] but I think we underestimated the commitment and involvement further on.

Deputy head teacher, Whitehorse Road School

The issue of ‘timing’ was a recurring theme in intervention school staff members’ accounts and seen as a challenge to implementing and co-ordinating multiple components across the whole school. There was a consensus that the intervention would benefit from earlier recruitment and randomisation of schools to support forward planning. This, in turn, would support more timely and effective delivery. One facilitator was previously a head teacher and this enabled him to highlight the challenges associated with not being able to organise staff training and meetings in advance of the new school year:

[Head teachers’] planning starts at the end spring, beginning of summer term [for the next academic year]. A lot of the working through [so the head teacher] really needs to have that written into their planning and because it wasn’t written in it made trying to get in space for more training days, in meetings with staff, very, very difficult.

Bryn, action group facilitator

It was acknowledged by school staff and intervention facilitators that a major strength of the intervention was its responsiveness to local needs and priorities, and that it recognises schools ‘start from different places’, as one of the intervention team put it. However, the intervention co-ordinator felt that the intervention should have begun more quickly after randomisation. This would have enabled facilitators to draw on the needs assessment data to engage school SMT and other staff:

The schools didn’t get the data quickly enough, and one of the things I was selling them on was that they were gonna get high quality data that was expensive and they were getting free.

Vanessa, intervention manager

The pilot study thus established that a full trial must begin recruitment no later than midway through the academic year before the academic year in which the intervention would be implemented, with baseline surveys undertaken before the summer holidays, in order to support timely intervention planning and implementation. There was also a consensus that the intervention should extend beyond one school year. School and intervention staff emphasised that few benefits would be expected after only one academic year because it takes longer than that not just to implement all the components, but to modify the school culture:

This project, it’s just over 1 year [ . . . ] This is all about changing the school ethos. Change school ethos in a year? Well, you’re gonna start nibbling at it but [ . . . ] to do it after 1 year I still am very sceptical about that, whether in fact you can measure any impact.

Bryn, External Facilitator

A recurring theme was that the pilot intervention had been successful at initiating changes, particularly considering the truncated timescales, but that real impacts in the school would manifest only in the following school year. For example, all the intervention schools described how they expected to see greater use of restorative practices the following year, after the pilot trial had finished, which is further evidence that both a longer-term approach and a longer period of follow-up are required than were possible in this pilot trial:

We’ll see that more next year, because like I say, the way in which the project has worked out for us, is very much putting things in place for next year, and I think one of the key things, will be restorative language and how well all of our staff are able to adapt to that and use that.

Deputy head teacher, Whitehorse Road

What I wanted to achieve is restorative justice. This is another strategy and we have trained professionals who will now train you on that technique and we will try and use it, and [ . . . ] then next year it will be embedded into the school and it will be [in] our new behaviour policy, it will be mentioned in our pastoral handbook and it will be mentioned in guidelines for new teachers . . .

Head teacher, Williamson High School

The intervention team’s views: getting a ‘head start’ to build on a ‘success’

The training provider, the intervention manager and the other two educational consultants who also facilitated school action groups during the intervention were highly supportive of this whole-school restorative approach, and all reported that they felt it was a ‘success’ based on their observations. They all reported that the intervention was acceptable and feasible across all four schools, with all participants supporting the notion of a ‘universal’ approach that could be adapted to different institutional contexts and was responsive to local needs. The intervention team also suggested that a whole-school restorative approach may have a particularly ‘natural’ place in the most ‘challenging’ and ‘needy’ schools with very high proportions of disadvantaged students, which suggests the intervention’s potential to address inequalities in adolescent health outcomes:

I think, it’s quite interesting that the schools that were arguably most needy implemented it best.

Vanessa, intervention manager

The intervention team were also positive about sustainability. One action group facilitator explained that the intervention was not ‘just going to stop’ at Railside High School, which was a testament to its feasibility, acceptability and value. It was also seen as addressing an important gap in secondary schools by specifically giving younger students a voice and greater responsibility, which typically does not happen until they enter year 11:

A lot of pupils in primary school have a lot of roles and jobs, and then as soon as they get to secondary school, they don’t. They’re not given these roles and responsibilities until year eleven. And [the action group students] very much took on board this and got people responsible.

Vanessa, intervention manager

It was suggested that we capitalise on the adaptability of the intervention to the local context as a source of acceptability and motivation by making it ‘even more bespoke’ and locally adaptable, including through greater integration with school improvement plans:

It has to be more bespoke. [ . . . ] I mean, essentially, what you have, we should have gone in and asked the schools to talk to us about their School Improvement Plan to start off with, or their School Self-Evaluation [for Ofsted], that’s supposed to be their core document, and say, ‘What are your priorities?’ and consequently, ‘Which ones are going forward? Which ones are you resolving? Which ones have you got obstacles? And how do we fit into that?’.

Vanessa, intervention manager

Intervention team members also saw the broader policy environment as a major facilitator to implementation, as reducing aggressive behaviour is part of schools’ ‘core business’ and a priority for SMTs, especially (although not only) at the most ‘challenging schools’. For example, it could give them an important ‘edge’ for future Ofsted inspections, as well as address ongoing concerns for all secondary schools regarding ‘disaffection’:

I mean if you think how important Ofsted and that sort of thing is to them, anything that can give them an edge is really important!

Dawn, action group facilitator

Every secondary school has an issue with disaffected kids from year 9 onwards, every single school. I don’t think there’s a school in the country that could say they didn’t. [ . . . ] No one’s solved disaffection yet.

Vanessa, intervention manager

The greatest testament to the intervention’s acceptability among staff and students, and their motivation for a process of whole-school restorative action, was that Whitehorse Road continued to implement all the intervention components during a turbulent year in which staff and students discovered the school was to be closed down:

It’s a school that halfway through the intervention they decided, the governors decided, they were going to close the school. They’re not taking any new intake from this September, and that [ . . . ] of course that had an impact on, around the school and it’s to their credit that they want to remain in the project and indeed want to continue developing their work with restorative justice, despite that, you know, that distraction.

Bryn, external facilitator

Local needs assessment survey: identifying priorities for action

School staff consistently reported that the needs assessment survey allowed them to see the ‘big picture’ and identify priorities for action, in terms of both the nature of aggressive behaviours at their school and the extent of ‘risk’ and ‘protective’ factors for conflict and aggression within the school environment. In some cases, the needs assessment data provided confirmation of existing concerns and additional motivation to mobilise the school community towards a ‘positive’ change:

Did we believe the results? Yes probably. It was done with year 8s, which was our most concerning year group at the time. [The survey found that] they were feeling quite negative about the school and I think that’s because they were getting lots of negative messages again, and it was because we were enforcing them: ‘you are the worst year group in the school, your behaviour is terrible, stop behaving so badly’, all from a negative perspective. We’re hoping that the second round of the results comes back better and shows improvement, because what we’ve tried to do, alongside all of this work, is to keep re-emphasising the positive.

Head teacher, Goldstone Park

In other cases, some of the findings were reported to be a ‘surprise’, which led to new strategies and actions to address problems likely to be associated with student aggressive behaviours. For example, at Railside High School students were reporting high levels of disengagement and low aspirations for the future, which informed the action group plan:

The issue around aspiration and hope, and their futures . . . I think that actually was the biggest surprise, because as a school, that’s something we do work on all the time so that, kind of, well that’s not good! [ . . . ] And that’s why we planned the horizon day, which was designed around looking at them, making them feel first of all better about themselves, better about school, better about the future, and that’s what the whole day was about.

Assistant head teacher, Railside High School

Staff and students in the action groups were also supportive of the use of a large-scale survey. For example, they knew the data were representative of the wide ‘range’ of year 8 students at their school, and this was a ‘useful’ source of motivation to address ‘aggression levels’:

It was quite useful to us because now like, like the aggression level was quite high so now we’re trying to think of ideas to like to get it back down.

Action group student, year 8 female, Railside High School

Our facilitators all reported that the needs assessment process at the start of the intervention was highly appropriate, ‘powerful’ and instructive for identifying priorities:

I think it was useful, I think it’s always good to see data [ . . . ] and I think without the data they probably would have just completely dismissed [some issues] so at least we did have something there, to say, ‘Well they said this you know’!

Dawn, action group facilitator

Although schools were keen for an ‘external push’ (a key source of acceptability, as reported above), some SMT staff did report that the needs assessment felt too ‘negative’ at times, especially among more established SMT members who had been in their post for several years and might have seen it as a reflection on their many years of work and leadership at the school. Intervention facilitators also picked up on this and felt that schools may ‘just see the negatives’ if data were not presented appropriately and may become ‘too defensive’, although this ‘defensiveness’ also prompted them to mobilise and analyse other data sources.

The educational consultant who co-ordinated the intervention across all four schools and facilitated the action group at Goldstone Park highlighted another potential problem of the needs assessment report relating to how schools (and their data) are now scrutinised by Ofsted inspectors. She felt that not only were some staff not always ‘happy’ with the issues reported, but also this survey could reflect badly on the institution during a future Ofsted inspection, which was an additional reason why school staff would be ‘uncomfortable’ with the needs assessment survey if it was too ‘negative’:

They [Ofsted] expect to see the school surveys [so] if anything comes out that says their kids feel unsafe, they’re very worried about it, not only because they don’t want the kids to feel unsafe but if Ofsted were to come in, they could rate them ‘Inadequate’, so they were very uncomfortable about that, that kind of thing.

Vanessa, intervention manager

The process of assessing local problems and needs appeared to be more acceptable when it was not only informed by the student survey but also involved drawing on other sources of routinely collected data and existing documents (e.g. attendance and exclusion data, parent surveys, auditing incident reports, etc.) as well as other ‘softer’ sources of information, such as the views of students in the action group and other ‘student voice’ groups. For example, at Williamson High School the external facilitator reported that the action group audited and analysed its own, routinely collected, statistics on behaviour and exclusions. Other schools established new ‘forums’ and/or carried out ‘focus groups’ to gain more in-depth views from students following the needs assessment survey report. The greater use of data and the establishment of a new action group also inspired further large-scale surveys at Goldstone Park, which the head teacher commissioned to monitor and celebrate the ‘positive’ changes, as well as to continue to identify ongoing challenges:

The survey [at the end of the year] that we did with students and parents, came out very positively [ . . . ] We did it after Easter, so April, May time. We did the whole school. And we had a huge response from parents. So I think some of the senior leadership team were a bit shocked . . . but we had a third of parents respond [ . . . ] Broadly, the parents feel that the school does care, and that students achieve well in the school, and teachers want them to do well.

Head teacher, Goldstone Park

The only note of caution regarding the use of other sources of data came from the head teacher at Williamson High School. She explained that English secondary schools were increasingly ‘data-rich’ environments and triangulating multiple ‘sources of information’ could be problematic.

In addition, although the presentation of the needs assessment data was generally well received, various ways to improve this were suggested. First, more careful and judicious use of ‘benchmarking’ against the average across intervention schools is required when presenting data to schools, particularly in terms of students’ aggressive behaviour and what is ‘not good’:

[Benchmarking each schools data against the ‘average’ across the four schools] kind of confused the issue sometimes. [For example, if] sixty percent of year 8 students felt they were being bullied and at school X and you’re below the average, that’s still not good! [ . . . ] I know why the project used that comparison data because if I’d been a head I’d have asked that question, ‘Well what about the other schools?’ – but that can lead you up some blind alleys.

Bryn, action group facilitator

The data must also be reported clearly and presented in a way that is also accessible to students. Some reported that the data would have made more sense to them if they had been presented differently. For example:

It is and it isn’t [useful], because, like the percentages. You don’t really pay attention, ‘cos normally they’re just like in a little space in the page or somewhere and you won’t really see them. Like I remember, I got the thing back with the class average [school mean] or whatever it was and we didn’t notice them until the teacher had pointed out them.

Action group student, year 8 female, Goldstone Park

Action groups: including students and staff as agents of change

An action group was established at each school and met at least six times during the school year. Each action group included at least six student representatives. Members of schools’ SMTs acknowledged that this was an important innovation for their school. For example, one head teacher explained:

The action group has been really important. It’s the first time in the school, as far as I understand, that you’ve had students working with staff. And properly working with staff.

Head teacher, Goldstone Park

School managers who were responsible for implementing the intervention described recruiting a diverse range of action group members and how ‘positive’ this process had been:

I think in terms of people coming onto the [action group] team, initially it was very positive and we had, in fact, we had too many people probably. I got as diverse a range as possible; there were quite a few students who were keen and involved and they came to the half-day training as well. [ . . . ] I wouldn’t say [it was just] goody-goody high-attainers.

Assistant head teacher, Railside High School

The recruitment and retention of a range of students was seen as central to the success of the action group. Increased student participation in decision-making was central to our logic model, as it was hypothesised to improve staff–student relationships and students’ commitment their school, resulting in reduced aggression and bullying. Three out of the four schools chose to recruit year 8 students only, thus allowing them to build on the year 8 needs assessment survey, although limiting representation from across the wider student community. As with the needs assessment process (described above), several staff members and the intervention team felt that broader representation from students across years 7–9 would be appropriate and this could then be ‘branded’ as a key stage 3 intervention to support its sustainability. An action group member explained why she felt it was particularly important to include the younger students:

Year 7s would have a different perspective to it: they’ve just come in, they’re the ones who, who face all those things. And then they’re the ones who will tell you ‘when I . . .’, the period of transition, ‘when I was in year 6 that’s what happened, and now I’m in year 7 this is . . . these are my difficulties’. Bullying, all those things, they happen to be affected more because they’re the ones who enter into the system

Action group member, English teacher, Railside High School

Students in the action group at Williamson High School, in which some older students were included from year 10 and the sixth form, felt that involving older students was helpful for a ‘third perspective’ alongside that of younger (key stage 3) students and staff:

They’ve had more of an experience at the school already, they’re sunk in to the school, and to hear our views about how we feel because we’ve, we’re coming up to two years now and so, yeah, it’s good to get different views from different years.

Action group student, year 8 male, Williamson High School

It appeared that using multiple methods of recruitment is acceptable to students and may be the most appropriate method for recruiting a diverse range of students. Students reported that they either ‘got chosen’, ‘got asked’ and agreed or volunteered after finding out about the group in an assembly, newsletter or via word of mouth from a friend. One school used a drama workshop on bullying to publicise the new project, which was popular with students and could be used to launch the project elsewhere in the future. For instance, Goldstone Park pro-actively ‘asked’ and encouraged some students who may not have volunteered to join the group to ensure diversity in terms of more and less engaged year 8 students. A pastoral support manager at the school explained why she thought this was appropriate:

It was announced obviously in the assemblies and things like that but we do approach students sometimes and say, ‘look you’d be really good on that’ because I think especially with your more colourful students, they don’t apply, thinking, ‘well, I’ll never be allowed’. [Then you’ve] just got to sort of have a chat with them and say, ‘you’d be really good at that, why don’t you . . . ?’ – [they say] ‘we’re not doing that’ –’no, go and apply!’. And they do. And as you can see at those meetings, thoroughly enjoying it [ . . . ] And you do need them involved, because if they’re on-board they’re going to make sure that everybody else is involved, do you know what I mean? All the other students . . . because they are known to have some issues and problems from time-to-time, then students – I don’t know – respect them more, listen to them more. If they think it’s a good idea then it can’t be all that bad!

Action group member, pastoral support manager, Goldstone Park

Students in the action groups also reported that it was appropriate for some students to be asked to participate, to ensure diversity, but still allowed to volunteer rather than being coerced:

[The deputy head] he asked me [but] people, like, volunteered to do it and they weren’t just [made to . . .] random people weren’t just told to go! They wanted to be there. So, they contribute instead of just sitting there ‘cos they don’t want to be there.

Action group student, year 8 male, Goldstone Park

Students and staff also suggested that the requirement for the action group to include management, teaching, support and administrative staff ensured that the group represented ‘different perspectives’ and could ‘paint a picture’ of the school to inform the development of an appropriate action plan:

There were governors. There were all different [staff]. There was parents, and there was some non-teaching staff as well, like the ones who do not teach in classes were there – one of them is a lab assistant [ . . . ] so these categories, they make up the school staff if you like [they are] going to have a different perspective . . . It’s going to be a different picture that we can paint.

Action group member, English teacher, Railside High School

The presence of senior management staff was also seen as particularly critical. The intervention team felt a key learning point was that a senior member of the SMT (i.e. head or deputy head) with ‘power’ to change wider policies was likely to be essential to the composition of an effective school action group. This was not the case at Williamson High School, where a relatively new ‘assistant head’ represented the SMT and changes to the school policies and ethos were much harder to achieve. The educational consultant who facilitated the action group at Williamson High School explained the consequences of this for the group, comparing it to another school (Whitehorse Road) where significant school policy changes were achieved:

My reservations are to do with the composition of the group. [At Williamson] they had an assistant head teacher who was part of the senior management team but what I think is important [is that] you have somebody on-board who has responsibility for behaviour policy in there [ . . . ] you really need to have a person who can lead as a power and [they] need to be there and particularly when you’re trying to change the ethos because there was a certain tension between the action group and the deputy head [who was not on the action group] who had responsibility for student behaviour [ . . . ] Whereas at Whitehorse Road they came up with a school policy which incorporated a behaviour policy which incorporated the principles of restorative justice. I spent some time with the head there working on that and they did that well.

Bryn, external facilitator

The weakness of existing student voice groups (e.g. school councils), which were seen as highly unrepresentative and powerless, also provided a strong source of acceptability for the action group and motivation for joining it, both for students and for staff keen to promote student voice. At all four schools, the action group was seen as ‘new’ and ‘different’ from the school council:

Most people don’t feedback from the school council, and, to be honest there’s not really – well I don’t see the point in having the school council.

Action group student, year 8 female, Goldstone Park

Like they say, student council, like ‘oh yeah we do it every week, we’ve done a few things’. And it’s like ‘what have you done?’

Action group student, year 8 male, Railside High School

The topics are good on the action group [compared to the school council . . . ] We actually talk about things that we do here, what’s good for the future, stuff like that, like I said in the first place about the CCTV, the cameras and also the teachers is going to be improved and, you know they listen.

Action group student, year 8 male, Whitehorse Road

The schools used the new action group to promote student involvement in decision-making and leadership via a new, more student-led and student-centred environment (Box 3 shows a case study of the achievements at one school). This included encouraging students to chair meetings and allowing students to talk first at the start of meetings:

One school [Williamson High School], they always had a student who chaired the meeting, and [Whitehorse Road] they took the first part of the meeting so it was just students who talked and that gave them time, rather than them trying to make a contribution in a perhaps more adult kind of scenario, both those methods worked very, very well.

Bryn, external facilitator

Senior managers, including head teachers, also recognised, and were motivated by, the need to increase student voice and the limited representation of students in existing groups. The action groups were seen as extremely useful in addressing this deficit in students’ ‘perspectives’:

It was very much a ‘done-to’ climate the ethos in the school, and staff thinking they knew best. And some of those staff, and some of my senior team who don’t have their own children as well, and I’ve talked a lot to my deputy about this, and those of us who have our own children, do tend to see it a bit more from the children’s perspective . . . .

Head teacher, Goldstone Park

These data support the view that the emphasis within the intervention on student voice was highly acceptable and potentially a driver of improved staff–student relationships and increased student engagement. At Williamson High School, the students in the action group also attended the enhanced 3-day restorative practices training, which appeared to be an appropriate innovation for supporting students’ ‘understanding’ and engagement with the concept of whole-school restorative change.

Several students reported that they felt that they had gained ‘good experience’ by participating in the action group and other related activities, such as the training. Students and staff also consistently reported that they felt that they had got ‘credit’ for their involvement. However, both staff and intervention \team members suggested that involvement and retention may be further improved by more formal accreditation and ‘additional rewards’ for students. The intervention manager also suggested recognising staff involvement through better integration of the intervention with continuing professional development initiatives.

Finally, it is also important to recognise the practical barriers some schools face when organising meetings after school for students to attend, especially at schools in London and/or with large catchment areas, where students have long journeys home via public transport.

Some students suggested lunchtime meetings may have been better at Whitehorse Road, but, at the other schools, students felt that lunchtime was ‘the only time to socialise’, which further supports an adaptable approach to organising these meetings based on schools’ local context and students’ views. Providing tea, coffee, juice and biscuits also appeared to be popular with students and some staff, encouraging them to stay after school and also facilitating a ‘grown-up’ environment.

External facilitation: supporting school managers, advocating for students

Three external facilitators (one was also the intervention co-ordinator) worked across the four schools to support each action group and help co-ordinate the intervention locally, each working with his or her own school or schools. School SMT members’ accounts of implementing the intervention included strong support for their action group’s external facilitator, who they felt supported feasibility. Facilitators’ experience and expertise was ‘handy’ to school managers who were co-ordinating the intervention locally, especially at more challenging schools and where it may be practically more difficult to organise new events quickly:

[Bryn] helped me with the agenda, and he takes part, particularly with the students, he asks a lot of questions to the students [. . .] there’s been some other things where Bryn’s expertise, if you like, his experience, has proved handy [For example] I make things fit to my timetable so it’s handy to have someone like Bryn because there were a couple of times when he was a bit more assertive and I thought, OK! I will do it. So it is handy.

Deputy head teacher, Whitehorse Road School

Those schools that were explicitly motivated by the need for an ‘external push’, such as Goldstone Park, also strongly welcomed the external facilitator, and explained that the role ‘fits’ with their motivation for participating in such an intervention and their other priorities:

It just fits with what we need at this moment in time, and what we’re doing and this is a huge strategic direction push for us as a school, so it blends in very nicely and helps us with what we’re already trying to achieve

Head teacher, Goldstone Park

One of the facilitators explained that she felt that this external role was particularly valuable, as facilitators could ‘challenge’ schools and provide fresh, ‘creative’ input outside the local/internal ‘politics’ of the school:

I think really good listening skills are important and not being afraid to challenge, being objective and not getting sort of sucked into politics, being trustworthy so they feel that you can, they can really say what they need to say in front of you so building that rapport and the trust. And I think maybe a bit of creative thinking because I think you have to help them think of ideas and ways they could work which are different to the ways they work now.

Dawn, action group facilitator

As well as providing an ‘external push’, the facilitators’ role was also seen as flexible enough to allow them to play multiple other roles and to adapt to different school contexts. For example, one facilitator explained how she could simultaneously act as an ‘objective’ source of support and help ‘anchor’ the action group to keep it focused.

The external facilitators could also support students in ‘having a say’ and ‘having respect’ as part of the process of implementing a new whole-school restorative approach at their school. The facilitators recognised the importance of their advocacy role to support student voice, especially when teachers were defensive. However, it was also suggested that change was most likely to occur and be student-led when teachers and other school staff, as well as the students, were all able to ‘have a say’ during action group meetings.

Two of the three facilitators had been school managers (one a head, one a deputy head) at secondary schools and the other was an educational consultant who worked for a number of education authorities. She felt that because of her knowledge of secondary schools from advising local education authorities, her ‘skills’ were important in terms of facilitation, advocacy and ‘helping groups work’. However, some SMT members suggested that an external facilitator should ideally have school leadership experience.

Finally, all those involved with delivering the intervention reported that facilitators’ implementation needed better co-ordination across schools, with some ‘collective meetings’ to share learning and example of ‘best practice’ in terms of facilitation and advocacy. The intervention manager described her vision for what this might look like and how greater use of technology might also make the intervention more sustainable:

I think we need a team of facilitators that are kind of coordinated [ . . . ] I think it would be good as well capacity-wise to be able to build in the cost of them meeting occasionally together and sharing ideas and things, because that didn’t happen very much. The other thing [would be] if you were to have a virtual learning environment that was shared between all the schools that were doing it, where they could share resources – I think that would really, the schools would really benefit from that too I think. That would be a good way to support the project, but that wouldn’t have to be people-rich!

Vanessa, intervention manager

Restorative training: room for improvement

It was widely agreed by both the school staff and the intervention team that high-quality training in restorative approaches was a crucial ‘input’ if they were to adopt this philosophy and operationalise it across the whole school. The quality of the restorative practices implemented in schools would therefore strongly be determined by the quality of training. Staff and the training provider agreed it was appropriate to undertake an initial half-day introduction for all staff on restorative principles, language and classroom methods (e.g. checking in, checking out and circle time), followed by 3-day, enhanced training for a smaller group on implementing practices such as restorative conferences. The trainer felt that the enhanced training worked best with a ‘mix’ of staff:

There’s certain key roles to be identified but there’s some of it to be opened up for these very naturally restorative people that are very passionate about it [ . . . ] because they are worth their weight in gold. So you do need a mix . . .

Brenda, restorative justice trainer

As well as the problems in scheduling the training reported above, there were also problems reported regarding its quality. School staff had divergent views on the content and delivery style of the training. Several SMT members reported their frustration that, although training was a critically important component and one that demanded considerable staff time and resources, it was also the weakest intervention component. A key theme that emerged from participants’ accounts was that the training must be improved by making it more interactive and engaging for staff. The half-day whole-school staff session was viewed as a ‘disappointment’ because it was ‘too didactic’:

I found the training very disappointing, I think the first morning was very disappointing. It was the whole staff, it was everybody, and I just, I felt it was something that could have really fired people, and I know that there were some positives that came up, I know that some staff went away and started using checking in and checking out etc., but I think the training was too didactic, it wasn’t relevant enough for a high school situation. [ . . . ] It was a large group, but we could have been divided up into groups and that only happened once in the 3 hours.

Assistant head teacher, Railside High School

Students in the action groups who attended the initial half-day training at their school to find out more about restorative approaches also reported that it needed to be more interactive. For example, students at Railside reported that it was ‘boring’ and the trainer should ‘have made it more active, we was like sitting there for four hours’. Students at Goldstone Park also complained that it was just ‘PowerPoint and someone talking’ and ‘all you got was a cup of tea’.

The enhanced 3-day training in conferencing methods also needed to be more engaging and resonate with secondary school teachers’ everyday experiences. Some of the school staff suggested how the methods could be improved to make it more engaging:

It really was death by PowerPoint. And at the break time, which was at 11 o’clock, which was [after 2 hours] you know 9 o’clock to 11, we told her. And by 20 past 11, she had made it practical and interactive and worked out ways the theory could be covered on the tables rather than everybody sitting . . . and you know, fair play to her, it was the first time she’d ever worked with students and adults in a group, you know, she did really well.

Assistant head teacher, Williamson High School

As well as avoiding large-group, didactic methods, the assistant head teachers at Railside also suggested that the training would be more acceptable if it was delivered by:

Somebody who has established this in their school and can talk about nitty gritty things, like the systems, the processes, how they operate the structures on a day to day basis, then also talk about the impact it’s had on children, on staff.

Assistant head teacher, Railside High School

Several SMT members felt that the training may be better received if it were delivered by someone with a teaching background. The head teacher at Goldstone also suggested that another way to address the ‘credibility issue’, if training were delivered by someone who did not have a teaching background, was to ensure it was more realistic.

Related to this was another recurring theme that the training lacked tangible examples focused on challenging secondary school environments:

[Although] the trainer does have examples of secondary, she talked to them in quite a primary way, I think. I think as well, when I sold it to schools, [ . . . ] I mean, training time for teachers is so rare and so valuable to them, and even if you pay to cover their class, which we did, they don’t like it, they’d rather be in the class often. But I think they felt they didn’t quite know why, why it was kind of fitting in, really.

Vanessa, intervention manager

The trainer acknowledged that starting with a more comprehensive audit of existing practices would be beneficial for understanding different, challenging secondary school environments, for example by asking initially: ‘What is the school hoping to achieve?’. It was also suggested that a more detailed pre-training audit could help tailor the training to schools’ needs regarding how they would ‘cascade’ and ‘formalise’ new restorative approaches and methods through the whole school. Furthermore, one staff member who attended the training sessions at Goldstone Park highlighted that the restorative training did not identify sufficiently realistic examples:

The last session for instance, the afternoon was, erm, we were given scenarios for role play, and I have to say I’m not a great lover of role play, but we’ll do what we have to do. Erm, the scenarios, a lot of them, were too weak to stand up to role-play [ . . . ] it was more in primary level.

One observation was that training participants needed to be encouraged to engage in further learning and practice their skills between the (3-day) enhanced training sessions.

The curriculum component: support for the ‘tailored’ approach

All schools welcomed the flexible needs-led basis of this curriculum. This consisted of learning modules that schools could address using either the INCLUSIVE teaching materials or other existing materials. This allowed schools to tailor the curriculum to their needs. Schools implemented 7–12 hours of the curriculum, which is impressive bearing in mind the late initiation of the project. The intervention manager felt this represented a success also in light of the limited number of hours available for PSHE:

Citizenship has been cut right down, you know, completely cut down.

Vanessa, intervention manager

Whitehorse Road was already delivering the ‘Opening Minds’ social and emotional well-being curriculum in PSHE. Nonetheless, teachers found the INCLUSIVE social and emotional curriculum for year 8 students useful and addressed PSHE with a combination of ‘Opening Minds’ and INCLUSIVE teaching materials. The school delivered 10 hours of the INCLUSIVE curriculum and rated it very positively.

As ‘restorative practice’ was seen as a major gap in existing PSHE curriculum resources across all schools, and, because of the opportunity to choose modules based on schools’ needs assessment data, this curriculum was seen as a highly attractive opportunity to ‘refresh’ PSHE in line with the restorative principles underlying the intervention:

As somebody who has recently taken responsibility for PHSE in the school, I found getting a whole programme absolutely marvellous and particularly a programme that will add, with adaptation will work for us we’re up to our sixteenth hour [with the year 8 students], so we’ve managed to get through it, but that was a really difficult thing to do, because of the timing, but actually it worked out OK in the end [because] our year 8 [PSHE] programme needed a refresh, so it was a good thing to happen.

Assistant head teacher, Williamson High School

There was also evidence of the curriculum component’s popularity and sustainability at Williamson, where 8–12 hours were delivered to each year 8 class. The PSHE teacher leading this reported that they would ‘use the units again’:

They’ve had altogether about 8 hours each (year 8 class), maybe a little bit more, some of them would have had more because I have them every week so they’ve probably had about 12 lessons with me [ . . . ] We’re also going to use the units again because they were very well received as well.

PSHE teacher and action group member, Williamson High School

A discussed at length above, greater planning would facilitate implementation earlier in the academic year (largely through the opportunity for preparation in the previous year). Some staff also suggested further improving the curriculum resources in the future through more interactive methods, which would support both ‘activity and action’. For example:

We need more activity and action in it. A wider variety of learning styles. There needs to be more activities.

Assistant head teacher, Williamson High School

Trial recruitment

Head teachers consistently reported that the process of recruiting schools into the pilot trial via initial contact from the intervention manager (a former secondary school deputy head, now an educational consultant and Ofsted inspector), followed by a more detailed discussion of what the research involved with the trial manager, was appropriate. They felt that other methods of recruitment (e.g. via local authorities) were unlikely to be more effective and seemed less appropriate:

Local authorities are one route in, although it’s become more disparate because of the academies and free schools.

Head teacher, Goldstone Park

It was, however, suggested that ‘networks’ of head teachers could also support recruitment into a larger trial via ‘word-of-mouth’ communication. The head teacher at the Bell Street Academy also highlighted the potential of new partnerships between schools and universities to facilitate ‘research readiness’, although this may limit generalisability if it were the only source of recruitment for a trial:

Lots of schools I know have very close relationships with, with academic institutions and they do lots of in-school research. I actually think it’s an opportunity area.

Head teacher, Bell Street Academy

Although the recruitment process was acceptable to schools, it was ‘rushed’ during the pilot trial, reflecting the later-than-planned July start. These challenges were compounded by the purposive sampling criteria for the pilot, which required us to recruit very particular combinations of school. The intervention manager, who recruited schools, suggested a larger team of education consultants could recruit schools for a full trial over a longer time period:

You needed a team of consultants who’ve got the contacts in the areas that you want. You see, I mean, on Ofsted I work with quite a lot of people who do a lot of coaching and other consultancy in schools, and so they, if you had a team of say five people and a year to do it, you’d do it without any problem . . .

Vanessa, intervention manager

Recruitment of schools further in advance and over a longer time period would also provide more time for presentations to staff from the intervention and research teams to get them ‘on board’. However, the central message was that it was acceptable and feasible for the intervention team to recruit schools.

Randomisation and experiences of being in a trial

Schools’ motivation for participation was usually for them to build on existing work and to ‘add value’ to their core business. As a result, schools that were assigned to the comparison group were disappointed:

We were disappointed. We were disappointed because we did make an investment although as I said I didn’t overdo it in case we didn’t get, uh, selected. We were also disappointed because we thought it would give, erm, us a real boost and be something that was of genuine value to us. We wouldn’t just be engaging as a favour or as an experiment [ . . . ] I’m disappointed, our Assistant Principal, was disappointed. Not aggrieved, and certainly not to the extent where we felt we’d been let down or anything like that.

Head teacher, Bell Street Academy

Despite disappointment, the process of informing schools that they would not be in the intervention group and what this meant for them in term of the rest of study was deemed to be acceptable to SMTs at comparison schools. For example:

I was quite happy with the information that I got. And I felt all along that if I had any questions it was made very clear to me that, you know, just ask and we’ll tell you what the information is. Everything I asked, I got satisfactory answers. So there was nothing that you did that was wrong in any way. I was quite happy with the information that I got.

Deputy head teacher, Woodhouse School

There was also evidence that there is support for cluster RCTs from the current generation of head teachers who are more ‘passionate’ about ‘evidence’, which also made it acceptable for them to be in the comparison group:

I’ve no problem with [being in the comparison group] because it’s the only sound way of getting information which becomes professional, rather than personal [ . . . ] I’m firmly of an opinion that all schools should be run on evidence-based ideas, rather than fly by your pants kind of approaches! [ . . . ] I’m passionate about evidence-based work in schools [ . . . ] we’re an all-graduate profession and we should be based on evidence-based work.

Head teacher, Jamestown School

None of the eight schools dropped out of the study and follow-up survey data were collected as planned at each school. One comparison school (Milton Down) did not comply with all elements of process evaluation data collection as per protocol, as they were experiencing a period of transition following the announcement of a merger with another school during the 2011–12 academic year.

The money provided to comparison schools retrospectively at the end of the study to cover any administrative and teaching costs of participating (£500) appeared to be an incentive for Milton Down School to stay in the study and complete the follow-up student survey. The money appeared to be less relevant for motivating the other comparison schools:

Five hundred pounds is nothing. It’s a drop in the ocean. You know, what you will get in a school for five hundred pounds is negligible really [ . . . ] people weren’t doing it to get 500 pounds. They were doing it to get the investment in the programme [ . . . ] We didn’t say ‘oh, let’s do this ‘cos if we’re not selected we get 500 quid’. We said, ‘oh, let’s do this because hopefully we’ll be . . . in the right half’.

Deputy head teacher, Jamestown School

I think, and then you did the, um, administration of it, so I would suggest that 500 pounds is quite generous considering what it, it didn’t . . . What we had to do.

Assistant head, Bell Street Academy

Overall, however, it appeared that it would be worthwhile to retain this in order to encourage all comparison schools to remain in the study at follow-up.

Survey methods

Student participation was exceptionally high in both baseline and follow-up surveys, suggesting high overall acceptability among students. However, some students reported concerns regarding the confidentiality of survey data when completing their questionnaires in school:

You know when we wrote our name and signed it, they could like count through and, count through the papers . . .

Male student, Year 8, Williamson High School

This suggests the need to ensure that information is provided to students in more accessible language about how anonymity is maintained within the trial. Some students thought that paper questionnaires ‘waste trees’/’waste paper’ and that an online survey would be more ‘green’. However, it appears that not all schools would be able to organise students to complete web-based surveys during the same period, which is likely to be a barrier to using them as a replicable data collection method for a Phase III trial.

The most consistent theme among school staff and the intervention team was that we need to change the timing of the baseline surveys:

It’s all about that build-up. I mean, bearing in mind that heads have no time, if you were to build that up and start getting heads together and explaining what you would be asking for, or other staff involved before that time, then the seeds would be sown much more about, you now, the input. It’s just laying that, and also you know what it’s like, it’s the beginning of [autumn] term [in September].

Vanessa, intervention manager

Undertaking surveys in the summer would also have the added advantage that comparisons with follow-up surveys would be less likely to be affected by seasonality in reporting.

Teacher surveys were also piloted during this study. Although organising teacher surveys was a challenge, these occurred at baseline and follow-up as planned at seven out of the eight schools. The time constraints on these surveys were a recurring theme, and there was a widespread agreement that whole-school teacher surveys were most feasible when all staff were together at a regular staff meeting or briefing:

From the [co-ordinating] teacher’s point of view I would like to have a time when they’re all present and all hand it in at the same time.

Deputy head, Woodhouse School

The only other times that all teaching staff get together, outside staff meetings and briefings, are the ‘inset’ training days during which there is limited time for external, non-essential activities:

We’ve only had three ‘inset’ days this year, and the time is really, really kind of fought over.

Deputy head, Woodhouse School

A SMT member at one of the comparison schools (Jamestown School) reported that staff were concerned about the confidentiality of the surveys:

Sometimes some people are suspicious about surveys and data, and think ‘what are they looking for? What should I say? What would look and sound right for me and the school, and the children?

Head teacher, Jamestown School

Finally, an important finding was that all schools would have preferred non-teaching as well as teaching staff to have been surveyed, particularly at schools trying to change perceptions and ‘break down barriers’:

There was division in the school and there often is in many schools between support staff, non-teaching staff, teaching staff and traditionally some people having the views that non-teaching staff shouldn’t be doing certain things [ . . . ] We’re trying to break down all those barriers, we’re trying to say we’re one team, everybody is supporting improvement in behaviour, whether they’re the office staff or not students should respond in the same way to all adults [ . . . ] of course in our staff meeting [when the survey was undertaken] we have non-teaching staff, and we were only giving out the surveys to the teaching staff. And that just re-enforced what we didn’t want to re-enforce.

Head teacher, Goldstone School

Completion rates and prevalence

Completion rates were calculated for each outcome measure, overall and stratified by sex. Low completion rates can indicate that an item is not generally well understood, or that respondents did not feel comfortable answering it. Table 8 shows the prevalence of missing data on items needed to calculate the GBS, AAYP and ESYTC school misbehaviour subscale scores overall and by sex.

Table 8 shows that the number of missing data on items needed to calculate the overall scores of the three pilot primary outcome measures (GBS, AAYP and EYSTC school misbehaviour subscale) ranged from 5.8% to 8.3% at baseline. Non-completion rates for these measures were therefore low at baseline, although non-completion rates were slightly higher for male students (8.0–10.1%) than for females (3.2–6.4%). Missing data were fewest for the GBS total score at baseline: 94.2% of students who completed the baseline questionnaire provided usable GBS score data in full. The non-completion rate remained relatively low for all the pilot indicative primary outcome measures at follow-up (5.5–9.2%), decreasing somewhat for the AAYP measure (see Table 8). Girls were, once again, more likely to respond to all items on these measures than boys: at follow-up the proportion of male students who did not complete all items needed to calculate the ESYTC school misbehaviour subscale was nearly 12%. The non-completion rate for the total GBS score at follow-up was 10.3% for males compared with 6.7% for females.

Table 9 shows the responses for each item, including missing data, according to sex at baseline and follow-up. Overall, items on the GBS were consistently answered, although at follow-up 7.7% (n = 45) and 8.2% (n = 48) of male students did not complete the third and fourth items, respectively (‘Have you been deliberately left out of things at this school in the last 3 months?’ and ‘Have you been threatened physically or actually hurt by another student recently at this school?’). The percentage of students who reported that they had been a victim of teasing, in the past 3 months, was similar across males and females, and across time (41.8–46.3%). About one-third of respondents stated that other students had spread rumours about them and one-fifth had been deliberately left out of things at school. Approximately 15% of girls at both time points reported having been physically threatened or hurt by another student in the last 3 months; this figure was slightly higher for boys (21.4% at baseline and 18.2% at follow-up).

The highest rate of missing data for the AAYP measure was for the third item (‘Have you ever threatened to cut, stab or shoot anyone?’), which 43 males did not answer (7.1%) at baseline. Analysis of students’ responses to items on the AAYP measure found that, overall, students were more likely to report aggressive behaviour at follow-up (see Table 9). For example, at baseline, 9.1% of males and 4.3% of females reported that they had threatened to beat someone up at least once in the last month, and this figure was slightly higher at follow-up (11.1% and 6.7%, respectively). However, the rates of ‘threatening to cut, stab or shoot someone’ and having ‘cut or stabbed someone’ in the last 3 months were almost zero for both males and females at both time points (range 0.4–1.6%). Exploratory analyses of the ESYTC school misbehaviour subscale found the non-completion rate was low for all items (< 7%) at both baseline and follow-up. The lowest rates of missing data across all three measures were for the sixth and seventh items of the ESYTC school misbehaviour subscale, with only seven females (1.4%) not answering at follow-up. ESYTC items relating to aggression provide stable but relatively low estimates of aggressive behaviour over time in this population.

Discrimination

Table 10 provides the mean scores and standard deviations for all the pilot primary outcome measures at baseline. The lowest possible value for all three scales is within one standard deviation of the mean, suggesting all scales may suffer from ‘floor effects’ (i.e. the scales may not be sensitive to low levels of reported bullying). This is particularly true for the AAYP score and the ESYTC school misbehaviour subscale score, both of which had standard deviations that were substantially higher than their means.

The distribution of responses was explored, and this also suggested the possibility of floor effects: responses were not normally distributed; and, for the AAYP measure, the vast majority of respondents in this study have a score of zero. For GBS and ESYTC, zero is also the most common value, but the distribution of responses is more even than that of the AAYP measure. In regards to discrimination, GBS performs best, followed by ESYTC, then AAYP.

Reliability

Reliability was assessed by examining the ICC for each score (as an indication of the stability of the measure over time) and by examining Cronbach’s alpha statistics at baseline and follow-up (indicating each scale’s internal consistency).

Intraclass correlations give an indication of the proportion of the variance explained within the participant responses (across time) relative to between-subject variance. They are used as an indication of the stability in a measure over time. None of the variables performed particularly well in this area, with surprisingly high ICCs for all three (Table 11). However, GBS had a substantially higher ICC than AAYP and ESYTC, suggesting that it is more stable across time.

We calculated Cronbach’s alpha statistics to provide an indication of each scale’s internal consistency, a measure of the extent to which items within the scale are measuring the same latent construct. Alpha values of 0.6–0.7 are considered acceptable and values of 0.7–0.9 ideal. 94 In this regard, at both reporting periods, GBS and ESYTC performed well, but the AAYP did not (see Table 11). This may be because of items on the AAYP scale that do not fit the typical British conceptualisation of youth violence, and these may have low correlations with other items. The results suggest that, as for the consistency of the items, GBS and ESYTC appear substantially preferable to AAYP. However, in considering these internal consistency coefficients (Cronbach’s alpha statistics), we should be cautious. First, a number of items had very low prevalence rates, and scales containing a large proportion of such items will have a lower level of internal consistency. Alpha coefficients are also dependent on the number of items on the subscale, with shorter scales yielding lower coefficients.

Pilot indicative primary outcomes

Table 9 shows that the prevalence of being bullied and aggressive behaviour towards others was relatively high, with relatively few differences according to sex. At baseline, > 40% of students reported being teased in the last 3 months, and 26.4% of males and 22.5% of females reported being teased once a week or more frequently. Moreover, at baseline, 26.6% of males and 29.4% of females had had rumours spread about them in the last 3 months, and 17.8% of males and 22.8% of females reported being deliberately excluded by others in the last 3 months. The rate of reporting being threatened or physically hurt in the last 3 months at baseline was 21.4% for males and 14.4% for females.

The AAYP scale found that, at baseline, 9.1% of year 8 males reported having been aggressive towards others (not including their siblings) during the past 3 months. This figure was lower for year 8 females (4.3%). Very few year 8 students reported that they had threatened to cut, stab or shoot someone in the last 3 months (< 2%) or had actually carried out these aggressive acts (< 1%). Response rates were higher for low-level provoking and school misbehaviour as measured via the ESYTC, such as using bad language, which was reported by > 15% of male students and 11.6% of female students at baseline. The majority of students reported that they had not threatened, hit or kicked a school teacher (see Table 9).

Table 12 provides the mean scores and standard deviations for all the pilot primary outcome measures at baseline for the intervention and comparison groups. There was a consistent trend across all three pilot primary outcome measures for the intervention group to report more prior experience of bullying and higher levels of aggression and school misbehaviour at baseline (see Table 12). On average, mean GBS overall scores varied from 0.91 for students in the comparison group to 1.04 for students in the intervention. With regards to aggression and school misbehaviour, the mean AAYP overall score and the mean ESYTC overall score were also both greater for intervention students. These imbalances are not unusual in a pilot cluster trial that includes a small number of randomised units (n = 8).

Table 13 reports the primary outcome score by arm for male and female students at baseline. This shows that rates of prior experience of bullying and levels of aggression and school misbehaviour were similar for males in both arms at baseline: the mean GBS overall score for males in the comparison group was 0.92 and for males in the intervention group was 0.91; for the AAYP the mean was 0.87 in the comparison group and 0.92 in the intervention group; and for the ESYTC school misbehaviour subscale the mean was 3.10 in the comparison group and 3.11 in the intervention group.

Although male students were fairly evenly matched in terms of these pilot primary outcomes, there were clear differences between females in the comparison and intervention groups in terms of all three measures. Mean rates of victimisation in the past 3 months measured by the GBS were much higher in the intervention group (1.17) than in the comparison group (0.91). Similarly, the mean overall AAYP and ESYTC school misbehaviour scores were higher for females in the intervention group (see Table 13).

The study did not aim to detect intervention effects, and lacked both the statistical power and intervention duration to be able to do so. Table 14 presents the pilot primary outcomes for the intervention and comparison conditions, and overall, at follow-up. AAYP and ESYTC scores increased between baseline and follow-up in both trial arms, as is expected normatively in year 8 students.

Response rates and distribution of results

Missing data from the CHU-9D are problematic, as the utility algorithm does not currently recommend they are replaced with suitable estimates. Of 1144 questionnaires issued at baseline and returned, 252 (22%) contained at least one missing CHU-9D response, meaning that overall utility scores could be calculated only for 892 respondents. However, at baseline it was the very final item on the questionnaire, and the proportion of incomplete questionnaires decreased to 16% in the follow-up survey, in which it was moved forward to precede other pilot secondary outcome measures. There did not appear to be any discernible indicators as to which of the nine questions was more likely to be left incomplete. This further suggests that time and questionnaire length were the only barriers to completion, and these can be addressed.

Analysis of the baseline data indicated that > 64% of year 8 students indicated ‘no problems’ on the worry, sad, pain, annoyed, work, sleep and routine CHU-9D subscales (Table 15), indicating potential ceiling effects on these domains. This is unsurprising because the young people in the pilot were generally considered to represent a ‘healthy’ school-going population. However, only 31% of respondents who completed the question indicated that they were not at all tired (31%), and almost 10% indicated that they were ‘extremely’ tired. The potential for floor effects on each of the domains was not shown to be an issue, as few individuals reported being in any of the ‘worst’ health states. Only 13% (112/892) of participants indicated that they currently had ‘no problems’ on all nine domains at baseline, and only 66 individuals (7%) reported having no problems on all domains at both baseline and follow-up.

A histogram of the baseline survey CHU-9D utility score showed that the distribution of results was skewed (Figure 6). Whereas the mean baseline CHU9 utility score was 0.88 (95% CI 0.88 to 0.89), the median was 0.90 (interquartile range 0.83 to 0.95). Females reported a lower median utility score than males (p = 0.0001). The lowest utility score recorded by a single individual was 0.33 (recorded in the follow-up study), which is the lowest possible health state. These data do not therefore suggest potential ‘floor’ or ‘ceiling’ effects.

FIGURE 6.

Distribution of CHU-9D utility scores.

Construct validity

The results of tests for discriminant validity indicated a strong positive correlation (0.57; p < 0.0001) between the CHU-9D utility and PedsQL total score, suggesting that the two scales are measuring similar items (Table 16). The correlations between CHU-9D utility and PedsQL total and primary trial outcome measures were all negative and in the expected direction: as CHU-9D utility/PedsQL scores increased (indicating better health), the outcome measure scores decreased. However, although the correlation coefficients were statistically significant (p < 0.0001 in all instances), their sizes can be considered moderate at best.

In tests of convergent validity, most of the hypothesised relationships were statistically significant. However, the level of convergent validity as assessed using the PedsQL subscale scores was arguably only ‘moderate’: the highest was –0.38 (Table 17).

Correlating changes in the CHU-9D utility scores and changes in the three primary outcome measures showed that the relationships were statistically significant (p ≤ 0.02 in all instances) (Table 18). However, again, the degree to which the scores were related was subjectively weak; the largest correlation was –0.11. The changes in the PedsQL and CHU-9D utility score were much more closely related. Although ideally there would be a level of correlation between all outcome measures, by definition they have been designed to measure different outcomes. For example, the AAYP asks questions regarding the frequency of aggressive behaviour rather than questions about ‘health’ per se. Moreover, the period of recall is over the last 3 months, whereas the CHU-9D specifically refers to ‘today’. Similarly, the GBS asks specific questions regarding experiences associated with bullying in terms of frequency and the degree of upset they caused, also over the last 3 months.