Increasing boys' and girls' intention to avoid teenage pregnancy: a cluster randomised control feasibility trial of an interactive video drama based intervention in post-primary schools in Northern Ireland

Primary aim

The aim of the study was to determine the value and feasibility of conducting an effectiveness trial of the If I Were Jack Relationship and Sexuality Education (RSE) intervention in post-primary schools.

Secondary objectives

The research aimed to achieve the following secondary objectives:

• assess the acceptability of the intervention to schools (school principals and RSE teachers), male and female pupils and parents

• identify optimal delivery structures and systems for the delivery of the resource in the classroom

• establish participation rates and reach including equality of engagement across schools of different socioeconomic and religious types

• assess trial recruitment and retention rates

• assess variation in normal RSE practice across the participating schools

• refine survey instruments as a result of cognitive interviews with male and female pupils

• assess differences in outcomes for male and female pupils

• identify potential effect sizes that might be detected in an effectiveness trial and estimate appropriate sample size for that trial

• identify the costs of delivering If I Were Jack and pilot methods for assessing cost-effectiveness in a future trial.

Stakeholder group

Ms Joanna Brown, RSE and Local Area Coordinator, Sexual Health Training Team, Belfast Health and Social Care Trust (BHSCT), Northern Ireland (NI), UK.

Dr Bernadette Cullen, Consultant in Public Health, Public Health Agency (PHA) of NI.

Ms Kathryn Gilbert, Curriculum and Assessment Programme Manager, Council for the Curriculum, Examinations and Assessment (CCEA), NI.

Dr Naresh Chada, Senior Medical Officer, Department of Health, Social Services and Public Safety (DHSSPS), NI.

Ms Janet Moore, DHSSPS.

International Advisory Group

Trial Steering Group meetings

The Trial Steering Group met three times, on 9 October 2014, 11 May 2015 and 24 November 2015. For Trial Steering Group minutes see Appendix 4.

Evidence supporting a theory-based approach

Providing a theoretically informed foundation for sexual health education programmes is considered key to effectiveness because it ensures that the most important determinants of young people’s sexual behaviour are targeted. 22,32,34,37–39 The underpinning theoretical framework for the intervention used in this study combines the well-established theory of planned behaviour40,41 and recent updates to this theory,42,43 which focus on the individual behavioural antecedents of an unplanned pregnancy along with an understanding of the broader socioenvironmental factors (such as social class) and underlying values (such as religiosity and gender ideologies) associated with the occurrence of teenage pregnancy. 18 Interventions should target six psychosocial mechanisms that research indicates are related to a reduction in risk-taking behaviour: knowledge, skills, beliefs about consequences, social influences, beliefs about capabilities and intentions. 19,44,45

Evidence supporting the use of culturally relevant and gender-specific interventions

The World Health Organization (WHO),46,47 recent systematic reviews commissioned by the US-based National Campaign to Prevent Teen Pregnancy,48,49 and the NIHR,50 among others,1,22,51–54 recognise that adolescent men have a vital yet neglected role in reducing teenage pregnancies, and that there is a pressing need for RSE interventions designed especially for them.

In relation to cultural relevance, research suggests the need to engage with young people both empathetically and cognitively in order to increase the relevance of the issues being raised. 14,23,30,31,55 As Ingham and Hirst24 have noted, it is important to harness the potential for sex education to be keenly anticipated, especially by those who are less engaged in the wider school curriculum, a factor that was identified as a possible barrier to impact. 50

Evidence relating to socioeconomic position and inequalities

Research suggests that interventions targeting only schools in disadvantaged areas are insufficient for achieving population-level reductions in teenage pregnancy. 56,57

Evidence supporting the use of interactive computer-based interventions

Recent systematic reviews have shown the value of interactive-computer-based interventions,34,35,55 and a meta-analysis examining these reviews in relation to the theoretical mediators of safer sex36 concluded that they were successful in impacting knowledge, attitudes and self-efficacy relating to sexual health.

Evidence supporting the use of skills-building components

Reviews and trials of health promotion and educational interventions show that simply providing information does not lead to behaviour change and that instead it is necessary to support young people to develop their own communication skills in relation to preventing risky sexual behaviours. 14,17,19,21–23,30,34,47,58 A recent NIHR Health Technology Assessment systematic review19 of the effect of interventions aiming to encourage young people to adopt safer sexual behaviour found that school-based interventions that provide information and teach young people sexual health negotiation skills can bring about improvements in behaviour-mediating outcomes such as knowledge, attitudes and self-efficacy. The authors noted that these variables are no less valuable than behavioural variables because they provide young people with a solid foundation on which to make sexual decisions.

Evidence supporting the involvement of parents in Relationship and Sexuality Education

Although evidence suggests that schools are an important context for sex education,14,48,59 recent systematic reviews have also shown that programmes that reach beyond the classroom (including multifaceted approaches with a parental or community component) are more effective,19,25,60 particularly with adolescent men. 48 In particular, factors such as parental monitoring and supervision and familial communication have been associated with adolescent sexual behaviours. 61,62 Teenagers who can recall a parent communicating with them about sex are more likely to report delaying sexual debut and increased condom and contraceptive use. 63–65

Most parents do, however, recognise the importance of speaking to their children about sex and most would like more resources to help them in this regard. 66 Furthermore, many teenagers report parents as a primary source of information on sexual health matters. 67 The unprecedented growth in the use of online and mobile technology over the past few decades by people of all ages presents opportunities for increasing the reach (and decreasing the perceived embarrassment) of parental involvement in sex education. 68 More than 20 years ago, Miller et al. 66 found that a home-based video sex education intervention designed to make it easier for parents to talk to their teens about sex was effective in increasing the frequency of parent–teen communication regarding sexual topics. The use of ‘education entertainment’ for addressing health issues has become increasingly common since then69 and, in recent times, studies61,68,70 have demonstrated the importance of embracing such modalities as engaging adjuncts to school-based education.

Previous research by members of the study team

When we used an earlier version of this interactive video drama (IVD), a component of the intervention in this study (see Chapter 3) as part of a research project on teenage males and unintended pregnancy, 85% of a sample of male pupils in Ireland (n = 360) and 72% in Australia (n = 386) agreed or strongly agreed with the statement that it ‘helped me understand the effect an unplanned pregnancy would have on a guy like me’. In Ireland and Australia, 79% and 69%, respectively, agreed or strongly agreed that the IVD ‘made me realise that I should never get myself in that situation’. 71

The public health concern of teenage pregnancy and the potential reach of targeted school-based interventions to provide young people with a solid foundation on which to make sexual decisions

The core aim of the If I Were Jack intervention is to use an evidence-based process to allow schools to provide pupils with opportunities for exploring the issues surrounding teenage pregnancy and parenthood with reference to the individual, family, community and society. If it is acceptable and effective, it has the potential to be rolled out to large numbers of boys and girls attending schools across the UK and to be of benefit to several groups of people. In 2011/12 there were 24,251 pupils in Year 11 in the 216 mainstream post-primary schools in NI72 and 551,800 pupils in the 3268 state-funded secondary schools in England. 57 Pupils will benefit from an engaging educational resource. Parents will benefit from a process designed to involve and support them in communicating with their child. RSE teachers will benefit because the resource will provide them with a stimulus for engaging pupils in an immersive decision-making experience from the point of view of a teenage male, followed by guided classroom-based activities.

Teenage pregnancy is both an outcome of, and a contributor to, inequalities in health. 12 The aggregate projected spend for 2013–20 for treating both unintended pregnancy and sexually transmitted infections (STIs) (for all ages) across the UK is estimated to be between £84.4B and £127B. This is based on a projected spend of £11.4B of NHS costs as a result of unintended pregnancy and STI costs, and between £73B and £115.3B of wider public sector costs. 73 The NI government estimates the cost to the Exchequer of teenage pregnancies at £20,000 per teenage mother on the basis that a teenage pregnancy effectively withdraws the mother from the labour market (not in education, employment or training) for at least 18 months and accounting for unemployment benefits and administration, plus tax revenue forgone. 26 Our population-based approach is consistent with the evidence that universal interventions are necessary to achieve population-level reductions in teenage pregnancies. 56,74–76

This study will begin a process of robust evaluation that may also lead to the roll-out across the UK and international applications of the intervention

Internationally, policy-makers and researchers have called for scientifically evaluated school-based RSE interventions. 16,21–25,77,78 A successful feasibility would underpin the development of a UK-wide Phase III trial. The intervention has been designed specifically for use in NI, but it is based on international research regarding effective RSE interventions. Although the video drama features NI actors, there is no specific reference to NI in the video and we believe that it would have strong cultural resonance in the rest of the UK. Furthermore, although the accompanying materials make specific reference to the NI curriculum and legislation in NI governing termination of pregnancy, the costs of making changes to these references so that the materials could be used in contexts across the UK would be negligible.

The need to develop and evaluate age- and gender-specific Relationship and Sexuality Education resources

The need for gender-sensitive interventions to address teenage pregnancy has been highlighted as a global health need by the WHO46,47 and has been recommended in systematic reviews. 19,48,49 Addressing teenage males’ RSE is an important mechanism for promoting positive development and improving the lives of all young adults, especially those suffering the effects of various types of disadvantage. 48,51–53,79

A recent inspection by the Education and Training Inspectorate of RSE implementation in post-primary schools in NI identified the need for RSE resources for older adolescents, especially at Key Stage 4 (age 14+ years). 80 The absence of such resources is thought to be a demotivating factor, making RSE more difficult for some schools to implement. Internationally, there has also been demand for age-appropriate school-based RSE interventions. 16,21–25,77,78

We aim to initiate a process of robust scientific evaluation, which will ultimately produce generalisable findings especially relating to gender-specific interventions. The intervention we propose is gender-inclusive in terms of intent and impact and it can be used in mixed-sex classrooms, but is gender-specific in that it specifically targets the inclusion of adolescent men by developing an interactive film from a man’s point of view.

The strengths of the If I Were Jack intervention

The If I Were Jack intervention is grounded in empirical research on adolescent men’s attitudes to UTP; was developed in consultation with key health and education experts, pupils, teachers and parents; and is informed by the best available evidence regarding the development of classroom-based RSE interventions. It is predicted to impact on a number of behavioural and psychosocial mediating variables, which research suggests decrease sexual risk-taking behaviour. It uses an innovative combination of intervention components which address deficits in existing RSE interventions and aim to maximise potential impact.

The right place for this study

This research is necessary to address the deficit of research on high-quality RSE interventions in NI. Although RSE is mandatory in NI, its provision in schools is known to be underdeveloped and highly variable. 9,80–83 In addition, given that RSE in NI is underdeveloped, we are more likely to be able to test the efficacy of the intervention in NI as a prelude to a Phase III trial to test its effectiveness. In our preliminary research and in working through a partnership model to develop this study, we believe that we are in a strong position to overcome the problems in implementing RSE in NI and have already found the resource to be culturally sensitive and acceptable to statutory stakeholders and schools.

The educational and policy demands for such an intervention

The letters of support from across the health and educational sector of the UK (including, in NI, the Chief Medical Officer, PHA NI, DHSSPS and CCEA) identify the need for and support this study. Several stakeholder organisations are also offering in-kind contributions in terms of membership of a study stakeholder’s group along with a contribution of £20,000 co-funding from PHA. This stakeholder investment in this project attests to the widely held need to evaluate evidence-based, theory-informed RSE resources that clearly target boys as well as girls in relation to unintended pregnancy and improving sexual health and well-being.

Stage 1: preliminary research and development of the interactive video drama

Stage 2: development and optimisation of the intervention

Stage 2 focused on developing the intervention. This involved redevelopment of the IVD and design of accompanying classroom materials based on evidence regarding the characteristics of effective RSE interventions and theoretical understanding of the determinants of behavioural change. Also central to stage 2 was consultation in relation to the content (see Chapter 1, Public and patient involvement).

Limitations

A recent systematic review of effective sexual health interventions19 indicated the importance of building connections with community-based sexual health services and providing longer-term programmes. The If I Were Jack intervention was unable to address such components owing to the strategy employed to develop and test an existing IVD. Such issues and other evidence-based components could be addressed in future development work.

As the If I Were Jack intervention strongly emphasises the potential negative consequences that having a child might have on a teenager’s current life or future goals, another potential limitation of the intervention is that it may have a possible negative impact on teenagers who have had, or are about to have, a child. It has the potential to reinforce or indeed instigate stigmatisation processes. Design features of the intervention aim to off-set this potential negative impact through the teacher training process, which emphasises the importance of reminding and discussing with pupils that the resource refers to UTP (i.e. a pregnancy that is unexpected and unwanted) rather than a teenage pregnancy in general, which for some young people can be a planned and positive experience. Pupils are also informed by the teachers and researchers involved that they can and should contact the school counsellor/pastoral care (PC) officer should they find any part of the intervention upsetting.

Rationale

A key outcome of this study was to pilot test the feasibility, usability and acceptability of providing demographic data and answering questions measuring the proposed primary and secondary outcomes of a future Phase III trial (see Chapter 6), that is, sexual behaviour data, including engagement in sexual intercourse, contraception use and diagnosis of STIs; data regarding knowledge, attitude, skills and intentions relating to avoiding UTP; and data on socioeconomic status (SES). The pilot study was not intended to assess the design and usability of the format of the questionnaire itself.

Methods

Unlike the finalised version of the questionnaire (for the feasibility study), the pilot version was of a basic format using the SurveyMonkey™ (Palo Alto, CA, USA) platform as the feasibility trial website was still in design.

To pilot test the questionnaire prior to the beginning of the feasibility trial, a school was approached that had been identified as willing to use the questionnaire, through prior liaison, with 50–60 of their Year 11 students (male and female, aged 14–15 years), who had proactively volunteered. In order to facilitate completing the questionnaire online, the Year 11 students were spilt into two groups across two different computer rooms and a researcher and a Learning for Life and Work (LLW) curriculum teacher (see Appendix 7) were present at all times in both groups. The session lasted for 1 hour. The data were analysed quantitatively by use of descriptive statistics and analysis of the reliability and spread of the scales.

Qualitative analysis was also made possible via focus group methodology. Focus group interviews are used extensively within social sciences to elicit information, perceptions, beliefs and attitudes. The group dynamic allows for a more natural conversational style rather than a traditional rigid interview technique. Researchers often use this technique to engage the end user in instrument design. Pupils who had completed the questionnaire were asked to participate in the focus group session regarding the acceptability and feasibility of the questions. These interviews lasted for 25 minutes. The students gathered around in a circle and the discussion was audio-recorded to facilitate note taking at a later stage. To begin with, the researchers reminded the students of the purpose (i.e. to find out what they thought about the questionnaire and not to be afraid of giving their honest opinions). The researchers also went over some ground rules, following the Respect, Involvement, Confidentiality, Equality (RICE) rules below:

R: Respect for an individual’s contributions, which we can demonstrate by listening to one another’s views.

I: Involvement in the discussions and activities.

C: Confidentiality – we are not going to talk about our own personal/individual stories but we can talk about relationships in general.

E: Equality – we want to acknowledge that everyone’s experience and opinions are equally valid.

The questions asked during the course of the cognitive interviews are listed below.

1. Would you prefer to complete the questionnaire on the computer or on paper?

2. What did you think about the length of the questionnaire?

3. What did you find useful or interesting about the questionnaire?

4. What did you find difficult or confusing about the questionnaire?

5. Is there anything that you think should be removed from the questionnaire?

6. When you talk about these issues with your friends what terms do you tend to use?

7. I have some specific terms that I wanted to check your understanding of. Ask what would that mean to you? Do you think there are some people your age who would not understand what it meant?

For invitation letters, information sheets and consent forms for schools, teachers, parents/guardians and pupils regarding questionnaire development, see Appendix 2. For the focus group topic guide, see Appendix 3.

Results

The participating school was a grant-maintained integrated co-educational secondary school (which did not go on to be involved in the feasibility trial). The school was located in an urban area and 17.8% of its students were eligible for free school meals. In total, 38 students (mean age of 14.72 years) agreed to take part (in two rooms containing 22 and 16 pupils, respectively), provided parental consent and completed the questionnaire.

Quantitative analyses

The estimated time taken to complete the questionnaire was 40 minutes; the actual time taken was around 25 minutes, including time taken to log in. The first section contained sociodemographic questions. The sample of respondents comprised 55% male and 45% female students and was spilt between 14- and 15-year-olds (26% and 68%, respectively). All students completed these questions, although two incorrectly inputted the date of the pilot study instead of their date of birth. Owing to this error, the online questionnaire was developed to bring up an error message if the year of birth does not equate to a Year 11 student. Sixteen students skipped the question that asked them to write in their postcode. As this information is important for a check to ensure that questionnaires can be correctly linked across the trial and potentially used for gathering multiple deprivation score, it was necessary to keep this in but it was accompanied with more explanation as to why this information was being collected. One question asked about the educational achievement of the respondent’s mother and father (or guardian) and was broken down into six questions asking specifically about General Certificates of Secondary Education (GCSEs), Advanced levels (A levels), degrees, apprenticeships, diplomas or no qualifications. Five participants skipped these questions and 18 did not know the answer. It was therefore decided to incorporate these questions into a single item referring to highest achieved qualification.

Owing to limited variation in responses received to the categorical tick box responses for the question ‘Whose responsibility is it to prevent unintended pregnancy?’, it was decided to change the response format of the question to the following visual analogue scale (Figure 3), with the following instructions: put an X on the line below to show how much responsibility you think boys and girls should have. For example, if you think that girls have more responsibility than boys, put an X on the line closer to girl; if you think that boys have more responsibility than girls, put an X on the line closer to boy; if you think that boys and girls should have equal responsibility, put an X in the middle of the line.

FIGURE 3.

Whose responsibility is it to prevent unintended pregnancy? A visual analogue scale.

Only 14% (five participants) of the sample said that they had experienced penetrative sex and only three answered the question regarding their age at which they first had sex, which was 15 years. When asked how many times they had had sex, two participants responded that they had had sex once; however, one participant said that they had had sex 30 times. It could be likely that they have been sexually active 30 times in the past but it could also be the case that this may have been an error and was perhaps meant to say ‘three’. To minimise this type of error and therefore to ensure that we could correctly interpret the data, it was decided to use fixed-category responses rather than free responses.

As the pilot study sought to determine participants’ level of comfort in answering potentially personal questions about a sensitive topic, the question ‘how comfortable do you feel in answering these questions?’ was added to the pilot questionnaire to assess this. Most participants felt comfortable answering all or most of the questions (n = 18), but some did report experiencing discomfort (n = 7).

Summary of changes made to questionnaire for feasibility study

In conclusion, the Year 11 students who assisted with the pilot development stage of the questionnaire were mostly positive about their experiences in completing the questionnaire. Feedback on their understanding of key terms used was critical to ensure that future participants can understand what is being asked and also to enable reliable estimates of effects. A number of changes were made to the questionnaire in light of this work, which are summarised below.

• changes made to terminology used

• ‘hover over’ definitions used on terms on the online questionnaire and a separate definitions page on the paper version of the questionnaire

• response format from one questionnaire changed to a visual analogue scale

• more description about a Likert scale and how to answer them

• removed Cost of Unintended Pregnancy scale

• the knowledge items two, four, nine and 10 were removed

• changed ‘How many times have you had sex?’ question from free response to fixed-category tick boxes

• included ‘not applicable’ option to question on paternal and maternal education?

• greater explanation of why we are asking for postcodes

• inserted reminders in the online questionnaire if any items are left blank unintentionally

• validity checks for date of birth inserted into online questionnaire

• explained why the questionnaire contains questions relating to parental education and finances

• broke up how longer scales are displayed – no more than six questions per page

• added progress bar to online version.

Rationale

The aim was to further refine the survey instruments by assessing their acceptability and usability among the pupils now completing them within the setting of the feasibility study.

Methods

An electronic version of the questionnaire was developed post pilot and offered to the schools as an alternative to the use of paper-based questionnaires.

The data were analysed quantitatively following baseline data collection, by use of descriptive statistics and analysis of the reliability and spread of the scales. The research team also made minor amendments to the survey instruments to aid user understanding after each data collection wave; these changes were based on analysis of responses and qualitative experiences of watching the pupils complete them.

As with the pilot study, qualitative analysis was also made possible via focus group methodology. This was conducted post trial. Two separate focus groups were conducted within two of the control group schools. The focus groups were audio-recorded to ease transcription and lasted no longer than 45 minutes. Soft drinks and snacks were provided and the interviewer tried to create a relaxed atmosphere to facilitate a discussion in which everyone had a voice. The focus groups were timed to coincide with the end of data collection for the feasibility trial and occurred either directly after data collection or within a few days of this. This timing ensured that issues the participants may have had and the survey instrument itself would be fresh in their minds. Although they were already familiar, sample questionnaires were given out to all participants so that they could examine it in detail.

Participants were given an overview of what a focus group is as well as a topic guide (see Appendix 3). They were reminded that although they would be discussing the questionnaire, for example, whether the questions were easy or difficult, they would not be asked to discuss their own personal answers. For relevant invitation letters, information sheets and consent forms for schools, teachers, parents/guardians and pupils, see Appendix 2.

The focus group schedule (see Appendix 3) was divided into five topics: questionnaire instructions, questionnaire design, terminology used, individual questions and other issues.

Results

Summary of changes made to questionnaire for first follow-up data collection (5 months after baseline data collection)

1. Added ‘If there are any questions you do not understand, look at the list of definitions at the end of the questionnaire but please ask us if you need any help ‘ to the instructions.

2. Questions 4 (What is your ethnic background?), 5 (How religious do you think you are?) and 6 (What religion where you brought up with?) removed from the baseline questionnaire.

3. Question 7: changed to include ‘(By this we mean how well off your family is in terms of money)’.

4. Question 11: changed from ‘Have you ever drunk alcohol?’ to ‘Have you ever taken an alcoholic drink (not just a taste or a sip)?’.

5. Question 20: certain questions within changed from ‘For girls/boys only’ to ‘If you are a girl/boy’.

6. Question 21: changed font size and highlighted important instructions.

7. Question 22: removed ‘This section concerns beliefs about being a man.’ From the description.

8. Question 23: changed ‘ your degree of agreement/disagreement with’ to ‘how you feel about’.

9. Question 43: moved to the next page.

10. Moved Thank you message up to above Question 46.

11. Question 46: removed lines and shortened box size.

12. Added definitions page.

Summary of changes made to questionnaire for second follow-up data collection (9 months after baseline data collection)

1. Added NIHR logo.

2. Removed Participant identification number (ID) and Date completed from front cover.

3. Added ‘If there are any questions you do not understand, look at the list of definitions at the end of the questionnaire but please ask us if you need any help‘ to the instructions.

4. Questions 4 (What is your ethnic background?), 5 (How religious do you think you are?) and 6 (What religion where you brought up with?) removed from the follow-up questionnaire.

5. Question 7: changed to include ‘(By this we mean how well off your family is in terms of money)’.

6. Question 11: changed from ‘Have you ever drunk alcohol?’ to ‘Have you ever taken an alcoholic drink (not just a taste or a sip)?’.

7. Question 20: certain questions within changed from ‘For girls/boys only’ to ‘If you are a girl/boy’ and added ‘Boys & Girls’ to other questions.

8. Question 21: changed font size and highlighted important instructions.

9. Question 22: removed ‘This section concerns beliefs about being a man’ from the description.

10. Question 28: added additional option ‘not since June 2015’ as the first option.

11. Question 43: moved to the next page.

12. Moved Thank you message up to above Question 46.

13. Question 46: removed lines and shortened box size.

14. Added definitions page.

15. Changed all ‘Go to Q…’ references to ‘Go to Question…’.

16. Changed word search.

Mode of administration: electronic versus paper based

Consenting pupils were tested using either a paper questionnaire or an online survey under exam conditions. When making arrangements to facilitate the testing, schools were given the option of how they would prefer to administer the questionnaire. Three schools agreed that they would prefer to use the online survey. Although using an electronic version of the questionnaire had numerous benefits (informed by use of the electronic SurveyMonkey™ online platform in the pilot study), including increased data completeness and the hover-over definitions, its use was discontinued after the first follow-up owing to technical difficulties. Only some pupils from two comparison schools were able to use this mode. During the baseline data collection there were numerous administrative difficulties in using the electronic questionnaire. All three schools had issues, including not being able to log-in and having timing out issues, and a significant number of participants in these classes had to transfer to using a paper questionnaire. For the first follow-up only one school of the original three was happy to try the electronic method again. The software developers were present throughout this data collection to troubleshoot problems and again the same issues occurred. After this session the data collected were not available on the server and these data were unfortunately lost. Owing to these issues, the research team was not confident that the electronic system was fit for purpose and its use was discontinued in order to ensure the safety of the collected data and to maintain positive relationships with both schools and participants who were frustrated with the software issues. In total, 2231 paper versions of the questionnaire were completed and 82 questionnaires were completed online (data from 11 of these were lost as a result of a technical error).

The online survey was able to record the length of time taken to complete the survey; this was recorded at 23 minutes and 45 seconds (range 8–43 minutes). Detailed time records were not available for those completing the questionnaire in paper format but the majority of pupils were able to complete the questionnaire within the constraints of one timetabled lesson.

Why is successful recruitment and retention important: a background literature review

Randomised controlled trials generally require large numbers of participants and follow-up over long time periods. Recruiting adequate numbers of participants and retaining them in the study for its entire duration, although challenging, is essential for ensuring internal and external validity and reducing bias that is introduced if certain groups of participants refuse participation or withdraw from the study.

However, participation rates in general have been decreasing over the past decades and it is increasingly difficult to recruit and retain participants to RCTs. 96 Fewer than one-third of publicly funded RCTs reach their recruitment targets within the specified time frame and more than half need to be extended owing to recruitment difficulties. 97 This can result in extended study duration, increased expenditure and findings that are not statistically accurate, because recruitment may be biased towards particular groups. 98 In worst case scenarios, insufficient recruitment or retention can result in a trial being stopped or a delay in the roll-out of an intervention with serious public health benefits. 99

Factors influencing recruitment and retention

It is widely recognised that the proliferation of research studies, marketing surveys and opinion polls over the past decades, and the resulting increase in the number of participation requests received by both individuals and sites, has led to a situation whereby increasing numbers of people refuse all (or all but the most immediately relevant) requests to take part in research. 100 Other factors thought to influence research participation include the characteristics and demographics of potential participants, factors related to the study design, and characteristics of the investigator(s). 101,102

Demographic factors that are associated with increased likelihood of participating in research include being younger, female, married, of higher SES, of higher educational attainment and in employment. 96,101,103 Factors relating to the study design include the length of time participants are expected to engage with the study, the potential burden placed on them, the treatment being offered, concern about allocation to the control group and the focus of the study (in particular whether or not it is sensitive/controversial). 98,101 The characteristics of the organisation responsible for the research and its representative investigators, particularly perceptions of their credibility, is also thought an important influencing factor. 98,104

In reference to school-based research, Pound et al. 104 found that perceptions of the social value of the study (i.e. value to students and the community and perceived benefits of taking part) and confidence in the organisation responsible for the research strongly influenced the decisions of school management to take part in a trial. Barriers to recruitment included the amount of effort involved for the school, interest in the project and commitment to other research. In a qualitative investigation of the views of school superintendents and principals, Befort et al. 105 found that school management was interested in research projects that (1) provide tangible benefits to their school; (2) are consistent with their academic mission; (3) are not burdensome; (4) do not take place during exam time or other busy periods; and (5) are credible and non-controversial.

Strategies to optimise recruitment and retention

In a recent systematic review of methods to improve recruitment to RCTs, Treweek et al. 98 concluded that effective interventions include telephone reminders to non-respondents; the use of opt-out rather than opt-in procedures; and open designs whereby participants know which treatment they are receiving in the trial. Over the years, others have reported that recruitment facilitators include engagement with the target population when developing the intervention and preparing participant information about the study;106 a personalised and culturally sensitive approach to participants;99,107 designing recruitment processes that are aimed towards clearly highlighting to potential participants the beneficial outcomes of taking part and addressing any barriers or perceived negative outcomes;108 providing research and implementation support by providing a dedicated research team contact;109 minimising the workload for participants by adapting trial processes so they are in line with usual ways of working at the site;109 providing financial or other incentives;29 and good communication and expressions of appreciation to site facilitators and participants. 108,109 Although some have suggested the value of using technology such as internet-based questionnaires110 and the use of video to provide trial information111–113 to improve recruitment, a recent systematic review98 concluded that their effect is not clear. At the very least, however, the use of video does have obvious benefits in terms of recruiting participants with low literacy levels. 114

A number of studies have reported recruitment and retention facilitators that are specific to schools. Lytle et al. 108 argue that an essential strategy is to adjust the recruitment process to suit the culture of each school and to plan to impact on a number of individual-, environmental- and behavioural-level factors. In addressing individual factors, they sought to help the school decision-makers recognise the importance of the research projects goals and objectives. They did this by emphasising the proximal and distal public health benefits of the programme intervention and highlighting the value and credibility of the programme using expert testimonials. In addition, they increased the primary decision-makers’ self-efficacy in being able to ‘sell’ the programme to implementers by offering to speak to implementers themselves. The researchers sought to influence environmental factors by ‘reducing external barriers to acceptance, enhancing social norms, and by committing future support to schools through training and materials provision’. 108 To achieve this they shared testimonials from participants who took part in the pilot trial, adjusting recruitment procedures to suit individual schools, providing a clear outline of roles and responsibilities of both school and research team members, facilitating discussion on the benefits and perceived barriers to taking part and providing external support (e.g. training, paying for substitute teachers), which would enhance implementation in intervention schools. Targeting behavioural factors, the team provided incentives in the form of training, research and intervention materials and support during implementation. For schools randomised to the control group, materials were provided at the end of the trial. In addition, providing refreshments at any meetings was seen to increase attendance as was sending personal thank you notes to all involved from the research team following such meetings. Obtaining ‘intention to act’ was operationalised in the recruitment process by asking individuals to sign a contract. Although these were not considered legally binding they were part of a behavioural process important for formalising the agreement. Lytle et al. 108 also imply that marketing techniques can be of value in recruiting for schools. For example, they sent a letter to schools saying that ‘only two more slots’ were available and used videos with end-user testimonials to increase credibility and perceived acceptability.

Aarons et al. 107 argue that it is essential to consider each participating school’s needs in planning intervention and data collection activities. They argue that it is essential to communicate to decision-makers at the recruitment stage that the research will not interrupt the ‘ebb and flow’ of the school. In particular, time and timing are important considerations for both recruitment and implementation planning in schools; although study deadlines must be met, this needs to be balanced with the school timetable. 107 Summer, Christmas and Easter closures as well as half-term breaks in autumn and winter must all be considered, along with school-specific staff development closures, public holidays and exam periods. Furthermore, the school curriculum is often overcrowded, with some schools less willing than others to remove pupils from exam subjects. In this regard, Befort et al. 105 suggest that, from the outset, researchers should be very clear about time commitments and how the study may be used to improve the school’s academic mission.

Financial incentives have been reported to be beneficial for recruiting and retaining both schools and pupils. 29 Harrington et al. 115 suggest that if schools are concerned about possible allocation to the control group (thereby depriving their pupils of a potentially beneficial intervention), a delayed treatment condition rather than no treatment may be an added incentive for schools to participate.

Potential barriers to and facilitators of successful recruitment and retention

Table 18 details potential barriers to and facilitators of successful recruitment and retention.

In conclusion, as outlined in Table 18, it was anticipated that a number of barriers to and facilitators of recruitment could impact on our ability to reach recruitment targets and retain participants in the study. In general, it was anticipated that key influences would be whether or not (1) the school was currently or recently involved in a research project; (2) school management, teachers and parents saw RSE as a priority subject and teenage pregnancy as a current concern; and (3) school management, teachers and parents perceived both the research project and intervention as credible, beneficial, compatible with their needs, ways of working and ethos, and not burdensome in terms of time and resources. Although the initial intervention design and development process sought to overcome potential intervention-level barriers (see Chapter 3), it was also anticipated that important research-level facilitators of recruitment and retention would include (1) approaching schools that were more likely to see RSE as a priority subject and teenage pregnancy as a current concern; (2) encouraging buy-in and intention to act by making roles and responsibilities clear from the outset, assigning a trial champion in each school and building positive relationships with all participants; and (3) providing token incentives, reimbursement of expenses and research support.

Schools

Initial and revised recruitment targets are described below. In 2013, there were 210 registered post-primary schools in NI. NI has a unique education system and post-primary schools can be broadly categorised as ‘secondary’ (n = 133) or ‘grammar’ (n = 68), with the latter using academic ability to select pupils. Various management structures also exist, with ‘controlled’ schools (n = 75) managed by one of the five NI Education and Library Boards and ‘voluntary’ and ‘maintained’ schools (n = 135) managed by a board of trustees. The Roman Catholic (RC) Church manages a significant number of voluntary (RC maintained) secondary (n = 68) and grammar (n = 29) schools. Although religion is not a criterion for attendance at NI schools, most pupils at controlled schools are from Protestant denominations and most of those attending RC maintained schools are Catholic. There are also a number of ‘integrated’ schools (n = 20), which aim to provide a religiously and culturally mixed environment.

Eight post-primary schools (five controlled secondary schools and three RC maintained secondary schools) in NI were excluded from target recruitment because they had < 30 Year 11 pupils and one was excluded because it was an Irish language school (secondary other maintained). The remainder of registered post-primary schools were included as targets for recruitment (Box 1).

In order to capture this diversity, and reflecting the aim to examine the acceptability of the intervention in different school types, the following seven schools were initially sought for recruitment: one integrated; one controlled grammar; one voluntary ‘other managed’ grammar; two voluntary RC grammars; and two schools of any type in deprived areas. Our preliminary research indicated potential challenges in recruiting RC schools owing to the fact that some may perceive an ‘abstinence-plus’ intervention such as If I Were Jack to be incompatible with the Catholic ethos. Conversely, it was anticipated that there may be greater uptake of the intervention in deprived areas where teenage birth rates are higher. For this reason two RC grammar schools and two schools in deprived areas were targeted for recruitment and one of each was randomly assigned to the comparison and intervention groups.

Teachers and other school staff

One teacher who would act as main trial contact within each participating school was targeted for recruitment. This individual would be asked to work with the trial manager to co-ordinate baseline and follow-up data collection with pupils, to recruit teachers to deliver the intervention (if necessary) and to participate in an interview regarding their perceptions of taking part in the research and (if appropriate) current RSE provision in the school. In intervention schools, at least one teacher was targeted for recruitment to deliver the intervention to Year 11 pupils and to take part in an interview about their experiences of using the intervention. In addition, within each school at least one school administrator/secretary to support the research team and make contact with parents was targeted for recruitment. Although not an initial consideration, the need arose to recruit a school Information Technology (IT) technician who could support use of the interactive film in intervention schools and use of the online survey in all schools opting to use this method of data collection.

Pupils

The target population was all Year 11 pupils aged between 14 and 16 years in participating schools. Pupils were excluded if their parents/guardians withdrew them from the study, if they declined to take part or if they were unable to read the questionnaire owing to language difficulties (i.e. if English was not their first language). Pupils with reading difficulties and/or learning disabilities were included if they wanted to take part and if teachers thought that they would be able to understand and complete the questionnaire (with or without the aid of a teaching assistant or researcher reading the questionnaire aloud and explaining problematic words). Based on the average size of Year 11 in schools in 2011/12 (mean = 114; median = 113) and allowing for an 80% consent rate, it was estimated that approximately 730 pupils would be recruited to the study.

Parents

At least one parent or guardian of each participating Year 11 pupil in the intervention group was sought for recruitment to attend a 1-hour parents’ and guardians’ information and discussion session at their child’s school. Based on anecdotal evidence from teachers regarding difficulties engaging parents in non-academic activities, it was estimated that parent/guardian representatives of approximately 50% of Year 11 pupils would attend these sessions, which would result in around 200 participating parents.

Recruiting schools

Three strategies were used to recruit targeted schools: (1) RSE teacher training events; (2) personal introductions by members of advisory and steering groups; and (3) ‘cold-call’ invitation. Regardless of the strategy used, all principals [and, when applicable, the RSE co-ordinator or PC vice principal (VP)] were posted a formal letter of invitation and information sheet, which included detailed information on what would be involved on consenting to participate, including the estimates of required time and resources (see Appendices 2 and 3). This was followed, within 1 week of posting the letter, with a telephone call to the principal. In most instances, several more attempts to make contact with the principal and/or other contact (if suggested by the principal at first contact) were required before a decision on whether or not they would like to participate was made. In most cases the decision was made following a meeting with the principal or PC VP and, for the majority of schools, a teacher [such as the designated RSE co-ordinator, Head of Year 11, Head of LLW curriculum/personal development (PD), or Head of Religious Education (RE)]. During this meeting the research fellow described the intervention and the research study and explained what would be expected of the school, teachers, pupils and parents, should they agree to participate. In order to promote the credibility of the research project potential participants were told that the study was funded by the NIHR and the PHA NI and had received ethics approval from QUB. In order to promote the credibility of the intervention and impact perceptions of its potential benefits, the meeting also included the presentation of a 5-minute video of health and education experts talking about the resource and its potential benefits for use in the classroom and testimonials from teachers who had used the intervention during the pre-piloting development phase.

Recruiting teachers and other school staff

The initial meeting between the trial manager and the school generally included either the principal or PC VP and a teacher who had attended the RSE training event or had responsibility for RSE/LLW. The main school contact following school consent to participate (the trial champion) was either the head of LLW and RSE co-ordinator (five schools); head of RE (one school); head of Year 11 (one school); or a senior teacher (one school).

Participating teachers who would deliver the intervention if randomised to the intervention group and/or facilitate delivery of the questionnaire at baseline and follow-up were identified by the principal/PC VP or trial champion prior to baseline data collection and randomisation. These teachers were then invited to attend a 30-minute meeting with the trial manager at the outset of the study. During this meeting they were provided with information sheets and the research was explained to them with the aid of Microsoft PowerPoint version 14 (Microsoft Corporation, Redmond, WA, USA) slides and a study flow chart. They were offered an opportunity to ask questions and consent to taking part in the research. In only two schools did the RSE co-ordinator attend this meeting, both of whom indicated that they had not yet identified other teachers to take part in the research. Both of these schools assigned teachers to facilitate the questionnaire data collection without any input from the researcher. One of these schools was later assigned to the intervention group, at which point an additional teacher was provided with an overview of the research during the teacher training session. In one school all members of the teaching staff were provided with an overview of the research during a staff training day. During the information sessions teachers were asked to consent to take part in an interview or focus group discussion and these were arranged with individual teachers or via the trial champion at the appropriate time. At the end of the study the trial champions in all schools were presented with a box of chocolates or bunch of flowers and an individualised thank you card from the research team.

At the initial meeting with school management the trial manager asked if she could be introduced to a school administrator who might help with administrative aspects of participation (i.e. provision of class lists, contact with parents, expenses). In five schools the trial manager was introduced to this person and in three schools contact only ever took place via e-mail or telephone. Following initial labelling and posting of information sheets and consent forms to parents, these individuals were presented with a small plant and note of thanks from the research team.

Although not initially anticipated, in some schools the trial manager had contact with the school’s IT technician. In intervention schools this involved liaison with the technician to ensure that pupils could access the If I Were Jack film and online resources and, in schools that opted to allow pupils to complete the questionnaire online, the IT technician was asked to test access to the survey and help with troubleshooting problems with this (see Chapter 7).

Recruiting pupils

After the school and teachers had consented to take part, parents/guardians of all Year 11 pupils were posted information sheets and given 2 weeks to withdraw their child from the study before the research was brought to the attention of pupils. During the teacher’s information session, the trial manager offered to spend a class period with pupils prior to data collection, during which the research would be explained in detail to them. All schools declined this offer stating that, owing to time constraints, it would be preferable to provide this information at the beginning of the baseline data collection session. Teachers were therefore provided with information sheets to distribute 1 week prior to baseline data collection to Year 11 pupils whose parents had not withdrawn them from the study. They were asked to encourage pupils to read the information sheet and inform them that they would have an opportunity to ask questions about it and decide whether or not they wanted to take part before completing the questionnaire the following week.

At the beginning of the baseline data collection sessions, either the trial manager or a trained research assistant spent 15 minutes explaining the research to pupils, giving them an opportunity to ask questions and asking them to complete a consent form indicating whether or not they wanted to take part. As a means of saying thank you to pupils for taking part in the study (and an incentive suggested by teachers), chocolates and sweets were provided to students after they had completed each questionnaire. Pupils who declined to participate or whose parents opted them out were supervised by a teacher in a separate classroom or the school library during data collection.

Participating teachers were asked to recruit volunteers from their class groups (six pupils minimum) to take part in a focus group discussion with a researcher (immediately following implementation in intervention schools and immediately following the 9-month follow-up in comparison schools). Pupils were provided with drinks and snacks during these sessions as an incentive for participating.

Recruiting parents

Parents or guardians of participating Year 11 pupils in the intervention group were invited via letter to participate in a 1-hour parents’ and guardians’ information and discussion session at their child’s school. At the request of the research team (and owing to a very small number of reply slips indicating attendance or non-attendance) a reminder text message was sent from the school to parents the day before the session. In three schools this session was managed and facilitated by a teacher and in one school (owing to teacher sick leave) by the PI. Schools were encouraged to run the event in the evening to facilitate attendance by working parents.

In an attempt to reach parents who did not attend the session, a 6-minute video for parents/guardians containing key messages was posted on YouTube. School administrators texted a link to the video to parents.

Parents who attended the parents’ session were invited to participate in a focus group discussion with a researcher immediately following the session if that suited them or to provide their contact details if they wished to take part at another time. In addition, the YouTube video provided contact details for any parents wishing to take part in a focus group.

Following implementation, parents of participating pupils were sent a text message containing a link to a short survey which asked parents for their views on the intervention.

Data collection consisted of three elements

1. Number of expressions of interest at RSE training days, invitation letters sent, responses received, telephone calls made, e-mails sent, participants declining participation and participants agreeing to take part in the study were all recorded in a database to allow the calculation of recruitment and retention rates and calculation of time taken for participants.

2. Time spent by the research team on recruitment activities, including delivery of recruitment presentations, preparation and posting of invitation letters, making follow-up telephone calls and sending e-mails to obtain expressions of interest, and arranging and conducting introductory meetings were recorded in researcher activity logs and used to estimate total time spent on recruitment activity.

3. Qualitative data were obtained from a combination of field notes, records of e-mail communication and contact notes following telephone conversations with participants who declined to participate, documented meetings with steering group members, and semistructured interviews with principals, VPs, teachers and parents recruited to the study.

Data analysis

To calculate recruitment and/or retention rates, a percentage was derived from the total number of invitations sent by school management type and the resulting number of schools, pupils and parents recruited.

To calculate the time associated with the various recruitment strategies and overall time spent on recruitment and retention activities, estimated time spent was totalled and divided by the number of schools, pupils and parents successfully recruited. This resulted in the total amount of time spent by the research team per school randomised.

Qualitative data were organised using NVivo 10 (QSR International, Warrington, UK) and analysed using a form of thematic analysis based on the six steps proposed by Braun and Clarke. 116 This involved moving between inductively derived codes emerging from the data and searching for data on pre-defined themes outlined in our topic guides. These inductively and deductively derived codes were analysed to form overarching themes.

Randomisation and allocation

Schools were the unit of randomisation, with four randomly allocated to the intervention group and four to the control group. After baseline data collection, schools were randomised as pairs to ensure that one of each type was in the intervention arm. The schools were grouped into four pairs based on their school management type: secondary schools in deprived areas, Catholic schools, grammar schools and other types. This process was conducted by the Northern Ireland Clinical Trials Unit (NICTU), a UK Clinical Research Collaboration-registered clinical trials unit, overseen by the trial statistician, using a method of simple randomisation completed using NQuery version 3.0 software (Statsols, Cork, Ireland).

Recruitment of schools

In October 2014, following unsuccessful attempts to recruit a second RC grammar school, recruitment targets were revised to maintain the diversity of schools included in the study but to broaden the scope to include RC secondary schools that, despite being RC schools, are not as academically driven as RC grammar schools. The revised target was eight schools of the following types: two secondary schools of any management type in deprived areas, two controlled secondary schools, two RC maintained schools and two grammar schools.

A total of eight schools were recruited (see Figure 4). The over-recruitment of one additional school was the result of the yield from the final attempt to recruit RC schools of any type following difficulties recruiting RC grammar schools.

Invitations were sent to a number of RC maintained schools and two responded on the same day with a decision to take part. It seemed unwise, and possibly unethical, to contact schools about a study and not invite them to participate once they had expressed an interest in doing so. Recruitment of schools via RSE training events was by far the most successful strategy, leading to 70% of schools invited by this method (n = 7) being randomised into the study. Invitation following introduction by a steering or advisory group member led to the recruitment of one additional school (a 20% recruitment rate). Cold-call invitation was unsuccessful, with all six schools approached declining to take part.

Targets for recruiting schools of different socioeconomic and religious types were met, but required more intensive work to reach particular school types and adjustments to initial recruitment specifications in order to achieve the recruitment target within the specified timeframe. As illustrated in Table 19, our main challenge was in recruiting RC grammar schools (i.e. RC schools that use academic ability to select pupils) and voluntary other managed grammar schools (i.e. non-RC Church or privately funded schools that use academic ability to select pupils). Although the overall school recruitment rate was 38% (100% for integrated, controlled grammar and controlled secondary schools), the recruitment rate for RC maintained secondary, RC grammar and voluntary other managed grammars was 67%, 13% and 17%, respectively.

Between June and October 2014 one of eight RC grammar schools approached and one of six voluntary other managed schools approached was recruited. Faced with a recruitment deadline of the end of October it was decided to revise initial recruitment targets to recruit RC schools of any type rather than just RC grammars. One RC grammar and one RC maintained secondary in a deprived area had already been recruited, and so it was decided to recruit one additional RC school of any type in a non-deprived area, which could be paired with the RC grammar for randomisation. This was achieved by inviting schools from the list of those attending the RSE training events.

As outlined in Table 20, a total of 13 schools approached declined to participate in the study. All but one of these schools made this decision prior to meeting with the trial manager. The most common reason for non-participation (n = 4) was that having considered the information sheets, school management and/or the teacher responsible for delivering LLW considered that there was no time available within the current curriculum at Year 11 to deliver the intervention and take part in the research. Three schools also reported that they were already involved in another research project and could not accommodate a further study at that time. Two other schools indicated that owing to staff changes (maternity leave and a change in the LLW team) they did not think it was an appropriate time to take part. Two schools declined to take part without providing a reason. The PC VP and Head of RE of one RC grammar school met with the trial manager prior to making their decision as they were concerned that the intervention might not be compatible with their school ethos. Having expressed positive views that they felt it was in line with their ethos at the meeting, they later declined to take part having consulted with the school Chaplin who felt that the intervention was not in fact compatible with the school ethos. Although a number of schools mentioned at first contact that they did not have any issues with teenage pregnancy in their school, only one cited this as their primary reason for non-participation.

The schools that did participate in the research reported doing so for a number of reasons, primary among which was their desire to improve RSE provision for their pupils and their belief that the If I Were Jack resource would be of benefit to their pupils. Schools in deprived areas were particularly interested in taking part because of their current and past experiences of teenage pregnancy in the community.

Principals indicated that they were approached several times a week to take part in research, and, although all appreciated the potential benefits of research they were often too busy to respond to such requests. All agreed that direct face-to-face contact with them or a subject teacher was the optimal strategy:

If [an invitation] just comes as an e-mail there is a fair chance it’ll end up in the bin […] If it’s preceded by a phone call there is probably a better chance [it’ll be considered].

If you just send something in and it comes to me, it could go over my head. If you send it to my teacher or you engage somebody or you phone the school up and say ‘Could I come in to chat to the Principal?’ or ‘Could I come in to speak to somebody?’ rather than sending a letter, I think that’s the way to hook people in . . . Personal contact is key.

One principal summed up how she went about considering research participation:

We would regularly get asked for research, maybe three or four times a week, so the volume of research out there, I think, is immense, and the time restraints that that places, depending on the type of research and what the investigator is looking for, specifically, can impact whole school activity, so we need to be very careful in terms of what we select and the benefits that are going to be achieved to the students. But then, also, I do think we’ve a wider responsibility as educators to look outside the students sometimes and possibly to learn from the practice of research and contribute to it on a more holistic educational basis. So, it’s, for me, it’s trying to balance up the benefits of that.

Principals agreed that a number of considerations were important in helping them to make a decision about whether or not to take part. Central among these were the potential burden on pupils, especially those in exam classes, the burden on teachers and the credibility of the project:

Firstly, if there’s any credibility in what’s being done. Secondly, if it can be fitted in with minimum disruption. If it’s going to affect exam classes there’s almost no chance.

Principals were asked what kind of things do they consider before deciding whether or not to take part.

Well, the first thing I would look at is, is it in an area of interest that we can contribute to that is very specific to my school? Is it an area that the students can benefit from? So, with the Jack project, I felt it was a wee bit innovative and that there were potential useful resources from it. The [other] things that I look at are what are the time constraints and the commitment for the students and the staff?

Several principals noted that ‘Area Learning Communities’ or other head teacher forums such as the ‘Catholic Heads Association’, which were attended regularly by principals, might be possible venues for presenting research and recruiting schools. Others noted that buy-in from teachers was a must and that often a good way of reaching the principal is through a subject teacher.

So, sometimes, if a specific teacher in an area is going to drive it and sees it as very beneficial, is going to inform practice and that, they would come to me and they would sell the research if they are going to lead it. The difficulty for me is sometimes, if there’s not an original buy-in from staff, it can prove very difficult to manage . . . so if you get the hook through somebody else, if it’s a subject-specific area, that could be a driver for running it as well.

Sometimes I get e-mails from folks I’ve never met or have no connection with, so they haven’t really much hope of being successful. Mmm, it depends on how the approach is made and on the reason behind it [the research].

I suppose for me the best time to get me is either early in the morning, between eight and quarter to nine or later in the afternoon, sort of, between half past four and six. That would be times when I’m not out and about or engaging with kids or staff.

Principals especially did not want pupils in significant exam years to be disrupted with external research studies.

If you’re wanting to do it with Year 11 to 14, especially Year 11 and 12, there’s hardly a week goes by where they’re not involved in some kind of controlled assessment . . . I won’t take kids out of English and Maths or maybe even other GCSE subjects to do [research].

Principals also had an especially strong message for researchers in terms of taking some of the responsibility for the organisation of the research within the school, especially in terms of conducting the data collection.

[I would immediately decline to take part in research projects] that are going to be very time-consuming . . . or, where projects put the onus on the school – ‘Could you get us a group? Here’s the list – could you go away and do it, and when you’re finished, could you bring it back to us and I’ll pick it up from you?’ and you’re like ‘What?!’

Principals were asked how important monetary incentives, such as getting the If I Were Jack resource to use at the end might be in a future trial. They agreed that monetary incentives would be an important facilitator of participation:

Very important. They would be the enticement to get you involved. It sounds very mercenary and I don’t mean to be mercenary in this day and age, but if I’m going to . . . if it’s going to be time and energy with students and staff and commitment, and if the benefits are not solely related to students and staff and outcomes, then there’s got to be a reason why you would do it.

One principal in the control group suggested that having access to the resource was a more important incentive than money to them:

We now have access to the Jack resources. From our point of view that is probably more valuable . . . that’s probably a better incentive than money.

Reasons principals chose to participate are detailed in Table 21.

Schools took between 1 and 19 days (mean = 6 days) to confirm their decision. Two schools did not respond with a decision despite several attempts at contact. Allowing 1 week following postage of the introductory letter, this means that it took an average of 2 weeks to receive a participation decision from schools.

For an overview of the number of contacts made, outcome and time take taken to decide by schools approached see Appendix 8.

Recruitment of teachers

A total of 40 teachers were recruited across the eight participating schools (see Figure 4). One teacher later opted out of implementing the intervention owing to an extended period of sick leave. For school personnel involvement in a semistructured interview or focus group discussion regarding their experiences of taking part in the research (comparison and intervention) following the final follow-up see Chapter 7.

Recruitment of pupils, pupil opt-out and retention

Recruitment of 831 pupils to the study at baseline represents an overall pupil recruitment rate of 80.9%. Parental withdrawal of consent accounted for 6.8% of the loss (n = 70) and pupil opt-out for 3.1% (n = 32). Pupil absence or unavailability at baseline in combination with pupils who chose not to complete the questionnaire or who were unable to because they had not returned consent forms to the research team accounted for the remaining 9% of loss (n = 94). In one school a number of pupils indicated that they were unable to participate because they did not have sufficient English to understand the consent form.

In two schools, teachers had forgotten to distribute the information sheets to pupils. They also had not given them any indication that the school was taking part in the research or that data collection was scheduled for that day. Although the researchers made an effort to spend a little more time explaining the research in these two schools, given time constraints there was a limit to the amount of additional information that could be imparted to pupils at this juncture. There was a noticeable difference among pupils who had prior warning and a chance to read the information sheets, both in their questions to the researcher and their expressed satisfaction with taking part.

As detailed in Table 22, pupil participation also varied by school type, with the highest recruitment rates in the three RC schools (> 90%) and the controlled secondary school (92%). The lowest rates were in the controlled grammar school (55%) and one of the integrated schools (62%), both of which had high pupil absentee rates on the day of baseline data collection. The controlled grammar schools’ absentee numbers at baseline (n = 40) were accounted for by the fact that one class group were undergoing an observation for teacher training purposes and were not permitted to leave their class to take part, while the remaining pupils had to attend a sports event. In the integrated school the absences at baseline (n = 25) were accounted for by the fact that data collection took part first thing on a Monday morning when absences were usually higher. Trial champions were provided with questionnaires and consent forms for absent pupils and asked to return them on several occasions before the cut-off point of 23 January. Teachers reported an inability to find time to administer the questionnaires to pupils as the reason for not returning them.

Pupil opt-out rates ranged from 0% to 13% across the schools, with the highest rates in both integrated schools (Table 22). Pupils were not asked to give a reason for non-participation. Observations indicated that pupil opt-outs tended to occur in friendship groups (i.e. it was rare for a single young person to opt out; instead, friendship pairs or groups tended to opt out together). In the school with the highest opt-out rate (n = 17), pupils had not been provided with the information sheet prior to the study and baseline data collection was scheduled by teachers to immediately follow an exam in a large exam hall, which made providing an overview of the study at a sufficiently loud volume for all students to hear very difficult. In the words of one of the attending teachers, decisions to opt out ‘spread like wildfire’.

In another of the schools a teacher told the trial manager after the baseline data collection session that she had had ‘around 25’ pupils outside the hall prior to the research team’s arrival saying that they did not want to take part. The teacher asked them to listen to the explanation of the study prior to deciding. Two pupils opted out following the overview of the study.

Pupil retention rates per school are detailed in Table 23.

In the intervention cluster pupil retention was 74.0% by follow-up 1 and 91.4% by follow-up 2. In the comparison cluster, pupil retention was 96.6% by follow-up 1 and 94.9% by follow-up 2. Pupil retention at trial level was 85.2% by follow-up 1 and 93.1% by follow-up 2. The lower follow-up at follow-up 1 was mostly because it occurred too near the holidays when pupils’ attendance had already started to drop off.

We originally aimed to recruit 80% of eligible children into the research trial (based on pupil ‘opt-in’ and allowing for ‘opt-out’ parental consent). Of the eight schools recruited, there were 1027 eligible pupils in Year 11 who could participate at baseline data collection. (For an overview of pupils responding to the questionnaire at each data collection time point, see Tables 28 and 29.) The proportion of eligible pupils (at baseline) who provided questionnaire data was 81% (for both the intervention and control group). At follow-up 1 data collection (allowing for loss of eligible pupils who had moved school or were now being home-schooled), these figures were 60% and 78%, respectively (the reason for this lower return is because we left data collection too close to the summer holidays and many of the pupils were already absent from class). At follow-up 2 data collection (allowing for loss of eligible pupils who had moved school), these figures were 76% and 77%, respectively. At each data collection time point (baseline, follow-up 1 and 2) 97% of pupils who completed the questionnaire answered the questions relating to the main composite primary outcome measure (see Chapter 6). At baseline, this involved 803 pupils and at follow-up 2 745 pupils (representing a reduction in numbers of 7%). Of the 803 pupils who answered the main composite primary outcome measure at baseline, 629 (22% drop-out) went on to answer the same questions at follow-up 1 data collection and 699 (13% drop-out) at follow-up 2 data collection (Tables 24 and 25).

The trial ended following successful completion of follow-up data collection with all pupils by 30 October 2015.

Recruitment of parents and parental opt-out

Recruitment of parents and guardians to attend the school-facilitated information and discussion session was extremely low, with an overall recruitment rate of 2.3% (i.e. 9 mothers and one father representing 10 different pupils) assuming potential recruitment of one representative parent or guardian for each eligible pupil in the three intervention schools that held the session (n = 428). All three schools chose to run the event in the late afternoon (two schools started at 15:30 and one at 16:30). Reasons for this early start included difficulties keeping the school open late for such an event and a lack of desire on the part of the teachers to stay late themselves. One school did not hold the parents event because the teacher who was responsible for organising and delivering it went on sick leave and failed to ‘catch up’ on organising it on her return 1 week later.

In the three schools that held the event, parental attendance was extremely low (between one and six parents/guardians attended). Based on experiences of similar events, teachers in two of the schools anticipated that attendance would be very low. One teacher said that ‘our parents are not very supportive of their children when it comes to things like this’.

In response to poor attendance at these sessions, recruitment targets were revised. A 6-minute information video was recorded which contained the key points from the information session. This was posted on a locked YouTube channel and parents were sent a link to access it via e-mail or text message. From parents who attended these sessions we aimed to recruit a total of 25 parents to take part in a focus group interview with a researcher. In order to capture the views of those who did not attend the information session or focus groups all parents/guardians were sent a link to a short online survey. It was estimated that 40% of parents would respond to this survey. Parents of pupils in the comparison group were not recruited to the study.

The information session video on the YouTube channel was viewed by 45 parents (12.5%, assuming potential recruitment of one representative parent or guardian for each eligible pupil in the three intervention schools alerted to the online video). Owing to an administrative error, parents in one school did not receive the information video but only a printed copy of the information sheet. This school (school ID 17) had the highest number of parental opt-outs.

In total, 29 parents responded to the parents’ survey, representing an 8.3% response rate [all parents whose children were eligible to receive the intervention (eligible pupils minus parental and pupil opt-outs) received the text alerting them to the survey]. In school 17 this figure was reduced to n = 21, owing to only one class being able to receive the intervention as a result of teacher sick leave. Of the responding parents, 22 indicated that they had not attended the information and discussion session (see Chapter 7).

See Chapter 7 for more information provided by parents who took part in a semistructured interview or focus group discussion following one of the information sessions.

As indicated in Table 25, parental withdrawal of consent rates ranged from 2% to 11% with the lowest rates in one of the RC maintained secondary schools and the controlled secondary (n = 2) and the highest in the voluntary other managed grammar (n = 18). Most parents did not include a reason, but of those who did (n = 16) reasons provided were variations of:

my child does not want to take part (n = 10)

I do not want my child to take part (n = 2)

my child wants to focus on his/her exam subjects (n = 1)

my child is busy with extra-curricular activities (n = 2)

my child has a learning disability so I do not think it is appropriate for him to take part (n = 1)

At the time of data collection teachers in four of the schools reported that some parents had been in contact with them because they were confused about whether or not to send back the withdrawal of consent form if they were happy for their child to take part. In the voluntary RC grammar school, five parents who had initially opted out contacted the school after their child had received the information sheet to say that they were now happy for their child to take part. Teachers theorised that this may have been at the request of the child but one teacher reported that a parent to whom she had spoken had previously thought that her child was being ‘singled out’ for participation in the study and when she realised everyone would be taking part she changed her mind.

Baseline characteristics of participating pupils

In total, 831 Year 11 pupils aged between 14 and 16 years took part in the baseline evaluation. Just < 50% (49.5%) were female and just > 50% (50.5%) were male. Overall, 420 pupils were randomly allocated to the intervention group and 411 to the control group (see Chapter 5 for the randomisation process). Table 26 shows the breakdown of the sample by sex and group allocation.

Table 27 shows the baseline demographic characteristics of the sample. This shows that the mean age was 14.4 years (SE = 0.019) in both the intervention and control groups. The ethnicity of the majority of participants was white (British/Irish) and the next largest group was Chinese (8% in total). For those whose ethnicity fell into the ‘other’ category, the most commonly cited ethnicity was Polish. In terms of religion, a wide range of religions was listed, with the most common being Christian, Atheist and Mormon. The most common religion in the sample was Catholic (n = 422, 50.8%). The second most common was Protestant (n = 307, 36.9%). The majority (41.6%) of pupils in the intervention group identified themselves as ‘not very religious’, whereas the control group (46.8%) most commonly identified themselves as ‘fairly religious’. Most of the pupils were not in receipt of free school meals; however, a slightly larger proportion in intervention schools were obtaining free school meals (16.4% vs. 10.6% in control schools). Here, we also present two measures of SES, one of which is based on parental highest level of education and one of which is based on children’s future educational aspirations (for further discussion on the issues encountered relating to a measure of SES, see Chapter 4).

Stopping rules/discontinuation criteria

The following were the trial stopping/discontinuation criteria as documented in the study protocol.

1. We shall monitor for harm by examining outcome data as soon as possible after each follow-up. If deterioration in scores indicated in the outcome measures in the intervention group are not matched by deterioration in scores in the control group, we would consider stopping the intervention.

Defining primary and secondary outcomes

Secondary outcomes

The following secondary outcomes were drawn from the theoretical framework underpinning the intervention (theory of planned behaviour; see Chapter 3). To measure these outcomes we used some existing measures with robust reliability and validity (see Appendix 1). The baseline questionnaire is included in Appendix 2.

The first outcome chosen in this questionnaire was Knowledge. The purpose of this was to determine if pupils had increased knowledge about UTP. Knowledge scores represented the total score on the Knowledge scale (Q.15) and the sum of correct answers to the following questionnaire items:

Q.16 What is the legal age of consent (the age at which you can legally first have sex) in NI?

Q.17 Can you think of any health services, clinics or places you could visit if you wanted to get the following? Free Contraception; advice on sexual health (e.g. pregnancy, STIs).

Q.18 Can you think of any websites which provide information on sexual health, relationships and pregnancy?

The second outcome, Attitudes, aimed to determine if pupils:

• had increased awareness that men and women have roles and responsibilities in avoiding UTP;

• hold less stereotypical gender-related attitudes and have increased perception that most peers are not sexually active and use contraception when they are;

• have increased perception that both parents (parent/child communication) and peers support abstinence and/or use of contraception.

Scores resulted from the incorporation of one questionnaire item (Q.19; Box 2), and the Comfort Communicating about Pregnancy scale (Q.20), the Comfort Communicating about Contraception scale (Q.20), the Male Role Attitudes scale (Q.22).

BOX 2 - Question 19: Whose responsibility is it to prevent unintended pregnancy?

Put an X on the line below to show how much responsibility you think boys and girls should have. For example, if you think girls should have more responsibility than boys, put an X on the line closer to girl; if you think boys should have more responsibility than girls, put an X on the line closer to boy; If you think boys and girls should have equal responsibility, put an X in the middle of the line.

The third outcome, Skills, aimed to determine if pupils had increased self-efficacy in communicating about UTP and avoiding UTP. Scores resulted from the incorporation of the Sexual Self-Efficacy scale (Q.21) the Sexual Socialisation scale (Q.23), the Sexual Socialisation Peer subscale (Q.23) and Sexual Socialisation Parent subscale (Q.23).

The final outcome, Intentions, aimed to determine if pupils had higher anticipated regret about UTP and increased strength of intention to avoid UTP. For this construct, the questionnaire incorporated a measure of intention to avoid UTP, developed as part of the research process (see Chapter 4). Scores incorporated the Intentions scale, Intentions Self-Efficacy subscale, Intentions Attitudes subscale and the Intentions Norms subscale.

Descriptive statistics for the secondary outcomes are presented below (Tables 34–36). As the study was not powered to detect statistically significant differences between groups, we have not conducted mediation analysis or logistic regression to test the impact of these variables on the primary outcome measure.

Compared with pupils in the control group, pupils in the intervention group had more of an increase in knowledge (Knowledge scale score) about UTP at follow-up 1 [0.21 (95% CI –0.14 to 0.55)] and at follow-up 2 [0.15 (95% CI –0.19 to 0.48)]. When asked about the legal age of consent (the age at which you can legally first have sex) in NI, 65.5% of respondents answered with the correct age of 16 years. Thirty-eight per cent of pupils correctly identified a health service, clinic or place where one could obtain free contraception [e.g. GP surgeries, Brook clinics (Brook Young People, Liverpool, UK)] and 40% were able to name a source of advice on sexual health (e.g. GP surgeries, sexual health clinics, internet). Pupils listed a wide variety of websites that provide information on sexual health, relationships and pregnancy. Eighty-nine per cent incorrectly (as determined by the research team) interpreted particular websites (e.g. named pornography sites) as sources of such information and 10.6% correctly identified websites that do provide such information.

At baseline, when asked whose responsibility it is to prevent UTP, 75.5% of pupils who responded to Q.19 attributed a 50 : 50 ratio of responsibility to boys and to girls. Less than 1% (0.6%) of responding pupils answered that it was only the boy’s responsibility to prevent UTP, and 0.7% answered that it was only the girl’s responsibility. At follow-up 1, 80.7% of pupils attributed a 50 : 50 ratio of responsibility to boys and girls, with 3.8% indicating that it was only the boy’s responsibility to prevent UTP and 0.2% that it was only the girl’s responsibility. At follow-up 2, 83.4% of pupils attributed a 50 : 50 ratio of responsibility to boys and girls, with 0.1% of responding pupils indicating that they thought it was only the boy’s or only the girl’s responsibility to prevent UTP. Both the intervention and control group demonstrated an improvement towards attributing an equal 50 : 50 ratio of responsibility over time, although the improvement was more pronounced in the intervention group (13.3% vs. 10.4% in the control group).

Scores on the Male Roles Attitudes scale at follow-up 1 indicated that pupils showed a decrease in stereotypical gender attitudes to the roles of males and females in society (e.g. men work and women care). This decrease in awareness was more evident in the intervention group [–0.55 (95% CI –1.26 to 0.16)]. At follow-up 2 pupils also had a decreased awareness of gender-specific roles, but on this occasion the decrease was more evident in the control group [0.87 (95% CI 0.32 to 1.42)] than in the intervention group, although see below that there was a sustained decrease of stereotypical gender attitudes among males in the intervention group.

Increased scores on the Comfort Communicating about Pregnancy scale indicated that more pupils in the intervention group than in the control group were comfortable communicating with parents and peers about avoiding UTP at follow-up 1 [0.06 (95% CI –0.21 to 0.34)] and follow-up 2 [–0.04 (95% CI –0.28 to 0.20)]. Similarly, scores on the Comfort Communicating about Contraception scale indicated that at follow-up 1 more pupils in the intervention group were more comfortable communicating with parents and peers about the use of contraception [0.08 (95% CI –0.29 to 0.45)]. However, at follow-up 2 more pupils in the control group were more comfortable communicating with parents and peers about the use of contraception [–0.03 (95% CI –0.38 to 0.32)].

In relation to sexual self-efficacy, pupils in the intervention group scored lower than those in the control group at both follow-up 1 [–0.04 (95% CI –0.26 to 0.19)] and follow-up 2 [–0.01 (95% CI –0.24 to 0.22)]. Scores on the Sexual Socialisation scale indicated that pupils scored better in the intervention group only at follow-up 2 [0.84 (95% CI –0.43 to 2.10)], as was the case with each Sexual Socialisation subscale: peer [0.23 (95% CI –0.70 to 1.16)]; parent [0.61 (95% CI –0.001 to 1.22)].

Looking at the Intentions scale as a whole, pupils scored higher in the control group at both follow-up 1 [–0.51 (95% CI –1.85 to 0.82)] and follow-up 2 [–0.56 (95% CI –1.78 to 0.65)]. When exploring the Intentions subscales, a difference was evident. Pupils in the intervention group scored higher at both follow-up 1 and at follow-up 2 for both the Intentions Self-Efficacy subscale [follow-up 1 0.26 (95% CI –0.33 to 0.84); follow-up 2 [0.11 (95% CI –0.45 to 0.67)] and the Intentions Attitudes subscale [follow-up 1 0.07 (95% CI –0.32 to 0.46); follow-up 2 [0.08 (95% CI –0.25 to 0.41)], whereas pupils in the control group scored higher at both follow-up 1 [–0.42 (95% CI –1.15 to 0.31)] and follow-up 2 [–0.62 (95% CI –1.29 to 0.06)] in relation to the Intentions Norms subscale.

Differences among male and female pupils in relation to secondary outcomes were also examined. Female pupils in the intervention group scored notably higher than their male counterparts in the Knowledge scale at both follow-up 1 and 2 [follow-up 1: female 0.36 (95% CI –0.14 to 0.87), male 0.09 (95% CI –0.39 to 0.56); follow-up 2: female 0.39 (95% CI –0.08 to 0.85), male –0.07 (95% CI –0.55 to 0.41)], the Intentions scale at follow-up 1 [follow-up 1: female 0.26 (95% CI –1.63 to 2.16), male –1.29 (95% CI –3.19 to 0.60)], the Intentions Self-Efficacy subscale [follow-up 1: female 0.55 (95% CI –0.22 to 1.32), male –0.05 (95% CI –0.92 to 0.82); follow-up 2: female 0.58 (95% CI –0.16 to 1.31), Male –0.37 (95% CI –1.20 to 0.47)] at both follow-up 1 and 2 and the Attitudes subscale [follow-up 1: female 0.49 (95% CI –0.11 to 1.09), male –0.32 (95% CI –0.84 to 0.19); follow-up 2: female 0.10 (95% CI –0.37 to 0.58), Male 0.06 (95% CI –0.41 to 0.53)] at both follow-up 1 and 2. Male pupils in the intervention group scored significantly higher on the Male Role Attitudes scale (p = 0.004) than male pupils in the control group at follow-up 2 and notably higher than their female counterparts in the intervention group [male 1.27 (95% CI 0.41 to 2.13), female 0.47 (95% CI –0.22 to 1.17)]. This was not evident at follow-up 1.

The total number of pupil participants who responded to the questionnaire at both baseline and follow-up 1 data collection was 666, and at both baseline and follow-up 2 data collection was 744 (see Chapter 5). The n values in the above table refer to the total number of participants who answered all the questionnaire items relative to the associated outcome measure at baseline, follow-up 1 and follow-up 2. Missing data can be deduced from this.

Secondary behavioural outcomes relating to the health economics evaluation model

Overall pupil responses (by treatment arm) to questions relating to secondary behavioural outcomes are detailed below in Table 37.

In the present feasibility study, respondents reported on having had STIs within the 6 months prior to baseline data collection, since follow-up 1 data collection, or since follow-up 2 data collection (Table 38). This question was poorly responded to, with 93.4% missing data at baseline, 91.3% missing data at follow-up 1 and 88.7% missing data at follow-up 2.

Reported UTPs were very low. At baseline data collection, only one respondent (5.9%) in the control arm reported having intentionally gotten a girl pregnant within 1 month prior to baseline data collection, and six (23.1%) and two (11.8%) in the intervention and control arms, respectively, reported that they did not know if they had ever gotten a girl pregnant. Only one respondent (12.5%) in the intervention group reported having been unintentionally pregnant between 10 and 12 months ago. At follow-up 1 data collection, three (12%) respondents in the control arm and one (7.1%) in the intervention arm reported having gotten a girl pregnant without intent, while one (7.1%) and three (12.0%) in the intervention and control arms, respectively, reported that they did not know if they had ever gotten a girl pregnant. One (8.3%) female respondent reported not knowing if she had ever been pregnant at follow-up 1 data collection. At follow-up 2 data collection, two (6.9%) respondents in the intervention group reported having gotten a girl pregnant (one intentional and one unintentional) and one respondent (4.2%) in the control group reported having gotten a girl pregnant (also unintentional), while two respondents in the intervention group (6.9%) and one in the control group (4.2%) did not know if they had. One (3.7%) female respondent reported being unintentionally pregnant at follow-up 2 data collection and one (3.7%) reported not knowing if she had ever been pregnant.

At baseline data collection seven (21.9%) respondents in the intervention arm and seven (28.0%) in the control arm reported having had sex when they did not want to within the past 6 months. These figures were seven (14.9%) and four (10.3%), respectively, at follow-up 1, and two (8.3%) and six (16.2%), respectively, at follow-up 2.

Aims and objectives

The aims of the process evaluation were to (1) assess the ways in which the intervention was implemented; (2) assess how it was received and how it could be further improved; and (3) collect details on the context of participating schools that might help to explain differences in perceptions of feasibility and acceptability.

The objectives of the qualitative component were to:

• record the current policies on, and provision of, RSE in schools and to assess what RSE was delivered in schools during the feasibility trial

• inquire into teachers’ experiences of using the intervention with pupils and parents/guardians (intervention group)

• record pupils’ experiences of using the intervention at school and home (intervention group)

• gain further information on implementation fidelity and teachers’, pupils’ and parents/guardians’ reactions through researcher observation

• inquire into parents’/guardians’ reactions to their children using the intervention, as well as the barriers to and facilitators of their own participation

• record participants’ experiences of taking part in the trial

• obtain teachers’ views on the quality of the training session offered on the intervention

• test the face validity of the quantitative instruments among pupils in the control group by conducting cognitive interviews on the topic of the questionnaire (see Chapter 4)

• record experiences of recruiting schools, also paying attention to refusals (see Chapter 5)

• record schools’ reactions to being assigned to control or intervention group (see Chapter 5 for randomisation and allocation).

Intervention schools

• RSE provision: semistructured interviews with teachers (RSE-coordinators, heads of Year 11, PC VPs)

  • the nature, quantity and quality of current RSE

  • facilitators of and barriers to teaching RSE.

• Research participation: semistructured interviews with principals/heads of Year 11

  • perceptions of the conduct of research in schools in general and experiences of participation in the trial

  • barriers to and facilitators of research participation

  • engaging parents.

• Feasibility and acceptability of the intervention and implementation process and taking part in the trial: individual interviews and focus group discussions with all teachers who delivered the intervention, focus group discussions with a sample of pupils who received the intervention and a sample of parents/guardians who attended the information and discussion sessions, comment sheets provided after last follow-up and an online questionnaire sent to all parents

  • strengths and weaknesses (perceptions) of the intervention materials

  • experiences of delivering/receiving the intervention

  • experiences of the teacher training session

  • experiences of facilitating the parents/guardians’ session

  • experiences of taking part in the trial

  • experiences of parent/pupil homework activity

  • suggested improvements for future trials

  • impressions of the information and discussion sessions

  • barriers to and facilitators of engaging parents in RSE

Control schools

• RSE provision and participation in the trial: semistructured interviews with teachers (RSE co-ordinators, heads of Year 11)

  • the nature, quantity and quality of current RSE

  • facilitators of and barriers to teaching RSE in school

  • experiences of taking part in the trial.

• Research participation: semistructured interviews with school principals/heads of Year 11

  • perceptions of the conduct of research in schools in general and experiences of participation in the trial

  • barriers to and facilitators of research participation

  • engaging parents.

• Feasibility and acceptability of the questionnaire and taking part in the trial: focus group cognitive interviews with a sample of pupils from two of the four schools in the control group and comment sheets provided after last follow-up

  • overall impressions of the questionnaires

  • focus on questions/words/formatting that was hard/easy to understand

  • experiences of taking part in the trial and suggested improvements for future trials.

Sample

A total of 13 teachers across the four intervention schools delivered If I Were Jack. A total of 26 school personnel, 51 pupils and 10 parents took part in interviews or focus group discussions (across both the intervention and control schools).

The context of participating schools

As indicated in Table 39, of the eight schools that took part in the study, seven were located in urban areas and one in a rural area; five were secondary schools and three were grammar schools, with the former enrolling pupils regardless of academic ability and the latter accepting pupils with the highest marks in entrance exams. In terms of management type, three were controlled schools (i.e. managed by the Education Authority), two were RC maintained schools, two were voluntary (i.e. managed by a board of trustees) and one was grant maintained. Three schools were in affluent areas, with the percentage of pupils eligible for free school meals ranging from 4% to 15%, and five were in deprived areas, with the percentage of pupils eligible for free school meals ranging from 32% to 48%.

Relationships and sexuality education in participating schools

Three overarching themes emerged relating to the provision of RSE: (1) existing provision – content, frequency, co-ordination and delivery; (2) attitudes towards RSE; and (3) obstacles to and facilitators of quality RSE. In general, it emerged that provision was extremely variable across the eight schools and, in most cases, not seen as a priority subject, although those staff members tasked with co-ordinating RSE were all motivated to prompt changes in this regard. In some schools there was a clear sense of a considered, planned, well-structured RSE programme, while in others, provision was so haphazard that even the RSE team had difficulties articulating the content and frequency of provision. In most of the schools pupils remembered learning about reproduction in Biology but could recount few, if any, other instances in which teenage pregnancy was mentioned.

Acceptability of the intervention

Although some pupils felt that the whole idea of RSE was embarrassing and awkward in relation to teachers and fellow classmates in general, when asked about their impressions of the intervention overall, most young people and teachers in intervention schools were positive and provided suggestions that they felt would help improve the various components.

Strengths and limitations of the core intervention (film and accompanying activities)

Teachers agreed that pupils enjoyed and easily engaged with the IVD and, although there were variations across class groups, pupils interacted well with most of the accompanying activities. All participants agreed that the overall concept of the intervention was acceptable and, allowing for minor changes to some of the activities, most said that they would use it again.

Positive aspects included the fact that it was locally set, realistic, relatable, interesting, informative and helpful.

I think, as resources go, it’s one of the best I’ve seen out there because anything we have is quite ancient, dated or from, like . . . England, Australia . . . We’re very lucky to have a resource that is very current and set, you know, in the local area.

I think it’s actually a very good resource, and I think the fact that it’s coming from a boy’s point of view is unique and the pupils took to that well, you know, because they’re always hearing about teenage pregnancy always more geared from the girl’s point of view. So, I think they found that maybe kind of . . . made them more interested in it, just from the very start.

The most common complaint about the IVD from pupils, which was supported by some teachers, was that there were too many questions embedded within it, too many stop-starts and that some of the questions seemed repetitive or did not include the specific option that the pupil wanted to pick.

It would have been nice to have the whole film and then to go back to run through it, without the questions, because the questions, you were kind of . . . it took a long while just to get through them.

Some pupils thought that the film was outdated, a bit ‘cheesy’ and the behaviour of the actors within a bit ‘childish’ and in need of updating, whereas others said that updating the film was not necessary because, when they compared it with other resources they used in school, it seemed very contemporary. Some pupils attributed the outdated feel to the assumption that young people had not been involved in writing the script.

Although participants did not spontaneously mention the focus on the male protagonist’s perspective as a limitation, when prompted, all agreed that more focus on the female protagonist’s perspective would perhaps be helpful. One suggestion was that an activity focused on the girl’s perspective might be beneficial. Male and female participants also agreed, however, that the focus on the boy’s perspective was both acceptable and relevant to them.

It would be better if you could, like, go from, like, each side, each perspective, if they took, like, the girl and the boy. It would be better because you’d get a rounded perspective of what it would be like.

Some pupils were disappointed by the fact that the story did not have an ending and saw this as one of the biggest weaknesses of the film. Teachers reaffirmed this feeling among pupils.

The only thing that really annoyed me about it is they don’t actually tell you what she does with the baby at the end. Like everybody wanted to know!

Most teachers agreed that providing pupils with a worksheet to complete the ‘Pause: Fast Forward: Rewind’ activity worked best. Pupils agreed that they would prefer less written and more interactive or discussion activities, although some did say that they enjoyed the opportunity afforded them by the worksheets to think things out in private before discussion.

Regarding homework, the majority said that homework was a bad idea, mainly because they did not usually have homework for LLW, PC or RE, and teachers confirmed that homework was not a good idea because pupils resisted this as a result of not being accustomed to having homework in this context. Although some pupils said that they did the homework without difficulty, most said that they did not do it at all or that their teacher asked them to do it during the class period. Pupils from the grammar school weighed up the time involved in completing homework against other priorities.

Four of the 10 additional activities sparked particular conversation: (1) ‘If I had a baby’; (2) ‘Making a difficult decision’; (3) role-plays; and (4) the parents’ survey. There was general agreement that the ‘If I had a baby’ activity, which asks young people to compare their own schedule with that of a parent with a 6-month-old child, was enjoyable and for some even shocking.

Yeah. That was . . . that was probably one of the most effective because it was a real eye-opener. It was something that they talked about even after we’d done it, even the following week – ‘I couldn’t believe that . . .’ So, em, something like that, where there’s a comparison of their own life, I think really works.

In relation to the ‘Making a difficult decision’ activity, which asks young people to consider the pros and cons of keeping the baby, having the baby adopted or having an abortion for Jack, Emma and the baby, teachers noted, and many young people fed back, that they found the exercise difficult, particularly when trying to think of the pros of abortion for the baby.

I think they were embarrassed to say because this was like is there any pros for the baby with, for example, abortion. For some of them, they found it quite . . . well, they would say, ‘But how could there be a pro if they’re dead?’ and for them to say that out loud, some of them struggled to put that into words . . . It was as if we were nearly tricking them – ‘Is there a positive?!’

The role-play activity sparked a mixed reaction, with some young people saying that they loved it and others saying that it was ‘embarrassing’ and an opportunity for members of the class to ridicule those who ‘went to the top of the room’ to act out the parts. Although some teachers were adamant that their pupils would not engage in role-play, others felt that it was worth keeping in because some pupils enjoyed it and through watching others, those who will not participate can still benefit.

Finally, although some pupils appreciated the potential benefit, the parents’ survey was associated with predominantly negative views. The activity asked pupils to interview a parent/guardian (or other trusted adult) regarding their views of the If I Were Jack film. Although some young people said that they had never received the worksheet, others said that they consciously did not do it or ‘forgot’, with only a few girls admitting that they had completed it with their mother and one boy saying that he had done it with his older brother. The majority mentioned embarrassment and fear of awkwardness as primary reasons for avoiding the activity. In addition, many anticipated that bringing up such a subject had the potential to encourage further discussion from parents – the thought of which put off most pupils. For some pupils, the concern was less about embarrassment and more about their parents’ actual or perceived unwillingness to engage with them. Some pupils thought that it would better to do the activity with friends or a sibling.

I’m like close to my parents – I can tell them anything, but I wouldn’t do it. Like I don’t know . . . it’s weird. It’s weird talking to your parents, you know, especially when you’re like 15. Like sitting down and just be like, ‘Yeah, by the way, I learned about sex today’.

I got it, but my Mum said that she couldn’t fill it in because she would be hypocritical because she got pregnant at 17.

Teachers reported mixed views on the parent survey homework exercise, with many reporting that the majority of pupils chose to leave the material for the exercise behind and not even consider taking it home to discuss with parents. When asked if the teachers got a sense that this was the first time the pupils had discussed such matters, responses included:

Yes . . . One of the kids was like, ‘My dad was like really proud of me for going to him to tell him that this had happened! ‘And it was just like . . . because that had led into a conversation where his dad had said to him, ‘If this did happen to you, would you feel that you could come and ask me for advice and help?’ and he said, ‘Well, on the basis of this response, yes,’ and his dad was going, ‘Good, I’m really pleased – this is fantastic, because this is what I’m here for! OK, I’d kill you, but [laughing] . . . I would be angry at you, but it’s really good that you would come and speak to me about that’. So, there was a bit of sort of the kids going . . .’. My dad’s actually reasonable . . .’ do you know [laughing]?! There was some things like that, where they have this idea that their parents are terrifying, and actually, they just care, you know.

Regarding the controversial statements activity, teachers reported that most students enjoyed it:

Yeah, and then they got a little bit of feedback. But I just wandered around, without being too intrusive, letting them . . . It was interesting what they picked up on, what they chose, what was important . . . They got a laugh with some of them as well, you know, like ‘Should women shave their legs’ . . . I mean, they were really talking about it, instead of just sitting hoping that I wouldn’t come round. I was listening and they really were discussing, so that was good.

One teacher, however, did not do the controversial statements activity, as she felt that it would not work with her class group and was unrelated to the rest of the activities.

Teachers agreed the intervention could have been broader in scope.

I do think there could have been much more discussion on other things, you know, like if you do sleep around, there’s other things can happen, like STIs and all of that, and I really felt that was missing in it. I mean, I know you’re just talking about the consequences, but they’ve slept together, do you know, and they’re . . . so I just felt that we could have taken it a wee bit further in that regard.

Participants offered suggestions for minor changes to the intervention (see Appendix 11).

Personal lessons learned

The most commonly mentioned personal gain was that the intervention offered pupils more insight into the possible consequences of an UTP on their lives.

And realising you probably wouldn’t have a . . . like a career . . . Well, you would, like you’d probably get like a job that you wouldn’t really need much qualifications, but if you had a baby, you’d probably be less likely to go to university when you had to take care of a child.

For some, the intervention seemed to challenge existing beliefs, such as the following.

• Babies are not hard work:

But people are always saying babies are worth it, they’re genuinely worth getting up at, you know, 4am.

• Abortion is an option for some people:

I understand that it has to be put in because it is an option, but I do like think it’s really stupid, especially since it’s like allowed up to like 24 weeks or something I think it is, which that is technically murder because like it’s breathing, which is . . . But the boys were being really immature about that question, like saying stupid things like ‘Yeah, do that now’, and it annoyed like quite a lot of us.

• Men have feelings and responsibilities when it comes to UTP:

Yeah, it was good to see how . . . because, usually, you focus on the girl, like how it would affect the girl, but there, you see . . . it isn’t affecting the guy physically at all, but it’s . . . like the moral implications of it, so how he’d think . . . what would be right, and usually you don’t think of that. Usually, guys are portrayed as just leaving.

In addition, some participants mentioned that there were advantages to putting yourself in Jack’s shoes and that the intervention had prompted them to consider teenage males’ roles and responsibilities.

I’d never thought about it that way, like from a guy’s point of view. It was all from a girl’s point of view, like going through that.

Strengths and weaknesses of the teacher training session

Although process evaluation findings indicated that teachers felt that they could implement the intervention with the help of the detailed ‘teacher’s information booklets’ and an online video, they were adamant that the training session was useful for them and suggested that, for the purposes of a future trial, the face-to-face training session should be retained.

Strengths and weaknesses of the parents/guardians’ information and discussion session

As noted in Chapter 5, recruitment of parents and guardians to attend the school-facilitated information and discussion session, to view the information session video on the YouTube channel and to respond to the parent’s online survey was extremely low. Twenty-nine parents/guardians responded to the online survey, with 22 indicating why they had not attended the information and discussion session (Table 40).

One mother wrote that her son had asked her not to attend:

My son was a little embarrassed by the subject matter being shown and discussed and requested that I did not attend.

Mothers and teachers attending focus group discussions shared speculations about why parents may not have attended and why the general experience of the teachers was that parents were difficult to engage in non-curricular activities in general. Reasons included that they were generally disinterested or apathetic, or that they did not feel the need to talk about such issues because they have already addressed them with their child or that they did not have time to attend owing to work commitments (as a result of the early timing of the event), or an unwillingness to give up other commitments, the distance from school and language barriers.

One teacher in the intervention group spoke of previous successes using short animations relating to online safety. Other teachers agreed that sending a link to a video via text message might reach more parents. Some teachers suggested alternative ways of engaging parents, including tagging on the session to other events that are more likely to be attended, such as parents’ evenings (when parents meet with their children’s teachers regarding progress).

Feasibility of implementation

Fidelity of implementation of the protocol

As detailed above, teachers often felt the need to adapt the resource to their own class situation, or were logistically forced to adapt in the moment. In one school both teachers decided not to deliver lesson four. Although both teachers agreed that they felt the pupils had had enough of the intervention after three lessons, one conceded that a lack of time was also a factor. Some teachers noted that they took longer than expected to deliver the intervention and that some pupils missed out on some activities owing to interruptions such as closures as a result of heavy snow, bus strikes, fire drills and pupils being taken out of class for sports and music activities. Some teachers extended the lessons in order to ensure that the homework activities were completed within school time. One teacher decided to focus on one dilemma rather than the four provided in the pack. Some teachers felt that allowing small group discussion prior to whole group discussion worked best for their pupils.

One pupil suggested that his teacher had not implemented the intervention as instructed, suggesting that the pupils were sat down in front of the IVD without explanation, provided with the worksheets and left to their own devices to figure out what to do.

Most of the teachers said that they delivered all activities, even if they had to extend the number of periods in order to do so. In one school, however, the teachers decided to leave out various activities, including the scavenger hunt, the parental survey, the controversial statements activity and activities involving role-play.

Fieldworker reflections on data collecting in schools

The fieldworkers reported that the majority of data collection was carried out in accordance with protocol and that most schools and children were accommodating and happy to participate. When asked about any problems they encountered, three overarching themes emerged from their reflections. These were school environment, pupil issues and researcher issues. First, in relation to school environment, fieldworkers initially experienced difficulties with computers and online questionnaires and reverted to paper questionnaires. In addition, pupil proximity in classrooms during questionnaire completion was thought by some to be too close and to potentially compromise confidentiality. Although teachers were thought of as helpful during the initial stages, some thought that their continued presence in the classroom may have affected the children’s confidence to ask questions and seek help. In one school, examinations ran over time, which resulted in timing issues, and buses arrived to take children home before they could complete the questionnaire. Second, in relation to pupil issues, fieldworkers noted that a few were confused about the confidentiality of the study, some had difficulty describing their family circumstances within the questionnaire, and some struggled with the wording of questions and terms used. Third, in relation to researcher issues, a few fieldworkers commented on dealing with challenging behaviour in the classroom and the extra time required to clarify the process, define terms and provide additional support for individual pupils (see Appendix 13 for categorised raw data and themes).

Feasibility of participation in the trial

Teachers

Teachers were asked about their experiences of taking part in the research. One teacher in a comparison school was disappointed not to receive the resource. The designated trial champion in one school spoke about the interruptions to her own work owing to technology problems experienced by those delivering the intervention and the importance of support from the research team. In another (grammar) school in the control group, the trial champion indicated that, although it had been challenging to begin with, he had learned to deal with the organisation of data collection over the course of the year and had really valued help received from classroom assistants.

Several teachers felt that offering pupils the opportunity to opt out was not in line with their usual way of doing things and led to disruption.

I know that we had a significant number of pupils who were given the opt-out option. Ehh I’m not sure if I would give them that option. I can probably understand the reasons behind it but, ehh, that option to opt-out is easy and some of those pupils who opted out are the very pupils who you would want to get a response from.

Some teachers noted that pupils found it hard to understand the connection between the questionnaire and the resource.

The baseline questionnaire, I think [was confusing] for some pupils, because then, whenever . . . whenever they went to then do the resource, they felt, well, what had that got to do with . . . where was the connection? They didn’t see that there was a connection in there.

One principal felt that it would have been more convenient to allow pupils to complete the questionnaires online at a time that suited them. Teachers also noted the fact that those pupils who took part in the focus groups enjoyed ‘having their say’. The principal of one of the schools in the control group reflected on the inconvenience of taking part in two research studies at the same time but also noted that the If I Were Jack trial was well organised.

Methods

We calculated the economic cost of If I Were Jack according to the principle of opportunity cost. That is, when resources have alternative beneficial uses (i.e. an opportunity cost), it is important to represent this with some monetary value, regardless of whether the resources were directly purchased in the market place. A microcosting approach was taken from a public sector decision-maker perspective. Relevant costs were identified, measured and valued in monetary units using the 2013/14 price year. The identification of appropriate intervention costs was guided by a number of sources. These included the practical guide to costing behavioural interventions of Ritzwoller et al. 132 and methods used in other behavioural interventions such as the Diabetes Education and Self-Management for Ongoing and Newly Diagnosed (DESMOND) programme,133 A Stop Smoking In Schools Trial (ASSIST)134 and the First-aid Advice and Safety Training (FAST) parent programme. 135 In keeping with both ASSIST and FAST trials, costs were categorised according to the stage at which they were incurred in the research process: planning and preparation for delivery (stage 1) and delivery itself (stage 2).

Pre-start-up costs associated with the development of If I Were Jack were not included in the analysis (e.g. lesson plan development, designing of materials, DVD production2) as these are non-recurring costs that were incurred prior to the start of this study. Stage 1 costs in the If I Were Jack intervention covered recurring costs associated with printing the classroom materials for teachers, providing the DVD and the delivery of teacher training. As the identification of relevant resources occurred prior to the implementation of the intervention, the research team were able to record the resource use prospectively.

Stage 2 costs covered the costs associated with teacher time input for lesson preparation, printing/photocopying of worksheets and teaching, and administration time input for printing/photocopying of worksheets. Resource use was gathered prospectively via a specifically designed Teacher Resource Use Questionnaire (see Appendix 3). The questionnaire had two objectives:

1. to measure the resources actually used in the delivery of each component of If I Were Jack in terms of time input and materials

2. to establish what teaching activities may have been displaced in order to deliver If I Were Jack, or, indeed, if time was freed up as a result of the intervention being implemented.

These were intended to provide us with more information on the true opportunity cost of delivering the intervention.

The quantity of resource use at each stage was multiplied by unit costs at 2013/14 prices to estimate total costs, mean cost per teacher, mean cost per school and mean cost per student. In accordance with the principle of opportunity cost, the cost of teaching time was included, as this reflects the benefits to the children forgone as a result of If I Were Jack being taught instead of something else. The trainer in the trial was a post-doctoral research fellow employed by the university. Training took place at schools within the staff room, so no costs were associated with the training location. Teachers were supplied with the intervention materials by the research team. The grade of teacher delivering pupil and parent components of If I Were Jack varied; thus, we used the mid-scale point on the upper pay scale (UPS2; Department of Education Northern Ireland). 136 We assumed that a clerical officer at point 11 on the National Joint Council pay scale carried out the photocopying of classroom materials when the teachers did not. All staff costs included 24% for employers’ costs.

Results

A qualitative exploration of the opportunity costs of delivering the intervention

When resources are allocated to be used in a particular way, they are no longer available to be used in another way. As a result, any benefits that would have been received are forgone. The opportunity cost is the term used to reflect the value of the benefits that have been forgone. Although costing the resource use as detailed in objective 1 is a pragmatic approach to measuring this opportunity cost, we wanted to explore the potential sacrifices more directly via the teachers.

Exploration of alternative methods of delivering intervention training in a future trial

In this feasibility study the intervention was delivered to four schools by a university researcher with a total of 16 teachers involved. Even with this relatively small number of schools and teachers, the organisation of teachers’ training sessions proved difficult. Principal barriers included identifying a time that suited both teachers at each school and the researcher delivering the training, and finding a suitable venue for the training within the school. These issues would only be exacerbated by increasing the number of schools and teachers, which would be required for a future Phase III trial. We also acknowledge that using a university researcher to deliver training on a large scale may not be realistic. Thus, using scenario analyses we explored alternative approaches to training for a future Phase III trial in order to estimate the financial impact of the alternatives.

The framework: from trial outcomes to social outcomes

Owing to the social impacts of many interventions, especially public health interventions, extending beyond the life of any single randomised trial, it is often the case that funders and decision-makers are interested in ‘modelling’ what these future impacts – usually expressed in terms of costs (to reflect net resource impacts) and benefits (to reflect future impacts on health and other aspects of well-being) – might be. For the If I Were Jack intervention, we have attempted to map out the important events subsequent to engaging with If I Were Jack. This is presented in Figure 6, along with the consequent data needs and possible sources of such data. Some of the required data can be sourced from a trial, whereas other data would have to be sourced from elsewhere.

FIGURE 6.

Current and future impacts of If I Were Jack on costs and benefits. HRQoL, health-related quality of life.

The framework in Figure 6 should be read from left to right and, thus, starts with a defined population that would engage with the intervention. The boxes represent events, or collections of events, that could have well-being and/or resource consequences. Data needs associated with the boxes are displayed at the top of the framework and labelled ‘Information requirements: events’. So, for example, all one would need to know at the stage of engaging with the intervention are the costs of the If I Were Jack intervention.

The lines between the boxes represent changes in rates at which the events might occur, and the data needs associated with these (at the bottom of Figure 6) are labelled ‘Information requirements: changes in rates of occurrence’. For these rates, ‘x’ and ‘y’ should always sum to 100, as they represent percentage amounts by which the original population, and subsequent sub-divisions of that population, is spread across the set of events occurring in the next stage of the framework. The subscripts on ‘x’ and ‘y’ simply represent the progressing stages to which our ‘x’ and ‘y’ values apply.

For example, an important secondary outcome from the trial is likely to be related to ‘awareness raising’. Awareness is regarded as an important secondary outcome because it is believed to mediate the primary behavioural outcome of safe sexual intercourse/no sexual intercourse. For a certain percentage of the targeted population (x¹), awareness will be raised, whereas for a corresponding percentage (y¹ = 100 – x¹), there will be no such impact. Trial data would indicate what these percentages might be and, for any individual who is positively influenced, what the magnitude of the change in awareness might be. For those whose awareness is not raised, we might assume that they revert to baseline rates for future events as we move from left to right in Figure 6. These rates are merely indicated by the word ‘baseline’ (issues related to ‘baseline’ data are discussed below).

More positively, however, any raised awareness might subsequently lead to changes in behaviour. It is important to note that such changes might conventionally be seen as leading to greater abstinence and increased condom use. However, a more positive outlook might be to view increased awareness as leading to increased confidence with respect to decision-making about sex. This might lead to increased condom use, but is better expressed as leading to increased levels of contraceptive use. Alongside abstinence, we need also to think of ‘delay’ of sexual intercourse and ‘frequency’. For these, we would need to know how higher awareness relates to changes in contraceptive use, delay and frequency behaviours, that is, values for x² and y² – which could be obtained from a subsequent trial and for which there is an existing literature (see Chapter 3). 19,21–23,25

We would also require data on the costs of contraceptive use, including condom use. Once again, for those going down the ‘y²’ pathway, we might assume that they revert to baseline from that point onwards.

The fact that programme participants might feel more in control of their sexual behaviour and decision-making, of course, has value in and of itself. However, the next stage would also involve estimating the extent to which increased contraceptive use (and changes in delay and frequency) relates to more immediate (health) events. For each of these, a separate value of x³ (and, correspondingly, y³) would be required. These events would be as follows:

• STIs, for which data would be obtainable from a trial as well as literature linking contraceptive use, delay and frequency changes to STI occurrences137,138

• pregnancies before the age of 17 years25

• terminations associated with pregnancies, for which we would require data on which to base an assumption about the continuation, and corresponding termination, rates.

For each of these, we would require data on costs (of a STI, a delivery and a termination) and we would have to make assumptions about what costs to include (e.g. do we go as far as counting the lifetime costs of a birth and what assumptions do we make about replacement; that is, does an early birth now replace another birth that might have happened later?). 139 We would also require data on health-related quality of life and other health outcomes associated with STIs, pregnancy and termination.

The final set of events in Figure 6 comes under the general heading of ‘life chances’. Once again, we would require estimates of the percentages x⁴ and y⁴; essentially, for what proportion of people are life chances negatively impacted by an early pregnancy? The impact of such an event is portrayed currently in terms of withdrawal from education and the consequent effects of that on increased welfare payments and worse job opportunities, for which we think there are data available from government sources in NI. On the basis that a teenage pregnancy effectively withdraws the mother from the labour market for at least 18 months, the DHSSPS estimates that the cost of a teenage pregnancy to the Exchequer (unemployment benefits and administration, plus tax revenue forgone) stands at £20,000 per mother. 26 In 2006, it was estimated that teenage pregnancy costs the NHS £63M per year. 140 In 2009–10, £26M was paid in benefits to teenage mothers on income support (England and Wales-based data). 141 Other adverse sexual health outcomes also cost the NHS large sums. For example, it is estimated that the UK spends between £700M and £1B per year treating STIs and their sequelae. 142,143

Other data needs: the ‘baseline’

Depending on the ‘level’ addressed by the modelling, and in order to address the issue of the ‘baseline’ for many of the variables, other requirements might be for NI-level data – for example, things like current rates of condom use and abstinence, and for STIs, age-specific pregnancies and associated terminations, as well as data on withdrawals from education.

Study limitations

In this exploratory trial we assessed the feasibility and acceptability of an effectiveness trial of If I Were Jack. The study was therefore designed to generate learning that could inform the decision about whether or not to proceed with an effectiveness trial, and also – if a decision to proceed was made – learning in relation to optimising the design and implementation of an effectiveness trial. Nonetheless, although this study was a learning exercise, the study itself also has a number of limitations that should be borne in mind when interpreting the results presented in this report.

Primary aim

The aim of the study was to determine the value and feasibility of conducting an effectiveness trial of the If I Were Jack RSE intervention in post-primary schools.

We now break down the main findings of the study under the study objectives.

Secondary objectives

To assess the acceptability of the intervention to schools (principals and RSE teachers), male and female pupils and parents.

The intervention proved to be acceptable to schools (principals and RSE teachers), as evidenced by the very favourable remarks by teachers and school principals right across all participating schools.

I think, as resources go, it’s one of the best I’ve seen out there because anything we have is quite ancient, dated or from, like . . . England, Australia . . . We’re very lucky to have a resource that is very current and set, you know, in the local area.

I think it’s actually a very good resource, and I think the fact that it’s coming from a boy’s point of view is unique and the pupils took to that well, you know, because they’re always hearing about teenage pregnancy always more geared from the girl’s point of view. So, I think they found that maybe kind of . . . made them more interested in it, just from the very start.

If I Were Jack was also an acceptable intervention for pupils, both male and female. Pupils commented on specific positives of the intervention and these included that it was ‘locally set’, ‘realistic’, ‘relatable’, ‘interesting’, ‘informative’ and ‘helpful’.

It didn’t give you one point of view, it gave you many, and it gave you the way that he was worried, and that she was worried. It didn’t show you everything would be fine.

Male

Like, if it was to happen to someone, it’s kind of a difficult situation to deal with and it shows you that. It’s almost like em . . . you get a first-hand experience of what would happen and it’s scary.

Female

In relation to parents, the low rate of parental/guardian withdrawal of children from the study (6.8%) suggests that the vast majority of the parents were happy for their child to participate in the intervention and the associated research. However, the results also suggest that parents were not willing to participate themselves in the intervention in the planned way (i.e. in a face-to-face information session about the intervention). Thus, a key change to the intervention, which we will undertake as a result of this feasibility trial, is to change the face-to-face information sessions to an online format. The parental component of the intervention will instead be delivered through three short (< 5 minute) online videos to parents. The links for these videos will be texted and/or e-mailed (with one additional reminder) by participating schools to all parents/guardians of the intervention year group on school communication lists. This electronic communication approach was successfully piloted in the feasibility study. Recent studies 68,69,148 demonstrate the importance of embracing such ‘education entertainment’ modalities as engaging adjuncts to school-based education. This refinement has been funded by the Health and Social Care Research and Development Office of Northern Ireland (HSC R&D). It was completed in 2016 and we would plan to pre-test this approach as part of a main trial.

Another secondary objective was to identify optimal delivery structures and systems for the delivery of the resource in the classroom. Teachers valued the fact that they considered If I Were Jack to be an ‘off-the-shelf’ product that required little preparation time, apart from some familiarisation with the content and photocopying. Teachers indicated that if the resource was being made more readily available outside the research environment, then teacher training could become accessible online, with a contact number to address any enquiries. They did, however, acknowledge that, in the context of participation in a research trial, a researcher-led session was essential to reinforce fidelity of implementation, etc. The evaluation also revealed that more time could have been dedicated to the teacher training session, as teachers reported a lack of time to address all questions.

Process evaluation indicated that there would be advantages to conservatively forewarning teachers of the time requirements of component elements of the intervention, thereby enabling forward planning (i.e. avoid using last lesson session of the day). One school had a 35-minute lesson time and struggled to complete the IVD and other accompanying activities within a single lesson, as hoped. Time-related issues were prevalent, with teachers reporting incidents such as examinations running over time and pupils being unable to complete questionnaires because school had ended for the day and they had to race off to catch their buses. Teachers valued the presence of the research team and the assistance made available through classroom assistants. Pupils valued incentives, such as sweets, but some suggested that a ‘contraceptive goody bag’ might be better.

Finally, in relation to the delivery of the intervention in the classroom, there was a consensus that pupils were receptive to teachers as deliverers of the intervention. However, process evaluation findings also revealed that some pupils had an aversion to engaging with parents in the homework activity, in addition to an actual or perceived perception that their parents would not be willing to engage in the homework activity with them. Although teachers reported that some students did undertake the homework activity, this was regarded by the teachers as an optional bonus, rather than a requirement. Thus, for a future trial we must consider the necessity of this ‘beyond classroom’ component to the overall logic model of the intervention.

As detailed above, technical difficulties meant that the questionnaire was delivered predominantly in paper form. Technical difficulties, mostly associated with school networks, also led to all but one school showing the IVD via overhead projector as opposed to online. Process evaluation revealed this was an acceptable method of delivery, with most teachers acknowledging that there were additional benefits to delivering in this way (i.e. control of pace, the opportunity to stop and explain, and being better for pupils with reading difficulties). In a future trial, we would still encourage use of individual personal computers or iPad (Apple Inc., Cupertino, CA, USA) computers, and we would seek to build in a mode of delivery into the memorandum of agreement. However, owing to this positive evaluation of overhead method, we would not exclude a school if this was the only feasible way of delivery.

A further secondary objective was to establish participation rates and reach, including equality of engagement across schools of different socioeconomic and religious types. The study demonstrated that it was feasible to recruit the full range of the most common types of schools in NI, namely schools with differing school management systems, including faith-based schools, schools that are especially academically oriented and those that are not, and schools that are in socially deprived as well as socially affluent areas. The inclusion of faith-based schools is important because we believe that this is the first RSE intervention trial to include faith-based schools in the UK and > 50% of schools in NI are faith based. Nonetheless, we also learned that the hardest schools to recruit for this trial were schools that were both faith based (i.e. RC) and high academic achieving schools. These are what are known as ‘Catholic Grammar Schools’ (RC Grammar) in NI. The overall recruitment rate for RC schools was 30%. However, the recruitment rate for RC grammar schools was 13%, while the recruitment rate was 67% in RC maintained secondary schools. The latter are RC schools for which the pupil’s entry selection is not dependent upon high academic achievement at 11 years of age. Thus, although the trial succeeded in recruiting a breadth of schools across crucial sociodemographic criteria, we are acutely aware that we also then need to reach potentially socially excluded groups of pupils within these schools, those students who are most likely to become a teenage father/mother (e.g. those from more deprived groups, those with low literacy, low attainment and poor school attendance, and those with learning disabilities), and we address this in the discussion below.

Another secondary objective aimed to assess trial recruitment and retention rates. In terms of school recruitment, the feasibility study demonstrated that our target aim of recruiting seven schools, four to the intervention arm and three to the control arm, was realistic and that we could achieve our school recruitment target rate (target = 25%, achieved 37%). In terms of pupil recruitment, the target was also met, with an overall pupil recruitment rate of 80.9%. Absentees who did not return completed questionnaires accounted for 9% of non-participants, and parental and pupil opt-outs accounted for the remaining 9.9% (6.8% and 3.1%, respectively). However, one conclusion of the trial has been a commitment to work with the trial steering group in the main trial to enhance clarity around the consent process for parents and guardians, as some indicated confusion regarding the opt-out form (assuming it was an opt-in form). Pupil information sessions also need to be better monitored by researchers to ensure that they follow protocol. We shall consider a short audio-visual presentation, which could be easily shown by teachers in addition to information sheets. In addition, we would try to avert data loss to pupil absenteeism by not conducting data collection on Monday mornings or Friday afternoons or in the weeks preceding holiday periods. In terms of parental recruitment, as reported above, we failed to adequately engage parents in the study (parental information sessions), recruiting only 2.3% of all parents/guardians. The recruitment of teachers for semistructured interviews within schools was unproblematic. Retention of schools was successful (100%) as was retention of pupils (93.1%).

We also aimed to assess variation in normal RSE practice across the participating schools. In this feasibility study, existing provision of RSE (content, frequency, co-ordination and delivery) was assessed. In NI there is no official curriculum material targeted to Key Stage 4 (Years 11–13). Year 11 was the participating year group in this feasibility study, although there was a consensus among participating teachers that they would also like to use the intervention with Year 10. In general it emerged that RSE provision was extremely variable and not perceived as a priority subject. Some schools had well-structured RSE programmes, whereas others had difficulty articulating the RSE provision within the school. Pupils recalled little to no attention or emphasis placed on UTP. All schools provided most of their RSE provision using the ‘Insync’ Personal Development programme, developed by CCEA, of the LLW statutory component of the NI curriculum. This programme was developed for Key Stage 3 to be targeted to Years 8–10 (see Appendix 7). Although RC schools described their approach as always in line with the RC Church’s teachings and ethos, this did not appear to alter the nature or attitude towards RSE provision compared with non-Catholic participating schools. Participating grammar schools, although academically focused, were not distinguishable in their existing RSE provision. Teachers reported a general sense that RSE provision remained somewhat sensitive and controversial and that attitudes towards RSE were highly variable, with some reports of religious conservatism and homophobia, and that improvements could be made. There was unanimous willingness to embrace new approaches and improve existing provision. Post-qualification training and opportunities for training for teachers appeared to be all but absent.

A further secondary objective was to refine the survey instruments as a result of cognitive interviews with male and female pupils. The pupil questionnaire has already undergone substantial redevelopment, as informed by the pilot and feasibility study (detailed in full in Chapter 4). Based on cognitive interviews (and other feedback mechanisms such as the pupil comment sheets and fieldworker observation sheets) after the final data collection in the feasibility study, we determined that further changes to the pupil questionnaire were required. These included simplifying and standardising the Likert-scale options and increasing the privacy of the participant by ensuring that skipped sections would be replaced with an equivalent number of questions and by randomising sections that would be incorporated into the questionnaire for a Phase III effectiveness trial. This was suggested by pupils so as to disguise who was answering what questions at any particular time point during data collection sessions, thereby negating any presumptions being made by fellow pupils about sexual activity and time taken to answer certain questions as a result. For an overview of the suggested changes to the questionnaire that emerged from the focus group activity see Table 17.

We also aimed to assess differences in outcomes for male and female pupils. In the subgroup analysis of the primary outcome measure by gender, there was an improvement in boys in the intervention arm (2.2% improvement at follow-up 1 and 0.5% improvement at follow-up 2), but a deterioration in females in the intervention arm was evident (1.3% deterioration at follow-up 1 and 4.4% deterioration at follow-up 2). Female pupils in the intervention group scored notably higher than their male counterparts in the Knowledge scale at both follow-ups 1 and 2; in the Intentions scale at follow-up 1; in the Intentions Self-Efficacy subscale at both follow-up 1 and 2; and in the Attitudes subscale at both follow-ups 1 and 2. Male pupils in the intervention group scored significantly higher (p = 0.004) than male pupils in the control group at follow-up 2 and notably higher than their female counterparts in the intervention group in the Male Role Attitudes scale (male 1.27, 95% CI 0.41 to 2.13; female 0.47, 95% CI –0.22 to 1.17). This was not evident at follow-up 1. The Male Role Attitudes scale is essentially an analysis of emerging masculinities that are more gender equal versus the traditional gender-related views that men and women are unequal.

The questionnaire also included a specific attitudinal question relating to whose responsibility it was to prevent an UTP, allowing for the balance of responsibility to be moved towards males or females. Both the intervention and control group demonstrated an improvement towards attributing an equal 50 : 50 ratio of responsibility over time, although the improvement was more pronounced in the intervention group (13.3% vs. 10.4% in the control group).

Another secondary objective was to identify potential effect sizes that might be detected in an effectiveness trial and estimate an appropriate sample size for that trial. The results of the feasibility study provided us with some useful parameters to calculate a sample size calculation for the main trial. To take this in sequence, we first report the effect size results in the feasibility trial. In the feasibility study, approximately 3% of our sample reported unprotected sex at baseline (3.6% in the intervention group and 2.5% in the control group) and approximately 5% reported unprotected sex at follow-up 2 (5.4% in the intervention group and 5.6% in the control group). This amounted to a 50% increase in unprotected sex by follow-up 2 in the intervention group (3.6% at baseline rising to 5.4% at follow-up 2) compared with a more than doubling of unprotected sex by follow-up 2 in the control group (2.5% at baseline rising to 5.6% at follow-up 2). We used this potential effect size to inform the sample size calculation for the main trial. The main trial will be powered to detect a 50% reduction in the incidence of unprotected sex (from an expected rate of 2.8% to 1.4%) by 15 years of age. A difference of 1.4% in unprotected sex has been shown to have a meaningful impact on pregnancy rates. 18,122,124,149 The between-group difference in the incidence of unprotected sex of 1.3% (95% CI 0.5% to 2.2%) by 9 months in our feasibility trial demonstrates that such an effect size is plausible and is consistent with effect sizes seen in the literature. 122 The study will take account of clustering. In the feasibility trial data the intraclass correlation coefficient (ICC) was 0.01. As pilot studies can provide imprecise estimates of ICCs,150 we re-estimated using ICCs from three sources: (1) the Randomized Intervention of PuPil-Led sex Education (RIPPLE) cluster RCT; (2) data from the WHO Health Behaviour in School-aged Children Survey GB;129 and (3) the Young Persons’ Behaviour and Attitudes Survey 2011. 9 The data from the WHO and Northern Ireland Statistics and Research Agency (NISRA) studies were combined. Both the RIPPLE and combined WHO and NISRA studies found an ICC of 0.004. In terms of our sample size calculation based on these data, first, based on the ICC of 0.01 (95% CI 0.01 to 0.04), and assuming 120 students per year group in school, and a 7% rate of attrition (based on the conservative attrition rate found in the feasibility study plus two additional schools), a trial involving 33 schools per group will provide 80% power at a 5% significance level. The power would rise to 93% if we used the alternative ICC of 0.004. Table 46 shows the sample size calculation based on the first ICC (0.01).

The final secondary objective was to identify the costs of delivering If I Were Jack and generate methods for assessing cost-effectiveness in a future trial. The mean cost of delivering If I Were Jack per pupil after adjusting for school cluster was £13.66. The economic cost was calculated to include teacher and school costs according to the principle of opportunity cost and using a microcosting approach taken from a public-sector decision-maker perspective. Preparation for a future cost-effectiveness decision model that incorporates theories of behaviour change was also undertaken. The structure of the model is illustrated in Figure 6. In the main trial, this framework will be populated with data from future primary research, policy documents and published literature (see further elaboration under progression rules below).

Thinking through implementation and evaluation from the outset of intervention design

In providing guidelines for the development and evaluation of complex interventions of this kind, the UK’s MRC89 recommend that high-quality evaluation is preceded by a systematic approach to intervention development. The iterative four-phase approach outlined in the MRC framework involves developmental work (referred to as ‘Phase I’ research), which includes identification of the evidence base, identification or development of theory relating to the phenomenon of interest and modelling of processes and outcomes to inform optimisation of the intervention prior to evaluation. However, in our view, although the MRC guidelines89 suggest a very useful framework for conducting Phase I research, they provide little description of how this should proceed (relative to guidance provided on Phases II–IV). This effectively relegates the importance of this phase to brief summaries in evaluation research, despite growing consensus that the development of conceptually based, acceptable interventions is of vital importance before proceeding to trial. This is compounded by a dearth of published literature reporting the development of complex interventions more broadly but, also, specific developmental process in relation to RSE interventions are scant in the literature (for exceptions, see Carswell et al. ,153 Kirby et al. ,94 McCarthy et al. 154 and Wight and Abraham155). The result is a paucity of models that might guide comparable research. In Chapter 3 we report the intervention design process in detail to hopefully guide other researchers in the design phases of comparable interventions.

We also offer the following observation in relation to the MRC guidelines. 89 The MRC guidelines do not indicate the importance of considering implementation during Phase I, despite the fact that this might alert the researcher to factors that may impact on successful implementation. In discussing Phase IV, the guidelines indicate that ‘successful implementation depends on changing behaviour – often of a wide range of people. This requires a scientific understanding of the behaviours that need to change, the factors maintaining current behaviour and barriers to and facilitators of change, and the expertise to develop strategies to achieve change based on this understanding’. 89 It seems likely many of these factors should be considered during the development phase, rather than post evaluation and this is the approach that we took during the study. The current study builds on the team’s Knowledge Translation Award from the Economic and Social Research Council (RES-189–25–0300),156 through which the intervention was initially developed and a ‘translation study’ offering insights into generating a culturally generalisable intervention across the UK. 144 It also follows foundational empirical groundwork generating understandings of adolescent males’ attitudes towards and decision-making in relation to teenage pregnancy. 1,90,93 A hallmark of this preparatory work has been meaningful consultation with all stakeholders, from pupils to senior policy-makers, ensuring that the If I Were Jack intervention can be sustainably implemented if proven to be effective. We would argue that an essential part of the design process involves thinking about and planning for all aspects of the research process that will follow – feasibility testing; evaluation; implementation, follow-up and monitoring.