Adapting the ASSIST model of informal peer-led intervention delivery to the Talk to FRANK drug prevention programme in UK secondary schools (ASSIST + FRANK): intervention development, refinement and a pilot cluster randomised controlled trial

ASSIST

The ASSIST intervention is an evidence-based, informal peer-led smoking prevention intervention based on diffusion of innovations theory22 that aims to diffuse and sustain non-smoking norms via secondary school students’ social networks in Year 8 (age 12–13 years). 23 It is part of the tobacco control plans of the Scottish24 and Welsh25 Governments and is recommended by the National Institute for Health and Care Excellence. 26 In the UK, > 120,000 students have taken part in the ASSIST intervention and an estimated 2200 young people have not taken up smoking because of ASSIST who otherwise would have done so. The five stages of ASSIST, as currently delivered, are listed in Table 1.

Talk to FRANK

The Talk to FRANK website (see www.talktofrank.com) was set up by the UK Department of Health and the Home Office in 2003 to provide up-to-date, youth-friendly information and advice on the risks of illicit drug use. Between January and December 2014 there were 5.1 million visits to the FRANK website. 27 In 2013, 18% of 11- to 15-year-olds (increasing from 5% of 11-year-olds to 33% of 15-year-olds) reported that the website was a source of helpful information about drugs. 28 The FRANK website is the most commonly cited source of information by teachers (78%) for preparing lessons on drugs. 6

Information is provided using a number of interactive methods including videos showing the effects of specific drugs, regular updates on the legality and mechanism of effect of new drugs, an A-Z guide of drugs and a frequently asked questions (FAQs) section. Highly rated FAQs include those on the long-term effects of cannabis, the levels of drug use associated with dependence, criminal consequences and the half-life of specific drugs. FAQs are also tailored to young people, parents and health professionals. Support is offered through the website with instant messaging, live chat (available at specific times during the day), an online forum, a confidential 24-hour telephone service and an e-mail or text service. Trained advisors operate this free service and local help can also be found by entering a postcode.

CASE+

There was a previous attempt to adapt the ASSIST intervention to prevent cannabis use. An unpublished feasibility trial (CASE+) in six Scottish secondary schools found little change in intentions to use cannabis in 732 students aged 12–13 year over a 3-month follow-up (Munro A, Bloor M. A Feasibility Study for a Schools-based, Peer-led, Drugs Prevention Programme, Based on the ASSIST Programme: the Results. Centre for Drug Misuse Research Occasional Paper. Glasgow: University of Glasgow; 2009). The intervention included an extra day of education solely on cannabis use in addition to the 2 days of training on smoking in the ASSIST intervention. However, the study had several limitations: (1) no drug use data were collected, (2) 3 months’ follow-up is not long enough to ascertain a likely effect of the intervention over longer periods and (3) the intervention content was solely related to cannabis use.

The process evaluation indicated that, although implementation fidelity and acceptability to school staff was high, students were overwhelmed with the amount of information that they received in the extra day of training on cannabis. The peer supporters also rarely had conversations about cannabis but focused on smoking, because only a few students were experimenting with cannabis at age 12/13 years and there was sensitivity around discussing cannabis use as it is illegal. We decided to adapt the ASSIST model to deliver the information from the FRANK website rather than CASE+ as it has a database of information on illicit drugs and not just cannabis. The FRANK website has demonstrated a large number of hits per year and teachers and young people use it, such that it provides an accessible drug education resource that updates.

Setting

The study took place in seven local authorities in South Wales (Cardiff, Newport, Torfaen, Blaenau Gwent, Rhondda Cynon Taf, Merthyr Tydfil and Caerphilly). With a projected population of 1,155,800 in 2016, these local authorities represent 36.9% of the Welsh population. 29 Fourteen out of the 72 secondary schools across the seven local authorities participated. Two schools received one of the interventions each in stage 2. Twelve schools were involved in stage 3, the pilot cRCT.

Ethics approval and monitoring

The study was granted ethics approval by Cardiff University School of Social Sciences Research Ethics Committee (reference SREC/1103). NHS research and development (R&D) approval was also granted, as Public Health Wales (PHW) staff delivered the interventions (reference 2013PHW0015).

Trial monitoring

The ongoing conduct and progress of the study was monitored by an independently chaired Trial Steering Committee (TSC). The membership included independent scientific experts, representation from the Department of Health, policy leads for children and young people in PHW and a parent representative.

Interventions

We present here an outline of each intervention provided at the time of the submission of the proposal for funding. The content of the activities and the method of delivery had yet to be developed. Both interventions underwent revision during stages 1 and 2.

Phase 1: evidence review and stakeholder consultation

Phase 2: coproduction

An intervention development group (IDG) was established consisting of members of the research team and the PHW ASSIST delivery team. The PHW team had delivered the ASSIST intervention to > 350 schools over a period of 7 years and so had extensive experience of intervention delivery within schools. The aim of the IDG was to adapt the ASSIST intervention materials to deliver information from the Talk to FRANK website, informed by the findings from phase 1.

Coproduction of intervention content took the form of an action research cycle over a series of meetings of the IDG at which findings from stage 1 were considered, ideas were presented by all members, feedback on ideas was sought and refinements were made and presented again, until the final content was agreed. Five face-to-face meetings were held over the course of a 4-month period. These were supplemented by communications by e-mail when face-to-face meetings were not possible or when matters arose that required discussion between meetings.

Phase 3: prototyping

After a first draft of the +FRANK and FRANK friends interventions had been developed, the manuals and resources (e.g. fact sheets on the effects of cannabis on mental health) underwent expert review. Two experts were asked to examine key uncertainties identified during phases 1 and 2. The lead author of the ASSIST cRCT23 examined the fit of the activities with diffusion of innovations theory,22 on which the ASSIST intervention was based. The lead trainer of DECIPHer IMPACT (Bristol, UK), the company that licenses ASSIST, and reviewers examined the age appropriateness of the activities and the suitability of the timings and sequencing of activities.

Preliminary feedback on acceptability and feasibility was collected during training of the PHW intervention delivery staff. Training involved the delivery of the two interventions to the IDG. Feedback was also sought from ASSIST trainers at a DECIPHer IMPACT ‘Train the Trainer’ course. The interventions were also delivered to a young people’s public involvement group (aged 16–19 years). Feedback was sought from young people on each activity, with a particular emphasis on relevance for their age group and interest in and engagement with the content.

Structured observations, evaluation forms and interviews

Independent structured observations were undertaken by two researchers of all intervention activities, including recruitment, re-engagement, training and follow-up visits. Observation forms captured whether or not objectives for the sessions and learning outcomes for activities were met using a traffic light system. Green indicated that the activity had been fully delivered with all objectives met, amber indicated minor deviations in delivery and red highlighted that activity objectives had not been met. The timings of the activities delivered were also recorded along with notes on any deviations to the timetable and/or activity instructions. Trainers completed self-assessment forms following each delivery episode to capture their perceptions of the delivery for each aspect of intervention delivery and training.

Evaluation forms were completed by peer supporters and trainers at the end of the off-site training days and final follow-up visit. Interviews were conducted with peer supporters, trainers, contact teachers and members of the SMT after the last follow-up. Interviews with peer supporters were conducted individually, in pairs or in small groups, to suit the requirements of the school. Additional interviews were conducted with trainers and students after delivery in the two schools that took part in the initial feasibility testing.

Interviews were conducted with peer supporters using a semistructured topic guide to explore all parts of the interventions, particularly what did and did not work, with a view to identifying areas for potential refinement. Interviews with school staff explored the intervention delivery, methods and processes. Interviews with SMT members explored their awareness and understanding of the interventions, the information that they had received and their overall impressions of the intervention. Audio recordings of interviews were transcribed verbatim and anonymised at the point of transcription. This included removing personal names, school names and any other identifying information. Qualitative data analysis was facilitated by the use of NVivo 10 (QSR International, Warrington, UK), a qualitative data analysis software programme.

A framework analysis was employed. 32 A coding matrix was used to index and categorise data, with categories and themes being predetermined based on the research objectives (e.g. feasibility of delivery of the intervention components, acceptability of the intervention components to peer supporters) and with any unexpected themes identified being added. Data collected from the structured observations and self-assessments were organised in descriptive ‘chunks’ for each component/activity within the interventions. Identification of recurring and salient themes was examined against this framework. Two coders agreed on the framework and then referred to an additional two independent members of the research team to discuss/resolve any discrepancies and reach a consensus. To agree on refinements to intervention resources and delivery mechanisms, the results were shared with the Trial Management Group (TMG) and IDG.

Eligibility criteria and recruitment

The pilot trial was embedded within the 2014/15 delivery of the ASSIST intervention by PHW. As part of the Welsh Government’s Tobacco Control Action Plan, PHW was funded to deliver the ASSIST intervention to 50 schools a year. The Welsh Government provided PHW with a list of approximately 160 schools eligible for the ASSIST intervention out of the 220 secondary schools in Wales. The Welsh Government informed PHW that schools were selected on the basis of having a high percentage of children in receipt of free school meals (FSMs) and schools were in relatively deprived areas according to the Welsh Index of Multiple Deprivation (WIMD). 33 The Welsh Government did not provide the exact cut-off points applied for FSMs or the WIMD to exclude schools. From this list PHW recruited schools from the counties of Cardiff, Newport, Torfaen, Blaenau Gwent, Rhondda Cynon Taf, Merthyr Tydfil and Caerphilly, inviting those that had not received the ASSIST intervention in the past 2 years. Out of the 72 schools in these counties, 40 had not received the ASSIST intervention in the last 2 years and formed our sampling frame. These schools were sent a project information sheet, reply envelope and form asking them to contact PHW or the principal investigator if they wished to be involved in the study. Non-responders were followed up with a reminder and telephone call from the study manager (KM). All interested schools were visited by the study manager to discuss the study in more detail.

Consent

Head teachers signed a memorandum of understanding before taking part in the study describing the roles and responsibilities of both the intervention delivery and the research teams and the timeline of intervention delivery and assessments. Letters were sent to parents/guardians asking them to contact the school if they did not wish their child to participate in the trial. Parents who did not wish their child to participate were able to opt their child out of data collections. Written consent detailing the right to withdraw was sought from all participants. At all data collection points, age-appropriate information sheets were provided, together with a verbal explanation by researchers on the right to withdraw.

Sample size

As this was a pilot trial a power calculation was not required. The estimated sample size at baseline was 1440 students across 12 schools (n = 360 per arm), chosen to provide some information on variability within and between schools at baseline and follow-up. This sample was not anticipated to provide adequate power to detect a statistically significant difference across groups. However, the sample was used to indicate the likely response rates and permit estimates [with 95% confidence intervals (CIs)] of intraclass correlation coefficients (ICCs) of drug prevalence in anticipation of a larger cRCT. 34

Randomisation

Schools agreed to take part prior to randomisation. The study used an assignment ratio of 1 : 1 : 1 : 1. Allocation was conducted by the study statistician, blind to the identity of schools, and minimised on the median percentage of students in receipt of FSMs (below/above median) and median school size (below/above median) to balance the randomisation. Optimal allocation was used to carry out the randomisation. 35

Interventions

The +FRANK and FRANK friends interventions delivered in the external pilot cRCT are described in Tables 2 and 3, respectively. Delivery of the +FRANK and FRANK friends interventions occurred between September and November 2015. In the +FRANK arm, peer nomination was repeated to examine the proportion of students nominated in Year 8 who were renominated in Year 9.

Usual practice

Children participated in their usual personal, social, health and economic education (PSHE) lessons provided by the schools. All schools followed a national PSHE curriculum, which may include education on drug use and smoking.

Quantitative data collection process

The consent procedure and questionnaires were self-reported. All data were collected by fieldworkers. Questionnaires were completed in school halls or classrooms under examination conditions. Researchers, fieldworkers and dedicated school staff helped children with literacy problems and special educational needs. Baseline data collection took place between 17 September and 20 October 2014, prior to randomisation. Follow-up data collection took place 18 months later between 22 March and 5 May 2016. To increase response rates, additional collections were made for students who were absent. Schools were paid £300 for staff cover for the data collections after the 18-month follow-up.

Outcome measures

The main study outcomes were operationalised as progression criteria. Progression criteria were agreed by the TMG, TSC and NIHR Public Health Research (PHR) Research Funding Board. Assessment methods used a mix of quantitative and qualitative data. Table 4 shows the research questions that the progression criteria addressed, the data collection methods and the sources of data used.

Self-reported outcome measures

The self-report questionnaires included items to assess the anticipated primary outcome in a future full-scale trial, the lifetime prevalence of illicit drug use, using questions from the ALSPAC cohort. 36 At baseline, students were asked whether they had ever tried 10 drugs that had a > 1% prevalence in 13- to 14-year-olds in the 2013 Smoking, Drinking and Drug use survey,37 with an additional ‘other’ open response category. At follow-up an additional seven drugs were added. Street names were also provided for all drugs and a fictitious drug (semeron) was used to examine false responding. We also examined a number of potential secondary outcomes in this population: frequency of use of each drug in the past 12 months, last 30 days and last week; cannabis dependence using the Cannabis Abuse Screening Test (CAST);38,39 lifetime smoking and weekly smoking status (weekly defined as smoking at least one or more cigarette a week40); number of cigarettes smoked every day; nicotine dependence using the Fagerström Test for Nicotine Dependence (FTND)41 and Heaviness of Smoking Index (HSI);41,42 lifetime alcohol consumption; the frequency of heavy episodic alcohol use using the Adolescent Single Alcohol Question (A-SAQ), which is a modified version of the modified Single Alcohol Screening Questionnaire;43 and health-related quality of life, using the Child Health Utility-9 Dimensions (CHU-9D) measure. 44,45

Self-reported measures of intermediary factors

To examine the hypothesised mechanisms of action described in the logic models, we examined a number of intermediary factors. These included the perceived lifetime prevalence of drug use in Year 9, the frequency of drug offers, conversations with friends about drugs and visiting the Talk to FRANK website, ever having talked to a peer supporter about drugs, whether students would get help for themselves or a friend from the Talk to FRANK website if they had a problem and knowledge about drugs calculated as the number of correct answers from eight true or false questions about drugs (see Appendix 1, Table 16).

Statistical analysis

Statistical analyses were largely descriptive. The eligibility, recruitment and retention rates for schools and students were summarised using a Consolidated Standards of Reporting Trials (CONSORT) diagram (see Figure 7). 46,47 The data collected for trial participants were summarised by trial arm and combined across arms. To examine the acceptability of potential outcome measures the percentage of missing values are reported for all variables. Categorical variables were summarised using the percentage in each category. Numerical variables were summarised using the mean, standard deviation (SD) and a five-number summary (minimum, 25th percentile, median, 75th percentile, maximum). We present mean and median values to examine the shape of each distribution. We estimated the percentage of recanted responses, whereby individuals indicated having never used a drug at follow-up but indicated that they had used at baseline, as a measure of reliability. Comparisons were made between those who completed the study and those who dropped out of the study. ICCs were calculated. Exploratory effectiveness analyses using multilevel linear and logistic regression models adjusting for gender, age, FSM entitlement and residence with an adult in employment were conducted for indicative outcome and intermediary variables. We fitted an interaction term to examine effects in students who had and had not used drugs at baseline. If the ICC was < 1 × 10^–8, a single-level model was used. All analyses used intention-to-treat populations.

Cost analysis

We estimated the cost of delivering the +FRANK and FRANK friends interventions. The cost and cost-effectiveness of the ASSIST intervention has been previously reported. 48 We aimed to provide preliminary evidence on the likely affordability of these interventions and also pilot methods of data collection. We focused on additional costs, over and above those of usual practice, that would be incurred by the public sector (i.e. PHW and schools/education authorities). We did not track or cost the resources used in development or the feasibility testing of the +FRANK or FRANK friends interventions (i.e. stages 1 and 2) as once manuals and resources are developed these are sunk costs. We also excluded the costs of the ‘Training the Trainers’ session as these are artificially high in a pilot trial setting where the training costs are spread over a small number of schools. However, it is important to recognise that some costs associated with updates to the intervention and PHW staff training, because of staff turnover, would be needed if the intervention is used routinely.

We recorded the time spent and other costs of administrating (e.g. contacting schools, arranging venues) and delivering (e.g. intervention days and follow-up sessions) the intervention. The cost of delivering the intervention was predominantly for room hire, catering, PHW staff time, transport costs and intervention consumables (e.g. text messaging service). PHW estimated unit costs for PHW staff time and transport, as well as expenses incurred for room hire, catering, student transport (i.e. coach hire) and consumables.

Process evaluation

The process evaluation examined the feasibility and acceptability of the two interventions from the perspectives of peer supporters, schoolteachers, intervention delivery staff, parents and a public health commissioner. Researcher notes were taken at each stage of intervention delivery, including the training of intervention delivery staff, nomination of peer supporters, training of peer supporters and post-intervention delivery (to assess objectives 3a–c; see Chapter 1).

Two staff members conducted structured observations to assess whether the defined learning outcomes of all intervention activities were met. Observations of the delivery of all intervention activities, across all sites, were made by two members of the research team to examine the fidelity of intervention delivery.

In each school, one member of the SMT plus the contact teacher, head of year or PSHE lead were interviewed. Post-intervention interviews were also conducted with a purposive sample of peer supporters in schools with the highest and lowest prevalence of drug use. All interviews were audio recorded, fully transcribed and anonymised and electronically stored on a secure server. Table 5 provides an outline of the methods used at each stage of the process evaluation in the external pilot, as well as the issues examined.

Qualitative data analysis

Thematic analysis was employed in the qualitative analysis. 32 The deductive coding framework developed in stage 2 feasibility testing was adapted to examine the feasibility and acceptability of the intervention against progression criteria. We used a coding matrix to index and categorise data, with categories and themes being predetermined based on the research objectives (e.g. feasibility of delivery of the intervention components, acceptability of the intervention components to peer supporters), with flexibility maintained so that any unexpected themes identified from the data could also be added. A sample of six transcripts was coded and compared and regular data analysis meetings were held by the qualitative research team to discuss emerging themes and ensure consistency in coding. Analysis was conducted using NVivo 10 software to assist with the systematic coding of data to identify patterns in the narrative provided by teachers, peer supporters, students, parents and schools.

Changes to the protocol

There were no major changes to the protocol. 49 In the statistical analysis section we stated that we would adjust for baseline levels in the analysis of lifetime drug, smoking and alcohol use at the 18-month follow-up. However, we did not adjust for lifetime measures of use at baseline or conduct subgroup analyses in baseline users. This is because with lifetime measures adjustment creates a situation of perfect prediction. As baseline users can only remain users at 18 months, estimates for students who report lifetime use at baseline cannot be calculated. A request for a 3-month non-financial extension was granted towards the end of the study.

Phase 1: evidence review and stakeholder consultation

In line with Medical Research Council (MRC) guidance on developing complex interventions,50 we reviewed the existing literature. Two systematic reviews of school-based drug prevention programmes found small effects on cannabis use in the short term. 18,19 Researchers working on one RCT noted evidence of poor implementation of interventions21 and, in another peer-led programme, evidence was found of iatrogenic effects20 (see Chapter 1, Evidence on the effectiveness of school-based drug prevention, for further details).

Population-based studies on the prevalence of drug use in secondary school-aged children in the UK have shown that the lifetime prevalence of any illegal drug use in England doubled from 6.8% to 12.4% and then to 23.1% from 13 to 14 to 15 years, respectively. 28 The lifetime prevalence of drug use was > 1% only for cannabis, GGAs, ecstasy, poppers, cocaine, ketamine, mephedrone and magic mushrooms. Cannabis had the highest lifetime prevalence, which increased from 2.7% to 7.5% and then 18.7% from age 13 to 14 to 15 years, respectively. GGA prevalence was 3.5%, 4.0% and 4.4% for 13-, 14- and 15-year-olds, respectively. No equivalent data were available in Wales. This informed our decision to deliver the intervention to UK Year 9 students (aged 13–14 years) and focus the intervention content on cannabis and GGA use.

Consultations with young people and practitioners in drug charities noted a local issue with steroid use in older age groups, which was not apparent in prevalence data as these were gathered in England and did not sample from Welsh schools. This led us to include information on steroids in the interventions.

The consultations and focus groups with young people suggested that 13- to 14-year-olds were relatively familiar with the potentially harmful effects of drugs on health:

Like we all know weed is bad, we all know what it does to you as well.

Young person 6

Young people were less familiar with the potential legal consequences of being caught in possession of an illegal drug in the UK:

When it says unlimited fine, does that mean the police can just charge you?

Young person 9

The familiarity of young people with the harmful effects of drugs on health prompted us to expand the focus to include the harms associated with drugs being illegal and therefore unregulated. These included the possibility of unexpected effects brought about by consuming an unknown compound, of unknown purity and dose. Other concerns that young people voiced included the potential harmful effects of drug use on family relationships, future education and employment:

I mean that’s your mum, that’s one of your parents, they put a roof over your head. If you get drove away from them you don’t get food for yourself, you don’t get a roof over your head, you’re out on the streets. You don’t have anyone to get you a meal or look after you ‘cause you’re on your own.

Young person 29

’Cause then you’re getting a criminal record that’s stopping you from getting a job and loads of stuff.

Young person 19

A number of factors that might influence the engagement of students during peer supporter training were found in both the interviews with the ASSIST delivery team and the independent observations by the research team of the delivery of the intervention. In particular, flexibility in delivering intervention activities to different groups and the need for engaging, interactive content were noted:

We work to the same objectives, but in terms of how we run some activities, we might change them a bit . . . with different groups you know, how they react to a certain activity you might change it round to help the running of it.

Trainer 2

Making sure that they’re interactive . . . so they’re up and about, they get moving around, break off activities, um, just making it as interactive as possible.

Trainer 6

Phase 2: coproduction

During coproduction, the IDG reflected on findings from stage 1 to adapt the content from the ASSIST intervention or develop new content. In interviews with the ASSIST team it was noted that it was important to provide peer supporters with interesting and memorable facts about smoking that they could use in conversations with their peers:

So if we can give them facts that sort of link into what they could be talking about with their friends, it makes it easier for these conversations to happen. In ASSIST, one of the facts they always remember, is that smoking could affect your ability to get an erection. That is the one that sticks with them, and you might not have done the training for 10 weeks, and they will still remember that.

Trainer 2

In ASSIST, we know that young people will leave knowing the ingredients of a cigarette, long-term, short-term health effects, is it guaranteed. We know that you’d go up to any young person that had done the training and you’d ask them how many ingredients are in a cigarette and they’d be able to tell you.

Trainer 2

This led us to adapt information from the Talk to FRANK website about the risks of drug use into memorable factual statements. These key statements were then used across several activities within the peer supporter training and were added to the peer supporter diaries as a reminder.

Phase 3: prototyping

Expert peer review, independent from the IDG, by the lead author of the ASSIST RCT23 and the lead trainer at DECIPHer IMPACT, was used to examine and address key areas of uncertainty. The feedback led to refinements in the timing of the intervention activities and the presentation of instructions in the intervention manual.

Feedback provided by the trainers at the end of the two training sessions indicated that the training was well received and that some of the issues that were raised at the end of the first training session, such as concerns about the need for an encyclopaedic knowledge of drugs, had been addressed during the second training session. This was achieved by providing additional drug education during the session along with the inclusion of a ‘new to trainers’ worksheet that trainers could use to demonstrate to peer supporters the difficulty of keeping track of new drugs and street names. Suggestions raised by trainers for improving activities were added to the intervention materials during refinement prior to stage 2. A final training session was delivered to run through the finalised intervention materials and schedules in advance of stage 2 delivery in two schools so that trainers could become more familiar with the running order of the +FRANK and FRANK friends training days and the resources required for carrying out the activities.

+FRANK intervention

Seven structured observations of delivery were completed for the +FRANK intervention, eight self-assessments were completed by the trainers and 14 +FRANK peer supporters completed an evaluation form at the end of the off-site training day. At the end of intervention delivery (i.e. after the final follow-up), 12 peer supporters completed evaluation forms. Interviews were also conducted after intervention delivery with six peer supporters, three trainers (at the end of the off-site training and the last follow-up), two of the contact teachers and two members of the SMT.

FRANK friends

Fifteen structured observations were made of delivery of the FRANK friends intervention, nine self-assessments were completed by the FRANK friends trainers and 26 FRANK friends peer supporters completed an evaluation form at the end of the second off-site training day. At the end of intervention delivery, 21 peer supporters completed evaluation forms. Interviews and focus groups were also conducted after intervention delivery with 20 peer supporters, five trainers (at the end of off-site training and the last follow-up), two contact teachers and two members of the SMT. Five focus groups were conducted in the FRANK friends school.

Refinements to the +FRANK and FRANK friends interventions: what changed from stage 1

The outcomes of the process evaluation were presented to the TMG. The TMG then agreed some amendments to the +FRANK and FRANK friends interventions. Nearly all of the refinements were made to both the +FRANK intervention and the FRANK friends intervention as a result of the process evaluation undertaken at stage 2 (Table 7). Peer supporters were contacted by e-mail or text message after training with the contact details for the Talk to FRANK website and were sent updates when news items or new drugs were added to the website. Peer supporters were also sent reminders to have conversations and bring their diary to follow-up sessions. The training day agenda was reordered and the timings of activities were amended to accurately reflect the time recorded during observations. Activities deemed to be non-essential were changed to be ‘optional’ and could be delivered at the trainers’ discretion if time allowed.

Refinements unique to the +FRANK intervention included e-follow-up sessions 2 and 3 changing to in-person follow-up sessions in which trainers go into the school to deliver tasks. Follow-up session 4 was dropped and the content was covered in follow-up session 3.

Figures 5 and 6 show the refined intervention logic models for the +FRANK and FRANK friends interventions, respectively.

FIGURE 5.

+FRANK logic model.

FIGURE 6.

FRANK friends logic model. The ASSIST intervention is not given as part of the FRANK friends intervention.

Recruitment and retention

Forty schools were invited to take part in the external pilot cRCT (Figure 7). Because of delays in contracting and the requirement to deliver the intervention in the autumn 2014 term, we oversampled schools to reduce the time needed to recruit the required 12 schools. Eight schools declined to participate, with seven citing staff workload and another already having a smoking prevention intervention, Operation Smoke Storm,51 planned. Twelve schools did not return telephone or e-mail messages. Twenty schools responded that they were interested in participating. Two schools were unable to meet the study manager and the lead ASSIST trainer, one because of staff absence and the other not providing a reason. Eighteen schools were visited, with one unable to commit that year because of changes in staffing and one withdrawing as the amount of money offered to recompense for time was not considered to be enough. Two schools were recruited to take part in the feasibility testing of the interventions and two were held in reserve.

FIGURE 7.

Trial CONSORT diagram. a, Because of staff absence; b, because the amount of money offered to recompense for time was not considered to be enough; c, three students started home schooling between baseline and follow-up; d, one student was suspended from school during the follow-up data collection period.

Twelve schools were randomised into the four trial arms, three to the ASSIST intervention, three to the +FRANK intervention, three to the FRANK friends intervention and three to the usual practice arm. There were 1692 students Year 8 aged 13–14 years across the 12 schools. Of these, 125 (7.4%) were withdrawn by the schools, were opted out by parents, opted out themselves or were not at school when the main or absentee surveys took place. This left 1567 students (92.6% of those eligible) who completed the baseline assessment, with 347 attending ASSIST schools, 419 attending +FRANK schools, 440 attending FRANK friends schools and 361 attending usual practice schools.

All schools allocated to the intervention arms received the intervention as planned. All 12 schools took part in the 18-month follow-up. Across the four arms, 107 students (6.8% of the baseline respondents) did not take part in the follow-up, with 84 moving school, 20 not being at school when the survey was administered and three starting home schooling. This left 1460 (93.2%) students who completed the 18-month follow-up, 315 (90.8%) in the ASSIST arm, 391 (93.3%) in the +FRANK arm, 412 (93.6%) in the FRANK friends arm and 342 (94.7%) in the usual practice arm.

Characteristics of trial participants at baseline

Characteristics of the outcome measures at the 18-month follow-up

Of the 107 students who did not take part at the 18-month follow-up, 14 (13.1%) reported lifetime illicit drug use at baseline. Of the 1460 students retained, 51 (3.5%) had used illicit drugs at baseline, suggesting that drug use was more common in students who did not take part at follow-up.

Exploratory effectiveness analysis

We examined the effectiveness of the interventions on the indicative primary and secondary outcomes and intermediary variables (see Appendix 1, Tables 25–27). It was not possible to estimate effects for cannabis or GGA use in the past 30 days, cannabis dependency, weekly smoking status or ever having visited the Talk to FRANK website, as there were too few cases at baseline per arm.

The odds of lifetime drug use at the 18-month follow-up were marginally lower in the +FRANK arm than in the usual practice arm [n = 561, 12.4% vs. 13.4%; odds ratio (OR) 0.96, 95% CI 0.58 to 1.59]. The odds of lifetime drug use were lower in the FRANK friends arm than in the usual practice arm (n = 576, 9.3% vs. 13.4%; OR 0.70, 95% CI 0.39 to 1.24). The odds of lifetime drug use were also marginally lower in the ASSIST arm than in the usual practice arm (n = 497, 8.7% vs. 11.1%; OR 0.81, 95% CI 0.46 to 1.42). The overall direction of effects for the FRANK friends arm across the secondary outcomes and hypothesised intermediary variables indicated a potential positive effect; for the +FRANK and ASSIST arms the pattern was mixed.

In exploratory subgroup analyses, interaction terms indicated that the effects of the +FRANK and FRANK friends interventions were comparable in baseline lifetime drug users and baseline lifetime non-drug users (see Appendix 1, Table 28).

Reliability of measures of illicit drug use

We examined the possibility of false-positive responding by including a fictitious drug, semeron, in the measure of lifetime drug use. False-positive responding was very rare, with only two participants at baseline and three at follow-up saying that they had ever taken the fictitious drug.

We also examined the reliability of measures of lifetime drug use by calculating the proportion of lifetime drug use that was recanted, whereby baseline drug users responded that they had never used drugs at follow-up. Out of the 65 drug users at baseline, 27 (41.5%) recanted reports of lifetime use at follow-up. The level of recanting was 14.8% (n = 4) in the +FRANK arm and 29.6% (n = 8) in the FRANK friends arm, with levels of 48.2% (n = 13) in the usual practice arm and 7.4% (n = 2) in the ASSIST arm. There was a wide variation across sites and within arms; no recanting was found in five schools, one school each had one two, three and four cases, two schools had five cases and one school had seven cases. As two-thirds of lifetime drug use occurred after the baseline assessment, the removal of students who recanted had a minimal effect on drug use prevalence (removed 12.0%, not removed 11.6%). The low number of baseline users and students who recanted mean that these results should be interpreted with caution as small differences in absolute numbers can appear to be large differences in prevalence.

To further explore recanting we examined changes in other time-invariant variables. Between baseline and follow-up, the gender of seven students changed and 109 (7.0%) provided a different date of birth. After removing errors in the date of birth provided, including putting the date of the data collection or using an adjacent month but the correct date and year (e.g. 24 June 2002, 24 May 2002), 47 (3.0%) dates of birth changed between baseline and follow-up.

Assessment of costs

Process evaluation

Quantitative and qualitative data were used to determine whether or not the progression criteria were met (see Table 4 for criteria). Quantitative data are presented first, followed by qualitative data, organised according to the progression criteria that they address.

Assessing the acceptability of the +FRANK and FRANK friends interventions to trainers, students, parents and members of school staff and exploring the barriers to and facilitators of implementation

In stage 2 we conducted a process evaluation examining the delivery and receipt of the +FRANK and FRANK friends interventions in one school each. Two members of the research team made independent observations of all intervention activities, including follow-up sessions. In the +FRANK intervention we carried out seven observations, collected 34 evaluation forms and conducted 13 interviews with peer supporters and trainers. In the FRANK friends intervention, we carried out 15 observations, collected evaluation forms of the training from 47 peer supporters and trainers, conducted 29 interviews with peer supporters, trainers and teachers (including members of the SMT) and held five focus groups with peer supporters.

We assessed peer supporter attendance during the interventions. In the +FRANK school, 12 of the 14 peer supporters attended follow-up sessions 1 and 4, which were carried out in person. Only one peer supporter completed the e-follow-up sessions. In the FRANK friends school, between 16 and 21 of the 26 trained peer supporters attended the four follow-up sessions. Independent observations of intervention delivery and interviews with trainers found that some activities were too long, others were too short and the sequencing of activities could be improved.

Fidelity of delivery

In the +FRANK intervention, across the 15 activities, five were delivered in full, eight had minor deviations and two were not delivered at all. In the FRANK friends intervention, across the 25 activities, 13 were delivered in full, nine had minor deviations and three were not delivered at all. The observations suggested that some trainers had struggled to switch off from the coproduction stage.

Refining the interventions

The attendance records, interviews and structured observations led to changes in the +FRANK intervention. We removed the electronic follow-up sessions and dropped the final follow-up in which peer supporters received a certificate from the trainers. These were replaced with three face-to-face follow-ups. Other changes applied across both interventions, including changes in the content and sequencing of peer supporter training activities and highlighting in the instruction manual which activities were essential and which were optional. These barriers to delivery would not have been identified and the accompanying refinements would not have been made without this initial feasibility testing.

Assessing the feasibility and acceptability of the refined interventions to trainers, students, parents and school staff

The process evaluation involved 66 interviews with peer supporters, PHW intervention delivery staff, school staff including members of the SMT, PSHE leads and contact teachers and a commissioner of public health interventions. Independent structured observations of the delivery of all intervention activities (n = 195; n = 111 for the FRANK friends intervention) were made by two members of the research team.

In the +FRANK arm, 92% of the ASSIST peer supporters were recruited and retrained and 92% of the peer supporters reported at least one conversation and all reported a contact with intervention delivery staff. In the FRANK friends arm, 82% of the peer supporters were trained and 94% of the peer supporters reported at least one conversation and all reported a contact with intervention delivery staff. The success in meeting these criteria is likely to reflect that the ASSIST training materials that we adapted had been adequately tested and refined. Our study design also allowed for stakeholder concerns to be elicited and addressed before implementation.

The qualitative analysis suggested that most elements of the interventions were acceptable to students, teachers and parents:

• School setting. Teachers and parents noted that schools were an acceptable environment to deliver drug education to young people. Trainers and teachers thought that involving someone from the SMT and having a committed contact teacher were needed to facilitate intervention delivery.

• Peer nomination. The ASSIST peer nomination approach to intervention delivery was regarded as preferable and more credible by parents and the public health commissioner than didactic drug education in schools.

• Peer supporter role. Peer supporters reported feeling comfortable having conversations about drugs with their school friends, particularly with regard to the risks of drug use for health,. They thought that the training on communication skills would help in oral examinations and with their future employment. Teachers reported that they thought that the peer supporter training had increased some students’ confidence and they had involved them in other school activities.

• Follow-up sessions. Peer supporters noted that follow-up sessions were helpful and acted as a prompt to continue having conversations with peers. This may in part be because trainers were perceived to be friendly, credible and knowledgeable.

• The Talk to FRANK website. The website was perceived positively, with peer supporters noting that it was easy to access (including by mobile phone) and to understand. In particular, the A-Z feature was cited as a commonly used feature when they were asked a question about a drug they did not know.

Assessing the fidelity of delivery of the interventions by trainers

All +FRANK and FRANK friends intervention activities (100%) were delivered as intended. This is a substantial improvement in meeting the essential learning objectives from that found for intervention delivery in stage 2. This may be attributed to amendments made to the intervention manuals, content, timings and sequencing based on requests by trainers and discussions by the IDG. Trainers are likely to have improved their knowledge of the materials and delivery and the questions that are commonly asked by students. Trainers’ awareness of being observed and scored may also have improved adherence.

Comparing the feasibility and acceptability of the +FRANK and FRANK friends interventions

The progression criteria relating to feasibility and acceptability were met for the +FRANK and FRANK friends interventions. However, the process evaluation indicated that the hypothesised intervention logic may not hold as well for the +FRANK intervention as for the FRANK friends intervention. In the +FRANK schools, students completed the peer nomination process in Year 8 and Year 9. Around one-third of the +FRANK peer supporters were not nominated as being the most influential by their peers in Year 9. This meant that other students who were perceived to be influential in Year 9 were not trained to be peer supporters. This is a potential barrier to the diffusion of the messages from the Talk to FRANK website as the intervention logic model for the ASSIST intervention is predicated on peers being perceived as influential and a sufficient proportion (17.5% in the ASSIST intervention) of these influential peers being trained.

Although we were able to re-recruit 92% of Year 8 ASSIST peer supporters to reprise their role as a +FRANK peer supporter in Year 9, the number of peer supporters available to train in the +FRANK intervention is limited by the number of ASSIST peer supporters. In the FRANK friends intervention, if a parent or student does not consent to attend training, the student with the next highest number of nominations is invited, thus maintaining an ideal 17.5% of the year group trained. In the +FRANK intervention this cannot occur and so a lower proportion of students may be trained. Consequently, the diffusion of information is reduced as there would be a smaller number of peer supporters relative to other students.

Trainers reported feeling rushed in terms of delivering the content in the +FRANK intervention as training took place over 1 day, whereas the FRANK friends training took place over 2 days and the trainers could spend more time on activities and answering questions. One trainer and two peer supporters mentioned that more training on communication skills was needed. One trainer also noted that +FRANK peer supporters were disappointed that training took place over only 1 day as they were expecting the training to last for 2 days, as in the ASSIST intervention.

For these reasons, we recommend that the FRANK friends intervention is taken forward to a full-scale trial.

Assessing trial recruitment and retention rates

Of the 40 schools invited to participate, 12 provided no response and eight declined to participate. Of the 12 who provided no response, it is not clear whether they would have participated. Of the 18 schools visited by the research team, only one withdrew and another could not commit to the data collections. Of the 16 remaining schools, 12 took part in the cRCT, two were kept in reserve and two were used in the stage 2 feasibility testing. Because of delays in the contracting process, we deliberately oversampled schools to be invited to ensure that 12 schools were recruited by the start of the school term. In any subsequent trial we would have a longer recruitment period and would anticipate a higher rate of recruitment.

The 12 schools recruited were randomised and retained at the 18-month follow-up. In total, 93% of eligible students were recruited at baseline and retained at the 18-month follow-up. The retention of schools might be attributed to the signing of a memorandum of understanding by a member of the SMT. The memorandum of understanding clearly laid out the timing of the intervention delivery and data collections. Schools were also paid to supervise data collections. The high recruitment and retention rates were probably the result of the recruitment and data collection processes that we had refined during other school-based studies. We monitored recruitment and retention rates and conducted multiple follow-up collections for absent students.

Pilot outcome measures

We found low rates of missing data for almost all variables. The highest rate of incomplete data (23%) was for the CAST measure of cannabis dependency at baseline. As the lifetime prevalence of cannabis use was only 2.4%, this represented eight people not completing the measure. This measure may not have been completed because of a combination of the measure requiring six responses, a need for clearer signposting in the filter questions or lack of comprehension. At follow-up, 13.7% of responses were missing on the CAST. This reduction in the proportion of missing responses on the CAST may be the result of changes made to the follow-up questionnaire to improve the filtering of cannabis users or improvements in understanding of the instructions. There was some evidence of floor effects for the measures of nicotine dependence, with median scores at baseline of 0 on the HSI and 0.5 on the FTND. At follow-up, the median score was 2.0 on the FTND and 0.0 on the HSI. This suggests that there may be limited value in assessing nicotine dependence in this age group in the UK as levels of dependency are low.

The prevalence of lifetime drug use at baseline was 4.1%. The most commonly used drugs were cannabis (2.4%) and GGAs (2.2%); all other drugs had a prevalence of < 1%. At the 18-month follow-up, the prevalence of lifetime drug use was 11.6%. The ICC for illicit drug prevalence at follow-up for the comparison between usual practice and the FRANK friends intervention was 0.003 and for the comparison between usual practice and the +FRANK intervention was < 1 × 10^–8. These estimates suggest that, between 13 and 15 years of age, the prevalence of lifetime drug use nearly triples, which provides justification for our decision to deliver the interventions to this age group.

The most common drugs used were cannabis (8.0%), GGAs (4.0%), legal highs (1.7%) and cocaine (1%). Of those who used, the majority had done so in the past 12 months. Use of cannabis in the last 30 days (21.9%) was more common than use of GGA (5.3%) in the last 30 days, with cannabis (8.9%) and cocaine (1.2%) the only drugs that were used by > 1% of students in the last week. In total, 35% of the 117 students who reported ever trying cannabis screened positive for cannabis dependency. The pattern of lifetime prevalence of specific drugs, with cannabis and GGA being the most prevalent, replicates that found in the most recent population-based study in the UK. 6

The lifetime prevalence of illicit drug use at 18 months was 9.5% in the ASSIST arm, 9.7% in the FRANK friends arm, 15.1% in the +FRANK arm and 11.7% in the usual practice arm. There was an imbalance in baseline risk factors for drug use, with living without an adult in employment53 and in social housing54,55 more common in the +FRANK arm than in the other arms. This imbalance is likely to be attributable to the small number of schools per arm. In a larger full-scale trial, a larger number of schools would be randomised into each arm, thus reducing the potential for imbalances. As the sample size in this study was not determined by a formal power calculation but by convenience, effect sizes should be interpreted with caution as there is a risk of a type 2 error with underpowered comparisons. In an adequately powered trial we could be more confident that a sufficient number of students were included in the analysis to detect a difference across arms.

Of the 65 drug users at baseline, 27 (41.5%) recanted use in the follow-up survey. The recanting of lifetime drug use has been found in a number of prospective cohort studies. 56,57 The rate that we found was slightly lower than those recorded in other UK-based studies, such as the Belfast Youth Development Study (63.8% for 10 illicit drugs assessed at between 12 and 15 years of age),58 but higher than those in the ALSPAC birth cohort [14.4% cannabis, 25.2% other illicit drugs (cocaine, amphetamines, inhalants, sedatives, hallucinogens, opioids), assessed at between 14 and 18 years of age],59 although our survey was undertaken with students of a different age range. These studies found that recanting is less common in adolescents reporting higher levels of cannabis use. 58,59 As prevalence was lower in this study than in the previous studies, partly because of population-level declines in the prevalence of drug use,58,59 selection of a younger age group than in these analyses and a shorter follow-up period, we might expect higher rates of recanting in our study. Importantly, we found the highest rates of recanting in the usual practice arm, suggesting that this was not attributable to performance bias, in which participants change responses after exposure to the intervention.

It has been suggested that recanting is less likely to be the result of intentional distortion but rather the result of poor comprehension, forgetfulness or even carelessness. 56,57 Consistent with this suggestion, we found that between baseline and follow-up, seven students’ gender changed and 47 (3.0%) students changed their date of birth. An alternative explanation is that recanting may be logical. The perception by students of whether their use of a drug constituted ‘trying it’, or in the case of GGAs ‘to get high’, may change with age, comprehension and exposure to drug use. One suggestion for addressing the issue of recanting would be to use objective biomarkers of drug exposure. Unfortunately, objective measures of cannabis use have a poor sensitivity in population surveys, even for heavy daily users, over periods of > 3 months. 60,61 This would preclude identification of a significant proportion of lifetime cannabis use in our study as around 40% had not used cannabis in the past 12 months. This is in addition to other problems of using biomarkers of lifetime drug use, including there being no test currently available for all controlled substances if taken > 3 months ago and there being no test at all for novel psychoactive substances. These limitations are in addition to the likely pragmatic, methodological (e.g. response bias) and ethical (e.g. acceptability to students and parents) barriers to collecting hair samples on thousands of young people. For these reasons, self-report measures have recently been described as still representing the ‘gold standard’ in assessing the use of illicit drugs. 61

Recording the delivery costs and piloting methods for assessing cost-effectiveness

The interventions were developed and delivered at an affordable average cost per school. The estimated cost per school for the +FRANK intervention was £1942 (£13.87 per student) and for the FRANK friends intervention was £3041 (£20.69 per student). It is important to note that schools could not receive the +FRANK intervention alone as there is a requirement to receive the ASSIST intervention first. As the ASSIST intervention is of the same intensity as the FRANK friends intervention it has the same cost implications as the FRANK friends intervention. This means that the approximate costs of the ASSIST plus +FRANK interventions are those of the FRANK friends plus +FRANK interventions (£4983).

Assessments of the cost-effectiveness of school-based drug prevention interventions are relatively scarce. When economic evaluations have been performed, cost-effectiveness analyses are available only for studies conducted in the USA,62 and neither of the two peer-led programmes20,21 have examined cost-effectiveness. Our study provides preliminary information on the costs borne by the NHS, as public health services reside in the NHS in Wales. The study has demonstrated that it is feasible to conduct an economic evaluation alongside a pilot cRCT of a school-based intervention to prevent drug use. However, to fully judge the cost-effectiveness of the intervention the within-trial estimates of intervention costs, and the prevalence of drug use, would need to be extrapolated over the lifetime of participants. 63