Undertaking Studies Within A Trial to evaluate recruitment and retention strategies for randomised controlled trials: lessons learnt from the PROMETHEUS research programme

Article history

This issue of the Health Technology Assessment journal series contains a project commissioned/managed by Better Methods, Better Research (BMBR). The Medical Research Council (MRC) is working with NIHR to deliver the single joint health strategy, which includes BMBR as part of the delivery model. This was launched as the Methodology Research Programme (MRP) in 2008. MRC is lead funding partner for MRP and part of this programme is the joint MRC–NIHR funding panel ‘The Methodology Research Programme Panel’.

To strengthen the evidence base for health research, BMBR aims to ensure optimal research methods are being used to advance biomedical, health and care focused research, policy and delivery. In addition to the MRC and NIHR funding partners, the BMBR takes into account the needs of other stakeholders including the devolved administrations, industry R&D, and regulatory/ advisory agencies and other public bodies. The BMBR funds investigator-led and needs-led research proposals from across the UK that develops and delivers ways to improve the research methods being used by others. In addition to the standard MRC terms and conditions on governance of good research practice, projects commissions/ managed by BMBR are expected to provide a detailed report on the research findings and may publish the findings in the Health Technology Assessment, if supported by NIHR funds.

The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Copyright statement

Copyright © 2024 Parker et al. This work was produced by Parker et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. See: https://creativecommons.org/licenses/by/4.0/. For attribution the title, original author(s), the publication source – NIHR Journals Library, and the DOI of the publication must be cited.

2024 Parker et al.

What is covered in this report?

This report focuses on randomised Studies Within A Trial (SWATs) testing the effectiveness of recruitment and retention strategies. The structure of this report begins by presenting the background to the PROMoting THE Use of Studies Within A Trial (PROMETHEUS) programme (see Chapter 1). In Chapter 2, we present the methods used in PROMETHEUS, followed by the results (see Chapter 3) and then the discussion of the programme (see Chapter 4).

Subsequent chapters focus on the lessons learnt from PROMETHEUS and how this can inform future work focused on SWATs. This includes lessons learnt for; patient and public involvement (PPI) (see Chapter 5), the development of SWATs and its funding (see Chapter 6), and ethical and other governance approvals (see Chapter 7). In Chapter 8, we focus on different approaches to implementing SWATs, using case studies of standalone SWATs, as well as case studies of co-ordinated SWATs, undertaken in a co-ordinated way across multiple host trials simultaneously. Chapter 9 reports lessons learnt for analysing SWATs, and Chapter 10 on lessons learnt for reporting SWATs.

At each point, we provide an outline of key points to consider, using specific examples of work from the PROMETHEUS programme to illustrate our point. Wherever relevant, we highlight what different stakeholder groups such as trial management teams, PPI partners, trial oversight committees, those overseeing trial governance, statisticians, journals, and peer reviewers can learn from the PROMETHEUS experience.

Finally, we discuss recommendations for future research practice, direction and support (see Chapter 11), and then present conclusions (see Chapter 12).

Who should use this report?

This report has been written for anyone with an interest in using randomised controlled trials to inform evidence-based health and social care. Lessons learnt that we detail in this report will be of use to those interested in how the design and delivery of future trials can be improved to benefit patient health and social care. This includes individuals and organisations involved in funding trials; those who design, plan and undertake trials such as Chief Investigators and Trial Managers; trial oversight committees; those providing ethical and other regulatory approvals; patient and public partners in trials; trial infrastructure organisations supporting trial delivery such as the UK Clinical Research Networks (CRNs), Trials Units and the Research Design Service. The report will be of use to those recruiting and retaining trial participants, as well as those interested in trials methodology.

The report has been written for those with an understanding of trials who are new to SWATs; those with some experience of SWATs who want to learn more; and those with SWAT experience who may wish to use our experience to consider possible methodological innovations of their own. This report is also written for journal editors publishing SWATs, as well as those undertaking peer reviews of SWATs.

The importance of randomised controlled trials

Randomised controlled trials (‘trials’) attempt to improve the health and care of populations and minimise the potentially harmful effects of treatments and interventions. Trials are undertaken on the basis that there are uncertainties about the effects of treatments, and such treatments are tested to adequately reduce these uncertainties. Consequently, trials are usually accepted as the foundation for evidence-based practice,1 the most complete paradigm for delivering safe and effective health and care for patients and citizens. Globally in 2022, there were more than 400,000 trials being undertaken across 220 countries. 2 The UK National Institute for Health and Care Research (NIHR) invested £250 million on research, predominantly in the form of trials, in 2019/2020, and £373 million in 2020–1. 3,4 To be efficient, trials need to recruit large numbers of participants quickly – often in excess of 1000 participants. 5 To have external validity, a generalisable sample needs to be recruited from all relevant patient and population groups, and to avoid post-randomisation selection bias, retention needs to be kept as high as possible.

Recruitment and retention difficulties in trials

Despite the importance of trials, they are susceptible to both poor participant recruitment and retention. As a consequence of this, many trials fail to recruit to both time and budget, often with lower participant retention than anticipated. 5 The cost of poor participant recruitment can be huge,6 contributing to significant research waste. 7,8 Furthermore, poor recruitment has a detrimental impact on mortality and morbidity, as this increases the time to the implementation of effective care and treatments. An analysis of the randomised evaluation of COVID-19 therapy (RECOVERY) trial, which recruited only 10% of eligible patients, showed that had recruitment been 50% or more of eligible patients, there would have been a significant improvement in lives saved due to earlier reporting. 9 Similarly, poor participant retention, which is often overshadowed by the research focus of poor participant recruitment,10 can also be detrimental to the success of a study. Poor retention may lead to the underpowering of a study and cause the estimates of an intervention’s effect to be biased. 11 Aligned with this, a priority-setting exercise involving 85% of UK Clinical Trials Units (CTUs) placed recruitment and retention as the top two priorities for methodological research,12 cementing the need to identify and implement tools within trials that help to improve both.

Study Within A Trial methodology

To date, there has been a distinct lack of robust evidence to inform trial design and conduct, including for recruitment and retention. However, there is an increasing international movement to improve the efficiency and successful delivery of trials by using robust research methods. 13–15 These robust studies use methods that are embedded within real ‘host’ trials and are referred to as ‘SWATs’. A SWAT has been defined as a: ‘self-contained study that has been embedded within a host trial with the aim of evaluating or exploring alternative ways of delivering or organising a particular trial process’. 15

Treweek et al. (2018)15 outline several key features of a SWAT. Firstly, SWATs aim to resolve key uncertainties around trial processes. Secondly, a SWAT is embedded within a host trial, but it should have a formal protocol, in the same way as the host trial. Thirdly, a SWAT must not affect the scientific integrity of the host trial, its rationale or outcome measures. Fourthly, a SWAT can be either tested in a single trial or embedded across multiple host trials in a co-ordinated way. Finally, SWATs provide evidence to inform the design and conduct of future trials; however, they may also produce evidence to inform decisions about the host trial.

SWATs can adopt a range of different methods including qualitative,16,17 mixed methods18 and before-and-after designs. 19 SWATs can be used to understand and refine implementation processes. 20 SWATs can also be non-randomised21 or randomised. 22

Addressing recruitment and retention difficulties using SWATs

Randomised SWATs are the most rigorous methods for evaluating the effectiveness of strategies for improving participant recruitment and retention in trials. 23 Two Cochrane reviews identified 150 SWATs of strategies to increase recruitment and/or retention in trials;23,24 however, effective, evidence-based strategies are rare and where evaluations exist, they tend to occur in the context of a single trial, meaning that they usually have limited statistical power and their effects across different trial contexts are unclear. 23 The most recently published Cochrane review on retention interventions concluded that there was no high-certainty evidence for any of the evaluated strategies, as assessed by Grading of Recommendations, Assessment, Development and Evaluations (GRADE). 24

Prior to the PROMETHEUS programme, the Medical Research Council (MRC) START programme was launched25,26 – a feasibility study that successfully developed the conceptual, methodological and logistical framework to improve recruitment through the embedding of two recruitment strategies in 12 host trials in primary care and developed reporting guidelines for embedded trials. 25,27 In addition, since 2014, the Health Research Board – Trials Methodology Research Network (HRB-TMRN) Ireland has supported and funded Irish researchers to conduct methodological studies to improve the efficient conduct of future trials including SWATs, offering funding of up to €25,000 for evaluations. 28 More recently, the NIHR Health Technology Assessment (HTA) programme is encouraging applications to embed at least one high-quality SWAT with funding of up to £10,000 per host trial. 29 The TRials Engagement in Children and Adolescents (TRECA) study, also funded by the NIHR, aimed to develop multimedia interventions to improve the quality of decision-making about recruitment to trials involving children and young people with long-term conditions and subsequently tested them using SWATs. 30

Aims and objectives

The PROMETHEUS programme aimed to build on the work of these previous initiatives by making the embedding of recruitment and retention SWATs within host trials standard practice across multiple CTUs, working to at least double the number of SWATs produced by the MRC START project, given a similar level of funding. This was to be achieved through pump-priming and facilitating the start of at least 25 SWATs within 30 months. The ultimate aim was to make the inclusion of SWATs routine practice when conducting a trial in a CTU.

PROMETHEUS preparatory work

Prior to the initiation of the PROMETHEUS programme, a network of 10 CTUs, 1 primary care research centre in the UK and the HRB-TMRN in the Republic of Ireland was established, each of whom committed to embedding either a randomised controlled recruitment and/or retention SWAT within at least two host trials.

Promising recruitment and retention strategies (which had some evidence of benefit but with substantial uncertainty) were identified from a variety of sources. These sources included Cochrane systematic reviews,23,31 the SWAT Repository Store,32 and the priorities of recruitment and retention strategies identified by CTUs. 33 In addition, the PRioRiTy list of top 10 unanswered questions on trial recruitment, for which there is no current evidence, was also reviewed. 34 A PPI panel was also convened to highlight the top priority strategies for evaluation (see Chapter 5).

The identified strategies were then prioritised if they met one or more of the following criteria:

1. Strategies that had already been evaluated with results published in peer-reviewed journal publications.

2. Strategies currently under evaluation.

3. Easy to implement strategies, that is those requiring little additional resource (input or cost) from the host trial, as assessed by the PROMETHEUS investigators. Such strategies might not have a large impact; however, they might still make a useful contribution to the evidence base, based on marginal gains and cost effectiveness.

4. Had the potential to significantly impact participant retention or recruitment (which are often the more challenging, expensive strategies to implement).

5. Strategies identified by host trial teams as suitable for testing in a SWAT within their trial.

Prioritisation decisions were made through group discussion and consensus. These priorities formed an initial strategy priority list of seven recruitment and eight retention strategies (Table 1). The priority of these strategies was reassessed and rearranged accordingly throughout the programme, based on emerging SWAT evidence.

Eligibility criteria

SWAT interventions

PROMETHEUS prioritised a broad list of recruitment and retention strategies that could be evaluated (see Table 1); however, host trials could also evaluate their own strategies if they wished to. Support included assistance in writing SWAT protocols; provision of templates and guidance in achieving Research Ethics Committee (REC) and Health Research Authority (HRA) approvals; guidance on writing and submitting SWATs for publication.

PROMETHEUS programme outcomes

PROMETHEUS was designed to analyse the effectiveness of recruitment and retention interventions within the context of a single host trial, as well as across a range of different trials through the synthesis of similar SWATs results.

Sample size

The programme aimed to implement at least 25 SWATs across eligible host trials. Generally, as the sample size of SWATs are driven by the size of the host trial, single SWATs are often underpowered; a known and accepted attribute of most SWATs is that while sample size calculations may be made, formal power calculations are not required. 36 Instead, the SWAT’s sample size is usually driven by the host trial. Therefore, it was always planned that, where appropriate, SWAT intervention results were to be aggregated. To reduce study heterogeneity, studies were encouraged to follow common SWAT protocols to promote homogeneity in interventions in readiness for pooling data (see Chapter 9). This aggregation of SWATs was intended to help with the detection of small differences and provide evidence as to whether results might generalise to a range of contexts, providing evidence with greater external validity.

Host trial recruitment

Eligible host trials were largely identified and recruited through a combination of the programme collaborators, advertisement on the University of York Trials Unit (YTU) web page, e-mails to all registered UK CTUs and conference presentations such as at the International Clinical Trials Methodology Conference 2019. Following their identification, trial teams were provided with a PROMETHEUS Information Sheet, invited to submit an expression of interest form, and apply for funding from the programme for up to £5000 per SWAT embedded in their host trial. The funding application consisted of submitting a SWAT protocol, including a project timetable and an outline of costs. To support the development of the SWAT protocols, host trial teams were provided with the opportunity to meet with PROMETHEUS team members for support to determine an appropriate recruitment and/or retention strategy.

Two independent members of the PROMETHEUS programme peer-reviewed each host trial application and protocol to ensure methodologically robust replicable research was planned. Reviewers were asked to report their peer review comments and scores using a Peer Review Assessment Form, which was adapted from the peer review form used by the HRB-TMRN. 28 Peer reviewers were asked to comment on the following:

1. eligibility;

2. priority and scientific quality;

3. costings; and

4. overall rating of the application.

Randomisation

Typically, randomisation was completed by the host trial team and therefore the methods of randomisation, allocation concealment, and implementation were applied at the level of the host trial.

Blinding

Often it is deemed that SWATs do not require individuals to provide participation consent due to them being generally low risk, rarely imposing additional burden on a participant and due to potential risks of confusing the participant as to what they are consenting to. 23 As such, participants were not blind to their received intervention [e.g. a thank-you card or short message service (SMS)], but they were unaware of their participation in the SWAT.

Data collection methods

Studies Within A Trial data were collected in line with both the relevant SWAT and host trial protocols. The PROMETHEUS team provided trial teams with support to determine appropriate data collection methods where required. The funded teams were asked to provide the PROMETHEUS team with the data for each SWAT, with the aim of collating these in pooled analyses.

Statistical methods

To enable pooling of findings, a standardised framework was established, similar to that of MRC START. 25,27 Template protocols with standard outcomes and a template Statistical Analysis Plan (SAP) were designed. Trial teams were encouraged to analyse the outcomes in terms of absolute differences in recruitment and retention rates, as appropriate and to provide a cost per recruited or retained participant where possible. The primary outcome measure was compared across intervention and control arms using logistic regression. For SWATs of retention, teams also looked at elements such as: time to, and completeness of, and responses to follow-up outcome measures. Where possible, odds ratios from multiple SWATs on the same or similar interventions were combined using a random-effects meta-analysis, ideally using a one-stage approach if the individual patient-level data were available.

Data monitoring

A Project Management Group (PMG) was established to oversee the management of the SWATs work consisting of the project Statistician, Research Fellows, and other co-applicants, and was chaired by the Chief Investigator (author DJT). The role of the PMG was to monitor all aspects of the conduct and progress of the study. The PMG met quarterly by teleconference, with annual face-to-face meetings where feasible, meeting more frequently when there was a need to monitor the programme’s progress more closely.

Harms

Due to the nature of the SWAT strategies, no strategy was likely to be responsible for any health-related adverse events. Therefore, adverse event reporting and review remained with the host trial teams with data not being collected for the PROMETHEUS programme.

Auditing

The PROMETHEUS team maintained contact with each of the host trials throughout the duration of their SWAT conduct, requesting and routinely monitoring their SWAT progress.

Research ethics approval

Each individual SWAT obtained approval from the host trial’s REC, and institutional governance committees as needed. This approval was sought as part of the initial study application or as an amendment, depending on the status of the study at the time of SWAT implementation and/or the nature of the SWAT. Careful consideration was given, during the application as to whether informed consent was required for SWAT participation. Most SWATs also required approval from National Health Service (NHS) site Research and Development (R&D) departments, in line with HRA procedure, prior to implementation. Given that participants may not have provided informed consent for SWAT participation, patient identifiable information was held by the host trial and not released as part of the SWAT data set.

Consent or assent

Participants were not informed of their involvement, as it was not appropriate and may have affected the host trial outcomes by resentful demoralisation. 37 Providing the SWAT did not involve knowingly withholding pertinent information about participant involvement within it, and involvement was non-invasive, it was deemed appropriate that patients were not informed about the SWAT and informed consent for involvement was not obtained.

Confidentiality

All SWAT data transferred to the PROMETHEUS team via trial teams was done anonymously; for instance, by removing identifiers such as date of birth (e.g. simply putting age) and participant identity number (e.g. hospital number) and then randomly sorting the data. This ensured that it would not be possible to re-identify participants in the data set, in line with General Data Protection Regulation (GDPR) requirements with individual participants being identified by their host trial identification number only, or linked SWAT identification number if the host trial identification were removed. All electronic records were stored on a secure server.

Access to data

PROMETHEUS access to the data for each individual SWAT was obtained via signed data-sharing agreements in line with the principles for access to, and use of, MRC-funded research data report,38 negotiated at the outset. Ongoing access to the data depended on agreement with the individual host trial teams, with agreements about authorship and dissemination of results from the individual studies and the combined data set. Subject to the consent of the host trial teams, primary data were made available in an anonymised format suitable for release in the public domain.

Host trials funded, numbers of recruitment Studies Within A Trial funded, number of retention Studies Within A Trial funded

In total, the PROMETHEUS programme supported 42 SWATs, which were implemented within 31 host trials, across 13 CTUs and Research Centres (Figure 1).

FIGURE 1.

Flow chart of SWATs funded by the PROMETHEUS programme.

Collectively these 31 host trials spanned 17 research areas (Table 2; see Appendix 1 for a full list of the funded host trials and their characteristics). It is important to note that not all participants in host trials were available for nor included in the SWATs. Five of the host trials implemented more than 1 SWAT, a further 4 implemented a factorial design SWAT and 12 were involved in the co-ordinated evaluation of 2 SWAT strategies.

In total, 12 of the funded SWATs evaluated recruitment strategies (see Results: findings from the funded SWATs, Tables 4 and 5) and 30 SWATs evaluated retention strategies (see Results: findings from the funded SWATs, Tables 6 and 7).

SWAT costs

Results: findings from the funded SWATs

Summary of main findings from PROMETHEUS

The PROMETHEUS programme successfully embedded 42 SWATs within 31 host RCTs, exceeding the original target of 25 SWATs. This is the biggest single effort to generate SWAT evidence in the world, representing a substantial increase in the global methodological evidence base. The PROMETHEUS-funded SWATs represent an increase of 18% (12/68) more SWATs to the Cochrane systematic review of recruitment strategies,23 and 79% (30/38) more SWATs to the Cochrane systematic review of retention strategies. 24

The main findings from the PROMETHEUS SWATs reported to date show that for recruitment, there was no evidence of a significant difference in recruitment rates in any of the strategies tested. For retention, we found that pre-notification of trial participants by sending a card may result in a slight increase in attendance at a face-to-face primary outcome measurement visit; and that sending a pre-notification letter or e-mail before sending a self-report questionnaire by post increases response rates, compared with no pre-notification. We also found that participants randomised to receive a personalised text message were more likely to return their initial postal questionnaire than those who received a non-personalised text message; and that men aged under 65 years were most likely to return their initial questionnaire if they received a personalised text message. Another SWAT comparing personalised text messages versus non-personalised text messages, found evidence that a personalised text would result in fewer telephone follow-ups, which were more resource intensive. When participants were randomised to receive a pen compared to no pen with their postal questionnaire, combined results from PROMETHEUS together with other published SWATs found that including a pen probably increases retention and response rate (pooled OR 1.21, 95% CI 1.09 to 1.35). One SWAT showed evidence of an adverse impact on retention when printing the trial primary outcome on pink paper compared to printing the primary outcome on white paper; this showed a decreased response, and it was also more burdensome to collect postal data as it increased the need for reminders. The other strategies tested showed no evidence of a significant difference in retention rates.

This was possible to fund more SWATs than initially anticipated due to the majority of the SWATs costing less than the proposed funding limit, with the mean SWAT cost being £3535; less than the proposed £5000. We have shown that typically SWATs can cost less than the £10,000 SWAT funding made available by the HTA. Therefore, PROMETHEUS is currently the largest programme of work to act as a central co-ordination point to offer both funding and practical support for the embedding of SWATs, independently contributing the largest amount of evidence to the recruitment and retention strategy evidence base. These SWATs were collectively implemented within a large number of host trials across multiple CTUs, demonstrating the wide reach of the programme. The programme’s success confirms the feasibility of implementing methodological research within a vast range of research areas when appropriate resource and infrastructure support is made available. It also confirms the feasibility and acceptability among trial teams of conducting co-ordinated SWATs, enabling a more rapid evaluation of recruitment and retention strategies.

As host trial teams were often inexperienced in conducting and implementing SWATs, the PROMETHEUS programme acted as an invaluable co-ordination point, providing teams with the confidence, knowledge and resources to do so. Furthermore, the programme identified that the lack of SWAT funding was often a barrier to implementation; a concept which was reinforced as the number of conducted SWATs increased following the introduction of PROMETHEUS funding. However, despite providing both funding and guidance, the encouragement of external researchers to conduct a SWAT proved difficult; with the majority of the SWATs being conducted by both YTU (18 SWATs, 43%) and the CTUs which PROMETHEUS co-applicants were associated with (16 SWATs, 38%). It is important to place this seeming lack of poor external engagement in context, given that the original PROMETHEUS focus was to support our collaborating network of CTUs and centres, who had already committed to starting at least two SWATs.

Strengths

Five of the host trials implemented more than one SWAT, and a further four implemented a factorial design SWAT, allowing for the assessment of two SWATs in a co-ordinated way. This finding demonstrates that, within an individual host trial, there is often a capacity to address more than one SWAT question, either separately (such as testing a strategy aimed at recruitment during the recruitment phase; and then testing a retention strategy during the follow-up phase) or simultaneously, using a factorial design. This suggests that there is capacity to significantly speed up and strengthen the evidence base through teams undertaking more than one SWAT in their trials where relevant. Another pathway to speeding up and strengthening the evidence base is through the co-ordinated SWAT approach that we pioneered, where the same recruitment or retention strategy is tested across multiple host trials simultaneously. Thus, our work highlights that given finite resources, robust evidence can be generated at speed in two key ways: by doing more SWATs in the same host trial, or by testing the same SWAT question across different host trials in a co-ordinated way.

Aside from the provision of SWAT funding, the work undertaken within PROMETHEUS also presents added value through working with CTUs, which are the key centres in the UK supporting the delivery of trials, to support the uptake of SWATs. Uniquely, PROMETHEUS focused on generating evidence for trial recruitment as well as retention, both of which are crucial for the successful delivery of trials, but which until PROMETHEUS had not been addressed together. Despite similar funding as other programmes such as MRC START (which funded 12 SWATs) and TRECA (which funded 6), PROMETHEUS has been able to generate more than three and seven times the number of SWATs, respectively, adding substantial, much-needed evidence to support trial process decision-making.

Limitations

Our experience of five funded SWATs failing to complete highlights the reality of SWATs being entirely subject to the fortunes of the host trials; if the host trial stops for any reason or changes its mode of follow-up, it places the SWAT in jeopardy. One study commenced the SWAT but technical errors in the software associated with the intervention (text messaging) resulted in a significant proportion of participants not receiving the intervention and so the SWAT had to be terminated. For two other SWATs recruitment to the host trial was stopped prematurely (one on direction of the associated oversight committees; one by the Funder) and so planned retention SWATs could not proceed. Finally, although we had awarded funding to all of our host trials prior to the COVID-19 pandemic proliferating in the UK in March 2020, the widespread societal changes that this forced resulted in the recruitment pathway of many of the PROMETHEUS host trials either pausing or altering their study pathways. As a result, 2 SWATs did not proceed, and 10 others remain in progress at the time of writing this report. We will work with the host trial teams of the ongoing SWATs to ensure their findings appropriately contribute to the evidence base.

Further to this, the funded SWATs evaluated a wide range of recruitment and retention strategies. While this led to an overall increase in the evidence base, the evidence generated did not allow for enough replications to have been conducted to conclude the effectiveness of all SWAT strategies evaluated via meta-analyses. This limited our ability to answer most SWAT questions definitively. Despite this, significant progress has been made with some SWATs (e.g. pens for retention) for which only a limited number of further replications in specific populations (male populations, younger adult populations) are now required to reach a definitive evaluation of strategy effectiveness. The development of the Trial Forge SWAT Network should enable additional replications to be completed more rapidly to fully answer these SWAT questions.

We found a greater interest and conduct of SWATs evaluating retention, rather than recruitment strategies. This may be due to retention SWATs being potentially less challenging to undertake than recruitment SWATs, as logistically, there is likely to be more time to introduce a retention strategy when there are multiple follow-up time points. Recruitment SWATs may also present additional challenges, such as the inaccurate perception that multiple levels of consent are required from individuals being approached to enter a trial (e.g. consent to be invited into the trial, consent to be included in the SWAT, and consent to be enrolled into the trial). Alternatively, there may also be greater time pressure associated with the embedding of a recruitment rather than retention SWAT, due to the additional tasks of site set-up. Lessons learnt from the PROMETHEUS programme are also discussed by Clark et al. (2022). 108

As all the funded SWATs were also conducted within the UK, the SWAT evidence generated through PROMETHEUS is not necessarily applicable to populations within other countries, limiting the reach of the programme.

We included only randomised host trials, which may have limited our ability to identify variation in effectiveness of recruitment and retention strategies across different research contexts. However, we believe that focusing on randomised controlled trials in this particular programme was appropriate, since trials are particularly difficult to recruit to, with randomisation and strong treatment preferences leading to issues around both clinician and participant equipoise, something not experienced in non-randomised studies. A direction for future research is to include non-randomised studies as host studies. We also focused only on randomised SWATs. Non-randomised SWATs such as qualitative studies, observational studies, and surveys have a substantial role to play in improving the evidence base for recruitment and retention. One strong way that non-randomised SWATs can help with the evidence base is through process evaluations, which might include interviews to elicit participants’ experiences of being exposed to a recruitment strategy, as well as the experiences of patients who decline the invitation to participate in a trial. This will be crucial to include in future SWATs to ensure both effectiveness in terms of recruitment and retention but also acceptability for participants.

Comparison with existing programmes of SWAT research: Medical Research Council START and TRECA

Building on and improving on the SWAT work initiated by MRC START, which established the feasibility of testing recruitment strategies across multiple host trials,25,26 as well as TRECA,30 which aimed to undertake a programme of SWATs in trials recruiting children and young people, PROMETHEUS has successfully undertaken more SWATs at a faster pace and successfully disseminated findings. However, PROMETHEUS faced challenges that were similar to those in MRC START, such as a large proportion of the SWATs being undertaken by researchers linked with the PROMETHEUS team. There is a range of reasons for the success of PROMETHEUS, which includes prior learning on SWATs gained from MRC START and other work undertaken by its collaborators, the financial support to undertake SWATs, as well as the support mechanism provided to host trial teams from a well-established registered CTU. This suggests that speed and efficiency for undertaking SWATs can be improved further, but it is important that the momentum and skill set of staff are maintained.

Patient and public involvement

Patient and public involvement was an important part of this programme, both for the overarching PROMETHEUS programme and for the SWATs undertaken. We actively involved PPI members in identifying, prioritising and implementing the SWATs, and we have dedicated Chapter 5 to outline the PPI undertaken, as well as lessons learnt for the trials community, and PPI partners.

Equality, diversity and inclusion

Increasing the diversity of trial participants is fundamental to increase participation rates in trials. An ambition in testing recruitment and retention strategies is not only to increase the numbers of people taking part in trials, but also helping to recruit participants that better reflect those that might benefit from the trial results, or to avoid harm. Our SAP reflected this and for each host trial funded, we asked that wherever possible, demographic data on the age, gender and ethnicity of participants should be recorded. The importance of capturing these data can be illustrated in the getting it right: addressing shoulder pain (GRASP) SWAT, which tested personalised versus, non-personalised text messages. 77 In this SWAT, post hoc subgroup analysis showed that males under 65 years were the group most likely to return their initial questionnaire if they received a personalised text message. Capturing such data in all SWATs will also allow subsequent future patient-level meta-analysis to be undertaken, and differences in effect between different populations to be assessed. In time, there may also be a need for testing of population-specific SWATs if there is evidence of differential effects in distinct population groups.

Conclusions

PROMETHEUS originally aimed to fund 25 SWATs and supported 42 SWATs, generating the single biggest body of SWAT activity in the world. When SWAT funding was made available, we found that many teams embedded SWATs into their research. The mean cost of each SWAT was £3535. In addition to funding SWATs, PROMETHEUS successfully demonstrated the methodological feasibility of undertaking co-ordinated SWATs, a powerful new tool with the potential to rapidly accelerate the evidence base for recruiting and retaining trial participants. In addition to money, the co-ordination PROMETHEUS provided was crucial to increasing the recruitment and retention evidence base.

The PROMETHEUS SWATs reported to date found that there was no evidence of a significant difference in recruitment rates in any of the strategies tested. For the retention strategies, we found evidence of effectiveness for sending pre-notification cards, letters and e-mails to trial participants; that personalised text messages were more effective than non-personalised text messages, especially for men aged under 65 years; that sending a pen compared to no pen with postal questionnaires was more effective, although this result was not consistent as another SWAT found no difference. One SWAT comparing pink versus white paper for printing the primary outcome showed evidence of a decreased response in the pink paper group, and that it was also more burdensome to collect postal data in this group. We found no evidence of a significant difference in retention rates for any of the other strategies tested. More replications of the recruitment and retention strategies funded by the PROMETHEUS programme are required to provide definitive evidence for these strategies.

In the chapters to follow, we reflect on lessons learnt from undertaking the PROMETHEUS programme.

Chapter overview

PPI is important for developing and evaluating interventions aimed at recruiting and retaining patients in RCTs. In this section, we report the PPI in PROMETHEUS using the guidance for reporting involvement of patients and the public 2 (GRIPP2) Reporting Checklist109 and reflect on the lessons learnt for involving PPI partners in SWAT research.

Defining patient and public involvement in PROMETHEUS

The traditional aim of PPI is to involve end users in the research process. Given the methodological focus of PROMETHEUS, we defined ‘patients and the public’ broadly, to include those who would potentially be impacted by the recruitment and retention strategies, such as patients, potential patients, carers and members of the public.

We acknowledge the importance of stakeholders from the trial community who may be involved in undertaking SWATs, recruiting participants, or applying the evidence from SWATs, such as principal investigators, statisticians, trial managers and staff based within CTUs and other centres undertaking rigorous RCTs. However, we consider the contributions of these stakeholders as separate from PPI.

Aims

The aims of the PPI in PROMETHEUS were to:

1. Inform the overarching prioritisation of the recruitment and retention strategies that host trial teams were funded to undertake.

2. Inform how individual host trial teams could select, adopt and/or adapt specific recruitment or retention strategies for their specific patient population and recruitment/retention contexts.

Patient and public involvement: methods

Patient and public involvement: results

Patient and public involvement: discussion and conclusions

There was significant PPI in PROMETHEUS from its earliest stages to prioritisation of the strategies to be funded, to implementation of the funded SWATs. Involving patients and members of the public in methodological research can be challenging, as methodological research is often several steps removed from direct patient care. 115 However, we were able to successfully incorporate PPI in a way that members felt able to contribute to and found rewarding.

Patient and public involvement: reflections and critical perspective

Because of the methodological focus of PROMETHEUS, our PPI panel members were more ‘experienced’, and had had prior involvement in trials, and some also had prior involvement in methodological research. While this allowed our group to quickly understand and appreciate the aims, objectives and methods involved in PROMETHEUS, and to contribute in a way that was meaningful and rewarding, this also meant that we may have excluded more ‘naïve’ PPI members who could potentially have brought novel perspectives to the process. However, on the whole, we felt that having panel members who had prior experience was more likely to have benefitted rather than hindered our programme.

Lessons learnt for involving patient and public involvement partners in SWAT research

• Trial teams should include PPI when planning SWATs: In the same way that PPI is expected for the main trial, PPI should be an important consideration when planning the SWAT and should include those who would potentially be impacted by the recruitment or retention strategy being evaluated.

• Depending on the nature of the SWAT, PPI members can include potential or enrolled trial participants.

• Helping to identify novel or adapt existing strategies: PPI partners can help trial teams to identify novel or untested strategies to be evaluated. In PROMETHEUS, we asked PPI members to consider potential strategies that they would be interested in getting tested; a strategy suggested in response to this, which was to thank trial participants, led to the development of SWAT 119, which is testing the effectiveness of a thank-you card following each study visit. Where a strategy has been tested in a different population, format or context, PPI partners could be particularly valuable in assisting trial teams to adapt that strategy to be tested within their particular trial.

• Accessing PPI partners: For SWAT strategies targeting participants or potential participants, the first port of call should be to approach the PPI members that informed the host trial. For strategies targeting staff, such as a recruitment training course for site staff, the key activity would be to approach staff undertaking the recruitment at sites.

Chapter overview

In this chapter, we outline how SWAT strategies should be prioritised or selected, as well as key elements to consider when designing a SWAT.

Selecting which strategy to test in a SWAT: the importance of prioritisation

When a decision has been made to include a SWAT within a host trial, the next step is to determine the most suitable SWAT to undertake. To do this, the PROMETHEUS group found that there were several areas where research teams wanted support, information, and clarity:

• to have clear SWAT research priorities;

• identifying an intervention suited to a host trial population and trial type;

• communication as to when no further SWAT replications are necessary;

• transparent costs of SWATs;

• SWAT interventions for low budgets.

This would ensure efficient resource use and maximum research output.

The PROMETHEUS priorities were based on SWATs that could be undertaken with funding of £5000 or less. For further information on how the PROMETHEUS priorities were identified, please refer to Chapter 2.

While the PROMETHEUS programme identified a list of priorities for this work (see Chapter 2), there remains a need for continually updated research priorities to allow researchers to address the questions relevant at that time. Collaboration is essential when undertaking SWATs. The priority list needs to be developed and finalised in collaboration with working groups that are involved in SWAT and methodology work such as Trial Forge116 and those involved in the Cochrane reviews. 23,24 PROMETHEUS has already proactively collaborated with Trial Forge and the MRC NIHR Trial Methodology Research Partnership (TMRP) to begin this work. 117 Together with Trial Forge we have set up the Trial Forge SWAT Network118 and have utilised membership of the TMRP Recruitment and Retention working groups to promote priorities for SWAT work. In addition to a priority list, to ensure research output is maximised, establishing a real-time method to communicate SWAT priorities to researchers is a recommended future development. The Cochrane reviews on recruitment and retention strategies23,24 are updated every few years and so establishing a more rapid method would enable the research gaps to be identified and addressed. Additionally, we identified that there is a need to work with research teams to help establish SWAT priorities based on their experiences as to what is important with their research areas, settings, and populations. Currently, some of the research priorities that have been identified by both the PRioRITY34,110 and PROMETHEUS groups are not suitable for all trial teams and therefore teams have expressed a reluctance to embed them. One such example is the use of SMS for participant retention. Many teams already use SMS and so do not want to undertake a SWAT testing SMS versus no SMS, which the Cochrane review24 states currently requires further evaluations. This further highlights the importance of working with teams undertaking methodological research to identify SWATs that are acceptable, thus increasing the likelihood of SWATs being undertaken and the findings implemented. There is also a changing landscape that trialists need to be mindful of, for example, SWATs relating to postal follow-up of participants are becoming less relevant with moves towards electronic data collection.

To ensure that SWATs are undertaken in areas of need, it is essential that there is a priority list that demonstrates not only which SWATs are needed but in which trial populations. This will prevent unnecessary SWAT replications from being undertaken and ensure resources are directed to areas that are needed to increase the evidence base. For example, there is high-certainty evidence that pens are an effective strategy for older females as a retention strategy but only moderate-certainty evidence in a younger population and males. 118 Therefore, it is recommended that if additional replications are undertaken, they are done so within these populations.

There is also a need for creating more awareness around embedding a SWAT which ‘de-adopts’ a strategy that has already been planned and budgeted for within a host trial, but further evidence is needed for evidence of its effectiveness, for example, newsletters. These are often sent to sites and participants so these could be tested as part of a low-cost SWAT. Additionally, in-person site initiation visits could be compared to remote site initiation visits and if site initiation visits are ‘standard’ practice then such a SWAT would save money. 119

SWATs prioritisation, GRADE and pragmatism

When considering SWAT priorities there is a need for pragmatism; the question needs to be asked as to whether additional replications are actually necessary. It is not possible to complete a SWAT within every population, each replication requires resources, and these resources could be directed to an area where there is still large uncertainty around a strategy.

When assessing the SWAT evidence base GRADE is used to rate the quality of evidence and grade the strength of recommendations. 120 The GRADE domains for rating a judgement down are based on five different criteria: risk of bias, imprecision, inconsistency, indirectness and publication bias, and result in a classification of either very low, low, moderate or high. Currently, the PROMETHEUS team has collaborated with Trial Forge and assessed SWAT evidence to create Trial Forge evidence packs118 that break down recommendations by population according to the GRADE assessments (see the ‘support teams to undertake SWATs: Trial Forge Evidence packs’ section and Figure 2). There is, however, a trade-off between high certainty and directing resources to maximise output and so it is suggested that we may have to accept that we do not have ‘high certainty’ for all populations and a pragmatic consensus will need to be made as to whether further SWAT replications are recommended.

FIGURE 2.

Summary example in Trial Forge evidence packs, used with kind permission from Trial Forge. 118

Supporting teams to undertake SWATs: Trial Forge evidence packs and an information repository

Trial Forge107 have developed a number of Trial Forge evidence packs. 118 These packs provide trialists with information on how many SWAT replications have been undertaken and the evidence base to date, including a summary of GRADE. Figure 2 is an example summary found in a Trial Forge evidence pack. Additionally, they provide teams with practical implementation advice on how to undertake and embed the specific SWAT. These packs enable research teams to make an informed decision – feedback from researchers that have used them is that they are a useful additional resource to support teams.

To further aid teams in selecting a SWAT, a repository of information could be developed to share necessary information on a freely accessible public platform. This would include not just protocols which are currently found on the Northern Ireland SWAT store,32 but additional helpful documents to aid trial teams such as those detailed in Table 9. Currently, steps are being made to progress a repository in collaboration with the Trial Forge SWAT Network. 121 This development will likely take several months to ensure that it is detailed and piloted, and a way to operationalise it presently and in the future is identified. Both the PROMETHEUS and Trial Forge initiatives will lead this and report on the progress via their websites and social media channels.

Design choices: cluster randomised versus individual randomised

In planning a SWAT, an important consideration is whether to randomise at the individual-level (e.g. the participant or potential participant) or at the cluster level (e.g. recruiters or hospitals). If the recruitment or retention strategy is focusing on participants, potential participants, or their proxies, an individual-level randomised design may be the most efficient and appropriate randomisation method. However, for logistical reasons cluster randomisation, perhaps by study site, might be more feasible. On the other hand, if the recruitment or retention strategy affects individual participants or potential participants, but is mainly happening at a different level, such as whether training trial recruiters improve recruitment rates then a cluster design may be more appropriate (see Chapter 8 for case study). Cluster randomisation is often used to minimise ‘contamination’ between intervention and control participants. 119 For example, in a SWAT where the intervention is to train research nurses recruiting participants on how to improve trial recruitment, it would be difficult to undertake individual patient randomisation within hospital sites (i.e. intervention patients are recruited using the training method; control patients recruited using standard practice) as nurses who receive the recruitment training might use some of the training received to recruit control patients. Nurses within the same hospital site might also share the recruitment training information (advertently or inadvertently) with colleagues recruiting control patients, who may also change their usual recruitment practice, leading to contamination. This contamination might result in dilution bias and consequently lead to type II error (i.e. erroneously concluding there is no effect when the intervention is effective), which cluster randomisation would guard against.

The design of the host trial, as well as the recruitment methods, clinical context and patient population of the host trial, can influence the decision on whether to use individual or cluster randomisation. If the host trial is a cluster randomised trial, the SWAT could use the same clusters as the host trial (although the trial clusters would be randomised separately for the SWAT stratified by the host trial’s cluster status so that half the intervention and half the control clusters would receive the SWAT).

For many SWATs, the decision about whether to randomise at the individual or cluster level is also influenced by the practicalities and ease of implementing the SWAT, which often focuses on minimising the burden on the staff delivering the SWAT. For instance, a retention trial of different letters, or incentives, such as pens, may use a cluster design for practical reasons to avoid centres having to individually randomise follow-up letters to different conditions. The MRC-funded START programme, which tested 2 recruitment strategies across 12 host trials anticipated that some host trial teams would prefer to use cluster allocation of the START recruitment interventions, to ease the logistical burden on the trial, as well as to reduce the risk of possible contamination. 25 This is also the approach we took in PROMETHEUS, so where individual randomisation was likely to be a challenge or burdensome to those undertaking the SWAT, we suggested to applicants that they consider cluster randomisation. Table 10 outlines examples of individual and cluster randomised designs in SWATs.

For statistical considerations of SWATs, please see Chapter 9.

Funding for SWATs

There are limited funding streams available for methodological work both in the UK and internationally. PROMETHEUS, funded by the MRC UK and NIHR as part of the MRC-NIHR Methodology Research programme was able to offer teams funding, lack of which can be a limiting factor to researchers undertaking a SWAT. 36 In Chapter 3 we outline the average cost of a SWAT funded through PROMETHEUS. Prior to PROMETHEUS, the UK MRC funded the MRC START programme, which tested the feasibility of undertaking SWATs in a co-ordinated way. 25

Some funding agencies have engaged with the need for SWATs to improve trial conduct. These include the Irish HRB which has a programme of funding SWATs up to €25,000 and the UK’s NIHR HTA programme that offers UK research teams up to £10,000 to embed a SWAT4 and up to £30,000 to embed Studies Within A Project (SWAPs) across a number of their funding streams for multiple long-term conditions. 5 Our experience with PROMETHEUS suggests that this level of funding should be suitable for many different SWAT interventions to be embedded within host trials. However, within the PROMETHEUS programme lower cost and easier SWATs were deliberately chosen by the PROMETHEUS Trial Management Group (TMG) as priorities. Additionally, the NIHR through its Health and Social Care Delivery Research (HSDR) and Fellowship streams has also funded SWATs. 30,126

Lessons learnt for trial teams and methodologists undertaking Study Within A Trial research

• When setting SWAT research priorities, methodologists should provide as much information as possible so that teams are able to make informed decisions. They should consider developing strategies to aid teams to identify and select suitable SWAT interventions for their individual host trials. SWAT priorities need to be clearly communicated to trial teams and funders.

• We would recommend that the priorities are presented alongside estimated costs, resource use (including the time needed to undertake the SWAT) and, if possible, a protocol. This was a strategy that we undertook during the PROMETHEUS programme, and feedback received from trial teams was that this was helpful.

• The allocation units of the host trial that can be randomised for the SWAT consist not only trial participants but also sites, recruiting staff and time periods. The decision on the best allocation unit will be informed by a combination of what is most appropriate methodologically, and what is practical and acceptable to the host trial team, PPI members of the host trial team and staff undertaking recruitment activities.

Lessons learnt for funders

• Trial teams want to know that they were undertaking a SWAT that is necessary and relevant to increase the evidence base. This will ensure efficient resource use and maximum research output. More funding is needed to support the development of these types of resources.

• When applying for funding, trial teams need to indicate whether the question they are addressing is a priority SWAT question, as well as provide a clear rationale for selecting that particular question.

Chapter overview

In this chapter, we discuss our experience of obtaining governance approvals for SWATs, including approvals needed from the host trial Sponsor, trial oversight committees, REC and HRA approvals. We outline Key Learning for Sponsors, oversight committees, Trial Management and for journals and peer reviewers. Additionally, we consider the operational ethical issues in relation to embedding SWATS at scale, rather than an empirical analysis of the conceptual ethical issues.

Governance approvals

Studies Within A Trial are designed to be embedded within a host trial with no implication on its integrity, rationale or outcomes15 and, as a result, the obtaining of governance approvals should, in theory, be relatively simple. Despite this, many difficulties including those similar to obtaining approvals for large multi-centre studies continue to prevail. 127–129 These include, but are not limited to, the need for patient consent and/or information regarding the SWAT, study power and a misunderstanding or lack of understanding with regard to the methodology such as the need for replication. These difficulties pertain at every level of study approval from oversight committees and patient advisory groups, through to Sponsors and governance bodies.

Throughout the PROMETHEUS programme, a range of queries around the approvals process for SWATs were raised. This chapter outlines the queries noted at each level of approval, details some potential ongoing developments in this regard, and outlines suggestions for each group involved in the approvals process for a SWAT.

Conclusion

The success of obtaining approvals of SWATs undertaken within the PROMETHEUS programme has varied, largely depending on the knowledge and understanding of this methodology. In the main, difficulties have been relatively minor, centring around delays to obtaining approvals, and the associated hassle for busy trialists of responding to ethical queries. There have however been a couple of more substantial issues where sponsorship or ethical approval was not provided. In such instances, this stands to limit the development of the evidence base and so increase or sustain existing levels of research waste.

First and foremost, the key learning for all stakeholders involved in SWAT approvals is developing a better understanding and awareness of the importance of such methodological research, and the nuances of SWATs versus multicentre trials. The co-ordinated SWAT approach proposed by the HRA is a step in the right direction and should over time help to improve and simplify the undertaking of SWATs moving forward.

In addition, there are specific learning points for individual stakeholder groups:

Lessons learnt for Sponsors

• The changes proposed by the HRA to SWAT approvals should be communicated clearly and applied consistently when made publicly available.

Lessons learnt for oversight committees

• TMG members should encourage and support the undertaking of SWATs within their trials.

• TSCs should routinely review SWAT progress. Whether a DMC review is required is dependent on the host trial and SWAT.

Lessons learnt for trial teams and methodologists undertaking SWAT research

• The input of PPI members with regard to the need for SWAT information and/or consent should be discussed during review of intervention design and content.

• The HRA approach to SWAT approvals should be utilised wherever possible. Where SWATs are prospectively registered with the SWAT Repository, consideration should also be given to obtaining approvals to enable building of the evidence base simply and effectively.

Lessons learnt for journals and reviewers

• Reviewers should consider the need to query informed consent with authors on the basis that this may not be feasible or appropriate for a SWAT. Transparent reporting (see Chapter 10) should help to facilitate this.

Chapter overview

In this chapter, we describe our experience of implementing SWATs, and present case studies of different approaches to implementing SWATs using a case study of individual, standalone SWATs testing the effectiveness of pens for retention; and case studies of co-ordinated SWATs undertaken – one testing a training course for staff recruiting trial participants, and the other testing the effectiveness of sending Christmas cards to participants on retention rates. This chapter also highlights the pros and cons of standalone and co-ordinated SWATs.

Introduction

Due to the sample size of SWATs often being constrained to the size of the host trial, SWATs should be designed for replication and, ultimately, to be meta-analysed together with other similar SWATs of the same strategies. This provides the statistical power needed to identify small yet meaningful differences and ensure generalisability across different patient groups. However, when undertaking a meta-analysis, careful consideration needs to be given about when it is appropriate to combine data from SWATs in different patient populations, health conditions, and trial contexts. 15

Studies Within A Trial can be undertaken in one of two ways: individually and co-ordinated.

Individual Studies Within A Trial

Individual SWATs can be undertaken by individual host trial teams on an ad hoc basis, based on the interests and needs of the trial team. Individual SWATs independently test the same or similar interventions, within host trials at a time or opportunity that is convenient to the host trial. Multiple evaluations over an extended period may therefore be required in order to reach a definitive answer.

Individual SWATs can also be linked, such as in the MRC START programme, which developed an optimised PIL intervention and a multimedia information intervention; testing them across multiple host trials in SWATs over several years. 25,26 The TRECA study similarly aimed to embed the use of multimedia information resources into six host trials in the UK in a co-ordinated way. 30 In both MRC START and TRECA, each host trial had a linked but separate SWAT protocol, a separate ethical approval, and analysis, with the aim of undertaking a meta-analysis to determine the effectiveness. A hybrid approach to individual SWATs can be undertaken, with SWATs initially being conducted on an ad hoc basis, and later individual linked SWATs might be conducted to provide a definitive answer to the research question. The case study of individual SWATs of pens for retention in this chapter is an example of such a hybrid approach.

Case study of individual Study Within A Trial: pens for retention

Background

Postal questionnaires can be susceptible to low response rates. These are frequently used in trials to collect patient-reported outcomes, and so it is important to use effective strategies to maximise response rates.

The Cochrane review by Brueton et al. (2013)31 found that non-monetary incentives (e.g. pens, certificates or other similar tokens) with postal questionnaires had no clear effect in improving response rates or retention. The review however grouped all non-monetary incentives together and so individual strategies within this grouping may subsequently prove to be effective.

Including a pen with a questionnaire, as a non-monetary incentive, has been suggested as a low-cost intervention that may improve response rates. In the context of trial retention, firstly providing a pen with a questionnaire facilitates completion of this. Secondly, the inclusion of a pen acts as an acknowledgement of the participant’s involvement in the study, and as a reminder, which may make the recipient more likely to complete the accompanying questionnaire. Thirdly, the theoretical basis of reciprocation, results in participants feeling obliged to respond to this positive behaviour with the same in return. 134–137

Prior to the PROMETHEUS programme commencing, there was some evidence (two SWATs)138,139 to suggest that this may be an effective strategy in improving questionnaire response rates and reducing the number of reminders required. Given the existing evidence, and its match to PRioRiTY II110 priority six, this strategy was selected as a medium-priority question for the PROMETHEUS programme.

Aims and objectives

The individual SWATs were designed to evaluate the effects of providing a pen with a follow-up questionnaire on retention rates.

Results from the series of pen SWATs undertaken prior to or within the PROMETHEUS programme were then combined via meta-analysis by the Trial Forge Initiative in order to determine a definitive answer as to the effectiveness of pens for retention.

Methods

Individual SWAT methods

All pen retention SWATs undertaken in PROMETHEUS followed a core protocol. The only difference was that two SWATs68,92 included additional secondary outcomes and associated analyses.

Participants, interventions and outcomes

Table 12 summarises the participants, interventions and outcomes associated with the individual pen SWATs.

TABLE 12 - Study Within A Trial participants, interventions and outcomes

Participants	Any participant in the host trial who was due to be sent a follow-up questionnaire at the study’s primary outcome time point was included in the SWAT substudy. Participants who withdrew from host trial follow-up before their follow-up questionnaire was due or those who had received their follow-up questionnaire prior to the start of the pen substudy were excluded.
Intervention	A pen printed with either the trial or university logo.
Control	Standard practice for the host trial, i.e. no pen.
Outcomes (Primary)	The proportion of participants in each group who return the questionnaire.
Outcomes (Secondary)	Time to response: defined as the length of time taken to return the questionnaire Completeness of response: defined as the number of questions completed.
	Cunningham-Burley92 and James68 also assessed additional secondary outcomes Whether a reminder notice is required (defined as the number of participants requiring a reminder mailing divided by the number of participants who were sent a questionnaire). Cost effectiveness.

Randomisation and blinding

In each SWAT, participants were allocated using simple randomisation 1 : 1 to either intervention or control. A statistician who was independent of the questionnaire mailing activity generated the randomisation sequence.

Participants were not informed of and so could not provide informed consent for their participation in the SWATs. Given the nature of the intervention, it was however not possible to blind the participants to receipt, or not, of a pen with the questionnaire nor was it possible to blind research staff implementing the SWAT.

Sample size

Given their nature, the sample size for each SWAT was determined by the number of participants due to receive their postal questionnaire at the primary outcome time point within the host trial.

Analyses

For each SWAT, data were analysed using the version of Stata current at the time of analysis, on an intention-to-treat (ITT) basis, using two-sided tests at the 5% significance level.

Primary outcome

• The proportion of participants in each group who returned the questionnaire was compared using logistic regression.

• As the SWAT by James et al. 68 was factorial (social incentive cover letter, pen), the interaction term was also assessed in the logistic regression model.

Secondary outcomes

• The difference in time to response between the two groups was compared by a Cox proportional hazards model.

• Completeness of response between the two groups was compared using a linear regression model68,92 or logistic regression. 81

• Requirement for a reminder mailing was compared using logistic regression.

• The difference in costs between the intervention and control groups including direct and indirect costs.

Each trial adjusted the analysis models for main trial allocation. In addition, James adjusted for age and gender,68 and Mitchell for trial site. 81

Study Within A Trial meta-analysis methods

A meta-analysis of all relevant published SWATs was undertaken by the Trial Forge initiative in 2020 as part of their evidence pack series. 118

Results of PROMETHEUS SWATs testing the use of pens for retention

In total, 4550 participants were randomised into pen SWATs funded by the PROMETHEUS programme: 2276 to receive the intervention, and 2274 to receive the control. Fifty participants either withdrew or were deceased before the SWATs could be implemented leaving 4500 participants included in the analyses: 2249 intervention and 2251 control. The majority of participants were female (67.1%), and the average age was 64 years.

The results of the individual studies are reported separately. 68,81,92

Comparison with existing literature

Prior to the PROMETHEUS programme two trials, consisting of 8512 participants, reported within a 10-year period,138,139 suggested that when pooled, as shown in Figure 3, the inclusion of a pen with a follow-up questionnaire may be an effective strategy for improving response rates (pooled odds ratio 1.21; 95% CI 1.05 to 1.40; p = 0.01).

FIGURE 3.

Pooled odds ratio for effectiveness of pen with a follow-up questionnaire (pre PROMETHEUS). 138,139

When the results of Cunningham-Burley,92 James68 and Mitchell81 were added to the meta-analysis,136 this contributed a further 4500 participants (an increase of 52.8%). The pooled effect as shown in Figure 4 indicated that including a trial pen with a questionnaire probably increases retention and response rate (pooled OR 1.21; 95% CI 1.09 to 1.35). 118 The evidence when combined has moderate-certainty GRADE evidence overall, but high GRADE evidence for an older population.

FIGURE 4.

Pooled effect of including a pen with a follow-up questionnaire. 68,81,92,138,139

Cost effectiveness

The cost of retaining one additional participant due to inclusion of a pen with a questionnaire mailing is £25, which is cost-effective if based on the cost of enrolling an average participant in a NIHR-funded trial being £2200. 118

Discussion

Over 14 years, five individual SWATs have been undertaken sequentially to ascertain whether the inclusion of a pen with questionnaire mailings is an effective retention strategy. 68,81,92,138,139 Four of the five SWATs were conducted within a 4-year period (2016–20),68,81,92,138 and three of these were conducted within the PROMETHEUS programme. 68,81,92

The pooled estimate of effect for all five pen SWATs for retention showed an increase in retention of 1.9% (95% CI 0.0 to 3.7%). 140 Depending on population demographics, there is moderate- or high-certainty evidence in these estimates and so the recommendation is that trialists should include pens to increase retention rates in trials that use postal questionnaires. The bulk of replications undertaken were in female and older populations and so evidence for younger and male populations should be the focus of any further replications.

The costs associated with the inclusion of a pen for retention were estimated, using generic trials information, to be around £25 per additional participant retained. 140 Cunningham-Burley et al. identified a cost of £10.56 per additional participant retained,92 while others have found lower rates, for example a cost of $6.98 (approximately £5.60) for an additional participant to be retained. 140 While these costs are not inconsequential, the intervention is relatively low cost compared to other retention interventions, and in a large randomised controlled trial, these costs may well be possible to accommodate.

Co-ordinated Studies Within A Trial

Simultaneous SWATs are also a type of co-ordinated SWAT, but they are undertaken across multiple host trials at the same time. Simultaneous SWATs have one protocol for all the host trials, one ethical approval, one analysis and a write-up.

Case study – Bah humbug! Association between sending Christmas cards to trial participants and trial retention: randomised Study Within A Trial conducted simultaneously across eight host trials

Background

Poor participant retention can adversely impact study validity. 24,33,36 Christmas cards are often used to encourage participants to continue participation in a trial. A survey of UK-registered CTUs found that 40% had used Christmas cards as a retention intervention. 33 However, there is no evidence that providing Christmas cards improves retention. For those trials that use resources (both time and money) to send Christmas cards, there is a potential opportunity cost in that these resources might be better invested in other potentially more effective interventions. On the other hand, there could be a case for routine use of this incentive if there was evidence to support it.

One way to evaluate the effectiveness of Christmas cards on retention rates, and allow for rapid collection of evidence, would be to plan to do the same co-ordinated SWAT in several host trials at the same time.

Aims and objectives

To determine the effectiveness of sending Christmas cards to participants in randomised controlled trials to increase retention rate at follow-ups and to explore the feasibility of doing co-ordinated SWAT across multiple host trials simultaneously.

Methods

Design

Randomised co-ordinated SWAT conducted across eight host trials.

Setting

Eight randomised controlled trials researching various areas including surgery and smoking cessation.

Participants and sample size

Three thousand two hundred and twenty-three trial participants who were still due at least one follow-up from their host randomised controlled trial.

Intervention and comparator

Participants in the intervention group received a Christmas card sent by post. Participants in the control group did not receive a Christmas card.

Primary outcome

Proportion of participants completing their next follow-up (retention rate) within their host randomised controlled trial.

Randomisation and blinding

Participants were randomised in a 1 : 1 ratio (separately by each host trial) to either receive a Christmas card in mid-December 2019 or not receive a card.

Statistical analysis

Intention-to-treat analysis was undertaken, using a 5% significance level. Analysis models adjusted for the host trial allocation and the SWAT allocation and were run for each host trial separately. The results of the individual trials were combined using a random-effects meta-analysis. The primary analysis compared the retention rate between the two arms using a logistic regression model. The average cost per card sent was calculated using staff time for preparing the cards, and costs for printing and postage. Assumptions for CO₂ emissions calculation were made, based on previous research were applied which assumes the card weighs 10 g, is printed on recycled paper and posted and recycled in the UK. 144

Results

Host trials and participants

Table 15 provides a description of the host trials involved. One thousand four hundred and sixty-nine (1469) participants [age 16–94 years; 70% (n = 1033) female; 96% (813/847) white ethnicity] across the eight host randomised controlled trials were involved in the analysis (cut short owing to COVID-19).

TABLE 15 - Descriptions of the host trials involved in PROMETHEUS Christmas card SWAT

Acronym	Area	Interventions	Follow-up method	Target sample size	Primary outcome
C-Gall	Surgery – gallbladder	Laparoscopic cholecystectomy or conservative management.	Postal	430	Short Form-36 at 18 months.
CPIT-3	Smoking in pregnancy	Both groups receive smoking cessation service support and contingent shopping vouchers. Intervention group receives additional shopping vouchers (up to £400).	Telephone	940	Self-reported abstinence from smoking for 8 weeks.
DISC	Surgery/ Drug – hand	Injection of collagenase or surgery.	Clinic/postal	710	Patient evaluation measure at 1 year.
FUTURE	Female bladder weakness	Urodynamics plus comprehensive clinical assessment or comprehensive clinical assessment only.	Postal	1096	Patient Global Impression of Improvement at 15 months.
ProFHER-2	Surgery – shoulder	Reverse shoulder arthroplasty or hemi-arthroplasty or non-surgical.	Clinic/Postal	380	Oxford Shoulder Score at 2 years
PUrE	Surgery – kidney	Extracorporeal shockwave lithotripsy or percutaneous nephrolithotomy or flexible ureterorenoscopy with laser lithotripsy.	Postal	1044	EQ-5D-5L at 12 weeks
REFLECT	Dental	Prescription of 5000 ppm fluoride toothpaste or usual care.	Postal	1174	Proportion of participants receiving dental care due to caries at 36 months.
SWHSI-2	Wound healing	Negative pressure wound therapy or usual care (normal dressing).	Postal	696	Time (days) to wound healing.

Impact of sending Christmas cards on retention rates

No evidence was found of a difference in retention rate between the two arms for any of the host trials when analysed separately or when the results were combined [85.3% (639/749) for cards versus 85.4% (615/720) for no card; odds ratio 0.96 (95% CI 0.71 to 1.29; p = 0.77)]. A cumulative meta-analysis of the results can be seen in Figure 5.

FIGURE 5.

Christmas card SWAT cumulative meta-analysis. 49

Costs and carbon footprint

The cost of this intervention was £0.76 (€0.91; $1.02) per participant, and it will have a carbon footprint of approximately 140 g CO₂ equivalent per card. One benefit of this approach was the need to only submit one ethics application.

Discussion

This co-ordinated SWAT investigated the effectiveness of sending a Christmas card to trial participants on retention rates across eight host trials and showed that sending a Christmas card to trial participants does not increase the retention rate.

This study showed that a co-ordinated SWAT can be embedded successfully by multiple host trials simultaneously. Not only does this increase the speed at which evidence can be accumulated, it has the additional benefit of allowing only one ethics application to be submitted to cover all the work, which will decrease the burden on individual trial teams.

When this co-ordinated SWAT was initially planned, it was hoped to involve more than 10 trials, and over 10,000 participants to ensure that the question could be answered by this one evaluation. For various reasons, the evaluation was delayed, and some host trials could no longer embed the SWAT which reduced the sample size.

This SWAT was successfully implemented across eight host trials, from two CTUs in the UK, at the same time. This is one of the first instances where a SWAT evaluation has been undertaken in this way, and its success should influence other researchers to consider undertaking simultaneous SWATs in the future, to allow answers to the methodological questions that SWATs pose to be obtained more quickly.

Conclusions

In the PROMETHEUS programme, we have demonstrated that it is feasible to adopt different approaches to undertake SWATs. Host trial teams can undertake individual SWATs focusing on their patient populations and recruitment context to add to the wider evidence base, which in time can be meta-analysed with similar SWATs to provide definitive evidence on the effectiveness of a particular strategy. An alternative is the co-ordinated SWAT, which potentially offers a much more efficient methodology than individual SWATs, requiring just one regulatory approval for all the SWATs through one central co-ordination point, providing economies of scale to significantly speed up evidence generation.

Both individual and co-ordinated SWATs have their advantages and disadvantages, and depending on the situation, one approach may be more appropriate than the other. Indeed, either approach may be suitable in some instances, depending on the preferences of the research team and the resources available to them (such as resources to recruit host trials and co-ordinate the SWAT), and the methods and interventions being used by the potential host trials. Strategies that are specific to the methods or interventions being used in a host trial, or are highly adapted to the host trial, such as offering a free yoga class to participants to enhance retention,145 or a patient and family co-developed participant information,146 may be more suited to an individual SWAT approach. Strategies that are more widely used, such as sending Christmas cards, text message reminders and financial incentives all lend themselves well to being tested using a co-ordinated SWAT approach. A mixture of both approaches may be used to provide a definitive answer to a question. For instance, a simultaneous SWAT was successfully undertaken for the recruitment training course to assess feasibility; however, a conclusion on the effectiveness could not be reached. A future evaluation of this training could take a co-ordinated SWAT approach, or a trial team may wish to undertake an individual evaluation by randomising its recruiting sites to receive the training or not.

Lessons learnt for trial teams and methodologists undertaking SWAT research

• Embedding a co-ordinated SWAT in multiple host trials simultaneously, using the same SWAT protocol, has been shown to be a feasible, efficient way to scale up and speed up evidence generation using SWATs. This approach should be used more often to build an evidence base to support the selection of recruitment and retention strategies. This SWAT design could be utilised for other methodological aspects such as protocol/medication compliance strategies, and is suitable within most, if not all, research fields.

• Teams wishing to undertake a co-ordinated SWAT should plan this carefully. Significant co-ordination is required to develop a common SWAT protocol that adequately covers all potential host trials, recruit host trials, apply for overarching ethical approvals, prepare and deploy the SWAT intervention and collect outcome data.

• For most trial teams, individual SWATs remain the most convenient method for testing the effectiveness of strategies for their trial population or trial team; however, where resources allow, a co-ordinated SWAT approach should be considered as it has the potential to provide definitive answers more rapidly and are more efficient.

Lessons learnt for funding bodies

• Funding bodies should fund SWATs, particularly co-ordinated SWATs as they are potentially efficient and are likely to provide robust evidence at speed, which in turn can help to reduce research waste.

Chapter overview

Randomised SWATs are like any other trial and as such, the statistical considerations for SWATs are similar to those for a typical trial. Although constrained within the context of the host trial, a SWAT should still follow principles for rigorous analysis, to ensure that the findings are considered reliable. Within this chapter, sample size, analyses and costing considerations when undertaking a SWAT will be detailed, alongside examples of our learnings from the PROMETHEUS programme.

Sample size considerations

It is well known that the sample size for a SWAT is constrained by the host trial. That is, for a recruitment SWAT the maximum number of participants will be the number approached to participate in the host trial, and for a retention SWAT, it is the number who are randomised into the host trial. However, SWATs are not necessarily implemented from the start of a trial, they are often embedded mid-way through recruitment/retention, and as such their sample size is reduced.

Typically, in UK trials funded by NIHR, recruitment of eligible patients is found to be between 50% and 88%, with retention seen to be between 80% and 97%. 5 If a SWAT was powered in a similar way to typical trials, with 80% or 90% power (using a two-sided 5% significance level), the host trial would need to be between 1438 and 26,256 participants, depending on the initial rates seen to detect a difference between 2% and 4% (Table 16). With the average target sample size of 1122 for trials published in the HTA Journal,147 sufficiently powering these assessments using one trial alone would usually be infeasible. Furthermore, should a SWAT have been undertaken using cluster randomisation, the number of participants required would be larger due to the likely correlation of the participants within the clusters. For example, in MRC START, the minimum sample size for cluster SWATs was appropriately inflated to consider cluster allocations, assuming an intraclass correlation of 0.02 in line with estimates from community studies. 25,26 Thus, multiple replications of a single SWAT evaluation are typically needed, where the results from each are combined to produce a more reliable result.

Although it may seem inefficient to need to undertake multiple evaluations of the same strategy, this is actually a beneficial approach. Firstly, this allows for the generalisability of the results – Treweek et al. (2020)148 suggest that to conclude further SWAT evaluations are no longer a priority, it must have been tested in a wide range of contexts. If it were possible to conduct a large enough SWAT in one host trial for example, the findings would only be directly applicable to that specific trial population, so there is a need for replication in further host trials with a different patient population to ensure the findings are externally valid. This allows for the effectiveness of the strategy as a whole to be determined. As such the PROMETHEUS programme planned to fund several implementations of the same SWAT – for instance, the Christmas Card SWAT – to allow for the results to be combined within a meta-analysis. 47 This allowed us to conclude that Christmas cards are inefficient at increasing retention in an adult population, as it has been evaluated within a range of different host trials, over different research areas and a range of patient populations, all at once. 49

Analysis considerations

The approach for the analysis should use ‘intention to treat’. Indeed, it is more usual to use ‘pure intention treat analysis’ in a SWAT than is often the case with the host trials. This is because the primary outcome in a SWAT is recruitment or retention, so there is usually no missing data problem that afflicts most other forms of trials.

As with any analysis undertaken within trials, the analysis of a SWAT should be pre-planned, and detailed with a SAP. This could be within the main SAP for the trial, or as a separate stand-alone document – the choice may depend upon at what point of the trial the SWAT is being undertaken. This SAP should detail the data that are to be collected (i.e. the source of the data), the planned outcomes and the statistical methods to analyse each of the outcomes – similar to any SAP. Within the PROMETHEUS programme we created a template SAP for the host trial teams to adapt for their SWATs.

Typically, for a recruitment SWAT, the primary outcome will be the proportion of eligible participants consenting to be randomised. Although this is not necessarily always the case as was demonstrated in the ‘training’ SWAT which was a feasibility SWAT (see Chapter 8, case studies of simultaneous SWATs), and the primary outcome was measures of recruitment confidence, while actual recruitment and randomisation rates were secondary measures. But it is always important to report actual recruitment rates to enable all SWATs to be included in a future meta-analysis.

Similarly, for a retention SWAT, the primary outcome will most likely be the proportion of participants who are retained. The definition of a retained participant may vary; it could be those who have completed a follow-up, attended a clinical visit, or another way, but this should be explicitly stated. Ideally, however, one measure should include the number of participants that provided primary outcome data to, again, allow a meta-analysis. Equally, whether all participants, or only those participants that were due to be followed-up (i.e. excluding those who withdrew prior to the follow-up), are to be included should be detailed.

For both types of SWATs, a logistic regression model is an appropriate analysis model. For retention SWATs this model should be adjusted for the host trial allocation and any stratification variables that were included within the SWAT randomisation. The SWAT design should be taken into account when analysing; for instance, if it was a cluster-randomised SWAT the clustering should be accounted for in the analysis.

Thought should be given as to whether it is appropriate to undertake a sensitivity analysis, where those participants who did not receive the SWAT allocation are excluded (for instance, if there is a text message sending failure) – though in most instances this is not required.

Secondary outcomes may include time to events, which would be most appropriately analysed with a Cox proportional hazard model, or count data, such as the number of reminders required, which may require a Poisson or negative binomial regression model, although for each analysis thought should be given to whether the relevant assumptions are met, and planned analysis adjusted to be suitable (i.e. using a zero-inflated model).

Once several evaluations of a SWAT have been undertaken, a meta-analysis should be performed to combine the results. Typically, it would be expected that each SWAT would be analysed and reported within its own right, thus a two-stage meta-analysis approach, in which the results from each SWAT are combined, would be used. Should it be possible to obtain the individual data from each trial, a one-stage approach could be considered, though there are typically no additional benefits to this approach apart from considering interactions with pre-specified covariates (e.g. age, gender) and ensuring consistency in model adjustments. A cumulative meta-analysis can be useful in determining if the results appear to have converged, which can be used to help judge if another evaluation is required. 145 When planning a replication of a SWAT, thought should be given to the design and approach taken to ensure that the replications are similar. For instance, using similar wording if evaluating a message-based intervention, or consistent timing, so that the results of the meta-analysis are reliable. The same can be said about the definition of outcomes, is a questionnaire considered complete if it has been returned, if all the questions were answered, or if the primary outcome was completed? These differences can impact the results and should be consistent across SWAT replications wherever possible.

Meta-analysis

A key element of accumulating any evidence for trials is to combine similar trials in a meta-analysis. For SWATs, this is particularly important as, unlike a host trial, they are rarely large enough on their own to demonstrate any difference as being statistically significant. Consequently, for individual SWATs we have a high risk of a type II error, erroneously concluding there is no important difference when there is. To deal with this important issue we have to rely on combining our SWATs. Typically, this is done using a random-effects meta-analysis, as we cannot assume that the effect size will be the same across all SWATs – especially when undertaken in various contexts. In this programme of work, we planned to undertake a one-stage meta-analysis, as the individual patient-level data were to be provided to the team, as part of the funding agreement. However, typically it would not be possible to access this level of data, and thus a two-stage approach, where the estimates of effect from each SWAT are combined is most appropriate. We, therefore, would encourage researchers to undertake a cumulative meta-analysis if there are similar published SWATs available – this recommendation is in line with that provided previously. 145

Economics and costings

Economic evaluations alongside SWATs aim to assess the costs of the strategies that were being evaluated. Despite the very high costs associated with poor recruitment and retention in trials, there has been very little attention paid to the economic aspects of SWATs. It is not enough to simply identify effective recruitment or retention strategies; we need to identify which strategies are also cost-effective.

Within the PROMETHEUS programme, we encouraged host trials to undertake a cost analysis. To do this, we suggested they collect costs both in terms of resources required (for instance, the cost of printing cards), and associated staff time (such as preparing the cards for postage). The importance here is to highlight the additional costs that are being acquired by implementing the strategy, as opposed to the standard treatment. This allows a trial team to determine if, for their trial, the cost would be feasible. For instance, within the Christmas card SWAT we determined that the cost per card sent was on average £0.76 – so if you were to use this strategy in a trial of 1000 participants, it would be a direct cost of £760. However, this analysis can then be taken one step further to allow for an assessment of cost per additional participant recruited or retained – where the total associated cost is divided by the additional participants recruited/retained. However, in the Christmas card example, it was not found to be an effective strategy, and as such no additional participants were retained, so even a relatively small finance cost (such as £760), would be considered wasted resources when assessing retention. In some instances, the information from these evaluations from the PROMETHEUS-funded SWATs have helped inform Trial Forge Evidence packs,136 to allow trial teams to easily determine if the associated costs are worth the potential benefits.

There is additional work that can be done in this area. For instance, for a retention SWAT there may be additional cost benefits if the number of reminders sent also is decreased by the strategy, as this may lead to less paper and postage costs. However, to date, this level of in-depth economic evaluation has not been undertaken within SWAT research, and the area is of growing interest.

Conclusions

SWATs are generally straightforward to analyse. We should use an ITT approach with a pre-specified SAP. Power calculations are generally not required as the sample sizes are constrained by the size of the host trial. The prospect of each individual SWAT being included in a future meta-analysis should always be at the forefront of the researcher’s mind when the analysis is undertaken. Consequently, all the numbers of patients randomised should be reported, and the numbers who respond (i.e. for retention SWAT the number who provided the primary outcome) and the number who did not, and for recruitment, the numbers who were randomised into the host trial.

Lessons learnt for trial teams and methodologists undertaking Study Within A Trial research

• Generally, the analyses are relatively straightforward and are much less complex than the standard analysis for the host trial.

• A SWAT SAP should be produced before the analysis as per the standard practice in trial analysis.

• It is not necessary, generally, to undertake a power calculation because each SWAT is constrained by the size of the existing host trial. The lack of statistical power in most SWATs to show the small, but important, differences in retention and recruitment should be addressed by combining SWATs in a meta-analysis. Consequently, researchers are encouraged when writing up their SWAT to include a cumulative meta-analysis of similar SWATs.

Lessons learnt for funders

• The analysis of SWATs is generally a much simpler procedure compared with the usual clinical trial analysis. Consequently, the impact on funding for statistical input is relatively light.

Chapter overview

Research teams participating in the PROMETHEUS programme have noted challenges in reporting SWATs. These include accessing funding, identifying a journal that will publish a SWAT and having resources and dedicated time available to analyse and write up a SWAT publication. This chapter outlines these barriers and identifies progress made by the PROMETHEUS programme in mitigating these.

The need for transparent reporting of Studies Within A Trial

The need for transparent reporting of clinical trials is well known, and the same applies to SWATs. Given that SWATs are designed to be meta-analysed together with similar evaluations of the same strategies, the need for transparent and timely reporting is arguably more crucial, particularly given the limited evidence base around effective recruitment and retention strategies. Without this, the evidence base will not be increased rapidly, and accurate SWAT priorities cannot be recommended. The findings of a SWAT may be detailed within the host trial publication or in a standalone dedicated publication.

Reporting Studies Within A Trial

Concise, clear and structured reporting of SWATs in a timely manner is crucial to building the evidence base. Given these methodological studies utilise a randomised controlled trial approach, many trialists may use and adapt the CONSORT guidance when reporting SWATs. However, there are also guidelines, developed by Madurasinghe et al. (2016),27 primarily for the reporting of embedded recruitment trials, which offer a useful starting point for the structuring of both recruitment and retention SWAT-related publications. Noting the use of both guidelines in the reporting of SWATs, the PROMETHEUS team, in conjunction with Trial Forge, have developed concise SWAT reporting guidelines (see Appendix 4). The guidelines developed by the PROMETHEUS team are designed to be methodologically robust and encourage GRADE adherence, which is crucial to aiding evidence synthesis and so deriving definitive evidence. These have recently been piloted by those writing SWAT publications,80,81,92,149,150 and once feedback has been received the final iteration of the guidance will be submitted for publication.

Studies Within A Trial may be published within the host trial publication, which offers benefits in obtaining publication in a potentially high-impact journal. There are, however, disadvantages to this approach, particularly where the SWAT is recruitment focused and the host trial has a long follow-up period, thus resulting in delays to making the SWAT results publicly available. This, therefore, limits the prompt inclusion of SWAT findings in evidence synthesis, and this may be further exacerbated by the fact that the host trial publication may not note the SWAT in relevant parameters to allow it to be identified in a search strategy focusing specifically on SWAT methodology. Indeed, publishing the SWAT separately means that ‘SWAT’ can be included either in the title of the paper or the list of keywords to facilitate identification for future systematic reviews and meta-analyses. Furthermore, where SWATs are submitted as standalone publications, these offer an opportunity for early career researchers to obtain first-author publications. It also allows members of the host trial team to obtain peer-reviewed publications before the host trial is completed. For instance, the SCOOP trial of screening for osteoporosis published three SWATs of retention interventions,134,151,152 which were completed and published before the main SCOOP trial results publication. There is, however, a dearth of methodological journals, which would be an ideal resource for the publication of SWATs, and so population or condition specific journals may often be approached. From the experience of PROMETHEUS, these journals often have limited interest and may reduce the subsequent acceptance of SWAT publications.

Reporting platforms such as F1000Research153 can therefore be a simpler, and quicker, alternative to the traditional journal. For example, this platform undertakes pre-publication checks on submission, and once approved the publication is made available on the platform with open peer review and revision following accordingly, which means the publication is available promptly. The costs of platform publication are often calculated on the basis of word count and so may also offer an affordable alternative for open-access publication, although it should be noted that since the PROMETHEUS programme commenced costs for platform publications have increased. Alternatively, a SWAT might be reported in a preprint server while under review by a conventional journal.

Identifying peer reviewers

Once a journal has accepted the SWAT for review, an additional barrier is the difficulty in finding suitable peer reviewers to review SWAT publications. This has resulted in substantial delays in the submission of SWATs until the journal finds a peer reviewer. This not only increases the length of publication time, but it also slows down advances in the evidence base and could result in teams continuing to undertake strategies which have no effect or a negative effect on their trials.

The small pool of reviewers is due to several reasons. Firstly, some journals have strict criteria for those who are eligible for being a reviewer (e.g. the need to hold a PhD). Secondly, SWATs are a niche area and so many SWAT authors have collaborated that conflict of interests are commonplace, further reducing the pool.

Due to the difficulties observed in PROMETHEUS in identifying potential reviewers, a database of potential peer reviewers has been developed. Currently, this is not publicly available due to data protection and GDPR regulations,154 although it is hoped that this may be available in the future once permission has been sought from individuals.

Many journals require reviewers to list reviewing preferences or classifications which could be beneficial to journal editors in identifying and allocating publications for review. It is therefore useful for journals to include ‘SWATs’, ‘Trials Methodology’, ‘Embedded Trials’, or other similar fields to reviewer classification fields, and for those with an interest in this methodology to ensure that they have included this in their preferences.

Peer review feedback

As noted previously, SWATs are often constrained to the size of the host trial and so formal sample size calculations are often not completed. Often this leads to peer-review feedback, specifically noting the need for a sample size calculation and sample size justifications based on power criteria.

This may be as a result of the difficulties in obtaining appropriate peer review (see above) and so the additional use of methodological classifications may ensure that future reviewers are SWAT experienced rather than perhaps just trials experienced. Where reviewers have limited SWAT experience, it may be beneficial to provide additional guidance on the characteristics of a SWAT and what is expected to be reported methodologically, to assist with mitigating unnecessary queries.

Conclusions

As SWATs are designed to be meta-analysed, the need for transparent and timely reporting is necessary to develop the evidence base, find definitive conclusions on effective (or ineffective) strategies, and identify further SWAT priorities.

During the PROMETHEUS programme progress has been made with improving the reporting of SWATs, including guideline development, identification of publication routes and improving peer review. In addition, the publication platform F1000 has set up a SWAT Collection,153 and Research Methods in Medicine and Health Sciences155 is currently developing a SWAT special edition. From these developments, the following key learning points are suggested for relevant stakeholders:

Lessons learnt for trial teams and methodologists undertaking Study Within A Trial research

• SWAT-specific reporting guidelines should be used to ensure that sufficient detail is provided to support GRADE evaluation of SWAT interventions.

• Trialists should be encouraged to include SWAT or methodology classifications in their journal reviewer profiles if they have an interest in this methodology and are willing to provide reviews.

Lessons learnt for journals and reviewers

• When selecting reviewers, SWAT and methodology classifications should be used where possible.

• Journals should consider potential flexibility to reviewer credentials for niche areas such as SWATs; for example, a PhD can be substituted with relevant experience in the area.

Chapter overview

Below we outline a range of recommendations for stakeholders such as funders, trial teams who are planning and undertaking recruitment and retention activities and for trial methodologists interested in SWATs.

Recommendations

Contributions of authors

Adwoa Parker (https://orcid.org/0000-0002-2880-3935) (Assistant Professor) contributed to study conceptualisation, funding acquisition, study methodology and investigation, project administration, drafted the initial version of the manuscript, and read and approved the final version.

Catherine Arundel (https://orcid.org/0000-0003-0512-4339) (Research Fellow) contributed to study conceptualisation, study methodology and investigation, project administration, drafted the initial version of the manuscript, and read and approved the final version.

Laura Clark (https://orcid.org/0000-0001-9227-5447) (Research Fellow) contributed to study investigation, project administration, drafted the initial version of the manuscript, and read and approved the final version.

Elizabeth Coleman (https://orcid.org/0000-0003-4210-1865) (Statistician) contributed to study methodology and investigation, curated the data, carried out analysis and prepared the results for publication, drafted the initial version of the manuscript, and read and approved the final version.

Laura Doherty (https://orcid.org/0000-0001-5310-5808) (Trial Support Officer) contributed to study investigation, project administration, drafted the initial version of the manuscript, and read and approved the final version.

Catherine Elizabeth Hewitt (https://orcid.org/0000-0002-0415-3536) (Professor) contributed to study conceptualisation, funding acquisition, study methodology and investigation, analysis, and preparation of the results, drafting of the manuscript, and read and approved the final version of the manuscript.

David Beard (https://orcid.org/0000-0001-7884-6389) (Professor) contributed to study conceptualisation, funding acquisition, study methodology and investigation, and read and approved the final version of the manuscript.

Peter Bower (https://orcid.org/0000-0001-9558-3349) (Professor) contributed to study conceptualisation, funding acquisition, study methodology and investigation, and read and approved the final version of the manuscript.

Cindy Cooper (https://orcid.org/0000-0002-2995-5447) (Professor) contributed to study conceptualisation, funding acquisition, study methodology and investigation, and read and approved the final version of the manuscript.

Lucy Culliford (https://orcid.org/0000-0002-9255-6617) (Senior Research Fellow) contributed to study conceptualisation, funding acquisition, study methodology and investigation, and read and approved the final version of the manuscript.

Declan Devane (https://orcid.org/0000-0002-9393-7075) (Professor) contributed to study conceptualisation, funding acquisition, study methodology and investigation, and read and approved the final version of the manuscript.

Richard Emsley (https://orcid.org/0000-0002-1218-675X) (Professor) contributed to study conceptualisation, funding acquisition, study methodology and investigation, and read and approved the final version of the manuscript.

Sandra Eldridge (https://orcid.org/0000-0001-5638-2317) (Professor) contributed to study conceptualisation, funding acquisition, study methodology and investigation, and read and approved the final version of the manuscript.

Sandra Galvin (https://orcid.org/0000-0003-3923-2506) (Programme Manager) contributed to study conceptualisation, funding acquisition, study methodology and investigation, and read and approved the final version of the manuscript.

Katie Gillies (https://orcid.org/0000-0001-7890-2854) (Reader) contributed to study methodology and investigation, and read and approved the final version of the manuscript.

Alan Montgomery (https://orcid.org/0000-0003-0450-1606) (Professor) contributed to study conceptualisation, funding acquisition, study methodology and investigation, and read and approved the final version of the manuscript.

Christopher J Sutton (https://orcid.org/0000-0002-6406-1318) (Senior Lecturer in Clinical Trial Statistics) contributed to study methodology and investigation, and read and approved the final version of the manuscript.

Shaun Treweek (https://orcid.org/0000-0002-7239-7241) (Professor) contributed to study conceptualisation, funding acquisition, study methodology and investigation, and read and approved the final version of the manuscript.

David J Torgerson (https://orcid.org/0000-0002-1667-4275) (Professor) conceptualised the study, led on funding acquisition, contributed to study methodology, investigation, drafting the manuscript, provided overall supervision of the programme, and read and approved the final version.

Study registrations

All SWATs in the PROMETHEUS programme had to be registered with the Northern Ireland Network for Trials Methodology Research SWAT Repository.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: 13/55/80) and is published in full in Health Technology Assessment; Vol. 28, No. 2. See the NIHR Funding and Awards website for further award information.

Ethics statement

Required ethical approvals for all of the PROMETHEUS host trials were obtained separately by the host trial teams, and the SWATs undertaken were approved as part of the initial ethical application for the host trial, or through the submission of an amendment to the Research Ethics Committee. The following approvals for the SWATs were obtained:

• ACTIVE: NRES Committee Yorkshire and The Humber – Bradford Leeds (ref.: 18/YH/0014).

• ActWELL: East of Scotland Research Ethics Service REC 1 (ref.: 17/ES/0073).

• CHAMP-1: South West – Frenchay Research Ethics Committee (ref.: 19/SW/0082).

• CLEAR: North East – Tyne and Wear South Research Ethics Committee (ref.: 17/NE/0339).

• GRASP: Berkshire B Research Ethics Committee, substantive amendment to the host trial (ref.: 16/SC/0508).

• IBD-BOOST: London – Surrey Research Ethics Committee (ref.: 19/LO/0750).

• KReBs (pen SWAT): North East – Newcastle and North Tyneside on (ref.: 16/NE/0400; amendment 2).

• KReBs (text message SWAT): North East – Newcastle and North Tyneside on (ref.: 16/NE/0400; amendment number (16/NE/0400/AM14).

• L1FE: London – Harrow Research Ethics Committee (ref.: 19/LO/0555).

• MiQuit: East Midlands–Nottingham 1 Research Ethics Committee (refs: 13/EM/0427 and 17/EM/0327).

• MSS3: North West – Greater Manchester Central Research Ethics Committee (ref.: 18/NW/0422).

• OTIS: NHS West of Scotland Research Ethics Committee 3 (ref.: 16/WS/0154).

• PEP-TALK: South Central Oxford B (ref.: 18/SC/0423).

• PONOC: National Research Ethics Service Committee East Midlands – Nottingham 2 (ref.: 13/EM/0459).

• SARC: West of Scotland REC 1 (ref.: 19/WS/0087).

• SSHeW: Department of Health Sciences Research Ethics Committee at the University of York and the Health Research Authority (ref.: HSRGC/2016/187/A).

• SWHSI-2: Leeds East Research Ethics Committee (ref.: 19-YH-0054).

• TOPaZ: East of Scotland Research Ethics Service (EoSRES) (ref.: 16/ES/0110).

• UK-FROST: North East – Newcastle and North Tyneside 2 Ethics Committee, (ref.: 14/NE/1176). Substantial Amendment 1 – REC Favourable Opinion 2 February 2015.

• VITA: London – Harrow Research Ethics Committee on 9 September 2017 (ref.: 17/LO/1245).

• WORKWELL: West Midlands – Solihull Research Ethics Committee (ref.: 18/WM/0327).

The ethical approval for the co-ordinated recruitment training SWAT across all the host trials was obtained from the University of York’s Department of Health Sciences Research Governance Committee. No further approvals were needed as this was classed as an evaluation of a teaching workshop (ref.: 76/76).

The ethical approval for the co-ordinated Christmas card SWAT across all the host trials was approved by the Yorkshire and the Humber – Leeds West Research Ethics Committee (ref.: 19/YH/0349). A single application was submitted to allow the SWAT to be embedded within all the host trials.

Data-sharing statement

The data sets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Additional funding received

YTU receives core funding from the NIHR CTU Support Funding. Following the initial success of PROMETHEUS, YTU received additional funding to continue some of the work of PROMETHEUS, through an institutional award from the NIHR CTU Infrastructure Support (NIHR132547). The Health Services Research Unit (HSRU) is core funded by the Chief Scientist Office of the Scottish Government Health and Social Care Directorate.

Each host trial received its own funding, as follows:

• Cancer Research UK, the Chief Scientist Office, Scottish Government, Health and Social Care Public Health Agency Northern Ireland, Chest Heart and Stroke Northern Ireland, The Lullaby Trust, Public Health Agency Northern Ireland, and the Scottish Cot Death Trust: CPIT-III.

• Medical Research Council and National Institute for Health and Care Research – Efficacy and Mechanism Evaluation: STARTS:REACTS, TOPaZ.

• Cancer Research UK and National Institute for Health Research: MiQuit.

• National Institute for Health and care Research – HTA programme: ARTISAN, C-Gall, CLEAR, DISC, FUTURE, GRASP, L1FE, MAGIC, ProFHER-2, PUrE, REFLECT and SWHSI-2, UK FROST, VITA.

• National Institute for Health and care Research – Programme Grants for Applied Research: IBD-BOOST.

• National Institute for Health Research – Health and Social Care Delivery Research: MSS3.

• National Institute for Health Research – Public Health Research Programme: SSHeW.

• The Scottish Government: ActWELL.

• Versus Arthritis: WORKWELL.

Author KG’s time was funded by the Chief Scientist Office of the Scottish Government’s Health and Social Care Directorate (CZU/3/3). Author RE’s time was funded by NIHR Research Professorship (NIHR300051) and the NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London. Author APs time was part-funded by the Wellcome Trust (ref: 204829) through the Centre for Future Health (CFH) at the University of York, which partially supported the delivery of the recruitment training SWAT.

Publications

1. Coleman E, Arundel C, Clark L, Doherty L, Gillies K, Hewitt C, et al. Bah humbug! Association between sending Christmas cards to trial participants and trial retention: randomised study within a trial conducted simultaneously across eight host trials. BMJ (Clin Res Ed) 2021;375:e067742. https://doi.org/10.1136/bmj-2021-067742

2. Parker A, Mills N, Rooshenas L, Jepson M, Donovan JL, Arundel C, et al. Staff training to improve participant recruitment into surgical randomised controlled trials: a feasibility study embedded within four randomised controlled trials. In International Clinical Trials Methodology Conference. Brighton, 2019, p. PS8A. BMC.

3. Arundel CE, Clark L, Coleman E, Doherty L, Parker A, Torgerson DJ, et al. Challenges and solutions to the implementation of Studies Within A Trial (SWATs): the experiences of the PROMETHEUS programme. Res Health Sci Med 2022;4(0):16–23. https://doi.org/10.1177/26320843221106949

4. Parker A, Arundel C, Mills N, Rooshenas L, Jepson M, Donovan JL, et al. Staff training to improve participant recruitment into surgical randomised controlled trials: a feasibility study embedded within four randomised controlled trials. Res Meth Med Health Sci 2022;4(0):2–15. https://doi.org/10.1177/26320843221106950

5. Clark L, Arundel C, Coleman E, Doherty L, Parker A, Hewitt C, et al. The PROMoting THE USE of SWATs (PROMETHEUS) Programme: lessons learnt and future developments for SWATs. Res Meth Med Health Sci 2022;3(4):100–6. https://doi.org/10.1177/26320843221089632

6. Treweek S, Gallant S, Anderson AS. SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance. F1000Research 2021;10:84. https://doi.org/10.12688/f1000research.50890.2

7. Cureton L, Marian IR, Barber VS, Parker A, Torgerson DJ, Hopewell S. Randomised Study Within A Trial (SWAT) to evaluate personalised versus standard text message prompts for increasing trial participant response to postal questionnaires (PROMPTS). Trials 2021;22(1):1–10. https://doi.org/10.21203/rs.3.rs-262194/v1

8. Mitchell AS, Cook L, Dean A, Fairhurst C, Northgraves M, Torgerson DJ, Reed M. Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial. F1000Research 2021;9:321. https://doi.org/10.12688/f1000research.23018.2

9. Mitchell AS, Cook L, Dean A, Fairhurst C, Northgraves M, Torgerson DJ, Reed M. An embedded randomised controlled retention trial of personalised text messages compared to non-personalised text messages in an orthopaedic setting. F1000Research 2020;9:591. https://doi.org/10.12688/f1000research.24244.1

10. Coleman E, Whitmore R, Clark LK, Daykin K, Clark M. Pre-notification and personalisation of text-messages to retain participants in a smoking cessation pregnancy RCT: an embedded randomised factorial trial. F1000Research 2021;10:637. https://doi.org/10.12688/f1000research.51964.1

11. James S, Parker A, Cockayne S, Rodgers S, Fairhurst C, Torgerson DJ, et al. Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study Within A Trial. F1000Research 2020;9:623. https://doi.org/10.12688/f1000research.23767.2

12. Cunningham-Burley R, Roche J, Fairhurst C, Cockayne S, Hewitt C, Iles-Smith H, Torgerson DJ; SSHeW Study Team. Enclosing a pen to improve response rate to postal questionnaire: an embedded randomised controlled trial. F1000Research 2020;9:577. https://doi.org/10.12688/f1000research.23651.1

13. Sarathy PP, Kottam L, Parker A, Brealey S, Coleman E, Keding A, et al. Timing of electronic reminders did not improve trial participant questionnaire response: a randomized trial and meta-analyses. J Clin Epidemiol 2020;122:70–7. https://doi.org/10.1016/j.jclinepi.2020.03.001

Disclaimers

This report presents independent research funded under a MRC–NIHR partnership. The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, the MRC, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, the MRC, NETSCC, the HTA programme or the Department of Health and Social Care.

Study Within A Trial

Key principles

SWATs are run as discreet studies but with the expectation that they will be replicated (in order to boost statistical power and assess the generalisability of the findings). Therefore, it is important to make a distinction between:

1. The overall, overarching SWAT evaluation (see Glossary) which will evaluate data from multiple SWATs which are run in multiple trials, and

2. The individual evaluations of a SWAT intervention within individual host trials, each of which will contribute data to the overarching evaluation.

The expectation is that approval will be obtained for the overarching SWAT evaluation (1) and then each individual SWAT (2) will come under the approvals obtained for the host trial; this may be from the start of the trial or as a substantial amendment after the trial has commenced. The SWAT evaluation lead and host trial team roles are shown in Figure 7.

FIGURE 7.

Study Within A Trial evaluation lead and host trial team roles.

Study Within A Trial groups

Table 18 sets out which approvals should be obtained for overarching SWATs and for individual SWATs to be included with a host trial.

TABLE 18 - Guide to which approvals for should be obtained for individual and overarching SWATs

Group	Description
1	The activities targeted by the SWAT are not of a type that needs to be described in detail in a host trial protocol (and so have not required previous approval) but are operational or management activities generally done by the central trial team.
	Examples of this SWAT group include: Randomising the host trial Chief Investigator to visit some recruiting sites and not others to see whether this improves recruitment. Randomising monitoring visits to sites being undertaken by either Skype or in person.
2	The SWAT interventions affect participants but affect a process that does not need to be described in detail in a host trial protocol (and so have not required previous approval). These interventions are provided to the research team via operational or instruction documents, rather than a protocol, which are not reviewed by the REC. However, in some cases these interventions are also described in the protocol. Where this is the case, the intervention remains a Group 2 SWAT due to the low risk and unobtrusive nature of the intervention.
	Examples of this SWAT group include: Variation in the gender of the person who signs a trial invitation letter to assess whether this impacts recruitment. Randomising whether trial participants are sent a Christmas card or not to assess whether this impacts on trial retention.
3	These SWAT interventions change the design of the host trial, change procedures undertaken by participants, or the information received by participants.
	Examples of this SWAT group include: Randomisation of whether and when a gift card is sent to participants to assess whether this impacts on trial retention. Randomisation of whether potential participants receive an information sheet developed by the trial team and an information sheet developed using a PPI group.

Glossary