The Men's Safer Sex project: intervention development and feasibility randomised controlled trial of an interactive digital intervention to increase condom use in men

The Behaviour Change Wheel

The Behaviour Change Wheel (BCW) (Figure 1) is one method that can be used to guide this process. 23 It provides a framework for the process of developing complex interventions and it represents a synthesis of 19 theoretical frameworks of behaviour change pathways. It has the advantage of being comprehensive (i.e. it covers all the main intervention functions and policy categories), coherent (i.e. categories within it are consistent and it provides systematic steps for intervention design) and linked to an overarching model of behaviour. The BCW has been used to develop interventions that were both acceptable to users and effective in achieving their aims. 24–26

FIGURE 1.

The BCW. 22,23 © Michie et al. ; licensee BioMed Central Ltd. 2011. This article is published under licence to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

The main steps are as follows.

Data collection

Step 1: specifying the target behaviour and population

Step 2: identifying theoretical domains that explain the behaviour

Step 3: identifying how explanatory domains should be targeted

Step 4: selecting standardised behaviour change techniques

Barriers to condom use

Content

On first using the website, users were asked to select the reasons why they personally did not use condoms from 12 possible options, which were identified through our fieldwork (condoms too tight or uncomfortable, reduced pleasure, not knowing when or how to suggest it, being drunk or having taken drugs, losing erection, being in a relationship, difficulty stopping in the heat of the moment, partner not wanting to use condoms, partner might be offended, sex does not feel as good, STIs are easily treated and they often break or slip off). The homepage (Figure 2) was then tailored to each individual user by ensuring that the content that addressed the barriers selected was displayed prominently in the centre of the page.

FIGURE 2.

The Men’s Safer Sex homepage with carousel of personalised tailored content.

Condoms: the basics

Content

This section included a short video demonstration and a click-through slideshow, which provided advice about using condoms correctly, highlighting the key steps in condom use and the areas in which people often make mistakes (Figure 3).

FIGURE 3.

Condoms: the basics – skills in correct condom application.

Condoms: tailored for you

Content

The intervention website aimed to educate men about different sizes and types of condom, using a tailored feedback activity. In this activity, users were asked to identify problems they had with condoms and then offered tailored advice about and recommendations for condom types to help address those problems (Figure 4). For example, men suggesting that condoms were uncomfortable, small or problematic for breaking were offered advice about larger types of condoms. This activity was similar to the ‘barriers to condom use’ activity but focused on problems with the actual condom (rather than problems surrounding condom use in general), and gives specific condom-related feedback and recommendations.

FIGURE 4.

Tailored feedback on selected barriers to condom use.

Pleasure

Sexually transmitted infections: are you at risk?

Content

The intervention included two interactive activities addressing STI risk, emphasising that the likelihood of acquiring STIs cannot be accurately judged (Figure 5). First, in ‘What’s the risk of STIs?’ a quiz presented facts and figures regarding STIs and their transmission (e.g. the number of people with undiagnosed HIV). This conceptualised the BCT of ‘information about health consequences’ in an interactive and visually appealing manner. Second, in ‘Are relationships safe?’ two animated diagrams demonstrated the way that STIs may spread within a network, common methods of transmission that people may not be aware of (e.g. oral sex) and how relationships may not be ‘safe’. This activity encompassed the BCT of ‘information about health consequences’, and also used ‘vicarious consequences’, by demonstrating the impact of risky sexual behaviours on others.

FIGURE 5.

Sexually transmitted infections: are you at risk? Infographics illustrating potential risks of STIs.

What women think

Slip-ups

Sexually transmitted infections: the facts

Communication

Reminders and plans

Content

In each section of the website, users were offered goals to set which related to the website content (Figure 6). When selected, these goals populated users’ own personalised ‘Reminders and plans’ page. Users could opt to receive a reminder by e-mail at a specific time, set time-limited goals (e.g. ‘I will purchase my recommended condoms’ by a selected date) or choose event-specific goals by forming an implementation intention58 (‘if-then plan’), identifying a potential situation for which condom use may be unlikely and then selecting a response to that situation.

FIGURE 6.

‘Reminders and plans’ feature.

Strategies for engagement

Discussion

Triangulation between a targeted literature review, expert workshops and interviews with the target population, all led by a multidisciplinary team, ensured that a range of potential influences on condom use were captured and that feedback to men on barriers to condom use were relevant to them. The resulting intervention is extensive, tailored to individual needs and targets a wide set of influences on sexual health. This is in line with recommendations that sexual health interventions should use a holistic approach to sexual health and well-being. 12

As with many complex interventions, the Men’s Safer Sex website contains multiple components, targeting a number of influences on behaviour. Therefore, it can be difficult to determine which part(s) of the intervention are effective and via which mechanisms. Online interventions offer the possibility of easily monitoring patterns of intervention use, which can facilitate the analysis the mechanisms of action of an intervention. Furthermore, clearly describing the intervention aims and content assists in this analysis. BCTs provide a standardised method for this process of describing intervention content.

The utility of the Behaviour Change Wheel

The BCW provided a useful and manageable framework for integrating information from multiple sources to design an intervention, especially for organising the information conceptually, for guiding the process of identifying influences on behaviour, and for selecting and delineating standardised methods for targeting those influences.

The advantage of using an integrative theoretical framework, such as the TDF, over a single theory of health behaviour is that they encompass multiple explanatory domains and, therefore, provide a more comprehensive assessment of factors which are important to the target population. 23 This is illustrated in a recent review of theories of behaviour change which found that only 3 out of 83 theories were integrative and set out to be comprehensive. 62,63 Models such as the Theory of Planned Behaviour64 present a cognitive model of behaviour (attitudes, perceived norms and perceived behavioural control predict intentions which, in turn, predict behaviour), downplaying impulsive factors in decision-making, which may influence motivation (an important influence on behaviour). Beyond identifying potential influences on behaviour, the BCW also provides structured guidance for determining the content and format of interventions.

Although the TDF is helpful in that it offers an extensive list of potential influences on behaviour, some domains were difficult to conceptualise within this context. ‘Goals’ was included as an important domain, but largely as a method of encouraging people to implement the advice given in the website and to target other domains (such as memory, attention and decision processes). Although the aim of the intervention was to improve intention, this was approached by targeting other domains. The TDF is a framework, rather than a theoretical model, and so does not specify causal pathways between variables. The intervention developer must draw on other evidence and theories to decide and elaborate on potential relationships between variables.

The use of standardised BCTs to specify the intervention content provided two advantages. First, the BCTs provided ideas for website features and health promotion messages (so the authors did not start with a ‘blank canvas’). Second, the BCTs were used to specify the content in standardised terms to facilitate replication, judgements regarding quality and comparisons with other interventions. 30 Translating BCTs into interactive website features can be a difficult process and requires a certain level of creativity. Given that the content of complex interventions is often not described in detail,20 building a repository of examples of such features would be a valuable resource for intervention designers.

The study had some limitations. For example, although we collected detailed data to specify intervention content, the literature review was targeted, rather than fully systematic, owing to time and resource constraints. This means some relevant evidence may have been missed; however, the inclusion of a number of reviews in the literature review mitigates this concern. A second limitation is that all men interviewed during the development process were sampled from sexual health clinics within inner London, thus potentially limiting the transferability to other populations and settings. However, the findings from our literature review confirmed the importance of the emergent themes from the interviews.

This chapter has provided a detailed description of an evidence-based IDI for sexual health, including how BCTs were translated into practice within the design of the Men’s Safer Sex website. It is hoped that this will assist intervention developers in their development work and reporting in terms of BCTs.

Design

This study is a feasibility RCT with random allocation to the Men’s Safer Sex website plus usual sexual health clinical care compared with usual sexual health clinical care only.

Participants and setting

Participants were recruited from three English sexual health clinics, two in central London and one in the Midlands. These clinics were selected as they serve a diverse range of patients in terms of age, socioeconomic status and ethnicity.

Inclusion criteria

• Men aged ≥ 16 years (with no upper age limit).

• Able to read English.

• Have access to the internet.

• At a high risk of future STI [i.e. two or more partners in the past year (male or female) and non-condom use in the past 3 months, or symptoms of suspected acute STI, or seeking treatment for a STI].

• For whom at least half of their sexual partners are female.

Exclusion criteria

Men who were HIV positive and men with hepatitis B or C were excluded, as patients with these diagnoses are likely to receive more intensive input in the course of routine clinical care. Men who had ever had sexual experiences only with other men, men who had sexual experiences more often with males but at least once with a female, or men with no sexual experience at all65 were also excluded.

Procedure

Research procedures were conducted online: a tablet computer (iPad) was positioned in the sexual health clinic waiting rooms or a side room. Leaflets and posters advertised the study and invited potential participants to approach the iPad (see Appendix 7); research and clinic staff also directed patients towards the iPad. Enrolment was self-directed, with staff available to help with technical problems. Trial software on the iPad guided participants through eligibility questions, informed consent and baseline demographic and outcome questions (see Appendices 8–10). Most online questions were compulsory and free-text comment boxes were offered at the end of groups of questions. All participant data were stored online using secure external servers. Following completion of baseline questionnaires, an automated computer algorithm allocated participants randomly either to the control group (at which point they were thanked for their time and asked to log out) or the intervention group (at which point they were directed to the intervention website). Those allocated to the intervention group were asked to engage with the Men’s Safer Sex website for as long as they wished (usually until the point at which they were called in to see health professionals). They also had access to the website by logging in after leaving the clinic.

Intervention

Participants in the intervention group were invited to interact with the Men’s Safer Sex intervention website while in the clinic waiting room. The time available for this varied by participant but we expected that participants would have an average of 10 minutes to interact with the intervention. Participants in the intervention group were also able to access the intervention website after leaving the clinic, for the duration of the follow-up period (12 months), using a username and login that they had chosen. Monthly e-mails were sent to participants in the intervention group, reminding them to visit the website, giving ‘teaser’ web links to intervention content. In addition, participants received e-mails reminding them about goals they had set on the website (see Chapter 2).

At 3, 6, 9 and 12 months after their initial clinic visit, participants were invited via e-mail to complete an online follow-up questionnaire. If they did not initially complete the questionnaire, they received three further e-mail prompts, at 1-week intervals, as well as two short message service (SMS) messages to their mobile phone (again, including the web link) at 1-week intervals alongside the previous two e-mails. If participants still did not respond, the researcher telephoned them 1 week after the final e-mail and SMS, reminding them about the questionnaire. Participants were given a 7-week window to complete their questionnaire. This window was extended at 3 months (until the 6-month questionnaire was due) in order to maximise responses, as technical issues had caused participants some difficulty in logging in. Information about STI diagnoses were collected at all time points via self-report and at 12 months by recording diagnoses or suspected diagnoses over the past 18 months (to control for pre-baseline service usage) recorded in the clinical notes at the sexual health clinics that participants were recruited from.

Outcomes

We adapted the Sexunzipped online sexual health questionnaire66 to measure outcomes and mediators of condom use. We selected items for inclusion based on a literature search for established measures and consultation with experts. Interviews were conducted with men in sexual health clinics (n = 11) to gain feedback on successive versions of the outcome questionnaire.

In the light of evidence that measurement alone may prompt behaviour change,67,68 a limited number of outcomes were measured at baseline (condom use and the main mediators, and self-reported STI diagnoses) and a full range of outcomes were assessed at 3, 6 and 12 months (Table 4). The primary feasibility outcome was retention in the study at 3 months, which was measured by response to the 3-month online questionnaire. The main behavioural outcome of interest was self-reported number of episodes of condomless sex at the 3-month follow-up. All self-assessments had a recall period of the previous 3 months. A full copy of the follow-up questionnaire (which assessed all outcomes) can be found in Appendix 7.

Data from medical records

Participants were asked to provide consent to access their medical records held at the sexual health clinic from which they were recruited from the study. A study researcher located files from patient records, collecting information for sexual health clinic attendances 6 months prior to participants’ date of recruitment to the trial and for the duration of the trial. Data were recorded on a Microsoft Excel^® 2010 (Microsoft Corporation, Redmond, WA, USA) spreadsheet with no participant identifiable information included; the unique participant identifier linked the clinic data to participant questionnaire data.

Sample size

This was a pilot RCT with a primary aim of assessing the success of recruitment and retention, engagement with the intervention and the acceptability of trial procedures to participants and clinic staff. The possible effect size of key outcomes, including condom use over the past 3 months, were assessed to inform power calculations for a future Phase III RCT.

Power calculations were performed based on data from the Sexunzipped online trial. 66 The study was powered to allow estimates of the effect of the intervention on episodes of condomless vaginal sex over the past 3 months. A sample size of 166 (intervention, n = 83; comparator, n = 83) randomised 1 : 1 between experimental and control conditions is adequate to detect a reduction of 1.35 episodes of condomless sex with a conventional two-sided α of 0.05 and 90% power (1 – β). Allowing for potential loss to follow-up at 3 months, 122 participants (intervention, n = 61; comparator, n = 61) is adequate to find a reduction of 1.35 episodes of condomless sex with a conventional two-sided α of 0.05 and 80% power (1 – β). In addition, this sample size is also sufficient to detect a 1.65 difference in safer sex intention and a 1-point difference in self-efficacy on Likert scales, with a conventional two-sided α of 0.05 and 90% power (1 – β). A total of 176 participants were randomised and, after later exclusions and withdrawals, 159 remained.

Randomisation

Once participants had been checked for eligibility, given informed consent and submitted baseline data, they were allocated 1 : 1 using a computer algorithm randomisation system to either the intervention or control group. The allocation was automated and instant and, therefore, was concealed until the point of allocation. Participants were informed with an automated message on the iPad and this allocation was unalterable.

Blinding

It was not possible to blind participants to their allocation, as those in the control condition did not receive access to any website. Research staff (including those conducting the analysis) were blind to allocation condition, as allocation was performed using the software on the iPad; however, the researcher performing recruitment may have been aware of allocation owing to the amount of time participants spent on the iPad. Allocation was revealed only once statistical analyses were complete.

Statistical methods

We present descriptive statistics by group at baseline and follow-up for main and secondary outcomes. The treatment effect at follow-up was estimated using generalised linear models with a log_e link and Poisson error. The log_e of the baseline values were included in each model as explanatory variables to account for baseline differences. Standard errors were estimated using variance components to account for overdispersion in the models. Incidence rate ratios (IRRs) comparing the control with intervention groups were estimated from these models along with their 95% confidence intervals (CIs) and p-values.

Recruitment

Participants were recruited between 28 April 2014 and 7 July 2014. Follow-up data were collected between 28 July 2014 and 2 July 2015. Recruitment ceased once target numbers were reached. Participant flow is outlined in Figure 7. Owing to a technical error in the online recruitment software, reasons for exclusion were available only for 20 participants.

FIGURE 7.

The Consolidated Standards of Reporting Trials diagram showing flow of participants.

One hundred and seventy-six participants were randomised, of whom seven withdrew from the study, one was a duplicate account and nine were withdrawn owing to being later found to be ineligible (they had been erroneously classed as eligible because of a technical error in the eligibility screening questionnaire). Fifty-five responded to the baseline questionnaire and the 3-month follow-up questionnaire (the principal outcome point) (22 in the control group, 33 in the intervention group). Forty-six participants responded to the baseline questionnaire and to all three follow-up questionnaires (27 in the control group, 19 in the intervention group). Data on STIs from clinic records were available for 149 (93.7%) participants. We monitored free-text comments on the online outcome questionnaires and found no reason to terminate the trial early. We did not conduct interim data analyses as the primary aim was to assess retention at follow-up.

Participants

Characteristics of participants are shown in Table 5. The mean age of participants is in line with data that suggest people in the 25–34 years age bracket are more likely to attend STI testing76 and employment status and ethnicity were comparable with the general population. 77,78

Group comparisons for condomless sex and sexually transmitted infections

Measures of condomless sex with female and male partners were not significantly different between the intervention and control groups at 3 months (Table 8). Twenty-three diagnoses of acute STI (one or more over 12 months) were recorded in the clinical notes of 15 trial participants (non-specific urethritis, n = 7; Chlamydia trachomatis, n = 7; Neisseria gonorrhoeae, n = 3; epididymitis, n = 2; molluscum contagiosum, n = 2; and people treated as contacts of someone with chlamydia, n = 2). New acute STI diagnoses were recorded for 8.8% (7/80) of men in the intervention group and 13.0% (9/69) in the control group over the course of 12 months. The IRR indicated a 25% lower cumulative incidence in the intervention group than the control group, but this was not a significant difference (IRR 0.75, 95% CI 0.29 to 1.90).

Discussion

In this pilot trial we explored the feasibility of a full-scale online RCT (in sexual health clinic settings) of the Men’s Safer Sex website for MSW. Recruitment into the trial was a success: the target sample size was achieved within 2 months, with the highest rates of recruitment on days when a researcher was present to remind staff and patients about the study. Retrieval of clinic records for clinical STI diagnoses was excellent (94%); however, response rates for online outcome measurement were poor (36–50%) despite a number of strategies to improve follow-up, including offers of incentives, repeated e-mail prompts, text messages and telephone calls. There were no reported harmful effects from the Men’s Safer Sex website or online trial. We discuss the implications of this feasibility trial for the conduct of online trials of digital interventions in sexual health clinic settings.

Problems with software and patient Wi-Fi access

We encountered technical issues at all time points which hampered recruitment and registration, participant access to the Men’s Safer Sex website, online data collection and retention of men in the trial at 3-, 6- and 12-month follow-up points. The NHS lags behind many other institutions in terms of patient access to digital services;79 therefore, it was difficult to set up and maintain access to Wi-Fi for patients in clinic waiting rooms because the process for permission and set up was complicated, connections were often poor and staff were not always confident about remedying internet access problems. 80

There were substantial software development errors that had an impact on recruitment of eligible participants as well as data collection and access to the intervention (e.g. errors with links to the outcome questionnaire, login and password reset functions). Despite careful specification of requirements for the project, there were delays to intervention development, there was a faulty algorithm for participant selection, user experience of the website was not always positive, there was failure of automated analytic data collection, and the login/website access was occasionally faulty. Despite careful specification of requirements, and extensive manual testing by the research staff, the time available for automated testing and user testing was severely reduced and the extent of these problems was realised only after patients had been recruited. Reliable software frameworks for online RCTs are needed, which are customisable, user-friendly and which meet requirements for secure data protection and data storage.

Ethical issues

It is essential that participants in research are given the information that they need to give informed consent and that confidentiality and data security are ensured. 81 However, we encountered are several tensions between these principles and the practical realities of (online) research.

Engagement with the Men’s Safer Sex website

Engagement with digital interventions for health promotion can be a major challenge. 84 We placed the intervention in clinic waiting rooms to take advantage of the waiting time that is common in drop-in sexual health clinics; however, this depended on flow of patients into the clinic. One-third of the intervention group did not see the Men’s Safer Sex intervention website, which may have been a result of lack of available time. We shortened the length of the outcome questionnaire by capturing only key outcomes at baseline, but the research procedures still took time (especially if participants were affected by the technical problems described in Problems with software and patient Wi-Fi access). 80

We sent e-mails at 1-month intervals to prompt men to access the intervention and one-third of participants did visit the website after their initial clinic visit. We expected that the main opportunity for intervention would be at the time of a clinic visit but, as discussed above, the time available for this was limited. We offered financial incentives for follow-up data collection but not for engagement with the intervention itself. The best methods for engagement need to be established, for example integrating the Men’s Safer Sex website as a step in the patient pathway through the clinic. 79

Sexually transmitted infection diagnoses in clinic records

We located clinic records with usable data on STI diagnoses for 94% of participants, searching by names given, date attended, e-mail address or telephone number. The location of data from clinic records could be improved by requesting additional information from participants (e.g. date of birth, address, clinic number); however, soliciting additional identifiable data may discourage some people from participating in research. 80 Almost half of the sample (72/159) returned to the same clinic within 1 year; however, rates of STI acquisition and service use may have been underestimated as participants are free to attend other sexual health clinics and/or their general practitioner (GP) and records are not shared between clinics. This would undermine the validity of clinically recorded STI diagnosis as an outcome if patients in different experimental conditions returned to the original clinic at different rates. It might be possible to capture STI diagnoses made at other sites by using Hospital Episode Statistics and GP data. However, patients in sexual health clinics can choose pseudonyms and may be wary about confidentiality. 80

Recording STI diagnoses from clinical notes has several advantages over self-reported online outcome measurement: clinic record data are not subject to recall bias or social desirability bias, and data collection from records is cheaper in terms of researcher time and costs of participant incentives. In addition, filling in a sexual health questionnaire may prompt participants to reflect on their behaviour (for both control and intervention participants),68,80 potentially reducing the apparent effect of an intervention. Recording diagnoses from clinic records allows collection of cumulative incidence of STI over time, which is likely to be more accurate than self-reported outcomes. The cumulative incidence of acute STIs over 1 year was 9.4%, which compares with an annual recorded rate of 0.85% for new STI diagnoses for men attending sexual health services in England. 85 We were therefore successful in recruiting men at substantial risk of STI.

Online outcome measurement

Online trials have a number of potential advantages including automated enrolment, randomisation, data entry and reminders. 79 Online questionnaires can capture internally valid data on a range of outcomes including sexual well-being outcomes, which reflect participants’ priorities. 68,86

However, although online recruitment can be good, there can be high drop-out rates which limit validity. 79,87 The response rates for Men’s Safer Sex online questionnaires sent by e-mail were poor (maximum of 50%), despite an increased level of incentive for the final questionnaire (£30 vouchers). We used evidence-based methods to enhance retention, that is contacting participants multiple times via different routes (by personalised e-mail, text message and telephone calls). 66,87,88 However, we do not know whether or not e-mails ended up in junk mail or how many participants were put off by the initial technical problems with password access and malfunctioning links to the outcome questionnaire. The larger incentive offered at 12 months appeared to have a positive influence on the response rate (an increase from 36% to 50%), which is in line with studies which tested the effect of larger incentives. 66,89

Sexually transmitted infection-related health-care resource use and costs

Participants completed questionnaires on sexual health-related resource use at 3, 6 and 12 months. Differences in resource use at each follow-up point between groups have been tested using the chi-squared test and any significant differences at an α < 0.05 reported. Unit costs applied to resource use and their sources are reported in Table 9. It was not possible to tell whether or not STI tests and treatment occurred in the same or different appointments, so it was assumed that a blood test, urine test and swab would all occur in the same appointment and then additional appointments were counted for each subsequent time they had had the same test. Furthermore, the assumption was made that treatment happened in an additional appointment. For example, if a participant had two urine tests and one blood test but received treatment once, it was assumed that they had attended three appointments – two for the tests (one for each test type) and one for the treatment. For sexual health advice, this was added on as an additional £20 per instance in line with the NICE costing template for sexual health services. 1

Mean total cost per patient at each follow-up point has been calculated and reported for complete cases (questionnaire completion at all follow-up points). In addition, 95% CIs have been calculated using bootstrapping and 1000 iterations. 102 No baseline resource use data were collected via patient questionnaires so as to maximise the amount of time participants in the intervention group could spend on the intervention. As a result, sexual health-care resource use collected using questionnaires could not be adjusted for baseline differences between groups.

Data from participant clinical records were used to calculate (1) the cost of the appointment that directly followed recruitment to the trial and (2) the cost of any follow-up appointments that occurred between recruitment and follow-up. Information on appointments that occurred during the 6 months prior to recruitment was used to adjust for differences in resource use using regression analysis. Costing was carried out using a bottom-up approach costing for clinician time (including on-costs and overheads),97 diagnostic tests2,99 and medication. 100

All costs are reported as 2014 values. Costs prior to 2014 were inflated to 2014 values using the hospital and community health services index. 97

Cost of the intervention

The total cost of developing and maintaining the intervention website was collected as part of the project management of the trial. There is no definitive way to incorporate the cost of a digital intervention into an economic evaluation. 79 The complications with doing so are the result of a number of factors, including (1) the intervention has now been developed and, hence, could be considered a ‘sunk cost’ as no further costs are required to develop it, (2) it is free to patients and providers at the point of access, (3) it is unclear how many people will go on to use the site therefore the cost per patient is hard to calculate and (4) it may be hard to disentangle costs associated with the trial (e.g. developing questionnaires and databases to store trial-related data) from the costs of developing the intervention and day-to-day maintenance if it were to be made widely available. As a result, we report the different costs associated with developing the website and the number of individuals that accessed the website to give some estimation of cost per individual.

If QALY gains are seen in the intervention group compared with the control group, we can also report the number of patients that would need to access the website for it to be cost-effective at the lower end of the willingness-to-pay (WTP) threshold of £20,000 per QALY, which is the threshold generally deemed likely for NICE to approve the implementation of an intervention. 103

Incremental cost-effectiveness ratio

The mean costs and QALYs calculated above will be used to calculate the mean incremental cost per QALY gained, known as the incremental cost-effectiveness ratio (ICER), of the website intervention compared with control over the 12-month duration of the trial for all analyses specified above. The number of infections prevented has been taken from the statistical analysis reported in Chapter 4 and used to calculate the mean incremental cost per infection prevented.

Cost-effectiveness acceptability curve

A cost-effectiveness acceptability curve (CEAC) reports the probability that an intervention is cost-effective compared with the control. This is based on running the analysis a number of times, in this case 1000 times using bootstrap sampling from the data collected and recalculating the statistic of interest, and counting the number of times that the ICER is below the WTP threshold. This is divided by the number of times the analysis has been run to give the probability. The results of the bootstrap were used to construct CEACs for a range of values of WTP for a QALY gained. The probability that the website is cost-effective compared with the control at a WTP for a QALY gained of £20,000 is reported in Results.

Sensitivity analyses

The number of pregnancies (in female sexual partners) was also reported in the patient questionnaires. These are rare and expensive events that may be hard to capture and could skew the results; therefore, they have been included in the sensitivity analysis only. Only pregnancies after baseline were included given no other cost information is available at baseline. All pregnancies after baseline were reported as miscarriages which have an average cost of £551 per miscarriage. 99 The sensitivity analysis was run using the same methodology as above but including the cost of pregnancy.

Completion of questionnaires

There was a large proportion of missing data owing to poor follow-up rates. Analyses have been conducted on complete cases (follow-up data at 3, 6 and 12 months for costs and also including information baseline for QALYs) in line with the statistical analysis plan and assuming that cases are missing at random. The number of participants completing questionnaires at each time point and complete cases are reported in Tables 10 and 11. From 72 participants in each group at baseline, there were 27 (37.5%) complete cases in the control group and 19 (26.4%) in the website intervention group.

Quality-adjusted life-years

Utility scores and QALYs calculated from responses to the EQ-5D-3L and the tariff developed by Dolan94 are reported in Table 10. There was no significant difference in QALYs between the two groups with an adjusted difference of 0.03 for intervention minus control group (95% CI –0.02 to 0.08).

Utility scores and QALYs calculated from response to the SQoL questionnaire and the tariff developed by Radcliffe et al. 72 are reported in Table 11. There was no significant difference in QALYs between the two groups with an adjusted difference of 0.001 for intervention minus control group (95% CI –0.02 to 0.02).

Table 12 reports the linear regression relationship between the EQ-5D-3L (index and QALY), any condomless sex and STIs. Table 13 reports the linear regression relationship between the SQoL-3D (index and QALY), any condomless sex and STIs. The only statistically significant relationship at p < 0.05 is between QALYs and STIs at baseline using the EQ-5D-3L.

Sexual health-care resource use and costs: from questionnaires

Sexual health-related resource use and costs are reported in Tables 14 and 15, respectively. There was no significant difference in resource use between the two groups and there were only minor differences in costs between the two groups. At p < 0.05, the intervention group had significantly higher costs of self-tests for STIs: £1.85 (95% CI £0.00 to £7.45) versus £0.00. The control group had significantly higher sexual health clinic costs at the 6-month follow-up: £48 (95% CI £0.00 to £143) versus £12.70 (95% CI £0.00 to £54.70).

There was no significant difference in costs between the two groups with a total mean incremental cost of –£24 (95% CI –£145 to £97) for the intervention minus the control group if pregnancy costs are excluded, and –£65 (95% CI –£193 to £63) if the cost of pregnancy is included. Sexual health clinic costs accounted for 88% of the cost in the control group and 84% of the cost in the intervention group.

Sexual health-care resource use and costs: from clinical files

Clinical files could be located for the sexual health clinic appointment directly after recruitment for 69 (92%) participants in the control group and 80 (95%) of participants in the intervention group. Thirty-five participants (47%) in the control group had clinical records for appointments up to 12 months after recruitment and 39 (46%) in the intervention group. Participants who attended the same sexual health clinic after recruitment and whose clinical files could be located had an average of 2.3 appointments over the 12 months for the control group and 2.5 appointments in the intervention group. This is compared with the questionnaire data for which 40% of participants report a blood, swab or urine test at a sexual health clinic in the control group at 12 months and 55% in the intervention group, with an average number of attendances of two in the control group and 1.3 in the intervention group. The proportion with appointments combined with the number of appointments results in a significantly higher average number of appointments recorded in clinical files than in questionnaires for the intervention group (p = 0.04) but does not differ significantly for the control group (p = 0.45).

Costs using a bottom-up costing approach and number of appointments attended pre-recruitment, directly after recruitment and post recruitment using data only from participant clinical files are reported in Table 16. If no record is available, it is assumed that there was no appointment and, hence, the cost was zero. There was no significant difference in costs between the intervention and the control group. Adjusting for baseline differences the incremental mean cost per participant of the intervention group compared with the control group was £12 (95% CI –£8 to £32).

Quality-adjusted life-years decrement from sexually transmitted infections and resource use from clinical records

Out of the 42 participants with complete QALY data and data available from clinical records, 10 participants had a STI diagnosis at recruitment: genital warts, n = 6; genital chlamydia, n = 2; genital herpes, n = 1; and urethritis, n = 1. Patients with a STI diagnosis at baseline had significantly lower QALYs of 0.843 over the 12 months than 0.926 for participants with no diagnosis of STIs at baseline (p = 0.03).

To calculate QALYs for patients using the clinical files, a QALY decrement of 0.083 was applied to any patient with a STI at baseline or who had a new STI diagnosis during the 12 months after recruitment. All other patients had QALYs of 0.926 during the 12 months of the study.

There was no significant difference in QALYs between the control and the intervention group, with a mean difference in QALYs for the intervention minus control of 0.004 (95% CI –0.011 to 0.020).

Incremental cost-effectiveness ratios and cost-effectiveness acceptability curves

In analyses 1, 2 and 4, the intervention dominates the control in that it results in more QALYs and lower costs. In analyses 3 and 5, in which patient files are used to calculate costs, the intervention results in an additional adjusted cost of £12 and is more effective (but not by a significant margin).

In analysis 3, in which QALYs are calculated based on the disutility of having a STI, the intervention results in an additional 0.004 QALYs per participant for an additional cost of £12. This translates to a mean incremental cost per QALY gained of £3000.

In analysis 5, in which costs are calculated from patient files, the intervention prevents four infections per 100 patient-years for an additional cost of £12 per patient. This translates to a mean incremental cost per STI prevented of £291.

The CEACs are depicted in Figures 8 [for results calculated using the EQ-5D-3L (analysis 1) and those derived from patient files (analysis 3)] and 9 [for results calculated using the SQoL questionnaire (analysis 2)]. Using data from questionnaire responses and calculating QALYs from the EQ-5D-3L, in 88% of instances the intervention is cost-effective compared with control at the WTP for a QALY threshold of £20,000. This is the same regardless of whether the cost of pregnancies is excluded or included. Using patient files for resource use and calculating QALYs from the disutility of STIs, there is a 68% probability that the intervention is cost-effective compared with control. If QALYs are calculated using the SQoL questionnaire, there is a 61% probability the intervention is cost-effective if pregnancy costs are excluded and 69% if they are included.

FIGURE 8.

Cost-effectiveness acceptability curve of intervention compared with control: QALYs calculated from EQ-5D-3L and questionnaires (with or without pregnancy) or patient files.

FIGURE 9.

Cost-effectiveness acceptability curve of intervention compared with control: QALYs calculated from SQoL.

Cost of the website

The total cost of the developing the website was £101,515 (Table 17). Activities and costs not associated with the intervention aspect of the website have already been excluded from the costs. In the trial, 84 participants in the intervention group accessed the website, resulting in a cost per patient of £1209 for the website.

Using the results of analysis 1, QALYs calculated from the EQ-5D-3L tariff and using responses to patient questionnaires to calculate costs, the intervention resulted in an average cost saving of £24 per participant in sexual health service use. Taking away the cost savings from the cost of the website leaves a total mean cost per participant for the intervention group compared with the control group of £1185. There was an adjusted QALY gain of 0.03 in the intervention group compared with the control group. This equates to an incremental mean cost per QALY of £39,466.

There is no reason why access to the website should be limited to the 84 participants who accessed it as part of the trial. Assuming that there is an average cost saving per participant accessing the intervention of £24 and an average QALY gain per participant or 0.003, given there is no additional cost per participant to access the website as the costs have already occurred, an additional 26 participants would need to access the website for an ICER of £30,000, which is the upper end of the NICE threshold for cost-effectiveness. Using the results from the clinical files, that is assuming that the intervention costs an average adjusted cost of £12 more than no intervention and results in 0.004 QALYs, the current ICER is £305,128 per QALY gained. An additional 856 participants would need to access the website for the ICER to be below £30,000 with a cost per participant for the website of £108.

Qualitative study design

The study was a qualitative interview study, using semistructured interviews. We used three qualitative data sources to assess the acceptability and validity of the pilot RCT methodology: (1) individual interviews with 11 male RCT participants, (2) individual interviews with nine clinic staff who assisted with the study recruitment in each of the sexual health clinic research sites and (3) comments made in free-text boxes on the 3-, 6- and 12-month online outcome questionnaires.

Data source 1: male sexual health clinic user interviews

Male study participants were recruited in the same way as participants in the Men’s Safer Sex pilot RCT, registering on a tablet computer (iPad) placed in the waiting rooms of sexual health clinics. Forty-eight men were invited to participate in interviews. Their experience of the RCT study procedures were the same, except that e-mail follow-up was at 2 weeks only instead of at 3, 6, 9 and 12 months (to enhance recall of trial procedures). An additional question in the online recruitment process asked permission to contact the men for a post-study interview. Their quantitative outcome data were not included in the pilot RCT data analysis. Ethics committee permission was provided by the City and East NHS Research Ethics Committee (study reference number 13/LO/1801).

We interviewed 11 male sexual health clinic users who had enrolled for the Men’s Safer Sex pilot trial. In order to recruit a diverse sample, as the recruitment proceeded we undertook theoretical sampling on the basis of age, trial allocation condition (i.e. intervention or control) and response to follow-up (positive/negative responses to requests for follow-up data).

Data source 2: staff interviews

Analysis of qualitative interview data

ATLAS.ti software (version 7.5.11, Cincom Systems Inc., Berlin, Germany) was used to facilitate data retrieval, coding and linkage, and to record analytic notes. Thematic analysis was used to identify patterns and themes within the data. One researcher independently coded text from the transcripts, categorised data by theme and identified relationships between different elements of the text. A data meeting was held to discuss the coding schema and coding decisions were reviewed by a second researcher.

Data source 3: free-text questionnaire comments

Space was offered for voluntary free-text comments after each cluster of questions on the quantitative online outcome questionnaires. At the end of each follow-up survey, participants were also asked whether or not being in the study had any good or bad effects on their lives.

Quantitative survey data were available for 159 men after excluding men who withdrew from the study and those later found to be ineligible. A total of 281 free-text comments on outcome questionnaires for these 159 men were analysed. Forty-six men out of the 159 eligible participants commented on the good or bad effects of the study at the end of at least one of the 3-, 6- or 12-month outcome questionnaires. Comments were exported in a Microsoft Excel file and coded thematically in situ.

Reasons for participating in the research

Participants commonly cited boredom and a desire to help the researchers as their main reasons for participating. The promise of a voucher was described by many as an added bonus but most participants said they would have taken part without this incentive. Features of the study which made it more appealing were anonymity, the flexibility to fit around patients’ consultations and that it was short. For most men, the particulars of the study were unimportant, but many were reassured by the academic nature of the study and would have been less keen to participate had it been a marketing questionnaire for a private company.

Participants’ understanding of the purpose of the research

Participants’ understanding of the research was often incomplete. Many felt it was purely an exercise in data collection and did not realise that the website was intended as an intervention to change behaviours. However, participants were happy to proceed without full understanding of the purpose of the research. By virtue of the study’s affiliation with a university (UCL) and with the NHS, men trusted that the study would be trying to do something good and that the details were not particularly important. Two participants had initial concerns that the study may have been sponsored by condom manufacturers, which led them to feel cynical. Others agreed that they would have felt uncomfortable about participating in a marketing questionnaire.

Experience of completing the research in the clinic

All three of the clinics offer sexual health appointments on a drop-in basis and the waiting time varies by time of day and day of the week. The iPads were positioned so that men could access them while waiting to see clinical staff but the placement of the iPads varied. In one clinic, the research was conducted in the waiting room and in two clinics, participants were taken to a side room to complete research procedures and see the Men’s Safer Sex website in private. Staff described the tensions between making the iPad accessible in the main waiting room (and, thus, increasing access) and offering participants privacy. Some felt uncomfortable approaching potential participants and explaining the study in a public waiting room, especially as the study focused on MSW and staff sometimes knew that men were MSM who would not be eligible. Male participants agreed that privacy was important and most felt happy with the location of the iPad that they had experienced.

The staff who were interviewed had been concerned that patients might be worried about missing their appointments and one of their important roles was to ensure that this did not happen. Participants’ worries about missing appointments were pre-empted by the staff who explained the study.

Online outcome questionnaire

The questionnaire content was acceptable to interviewees and men said that they were honest in their questionnaire responses. Some commented that it was easier to be honest online than in person.

it’s just good for us to know, but were you honest in the answers that you gave in the questionnaire?

Yes . . . You know, I don’t see the point in saying, oh, I’m not talking about that, because it, at the end of the day, it can potentially harm the research, because you’re not getting the true picture, and it doesn’t . . . you know, it’s very unlikely that I’m ever, you know, going to knowingly meet any of the actual research team, you know, so . . .

Some interviewees did not realise the distinction between the Men’s Safer Sex intervention website and the outcome questionnaire (and so were not clear about whether they were in the intervention or control group).

Potential impact of the study on sexual well-being

Our quantitative results showed no significant differences in measured outcomes (although, as a feasibility RCT, the sample size was intentionally small). It was difficult for interviewees to unpick the influence of the research (outcome measurements with or without viewing the Men’s Safer Sex website) from the influence of visiting the clinic and the subsequent invitation for interview. However, participating in the study did seem to change some men’s attitudes and sexual health behaviours.

For some, the act of filling in survey questions prompted some changes:

. . . when I had to do the test and the questions are put in front of you then I started to realise how much I wasn’t being honest in my head . . . that’s what made me realise I had to change my life. That’s when the study helped me so much.

Participant 372 (interviewee)

I think it has been a positive experience. I found being submitted the same questions over and over and having to answer them, offered me the chance to ponder over this topic that often I might have overlooked. It has made me feel more conscious about my sex life.

Participant 332, control group, final comment on 12-month questionnaire

Made me think about protection and prevention before embarking on new relationship.

Participant 181, control group, final comment on 12-month questionnaire

It’s been good because it’s a little reminder to be careful if I were to cheat and it reminds me of going to get tested at the hospital and how much I hated it and it's much easier just to wear a condom.

Participant 215, control group, final comment on 12-month questionnaire

Good. I am more responsible in the fact I’m more open to talking about contraception and protection.

Participant 199, control group, final comment on 12-month questionnaire

Good effects. Nice to reflect on this aspect of life. I also spent the first £10 voucher on a new yoga mat. Thanks for that!

Participant 254, control group, free text comment on 6-month questionnaire

The outcome questions had no reported impact on other men, but the voucher incentives were valued.

I am not learning anything new, so it is not having any effect on my life beyond taking up some time and providing me with amazon vouchers . . .

Participant 167, control group, final comment on 3-month questionnaire

Among the intervention group participants, it is difficult to separate the potential impact of outcome measurement from impacts of viewing the Men’s Safer Sex website. Some participants learned new information as a consequence of participating in the study (presumably from the Men’s Safer Sex website, which contained multimedia features addressing STI transmission):

It has helped me understand more about the way that STI can be transmitted.

Participant 207, intervention group, free-text comment on 3-month questionnaire

There were differing views of the website’s promotion of condom use:

It has been helpful to revise good practice for safer sex. The study has reminded me of risk but not necessarily offered any new solution. I can't help but feel you are trying to promote condoms, an old message. Everything on the website comes back to ‘use a condom’ People, including me take absurd risks because sex with a condom is not the same thing as sex without. There are condoms that work a little better. Your website hedges around actually recommending a brand or type on the grounds of pleasure. That's what people want to know. Is there a condom that I can forget that I'm wearing?

Participant 208, intervention group, final comment on 12-month questionnaire

. . . in a good way I feel that this study has really challenged my sexual habits and made me question why I put my health at risk for a few seconds of intense pleasure when using condoms is not a chore and could save my life.

Participant 218, intervention group, final comment on 3-month questionnaire

However, some men felt that condom use/safer sex was not relevant to them:

I appreciate the study and fully support it, but as I am in a long-term relationship with a single partner, I find that a large amount of questions do not apply to me.

Participant 137, intervention group, comment on 6-month questionnaire

Some felt that participating in the study had no effect on their sexual health.

Final comment box: has being a part of this study had any good or bad effects on your life?

[N]ot really to be honest. I feel that generally speaking it was very obvious the advice that was given, although I do think it is very important to make people aware of some things that perhaps some people do not see as being obvious. Personally I found it obvious and I do not feel that it has had any affect on my sexual practice or my well being. However i am grateful to have taken part and I wish you all the success with your study:)

Participant 304, intervention group, comment on 12-month questionnaire

Potential adverse impacts of the study

Although most comments on the effect of participating in the study were neutral or positive, one man described an adverse impact of participating:

Final comment box: has being a part of this study had any good or bad effects on your life?

Not really, i mean, it reminds me of the sadness of losing my long term girlfriend and also imagining her having sex with other people, which is quite painful to think about but, in terms of my own sexual psychology i think i am fine . . . Honest answer . . . you did ask! ;)

Participant 304, intervention group, final comment on 3-month questionnaire

The place of digital interventions in sexual health care

Male clinic users and clinic staff were positive about the role that digital interventions could play in terms of being a useful resource for sexual health information either before or after clinic appointments. Offering a web-based resource in sexual health waiting rooms was felt to be appropriate.

I think the ideal time to do it would be the waiting room beforehand because you’ve almost got a captive audience. You’ve got like, a marketeer’s dream, haven’t you, because you’ve got people who can’t go anywhere, who are already actually by virtue of being there, a little bit engaged with the subject you want to talk to them about. So, it’s a little bit easier to talk to them about sexual health than just wondering up to them at Liverpool Street station and saying, I’m going to talk to you about sexual health.

Participant 353 (interviewee)

Clinic staff felt that the website could have a useful role in providing access to additional information for patients, at their convenience, but felt that a digital intervention could not and should not replace their role in communicating directly with clinic users.

I think anybody who comes to the clinic has . . . we should be promoting it. It’s not there to substitute any health education that we provide, it’s not there to substitute CBT [cognitive–behavioural therapy] that the psychologist might want to give to the patient, or motivational interviewing that the health advisor might want to do. It’s there as an additional tool to . . . that is accessible 24 hours a day to the patient. And there’s nothing wrong with us making an advert of it in the clinic or having a computer where patients can go and interact with in the waiting room, or having it as an app that they can access on their phones from home. [. . .] I don’t think it changes . . . you are what you are. We cannot be replaced by robots; this can only provide information but it almost ends up being almost one way because it’s quite a rigid system. It can only give you information based on the questions that are already set on the system. It doesn’t have the ability to pick up behaviour, it doesn’t have the ability to pick up any verbal cues that young people might be asking or be uncomfortable with so it can’t prompt, so it doesn’t replace human beings in any way, shape or form. [. . .] I think is has a role and we can use it as a tool. I don’t think it should be there to replace what we do.

Staff member 08 (clinic 2)

Clinic users agreed that face-to-face communication can offer advantages over digital interventions.

I don’t like the idea of it being used in a consultation because I, kind of, want to speak to the clinician then who’s the expert. You know, sometimes clinicians ask a second question, which actually prompts an outpouring of something that proves to be really useful and really, kind of, helps pinpoint a problem.

Participant 353 (interviewee)