E-health interventions targeting STIs, sexual risk, substance use and mental health among men who have sex with men: four systematic reviews

Review aims and questions

The aims of this review were to search systematically for, appraise the quality of and synthesise evidence to address the following research questions:

• Research question (RQ) 1. What approaches and theories of change do existing e-health interventions employ to prevent HIV, STIs, sexual risk behaviour, alcohol and drug use or common mental illness symptoms among MSM?

• RQ2. What factors relating to interventions, providers, participants or contexts promote or impede delivery or receipt of such interventions?

• RQ3. What are the effects of such interventions on HIV and STIs, sexual risk behaviour, alcohol and drug use, and depression and anxiety, overall and by intervention and client subgroup?

• RQ4. Are such interventions cost-effective in reducing these outcomes?

• RQ5. Does the existing evidence overall suggest that these outcomes can coherently, feasibly and effectively be addressed by a single, joined-up e-health intervention targeting UK MSM and, if so, what might such an intervention look like?

Review objectives

• To conduct electronic and other searches.

• To screen found references and reports for inclusion in the review.

• To extract data from, and assess the quality of, included studies.

• To develop a typology of interventions and synthesise theories of change and process evaluations.

• To consult with policy/practice and community stakeholders on the typology and theory of change/process synthesis.

• To synthesise outcome evaluation and cost-effectiveness data.

• To draw on these syntheses to draft a report addressing our research questions.

• To consult with policy/practice and community stakeholders on the draft report (to inform amendments and dissemination).

• To submit the final report to the National Institute for Health Research.

Types of participant

The review focused on gay, bisexual and other men (including trans men) who have sex with men, including those who have been diagnosed as HIV positive, as well as those whose last HIV test was negative or who have never tested for HIV. To determine study eligibility, we operationalised this focus by requiring that MSM constituted at least half of the sample.

Types of intervention

The review focused on interactive or non-interactive e-health interventions delivered via mobile phone applications (hereafter referred to as ‘apps’), the internet or other electronic media (i.e. electronic communication technology) that aimed to provide ongoing support to populations consisting entirely or principally of MSM to prevent HIV, STIs, sexual risk behaviour, alcohol and drug use or common mental illnesses. These could include interventions that also aimed to promote HIV treatment adherence or that addressed HIV testing or pre-exposure prophylaxis, as long as these also addressed sexual risk behaviour, substance use behaviour or mental health. It excluded e-health interventions facilitating merely one-off, as opposed to ongoing, support and those addressing HIV self-testing, clinic attendance or STI partner notification only. The e-health interventions were electronically delivered; interventions delivered by human providers via electronic media were excluded. The interventions could be distributed by commercial, statutory, academic or voluntary sector agencies.

Types of control

The review focused on treatment as usual, no treatment or other active treatment control groups.

Types of outcome

The review focused on HIV or STIs, sexual risk behaviour, alcohol consumption (e.g. self-reported alcohol consumption via questionnaires or diaries), legal and illegal drug use (e.g. self-reported drug use) and anxiety or depression (clinical or self-report measures). Studies were included if they addressed any, some or all of these outcomes. Outcome measures could draw on dichotomous or continuous variables, and self-reports or reports by other raters. Behavioural outcomes could use measures of frequency (monthly, weekly or daily), the number of episodes of use or an index constructed from multiple measures. Alcohol measures could examine alcohol consumption or problem drinking. Drug outcomes could examine drugs in general or specific illicit drugs, and could include drug convictions.

Types of studies

To address RQ1, we included process and outcome evaluations providing intervention descriptions or theories of change, as well as theoretical reports. To address RQ2, we included process evaluations. To address RQ3, we included outcome evaluations. To address RQ4, we included economic evaluations. To address RQ5, we drew on all of the above. Included theoretical reports described intervention theories of change, logic models or mechanisms of effect. Included process evaluations could employ any quantitative and/or qualitative design, but were required to report empirically how delivery or receipt varied by characteristics of intervention, provider, user or context using quantitative and/or qualitative data. These studies could report exclusively on process evaluations or report process alongside outcome data. Included outcome and economic evaluations employed prospective experimental or quasi-experimental control groups.

Database search strategy

The search string used in a limited preliminary search in PubMed (see Appendix 1) informed the development of a more sophisticated search strategy (see Appendix 2), thereby maximising sensitivity, as recommended by the Cochrane Handbook for Systematic Reviews of Interventions. 63

Search strategy for other literature sources

The following clinical trials registers were searched for relevant ongoing and unpublished trials:

• ClinicalTrials.gov (complete database as of 27 April 2020).

• World Health Organization International Clinical Trials Registry Platform (ICTRP) (complete database as of 26 November 2018). Owing to the COVID-19 pandemic, the ICTRP search functionality was removed. Therefore, this search could not be updated.

• EPPI-Centre Trials Register of Promoting Health Interventions (TRoPHI) (full database as of 27 April 2020).

Search terms were derived from the OvidSP MEDLINE search compiled for database searching. All trial details were examined for their relevance and included if they met our inclusion criteria.

To find other grey literature, the complete OpenGrey database was searched on 1 November 2018 and again on 27 April 2020, using a version of the OvidSP MEDLINE search compiled for database searching. Google (Google Inc., Mountain View, CA, USA) was searched on 21 November 2018 in incognito mode to look for non-governmental organisation and governmental publications. Search terms were derived from the OvidSP MEDLINE search compiled for database searching. The first 100 results for each search were examined for their relevance and included if they met our inclusion criteria. The Google search was not updated in 2020.

Full details of the search strings used for these sources can be found in Appendix 2 and Appendix 3 and are available at the London School of Hygiene & Tropical Medicine’s Data Repository. 67

We also carefully searched reference lists from all studies that met our inclusion criteria. We contacted subject experts during the initial search and again at the time of our updated search to identify relevant ongoing or completed research (see Appendix 5). Our protocol specified that we would hand-search journals that published included studies that were found only via reference checking, and which were not indexed on databases we had searched, but no journals met this criterion.

Assessment of theories of change

We assessed the quality of descriptions of intervention theories of change using a modified version of the criteria developed in our previous systematic reviews of school-based interventions integrating health and academic education, positive youth development and school health education interventions,77,78 modified in the light of our more recent work on realist methods. 79 The assessment focused on (a) the extent to which the theory of change described the path from intervention to outcomes, (b) the clarity with which theoretical constructs were defined, (c) the clarity with which causal inter-relationships between constructs were defined, (d) the extent to which the mechanisms underlying these inter-relationships were explained and (e) the extent to which the intervention theory of change considered how mechanisms and outcomes might vary by context. The two reviewers then met to compare their assessments, resolving any differences through discussion and, when necessary, by calling on a third reviewer.

Assessment of process evaluations

We assessed the quality of the qualitative and quantitative elements of process evaluations using an EPPI-Centre tool. 80 The assessment addressed the rigour of sampling, data collection and data analysis; the extent to which the study findings were grounded in the data; whether or not the study privileged the perspectives of participants; the breadth of findings (i.e. the extent to which the study explored a broad range of process issues); and the depth of findings (i.e. the extent to which the study provided in-depth insights into participant perspectives). This was then used to assign studies to two categories of ‘weight of evidence’. First, reviewers assigned a weight (low, medium or high) to rate the reliability or trustworthiness of the findings (the extent to which the methods employed were rigorous/could minimise bias and error in the findings). Second, reviewers assigned an additional weight (low, medium, high) to rate the usefulness of the findings (i.e. the extent to which these could shed light on how processes of intervention delivery/receipt varied with characteristics of interventions, providers, participants or contexts). Guidance was given to reviewers to help them reach an assessment on each criterion and on the final weight of evidence. The two reviewers then met to compare their assessments, resolving any differences through discussion and, when necessary, by calling on a third reviewer.

Assessment of outcome evaluations

For outcome evaluations, we assessed the risk of bias of each included experimental study using the tool outlined in the Cochrane Handbook for Systematic Reviews of Interventions. 63 For each study, two reviewers independently judged the likelihood of bias in seven domains: sequence generation, allocation concealment, blinding of participants or personnel, blinding of outcome assessors, incomplete outcome data, selective outcome reporting and other sources of bias (e.g. recruitment bias in cluster randomised studies). Each study was subsequently identified as being at ‘high risk’, ‘low risk’ or ‘unclear risk’ of bias in each domain. In cases of disagreement, the reviewers met to seek consensus and, when necessary, referred judgement to a third reviewer. The protocol did not originally specify a quality assessment tool for non-random studies. An amendment to the protocol (see Appendix 7) stipulated that such studies would be assessed using the Risk Of Bias In Non-randomised Studies – of Interventions (ROBINS-I) tool,81 but no non-random evaluations were eligible for inclusion in the review.

Assessment of economic evaluations

We initially planned to assess the quality of economic evaluations using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist,82 but amended the protocol (see Appendix 7) to do so using a more appropriate tool: an adapted version of the Drummond et al. 83 reporting guideline. It requires the assessor to answer 24 questions regarding each study, ranging from the type of economic evaluation (e.g. cost–utility analysis) to the time horizon and rationale for the choice of modelling approach. Although the questionnaire is detailed, we expanded a number of its questions to ensure that information that was particularly relevant to this review was extracted, such as identifying uptake rates and assumptions regarding the heterogeneity of risk (see Appendix 6). Two reviewers each conducted this assessment independently, then met to compare their assessments, resolving any differences through discussion and, when necessary, by calling on a third reviewer.

Typology of intervention approaches

Intervention descriptions and theories of change were first analysed to develop a typology of interventions. We intended to incorporate whether or not interventions were focused solely on the prevention of alcohol or drug use; HIV, STIs and sexual risk behaviour (sometimes referred to as ‘sexual health outcomes’ in this report, for brevity); or mental illness; and whether or not they had other aims such as access to HIV testing or adherence to HIV treatment. As planned, we identified interventions’ targeted outcomes. However, we ultimately developed a typology of interventions based on authors’ narrative descriptions of intervention methods because the strongest similarities and differences between interventions emerged in relation to the approaches used, and interventions addressing similar outcomes often took different approaches to doing so.

Research question 1: synthesis of intervention theories of change

Using thematic synthesis methods,84–86 we undertook a synthesis of author narratives and depictions describing theories of how interventions were intended to generate outcomes. The aim was to develop an overarching theory of change for e-health interventions, or multiple such overarching theories if this was more appropriate. The synthesis was not restricted to studies judged to be of high quality. Instead, conclusions drawing on poorer-quality reports were given less interpretive weight. We focused on prospective theories of change rather than retrospective reflections on how interventions might have worked in the light of study findings. Author narratives describing rationales for aspects and characteristics of the intervention in which the author did not link these to theorised mechanisms of change were also excluded. Theory of change synthesis commonly uses a meta-ethnographic approach, originally developed to synthesise findings across multiple qualitative studies. 87 As originally applied to qualitative research, meta-ethnographic methods draw on primary constructs (verbatim qualitative data presented in reports of primary research) and secondary constructs (author interpretations of data presented in primary research) to develop tertiary constructs (reviewer interpretations presented in syntheses). Applied to theory synthesis, such methods draw solely on primary constructs (author descriptions of theories of change).

Informed by our typology of included interventions, two reviewers undertook a pilot analysis of the theory of change extracted from each of the two highest-quality studies of the same intervention type. We initially planned to undertake line-by-line coding of theory reports in order to identify recurring narrative themes across theories of change, as we have done in previous theory syntheses. 77,78 Each of two reviewers independently applied line-by-line codes, beginning with in vivo codes that closely reflected the words used in the theory reports. They then grouped and organised codes, applying axial codes reflecting higher-order themes, and met to compare and contrast the resulting coding. Because this narrative-based approach did not readily capture the well-described and complex inter-relationships between theoretical constructs present in the reports (often in the form of a diagram), we instead decided to develop a novel diagrammatic approach to theory synthesis. This methodological innovation allowed us to summarise the components of each intervention’s theory of change and the explicitly and/or implied causal relationships between them, drawing on text and diagrams present in included reports. Summarising these diagrammatically also facilitated the comparison and synthesis of these components and relationships across the included theories of change.

Like the approach we had initially planned, this novel method of theory synthesis was a form of qualitative synthesis, but one that aimed to describe theories of change primarily in terms of constructs, interconnections and interactions, rather than as narrative themes. Like conventional thematic analysis, it involved an initial stage of ‘in vivo’ coding of author descriptions to identify theories of change for each intervention (but expressed diagrammatically rather than as a set of narrative themes), followed by a stage of ‘axial’ coding to explore interconnections between in vivo coding, identifying similarities and differences across interventions to develop overarching theories of change (again expressed diagrammatically). This represents a deviation from protocol, albeit one that added, rather than detracted from, the rigour of the analysis; this protocol deviation is reported in Appendix 4.

We read and re-read textual descriptions of theories of change and (when available) diagrammatic logic models of theories of change contained in the data extraction forms relating to each study. Two reviewers then independently drew an initial diagram of the theory of change underpinning each intervention. When more than one report addressed the same intervention, reviewers used theory of change descriptions from all relevant reports to inform the diagram. The two reviewers then met to compare their diagrams for each intervention theory of change and reconciled discrepancies through discussion. Drawing on the strengths of each diagram, they developed a single diagram of each intervention’s theory of change, which included intervention components, mediators and moderators (when described by authors) and intended outcomes. When author descriptions implied, but did not explicitly state, inter-relationships between components of the theory of change, reviewers made inferences and noted when the diagrams were, in part, based on such inferences.

The two reviewers grouped these theory of change diagrams by (1) intervention type and (2) targeted health outcomes (sexual health, mental health and substance use) to explore the scope for developing overarching theories of change within and across these categories. Finding that the theory of change approaches underpinning the interventions were not patterned by intervention type or targeted outcomes, we took an inductive approach, grouping diagrams of theories of change that shared important constructs and pathways. Then, using reciprocal translation (to identify and describe similar concepts occurring across theories of change underpinning different interventions), refutational synthesis (to identify contradictory or opposing concepts occurring across theories of change) and line of argument (to synthesise distinct elements occurring across theories of change that form part of a broader whole) approaches from meta-ethnography,88 each reviewer independently analysed the diagrams within each grouping. We did this by systematically examining the constructs and the relationships between constructs presented in each intervention-specific diagram and by examining whether these recurred, appeared only once or conflicted with those depicted in other intervention-specific diagrams in the grouping. Based on their analyses, each reviewer then independently drafted one synthesised diagram for each grouping of similar intervention theories of change.

We documented each stage of this process, noting when theory of change components or relationships between these components differed between individual diagrams in the grouping, the approach used to synthesise these components (i.e. reciprocal translation, refutational synthesis, line-of-argument synthesis, or the exclusion of a theory of change component) and the resulting decision for the synthesised theory of change diagram. The two reviewers then met to compare their synthesised diagrams of change for each grouping, reconciling discrepancies and drawing on the strengths of each to develop a single synthesised theory of change diagram for each theory of change grouping. To demonstrate this process, Appendix 8 presents the theory of change diagrams for each of two individual interventions in one grouping and the resulting diagram of the synthesised theory of change for that grouping. Each synthesised theory of change was given a descriptive title inductively drawing on the central approaches of the theories of change synthesised.

In this application of meta-ethnographic methods to the synthesis of theories of change, our first-order constructs were the theory of change information described in theory reports and represented in data extraction forms; our second-order constructs (analogous to in vivo codes) were the reviewers’ interpretations of these concepts, represented in the intervention-specific theory of change diagrams; and our third-order constructs (analogous to axial codes) were the higher-order interpretations, represented by the diagrams of the synthesised theories of change developed for each inductive grouping.

Thematic synthesis of process data

We undertook a synthesis of process data using thematic synthesis methods. 84–86 Syntheses of findings from qualitative and quantitative elements of process evaluations were used to understand characteristics of interventions, participants and contexts that acted as potential barriers to and facilitators of implementation and receipt, and which of these applied across or only within the domains of sexual health, substance use and mental health interventions.

Synthesis followed a meta-ethnographic approach. Second-order constructs (author narratives) were distinguished from first-order constructs (directly quoted qualitative data). In the case of findings from qualitative elements, we undertook line-by-line coding examining ‘first-order constructs’ (directly quoted qualitative data) and second-order constructs (author interpretations). In the case of findings from quantitative elements, we coded author interpretations, first checking as part of the quality assessment whether or not these aligned with the quantitative data presented (i.e. the extent to which study findings were grounded in the data). The synthesis drew these together through a thematic analysis, which developed third-order constructs by drawing connections between these data. This synthesis was not restricted to studies judged to be of high quality. Instead, conclusions drawing on poorer-quality reports were given less interpretive weight.

First, two reviewers prepared detailed results tables to describe the quality of each process evaluation report; details of the intervention examined (drawing on all included reports on the intervention, regardless of report type); study site and population; and study findings, including which aspects were explored in regard to how processes vary by intervention characteristics, providers, participants and/or context (see Appendix 9). Second, the two reviewers independently piloted coding of two high-quality process reports. They each read and re-read the study findings, applying line-by-line codes to capture the content of the data. Coding began with in vivo codes that closely reflected the words used in findings sections. The reviewers then grouped and organised codes, applying axial codes reflecting higher-order themes. They then met to compare and contrast their coding of these first two reports, developing an overall set of codes. Third, the two reviewers went on to code the remaining reports for each synthesis, drawing on the agreed set of codes, but developing new in vivo and axial codes as these arose from the analytical process. At the end of this process, the two reviewers met to compare their sets of codes. They identified commonalities, differences of emphasis and contradictions with the aim of developing an overall analysis that drew on the strengths of the two sets of codes and that resolved any contradictions or inconsistencies. We planned a priori to produce tables from our analysis demonstrating how first-, second- and third-order constructs related to one another, thereby enhancing transparency about emergent themes.

Synthesis of outcome data

We first produced a narrative account of the effectiveness of included interventions. Subsequently, we conducted a narrative synthesis by outcome, then, within this, by follow-up time and intervention type. Outcomes were categorised into sexual health outcomes, alcohol use, drug use, anxiety and depression. Two reviewers prepared detailed results tables describing intervention characteristics (drawing on all included reports on an intervention), study characteristics and study findings (see Appendix 10). We produced forest plots for each of our review outcomes, with separate plots for different outcomes and follow-up times, and pairwise comparisons between intervention types (e.g. intervention vs. no treatment control, or vs. another treatment type). Plots included point estimates and standard errors for each study, expressed as standardised mean differences (Cohen’s d) to ensure comparability across reports.

When data allowed, we calculated pooled effect sizes within each pairwise comparison (e.g. intervention type vs. control), accounting for the extent of heterogeneity among the studies (as determined both by a Cochran’s Q test and inspection of I²). 89 The results of statistical tests were evaluated in accordance with the Cochrane handbook. 63 If an indication of substantial heterogeneity was determined with fewer than three studies (e.g. study-level I² value of > 50%), we did not present a pooled estimate. When we had high levels of unexplained statistical heterogeneity (e.g. study-level I² value of > 50%) in any of our study groupings and a sufficient number of studies, we aimed to investigate this further using subgroup and sensitivity analyses. 90 As is appropriate for complex interventions, we used the random-effects model in meta-analyses; we did not undertake fixed-effects analyses because these would not have been interpretable given the effect estimates and statistical procedures used.

Where we produced pooled estimates, we used a robust variance estimation meta-analysis model to synthesise effect sizes. 91 This is because outcome evaluations are likely to include multiple measures of conceptually related outcomes and robust variance estimation meta-analysis improves on previous strategies for dealing with multiple relevant effect sizes per study, such as meta-analysing within studies or choosing one effect size, by including all relevant effect sizes but adjusting for interdependencies within studies. 92 Unlike multivariate meta-analysis, it does not require the variance–covariance matrix of included effect sizes to be known. We estimated separate models for each outcome: HIV, STIs, defined sexual risk behaviours, alcohol use, drug use, anxiety and depression. We regarded follow-up times of < 3 months (short term), 3 months to 1 year (mid-term) and > 1 year (long term) post intervention as different outcomes, pooling first by follow-up times and, when appropriate, overall across follow-up times. We ran these models for interventions overall and, when sufficient studies were found, we ran separate models for different intervention types. Categorisation by intervention type was into ‘clinically meaningful units,’ or subgroups of interventions that are useful and meaningful from a clinical and practice perspective, and were informed by the typology derived from our prior synthesis of intervention descriptions and synthesis of theories. 93 When meta-analyses were performed, we included pooled effect sizes in forest plots, with the individual study point estimates weighted by a function of their precision.

In addition, we aimed to consider all outcome evaluation evidence jointly in a network meta-analysis (NMA). A NMA integrates all pairwise comparisons for a specific outcome in the same model, to allow for the comparison of intervention types that may not have been directly compared head-to-head in empirical studies, and to combine direct, empirical head-to-head evidence with indirect estimates. However, we were unable to undertake these models with the body of evidence included because pairwise comparisons did not form an interconnected network with similar populations and outcomes. All included studies allocated individuals rather than clusters, so there was no need to account for clustering in any analyses.

We used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach, as described in the Cochrane Handbook for Systematic Reviews of Interventions, to present the quality of the evidence and the tables summarising the findings. 63 GRADE is a tool for researchers to describe how close the effect estimated in a meta-analysis is to the ‘true’ effect of the intervention. The downgrading of the quality of a body of evidence for a specific outcome was based on five factors: limitations of study, indirectness of evidence, inconsistency of results, precision of results and publication bias. The GRADE approach specifies four levels of quality (high, moderate, low and very low). If we had found a sufficient number of studies, we would have drawn funnel plots to assess the presence of possible publication bias (trial effect vs. standard error). Although funnel plot asymmetry may indicate publication bias, this can be misleading with a small number of studies.

We planned to undertake a sensitivity analysis to explore whether or not the findings of the review were robust in the light of the decisions made during the review process. This would have included assessing the impact of risk of bias in the included studies via restricting analyses to studies deemed to be at low risk of selection bias, performance bias and attrition bias. However, we did not undertake these analyses because too few studies would have been included in each meta-analysis to permit meaningful interpretation. When data allowed, we planned to undertake additional exploratory meta-analyses to determine intervention effects on theorised intermediate outcomes (such as goal-setting or self-efficacy) to examine the plausibility that these might mediate or otherwise precede behavioural effects, as well as to explore whether or not intervention effects on some of our outcomes (e.g. drug use) appear to mediate effects on other outcomes (e.g. risk of HIV infection). However, the reported intermediate outcomes were diverse between studies, poorly reported so as to preclude estimation of effect sizes and unlinked to estimates of outcomes, thereby precluding a meaningful synthesis of these outcomes.

If sufficient studies had reported appropriate information, we would have examined intervention effects by participant subgroups in terms of participant sexuality and gender identity, SES and ethnicity to explore potential impacts on health inequalities. This would have drawn on existing methods involving an ‘equity lens’ to examine evidence that equity-related characteristics (individual sexuality or gender identity, SES or ethnicity) moderate intervention effects, both in terms of evidence within studies and, if enough evidence had existed, between studies of significant effect modification.

Synthesis of economic data

Measures of costs, indirect resource use and cost-effectiveness were summarised in a table. If measures of resource use were judged to be sufficiently homogeneous across studies, we planned to synthesise these using statistical meta-analysis, but this was precluded by the sparse evidence available. 70 Measures of costs, indirect resource use and cost-effectiveness were adjusted for currency and inflation to the current UK context. These data were used to inform a narrative synthesis of economic evidence and applicability to the UK context (RQ4).

Interpreting overall findings

In Chapter 10, we draw together what we have learnt overall from the syntheses of theories of change, facilitators of and barriers to implementation, and outcomes of e-health interventions addressing sexual health, substance use and mental health among MSM. This discussion focuses on implications for intervention research. We assessed whether or not interventions addressing the various outcomes, or some subsets of these, appeared to have similar, or at least compatible, theories of change (e.g. similar mediating factors or mechanisms) that could pragmatically be combined to develop an overall intervention with a coherent theory of change addressing the syndemic or some sub-elements of the syndemic. We then assessed whether different or similar factors appeared to facilitate or impede implementation or receipt of interventions addressing the various outcomes in order to further develop our sense of whether these outcomes might be feasibly addressed by a single intervention or might be better addressed by separate interventions addressing some, but not all, outcomes. Finally, we draw on the findings from our outcome syntheses to judge whether or not there is scope for a single intervention addressing sexual health, substance use and mental health to have synergistic effects. We established a priori that we would judge this to be likely if there was evidence that e-health interventions addressing the various outcomes all tended to be effective, particularly when there was evidence that interventions addressing different outcomes appear to positively affect similar mediators. We determined a priori that when this was found not to be the case, for example because of evidence of a lack of, or even harmful, effect on some outcomes, or a lack of evidence for some outcomes, our recommendations would reflect this.

Years of report publication

Reports were published between 2006 and 2020, with more than two-thirds (26/37)96,97,103,104,106–108,110–115,117–119,123–132 published from 2015 onwards (Figure 2).

FIGURE 2.

Number of included reports, by year of publication.

Targeted health outcomes

In total, 20 interventions addressed sexual health outcomes,97–106,109–125,127–132 10 addressed substance use97,103,104,106,109,110,112–115,119,122,123,125–127,129,130,132 and seven addressed mental health. 96,97,104,106–108,119,123,126,127 Thirteen addressed a single health outcome of interest for this review,96,98–102,105,107,108,111,116–118,120,121,124,128,131 six addressed two of these outcomes103,109,110,112–115,122,125,126,129,130,132 (with five addressing sexual health and substance use,103,109,110,112–115,122,125,129,130,132 and one addressing mental health and substance use126) and four addressed all three outcomes. 97,104,106,119,123,127

Intervention development

Most interventions were informed by formative research, consultations and/or pilot testing with participants drawn from the target population (see Table 3). For example, gay community representatives provided the scenarios presented in the China–Gate HIV Prevention Program Online Intervention for MSM,124 and development of myDEx was informed by input from a sociodemographically diverse youth advisory board of three young MSM. 97 Gay Cruise developers used intervention mapping, a systematic method for developing interventions informed by evidence, theory and stakeholder input, and piloted materials with 15 professionals and 15 members of the target population. 105 The HealthMindr mobile app was informed by separate focus groups with MSM, HIV testing counsellors and key informants, then tested in focus group discussions. 118 Several interventions were informed by or adapted from other existing interventions. For example, the online mindfulness-based cognitive therapy intervention was modelled on an existing mindfulness-based cognitive therapy protocol and drew much of its content from the materials and trainers’ manual from that programme,96 and Rainbow SPARX was an adaptation for lesbian, gay, bisexual and transgender (LGBT) youth of the computerised cognitive–behavioural therapy (CBT) programme SPARX. 108

Intervention timing and duration

Interventions varied in the timing, intensity and duration of delivery (see Table 3). For example, TXT-Auto involved text message delivery five times per day over an 8-week period,114,115,132 whereas Gay Cruise involved downloading a game that users could play at their own pace over an indefinite period of time. 105 Three interventions were delivered via smartphone/mobile apps,118,119,125,128 which participants were to use over periods ranging from 6 weeks118 to 1 year. 128 The content of several interventions was delivered weekly over a defined period of time,96,104,106,124,127 ranging from 3 weeks (for part II of the China–Gate HIV Prevention Program Online Intervention)124 to 1 year [for the Mobile Technology and Incentives (MOTIVES) intervention]. 106 When authors reported on the expected or actual amount of time taken for users to work through defined programme content, this ranged from 10 minutes for the tailored version of the Cognitive Vaccine Approach to 3.5 hours for Rainbow SPARX. 107,108

Target populations

All included studies met our inclusion criteria stipulating that MSM account for 50% or more of the sample. Of the 23 included interventions, three targeted LGBT or sexual minority youth (aged 13–19,107,108 14–18126 and 16–20 years111) or young men (aged 13–18 years);129 four targeted young adult MSM aged 18–2497,101,103,112,113,122,123,130 or 18–39 years;100 one targeted HIV-negative young adult gay, bisexual and queer men (aged 18–29 years);131 three targeted MSM104,109,110,127 or people more generally119 living with HIV; two targeted single gay men;102 four targeted MSM/same-sex attracted men generally;96,118,124,125,128 two targeted men who use the internet to seek sex with men;105,116,120 one targeted sexually active, internet-using MSM,117 with an earlier iteration targeting this population in rural areas;98,121 one targeted Latino/a MSM and transgender women;106 and one targeted out-of-treatment methamphetamine-using MSM. 114,115,132

Geographical location of studies and implementation

Three-quarters (21/28) of included studies took place in the USA. 96–101,103,104,106,109–116,118–123,126–130,132 Two studies (7%) took place in the Netherlands,102,105 and one (4%) took place in each of the following: China,124 New Zealand,107,108 Singapore,131 Sweden117 and Taiwan (Province of China)125 (Figure 3).

FIGURE 3.

Spread of studies by country.

Intervention mode of delivery and interactivity

The 23 included interventions were delivered by a variety of electronic methods (Figure 4). Most were delivered via the internet (n = 15, or 65%),96–100,102–105,109–113,116,117,120–124,127,129–131 with others split across the following platforms: smartphone/mobile app (n = 3, or 13%),118,119,125,128 text messaging (n = 2, or 9%),106,114,115,132 computer download (n = 2, or 9%)101,126 and CD-ROM (n = 1, or 4%). 107,108

FIGURE 4.

Electronic mode of delivery, by intervention.

The vast majority of interventions (20/23, or 87%) were interactive,96–101,103,105–108,111–113,116–123,128,130 a further three (13%) used participant assessments to inform the delivery of non-interactive content102,109,110,114,115,132 and the remaining three (13%) were not interactive. 102,104,127,131

Time-limited or modular interventions

Seventeen interventions were designed as time-limited or modular interventions, guiding participants sequentially through intervention content from beginning to end. 96–105,107,108,111–113,116,117,120–124,126,127,129–131 Of these, 14 addressed sexual health,97–105,111–113,116,117,120–124,127,129–131 five addressed mental health96,97,104,107,108,123,126,127 and five addressed substance use. 97,103,104,112,113,122,123,126,127,129,130 These interventions could be subdivided into interactive96–101,103,105,107,108,111–113,116,117,120–124,126,130 and non-interactive interventions. 102,104,127,131

Open-ended interventions

In contrast to the time-limited/modular interventions, six interventions appeared, according to author descriptions, to be open-ended. 106,109,110,114,115,118,119,125,128,132 Rather than implying an infinite amount of content, this description merely indicates that these interventions were not designed as fixed and sequenced bodies of learning that all participants were intended to work through. All of these open-ended interventions addressed sexual health: one did so alone;118,128 three addressed sexual health and substance use;109,110,114,115,125,132 and two addressed sexual health, mental health and substance use. 106,119

Two open-ended interventions were delivered via SMS. 106,114,115,132 Although their content was limited (i.e. there was a finite set of SMS messages that could be delivered), we categorised these as open-ended because author descriptions suggested that they were not designed to guide participants from beginning to end through a sequential intervention. In TXT-Auto,114,115,132 users received five messages daily for 8 weeks and a weekly text-based assessment. The messages each user received were tailored to their risk profile at baseline. In the MOTIVES intervention, participants received a weekly SMS with HIV prevention information, followed 2 days later by a SMS asking a question about this information, as well as a HIV testing reminder message every 2.5 months. 106

Among the open-ended interventions, two determined which core content to deliver based on a user assessment109,110,114,115,132 and four did not use this approach. 106,118,119,125,132

‘Cognitive/skills’ synthesised theory of change

We drew on 13 intervention theories of change, which varied in quality from low to high, to develop an initial diagram for the ‘cognitive/skills’ grouping (Figure 5). 97–100,102–105,111–113,117,118,121–125,127–131 These theories of change underpinned 14 interventions: China–Gate HIV Prevention Program Online Intervention,124 Cognitive Vaccine Approach [(1) tailored and (2) non-tailored versions],102 Gay Cruise,105 Keep it Up!,103,112,113,122 HealthMindr,118 Hot and Safe M4M,100 myDEx,97 MyPEEPS Mobile,129 People Like Us,131 Queer Sex Ed,111 Safe Behaviour and Screening,125 Sex Positive!104 and WRAPP. 98,99,117,121

FIGURE 5.

‘Cognitive/skills’ synthesised theory of change. MH, mental health; SH, sexual health; SU, substance use.

Ten of the 13 intervention theories of change in this grouping referenced social cognitive theory98,99,104,105,117,118,121,127,128 and/or the IMB model,97–100,102,103,111–113,117,121–123,125,130 often in combination with other scientific theories. Three referenced the theory of planned behaviour102,124,129 and one referenced no existing scientific theories, but shared key constructs with other intervention theories of change in this grouping. 131 Although other interventions also drew on these scientific theories, the intervention theories of change in this grouping shared, as core components, constructs that are key to these three scientific theories. All interventions represented in this grouping targeted sexual health outcomes, either alone or in combination with substance use or both substance use and mental health. Of the 12 time-limited interventions in this grouping, guiding participants sequentially through intervention content from beginning to end, seven were online modular interventions,97–100,103,105,111–113,121–124,129,130 four were non-interactive interventions102,104,127,131 and one was a computer game. 105 The remaining two interventions in this grouping were open-ended and of the ‘general content’ subtype (i.e. with content not organised by an assessment). 118,125,128

The MOTIVES intervention106 and the internet-based safer sex intervention110 theories of change, assessed as being of medium and low quality, respectively, were primarily rooted in different approaches or scientific theories and were therefore not included in the ‘cognitive/skills’ grouping. However, these theories of change included components both overlapping with and complementing those in the ‘cognitive/skills’ grouping. Using the line-of-argument approach, we therefore drew on the theories of change underpinning MOTIVES and the internet-based safer sex intervention to augment findings within this grouping, and these additions (theorised moderators and the provision of information based on assessed stage of change) appear in the synthesised theory of change diagram.

Interventions in this grouping provided information and exercises and incorporated techniques to model, or demonstrate, desired behaviours. Included in the exercises are decisional balance exercises, which weigh the pros and cons of a particular behaviour; this recurred as an intervention activity in theory reports.

These activities were theorised to affect mediators including knowledge, outcome expectancies (i.e. positive and negative expectations of a particular behaviour),134 attitudes (including internalised homophobia),111,131 perceived norms, perceived vulnerability/risk (a construct combining, via reciprocal translation, perceived vulnerability102,103,111–113,122,130 and risk perceptions105,124,131) and behavioural skills. In addition, two interventions whose theories of change informed this grouping aimed to affect connectedness to the LGBT community,111,131 portrayed as a mediator in the synthesised diagram, although the reports were not explicit about how intervention activities aimed to engender this connectedness or how it might be related to sexual health outcomes.

Individual theories of change differed as to what specific activities led to the development of behavioural skills. We inferred that exercises would develop behavioural skills, indicating this as an inference by use of a dotted line in Figure 5, although behavioural skills could be developed through other activities. For example, in the video-based Sex Positive! intervention, characters modelled behaviours and this was theorised to increase self-regulatory skills. 104 Theory reports did not describe clear and recurring pathways from modelling to mediators, but we inferred (denoted by use of a dotted line) from reciprocal translation of recurring descriptions that these aimed to promote self-efficacy. 104,105

Although information provision was consistently linked to knowledge, theory reports provided no consensus on the activities that were intended to modify outcome expectancies, attitudes, perceived norms and perceived vulnerability. These mediators are shown in Figure 5 by the use of three hollow block arrows, indicating that they stem from intervention activities generally. Knowledge and behavioural skills were described as affecting behaviour either directly or via self-efficacy, so both pathways are shown in the synthesised model.

Outcome expectancies tended to be linked to motivation in theory of change descriptions, although the theorised relationship of motivation to other constructs varied. Outcome expectancies were theorised as a component of motivation in the theory of change for the WRAPP intervention99 and, although the report for the Sex Positive! intervention does not explicitly link these two constructs, the measures used to assess outcome expectancies in that report reflect users’ motivation [e.g. ‘I am more likely to have anal insertive sex (top) without a condom while drinking or high’] and intentions (e.g. ‘I am more likely to use a condom with men who are HIV-negative or of unknown status’). 104 The theory of change for the HealthMindr app highlighted outcome expectancies (referred to as ‘outcome expectations’)118 as a key mediator without discussing motivation. Given these differences, we drew on line-of-argument synthesis to include outcome expectancies as a mediator and inferred that it influences motivation in the synthesised diagram, indicating this as an inference by use of a dotted line in Figure 5.

The concept of self-regulation appeared variously as a mediator underpinning different theories of change. It is not included as a distinct construct in the synthesised diagram because its theorised role was not consistent. Self-regulatory skills were specified in the Sex Positive! intervention’s theory of change104 and are subsumed under behavioural skills in the synthesised diagram. Self-regulation was a key mechanism of change for HealthMindr,118 but did not recur in this way in other intervention theories within the theory grouping.

Motivation and self-efficacy both featured prominently in intervention theories of change within this grouping, often on the pathway between distal mediators and behaviour change. Although some reports suggested that motivation and self-efficacy influence each other in one direction or the other,99,100 most did not address their inter-relationship, either portraying the constructs as independent111,112 or including only one. 102,104,105,118 The synthesised theory of change diagram, therefore, portrays both constructs as affecting behaviour independently via a line-of-argument synthesis.

Although some intervention theories of change included either motivation124,131 or intentions,98–100,117,121 those that included both variably portrayed their relationship as motivation influencing intentions,97,123 intentions influencing motivation111 or both combined into one construct. 102,125 Given these differences, we have combined these constructs via reciprocal translation in the synthesised diagram, in which the combined construct of motivation/intentions is theorised to be engendered by exercises and influenced by outcome expectancies, attitudes, perceived norms and perceived vulnerability.

We interpreted the construct of perceived behavioural control97,102 to be similar to the concept of self-efficacy because the latter was variously described as including an ‘ability to refuse to have anal sex if a condom was unavailable’,99 ‘confidence in practicing safer sex behaviours’122 and/or the ‘ability to avoid the situational temptation to have unprotected sex’. 122 Self-efficacy and perceived control were therefore merged via reciprocal translation in the synthesised theory of change diagram.

Theorised moderators of the relationship between intervention activities and outcomes are shown at the top right of Figure 5, and include participants’ acculturation to the local majority culture; sociodemographic characteristics; psychological risk correlates, including mental health and substance use/abuse; and, for sexual health interventions, sexual partner type (casual encounter, romantic interest, or friends with benefits). In a few cases, theory reports discussed the direction of moderation and/or how these might operate. Bauermeister et al. 97 theorised that MSM’s ability to enact sexual risk reduction behaviours might vary by partner type and that MSM experiencing stressors related to being a sexual minority, experiencing psychological distress, or using alcohol or drugs might have less behavioural control, limiting the extent to which the intervention might promote their ability to ‘regulate their affective motivations’97 and enact sexual risk reduction behaviours. As noted previously, the theory of change underpinning the MOTIVES intervention was primarily rooted in a different approach and did not sit squarely within this grouping, but included components overlapping with and complementing those in this grouping. 106 It was therefore used to further develop this synthesised theory of change via line-of-argument synthesis. Linnemayr et al. 106 theorised that participants in the MOTIVES intervention who were more acculturated to the USA and could more easily communicate with health-care providers might face fewer barriers to HIV testing, thus benefiting less from the intervention.

Finally, some interventions in this grouping,100,122 and the two not included in the grouping but augmenting its synthesised theory of change,106,110 aimed to boost intervention impact, for example by recruiting participants at a critical biographical point for behaviour change (e.g. on receipt of a HIV-negative test result in the case of the Keep it Up! intervention),122 delivering content based on a participant’s assessed stage of change100,110 or using prize lotteries to incentivise knowledge retention. 106

‘Self-monitoring’ synthesised theory of change

We defined a grouping focused on ‘self-monitoring’ (Figure 6) by synthesising theories underpinning the smartphone self-monitoring intervention,119 an open-ended intervention with general content addressing all three health outcomes examined in our review among people living with HIV (with a theory report assessed as being of high quality), and the TXT-Auto intervention,115 an open-ended sexual health and substance use intervention for out-of-treatment men using methamphetamines, with content tailored by a user’s risk profile (with a theory of change description assessed as being of medium quality). The theory report for the smartphone self-monitoring intervention offered a more detailed outline of the self-monitoring behavioural change pathway;119 this was incorporated into the synthesised theory of change diagram.

FIGURE 6.

‘Self-monitoring’ synthesised theory of change.

Both interventions in this grouping asked the user questions about their behaviour to prompt self-monitoring, and both were theorised to be driven by self-monitoring as a key mechanism. Frequency of self-monitoring varied by study and outcome, ranging from four times per day119 to weekly. 115 The theory report for the smartphone self-monitoring intervention suggested the need for users to first establish criteria for their desired behaviour, such as personal norms or standards, against which they could assess their actual behaviour. 119 However, there was no evidence that this was a separate stage of the intervention; therefore, this stage does not appear in the synthesised diagram.

The synthesised theory of change posits that behavioural questions result in self-monitoring, which prompts reflection in terms of pre-established criteria. This reflection is theorised to result in either self-reward or self-critique, generating enhanced self-regulation, a theoretical mediator of behaviour change.

Although both theory reports included in this grouping suggested that processes of change are more complex than depicted in this synthesised theory of change, neither provided further details on other mechanisms. The theory of change underpinning TXT-Auto suggested that the intervention worked via two non-intersecting pathways, one captured in the ‘cognitive/skills’ synthesised theory of change and one featuring self-monitoring. Arguing that pathways from self-monitoring to behaviour change are underdeveloped, Swendeman et al. ’s119 research drew on qualitative data from their study to further explicate the theory of change for the smartphone self-monitoring intervention. This was not included in our synthesis, which focused on a priori intervention theories of change, rather than on theories refined through empirical research.

‘Cognitive therapy’ synthesised theory of change

We drew on theories of change underpinning the online mindfulness-based cognitive therapy intervention,96 an online modular intervention for same-sex attracted men, and Rainbow SPARX,108 a computer game intervention for sexual minority youth, to develop a ‘cognitive therapy’ theory of change grouping. Descriptions of both interventions’ theories of change were assessed as being of medium quality. These interventions were the only two in this review that targeted only mental health outcomes; both interventions drew on cognitive therapy techniques and aimed to reduce depression and improve mental health. CBT skills were at the core of Rainbow SPARX, while the online mindfulness-based cognitive therapy intervention combined mindfulness with cognitive–behavioural techniques. Both interventions aimed to improve emotional health via relaxation training and by addressing negative cognitions (including internalised homophobia), utilising differing pathways for the latter.

The synthesised theory of change diagram (Figure 7) depicts activities and mediators stemming from these two approaches. Activities include CBT and mindfulness-based cognitive therapy. Although we do not detail all activities comprising each of these approaches in the synthesised theory of change diagram, we include key components of each: behavioural training (an element of CBT), an emphasis on prioritising ‘being’ over ‘doing’ (an element of mindfulness-based cognitive therapy)96 and relaxation training (an element of both approaches).

FIGURE 7.

‘Cognitive therapy’ synthesised theory of change.

Theory of change descriptions for both interventions referred to recognising, paying attention to or developing awareness of thoughts, feelings and situations, with Rainbow SPARX107 then challenging negative cognitions and the online mindfulness-based cognitive therapy intervention focusing on accepting and ‘letting go’ of these negative cognitions. 96 We use differently shaded block arrows in Figure 7 to delineate these distinct pathways. Among the negative cognitions to be addressed, both interventions sought to reduce internalised homophobia and mitigate its effects on health. 96,107 The report for Rainbow SPARX suggested that this was addressed by providing information to help promote young people spending more time with those who accepted them and reducing exposure to homophobic bullying. 107 The report for the online mindfulness-based cognitive therapy intervention did not make clear how this would be achieved. 96 Descriptions of both interventions’ theories of change suggested that the reframing of distressing emotions was theorised as a key mechanism for improving mental health.

Intervention theories of change not included in synthesised models

Although informed by the sexual health model, the theory of change underpinning the online modular sexual health intervention Sexpulse (as described in theory reports assessed as being of low quality) did not specify components or mechanisms and so could not be synthesised. 116,120

Four other interventions, with theory reports ranging in quality from low to high, did not fit within any of the three inductive theory of change groupings. 101,106,110,126 These interventions cut across the targeted health outcomes considered in this review and represented three of the five categories of the intervention typology: computer games,101,126 ongoing interventions with content organised by assessment109,110 and ongoing interventions with general content. 106 Although some components of their theories of change overlapped with the three groupings described above, theories of change for these interventions featured elements that differentiated them from those included in these groupings. The MOTIVES106 intervention was based on behavioural economics and the SOLVE intervention was driven by the theorised role of emotions in decision-making, featuring ‘shame associated with sexual stigma’101 as a key mediator. Drawing on social cognitive theory, stress and coping theory and the social and emotional learning framework, the role-playing game (described in a theory report assessed as being of medium quality) aimed to improve mental health and reduce substance use via use of productive coping strategies with a focus on help-seeking. 126 Informed by the transtheoretical model, the theory report (assessed as being of low quality) for the internet-based safer sex intervention110 suggested that the intervention was primarily based on a stages-of-change approach, tailoring content based on responses to monthly sexual behaviour surveys. Although not included in the grouping, the MOTIVES intervention106 theory report (assessed as being of medium quality) and the internet-based safer sex intervention110 theory report informed the ‘cognitive/skills’ synthesised theory of change, as described above.

‘Cognitive/skills’ synthesised theory of change

The ‘cognitive/skills’ synthesised theory of change resonated most strongly with stakeholders and was thought to be the best reflection of how face-to-face interventions are currently conceptualised in the UK. Recruitment at a critical point in potential participants’ lives was felt to be important to the success of interventions. Ensuring that interventions met individuals’ needs at various stages of change was also recognised as important, and several stakeholders felt that including prizes would boost engagement.

The mediators and moderators were felt to be well considered and likely to be central to how these types of interventions could be designed and targeted, reflecting much current practice for face-to-face interventions.

‘Self-monitoring’ synthesised theory of change

At the time of the consultation, the draft ‘self-monitoring’ synthesised theory of change diagram included an additional activity component, ‘define criteria (perceived norms, personal standard)’, which was later removed from the final diagram because the intervention theory of change from which it derived did not explicitly include a criteria-setting activity. 119 This activity and its relationship to a theorised mediator were shown with dotted lines, denoting that they were inferred from, rather than explicitly described in, the author narrative. 119

The ‘self-monitoring’ synthesised theory of change did not align with stakeholders’ views on how best to conceptualise interventions for addressing syndemics among MSM. Stakeholders thought that approaches that relied on this theory of change risked encouraging participants to become focused on self-monitoring and comparing their behaviours with goals implicitly in line with public health and LGBT community injunctive norms more widely. Stakeholders felt that interventions would perform better if the motivation for behaviour change was improvement in well-being, rather than comparison with behavioural norms. Stakeholders were concerned that self-monitoring approaches could amplify existing stigmas surrounding behaviour, especially relating to drug use and sexual activity, with potentially negative impacts on the community as a whole. There was, however, acknowledgement that goal-setting was an important part of existing interventions and should be an integral part of any potential e-health interventions. However, it was felt that this should be a secondary activity or component, rather than the primary focus of an intervention.

‘Cognitive therapy’ synthesised theory of change

The ‘cognitive therapy’ synthesised theory of change was felt to be useful in interventions targeting mental health, but less useful for interventions addressing sexual health and substance use. Stakeholders valued the inclusion of CBT and mindfulness approaches among the intervention theories of change reviewed, which are likely to be recognised by the client populations they serve. There were concerns, however, that this model would not apply to sexual health and substance use, and so an intervention based on these theories of change might focus on mental health at the expense of these other health domains.

Intervention characteristics affecting intervention receipt

Nearly all process evaluations explored ways in which intervention characteristics affected the receipt of interventions, although the included reports tended to lack breadth of the areas explored and in-depth exploration of the findings that they did report. 96,103,107,108,111,118,119,121–123,130 However, several subthemes emerged in our analysis across studies.

Participant characteristics affecting intervention engagement and receipt

Evaluations of four online modular96,121,130 and open-ended (general content)118 interventions (two targeting sexual health only,118,121 one targeting mental health only96 and one targeting sexual health and substance use130) quantitatively explored the relationship between participant characteristics and intervention engagement.

A medium-reliability study of the HealthMindr sexual health mobile phone app found no differences in the time spent on the app by participant location (comparing different cities in the USA), age, race/ethnicity or knowledge of local HIV testing,118 while a high-reliability study of the Keep it Up! intervention targeting young ethnically and racially diverse MSM found that, among black users, those with graduate degrees spent more time on the intervention than those with high school or lower levels of education. 130 A study assessed as being of medium reliability found no significant variation in retention for the eight-session modular mental health online mindfulness-based cognitive therapy intervention by age, SES, ethnicity, internalised homonegativity or experience of homophobic bullying. 96 A study assessed as being of high reliability found no differences between participants completing one versus all three modules of the Hope Project (an extension of the WRAPP sexual health intervention, targeting rural MSM) by age, ethnicity, marital status, sexual orientation, education or student status, but did find higher completion rates among higher-earning participants. 121

Madkins et al. 130 conducted a high-reliability assessment of intervention receipt among different users of the Keep it Up! intervention, which was developed with the engagement of a diverse group of young MSM and designed for young MSM of all racial groups. 122 Researchers found several differences in the acceptability of the Keep it Up! intervention by race/ethnicity, education level, age and city in the USA. 130 Black, Latino and other non-white users reported higher acceptability in a range of domains than did white users, and Latino users rated content more highly than other non-white users. In the overall sample, users with high school-level education or lower rated the intervention more highly than those with a higher level of education. Exploring the interaction of race/ethnicity and education level, the study found that white users with higher levels of education reported lower acceptability, while no such differences were found among black, Latino or other non-white users. Older users and those in Atlanta tended to rate modules more highly than those in New York.

Exploring intervention receipt qualitatively, a study with high reliability found that, for Rainbow SPARX, a computer game intervention addressing mental health among sexual minority youth aged 13–19 years, some older users reported that some aspects were too easy and the programme ‘babied’ them. 107 Acknowledging the challenge of designing a programme appropriate for a range of young people, one participant, aged 19 years, said:107

[S]ome things were a little easy . . . Overall it wasn’t difficult to figure out what you needed to do. Those . . . puzzles were quite easy to do. I guess it would be hard to make them more difficult though because you would have to be careful that everyone could actually get it.

Qualitative research suggested that how users experience e-health interventions could also vary by individual circumstances beyond sociodemographic factors. For young people who were also receiving external support (typically in-person therapy), Lucassen et al. 108 found, in a high-reliability study, that Rainbow SPARX could play a unique role by complementing that support. In a medium-reliability study, Swendeman et al. 119 found that the smartphone self-monitoring intervention, addressing all three health outcomes, could either support users’ antiretroviral therapy maintenance or promote recognition of patterns of non-adherence, depending on a user’s current level of adherence.

Contextual factors affecting intervention engagement

Few studies explored how the context for using the intervention was associated with the experience of its use. Those that did focused on internet speed in high-reliability 200798 and 2010121 studies of two iterations of the WRAPP sexual health intervention, which targeted rural MSM in the USA. Bowen et al. 98 found that users with dial-up, compared with high-speed, internet connections were more likely to report taking too long to load programme graphics, while Williams et al. 121 found no differences among participants completing one versus all three modules by type of internet connection.

Context

E-health interventions were felt by stakeholders to be broadly acceptable among MSM across geographical contexts. This varied by region, with stakeholders in London and Cardiff identifying larger groups of MSM whom they thought could benefit than stakeholders in Scotland and the north of England. This was largely due to regional variations in perceived health service availability compared with the level of need, alongside preferences for service types. Stakeholders from London identified the region as one with a high level of needs and greater demand than could be met by existing health services, thus increasing the potential utility of e-health interventions. Services in Cardiff were similarly felt to be insufficiently developed to meet existing needs.

There were concerns that poor access to broadband/the internet were issues that would negatively affect uptake in Scotland and the north of England.

It was felt that these interventions had significant potential in the UK, given funding constraints in sexual health and increasing imperatives around self-management.

Usability/acceptability

In line with process evaluation findings, stakeholders felt that it was critical to the success of e-health interventions for MSM that interventions were well-designed with minimal technical problems. There was a strong feeling that, should even minimal problems arise, it would have a profound impact on intervention engagement and effectiveness. Significant concern was expressed around the level of funding required to develop a suitable platform to deliver these interventions.

Content, language and tone

Stakeholders recognised that, in line with the process evaluation findings, appropriate content was central to the success of e-health interventions. The tone of the interventions was felt to be important in engaging a diverse group of MSM; matching intervention language to colloquial language used by MSM was felt to be essential. Furthermore, in line with the process evaluation findings, a candid tone and sex positivity were valued. Involvement of potential users in setting the tone and direction of these interventions was highlighted as an important component of formative development.

In addition, all stakeholders stressed the need for e-health interventions to be available in a range of languages to meet the needs of a broad range of MSM from migrant communities. This was regarded as particularly important given the higher engagement in sex work of MSM from Latin American and eastern European countries, recognising significant crossover between issues pertaining to sex work, mental health, substance use and sexual health.

Privacy and intrusiveness

Significant concerns regarding privacy and intrusiveness did not emerge among stakeholders. With regard to evaluations, as opposed to interventions, there was some concern about the burden of data collection for participants and concerns that it could negatively affect acceptability of similar interventions in the UK. Stakeholders felt that the number of data required to conduct rigorous evaluation might lead to a disjointed experience for service users, and that some may have concerns about confidentiality. In addition, one participant raised as a concern the number of data that might be collected and not used.

Risk of bias of active intervention versus control arm trials

Effects on human immunodeficiency virus infections

Two studies presented estimates for HIV infection: Chiou et al. ,125 which presented short-term (< 3 months) estimates, and Mustanski et al. ,113 which presented mid-term (3 months–1 year) estimates (Table 5). Across studies, there was no consistent evidence of interventions reducing new HIV infections, although sparse events and short time scales may have precluded clearer evidence of an effect.

Meta-analysis

Effect sizes for HIV infection are presented in Figure 9 and drew from two studies, each contributing one effect size. 113,125 We undertook an overall analysis, rather than analyses by time, as only one effect size from each follow-up category was included. Included interventions did not have an overall impact on HIV infections, with an increase in incidence of HIV infections equivalent to 0.12 standard deviations, but with a wide estimated CI that included the point of no effect (95% CI –0.34 to 0.59). Heterogeneity was not meaningfully present in this meta-analysis (I² = 0%). Certainty in the assessment of the evidence ranged from very low to low because of the risk of bias (selective outcome reporting) and the imprecision of effect estimates (see Grading of Recommendations Assessment, Development and Evaluation analysis for detailed reporting on the level of certainty for outcomes reported).

FIGURE 9.

Effect sizes and meta-analysis for HIV infections in included studies. This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

Effects on sexually transmitted infections

Three studies presented estimates for STIs: Chiou et al. ,125 Milam et al. 110 and Mustanski et al. 113 (Table 6). Of these, Chiou et al. 125 and Milam et al. 110 presented short-term results, whereas Mustanski et al. 113 presented mid-term results. There was no evidence of short-term impacts on incident STIs, whereas there was some evidence, albeit from one study, of mid-term impacts on incident STIs.

Meta-analysis

Effect sizes for STIs are presented in Figure 10 for outcomes at < 3 months post intervention and in Figure 11 for outcomes between 3 months and 1 year post intervention. In both plots, negative effect sizes represent benefits. A meta-analysis of effect sizes with follow-up of < 3 months included two effect sizes from two studies110,125 and also suggested a non-significant increase in the number of STIs in the intervention group, compared with the control group (d = 0.17, 95% CI –0.18 to 0.52), with heterogeneity not meaningfully present in this meta-analysis (I² = 0%). 110,125 The certainty of evidence was very low as a result of risk of bias (details of randomisation, selective outcome reporting) and imprecision of effect estimates. The overall analysis across short- and medium-term follow-ups and intervention types drew on three studies110,113,125 contributing seven effect sizes, and suggested a small and non-significant increase in STIs in the intervention group, compared with the control group (d = 0.07, 95% CI –0.79 to 0.94) and low heterogeneity (I² = 16%). We did not meta-analyse effect sizes with follow-ups of 3 months to 1 year as only one study would have contributed to this. 113

FIGURE 10.

Effect sizes and meta-analysis for STIs in included studies, < 3 months post intervention. This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

FIGURE 11.

Effect sizes for STIs in included study, 3 months to 1 year post intervention. This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

Effects on sexual risk outcomes

A total of 11 studies presented estimates for sexual risk outcomes (Table 7). 100,102,110,113,115,116,122–125,127 Nine studies100,101,113,115,116,122,123,125,127 presented short-term results, whereas six studies102,113,115,116,124,127 presented mid-term results. One study117 intended to present short-term results relating to sexual risk outcomes, but did not estimate an effect because of an unexpectedly low sample size. The primary method of analysis in Reback et al. 115 was a longitudinal regression model, and we present these estimates as well. There was some indication of short-term impacts on sexual risk in included interventions, but no such effect was apparent in mid-term results.

Meta-analysis

Effect sizes for sexual risk outcomes are presented in Figure 12 for outcomes of < 3 months post intervention (short term) and in Figure 13 for outcomes between 3 months and 1 year post intervention (mid-term). In both plots, negative effect sizes represent benefits. A meta-analysis drawing on 32 effect sizes from eight studies100,113,115,116,122,123,125,127 with < 3 months’ follow-up found a suggestion of effectiveness, albeit not statistically significant (d = –0.14, 95% CI –0.30 to 0.03) with substantial heterogeneity (I² = 61%). The certainty of evidence for this meta-analysis was graded as very low because of risk of bias (details of randomisation, selective outcome reporting), inconsistency of studies, and publication bias arising from the non-inclusion of two studies. 101,117 A meta-analysis drawing on 22 effect sizes from six studies102,113,115,116,124,127 with 3 months to 1 year of follow-up suggested a significant impact on reducing sexual risk (d = –0.12, 95% CI –0.19 to –0.05), but with low heterogeneity (I² = 27%). The certainty of evidence for this meta-analysis was graded as low because of risk of bias (details of randomisation, selective outcome reporting).

FIGURE 12.

Effect sizes and meta-analysis for sexual risk outcomes in included studies, < 3 months post intervention. RVE, robust variance estimation. This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

FIGURE 13.

Effect sizes and meta-analysis for sexual risk outcomes in included studies, 3 months to 1 year post intervention. RVE, robust variance estimation. This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

We then pooled estimates regardless of follow-up time. Based on 54 effect sizes from 10 studies,100,102,113,115,116,122–125,127 interventions significantly reduced sexual risk compared with control groups (d = –0.15, 95% CI –0.26 to –0.05). This meta-analysis had substantial heterogeneity (I² = 56%). To explore this heterogeneity, we compared interactive interventions100,113,115,116,122–125 with non-interactive interventions. 102,127 A random-effects meta-regression did not suggest that this characteristic accounted for heterogeneity with non-interactive interventions not meaningfully worse than interactive interventions (β = 0.12, 95% CI –0.66 to 0.89).

Effects on alcohol and drug use

Two studies presented estimates for drug use: Chiou et al. 125 and Reback et al. 115 (Table 8), both focused on open-ended interventions. Neither quantified outcomes on alcohol use specifically. Although both studies presented short-term results, only Reback et al. 115 presented mid-term results. However, the primary method of analysis in that study was a longitudinal regression model; we present these estimates as well. Across studies, there was no consistent evidence of effectiveness of interventions in reducing drug use.

Meta-analysis

Effect sizes for alcohol and drug use are presented in Figure 14 for outcomes of < 3 months post intervention and in Figure 15 for outcomes between 3 months and 1 year post intervention. In both plots, negative effect sizes represent benefits. We do not present a pooled effect size for these estimates as meta-analyses included both substantial heterogeneity (I² = 95% both overall and only including effect sizes of up to 3 months’ follow-up) and too few studies to explore this heterogeneity.

FIGURE 14.

Effect sizes for alcohol and drug use in included studies, < 3 months post intervention. This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

FIGURE 15.

Effect sizes for drug use in included studies, 3 months to 1 year post intervention. This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

This figure has been adapted with permission from Melendez-Torres et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/.

Grading of Recommendations Assessment, Development and Evaluation analysis

Findings from a GRADE analysis of included outcomes are presented in Table 9. The evidence generated a score of low or very low confidence for most outcomes. This was primarily because of risk of bias and imprecision in the evidence.

Risk of bias of trials comparing active interventions

Narrative summary of trials comparing active interventions

Davidovich et al. 102 included a three-arm trial testing tailored and non-tailored versions of their time-limited non-interactive online modular interventions against a no-treatment control group. We calculated that participants receiving the tailored intervention (n = 128) were more likely than participants receiving the non-tailored intervention (n = 107) to report, at 6 months post baseline, the practice of negotiated safety (defined above), compared with CAI with other partners (OR 6.50, 95% CI 2.49 to 16.90), and the practice of condom use, compared with CAI (OR 2.98, 95% CI 1.74 to 5.12).

In their evaluation of time-limited interactive online modular interventions, Bowen et al. 99 tested the impact of ordering modules with content about HIV knowledge, content about risk in casual or new partnerships and content about contexts of risk on the proportion of anal intercourse partners with whom a condom was used every time. At post intervention, there was no statistical difference between modules on sexual risk, but specific group differences were not presented.

Typology of intervention approaches

We developed a typology of interventions that categorised those included in the review by the structure of their content delivery, and identified two overarching types of interventions: time-limited or modular, which guided participants sequentially through intervention content from beginning to end, and open-ended, which were not designed as fixed and sequenced bodies of learning that all participants were intended to work through. Within these we identified five core categories. The three time-limited/modular categories comprised computer games and online modular interventions, both interactive and non-interactive. The two open-ended categories comprised interventions with content organised by participant needs, based on assessment, and those with generic content delivered to all participants. There was no clear pattern of particular intervention types addressing one or other of our outcomes, except that both open-ended interventions with content organised by assessment addressed sexual health and substance use and no open-ended interventions addressed mental health.

In our stakeholder consultations, all participants preferred interventions that provided some degree of personalisation and tailoring to the specific needs of individuals, ideally based on a risk assessment and providing content most relevant to their profile and/or needs. Stakeholders felt that this was critical in maintaining engagement with a diverse group of MSM who could benefit from these approaches. Interventions that were interactive and provided feedback mechanisms were favoured for increasing engagement and a sense of personal connection. Computer game approaches were seen by most stakeholders as patronising and unsuitable, except if these were specifically targeting young MSM or were exceptionally well designed. There was significant concern that these interventions would alienate much of their target population.

Theory of change synthesis

For this review, we synthesised the theories of change underpinning e-health interventions targeting sexual health outcomes, substance use and mental ill health among MSM. We developed a novel approach to doing so by using diagrams, rather than narrative themes, to summarise theories of change within and across interventions.

Although we have previously conducted reviews synthesising intervention theories of change using line-by-line coding of descriptive text,77,78 we found that this approach did not readily capture the often precisely described and complex inter-relationships between theoretical constructs presented in the body of literature for this review. Theories of change included in our past reviews, which addressed the integration of health and academic education78 and positive youth development interventions,77 were relatively simple and were either not significantly informed by existing scientific theories78 or informed by theories that are not typically portrayed visually. 77 In contrast, theories of change underpinning the e-health interventions included in this review were more complex, more explicitly theorised and largely drew on existing scientific theories that are typically conceptualised in terms of diagrams indicating relationships between theoretical constructs, with constructs widely recognised and understood and therefore not always discussed at length. We found that, although thematic analysis is a good way of rendering explicit what is implicit, it is less appropriate when the literature itself is more explicit. We therefore developed a novel method of theory synthesis in which we created diagrammatic summative logic models of intervention theories of change. By inductively grouping these models according to their core constructs and using meta-ethnographic approaches, we identified three emergent theoretical approaches underpinning the included interventions and we created synthesised models of each approach. We have thus synthesised theories of change underlying interventions with similar approaches. These summarise and integrate existing theories of change rather than providing a novel overarching theory of change for such interventions.

Social cognitive theory and the IMB model featured most prominently in intervention theories of change, informing a small majority. Although reports cited a number of other existing scientific theories, each informed only a few interventions at most. In the ‘cognitive/skills’ synthesised theory of change, based on theory of change descriptions assessed as ranging from low to high quality, information and exercises were theorised to influence behavioural skills directly and via various cognitive factors relating to motivation/intention and self-efficacy/perceived control. All interventions represented in the ‘cognitive/skills’ grouping of theories of change targeted sexual health outcomes, either alone or in combination with substance use or both substance use and mental health. The synthesised theory of change also suggests that particular strategies can boost intervention impact, whereas other factors relating to the participant or (for sexual health interventions) to their type of partner (e.g. casual encounter, romantic interest or friend with benefits) moderate impact. In our consultations, this theory of change grouping resonated most strongly with stakeholders and was considered to be the best reflection of how face-to-face interventions with MSM are currently conceptualised in the UK.

Although represented in fewer interventions, two other distinct theory of change groupings emerged. The ‘self-monitoring’ grouping, based on theory of change descriptions assessed as being of low and high quality, focused more narrowly on the role of self-monitoring in triggering reflection, self-reward/critique and behavioural self-regulation. Interventions in the self-monitoring group of theories of change addressed all three domains of health outcomes for this review. Theory reports suggested that this synthesised diagram represents the basic components of more complex cognitive pathways119 and can sit alongside other distinct mechanisms of behaviour change in intervention theories of change. 115 Underpinning interventions focused solely on mental health and based on theory of change descriptions assessed as being of medium quality, the ‘cognitive therapy’ grouping is rooted in cognitive therapy techniques, which can be augmented by mindfulness. In this approach, activities promoted awareness and recognition of a participant’s thoughts, feelings and situations and, via either challenging or accepting negative cognitions, aimed to reframe negative emotions to improve mental health. In our stakeholder consultations, the ‘cognitive therapy’ theory of change was felt to be useful in interventions specifically targeting mental health, but less useful for interventions addressing sexual health and substance use.

Intervention theories of change included in this review were informed by scientific theories that have been associated with greater impacts in reviews of e-health interventions specifically, including the transtheoretical model45,142 and the theory of planned behaviour,142 although these approaches underpinned a minority of interventions. Several also featured modelling and self-monitoring, and behaviour change techniques,143 which some evidence suggests might be effective in e-health interventions. 142,144

As is common with e-health behavioural interventions,55,142 the theories of change underpinning interventions in this review tended to rely on individually focused psychological theories of behaviour or behaviour change. Although these are unlikely to address structural factors contributing to the syndemic among MSM such as marginalisation, homophobia and discrimination,22,145 the accessibility and anonymity of e-health interventions offer one approach to reducing barriers to service access stemming from stigma and discrimination146 and might form an important element of a broader mix of interventions addressing individual and structural factors.

Furthermore, we found that included interventions drawing on multiple theoretical approaches96,97,101,108,111,131 took into account sexual minority-related stressors. One accounted for such stressors as a theorised moderator of intervention impact,97 whereas others aimed to increase connectedness to the LGBT community111,131 and reduce internalised homophobia96,108,111,131 and associated shame101 to reduce sexual risk101,111,131 and improve mental health. 96,108 However, theory reports included little discussion of the relationships between these constructs, limiting our ability to explicate their roles in the synthesised theories of change.

It was notable that some existing scientific theories that informed theories of change were theories of behaviour change (e.g. CBT theory) whereas others were theories of behaviour and its determinants (e.g. the health belief model). Authors generally did not report drawing on scientific theories oriented more explicitly towards selecting strategies for enacting behaviour change, such as the behaviour change wheel147 or the elaboration likelihood model. 148 Intervention developers might usefully draw on such models.

Our theory synthesis aimed to develop overarching theories of change for different subtypes of e-health interventions to address sexual health outcomes, substance use and mental ill health among MSM, which we hope will help inform future intervention studies. Intervention developers might also draw on existing scientific theories that aim to integrate existing scientific theories of behaviour and behaviour change, such as plans–responses–impulses–motives–evaluation (PRIME) and capability, opportunity, motivation and behaviour models. 147,149 Because we found that similar theories of change could underpin interventions addressing the different outcomes constituting the ‘syndemic’ affecting MSM, this provides some confidence that interventions can be theorised and developed that aim to address these interventions in combination.

Intervention, participant and contextual characteristics affecting intervention receipt

One-third of reports included in the overall review reported process evaluation data. All but one process evaluation took place in the USA. Most interventions targeted a single health domain of interest for this review (sexual health, substance use or mental health), with the majority focused on sexual health. However, two aimed to address aspects of all three. 119,123 Some interventions employed personal tailoring, an approach that has been associated with effective e-health behaviour change interventions. 45,144

Process evaluations rarely explored how intervention receipt varied between contexts. We found no eligible reports examining what factors affected intervention delivery as opposed to receipt. This seems to reflect the emerging state of process evaluations in e-health literature, with other reviews of e-health interventions reporting a similar pattern. 150–153 There was some suggestion that slower internet speed could reduce acceptability of a multimedia intervention among rural MSM in the USA, who are less likely than non-rural residents to have high-speed internet at home. 154 We did not find evidence that the factors that process evaluations identified as affecting acceptability varied by intervention type or by outcomes addressed.

In terms of intervention characteristics, as with e-health interventions among general populations,153 participants appreciated when interventions were easy to use and free of technical problems, while incompatibility with mobile platforms detracted from acceptability and could impede participation. In line with these findings, stakeholder consultations suggested that well-designed interventions with minimal technical problems were critical to the success of e-health interventions for MSM. There was a strong feeling among stakeholders that even minimal technical issues would have a profound impact on intervention engagement and effectiveness.

Privacy also emerged as an important aspect of acceptability in our review, suggesting that, with respect to e-health interventions addressing sexual health, detailed partner-level questions on sexual behaviour could feel intrusive and that features protecting app access and obscuring the manifest purpose of apps (for sensitive health domains) promote acceptability. The importance of privacy is also supported by existing evidence on behaviour change interventions for MSM155 and general populations. 153

Participants liked content that was interactive and aesthetically pleasing, and they enjoyed the use of diverse media, such as animations, videos and graphics. However, among rural MSM, these media could also increase loading times for users with slower internet connectivity. Although modular approaches could support users to absorb programme content cumulatively, interventions that were too long detracted from acceptability and some users preferred that little or no time be required between sessions. The ideal number and duration of modules are likely to depend on a variety of participant, intervention and contextual factors.

Individual tailoring based on participant characteristics and risk profiles increased acceptability, highlighting this as a particularly promising approach and aligning with studies of e-health behavioural interventions for other populations. 144,152,156 Participants valued interventions that presented scenarios and other content that reflected their experiences as MSM, an approach that stands in contrast to most existing e-health interventions targeting mental health and HIV prevention. 56,157 When interventions targeted sexual minority groups more broadly, some participants suggested further tailoring based on the sexual and gender identities of its users. The language and tone of intervention content emerged as an important factor shaping acceptability for MSM, who appreciated the use of colloquial, direct, ‘up-beat’122 and sex-positive language. A candid tone and sex positivity were also highlighted in stakeholder consultations as important in engaging a diverse group of MSM, and matching intervention language to colloquial language used by MSM was felt to be essential. Stakeholders highlighted involvement of potential users in setting the tone and direction of these interventions as an important component of formative development.

Findings from our review also highlight the importance of paying careful attention to language and framing to ensure that these affirm sexual minority identities. That these concerns arose in interventions designed explicitly for sexual minority users, including one adapted for sexual minority young people using participatory approaches,107 suggests that this is an important area to explore during the pilot phase of intervention development.

As with studies of e-health interventions for general populations,150,153 perceived usefulness was key to acceptability. Participants liked gaining new knowledge and skills from e-health interventions and developing an awareness of the relationship between sexual behaviours and substance use.

Although reviews of e-health interventions for general populations report higher use and engagement among participants with higher levels of education,150,152,153 our findings suggest that in the context of a generally high use of electronic devices among MSM,47 the targeting of intervention content might be a more important determinant of the relationship between education level and receipt of e-health interventions than their electronic mode of delivery. 130 Similarly, findings on the greater acceptability of the Keep it Up! intervention among black, Latino and other non-white users, compared with white users, suggest that e-health interventions can be developed to enhance inclusive acceptability among racially and ethnically diverse users. 130 There was otherwise little evidence of engagement varying by sociodemographic factors, although findings on SES were mixed96,121 and Madkins et al. 130 found that overall, and among white Keep it Up! users, those with lower levels of education reported higher intervention acceptability. Qualitative data suggest that e-health interventions can play a role in complementing external mental health support among MSM107,119 and that interventions targeting all adolescents might struggle to pitch content appropriately across this age range. 107

Outcome data synthesis

Our systematic review of intervention effectiveness included 14 trials, of which 13 included active versus control comparisons. Trials included substance use, HIV, STIs and sexual risk behaviour outcomes, but not mental health outcomes. Substance use outcomes did not include alcohol use. Furthermore, all outcome estimates drew from ≤ 12 months of follow-up post intervention. A further two trials99,102 presented active versus active comparisons. Neither trial found a difference between tested interventions on sexual risk outcomes, and thus are not discussed further. The evaluation of the China–Gate HIV Prevention Program found a small but significant increase in benefits among participants with the least education, compared with those with university education or above;124 this is notable in the light of the process evaluation of Keep it Up!, which found that users with lower levels of education and white users reported higher intervention acceptability overall (but not non-white users). 130 However, equity-relevant characteristics, for example moderation of intervention effectiveness by income, ethnicity and other social variables, were not meaningfully addressed by this body of evidence.

In active versus control comparisons, analysis for HIV infection drew on two studies, one with a short-term follow-up125 and one with a mid-term follow-up. 113 Neither study suggested that an e-health intervention was effective at reducing infections, although short follow-up times and low event rates precluded meaningful comparison. The GRADE profile suggested that the certainty of these findings was low or very low. Analyses for STIs were similarly scant, drawing on two trials in the short term110,125 and one trial in the mid-term113 follow-up. Although a pooled analysis of short-term follow-ups suggested no impact of interventions on incident STIs with very low precision,110,125 the one trial informing the mid-term follow-up113 did suggest a meaningful and statistically significant reduction in incident STIs, with corresponding moderate certainty.

The largest analyses assessed sexual risk behaviour outcomes. Although the GRADE profile suggested that the certainty of conclusions was very low or low, primarily because of risk of bias in the included trials and possible publication bias, pooled estimates from mid-term follow-ups drawing on six trials suggested a small and statistically significant impact of e-health interventions in reducing sexual risk behaviour (d = –0.12). A pooled estimate from short-term follow-ups drawing on eight trials did not reach statistical significance, but suggested a trend towards reductions in sexual risk behaviour of similar magnitude (d = –0.14). These findings are in line with those from a 2019 review55 which concluded that e-health interventions targeting HIV/STI prevention among MSM could affect behaviours in the near term, but found few studies assessing change at 12 months, only one of which demonstrated a significant effect at that time point. We tested whether or not interactivity of interventions related to intervention effectiveness on sexual risk behaviours; however, a meta-regression did not suggest significant differences between interventions on the basis of this characteristic.

Findings for drug use drew on two studies with short-term follow-ups,115,125 which could not be meta-analysed because of extreme heterogeneity, and one study with mid-term follow-up. 115 Together these studies did not present consistent evidence of effectiveness, with only one of these studies125 reporting evidence of a short-term impact from the Safe Behaviour and Screening open-ended intervention with general content on reduced recreational drug use. Furthermore, the GRADE profile for both analyses suggested that the certainty of the evidence was very low.

We found only two studies that examined the effects of e-health interventions on outcomes that spanned sexual health and drug use, with Reback et al. 115 reporting no effects on sexual risk behaviour, but effects on one measure of drug use, and Chiou et al. 125 reporting effects on measures of sexual risk behaviour and drug use, but not HIV infections or STIs. This lack of evidence for the effects of interventions that address more than one of the outcomes constituting the syndemic affecting MSM means that further trials will be required of interventions addressing such outcomes.

Economic evaluation synthesis

Our search identified only one economic evaluation eligible for inclusion in this review. 132 This study suggested that the intervention may have been cost-effective in reducing episodes of CAI, but this finding was undermined by the lack of probabilistic sensitivity analyses examining the large degree of uncertainty around these results.

Search and study selection

The original review searches involved multiple sources and methods, and aimed to maximise sensitivity. However, the updated searches were necessarily narrower because of the limits imposed by the COVID-19 pandemic. However, the sources that yielded all of the included study reports found as a result of the original electronic searches were included in the updated searches, so we think it unlikely that any studies were missed because of this reduced scope.

Typology of intervention approaches

We took an inductive approach to developing the typology of included interventions, which we grouped according to the structure of content delivery. Grouping based on other characteristics, such as delivery platform (e.g. SMS, mobile app), might be equally valid, and other reviewers might have developed a different typology of interventions. Categorising interventions based on behaviour change approaches such as the level of tailoring144 or interactivity or the use of modelling143 could also be useful approaches, but this was not possible because of inconsistent levels of detail available for different interventions.

Theory of change synthesis

Our synthesis was limited by the quality of the existing theory reports, which sometimes did not describe clear pathways from intervention activities to intended outcomes. In some cases, reviewers inferred relationships between theory of change constructs (which are made clear in our theory of change diagrams) and, in others, the relationships between specific activities, mediators and outcomes could not be determined. Although our approach to theory synthesis enabled us to systematically explore constructs and the relationships between them across intervention theories of change, synthesised diagrams do not capture aspects that theory reports suggest influenced theories of change when their role was not clear enough to be included in intervention-specific diagrams.

Our assessment of reported theories of change did not include assessment of the parsimony of theories of change, because we have found in past reviews that this is very difficult to consistently operationalise as a criterion of quality assessment; however, this is an important feature of theories of change. We also did not aim to systematically assess the evidence base for each of the scientific theories underpinning the intervention theories of change, because this was outside the scope of this review and would require assessing not only the evidence for the scientific theory, but also the evidence for the application of that theory to the outcomes targeted in the intervention theories of change it underpins.

Process evaluation synthesis

Our process evaluation synthesis was limited by the size and quality of eligible reports. Most were assessed as being of medium or high quality in terms of their reliability and usefulness. However, studies often lacked depth and breadth of analysis, and only around half were judged to privilege MSM’s perspectives.

Although the vast majority of interventions targeted MSM only and all were evaluated principally among MSM, three were assessed among samples that also included cisgender women. 107,108,111,119 Author narratives and quantitative data did not always disaggregate MSM from other participants, raising the possibility that specific findings from these three studies might reflect data from other groups. The process evaluation of the smartphone self-monitoring intervention was the sole study contributing to findings on intervention benefits of self-monitoring and self-expression. 119 Although the intervention targeted people of all genders and sexual identities living with HIV, > 80% of study participants identified as male and > 80% identified as gay or bisexual. 119 In two studies, just under half of participants identified as female,107,108,111 but all themes informed by these studies also drew on other studies. The make-up of participants in these three studies is therefore unlikely to affect the validity of the themes to which they contributed. Studies of relevant interventions among broader sexual and gender minority populations might add further insight but could not be included as we could not be certain which findings reflected experiences of, or relevant to, MSM.

Outcome data synthesis

The meta-analysis drew on evidence of variable quality, with limited scope for meta-analyses. We were unable to account for heterogeneity between studies where this was present because of either scarcity of evidence or limitations of our evidence base. We were unable to undertake NMA, and most meta-analyses had too few studies to make meta-regression (e.g. comparing intervention type on outcomes) meaningful. In our analysis of sexual risk outcomes, which was the one model for which we were able to undertake meta-regression, we were unable to explain heterogeneity. Meta-regressions by outcome type to determine differential effectiveness on outcomes within sexual risk would have been uninterpretable because of the statistical methods used for meta-analysis, and because of multiple studies reporting outcomes across several domains. Probable publication and selective reporting biases across studies meant that several estimates of intervention effectiveness from included studies could not be included in our meta-analyses; in at least one case, outcomes stated in a trial protocol were not published in the main trial report. Finally, we were unable to locate evidence for some scoped outcomes.

Economic evaluation synthesis

The synthesis of economic evaluations was limited to one study132 eligible for inclusion in this review, which provided only limited information because of the uncertainty of its estimates.

Implications for research

Prior to this review, evidence suggested that e-health interventions are a feasible and acceptable approach for reaching MSM with targeted health interventions, particularly for men with lower-intensity needs or where access to face-to-face provision is limited. 146 Previous evidence drawn from the general population or populations other than MSM suggests that e-health interventions might be effective in reducing sexual risk behaviour43,45 and substance use34 and addressing common causes of mental ill health,35–41 but effectiveness for MSM had not been adequately synthesised.

Our synthesis of theories of change identified three distinct theory of change pathways underpinning existing e-health interventions for MSM targeting sexual health, substance use and mental health outcomes, two of which underpin interventions targeting all three of these outcomes. Our review of theories of change suggests that interventions addressing these different outcomes may aim to exert impacts via common mechanisms of action, further adding to the potential for e-health interventions targeting the syndemic of sexual risk, substance use and poor mental health affecting some MSM. The synthesised theories of change are non-exclusive and may be combined to inform development of an e-health intervention holistically addressing this syndemic of multiple, often co-occurring, health issues among MSM. Although the ‘cognitive therapy’ synthesised theory of change is applicable only to interventions addressing mental health, the ‘cognitive/skills’ and ‘self-monitoring’ synthesised theories of change are relevant to all of the outcomes examined in this systematic review. Our synthesised theories of change could be augmented by use of the scientific theories on which they are based to inform a nuanced understanding of these theoretical underpinnings and how they can be most usefully applied in specific interventions. In the case of the ‘cognitive/skills’ synthesised theory of change, this would probably involve selecting a subset of mediators on which to focus, because a single intervention would not be expected to address the full range of constructs presented in this synthesised theory of change.

The findings suggest that the quality of existing theory reports is low to medium, with limited discussion of the inter-relationships between theoretical constructs and little attention to how mechanisms might vary by context. Improving the quality of theory reports would enable a better understanding of how interventions are intended to work and the evidence supporting this. It would also facilitate evaluations by identifying appropriate mediators and moderators of effects, the use of which could help outcome assessment to identify which components are triggering which mechanisms of change, and to what effect among which of its users. 158 We suggest that intervention developers provide clear theories of change for their interventions, informed by existing scientific theories of behaviour and behaviour change relevant to the approach of the intervention and to the outcomes it seeks to address. Such theories of change can ensure that intervention activities align with their intended outcomes, and can ensure that evaluations are focused on the most appropriate measures of implementation, mediators and outcomes, and consider how mechanisms might vary by context and/or population.

The synthesis of process evaluations suggests that e-health interventions offer a feasible and acceptable approach to promoting the health of MSM, and allow MSM to access health promotion interventions privately, anonymously and at times that they find convenient. This synthesis identified several factors shaping the receipt of e-health interventions by MSM, which applied across interventions addressing substance use, mental ill health and sexual risk. These included ease of use, clear and comprehensive content, fun and enjoyable content, appropriate language and tone, interaction and personalisation, privacy and lack of intrusiveness, appropriate pacing and structure, content relevant to participants’ lives and relevance of intervention goals. Other factors should be considered carefully in designing interventions for MSM, including ensuring that language and tone are affirming of sexual minority identities and that content reflects the reality and experiences of MSM. The findings suggest that e-health interventions are acceptable for MSM across sociodemographic groups, although evidence in this area is limited and mixed. Different content for younger and older adolescents might be warranted. Variation in engagement and acceptability by participant characteristics, including ethnicity and level of education, should be explored in future research, and new interventions should be rigorously piloted to refine aspects affecting usability and acceptability. 55,150

This review found that participants valued interventions that addressed the reality of their lives and the inter-relationships between the different domains of health, and the consultation with stakeholders found unanimous agreement among clinical and voluntary sector stakeholders that new interventions should holistically address the range of outcomes included in the review. Therefore, the findings suggest that e-health interventions simultaneously addressing sexual health, substance use and mental health might be particularly acceptable. The findings on the factors promoting acceptability of interventions can inform the development of future e-health interventions to address the syndemic of substance use, mental ill health and sexual risk among MSM and guide research questions for pilot and process evaluation studies. Those developing e-health interventions for MSM should use co-production and testing processes that ensure that the above factors are addressed and that interventions are acceptable. This approach is supported by findings from stakeholder consultations, which emphasised the importance of co-design for developing interventions that are relevant and accessible to MSM. Process evaluations should explore a broader range of individual, intervention and contextual factors that might affect implementation, and they should collect more in-depth, ideally qualitative, data privileging the perspectives of intended beneficiaries. Outcome evaluations of such e-health interventions should conduct linked process evaluations whenever possible, which would shed further light on factors affecting how they are delivered and received. 158

We were interested in studies of interventions to address outcomes in the different domains of sexual health, substance use and mental health either in combination or separately. Studies addressing the domains in combination would indicate whether or not there is already good evidence for ‘holistic’ e-health interventions to address this syndemic of interclustered outcomes. Studies addressing them separately would provide some indication of the potential for developing and testing such a holistic intervention, particularly if no such holistic interventions have been evaluated to date. We found no outcome evaluations of holistic interventions addressing all of the outcome domains of interest. We found only two studies that examined the effects of e-health interventions on outcomes that spanned sexual health and drug use, with one115 reporting no effects of an e-health intervention on sexual risk behaviour, but an effect on one measure of drug use, and another125 reporting effects on measures of sexual risk behaviour and drug use, but not HIV infections or STIs. We found no evaluations of e-health interventions reporting effects for other combinations of outcome domains.

The quality and quantity of evidence supporting the effectiveness of e-health interventions for most of the outcomes that we set out to analyse was generally low or, in the case of alcohol or mental health outcomes, non-existent. The wide CIs surrounding many effect estimates suggest that many trials involved insufficiently large samples. Even when meta-analyses drew on multiple studies, issues with included trials precluded certainty in the evidence presented. Moreover, despite substantial heterogeneity in meta-analyses for sexual risk behaviour outcomes, we were unable to explain this heterogeneity. Although there was some evidence for intervention effects on sexual behaviours, there was inconsistent evidence regarding effectiveness preventing STIs and drug use, and no evidence for effectiveness preventing HIV infections.

Another key gap in this systematic review related to the inclusion of outcomes that accurately reflect current knowledge about minimising sexual risk. For example, a focus on condom use does not reflect that risk for HIV can be managed through effective biomedical means, such as adherence to HIV treatment for people living with HIV, or PrEP for those who are HIV negative. It is likely that interventions designed in the current context would more explicitly acknowledge biomedical approaches to managing risk.

One of the questions we set out to address in this systematic review was if the existing evidence suggests that our scoped outcomes could coherently, feasibly and effectively be addressed by a single health intervention addressing the syndemic of sexual risk, substance use and poor mental health affecting some MSM. It is clear, based on the meta-analyses presented, that the evidence does not, as yet, suggest that this is the case. This is largely because the majority of interventions were focused on individual, not syndemic, outcomes, as well as because of the patchy effects for outcomes that were assessed. Only two interventions115,125 assessed effects on substance use outcomes alongside sexual risk behaviour outcomes; for trials that otherwise reported outcomes over multiple categories, this was between sexual risk behaviour and either HIV or other STIs. No studies reported outcomes for depression or anxiety, despite poor mental health being a key syndemic condition, nor did any studies report outcomes relating to alcohol use, despite this being the most commonly used intoxicant in developed country settings.

Given the lack of evidence for e-health interventions to reduce risk across different outcomes, we cannot currently recommend scale-up of any e-health intervention aiming to address these outcomes synergistically. Given the lack of rigorous evidence for the effects of e-health interventions on certain outcomes (HIV and STIs, substance use and mental health in particular), a priority for future research is rigorously conducted studies of e-health interventions focused on these outcomes. Such studies may focus on a single domain of outcomes (e.g. mental health or alcohol use). However, given the syndemic of inter-related outcomes affecting some MSM, our findings that e-health interventions addressing diverse outcomes are often underpinned by similar theories of change and that several factors shape MSM’s receipt of such interventions applied across targeted health outcomes, our view is that it would be legitimate for future studies that focus primarily on one main domain of effects (e.g. alcohol use) to examine secondary outcomes in other domains (e.g. mental health) or to include multiple primary outcomes focused on these different domains. This could provide a more rapid means of identifying the potential for e-health interventions to address these syndemic multiple outcomes. Future trials of e-health interventions should also include several considerations. First, given the complete lack of evidence in this area, trials should consider how to address poor mental health among MSM, with a focus on how determinants of poor mental health among MSM relate to other outcomes considered here (sexual ill health and substance use) and to other antecedents (e.g. stigma). Second, trials should address a range of substance use behaviours that are syndemically linked to other relevant outcomes (mental well-being and sexual health), including alcohol use. Third, trials should involve interventions drawing on common and complementary theories of change to address multiple outcomes, and develop and test intervention content in collaboration with MSM. Fourth, trials should incorporate follow-up long enough, and sample sizes large enough, to detect a meaningful impact on HIV and other STIs, given the time needed to detect meaningful differences in HIV incidence. Fifth, trials should incorporate rigorous process and economic evaluations. Sixth, trials should ensure that interventions are not inequality-generating and that this is examined empirically by involving representative, diverse samples of MSM and examining how any intervention effects are moderated by factors such as gender identity, ethnicity, baseline health status and SES. This is important because interventions may unintentionally exacerbate inequalities between groups as a result of, for example, differential access to mental or sexual health services or substance use services. Seventh, to generate a joined-up, comprehensive e-health intervention that targets multiple outcomes, intervention evaluations should seek to generalise both mechanisms and components that are successfully used to achieve change in one outcome over multiple outcomes. Finally, e-health interventions are, of necessity, individualistic and inaccessible to men with limited internet access, and so should be complemented by other interventions, including community and structural interventions, addressing the broader upstream influences on the health of MSM.