Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update

Methods for stage 1: gap analysis

The aim of the gap analysis was to identify and summarise aspects of the previous guidance that required updating. We used these gaps as a starting point for discussion within the project team, SAG (for a list of members, see Appendix 3) and identified experts. It was, therefore, a method of agenda setting and, thus, did not aim to be comprehensive. The intention was that issues could be added as the work progressed.

Our first step was a brief horizon scanning review that focused on new approaches/progress since the previous guidance, criticisms of existing guidance and other gaps. Based on initial reading of the literature and the experience of the project team, the SAG were provided with a list of topics for update. This was discussed at the initial SAG meeting (24 November 2017) and the list of topics was updated for more in-depth exploration of the literature.

Separate literature searches were conducted for each of the identified topics using keywords (the topic of interest plus variations of ‘complex intervention’) in Web of Science, restricted to English language with the date limited to those published since 2008. Where there were existing guidance documents relevant to the development, implementation or evaluation of complex interventions, we used these as our starting point and limited our literature review to documents published after these existing guidance documents. For example, guidance existed for natural experiments,9 process evaluation5 and context,10 which we drew heavily on. We also discussed this new framework with those involved in developing other guidance at the time, for example for intervention development,11 exploratory studies12 and systems thinking. 13,14 We excluded guidance that did not provide substantive information on methodological issues. Criteria for including other publications were broadly that they provided relevant thinking that could be used to progress the work. A summary of the findings from each topic search was created and used to identify focal points for the expert workshop.

Findings from stage 1: gap analysis

Several limitations of the 2006 guidance were highlighted. These mainly related to (1) the focus on effectiveness; (2) considering randomised controlled trials (RCTs) as the gold standard research design; (3) the lack of detail on economic evaluation; (4) the lack of attention to mechanisms that deliver change; (5) the lack of acknowledgement of complex organisational systems or complexity theory; and (6) omission of the importance of policy context, including the impossibility of standardising context. In addition to these issues, there were several areas that had progressed since 2006, which were considered to be important to explore in more detail. Here we provide a brief summary of the gaps that were identified in 2017 and what we initially proposed to be discussed with experts at the workshop:

• Complex intervention definition –

  • Key issue for the update – definition is too narrow.

  • There are different dimensions of complexity. How can we improve the definition of complex intervention to better acknowledge contextual complexity and the system-level properties that add to this complexity?

  • Does the term ‘complex intervention’ make sense when complexity arises from the intervention context and the interplay between intervention and context as well as the intervention itself?

• Intervention development –

  • Key issue for update – little practical guidance on this phase and subsequent literature provides more detail on certain aspects, for example using a theory-driven approach;15,16 identifying and delivering the mechanism of change;17 co-production and prototyping of public health interventions;18 and optimisation of the intervention. 19

  • There is guidance under way on intervention development (INDEX study11) that is about identifying and assessing different approaches to developing complex interventions. Given that this is current and there are clear overlaps, are there any issues that are not covered in the INDEX guidelines that we should consider adding in this document?

• Pre-evaluation phase: appraisal of evaluation options or exploratory work –

  • Key issue for update – previous complex intervention guidance highlighted the importance of preparatory work, with the focus being on conducting a pilot study; however, detail on feasibility issues and how to develop the feasibility stage is required.

  • Similar to the development phase, there is work in progress to create guidance for Exploratory Studies of Complex Public Health Interventions (GuESt study12), which includes a lot of relevant information. Should the current update include anything further and how can we make it relevant beyond public health?

• Context –

  • Key issue to update – although the previous complex intervention guidance states the importance of taking account of context, this is mostly about how context affects outcomes at the evaluation phase and how interventions may require adaptation for implementation in different contexts. There is little attention to the consideration of context throughout the research phases or guidance on how to take it into account.

  • How do ‘context’ and ‘system’ differ/overlap?

  • Context is a critical construct; how do we ensure that we refer to it throughout the research process?

  • Do we want to go further than the recently published guidance on taking account of context in population health intervention research?10 What are the key points for considering context in complex intervention research more broadly?

• Ideas from complex systems science –

  • Key issue to update – this is an area that has received increasing attention in the last decade, and for this reason the previous complex intervention guidance did not draw on it.

  • Examples of using complex systems thinking in public health research have been limited to describing and modelling systems; this has not yet been taken further and been used to develop and evaluate interventions. 7

  • When is it critical to embrace a complexity perspective (and when it is not necessary: simple and complicated questions and approaches have their merits) and how can such a perspective be implemented methodologically?

  • How can a complex systems approach guide each phase of complex intervention research?

• Programme theory –

  • Key issue to update – the previous complex intervention guidance provided brief information on causal mechanisms and on developing a theoretical understanding of the process of change; however, this lacks the required level of information to guide researchers in developing programme theory from the outset.

  • Further detail is needed to illustrate the steps required to undertake a robust planning phase, including (1) identifying appropriate theories of change, (2) considering potential mechanisms of change, (3) anticipating important contextual factors that could influence the change mechanism and outcomes and (4) mapping appropriate methods to operationalise the chosen theory into practice.

• Implementation research –

  • Key issue to update – the previous complex intervention guidance has limited information on the practical implementation process and needs to understand and account for dynamic contextual factors.

  • Successful implementation is critical to the scaling up of interventions and the new framework should reflect this by emphasising implementation throughout the research process.

  • When do you stop doing effectiveness studies and start doing implementation studies?

  • How can we include the wider aspects of implementation that may enable or constrain desired change? For example, how much guidance do we provide on addressing intervention context and addressing future implementation on a greater scale?

  • How do we make the information palatable for decision-makers?

• Economic evaluation and priority setting –

  • Key issue to update – the previous complex intervention guidance did not go into any detail on how standard economic evaluation methods need to be adapted to deal with particularly complex interventions.

  • Issues around timeline – outcomes are likely to extend beyond the lifetime of an evaluation – can economists work with proxies to system change?

  • How do we best guide on issues for existing economic evaluation methods where interventions aim to change the properties of complex systems? That is, it is not appropriate to evaluate health outcomes only at the individual level if a component of the intervention is to effect change to the system; outcomes are broader than individual health and costs (is a societal rather than a health-care perspective required?).

  • (How) should we include equity issues and economic evaluation analytical approaches, which are growing and complicated methodological areas?

  • How can we guide on economic evaluation for priority setting? That is, what is the most efficient use of resources (to determine whether or not the additional cost of a research project or particular study design is justified)? Are decision-modelling and value-of-information analysis (VOI) practical propositions?

• Systematic reviews of complex interventions –

  • Key issue to update – the previous complex intervention guidance did not address issues related to the inclusion of complex intervention studies in systematic reviews, much beyond acknowledging that they can be problematic. Should we add more?

  • Systematic review methods may differ from standard methods and extra consideration is necessary where the systematic review includes complex interventions (if the review is about complexity), for example in defining the research questions, developing the protocol, the use of theory, searching for relevant evidence, and assessing complexity and quality of evidence (how to identify key components of complex interventions; how to assess study quality).

  • What should be the end point of a systemic review of complex interventions? For example, effect size, decision model, improved theory or supporting policy decisions?

• Patient and public involvement (PPI) and co-production –

  • Key issue to update – previous complex intervention guidance mentioned that stakeholders should be consulted at various points, but did not emphasise the need to engage relevant stakeholders throughout the research process or provide any guidance on how to do this.

  • How do we guide on effective engagement of stakeholders throughout?

• Evaluation –

  • Key issue to update – the previous complex intervention guidance focused on designing evaluations to minimise bias (i.e. with high internal validity) and, in doing so, did not consider how to maximise the usefulness of evidence for decision-making. These are not mutually exclusive concerns: could both be considered?

  • Should we take an approach that promotes ‘usefulness of evidence’ rather than hierarchy of evaluation study design?

  • Should we present evaluation options that go beyond individual-level primary health outcomes? For example, taking account of system change.

  • Evaluation study designs – what should be added to reflect development in this area? For example, n-of-1, adaptive trials. How much information should we present on individual study design?

These topic areas and questions were intended to be a foundation for discussion and further consideration, rather than an exhaustive or definitive list.

Methods for stage 2: expert workshop

A 1-day expert workshop was convened in London in May 2018. A list of those who attended the workshop is given in Appendix 3. The aim of the workshop was to obtain views and record discussions on topics that should be newly covered or updated. Participants were identified by the project team and SAG. We aimed to have at least two experts for each of the identified topics and include a range of people from across the UK, plus international representation as far as budget allowed.

In advance of the workshop, the participants were asked to provide two key points, each with one sentence of explanation, that they felt should be taken into account in the update. These key points, alongside findings from the stage 1 gap analysis and discussion with the SAG, were used to inform the agenda and develop content for an interactive, multidisciplinary expert workshop.

After an introductory presentation by the project team, participants were split into five groups (of seven or eight) for the morning session round-table discussion.

The topics covered for all groups (presented in a different order) were:

• the definition of complexity

• the overall framing and scope

• potential impact of the new framework

• the main diagram of the framework (key elements of the development and evaluation process)

• complex systems thinking.

For each of the two afternoon sessions, participants were split into five ‘expert groups’ aligned with their topic areas of expertise. Topics covered in these smaller specialised groups included:

• options for study design

• the previous guidance’s emphasis on ‘effectiveness’

• choice of outcomes

• considerations for economic evaluation

• pre-evaluation and development phases

• considerations for implementation

• key focus areas to improve applications for funding

• evidence synthesis of complex interventions

• considerations for digital health

• programme theory.

Each session was facilitated by a member of the project team and was supported by a colleague from the MRC/Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow. Colleagues assisted the facilitators by taking notes of key points during each discussion, clarifying main points with attendees and producing a written summary of each discussion after the workshop. SAG members were also present in each discussion. Round-table discussions were audio-recorded. Throughout the day, participants were asked to provide their thoughts on key points, case study examples and key references on Post-it^® notes (3M, Saint Paul, MN, USA) on dedicated noticeboards.

Data from each of the 15 workshop discussions and post-it points were thematically coded, and summaries drawing on all of the data were created for each theme. These workshop summaries were sent to workshop participants by e-mail as a follow-up consultation to ensure that the thematic summaries that we created from the workshops were accurate overviews of the discussions in which they were involved. Final summaries were discussed in detail with the SAG to support the decision-making on the content of the document.

Findings from stage 2: expert workshop

Seventy experts were invited to the workshop (with the aim of facilitating a workshop of around 40 participants). In total, 37 experts confirmed their attendance; one who accepted the invitation did not attend (owing to sickness), three people did not respond to our invitation and 30 people could not attend for various reasons, some of whom recommended others who did attend. In total, 36 participants attended the workshop. Key issues that were identified are summarised in Table 1.

Methods for stage 3: open consultation

The first draft of the updated document was made available for open consultation from Friday 22 March to Friday 5 April 2019.

Potential respondents were targeted, as follows:

• those invited to the expert workshop

• other experts identified from the suggestions of workshop participants, with greater focus on international experts

• early and mid-career researchers (identified via e-mail groups)

• journal editors

• funders

• service users/public

• policy-influencers/-makers.

We e-mailed potential respondents with advance notice of the consultation dates and a link to register their interest in participating, and sent a further message when the consultation opened. Two reminder e-mails were also sent. As well as targeted promotion, we used social media to publicise the consultation and encouraged others to pass on the link.

Consultees were informed that they were responding about an early draft of the revised framework and that their involvement was an important part of the process for its final development. We asked them to relate topics in the draft to a project that they had recently worked on and to provide feedback on its usability.

The online consultation was guided by a questionnaire that was developed by the project team (the questions that all consultees were asked to complete are presented in Appendix 4). Responses were anonymous.

Findings from stage 3: open consultation

We received 52 individual responses, plus some follow-up e-mail comments. This amounted to 25,000 words of response. The majority of responses were from researchers, but some identified as funders (n = 3), journal editors (n = 7), NHS (n = 7), policy-influencers (n = 3) and service users (patient or public, n = 5). Most of the respondents said that their main field of expertise was public health (n = 21) or health services research (n = 20), with others stating clinical medicine (n = 6), implementing policy (n = 3), systems-based research (n = 4), patient or public involvement (n = 4) and other (n = 7: statistics, sociology, health economics and triallist) as their main field of expertise. A summary of the consultation suggestions is provided below; however, it is important to note that there were conflicting views on some aspects, which we have noted.

Overall

• Overall layout: extra sections are required – an executive summary and a preface chapter that details how this is related to previous guidance and that this document is a standalone framework that does not require reference back to the 2006 version. Consider placing more emphasis on development in the earlier sections of the document rather than delve straight into evaluation.

• Definition of complexity: the distinction made between complicated and complex interventions was said to be unclear. Respondents stressed that a clear definition of complex intervention and a more accessible account of how complexity affects the research process are required.

• Key elements for developing and evaluating complex interventions (Figure 1): respondents felt that the ‘overarching considerations’ should all be highlighted as central to the research process and that some text detail should be added to each phase box to provide more information on what each means.

• Evaluation perspectives (shown in the x-axis of Figure 2): many respondents felt that the perspectives that we presented were shown to be mutually exclusive and hierarchical (which was not the intention). There was significant pushback on using the term ‘realist’ as an evaluation perspective. Respondents questioned whether or not we were advocating for evaluations that do not measure effectiveness, with some conflicting views on whether or not this was a positive change. It was felt that there was not enough focus on how the perspectives relate to intervention development or to the development of research questions.

• Framework for addressing complexity within an evaluation (see Figure 2): although some liked this framework, on balance respondents did not feel that this figure complemented the text or was very clear. Complexity does not increase in a linear fashion based on intervention components and perspective taken. Context and system were missing from the diagram despite being a large focus of the text.

FIGURE 1.

Key elements for developing and evaluating complex interventions (consultation version).

FIGURE 2.

Framework for addressing complexity within evaluation (consultation version).

Methods for stage 4: writing the updated framework

The writing of the framework was led by the project team and was supported by co-authors in the writing group and the SAG. Feedback was received at various stages throughout the writing process from members of the MRC’s Population Health Sciences Group (PHSG) and the MRC–NIHR Methodology Research Programme (MRP) Advisory Group.

Given that the document had changed substantially from the open consultation draft, we asked a further set of external individuals to provide comments on the near-final draft. We received feedback from eight people in May/June 2020. The final draft was then sent to all co-authors for approval.

Findings from stage 4: final approval and sign-off

The final draft was approved by the MRC’s PHSG in March 2020.

Patient and public involvement

This project was methodological; views of patients and the public were included at the open consultation stage of the update. The open consultation, involving access to an initial draft, was promoted to our networks via e-mail and via digital channels, such as our unit Twitter account (Twitter, Inc., San Francisco, CA, USA; www.twitter.com). We received five responses from people who identified as ‘service users’ (rather than researchers or professionals in a relevant capacity). Their input included helpful feedback on the main complexity diagram, the different research perspectives, the challenge of moving interventions between different contexts and overall readability and accessibility of the document. Several respondents also highlighted useful signposts to include for readers.

In relation to broader PPI, the resulting updated framework (see Chapter 2) highlights the need to include PPI at every phase of developing and evaluating complex interventions. We have drawn on and referred to numerous sources that provide further detail or guidance in how to do so.

Limitations

There was a huge amount to cover in developing this document. We have not provided detailed methodological guidance where that is covered elsewhere because we have tried to focus on the main areas of change and novelty. In many of these areas of novelty, methods and experience are in some parts quite limited. In addition, we have foregrounded the very important concept of ‘uncertainties’ and, although there are methods of doing this through, for example, decision-modelling and more qualitative soft system methodologies, this area is limited and specific guidance on how to determine uncertainties in a formal way may seem unclear. We recommend that due consideration is given to this concept and call for further work to develop methods and provide examples in practice. Invariably we may have missed something in our writing and, furthermore, the fields will inevitably move on at pace following publication of this document.

Complexity owing to characteristics of the intervention

Parts of this text have been reproduced with permission from Skivington et al. 26 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

Interventions become more complex in line with increasing the:1

• number of intervention components and the interactions between them

• range of behaviours, expertise and skills (e.g. particular techniques and communication) required by those delivering or receiving the intervention

• number of groups, organisational levels or settings that are targeted by the intervention

• level of flexibility or tailoring of the intervention or its components that is permitted (i.e. how dynamic or adaptive the intervention is).

For example, the Links Worker Programme27 was an intervention in primary care in Glasgow, Scotland, that aimed to link people with community resources that could help them to ‘live well’ in their communities. It targeted individual, primary care [general practitioner (GP) surgery] and community levels; the intervention was flexible in that it could differ between primary care GP surgeries. In addition, there was no single health or well-being issue that the Link Workers specifically supported: bereavement, substance use, employment and learning difficulties were all things that could be included. 27 The inherent complexity of this intervention had implications for many aspects of its evaluation, such as the choice of appropriate outcomes.

Complexity arising from context

. . . any feature of the circumstances in which an intervention is conceived, developed, implemented and evaluated.

Reproduced with permission from Craig et al. 10 Contains information licensed under the Non-Commercial Government Licence v2.0

Examples of features include social, political, economic and geographical contexts. Whether or not and how an intervention generates outcomes can be dependent on a wide range of contextual factors. These contextual factors will be intervention specific and may be difficult to anticipate. An example of an intervention that may seem simple, until context is considered, is the ‘Lucky Iron Fish’ (Box 1). 10

It is important to consider what features of context may be important in determining how an intervention achieves outcomes and under what circumstances an intervention may be more or less effective. This is important for all phases, that is for developing a new intervention, adapting or translating an existing intervention to a new context, evaluation and implementation.

Interventions as ‘events in systems’

a set of things that are interconnected in such a way that they produce their own pattern of behaviour over time.

Meadows and Wright31

Systems thinking can help us to understand the interaction between an intervention and the context in which the intervention is implemented in a more dynamic way. Systems can be thought of as complex and adaptive,32 in that they are defined by system-level properties, such as feedback, emergence, adaptation and self-organisation (Table 2). We can theorise interventions as ‘events in systems’24 and can conceive an intervention’s outcomes as being generated through the interdependence of the intervention and a dynamic system context.

Change in complex systems may be unpredictable. In a social system, people interact with each other and other parts of the system in non-linear and interconnected ways so that the actions of one person alter the context for others. 35 For example, removing hospital car-parking charges has clear beneficiaries. However, by encouraging people to drive, the policy may reduce demand for public transport, leading to a reduction or withdrawal of services. The net effect may be to reduce accessibility to those without a car. Demand for car parking will increase, possibly beyond capacity, and alternative options will have diminished. What might appear at first sight to be a positive intervention may have adverse effects on health and serve to widen inequalities.

Efficacy perspective

Research taking an efficacy or effectiveness perspective is principally concerned with obtaining unbiased estimates of the average effect of interventions on predetermined outcomes. Studies from an efficacy perspective aim to test hypotheses about the mechanisms of action of interventions. 38 Such research aims for high internal validity, taking an explanatory approach to test causal hypotheses about the outcome(s) generated by the intervention. This contrasts with effectiveness studies, which take a more pragmatic approach. The PRagmatic Explanatory Continuum Indicator Summary (PRECIS)-2 tool39 identifies nine domains on which study design decisions may vary according to the perspective or approach. Few studies exhibit all of the characteristics of a purely efficacy or a purely effectiveness perspective; there is not necessarily a dichotomy between efficacy and effectiveness studies, rather a continuum.

Studies from an efficacy perspective will typically evaluate interventions in idealised, controlled conditions among a homogeneous group of highly selected participants using a proximal outcome, such as disease activity, with the intervention delivered in a standardised manner with high fidelity to a protocol or manual, by highly trained practitioners, without the flexibility and variability that is likely to occur in real-world practice. Evidence from an efficacy study indicates whether or not an intervention can work in idealised conditions. Typically, efficacy studies control for contextual variation, so do not usually help to illuminate context dependence. An efficacy perspective could be taken to the development, feasibility and evaluation phases of intervention research. However, if found efficacious, this would not provide sufficient evidence for implementation. For example, testing whether or not a vaccine is efficacious in preventing infection under optimal conditions would be undertaken prior to developing a delivery programme, which would then need to be tested for effectiveness in practice. Another example is testing a psychosocial intervention for efficacy in optimal conditions, using experienced practitioners and carefully selected patients, a precursor to a further research cycle to refine and test the intervention for effectiveness in real-world settings.

Effectiveness perspective

Research from an effectiveness perspective seeks to answer pragmatic questions about the effects that are produced by interventions in routine practice to directly inform the choice between intervention options. 38 Effectiveness studies aim to test an intervention in samples and settings representative of those in which the intervention would be implemented in everyday practice, usually with a health or health service outcome. Flexibility in intervention delivery and adherence may be permitted to allow for variation in how, where and by whom interventions are delivered and received. Standardisation of interventions may relate more to the underlying process and functions of the intervention than on the specific form of components delivered. 40 For example, the INCLUSIVE trial41 assessed the effectiveness of the Learning Together programme of restorative justice to reduce bullying and aggression in schools. Although key intervention inputs were provided to all intervention schools with the aim that key functions were delivered to trigger the theorised mechanisms of change, each school was encouraged to ensure that the form of local implementation was appropriate for their students and context, with scope for locally decided actions. 41

Theory-based perspective

The primary aim of adopting a theory-based perspective is to provide evidence on the processes through which interventions lead to change in outcomes and what prerequisites may be required for this change to take place, thus exploring how and why they bring about change. This differs from developing or evaluating interventions using an effectiveness perspective, which focuses on identifying whether or not they ‘work’ based primarily on average estimates of effect. It also differs from an efficacy perspective, as theory-based perspectives explore interventions in practice, taking account of context, and often explore more than one single theoretical account of how the intervention may work. Such approaches to evaluation aim to broaden the scope of the evaluation to understand how an intervention works and how this may vary across different contexts or for different individuals. 42 In research taking a theory-based perspective, interventions are developed and evaluated through a continuous process of developing, testing and updating programme theory. Research from this perspective can generate an understanding of how mechanisms and context interact, providing evidence that can be applied in other contexts. For example, there are numerous mechanisms by which group-based weight loss interventions may bring about behaviour change. Change may be motivated by participants’ relationship with the facilitator or by interaction and sharing of experiences among members of the group, as well as by the specific content of the intervention. Whether or not and how such mechanisms generate outcomes will depend on the context in which the intervention is being applied. 43 Thus, the impacts of interventions cannot confidently be determined in the absence of knowledge of the context in which they have been implemented. 44

There are several approaches that take a theory-based perspective (Box 3 shows a selection of examples). They are ‘methods neutral’ in the sense that they draw on both quantitative and qualitative study designs to test and refine programme theories.

Systems perspective

A systems perspective suggests that interventions can be better understood with an examination of the system(s) in which they are embedded or the systems that they set out to change. A systems perspective treats interventions as events within, or disruptions to, systems. 24 The properties of a system cannot be fully explained by understanding only each of the system’s individual parts. 49 This perspective is concerned with an awareness and understanding of the whole system:

The essential point is that the theory driving the intervention is about the dynamics of the context or system, not the psyche or attributes of the individuals within it.

Hawe et al. 24

Key to a systems perspective is considering the relationships between intervention and context, engaging with multiple perspectives, and identifying and reflecting on system boundaries. 50 A systems perspective encourages researchers to consider how the intervention may be influenced by and impact on many elements of the system, and over an extended period of time. Rather than focusing on a narrow and predetermined set of individuals and outcomes within a fixed time period, a systems perspective will aim to consider multiple ways by which an intervention may contribute to system change. These may occur through multiple, often indirect, routes over an extended time period and through spillover and diffusion processes, some of which may be unintended. It is not usually possible to take account of a whole, often open, system; therefore, to make an evaluation tractable it will be necessary to limit its scope by determining a system boundary and restricting the range of potential mechanisms or explanations that are investigated.

Summary

Interventions rarely achieve effects evenly across individuals, populations and contexts. For complex intervention research to be useful to decision-makers it needs to take into account the complexity that arises both from the intervention’s components and from the intervention’s interaction with context. Intervention effects can be dependent on, can influence or change contexts, and this interdependence can vary over time. Complex intervention research should be conducted with an awareness of these multiple potential sources of complexity, with the design of any specific research study adopting the research perspective that is most suited to the research question that it aims to address (see Box 4 for examples). The preponderance of complex health intervention research to date has taken an efficacy or effectiveness perspective, and there will continue to be many situations in which research questions for which these perspectives are most suited will continue to be prioritised. However, for many interventions, the most critical research questions and the needs of decision-makers who use research evidence are not met by research that is restricted to questions of efficacy and effectiveness; therefore, a wider range of research perspectives and methods needs to be considered and used by researchers, and supported by funders. This may particularly be the case in health service delivery and public health research, in which organisational-level and population-level interventions and outcomes are research priorities that are not well served by traditional methods. 6,7 Key questions for evaluation include identifying generalisable determinants of beneficial outcomes and assessing how an intervention contributes to reshaping a system in favourable ways, rather than the dominant focus on the binary question of effectiveness in terms of individual-level outcomes. In the next section, we set out the revised framework that aims to help research teams consider the research perspective(s) most suited to the research challenge that they are aiming to address, to prioritise research questions, and choose and implement appropriate methods.

Context

. . . any feature of the circumstances in which an intervention is conceived, developed, implemented and evaluated.

Reproduced with permission from Craig et al. 10 Contains information licensed under the Non-Commercial Government Licence v2.0

Context is dynamic and multidimensional. Key dimensions include physical, spatial, social, cultural, digital, political or economic features of the circumstances in which interventions are implemented. Context ‘interacts, influences, modifies and facilitates or constrains the intervention’. 64 How an intervention works may vary across context and what works in one context may not be as effective, or may even be harmful, in different contexts (see Case study 2). The impact of a new intervention will depend on what provision already exists. Interventions may need to be explicitly adapted to fit different contexts (Box 5). As well as being influenced by context, many ‘interventions work to change the context within which health is produced and become part of the context for future health behaviours, interventions and outcomes’ (reproduced with permission from Craig et al. 10 Contains information licensed under the Non-Commercial Government Licence v2.0). Theorising interventions as interruptions in systems highlights the dynamic interdependence of intervention and context over time. Taking context into account is crucial throughout the research process:

• in theorising how the intervention will generate effects, intended and unintended

• in anticipating and understanding barriers to and facilitators of the implementation, and scaling up and scaling out of interventions

• in the development phase to make it more likely that the intervention is effective and implementable among the target population and setting

• at the evaluation phase, to understand how interventions ‘work’ in terms of how they interact with their context

• in the reporting of research, to help decision-makers understand whether or not the intervention will work and for whom in the context for which they are responsible. 10

Circumstances surrounding the intervention may change after the study has begun, for example through the introduction of a new policy or service or, in the case of evaluations with long-term follow-up, more fundamental changes in the social or economic context. The context may also interact with the intervention and interventions may be designed to be adaptive and responsive to context.

It is important to develop a clear understanding of what aspects of context may be important in a broader consideration of intervention outcomes and processes, and which of these are particularly relevant to the research questions to be addressed. Considering these wider contextual factors at the earliest stage can help to ensure that the research improves understanding of how an intervention works and how it might operate in or respond to different circumstances. It allows interventions to be developed, refined and adapted so that they fit with the contexts in which they are implemented.

Programme theory

Programme theory describes how an intervention is expected to lead to a set of specified outcomes and under what conditions. It articulates (1) the key components of the intervention and how they interact; (2) the mechanisms of the intervention; (3) the features of the context that are expected to influence those mechanisms; and (4) how those mechanisms may influence the context. 72

Case study 3 provides an example of how programme theory has been developed and used. For interventions designed to be adaptable across contexts, the programme theory should articulate the key functions that must be consistently delivered and indicate how far interventions can be responsive to different and changing contexts. 40,73 A programme theory may include aspects of mid-range or grand theories, but its aim is more specific: to articulate how and why an intervention is expected to work.

Stakeholders

Stakeholder engagement is required throughout the research process to maximise the likelihood that the research addresses relevant questions and leads to changes in practice or policy:

Those who are targeted by the intervention or policy, involved in its development or delivery, or more broadly those whose personal or professional interests are affected i.e. who have a stake in the topic. This includes patients and members of the public as well as those linked in a professional capacity.

Stakeholders should be involved early in the research process and remain engaged throughout. Reciprocity is vital, and the goal should be the development of collaborative relationships that continue throughout the phases of development, evaluation and implementation.

The purpose of stakeholder engagement will differ depending on the context and phase of the research, but could include drawing on stakeholders’ practical experience; obtaining stakeholders’ perspectives on the problem being addressed by the intervention; gaining stakeholders’ input to and shared understanding of the programme theory, including understanding of the context and system properties; identifying potential areas for change; choosing appropriate outcomes; and ensuring that all ‘voices’ are heard. It can also help with prioritising research questions, choosing the most useful evaluation perspective, building consensus and fostering relationships to make evaluation more straightforward, for example by accessing gatekeepers, supporting intervention refinement and considering issues around transferability and implementation. Where a systems perspective is taken, stakeholder engagement can help to define the system boundaries and, thus, the scope of the evaluation.

The risks of not engaging relevant stakeholders in complex intervention research include the intervention being theorised or considered from a narrow perspective only; the intervention not being readily implementable or supported by existing system context; the intervention being infeasible or unacceptable; and the chosen outcomes not being important to the patients, public, policy or practice. For example, outcomes of surgical interventions for severe and complex obesity depend on patient engagement with new lifestyle and diet, and work better with multidisciplinary care from psychologists and dieticians in addition to the traditional surgical teams. 88 The development and evaluation of a surgical intervention like this example, therefore, requires multidisciplinary stakeholder engagement, not least with the patients who may receive the intervention. Broad thinking is needed to identify a diverse range of appropriate stakeholders beyond the usual suspects.

Key uncertainties

A major challenge for complex intervention research is that, at any phase, there are multiple research questions that could be usefully addressed and alternative research perspectives that may seem equally plausible. Key to maximising the efficiency and value of research is to focus on addressing the key uncertainties that exist, taking into account what is already known and what the programme theory, research team and stakeholders identify as being the most important to find out. These judgements inform the framing of research questions that, in turn, govern the choice of research perspective. Even if the fullest possible assessment of an intervention depends on understanding its impact on the system in which it is embedded, that understanding may need to be developed through a series of studies that adopt a range of different perspectives, chosen at each phase to be the most relevant to addressing the key uncertainty at that phase. For example, a common approach that fits well with the linear stages of research that were proposed in the 2000 complex intervention framework,3 is to focus on establishing whether or not the intervention ‘works’ by obtaining an unbiased estimate of the average effect. This is undertaken initially through an efficacy perspective that is applied within tightly controlled conditions, and then through effectiveness research that is conducted in more realistic conditions. For interventions where the programme theory suggests that there are homogeneous effects across population subgroups, with a low degree of system and context dependence, efficacy and effectiveness are likely to be the key initial uncertainties, with implementation and transferability being the focus of later stages of research. However, for interventions that are designed to be flexible and adaptable or that interact with contexts and systems in a dynamic and variable way, the key initial uncertainties may relate less to efficacy and more to acceptability and compatibility with the wider systems in which they are embedded, with the assessment of impact on key outcomes taking place at a later evaluation phase.

At any given phase of research there may be more than one key uncertainty, which can be addressed through separate research questions and parallel paths of inquiry. For example, modelling the effectiveness and economic implications of scaling up an intervention may be worthwhile alongside a development or feasibility study, to reduce the risk of developing and testing an intervention that has little real prospect of being implemented in practice (Box 6).

Prioritising research questions and choosing a research perspective will be influenced both through the identification of key uncertainties and through the assessment of the likelihood of achieving an unequivocal answer to that question. There is a risk that research questions, research perspectives and research designs that are favoured by existing guidance and research funding mechanisms are those that have a greater probability of finding a certain answer, even though the question may be of less importance. One of the motivations for updating this framework was to answer calls for a change in research priorities, towards allocating greater effort and funding to research that can have the greatest impact on health-care and population health outcomes. This may require a shift away from, for example, efficacy trials of relatively uncomplicated interventions in tightly controlled conditions, for which research questions are answered with great certainty but for which the translation of this evidence into diverse, real-world settings is often highly problematic,119 requiring heroic assumptions about the potential for generalisation and implementation. In many areas of health intervention research, it may be more useful to give higher priority to an evaluation that is sensitive to complexity, from which improvements in understanding and important implications for decision-makers can be identified, albeit with caveats, assumptions and limitations. 7 Thus, theory-based or systems evaluations of complex population-level interventions and mixed-methods evaluations that emphasise implementation, context and system fit may provide more useful knowledge to address key uncertainties for decision-makers than evaluations that seek to provide unequivocal answers to tightly specified questions of less importance. For example, decision-makers may place greater value on research from a systems perspective that assesses an intervention in terms of its potential to contribute to long-term impacts on multiple outcomes, through multiple and often indirect pathways that will vary over place and time, than on research from an effectiveness perspective that measures short-term effects on a single prespecified outcome.

Identifying key uncertainties, priority research questions and optimal research perspectives will ideally involve the active participation of a multidisciplinary research team and a diversity of stakeholders, and ideally with independent oversight, perhaps from a funders’ steering group. Different stakeholders will have different views on what is the most important to find out. Some may not be in equipoise about the intervention’s effectiveness because they are satisfied with what is already known and may, therefore, be more focused on questions of reach, context dependence and issues of implementation; others may feel that greater certainty of effectiveness is required owing to limitations and potential bias in the existing knowledge base. Choice of outcome may also vary according to stakeholder perspective. For example, a local authority evaluating a suite of policies to increase physical activity may prioritise a natural experiment using routinely collected data on pre- and post-intervention use of leisure centres and green space in localities with and without the policy. Researchers may place higher priority on validated measures of physical activity in exposed and unexposed groups to be convinced of the effect of the intervention. An evaluability assessment would be a useful tool in this case to help identify the most worthwhile design and focus of evaluation (see Evaluability assessment).

Key uncertainties should be continually revisited as data accumulate and the programme theory and intervention are updated and refined. Formal methods, such as decision analysis, VOI analysis (see Box 6) or agent-based modelling (ABM) with uncertainty quantification, may be useful in prioritising research questions and optimising research design. 120–122 However, in many situations this will be largely a deliberative process underpinned by the purpose of the evaluation and the needs of research users. Traditionally, research funders and researchers have prioritised research questions and their associated designs and methods that have a good chance of being answered with a high degree of certainty and are strongly influenced by hierarchies of evidence. For many complex interventions, the needs of research users and the complexity of the evaluation may favour decision-making based on a greater diversity of evidence, thus valuing a wider range of research perspectives and methods, and on ‘balance of probabilities’ rather than ‘evidence beyond reasonable doubt’. By involving a wider range of stakeholders in the research team, a consideration of uncertainty in prioritising the next stage of research may lead to a greater focus on reducing uncertainty around important questions than on answering less important questions with certainty; the findings may be equivocal but will nevertheless be of greater value.

Intervention refinement

Within each phase of complex intervention research and on transition from one phase to another, there may be good reason, based on the data collected and/or the development of programme theory, to refine or optimise the intervention:

The process of ‘fine tuning’ or making changes to the intervention once a preliminary version (prototype) has been developed.

O’Cathain et al. 11

Intervention refinement (see Case study 9) in the development, feasibility and implementation phases can be relatively straightforward, provided that the changes are consistent with the intervention’s programme theory. Such refinements are covered by the argument that the form of an intervention can be flexible as long as the key functions specified in the programme theory are delivered. 40 However, where the changes are substantial and imply a significant change to the programme theory, such adaptation may require a more fundamental rethinking of the research process. This more fundamental kind of adaptation is distinct from refinement, in that it refers to an existing intervention being adapted to a new population, setting, target outcome or context, and is discussed in Developing an intervention.

Intervention refinement and flexibility are discouraged in an efficacy evaluation, in which standardisation and consistency of intervention delivery are prioritised to maximise internal validity. However, from a systems perspective, adaptability and responsiveness are likely to be desirable features of the intervention and a key focus of the research. The extent to which variation in programme delivery is desirable in an effectiveness or theory-based evaluation will depend on the extent to which a flexible form of delivery is allowed, while preserving the delivery of the intervention’s key functions. 40

In instances of an ‘identified intervention’ (i.e. one that is driven by policy or practice, rather than researcher led; see Identifying an intervention), limits on refinement may be set by the policy or practice context. Refinement of policy- or practice-led interventions will require negotiation with those co-ordinating the intervention and all other relevant stakeholders, but may be valuably informed by evaluation results.

Economic considerations

What are the comparative resource and outcome consequences of the interventions for those people and organisations affected?

Nearly all complex interventions will be costly to implement and will impose costs on a range of populations or organisations. Economic evaluation, the comparative analysis of alternative courses of action in terms of both costs (resource use) and consequences (outcomes and effects), should be a core component of all phases of intervention research. Early identification of the potential costs and outcomes can help with the setting of research questions and the design of an appropriate economic evaluation (see Case study 10). Methods for the economic evaluation of health interventions are well developed. 132 Here, we concentrate on aspects that are particularly important for complex interventions.

Early engagement of economic expertise will help identify which perspective to adopt to answer questions that matter most to decision-makers. 133 Recent National Institute for Health and Care Excellence (NICE) guidance134 recognises that complex interventions are likely to have costs and outcomes across different sectors, and recommends the adoption of a broad (e.g. societal) rather than a narrow (e.g. health service) perspective. Use of an impact inventory is a good way of making choices of perspective and of which costs and outcomes to include, explicit, transparent and comprehensive. 135 Another important initial consideration is what economic evaluation framework should be adopted. Again, the NICE guidance134 advocates broad cost–consequence analysis (CCA) or cost–benefit analysis (CBA) approaches because narrower approaches, such as cost-effectiveness analysis (CEA) or cost–utility analysis (CUA), are unlikely to reflect the full range of costs and consequences. These approaches and the choice between them are discussed more fully in Economic considerations.

Central to all forms of economic evaluation is the identification, measurement and valuation of the resources and outcomes of both the intervention being evaluated and the appropriate comparators. What to include may not be obvious to evaluators given the variety of resources required to deliver a complex intervention and the range of intended and unintended consequences for those directly and indirectly affected. 136 Engaging stakeholders (organisations and individuals involved in commissioning and delivering the intervention or expected to experience additional costs, savings, benefits or negative impacts) in the identification process is essential. Economic considerations are often very important to decision-makers because many of their decisions concern the optimal allocation of scarce resources. A decision-modelling framework may be useful to guide and synthesise complex intervention research.

This understanding of resources (costs) and outcomes, together with evidence from existing studies, such as economic evaluations of comparable interventions, should be incorporated in a programme theory. Programme theories can be used to identify the cost and outcome variables that should be taken forward into the economic evaluation. Interactions between an intervention and the context in which the intervention is delivered may lead to variation in costs and outcomes, underlining the importance of unpicking how causal pathways might work differently for different groups and incorporating this understanding in an explicit programme theory. 137 Use of programme theories in economic evaluation is considered further in Developing an intervention.

A further consideration in designing economic evaluations of complex interventions is the trade-off between equity and efficiency; this is important where differential impacts are likely. The costs and the benefits of complex interventions may vary across groups in the population, such that the most efficient interventions in terms of population health gain may not be the most equitable in terms of reducing (or at least not increasing) health inequalities. Considering who the beneficiaries might be and how costs and impacts might differ between them is a crucial part of planning an economic evaluation of a complex intervention, and methods have been developed for conducting economic evaluation in ways that take account of trade-offs between equity and efficiency. 138,139

Economic considerations that should be taken into account at particular phases in the process of developing and evaluating a complex intervention are addressed in the relevant sections below. Ways in which economic evaluation of a complex intervention might differ from standard approaches are identified and new methods highlighted.

Developing an intervention

The MRC-/NIHR-funded INDEX study has developed comprehensive guidance on intervention development. 11 Rather than replicate the INDEX guidance here, we provide an overview of the key steps and encourage readers to access the main INDEX document (see Signposts to further reading 8: developing or identifying an intervention).

The INDEX guidance recommends that a robust process of intervention development should involve the 11 steps outlined in Table 3.

In practice, intervention development approaches will need to be tailored to the capacity of the team, context and resources. Some of the steps in Table 3 may overlap or need to be repeated. It may not be possible or necessary to address all of them, but each should be considered for relevance and importance. Teams involved in developing interventions are encouraged to publish their development work so that learning can be shared.

Four additional sets of issues that may need to be addressed are (1) the adaptation of existing interventions to a new context; (2) wider system factors that may need to be taken into account; (3) the consideration of economic factors; and (4) mathematical modelling or simulation of the intervention.

Identifying an intervention

Often, interventions are policy-driven and/or designed in clinical practice. In such cases, researchers are not usually involved in intervention development, but evidence that they have generated may have informed this process. In some instances, implementation of policy- or practice-led intervention may have taken place without a prior evaluation, in which case researchers may need to plan a retrospective evaluation.

Even if researchers are not involved in the design phase, they should still consider the six core elements (Table 4). In particular, they should develop a programme theory and use this to identify key uncertainties. Some policy interventions are based on explicit theory, but many are not. In these cases, identifying the implicit theoretical basis of an intervention is essential to working out how it might be evaluated. This is an important step, even if the programme theory is developed after the intervention has been rolled out. This helps identify mechanisms, important contextual factors and relevant outcome measures.

Why is a feasibility phase needed?

The feasibility phase explores the feasibility of the intervention and/or the evaluation design to underpin decisions about whether or not and how to progress to evaluation. Although feasibility studies are most commonly undertaken prior to effectiveness evaluation, they can also be necessary prior to an efficacy, a theory-based or a systems perspective evaluation study. This vital preparatory work is sometimes overlooked or rushed,163 which leads to evaluation studies that are undermined by problems related to either the intervention or the evaluation design that could have been identified and overcome through appropriate feasibility research. On the other hand, feasibility studies can sometimes be overspecified and take more time and resources than are necessary to address the key uncertainties with the planned intervention or evaluation design.

A feasibility study is designed to explore the uncertainties that have been identified at the development phase (see Case study 12). The aim is then to address these uncertainties to ascertain whether or not it is appropriate to move onto the evaluation phase and optimise study design.

Progression criteria

A feasibility study should be designed to assess progression criteria that relate to the feasibility and acceptability of the intervention and/or the evaluation design. Progression criteria should ideally be developed with input from an independent steering committee and relevant stakeholders, and be agreed and signed off prior to the collection of data.

Progression criteria should be used to guide the decision on whether to proceed to the next stage of evaluation or to undertake further feasibility work, or to return to the development phase and/or terminate the research. Assessment of progression criteria may require both qualitative and quantitative data. Examples include (1) qualitative data, such as the acceptability of the intervention content and mode of delivery for participants and providers, and (2) quantitative data, such as recruitment and retention rates. See the HelpMeDoIt! study for an example of progression criteria. 164

Progression criteria should be worded in a way that provides scope for considered discussion among the study team and stakeholders, ideally with an independent steering committee making the final recommendation. The use of a traffic light system for each progression criterion is a useful approach (green = go, where there are no issues identified; red = stop, where issues identified cannot be resolved; and amber = amend, where issues identified can potentially be resolved).

In many cases, the key uncertainties to be addressed prior to the evaluation phase relate solely to aspects of evaluation design. This may be where the development phase has involved substantial empirical assessment of the intervention, where the intervention is not in the control of the researcher and so cannot be changed, or where the intervention has already been tested for efficacy. In these cases, questions of feasibility may relate to evaluation design only, given that there may be no reason to expect that further changes will need to be made to the intervention to successfully conduct the evaluation. If this is the case, feasibility work may take the form of a pilot study: a small-scale version of some or all of the future evaluation study. To expedite the research process, it may be advantageous to commission a full evaluation in which an initial pilot stage is included, rather than a separate feasibility phase incorporating a pilot study. All being well, the pilot stage could then be continued and expanded into a full evaluation, eliminating a substantial delay between the pilot and the evaluation. If there were problems that needed to be addressed in the pilot, such as problems with recruitment, retention or measurement of study participants, the pilot would be stopped and the research design or methods reviewed.

Feasibility of the evaluation design

In terms of evaluation design, a feasibility study should assess key aspects of research design and conduct, which typically include:

• recruitment – time frame, willingness of participants to be randomised, capacity of those involved (e.g. clinicians) to recruit participants and effective strategies

• retention – follow-up rates and reasons for attrition

• sample size – capacity to achieve an appropriate sample size

• outcomes – choice of outcome measures, data collection tools and processes (e.g. missing data on questionnaires or accessibility of routine clinical data), and duration of follow-ups

• analysis – capacity to collect and analyse data within a time frame, and floor or ceiling effects

• unintended outcomes – for example, potential harms or the impact of the intervention on inequalities.

Feasibility of the intervention design

Feasibility studies provide an important opportunity to implement the intervention on a small scale and assess any uncertainties regarding the intervention prior to embarking on a full-scale evaluation. Where prior stages of research have not tested the implementation of the intervention in the same kind of circumstances that will exist in the evaluation study, the feasibility phase is the first and an important opportunity to address questions related to optimal intervention content and mode of delivery; the acceptability of the intervention to participants and providers; adherence; the likelihood of cost-effectiveness; and the capacity of providers to deliver the intervention and fit within the setting. Where the intervention programme theory suggests that contextual or implementation factors may influence the effectiveness of the intervention, these questions should be addressed at the feasibility phase, even if the intervention has already been tested for efficacy.

Many intervention-related questions that might be assessed in a feasibility study relate to process evaluation, which aims to understand the functioning of an intervention by examining implementation, mechanisms of impact and contextual factors. 5,165

Depending on the results of the feasibility study, further work may be required to progressively refine the intervention before embarking on a full-scale evaluation. As discussed in Intervention refinement, any refinement should have agreed limits and be reported transparently. Ideally, incremental refinements should each be tested separately; however, in practice refinements can be made simultaneously if sufficiently rich data are collected to enable judgements about which refinements are helpful and which are not.

The programme theory should be refined in an ongoing way regularly throughout the feasibility study and indeed all phases of the research process.

Efficacy signals

To justify the expense of a full-scale evaluation from an effectiveness, theory of change or systems perspective, as well as evidence of feasibility it is also helpful to have convincing empirical or theoretical evidence to suggest that the intervention has the potential to be effective. Such evidence may come from prior research conducted from an efficacy perspective that has demonstrated that the intervention can work in ideal, controlled conditions. Such ‘efficacy signals’ or evidence of potential effectiveness may also be drawn from evidence of the effectiveness of similar interventions for which the programme theory suggests that the effects may be transferable to different populations, settings or outcomes. Feasibility studies should generally not aim to identify efficacy signals or estimate effect sizes for use in sample size calculations, given that these will be very imprecise. However, in some cases, a feasibility study may provide an opportunity to collect data to support the case for potential effectiveness, for example evidence of short-term impacts on proximal, sensitive and intermediate outcomes.

Economic considerations for the feasibility phase

The aim of an economic evaluation conducted within a feasibility study is to explore the likelihood that the intervention will be worthwhile, identify key areas of uncertainty and understand whether or not the data required for a full economic evaluation can be captured. Economic considerations that should be addressed during the feasibility phase include refinement of the programme theory that will underpin the full economic evaluation, development and testing of primary data collection tools, and identification of relevant sources of routinely collected resource use and outcome data. Economic modelling can be undertaken to assess the likelihood that the expected benefits of the intervention justify the costs and to help decision-makers decide whether or not it is worthwhile to proceed to a full-scale evaluation. A VOI study (see Developing an intervention) should be considered if one was not included in the intervention development phase. If a VOI study was included then it should be updated at this stage. It should be noted that such analyses may increase rather than reduce uncertainty by identifying new sources of variation in costs and outcomes. This can be helpful when a policy-maker believes that an intervention is cost-effective, despite a lack of evidence.

Case study 10 demonstrates how economic considerations at the design and feasibility phases were used to build the case for a full-scale trial of a complex intervention.

Evaluability assessment

An alternative or complementary approach to feasibility is evaluability assessment, a systematic, collaborative approach that is often used with policy-driven or practice-driven (identified) interventions that can equally be used to plan the development and evaluation of a researcher-developed intervention. It aims to determine whether or not an intervention can usefully be evaluated and, if so, which methods are the most appropriate. Its purpose is to ensure that evaluation resources are used efficiently and to minimise research waste on evaluations of poorly designed and/or implemented interventions, or those for which the available data or the method of implementation do not permit outcome evaluation. Evaluability assessment involves collaboration with stakeholders to reach agreement on what are the expected outcomes of the intervention, what data could be collected to assess processes and outcomes and what are the options for designing the evaluation. 75 The end result is a recommendation on whether or not an evaluation is feasible, should be undertaken and can be carried out at a reasonable cost, and by which methods this can be undertaken. 166 Five questions to assess evaluability have been suggested by Ogilvie et al. :166

1. Where is a particular intervention situated in the evolutionary flow chart of an overall intervention programme?

2. How will an evaluative study of this intervention affect policy decisions?

3. What are the plausible sizes and distribution of the intervention’s hypothesised impacts?

4. How will the findings of an evaluative study add value to the existing scientific evidence?

5. Is it practicable to evaluate the intervention in the time available?

Case study 13 provides an example of using evaluability assessment of existing community interventions to identify practice-based evidence. Table 5 summarises the core elements for the feasibility phase.

Purpose of evaluation

A key change in the field of evaluation has been a shift towards assessing the ‘usefulness’ of information for decision-making, in contrast to focusing exclusively on obtaining unbiased estimates of effectiveness. 172 In a trial, randomisation, coupled with other elements of trial design (such as concealment of the allocation sequence and blinding of participants and researchers), can minimise bias. A statistical model of the process that determines exposure to the intervention serves the same purpose in a natural experimental study (see Natural experimental designs); however, this may not be as effective as randomisation at creating balanced (or ‘exchangeable’) intervention and control groups. However, to maximise the usefulness of the evidence for decision-making, other elements are required, such as the use of programme theory and consultation with stakeholders to identify research questions that matter and a process evaluation to assess elements of implementation, context and mechanisms of impact. 5 We emphasise that here we are advocating for additions to the toolkit of methods, rather than replacements for traditional methods, when they are appropriate. A more fundamental challenge to the conventional idea that precise and unbiased estimates of effectiveness are the principal goal of evaluation is the proposal that improving theories about how to intervene should be the ambition. In this view, effect estimates are inherently context bound, so that average effects are not a useful guide to decision-makers working in a particular context. More useful may be contextualised understandings of how an intervention works and details on the most important enablers and constraints on its delivery across a range of settings.

Often, questions about the delivery, reach and maintenance of interventions are assigned to ‘implementation research’ and are explored separately from evaluation questions. Some frameworks, such as RE-AIM,158 try to pull them together. A key recommendation here is that implementation questions should be considered alongside evaluation questions from the outset. Moving away from a narrow ‘effectiveness’ focus should increase the relevance of the evidence produced by evaluation studies and increase the speed with which interventions that make a useful contribution can be implemented within policy and practice. This is about explicitly considering from the outset how the research will make a difference, who will use it and who could benefit from it. For example, who needs to know the answer to the research questions? One component of a research proposal should be a ‘Pathways to Impact’ statement or an implementation plan; these are often required by funders,173 but even if not it is a good idea to have given it due consideration.

Research perspectives and evaluation

Table 6 sets out the main research perspectives used in evaluation and relates them to study designs: (1) efficacy, (2) effectiveness, (3) theory based and (4) systems. Although some designs and perspectives tend to be closely associated with one another, the mapping is not fixed and designs can be combined to answer a range of research questions about any given intervention. Efficacy and effectiveness evaluations do not represent a dichotomy but a continuum,39 and it is ‘likely impossible to perform a pure efficacy study or pure effectiveness study’. 189 An effectiveness evaluation can draw on a theory-based perspective to help to explore the impact of context and mechanisms on the outcomes being assessed. For example, a theory-informed process evaluation was conducted alongside an effectiveness trial of the 3D intervention190 (to improve the management of multimorbidity in primary care, see Case study 14) to understand how and why the intervention was effective or not, as well as to understand the effect of context. An effectiveness evaluation can also draw on a systems perspective to help determine which outcomes should be evaluated. For example, in determining an evaluation strategy to explore the impacts of the SDIL, researchers drew on a systems perspective and methods (see Case study 7). 191 Theory-based and systems perspectives to evaluation can also be used in conjunction; for example, methods developed using systems thinking could be used to support and focus the involvement of stakeholders in a realist evaluation. 82

Links to case studies are included in Table 6 to highlight how different study designs may be used to answer different types of questions. A short description of a selection of these methods is given in Types of evaluation design and methods; however, before deciding on study design, the range of outcomes that an evaluation may include should be considered, which is described in the next section.

Choice of evaluation outcomes

A crucial aspect of the design of an evaluation is the choice of outcome measures or evidence of change. Evaluators need to assess which outcomes are the most important for stakeholders (e.g. patients, public and decision-makers) and how to deal with multiple outcomes in the analysis. A sharp distinction between a single primary outcome and a number of secondary outcomes is not necessarily appropriate, particularly where the programme theory identifies impacts across a range of domains. A good theoretical understanding of the intervention, derived from careful development work, is key to choosing suitable outcome measures. Rather than always choosing one health outcome and judging the effectiveness based on that only, effects on intermediate outcomes, such as health behaviour, health care or a determinant of health or a range of outcomes, should be explored and theoretically justified. These outcomes should be identified prospectively during the process of developing the programme theory and be incorporated alongside an a priori analysis plan in a published study protocol and reported comprehensively in the final evaluation. This approach allows a more broad-based judgement on effectiveness to be made.

Inconsistencies in measured outcomes makes synthesis of evidence more difficult; therefore, the use of core outcome sets is recommended where appropriate. 192 The Core Outcome Measures in Effectiveness Trials (COMET) initiative supports the development of core outcome sets, largely for clinical trials, although includes some resources that may be more widely applicable. 193

Outcome measures may capture changes to a system rather than changes in individuals. Examples of system-level outcomes include changes in relationships within an organisation, the introduction of policies or new ways of working and changes in social norms or normalisation of practice. They also include how changing the dynamics of one part of a system alters behaviours in other parts, such as the displacement of smoking into the home after a public smoking ban. It is possible to use traditional evaluation designs to answer questions relevant to a systems perspective (see Case study 19). It is unlikely that all aspects of complexity can be investigated in an evaluation;62 however, a broad awareness of the system should be considered at the development phase and system boundaries determined, with consideration of the usefulness of evidence that will be generated.

Pre-identified sources of variation in outcomes are important, and prespecified subgroup analyses should be carried out and reported. In the case of interventions that are expected to affect inequalities in health, analyses stratified by socioeconomic position, gender, ethnicity or geography may be appropriate. Even where such analyses are underpowered they should be included in the protocol and reported, as they may be useful as input into subsequent meta-analyses or for developing hypotheses for testing in further research.

A priori sample size calculations are made to ensure that there is enough statistical power to detect a meaningful target difference in the outcome between the intervention and the comparison groups. This calculation is often based on a single primary outcome, although different candidate outcomes can be considered in turn. The final decision for sample size will depend on practicality and usefulness of evidence, and ideally will be sufficient to detect a meaningful change in all specified candidate outcomes. The Difference ELicitation in TriAls (DELTA2) guidance194 provides guidance on choosing target differences in outcomes and on associated sample size calculations.

Another outcome of evaluation is refined or developed theory. Improved theory helps us to understand the possibility of transferability across settings and to produce evidence and understanding that is more applicable to informing future decisions by policy-makers. Theory is used to inform evaluations, but the purpose of evaluation should also be the gradual incremental testing and refinement of theory. Interventions are likely to require refinement or adaptation to be successful in different contexts, whereas the theory produced from the study can be generalisable. It is important that results from experimental trials are not overinterpreted; unbiased estimates may not provide evidence suited to the application of the intervention in different contexts. 172 Decision-makers operating in a context different from that of the research may be able to learn more from the theory that is developed from the intervention than from an effect estimate with little information on context or transferability.

Types of evaluation design and methods

There are many study designs to choose from and different designs are each optimally suited to different research questions and different circumstances. 195 As we have emphasised, there is no one-to-one mapping of research perspectives to evaluation designs. A purely quantitative approach using an experimental design with no additional elements, such as a process evaluation, is rarely adequate. This is particularly true in complexity-informed research using a theory-based or systems perspective; in such cases, qualitative and mixed-methods designs are essential for evaluation to answer questions beyond effectiveness.

Studies that use both qualitative and quantitative methods often analyse the data for each method separately and then consider them together in discussion of the findings. There are techniques for integrating qualitative and quantitative data and findings;196–198 however, this is not common practice in the context of RCTs for health interventions, and effort is ongoing to provide future guidance (see Richards et al. 199 for some examples).

A design may rarely be used in a particular field, but that does not mean it cannot be used. You should make your choice on the basis of the specific needs of your evaluation, in particular the research questions that you aim to answer. Awareness of the range of experimental and non-experimental approaches should lead to more appropriate methodological choices. In the following sections we elaborate on some of the evaluation designs from the case studies referred to in Table 6 and in Case studies. We also include social network analysis and modelling, which are not designs as such but provide an overview of further methods that can be used to evaluate complex interventions. We focus on experimental and theory-based designs that have come to the fore since the previous version of this document1 and the natural experiments guidance200 were published. The examples that we give of systems methodologies are those most commonly discussed in relation to systems methods for evaluation of interventions within health research, not necessarily an exhaustive list of systems methodologies.

Understanding processes

Process evaluation is a critically important component of the evaluation of complex interventions and is an essential adjunct to the study designs presented so far (see the case study examples set out in Table 6). Process evaluations can use qualitative and/or quantitative process data to answer questions around fidelity and quality of implementation (e.g. what is implemented and how?), mechanisms of change (e.g. how does the delivered intervention produce change?) and context (e.g. how does context affect implementation and outcomes?). 165

Within an efficacy study, a process evaluation would have relatively narrow questions. The key idea of efficacy studies is that there is a test of causal mechanism, often through mediation analysis. Such analyses are usually quantitative, but could use qualitative methods to assess how research participants understood the intervention and made use of intervention components. Because the focus of efficacy studies is to maximise internal validity, process analyses could usefully examine fidelity to assess whether or not the intervention was delivered with high fidelity to the theorised mechanism of action. Consideration of context would not be the focus of a process analysis from an efficacy perspective.

Process evaluations in an effectiveness perspective would be used to inform findings, for example to provide insight into why an intervention fails unexpectedly or has unanticipated consequences, or why a successful intervention works and how it can be optimised. This includes exploring the fidelity and quality of the intervention, clarifying causal mechanisms and identifying contextual factors associated with variation in outcomes (see Case study 14). Details on the type of measures to consider are provided in MRC guidance on process evaluation of complex interventions. 5

Process evaluations are key components of theory-based and systems evaluations, to the extent that they seek to answer research questions about how the intervention produces change in a generalisable way, taking the study further than assessing solely whether or not it works in a specific setting. An emerging approach is to conduct a realist process evaluation, specifically drawing on realist evaluation. 48 If undertaken alongside an effectiveness evaluation, a realist process evaluation can help to understand the effects of intervention components rather than only the intervention as a whole; examine mechanisms of change; explore the effect of intervention context; collect and analyse qualitative and quantitative data; and aim to build and validate intervention programme theories. 235 There are also increasing examples of taking a systems approach to process evaluations, with a framework developed by McGill et al. 236 offering some guidance.

Process evaluations should be conducted to the same high methodological standards and reported as thoroughly as evaluation of outcomes. The MRC guidance on process evaluation for complex interventions5 provides further information on how to conduct a process evaluation, including consideration of relevant theoretical approaches.

Economic considerations

As noted in Economic considerations, there are a range of economic evaluation frameworks (Box 7).

Each framework has different strengths and limitations. CEA measures outcomes in natural units that relate to, but do not measure directly, the health outcomes of interest; for example, reduced blood pressure is associated with reduced incidence of stroke, but it does not directly measure the incidence of stroke or capture the associated range of health improvements. This limits the usefulness of an incremental cost-effectiveness ratio when evaluating complex interventions with multiple impacts. CUA uses measures, such as quality-adjusted life-years (QALYs), to compare interventions with a range of physical and mental health outcomes in terms of their impact on length and quality of life. However, complex interventions often have outcomes that are difficult to capture fully using QALYs. Ascribing monetary values to all of the dimensions of benefit in a CBA is methodologically challenging. CCA is more straightforward and can be used to present evidence on a wide range of costs and outcomes; however, without valuation of these outcomes (e.g. preference weighting in QALYs or monetary valuation in CBA) decision-making requires judgement. Rare but costly outcomes may acquire too much influence if their frequency is not taken into account. Furthermore, if some outcomes improve more under some interventions and other outcomes fare better under others, the policy implications of a CCA may be ambiguous. The importance or ‘weight’ of differing consequences is likely to vary across stakeholders. A separate exercise, such as a discrete choice experiment (see Case study 26), could add additional evidence to the value of different consequences to different stakeholders.

Given that complex interventions typically have effects across a wide and diverse range of outcomes, recent guidance emphasises the use of a broad framework, such as CCA or CBA. 134 CCA has the intuitive appeal of providing data across a range of ‘consequences’ (outcomes) with which decision-makers are likely to be concerned when making a policy decision. This allows the analyst to adopt a broad public sector or societal perspective that is appropriate to many complex interventions. It is important to note, however, that the different frameworks are not mutually exclusive: CBA and CEA can be carried out and embedded in a wider CCA that provides data on a more comprehensive inventory of outcomes. The CCA can be useful for identifying the range of impacts likely to be of interest to researchers and policy-makers prior to deciding which outcomes will be measured and how. For this reason, it can be a helpful first step towards a CBA.

In addition, there are a number of emerging methodologies that can lend themselves to economic evaluations of complex interventions. They include Social Return on Investment237 and multi-criteria decision analysis (MCDA). 238 These are not usually considered to be standard tools for economic evaluation owing to theoretical weaknesses and their handling of outcomes, but they are becoming more commonly used in the area of complex interventions. MCDA is particularly helpful following a CCA because it involves a structured, explicit approach to decisions involving multiple criteria. 238

Decision analytic modelling120,239 is a useful way of combining data from an economic evaluation with data from other sources to extrapolate costs and outcomes beyond the study time frame. The use of advanced modelling techniques, such as microsimulation modelling, network analysis, discrete event simulation and ABM,153,240,241 is particularly useful for complex and public health interventions, although simpler model types can also be appropriate. 242,243

Reporting evaluations

Evaluations should always be reported in accordance with established guidelines because this will help to ensure that the key information is available for replication studies, evidence syntheses and guideline development. The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network maintains a comprehensive database of reporting guidelines [www.equator-network.org/ (accessed 2 March 2021)].

Relevant guidelines for reporting are as follows:

• The CONSORT statement is a well-established standard for reporting randomised trials. Relevant extensions are published for –

  • social and psychological interventions (CONSORT-SPI 2018)244

  • randomised pilot and feasibility trials163

  • cluster RCTs245

  • pragmatic trials246

  • n-of-1 trials247

  • cluster stepped-wedge trials248

  • non-pharmacological interventions249

  • patient-reported outcomes (CONSORT PRO)250

  • adaptive designs251

  • and the SPIRIT-PRO extension for patient-reported outcomes. 252

• Reporting of statistical analysis plans. 253

• Reporting process. 254

• RAMESES II for realist evaluation. 255

• SPIRIT 2013 statement provides evidence-based recommendations for the minimum content of a clinical trial protocol. 256

• Standards for reporting qualitative research. 257

• STrengthening the Reporting of OBservational studies in Epidemiology (STROBE). 258

• Consolidated Health Economic Evaluation Reporting Standards (CHEERS). 170

Table 7 provides a summary of the consideration of each of the core elements at the evaluation phase.

Implementation science

Within health care, implementation science is an important and developing field of research that undertakes studies that develop and evaluate strategies for implementing effective interventions that address local realities, can be implemented at scale and are potentially sustainable. 269 Implementation science has been defined as ‘the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services’;270 however, a wider definition would not be restricted to health care or to the limited range of designs and methods typically used in that context. With this wider definition, implementation science, therefore, includes interventions and strategies to increase the uptake and impact of research in policy, including knowledge mobilisation across a range of potentially relevant organisations, sectors and professions, as exemplified by the range of case studies of implementation research in a global health context. 271

A large part of implementation science research, therefore, involves the development and evaluation of complex interventions to maximise effective implementation in practice and/or the policy of interventions that have already demonstrated efficacy or effectiveness. The whole of this framework, therefore, applies to those implementation studies just as it does to other forms of intervention. The main specific characteristic of such implementation studies is that the main outcome measures used will not be health outcomes but measures of implementation effectiveness, such as practitioner or patient behaviour, uptake of services, reach and policy impact. For example, a cluster randomised trial of strategies to improve nurses’ hand-washing behaviour used hand-hygiene compliance and adherence as outcome measures rather than infection rates, with a nested process evaluation identifying which components of the implementation strategy and context particularly supported compliance. 272 The field of implementation science has many theories and frameworks that can importantly inform the development of implementation interventions and associated programmes (see Signposts to further reading 11: implementation). There are also specific standards for reporting implementation studies and other specific considerations for implementation-focused intervention research that complement this framework. 273

Implementation in all phases

A key motivation for this update was to maximise the usefulness of complex intervention evidence for decision-makers. For the evidence to be useful, it is important that interventions are developed with implementation considerations in the foreground at all phases (see Case study 27). A key part of this is involving stakeholders and being mindful of the realities of the intervention context at all phases/from the beginning and throughout. Early consideration of implementation increases the potential to develop an intervention that can be widely adopted and maintained in real-world settings. 245 For example, this may encourage the development of ‘very brief interventions’ that fit with routine practice and are effective and cost-effective when implemented at scale. 274 Such interventions may have more impact than very ambitious researcher-led interventions that demonstrate effectiveness and cost-effectiveness in a research setting, but are not ‘implementable’ for a variety of reasons, such as lack of fit to health-care practitioners’ usual practice. 130 The consequence of neglect is that issues, such as acceptability to the public or to those who deliver the intervention, may undermine implementation later on. 275 Other benefits of including stakeholders in all phases are to gain better understanding of the time and resource constraints of real-world evidence users and to engage with, understand and influence organisational, professional and social systems that may be resistant to change. 276–279

As well as anticipating implementation factors in intervention development and design, research questions in the feasibility and evaluation phases can help to understand and inform implementation. The intervention programme theory and process and outcome data collected in these phases can help inform dissemination of the findings and preparation of an implementation manual or plan to maximise the effective roll-out of the intervention in real-world settings. In addition to the publication of findings, sufficient details of the intervention and its underlying theory should be reported. 273 This improves the synthesis of evidence that may be more useful to decision-makers (e.g. theoretical outputs compared with effect sizes) and improves their understanding of the potential transferability of the intervention to their specific context (see The role of context in implementation).

Details of the intervention and key information related to effective delivery identified in feasibility and evaluation phases need to be communicated clearly in dissemination via methods that are accessible and convincing to implementers. 280 These may include resource requirements for prerequisites and delivery (e.g. equipment, staff training, skills and time); barriers to and facilitators of effective implementation; and potential flexibility in implementation while maintaining programme fidelity. This includes making recommendations as specific as possible. 281

Implementation and evaluation phases combined

In a natural experimental study, evaluation takes place within the implementation of the intervention in a real-world context. It may be possible to ascertain both the principal health outcomes during implementation to demonstrate effectiveness and/or the more proximal or process measures available through readily available data sources collected as part of the implementation. This would allow the assessment of which implementation or contextual factors support, amplify or undermine the uptake, (cost-) effectiveness or reach of the intervention.

In the case of highly pragmatic effectiveness trials or specific hybrid effectiveness–implementation designs, effectiveness and implementation outcomes can be combined in one study, with the aim of reducing time for translation of research on effectiveness into routine practice. Different hybrid designs have been proposed based on their focus on effectiveness versus implementation: conducting an effectiveness study while exploring and gathering data on implementation; dual focus on effectiveness and implementation; and conducting a study with a primary focus on implementation, while gathering data on effectiveness. 282,283 Green et al. 284 describe three studies to show how these approaches differ in practice.

The role of context in implementation

Understanding interdependence with the context of a given intervention is critical to the success or failure of implementation. Developing an understanding of context dependence throughout the development, feasibility and evaluation phases can, therefore, importantly inform what prerequisites need to be in place for the effects identified in the evaluation phase to be successfully reproduced in implementation.

Understanding context is also central to the transferability of an intervention into different contexts: an important aspect of long-term implementation. Developing and refining programme theory and considering wide-ranging contextual factors can help to identify issues related to transferability, for example that the intervention is unlikely to be effective when transferred to a given context. 285 Stakeholder involvement is then a helpful process to identify potential solutions to overcome identified issues. The ‘Context and Implementation in Complex Interventions’ framework that integrates context, implementation and setting can be used to guide discussions between researchers and stakeholders on the transferability of interventions. 70

Economic considerations of implementation

Economic issues at the implementation phase have not traditionally formed part of the guidance on economic evaluation, but they can and should be considered (1) at the early stages of intervention and study development and (2) in the reporting of economic results.

How the results of economic analyses are reported and presented to decision-makers can affect whether or not and how they act on the results. A key issue is avoiding and anticipating misconceptions in reporting the results of economic analyses. Interventions described as ‘cost saving’ may well be potentially cost saving, but savings may be difficult to realise in financial terms during implementation if there is an element of fixed costs or if the reduced demand for a service withdrawn on economic grounds is simply replaced with demand for services to meet previously unmet needs. The ‘cost saving’ may refer not to a ‘financial’ saving but to a release of resources for other potential uses, such as a freeing up of beds or the shifting of resource use from the NHS to another sector, such as social services (e.g. palliative care). 286

Another important issue that is relevant for implementation is how to deal explicitly with multisectoral issues. The payers of costs and the receivers of benefits of a multisector intervention may not be the same organisation, which can reduce the incentive to implement an intervention even if it is shown to be beneficial relative to its cost. Early identification and engagement with appropriate stakeholders and decision-makers to identify such cross-sectoral impacts will help frame appropriate research questions and pre-empt any implementation challenges that might arise. Recent work in this area has suggested methods, such as co-financing between and across sectors,261 to deal with these issues.

Studies to assess the cost of implementation can also be undertaken post evaluation to help aid the implementation process. This could be particularly important where an intervention is proven to be effective and cost-effective, yet implementation requires additional investment.

Other considerations specific to the implementation phase

At the implementation phase, observational or experimental studies are often useful even though there may not be any specific implementation strategies being assessed, as would be the case in implementation science studies (see Implementation science) or hybrid implementation/effectiveness designs (see Implementation and evaluation phases combined). Questions that can be assessed at the implementation stage include assessment of intervention receipt, ‘dose’ received, programme completion and the characteristics of those receiving the intervention and any inequalities in reach or completion. Similarly, barriers to and facilitators of effective implementation, including contextual, system or resource constraints, can be investigated. A key issue in understanding implementation is the extent to which the intervention has been delivered with fidelity,287 so that problems with implementation can be distinguished from a lack of effectiveness. Where an intervention is delivered with low fidelity, it is difficult to determine whether or not a lack of impact in implementation reflects an intervention that is genuinely ineffective, ineffective in that context or potentially effective but not observable in this instance owing to low-fidelity implementation. On the other hand, some flexibility in intervention implementation may support intervention effectiveness across contexts, provided that the key functions of the programme are maintained. 40 Where adaptations have been made to an intervention in a particular context, it is important to understand exactly what has been adapted. 145 Further detail on implementation related to fidelity can be found in the MRC’s guidance on process evaluation of complex interventions. 5

Few evaluation studies are powered to detect rare events, while unanticipated or adverse outcomes and potentially important longer-term system effects of interventions, such as spill over or diffusion of effects through networks, may also not be feasible to measure in an evaluation study within a specific sample and time frame. These can potentially be measured within the implementation phase, as they may have differential reach or effectiveness across population subgroups. 288 Effects are likely to be smaller and more variable once the intervention becomes implemented more widely, and unintended consequences may begin to emerge. 76,136 Long-term follow-up may be required to determine whether or not short-term changes persist, to determine whether or not benefits demonstrated from the original study do in fact occur and to capture any likely cost impacts (e.g. cost savings). Although long-term follow-up of complex interventions is uncommon, such studies can be highly informative. It is worth thinking about how to measure rare or long-term impacts, for example through routine data sources and record linkage or by recontacting study participants. Plans for the collection of appropriate outcome data and obtaining appropriate consents should be built into the study design at the outset. Implementation phase studies may involve randomised or non-randomised control groups or can be undertaken solely among those who receive the intervention.

Table 8 provides a summary of the considerations of the core elements at the implementation phase.

Key points

This efficacy trial can be differentiated from an effectiveness trial based on the following characteristics259 –

• The inclusion criteria for the trial were narrow. The main group of interest were those who were at high risk of stroke and those who were most likely to respond to the intervention. The exclusion criteria were wide-ranging; therefore, the intervention was tested on a very narrow group of patients.

• The intervention delivery was tightly constrained with little flexibility, and those delivering the intervention were highly trained. The surgeons taking part had to be approved by an expert panel and had to have completed at least 50 carotid endarterectomies in the last 24 months.

• The control intervention was not treatment as usual but included antiplatelet therapy and the same co-interventions applied to the intervention arm were also given to control patients.

• Follow-up was very intensive at multiple time points.

• Practitioner adherence to the study protocol was monitored.

• In real-world practice, much of the above would not happen; therefore, the impact of the intervention is likely to be different in a more pragmatic setting.