A 10-year impact assessment of the Efficacy and Mechanism Evaluation (EME) programme: an independent mixed-method evaluation study

Assessment of EME project applications

Applications to the EME programme may be submitted in response to researcher-led open calls or in response to EME commissioned calls that target specific health or technology areas. EME commissioned calls can be specific to the EME programme and are intended to fill gaps in health research that are identified by horizon scanning and/or communication campaigns (e.g. webinars, partnering events and site visits). EME commissioned calls can also be part of broader ‘themed calls’ across all NIHR programmes, which respond to the Chief Medical Officer’s priorities. In addition, applications made to other MRC and NIHR schemes can be transferred to the EME programme for consideration if they are better suited to the EME programme’s remit.

When applications are received, a subgroup of the EME Funding Committee conducts an internal remit and competitiveness check. Any applications that are out of remit or non-competitive do not progress any further.

For most applications, a two-stage review process is applied. Stage 1 (outline) applications are reviewed by the full EME Funding Committee. If shortlisted, applicants are invited to submit a stage 2 (full) application. Full applications are assessed by external peer reviewers and are considered at the following EME Funding Committee meeting. In some circumstances, applicants at either stage 1 or stage 2 are invited to resubmit a modified version of their application if the committee feels that the application has potential but is not yet of sufficient quality to progress in the funding process. The same proposal may undergo multiple rounds of feedback and resubmissions, termed ‘iteration of proposals’.

If an accelerated review timescale is of significant benefit or essential for the proposed research, the EME programme employs a fast-track scheme. This allows applicants who are approved for this process to bypass the two-stage review and submit a full proposal directly. A one-stage application process is also employed for ‘mechanisms of action’ commissioned calls for mechanistic study proposals. In these cases, applicants first complete an eligibility form and discuss the proposed research with an EME consultant advisor to check whether or not it is likely to be in remit. If this is the case, applicants prepare a full stage 2 application.

Following review of the full proposals, the EME Funding Committee makes funding recommendations to the Department of Health and Social Care (DHSC) and the MRC based on the general NIHR assessment criteria of (1) need for evidence, (2) value for money and (3) scientific rigour. 4

EME programme governance and implementation

The EME programme is led by the EME programme director, who calls on the expertise, knowledge and opinion of a large number of people to ensure that research areas are identified and funding decisions made in a clear and fair way. The programme director chairs the EME Strategy Advisory Committee, which provides advice to the programme director on the scientific strategy for the programme. The EME Strategy Advisory Committee currently has 13 members, with representation from the funders and the academic, clinical and patient and public involvement (PPI) communities.

The EME Funding Committee assesses stage 1 and stage 2 proposals and makes funding recommendations to the DHSC and MRC. It currently has a chairperson and 23 members, representing the clinical research areas in scope for the EME programme. This committee covers a broad range of clinical specialisms, as well as statistical expertise and PPI. If additional input and expertise is required, the review process may include external review or co-opting committee members.

The NIHR is the lead administrative partner on behalf of the funders of the programme and is responsible for preparing and publishing the EME calls for proposals, handling outline and full applications, arranging external referee reports and providing monitoring and oversight for studies post award. These operational aspects are carried out by the secretariat for the programme at NETSCC, referred to as ‘NETSCC’ throughout this review. NETSCC together with the broader senior management and leadership of the programme form the ‘EME programme team’.

Where relevant, EME projects are monitored by their own Trial Steering Committee (TSC) or Study Steering Committee, which supervises the trials and ensures that they are carried out to the appropriate standards. The day-to-day management of the project is the responsibility of the chief investigator (CI). Most projects also establish a Data Monitoring (and Ethics) Committee that report to the TSC/Study Steering Committee.

To monitor outputs and outcomes of EME studies, investigators are required to submit information to the outcomes collection system Researchfish^® (Interfolio UK Ltd, Cambridge, UK) on an annual basis, both during the award and for 5 years after award completion. In addition, NETSCC regularly monitor project progress and financials.

On completion, EME studies submit a final report. Since 2014, these are mandated to undergo peer review and publication in the open-access EME journal (part of the NIHR Journals Library). In addition, a study protocol has to be submitted, which is then made available in the public domain. The editorial office at NETSCC manages all related processes, co-ordinates the editorial boards and liaises with the production house. The NIHR Journals Library website provides a platform to showcase all information about the study in one place. 5 The CI can develop a ‘threaded publication’ and submit material (protocol, trial findings, secondary analyses, lessons learned, etc.) throughout the life of the project and beyond. This not only serves an archival purpose, but also facilitates reproducibility and enhances transparency, knowledge exchange and impact.

Chief investigator survey

Two online surveys were implemented to gather information and views of the following.

Programme of interviews

We selected 47 awards to gather in-depth information to inform the impact and process evaluations. This included all 41 projects that had completed at the time of the review, as well as six active projects that were found to have published their main findings at the time the interview programme was being implemented (see Desk research and database analysis). Two CIs led two awards each. Therefore, in total, 45 individuals were contacted and approximately half (51%, 23/45) were available for interview before the UK COVID-19 outbreak.

Interviews were semistructured in nature, with open-ended questions and the option to probe answers and specific aspects in more detail. Interviewees were first approached by NETSCC to request participation and then contacted by the evaluation team to schedule interviews. Interviews were conducted remotely, recorded, transcribed and analysed using the software tool NVivo (QSR International, Warrington, UK). First, broad categories of analysis were defined, aligning with the EQs, and transcript sections were assigned accordingly. The coding units were then refined through several iterations, taking account of themes and patterns emerging across transcripts. The full interview questionnaire and an analysis of the characteristics of EME projects led by respondents and non-respondents are available in Appendix 2.

In addition, five key opinion leaders and four members of the EME Impact Advisory Group (see Analysis and recommendations) were consulted to gather views on the fit of the EME programme in the wider research funding landscape and the programme’s design (all referred to as ‘key opinion leaders’ in the report). Key opinion leaders were selected to represent research institutions, research funding organisations, industry and the PPI perspective. As this part of the evaluation was scheduled to follow the CI consultation, the number of interviews was affected by the COVID-19 outbreak and, therefore, smaller than originally planned.

The ‘EME research funding/programme domain’

The ‘EME research funding/programme domain’ contains aspects pertaining to direct inputs from the NIHR, MRC and devolved administrations, and the design and implementation of the EME programme. These aspects include governance structures and strategy development, project funding and staff resource (inputs), active management of the EME programme, support for funded projects, dissemination of research findings through a dedicated NIHR platform and the EME journal (activities), enabling learning from programme implementation (outputs) to adjust and strengthen funding mechanisms (outcomes) and ultimately provide value for money for funders by maximising the impact of the investment.

The ‘research domain’

The ‘research domain’ contains aspects pertaining directly to the delivery of EME research projects by CIs and further R&D of the tested intervention. These aspects represent the core research pathway [i.e. research activities funded by the EME award (activities) lead to new knowledge and evidence (outputs), which in turn inform further research leading to scientific advancement and uptake by the other researchers (outcomes)]. The scientific advancement may relate to more efficient clinical trials, progress of the intervention tested along the technology readiness level scale and further de-risking of technologies (outcomes). This, in turn, may feed into high-quality translational research applications in the future (inputs).

The ‘commercial/health-care domain’

The ‘commercial/health-care domain’ contains aspects pertaining to knowledge creation for industry and/or public health and care services, with the aim of delivering benefits to patients and the public. Researchers and industry collaborators bring pre-existing IP into new project proposals and, with the support and oversight of the NIHR IP unit (inputs), may generate and register new IP (outputs) as part of the delivery of the EME project. The translation of outputs and outcomes of the ‘research domain’ and the new IP related to tested technologies can lead to improved health products and interventions, commercial exploitation, and take-up into clinical practice and policy (outcomes). Adoption of cost-effective technologies and interventions in the health-care sector, delivery of higher-quality care and more effective and efficient use of existing resources may ultimately lead to economic value and enhanced population health (impacts).

The ‘research ecosystem domain’

The ‘research ecosystem domain’ contains aspects pertaining to co-ordination within and strengthening of the wider research ecosystem (i.e. the environment that enables delivery of research). These aspects include existing infrastructure and other funding and resources used in the delivery of EME projects (inputs), through expert review of proposals, involvement of patients and the public in research design and delivery, collaboration between and across disciplines and sectors, including industry, hospitals and universities, and training of students and staff as part of the project (activities), leading to improved capacity to generate relevant knowledge (outputs), stronger collaborative networks and a change in research culture and the research ecosystem (outcomes). This ultimately influences the translational research landscape and funding agendas towards a more coherent and joined-up system (impacts).

Several aspects are relevant for more than one domain (see Figures 1 and 2).

The EME portfolio

Between 2009/10 and 2018/19, the EME programme published 118 calls for proposals that received at least one application, including 84 commissioned calls (of which 12 were ‘mechanisms of action’ calls) and 34 researcher-led calls. Thirty-four per cent (40/118) of these calls did not lead to any funded projects. The share of ‘unsuccessful’ calls is higher for commissioned calls (37/84, 44%, including four mechanistic calls) than for researcher-led calls (3/34, 8.8%). In addition, five fast-track proposals were considered, of which two were funded.

Across the 118 calls for proposals, a total of 854 applications entered the review process, of which 285 (33%) were shortlisted and 158 (18.5%) received funding (Figure 3). The success rate was similar for applications to commissioned and researcher-led calls [68/346 (19.7%) and 90/508 (17.7%), respectively]. A total of 30 applications entered the (one-stage) review process across 12 mechanisms of action calls, of which 11 (36.7%) were funded.

FIGURE 3.

Success rates of applications for all EME calls: 2009/10 to 2018/19. (a) All applications; (b) researcher led; and (c) commissioned. Source: Technopolis analysis of EME call success rates data.

With the exception of 2011/12, success rates of applications ranged from 11.0% (2009/10) to 24.0% (2014/15) (Figure 4). Overall, the proportion of applications receiving funding was similar for commissioned and researcher-led calls at just under 20% [68/346 (19.7%) and 90/508 (17.7%), respectively]. There was, however, some year-on-year variation, with a higher proportion of commissioned projects receiving funding in 2013/14 (commissioned, 30.3%; researcher led, 18.4%) and a higher success rate for researcher-led applications in 2018/19 (commissioned, 12.0%; researcher led, 24.0%) than in other years. (Given the very small number of applications considered, 2011/12 was excluded from this analysis.)

FIGURE 4.

Number of applications and shares funded per year. Data labels indicate the total number of applications considered by the EME Funding Committee per financial year. For each call, applications are counted in the financial year in which the final application was submitted. The smaller number of applications in 2011/12 is the result of applications that were deferred to, or resubmitted in, 2012/13. Unsuccessful applicants to an EME commissioned call may resubmit their proposal to a researcher-led call (‘legacy effect’). Source: Technopolis analysis of EME call success rates data.

From 2013/14, the success rate for applications to commissioned calls dropped steadily, from 30.3% to 12% (2018/19). The success rate for researcher-led applications remained relatively steady between 2012/13 and 2017/18, ranging between 18.4% and 20.6%, with an uptick in 2018/19 to 25%.

Applications

Between 2008 and 2016, 772 applications entered stage 1 of the review process (note that data for 2017 and 2018 are not available). These applications were submitted by 145 institutions, of which 33% (48/145) received funding for at least one application. Most of the applications were led by CIs affiliated with academic institutions (562/772, 73%), especially those that are part of the Russell Group (Cambridge, UK) (471/772, 61%), whereas the remaining 27% (205/772) were led by CIs from NHS hospital trusts. Nearly one-tenth of applications were by CIs at Imperial College London (London, UK) (72/772, 9%).

The success rate was much higher for applications from Russell Group institutions (99/471, 21%) than for applications from other universities (8/91, 8.8%). The success rate for applications from NHS hospital trusts was similar to Russell Group institutions, at 18% (36/205).

The Health Research Classification System (HRCS), developed by the UK Clinical Research Collaboration (UK CRC), is a two-dimensional framework for classifying research awards. One dimension of the framework, the ‘research activity codes’, classifies awards according to the type of research activity conducted. The other dimension, the ‘health categories’, classifies research according to 21 separate categories that encompass all diseases, conditions and areas of health. 6,7 A research project can be associated with up to five HRCS health category codes. For further detail on HRCS coding methodology and the approach to analysis employed in this evaluation, including caveats, see Appendix 4.

Individual applications to the EME programme were associated with between one and five HRCS categories. The largest number of applications was associated with the health category ‘cardiovascular’ (122/772, 16%), followed by ‘cancer’ (112/772, 15%) and ‘oral and gastrointestinal’ (94/772, 12%) (Figure 5). It should be noted that, throughout the analysis of HRCS codes, those that are more frequently assigned alongside other codes, such as ‘metabolic and endocrine’ and ‘infection’, may be overstated, whereas codes that are rarely assigned alongside other codes, such as ‘mental health’, ‘musculoskeletal’ and ‘respiratory’, may be understated.

FIGURE 5.

Share of total applications by HRCS health categories. Data labels indicate total number of applications associated with the relevant HRCS health category. Commissioned, n = 283; researcher led, n = 488. Source: analysis of call success rates and EME portfolio data.

There were some differences between applications to commissioned and researcher-led calls. Applications to commissioned calls were most often associated with the HRCS categories ‘oral and gastrointestinal’ (50/283, 18%), ‘neurological’ (38/283, 13%) and ‘cancer’ (36/283, 13%), whereas researcher-led applications were more frequently associated with HRCS categories ‘cardiovascular’ (95/488, 19%), ‘cancer’ (76/488, 16%) and ‘metabolic and endocrine’ (51/488, 10%). Therefore, applications to researcher-led calls more often than commissioned calls were associated with the HRCS category ‘cardiovascular’ (19% vs 10%), whereas applications to commissioned calls were more often associated with the HRCS category ‘oral and gastrointestinal’ than researcher-led calls (18% vs. 9%). Some of the applications in the ‘oral and gastrointestinal’ HRCS category can be traced back to commissioned calls targeting this health area, demonstrating that commissioning has an effect on the type of research ideas submitted. For example, applications submitted to calls ‘Inflammatory bowel disease’, ‘Bowel control and faecal incontinence in adults’ and ‘Very low energy diets’ were associated with the HRCS category ‘oral and gastrointestinal’. In addition, commissioning may influence the types of projects funded through the researcher-led workstream (‘legacy effect’), either directly (when unfunded proposals to commissioned calls are successfully resubmitted to researcher-led calls) or indirectly (by stimulating the community in a targeted research field, leading to an increase in submissions in following years). For example, six applications originally submitted to commissioned calls were ultimately funded through the researcher-led workstream, and a seventh through another relevant commissioned call. The following analysis of HRCS category associations classifies these six projects as researcher-led projects (i.e. it does not take into account the legacy effect).

Success rates were highest for applications associated with the HRCS categories ‘eye’ (8/24, 33%), ‘reproductive health and childbirth’ (16/50, 32%) and ‘skin’ (3/10, 30%) (Figure 6). Other ‘major’ categories (i.e. those associated with ≥ 10 projects) include ‘inflammatory and immune system’ (26% success rate, 11/42 applications), ‘respiratory’ (23% success rate, 11/48 applications) and ‘oral and gastrointestinal’ (21% success rate, 20/94 applications). For commissioned calls, the highest success rates were associated with applications belonging to the HRCS categories ‘eye’ (45%, 5/11), ‘skin’ (40%, 2/5) and ‘reproductive health and childbirth’ (35%, 8/23). For researcher-led applications, ‘blood’ (33%, 1/3), ‘reproductive health and childbirth’ (30%, 8/27) and ‘inflammatory and immune system’ (27%, 6/22) had the highest success rates.

FIGURE 6.

Success rates of applications per HRCS health category. The total number of applications is indicated to the right of the corresponding bar. Source: analysis of call success rates and EME portfolio data.

The lowest success rates among major HRCS categories (associated with ≥ 50 applications) were observed for ‘musculoskeletal’ (10%, 6/58), ‘mental health’ (12%, 10/84) and ‘neurological’ (13%, 11/86). Within these health categories, success rates for applications to commissioned and researcher-led calls were broadly similar. On the other hand, the ‘metabolic and endocrine’ area showed a notable difference, with applications to commissioned calls achieving a success rate of 31% (10/32), which was five times higher than applications to researcher-led calls (6%, 3/51). Each of these 10 commissioned projects was funded through a different commissioned call.

In terms of both average and median, commissioned calls received a smaller number of applications (n = 9, with a median of 6) than researcher-led calls (n = 15, with a median of 13), with 39% (33/84) of commissioned calls receiving between one and three applications only. This number is particularly small for the 12 ‘mechanisms of action’ calls, with an average and median of three applications. However, as these calls represent a ‘rolling opportunity’ (i.e. identical calls inviting application across a broad time window), a smaller number of submissions to each individual call is expected. In terms of the size of the project team, there was no significant difference between the average number of team members of the two types of funding streams (researcher led, n = 9.3; commissioned, n = 10.1).

Funded projects

Of the 145 projects contracted before 30 September 2018, 53% (77/145) are active and 37% (53/145) have completed and/or published their main findings [completed, n = 40 (28%); published, n = 13 (9%)]. In addition, one active project, a multiarm trial, has published main findings for one of its arms but not yet others. Of the completed projects and projects that have published the main findings, eight were funded through commissioned calls (including two focused on mechanisms of action).

Six projects (4%) were marked as ‘discontinued’ and nine projects (6%) were marked as ‘in editorial’ (n = 8) or in ‘post-contract set-up’ (n = 1). As can be expected, most of the completed projects were initiated in the first 5 years of the scheme, whereas active projects tended to be funded more recently (Figure 7).

FIGURE 7.

Number of projects by status and the financial year the project was funded. Numbers correspond to data provided by NETSCC, which were updated through desk research. Of the 16 projects marked as ‘in editorial’ in NIHR records, eight were found to have already published their main findings (two in the EME journal and six elsewhere). These projects were labelled ‘published’ and counted in the ‘completed/published’ set of awards. One project marked as ‘completed’ in NETSCC records was unable to recruit and stopped and, therefore, it was included in the ‘discontinued’ project count. Source: Technopolis analysis of EME portfolio data (n = 145).

The funded projects are/were led by CIs from 49 different institutions/organisations. Imperial College London was associated with the largest number of studies (13/145, 9.0%), in line with accounting for the largest share of applications submitted (72/772, 9.3%). King’s College London (London, UK) and the University of Birmingham (Birmingham, UK) came next, with CIs based at these institutions leading 10 (6.9%) and nine (6.2%) projects, respectively.

Efficacy and Mechanism Evaluation projects rely, to a large extent, on pre-existing teams and collaborations. The majority of survey respondents (72%, 33/46) indicated that they had worked with the EME project team before. CIs reported that ‘new’ co-investigators were included because they were experts in their field, and that they had been identified through personal contacts or NIHR recommendations.

Project costs

A total of £175.7M in funding was approved for 145 EME projects at review. The yearly amount of funding awarded peaked in 2014/15 at £29.6M, owing to more funds allocated through commissioned calls (Figure 8). The total funding amount approved was largely similar for commissioned (£81.1M) and researcher-led (£92.8M) calls, with the former being awarded across seven financial years, compared with 10 for the latter. The average funding amount per project peaked in 2011/12 for commissioned calls (£1.7M) and in 2015/16 for researcher-led calls (£1.6M).

FIGURE 8.

Total EME project funding (£M) awarded by financial year. ‘Transfer’ refers to projects transferred to the EME programme from other funding streams. Bars show total funding awarded. Source: Technopolis analysis of EME portfolio data.

The average project cost at the point of contracting was £1.2M across all projects. This value was slightly higher for projects funded through commissioned calls, at an average of £1.3M per award, than for researcher-led projects, at an average of £1.1M per award. Project costs ranged from £300,000 to £3.9M for projects funded through commissioned calls, with a median of £1.16M, and from £100,000 to £3.5M for researcher-led projects, with a median of £1.03M. Although there was an upwards trend in the average level of funding awarded to researcher-led projects (e.g. £960,000 for projects up to and including financial year 2011/12 vs. £1.4M for financial years 2012/13 to 2017/18), this was not the case for projects funded through commissioned calls.

Research areas

The 145 projects were associated with 18 HRCS health categories, with the greatest share of projects coded against the categories ‘cancer’ and ‘cardiovascular’ (21/145, 14% each), and ‘oral and gastrointestinal’ (20/145, 14%) (Figure 9). The category ‘metabolic and endocrine’ accounted for a higher proportion of commissioned projects (10/63, 16%) than of researcher-led projects (3/81, 4%), as did the category ‘oral and gastrointestinal’ [13/63 (21%) vs. 7/81 (9%), respectively]. Conversely, ‘cardiovascular’ projects accounted for nearly twice as many researcher-led projects as commissioned projects [15/81 (19%) vs. 6/63 (10%), respectively].

FIGURE 9.

Share of funded projects by HRCS health category. Data labels indicate the total number of projects associated with the respective health category code. Thirty-one projects were associated with more than one HRCS health category, resulting in a total of 190 associations. Projects associated with more than one code are represented accordingly (i.e. two or more times). The analysis gave equal weight to each association, irrespective of whether a project was assigned one or more codes. Further detail on the analysis approach and caveats is available in Appendix 4, Table 9. Source: Technopolis analysis of EME portfolio data (n = 145).

As can be expected from an EME programme, the majority of projects (126/145, 87%) were associated with the HRCS research activity code for ‘evaluation of treatments and therapeutic interventions’ (Figure 10). The next most common categories were ‘development of treatments and therapeutic interventions’ [associated with 62/145 (43%) projects] and ‘detection, screening and diagnosis’ [associated with 32/145 (22%) projects]. Similarly, the majority of studies were classified as ‘treatment’ trials in clinical trials registries (120/136, 88%), with diagnostic studies the next most popular type (9/136, 7%).

FIGURE 10.

Share of funded projects by HRCS research activity category. Data labels indicate the total number of projects associated with each research activity code. Seventy-six projects were associated with more than one HRCS research activity category, resulting in a total of 227 associations. Projects associated with more than one code are represented accordingly (i.e. two or more times). The analysis gave equal weight to each association, irrespective of whether a project was assigned one or more codes. Source: Technopolis analysis of EME portfolio data (n = 145).

Commissioned projects involved a larger share of ‘detection, screening and diagnosis’ [commissioned, 23/63 (37%), vs. researcher led, 9/81 (11%)], whereas ‘evaluation of treatments and therapeutic interventions’ accounted for a greater share of researcher-led projects [researcher led, 77/81 (95%), vs. commissioned, 48/63 (76%)].

More than half of the EME projects investigated pharmaceutical interventions (84/145, 58%) and were associated with the research activity codes ‘6.1 pharmaceuticals – treatment evaluation’ (79/145, 54%) and/or ‘5.1 pharmaceuticals – treatment development’ (44/145, 30%). Fifteen per cent (22/145) of projects were related to ‘4.2 evaluation of markers and technologies’.

Compared with researcher-led projects, a larger share of commissioned projects addressed ‘4.2 evaluation of markers and technologies’ [researcher led, 8/81 (10%), vs. commissioned, 14/63 (22%)] and ‘4.1 discovery and preclinical testing of markers and technologies’ [researcher led, 1/81 (1%), vs. commissioned, 8/63 (13%)]. By contrast, the share of researcher-led projects was larger for ‘6.7 physical’ [researcher led, 7/81 (9%), vs. no commissioned projects], including physical therapies and exercise.

Although research activity code 5 (‘development of treatments and therapeutic interventions’) and code 4.1 (‘discovery and preclinical testing of markers and technologies’) are considered outside the EME programme’s remit, these were probably applied to mechanistic studies or mechanistic components of studies. For example, for code 4.1, this is reflected in that fact that only three of the nine projects concerned used trial methodology.

Types of interventions tested

An analysis of funded projects was conducted to understand the types of research projects in more detail, based on the evaluation team’s assessment of project abstracts, study protocols (if published) and targeted online searches.

The majority (67/137, 49%) of EME-funded projects investigated whether or not an existing intervention can be used for a different health issue or to treat a different patient group (i.e. ‘repurposing’) (Figure 11). Thirty-four per cent of projects (n = 47) were directly involved in the development or validation of new therapies (i.e. interventions not yet in clinical use) or novel diagnostic/stratification/imaging approaches. Seventeen per cent of projects (n = 23) generated evidence to test or inform current clinical practice (i.e. interventions already in use). The average cost of projects in each of these categories was comparable, at £1.2–1.3M.

FIGURE 11.

Type of study intervention by share of funded projects. Source: Technopolis analysis of study protocols and final reports (n = 136; excludes eight mechanistic projects).

Over time, the share of projects that generated evidence to test or inform current clinical practice increased while the share of projects developing or validating new therapies or approaches decreased (Figure 12). Compared with the first 5 years of EME awards (i.e. financial years 2008/9 to 2012/13), the second 5 years (i.e. financial years 2013/14 to 2017/18) saw an increase in the share of projects that generated evidence to test or inform current clinical practice, rising from an average of 8% (5/66 projects) to an average of 26% (18/70 projects). Conversely, the share of projects developing or validating new therapies or approaches decreased from 41% (27/66) to 27% (19/70) of projects. The share of repurposing studies remained relatively constant.

FIGURE 12.

Type of study interventions per year by share of funded projects. Source: Technopolis analysis of study abstracts, protocols and targeted online searches (n = 136).

Of projects that target repurposing of an existing intervention, 76% (51/67) investigated drugs, 10% (7/67) biologics and 6% (4/67) different use cases for existing devices. For at least 75% (50/67) of projects targeting repurposing, the drugs and biologics were treatments available as generics.

Of the 47 projects that were directly involved in developing or validating novel therapies or diagnostic/stratification approaches (i.e. not yet in clinical use), 39% (n = 19) developed new diagnostic tests (n = 14, including eight based on imaging) or stratification approaches (n = 5). Eleven per cent (n = 5) targeted gene and cell therapies, 9% (n = 4) devices and 7% (n = 3) surgical procedures, behavioural therapies or imaging approaches to support surgery. Only one study took forward the development of a new pharmaceutical, a therapeutic antibody.

Twenty-three projects aimed to inform current clinical practice by assessing the efficacy of treatments routinely provided but lacking supporting evidence, including treatments that became recently available outside the UK and the NHS. Four of these projects assessed the efficacy of (routinely used) special diets or dietary supplements, and three investigated the degree to treatments were effective in specific patient subgroups.

Addressing patient needs and policy priority areas

Efficacy and Mechanism Evaluation projects addressed nine of the top 10 causes of disability-adjusted life-years (DALYs) in the UK (based on 2017 figures)8 (Figure 13). The only cause not addressed was ‘headache disorders’.

FIGURE 13.

Distribution of EME projects relative to 10 burdens of ill health (UK DALYs). Data labels indicate number of projects (n = 145). Source: Technopolis analysis of EME portfolio and data from the Institute for Health Metrics and Evaluation. 8

Commissioning has contributed to this alignment. For example, of the seven projects investigating diabetes-related conditions, five were funded through commissioned calls. Likewise, both of the projects addressing Alzheimer’s disease were awarded following a themed call. Eight themed calls, corresponding to NIHR-wide priority areas, attracted a total of 61 applications. Of these, six calls led to a total of 12 funded projects.

When the distribution of the burden of ill health (in UK DALYs) is compared with the share of EME projects associated with the relevant HRCS health categories, which are broader than the individual causes in Figure 13, coverage is variable (Figure 14). On the one hand, the two HRCS categories accounting for the largest share of UK DALYs (35% in total), ‘blood/cardiovascular/stroke’ and ‘cancer and neoplasms’, correspond to the largest share of EME projects (55/141, 39%). On the other hand, for ‘reproductive health’, ‘metabolic and endocrine’ and ‘oral and gastrointestinal’, the share of EME projects is higher than the corresponding UK DALY rates (8.5, 3 and 2.5 times more, respectively). Conversely, the areas ‘injuries and accidents’, ‘musculoskeletal’ and ‘mental health’ are less represented in the EME project portfolio, with the UK DALY rates 3.7, 2.2 and 1.8 times higher, respectively, than the share of EME projects addressing these issues. Further detail and caveats of the analysis are available in Appendix 4.

FIGURE 14.

Shares of UK DALY rates (2012 and 2016) and EME projects associated with relevant HRCS health categories. The HRCS category ‘inflammatory and immune’ has no equivalent World Health Organization Global Health Estimates code and, therefore, four projects that are associated with only this category were excluded from the analysis. Source: Technopolis analysis of EME portfolio and data from UK CRC UK Health Research Analysis 2018,6 which maps HRCS health categories to the World Health Organization Global Health Estimates to calculate the equivalent DALY rate per health category.

This variation is underpinned by two factors: (1) the number of applications received and (2) the success rate (Table 2). For example, the number of applications associated with the HRCS category ‘reproductive health and childbirth’ was near the average (n = 50), but a higher success rate led to a relatively large number of awards (n = 16). Similarly, the number of applications in the category ‘musculoskeletal’ was also near the average (n = 58), but a low success rate (10%) led to a small number of funded projects (n = 6). Conversely, although the success rate for applications in the ‘metabolic and endocrine’ category was below average (15%), a larger number of applications (n = 84) led to an above average number of projects (n = 13). The category ‘injuries and accidents’ received fewer applications (n = 13) and, despite a success rate slightly greater than the average (23%), only a small number of projects were funded (n = 3).

The success rates of applications in response to commissioned and research-led calls were broadly similar for the health categories shown in Figure 14. An exception was the ‘metabolic and endocrine’ area, with applications in response to commissioned calls achieving a success rate of 31% (10/32), which was five times higher than applications in response to researcher-led calls (6%, 3/51). However, each of the 10 commissioned projects was funded through a different commissioned call. The reason for this difference is unclear.

An analysis of areas where commissioned calls led to few or no awards (i.e. ‘unsuccessful calls’) points to different reasons underlying differences in the number of projects funded per health area and call:

• Two calls targeting chronic obstructive pulmonary disease (COPD) received a total of 18 applications, but nearly half of these applications (8/18, 44%) did not enter stage 1 of the review process (i.e. were out of scope for the EME programme). Of the 10 applications that entered the review process, only one was successful. A second application in response to a COPD commissioned call that was rejected was later resubmitted and funded through a researcher-led call. Therefore, although there was interest in the COPD calls (indicating that there are research groups in the UK working in this area), the applications were not of the required quality. The reason could be that the research field has not yet reached the appropriate stage for efficacy studies. If this is the case, funding needs to focus on earlier-stage research to understand underlying mechanisms and identify potential interventions. It may also signal a need for capacity building in the associated research community to design and undertake efficacy studies.

• Two calls focusing on sleep disorders received a total of 11 applications, all of which entered stage 1 of review (i.e. were considered in scope). However, none of the applications was successful. This suggests that the proposed projects were in scope but of insufficient quality. This might be addressed through a focus on capacity building or may signal that more supporting infrastructure is needed.

• A combination of small application numbers and low success rates was observed for two mental health-themed calls in 2017 and 2018. These received a total of five applications (compared with an average of 9.3 applications per call for all other themed calls), of which only three entered the review process and, ultimately, none was funded. As already noted, applications associated with the HRCS category ‘mental health’ had a relatively low success rate across all calls (10/84, 12%). Reasons could include alternative funding sources in the area of mental health that researchers can access, a low level of interest in mental health research in the UK, a lack of interventions to test in efficacy studies (e.g. applications to the themed call were predominantly to the HTA or other schemes) and low capacity to conduct efficacy research, including challenges in accessing the necessary research infrastructure and/or low awareness of the EME programme or call in the relevant research community. In addition, some researcher communities may be discouraged from applying because of the perception that a mechanistic component is required as part of the study.

Key priority areas identified by the UK government include conditions associated with ageing, maternal/reproductive health and mental health.

Conditions associated with ageing were identified as a key priority across multiple government entities,9–11 including the NIHR. 12 In particular, dementia was highlighted as a priority area in the Public Health England (PHE) strategic plan,13 the NHS long-term plan11 and the Prime Minister’s challenge on dementia. 14 Nine EME projects investigated conditions specifically affecting older patients (e.g. Alzheimer’s disease, age-related macular degeneration and musculoskeletal health), not including common indications encountered in the elderly (e.g. heart conditions or type 2 diabetes), all of which stemmed from commissioned calls. This accounts for 6.2% of EME projects, a relatively small share, given the UK government’s focus on this area.

The areas of maternal/reproductive health and mental health are also UK priority areas, as detailed in the white paper Healthy Lives, Healthy People: Our Strategy for Public Health in England,10 the NHS Five Year Forward View11 and the PHE strategic plan (2016–20). 13 In the area of maternal and reproductive health, the EME programme has funded 16 projects (11% of the portfolio), with half (n = 8) funded through commissioned calls. The scheme has also funded 12 projects on mental health and developmental brain disorders (8.2% of the portfolio), with a further project awarded but not yet contracted at the time of this evaluation. In total, 10 commissioning calls specifically targeted mental health and developmental brain disorders, with only four leading to one funded project each.

Origin of research idea

Surveyed CIs most frequently reported that the original idea for their research topic came from clinical researchers (33/45, 73%), followed by academic researchers (23/45, 51%) and health professionals (12/45, 27%) (Figure 15). Of those who provided further details (n = 28), eight reported that the idea was from their personal experience and seven explained that the research was based on a clinical need. Other responses referred to ideas stemming from a commissioned call (n = 3) or from workshops/meetings (n = 2).

FIGURE 15.

Source of research topic idea for successful and unsuccessful applications. Data labels indicate the number of projects. Successful applications, n = 45; unsuccessful applications, n = 28. Source: Technopolis analysis of survey data.

On average, unsuccessful applicants referred to a broader range of ‘origins’ for their research idea. As for successful applications, clinical researchers (23/28, 82%) and academic researchers (18/28, 64%) were most commonly mentioned as the source of the idea. However, the share of ideas stemming from patient groups was much higher for unsuccessful applications than for successful applications [13/28 (46%) vs. 8/45 (18%)], as was the share of ideas stemming from the literature [9/28 (32%) vs. 5/45 (11%)].

Types of studies funded

The majority of studies were RCTs (125/145, 86%), including some with novel or efficient designs (see Elements of innovative study design). Seven of the RCTs also included a pilot or feasibility component. The remaining studies were defined as cohort (n = 5), observational (n = 3), or stand-alone pilot/feasibility studies (n = 2) or substudies (n = 10, either substudies of larger clinical trials or studies using existing samples or data).

Most projects (121/145, 83.4%) included a mechanistic component, mainly sample analysis (n = 108, 74.5% of projects). Eight projects were fully mechanistic (5.5%).

Elements of innovative study design

Efficacy and Mechanism Evaluation projects included a range of innovative study designs. A total of 12 studies employed an adaptive methodology [i.e. studies in which an earlier phase of the project is used to inform either the decision to continue with the study (stop/go) or to prospectively adapt the study based on an interim analysis]. This includes a study with the primary objective of validating adaptive trial design in the context of surgical trials. One trial15 utilised a multiarm, multistage approach with the concurrent evaluation of biomarkers (Box 1). A further trial employed a stepped-wedge cluster design.

The funding environment

Sources of funding preceding the EME award

Among projects that indicated a source of funding for research underpinning the EME award (108/145, 74%), 58% (63/108) had received funding from UK public funders and 46% (50/108) from charities (Figure 16). The NIHR (n = 25, 23%) and the MRC (n = 20, 19%) were the most common sources of funding, whereas industry funding accounted for 17% (18/108). Of EME awards that were based on NIHR-funded research, 32% (8/25) had been supported by NIHR BRCs and BRUs (n = 6 and n = 2, respectively), 16% (n = 4) by the NIHR Research for Patient Benefit (RfPB) scheme and 8% (n = 2) by a NIHR Programme Grants for Applied Research grant. Interestingly, five awards were based on insights from NIHR HTA projects, demonstrating that research in the later stages along the translational pathway informs further research in earlier stages. Little detail was available on MRC-funded studies on which EME projects are based. Two EME awards had received support from the MRC Experimental Medicine Programme/Experimental Medicine Challenge Grant Programme, and one from a MRC Pathfinder scheme.

FIGURE 16.

Sources of funding preceding EME projects by share of projects. Data labels indicate number of EME projects (n = 108). Individual projects may report funding from more than one source. EC, European Commission. Source: Technopolis analysis of EME portfolio data classified according to EME programme internal coding.

Of charity funders, the British Heart Foundation (London, UK) supported the largest number of preceding research studies (n = 7), followed by the Wellcome Trust (London, UK) and Cancer Research UK (CRUK) (London, UK) (n = 6 each). Other charities provided funding for up to three projects each. These were primarily disease-specific charities, such as the Juvenile Diabetes Research Foundation (London, UK), Diabetes UK (London, UK), the MS Society UK (London, UK) and the Cystic Fibrosis Trust (London, UK).

Co-funding

Efficacy and Mechanism Evaluation project co-funding included agreements between funders or organisations to cover the research costs of a project. In addition, some EME projects were supported through other types of financial collaboration, such as discount agreements, and these are included in this analysis.

Information on co-funding reported in, or estimated from, applications was available for 62 projects (62/145, 43%). The most frequent co-funding source was the pharmaceutical industry (31/62, 50%), followed by ‘other industry’ (23/62, 37%) and charities (12/62, 19%) (Figure 17). ‘Other industry’ included primarily medical technology companies and health-care small and medium-sized enterprises (SMEs).

FIGURE 17.

Sources of co-funding by share of projects. Data labels indicate the number of projects (n = 145). The majority of projects received co-funding from one other source, with a maximum of four co-funders reported. Source: Technopolis analysis of EME portfolio data.

Efficacy and Mechanism Evaluation projects that had received funding from industry for research prior to the EME award were more likely to be co-funded by industry (11/18, 61%), potentially because these built on previous work, addressing research ideas of commercial interest or focus on interventions the companies have brought to market. EME projects that were funded by grants from the MRC, NIHR and charities prior to the EME award were less likely to receive industry co-funding [6/21 (29%), 7/24 (29%) and 17/50 (34%), respectively].

Most co-funding was provided in the form of free or reduced-cost interventions (42/62, 68%) or placebo (19/62, 31%). Contributions to research costs accounted for 18% of co-funded EME projects (n = 11). The majority of projects that reported co-funding specified the amount provided (54/62, 87%). The total overall value of co-funding secured, as reported at the application stage, was approximately £15M.

Forty per cent of CIs consulted for this evaluation (18/46 surveyed and 9/21 interviewed) reported that their EME projects had received co-funding from other sources, including charities, institutional funds and in-kind contributions from industry. This included three substudies to larger trials and co-funding from disease-specific charities (n = 3) (e.g. CRUK). Thirty-three per cent of interviewed CIs (n = 7) received no co-funding or in-kind contribution. Of survey respondents who did not receive any contributions from industry, 38% (12/32) had approached companies but the request was declined. Where reasons were provided, these included a lapse in the drug’s licence, the company not supporting the trial protocol, failure of the company and the NIHR to agree on IP terms and the inability of the company (SME) to financially support the work. Reasons given by CIs for not seeking industry funding (20/32, 63%) included that the treatment was not of commercial interest (e.g. off-patent, trial testing treatment withdrawal) and that it was important to conduct the trial independently from industry. One CI set out their considerations for choosing whether or not to work with industry in more detail, including a suggestion for how the NIHR could support such collaboration (Box 2).

BOX 2 - Example of CI considerations vis-à-vis industry collaboration

There were two main considerations that competed with each other:

1. We could [and eventually did] select the only drug that was available in generic form, with a view that in the event of a successful outcome, it would have a significant cost benefit to implement within the NHS. However, it had the upfront cost of manufacturing that proved a tedious and problematic endeavour to negotiate in the limited window period of the application for the specific funding call we had entered into.

2. We could approach a pharma partner to provide both the active drug and placebo at their cost, for a compound that is still under patent. We opted against this due to a lack of confidence in the commercial partner being sufficiently interested or agile and responsive to aid getting relevant agreements in place up front. We also were somewhat anxious that including a commercial partner might have the complication that in the event of a successful outcome, the commercial partner would stand to gain significantly financially [ . . . ] leading to a higher cost to the NHS in the future since the current patent, although due to expire in the near future, would have given the commercial partner scope to potentially amend or extend the scope of their patent. [ . . . ] The limited timelines between outline and final application were not conducive to properly exploring this option.

[It would be helpful if] the NIHR EME were able to assist in facilitating introductions and negotiations between academics and commercial partners, where the weight and negotiating power of NIHR would be stronger than that of individual academic teams.

Source: Technopolis stakeholder consultation.

Further funding

Further funding was reported for 52 projects (52/141, 37%), securing a total of 186 awards. The number awards received by projects ranged from one to 16, with a median of three. Follow-on awards ranged from £815 to £111M in size, amounting to a total value of £355M. The majority (47/52, 90%) of projects secured research grants. Other types of funding were fellowships (n = 11, 21% of projects), studentships (n = 7, 13%), capital/infrastructure funds (n = 6, 12%) and travel/small personal awards (n = 3, 6%).

The main source of further funding was the public sector. Of the 52 projects that reported further funding, 63% (n = 33) received further funding from the UK public sector, 56% (n = 29) from charities/non-profit organisations and 15% (n = 8 each) from industry and institutional sources (Figure 18). Similarly, the public sector accounted for 45% (84/186) of all reported follow-on awards, whereas charities provided one-third of follow-on awards (57/186, 31%) and industry 6% (12/186).

FIGURE 18.

Source of further funding by share of projects. Data labels indicate number of projects (n = 52). Source: Technopolis analysis of 2020 Researchfish data.

The largest share of projects (22/52, 42%) received further funding from the NIHR (Figure 19), followed by the MRC (17/52, 33%) and the Wellcome Trust (9/52, 17%). The NIHR also accounted for the largest overall number of awards and amount of funding, providing 22% (40/186) of awards for a total of £155M. This included four EME awards, three RfPB scheme awards and two HTA awards, as well as three NIHR Research Professorships and two NIHR Clinical Fellowships. The MRC funded 16% (n = 30) of further funding awards, with a total value of £40M. Most of these were research awards (23/30, 77%), including four that included a MRC Industry Collaboration Agreement, and two awards each from the MRC Developmental Pathway Funding Scheme (DPFS), the MRC Stratified Medicine Scheme and the MRC Proximity to Development Scheme. The Wellcome Trust accounted for 4% (n = 8) of grants, with a total value of £19M. Other commonly reported sources of further funding were the European Commission (Brussels, Belgium), Innovate UK (Swindon, UK), the British Heart Foundation and Boehringer Ingelheim (Ingelheim am Rhein, Germany) (3/52, 6% each).

FIGURE 19.

Funders of further funding by share of projects and total value of reported awards. Data labels indicate number of projects (n = 52). Source: Technopolis analysis of 2020 Researchfish data.

Of CIs of active projects who indicated that they were planning further research, two-thirds (22/33, 67%) reported that they intended to apply to the NIHR for funding and one-quarter pointed to funding from the MRC and/or industry (8/33, 24%).

Role of the EME programme in the wider research funding environment

Eighty-nine per cent of CIs [85% (38/45) of survey respondents and 100% (13/13) of interviewees] agreed that the ‘EME programme fills a gap’ in the funding landscape. Most interviewees (9/13, 70%) discussed the role of the EME programme in bridging the gap between early proof-of-concept and effectiveness studies.

Survey respondents highlighted the programme’s value as a funder of exploratory or novel studies and of mechanistic components (5/45, 11% each). Five CIs described the research pathway as leading from MRC-funded basic research to NIHR EME studies (combining the MRC-oriented mechanistic aspect with the NIHR-oriented efficacy study) to NIHR-funded HTA studies.

Overall, the EME programme was considered unique, with limited overlap or duplication with other funders. When asked about what CIs would have done ‘if the EME programme were not available’, respondents indicated that they would apply to the NIHR HTA programme (19/53, 36%), for MRC funding (12/53, 23%) or for funding from charities, such as the Wellcome Trust (12/53, 23%). A small number of CIs referred to funding from industry (n = 4), other NIHR programmes (n = 3) or the European Commission (n = 2) (Figure 20). However, CIs also highlighted caveats associated with these funding sources, including limitations on the types of studies that could be funded (e.g. NIHR HTA awards are not suitable for mechanistic research), limited budgets offered by charities and concerns over the independence of industry-funded research. One-third (12/40, 30%) of surveyed CIs indicated that there are no alternative funding schemes, a view shared by the majority if CIs interviewed.

FIGURE 20.

Alternative funding programmes to the EME programme by share of CIs. Data labels indicate number of CIs. Some CIs reported multiple alternative funding programmes (n = 53). EC, European Commission. Source: Technopolis analysis of survey and interviews.

When asked about remaining gaps in the funding landscape, a small number of CIs mentioned that it was still difficult to secure funding for proof-of-concept studies. One CI felt that the EME scheme ‘could connect better to DPFS on one end and HTA on the other’. Supporting this statement, none of the ideas underpinning EME projects was attributed to MRC DPFS funding. Similarly, a recent consultation of key opinion leaders in medical research found that ‘while funding streams in the UK were considered more linked up now than a few years ago, some issues were highlighted, e.g. moving from DPFS to EME’ (reproduced with permission from the MRC, 2021, personal communication). 24

A number of interviewees pointed to gaps in EME funding for specific research areas/disciplines or types of efficacy research, such as research focused on prevention, behavioural approaches/non-pharmaceutical interventions and quality-of-life measures, and funding for multidisciplinary research and work using animal models. For example, one interviewee described the EME programme as strongly focused on ‘conservative medical interventions’ (e.g. pharmaceutical or surgical approaches), rather than behavioural and ‘lifestyle interventions’ (e.g. exercise and diet). To bring in more innovative approaches, the suggestion was made to include social science expertise on the EME Funding Committee.

One-quarter (7/28, 25%) of researchers whose application to the EME programme had not been successful reported that they had gone on to conduct an efficacy trial, whereas 39% (n = 11) of respondents were continuing with aspects of the research idea, but were not conducting an efficacy trial (Figure 21). The remaining 33% of respondents indicated that they did not pursue the research idea any further, as they did not think that there were any alternative funding sources and/or because the window of opportunity for that project had now closed.

FIGURE 21.

Share of unsuccessful applicants who continued to work on research idea/intervention. Source: Technopolis analysis of survey data (n = 28).

When asked whether they had applied for funding elsewhere for the research idea underlying the unsuccessful EME application, the majority of CIs (16/27, 59%) reported that they had not done so and one-third (9/27, 33%) reported that they had successfully applied for funding from another funder. At least four of these applications led to trials funded by the NIHR HTA programme, the JP Moulton Charitable Foundation, the MRC DPFS scheme and the British Heart Foundation. Two further applications, relating to mechanistic aspects of an intervention, received funding, one from the Wellcome Trust and one from CRUK. Another two researchers were funded to conduct feasibility/pilot studies, one of whom received a grant through the NIHR RfPB scheme.

Of respondents who continued working on their research idea, one-third (7/19, 37%) reported that research on the intervention had progressed, with most (n = 4) showing that the intervention was beneficial. Five respondents (71%) reported that the research had led to a peer-reviewed journal publication, with one leading to changes in policy/practice (no further details of this were provided).

Challenges, enablers and learning

During the consultation phase, all CIs reported challenges in the implementation of their project.

The majority of CIs of active projects reported challenges with contracting processes (35/46, 76%), followed by the setting-up of study sites (30/46, 65%) and patient recruitment (28/46, 61%) (Figure 22). It should be noted that these active projects include some in the early stages of delivery and, therefore, challenges occurring ‘earlier’ in project implementation, such as contracting, may be over-represented. Contracting issues were encountered between a variety of organisations (e.g. between the NIHR and the lead university, between collaborating institutions, between the NIHR, the academic and the industry partner, and with the Health Research Authority). Frequently named issues were in relation to setting up study sites, for example coverage of excess treatment costs (ETCs) and working with the NHS (both NHS R&D departments during set-up and NHS staff during study implementation). In describing their challenges, CIs often discussed how one issue influenced another. For example, about one-quarter of CIs (11/46, 24%) described how delays to contracting, site set-up and/or ethics approval had caused delays in other areas, such as recruitment.

FIGURE 22.

Challenges reported during project implementation. Data labels indicate the number of projects (n = 46). Source: Technopolis analysis of survey data.

In interviews, CIs of completed studies most commonly reported challenges with recruitment (14/23, 61%), half of which were underpinned by problems at the NHS trial site, such as low priority of research within NHS and a lack of resources and/or research capabilities. A more detailed analysis of this issue is provided in Box 3. One interviewee commented:

None of the NHS hospitals seemed to have time to do research. So recruitment is a problem, because the sites do not take this as important. They do their work and have a clinic to finish. They don’t want to sit and talk to a patient about recruitment to a study. So they just skip recruitment.

Another CI explained that:

There were hospitals that seemed to be using the central labs as a cheap way of getting [sample analysis], but actually entered very few patients into the randomised controlled trial part of the study.

A third CI highlighted:

I think EME should know that NHS trusts just do not have the funding to do ‘that stuff’.

The last comment is also reflected in issues with coverage of ETCs, reported by eight CIs in the survey as a major barrier (8/46, 17%).

As a result of issues with recruitment, four CIs explained in interviews that they were unable to carry out the planned mechanistic study.

Thirty per cent (7/23) of interviewed CIs reported delays due to contracting processes, which were frequently described as ‘complex’ and ‘slow’, with overly burdensome administrative requirements on the CI, and were related to academic, clinical and industry partners. Issues with these processes often led to delays in setting up study sites (e.g. when dealing with R&D departments and around ETCs). A few CIs said that they would have liked more support from the NETSCC or EME programme team in overcoming contracting challenges. For instance, one CI suggested that funding be made available prior to finalising contracting with the NIHR to allow recruitment of a trial manager who can relieve some of the administrative burden.

Around 20% of CIs [22% (10/46) of survey respondents and 17% (4/23) of interviewees] reported delays due to regulatory processes [e.g. as a result of compliance with drug and medical device regulations, differences between countries, procedures for importing drugs for trial use and Medicines and Healthcare products Regulatory Agency (MHRA) inspections]. To illustrate this point, one CI explained that in Clinical Trials of Investigational Medicinal Products, sample analysis for outcome measurements has to take place in appropriately regulated laboratories. This includes any laboratory work carried out as part of nested studies (e.g. mechanistic substudies), which poses an administrative challenge for EME project teams. The CI suggested that this barrier be avoided by implementing the trial and mechanistic study via two separate study protocols.

Other issues CIs highlighted in the survey and interviews included:

• Delays in research progress due to limited access to data held by the CTUs, including from mechanistic aspects of the study. As a consequence, researchers are unable to make full use of the data (e.g. applying bespoke analysis tools) (5/69, 7%). In relation to this aspect, a CI commented that it was helpful to have a project manager who works directly for the CI rather than the CTU, particularly on the data management side, as this made negotiations easier.

• A change in the availability of the drug investigated (7/69, 10%).

• Delays caused by the NIHR’s slow award contracting processes (5/69, 7%), coupled with ‘unrealistic expectations of the time it takes to set up trial sites following the award’.

Chief investigators with no or little previous experience in leading trials frequently cited the steep learning curve they had to master, particularly with respect to ‘paperwork’, such as the Investigator’s Brochure and documents to import medicinal products for trials.

Issues were also often reported for studies that had to set up research structures, for example in ‘novel’ areas of investigation, across care settings or research communities, or for working with patient populations or providers that had not participated in research prior to the EME project. These issues often related to unexpected requirements, such as joint approval by NHS and academic units at each site, or a different level of certification of the pharmacy supplying the intervention for the trial compared with routine care.

Challenges to research resulted in 13 EME projects that were not implemented. Of the 53 completed/published studies, a total of 13% (n = 7) projects were initiated but stopped early. This includes two trials that were unable to recruit a sufficient number of participants [one because of a change in National Institute for Health and Care Excellence (NICE) guidance and another because of a competing trial], two trials that were terminated because of safety concerns, two that did not meet a milestone (therefore reducing unnecessary further costs) and one that had a very low level of adherence. A further six projects (not included in the 53 ‘completed/published’ count) were not initiated or were stopped very shortly after initiation. This includes two trials that failed to secure ETCs at study sites, two studies that encountered issues with the supplier of the drug under investigation, one trial that was unable to recruit patients because of a change in NICE guidance and one study for which research findings published by a different group during the approval process affected the planned trial.

Adjustments to the project plan after the start of the project were reported by 96% (43/45) of survey respondents. The most commonly reported adjustment was to the study timeline (34/45, 76%) and the recruitment target number (10/45, 22%) (Figure 23). Similarly, interviewed CIs explained that adjustments to the original project plan included extensions to the project timeline (10/23, 43%), an increase in the number of trial sites (6/23, 26%) or a broadening of inclusion criteria (4/23, 17%). Changes to the project plan were also evident in the number of costed extensions to EME projects. Programme monitoring data show that, of the 121 projects for which contracts had been signed by 2016, 40% (n = 48) were provided with additional funding, amounting to a total of nearly £6.5M at an average of £134,000 per project. The share of commissioned awards granted cost extensions was not significantly different from that of researcher-led awards [17/49 (35%) vs. 12/41 (29%)]. As more projects advance towards the later stages, additional cost extensions may be requested.

FIGURE 23.

Adjustments to the project plan after the start of the project. Data labels indicate the number of projects (n = 45). Source: Technopolis analysis of survey data.

Given their time again, the majority of CIs would make changes to the design and implementation of their projects [59% (13/22) of interviewed CIs and 76% (35/46) of surveyed CIs], with 56% (26/46) considering minor changes and 20% (9/46) major changes. The majority (21/35, 60%) of surveyed CIs who would make changes reported that they would adjust the study timeline to account for delays due to contracting and site set-up and to allow more time for recruitment (Figure 24). CIs would also make changes to the study design, such as the target recruitment number, the type of data collected and outcomes measured (6/35, 17%), or would increase the budget requested for the trial (stated under ‘other’) (6/35, 17%).

FIGURE 24.

Adjustments that CIs would make to their project with hindsight. Data labels indicate the number of CIs (n = 35). Source: Technopolis analysis of survey data.

Interviewed CIs who would make changes in hindsight explained that they would place stronger emphasis on relationships with, and commitment from, partners, such as NHS sites, CTUs or CRNs, in the early project phases (4/23, 17%), would broaden the trial inclusion criteria (2/23, 9%) or would focus on mechanistic research rather than conducting a trial (as their EME trial had been ‘too early’) (2/23, 9%). One CI elaborated on the changes envisaged, citing both a pilot to assess recruitment and hiring of dedicated staff to assist in study set-up and shut-down:

The recruitment rate to our study was much slower than expected and was based on overambitious rates suggested to us by sites. We would probably have an initial phase to assess and check recruitment rates for a fixed number of sites and then base the number of additional sites that might be needed on that pilot phase. I would also consider having additional Trial Manager staffing to help drive forward and focus mainly on the opening of sites at various periods and to support the overall set-up of the study, and similarly co-opt in additional support in the latter stages of the study to help with far-from-trivial work required to shut down the study, data collection, invoicing and fulfil the demands of the sponsor as well regarding monitoring requirements.

In terms of enablers, five CIs emphasised the crucial role of a funded CRF position, highlighting the ability of CRFs to understand the research, work autonomously and liaise authoritatively with third parties, such as regulatory agencies and committees. As one CI explained:

The CRF really was absolutely vital because there were so many day-to-day clinical governance [steps] or decisions required, at least in this first start of the trial. The CRF was really essential. And I think in general, these people seem to be very expensive – and they are because they are qualified doctors. But for me, this was invaluable and really very important.

Other comments included ‘The CRF was very valuable, absolutely key to running the project’ and ‘It [the project] would have been impossible without the dedicated CRF’.

EME project findings

A total of 53 EME projects had completed and/or published their main findings in the scientific literature by September 2020, with 43 projects implementing clinical trial methodology. Of the clinical trials, 14% (6/43) showed that the intervention tested had a positive effect, whereas 74% (32/43) showed that the intervention tested did not have an effect on the primary outcome of the trial. At least 12% (5/43) of projects demonstrated effects on secondary outcome measures in the absence of an effect on the primary outcome measure. Two trials were inconclusive because recruitment did not reach the required target number but the results indicated that the intervention did not have an effect. Two further trials were able to recruit only a very small number of patients. One of these trials provided some proof-of-concept evidence that supported further development of the intervention (but a conclusion on the treatment’s efficacy was not possible) and the other demonstrated that the trial, as planned, was not feasible. One additional trial showed ‘no effect’ in the development phase and was, therefore, discontinued, avoiding research waste. Trials that showed an effect on the primary outcome, as well as those that did not, resulted in important findings that can underpin scientific and health outcomes:

• Seven clinical trials provided strong evidence that current clinical practice is either ineffective (n = 3) or potentially harmful (n = 4).

• Four trials tested novel approaches to treatment, which are being taken forward for further development (Box 4).

• Two trials demonstrated that an adaptive trial design can inform the decision to proceed or stop clinical evaluation based on an early understanding of the efficacy of the intervention, therefore reducing the risk of ‘research waste’. Another (closed) trial successfully employed an adaptive randomisation algorithm to allocate patients to treatment arms.

Ten completed projects followed methodologies other than clinical trials methodology. Four studies tested or validated diagnostic technologies and approaches. Two of these led to findings that have informed clinical practice or guidelines, and one is currently being validated in a clinical trial. Although the fourth study showed that the diagnostic test was not predictive, development of the technology resulted in a patent. One study, funded through a ‘mechanisms of action’ call, developed a methodology that can support further research, and a feasibility study showed that a trial is feasible.

Four studies were substudies of larger trials. Three of these substudies involved the development of imaging approaches for assessment, or to support treatment, of patients. One study has led to changes in routine practice, another avoided unnecessary cost to the NHS by showing that a cheaper assessment method was appropriate and a third led to important mechanistic findings that have changed the community’s understanding of the disease. (The fourth substudy is marked as ‘completed’, but results have not been published.)

In addition, 36% (16/44) of CIs of active projects indicated that research had already led to findings, as yet unpublished. Twenty-five per cent (11/44) of projects had findings related to the primary outcome of the study, with four showing that the intervention tested had a positive effect and seven showing that it had ‘no effect’ on the primary outcome of the study. Fourteen per cent (6/44) of projects reported findings related to the mechanism of action of the intervention, and one study identified genetic factors associated with differences in patient response to the intervention.

Findings related to the mechanism of action of the intervention often confirmed the ‘no effect’ conclusion of the trial, as was reported by approximately one-third of CIs of closed trials (8/22, including findings on three surrogate outcome measures rather than separate mechanistic studies). Two CIs specifically emphasised that this was important, as it lent confidence to the trial outcomes and enhanced the profile of the research. As one CI explained:

I think ultimately, it [the mechanistic finding] was very important when it came to publishing the study. It is difficult to publish a negative study [i.e. a study showing no effect], and the scientific aspects make it more interesting. It meant we were able to get the findings into a fairly high-profile journal.

A further study changed assumptions about the mechanisms of action of the intervention. Two studies led to a change in the understanding of the underlying disease mechanisms and a further two studies identified markers of underlying genotypes associated with the disease under investigation.

Other findings include the discovery of outcomes-associated markers (four projects) and identification of new patient stratification approaches (three projects). Two studies challenged assumptions about the prevalence of the disease under investigation, one study uncovered strong divergence between current guidelines and practice, and one study developed best practice for patient recruitment.

Research outputs and uptake

Engagement activities

Engagement activities were reported for 93 of the 141 projects in Researchfish. The overall number of activities was 840, ranging from 1 to 87 activities per project, with a median of five activities. The most frequently reported forms of engagement were ‘a talk or presentation’ (75/93, 81%), followed by ‘participation in an activity, workshop or similar’ (54/93, 58%) and ‘a formal working group, expert panel or dialogue’ (37/93, 40%).

These dissemination activities were for a range of audiences, with most projects targeting professional practitioners (80%, 74 of 93), followed by public/other audiences (51%, 47) and patients, carers and/or patient groups (37%, 34) (Figure 25).

FIGURE 25.

Primary audience of engagement activities by share of projects. Data labels indicate the number of projects (n = 93). Source: Technopolis analysis of 2020 Researchfish data.

Awards and/or recognition

Awards and/or recognition were reported for 60 projects in Researchfish. The number of awards/recognitions per project ranged from 1 to 64, with a median of three. The awards/recognitions most frequently related to ‘personally asked as a keynote speaker to a conference’ (37/60, 62%), followed by ‘prestigious/honorary/advisory position to an external body’ (24/60, 40%) and ‘research prize’ (21/60, 35%) (Figure 26).

FIGURE 26.

Awards and recognitions, by type and by share of projects. Data labels indicate the number of projects (n = 60). OBE, Order of the British Empire. Source: Technopolis analysis of 2020 Researchfish data.

Many CIs highlighted that the awards and recognition had led to increased awareness of their research and had enhanced their profile in the research community (Box 5). Other comments included that the award had helped facilitate networking opportunities and dissemination.

BOX 5 - Supporting the career of CIs: the CLARITY trial

PDR is a complication of diabetes that can lead to severe vision problems. The CLARITY trial33,34 set out to compare the efficacy and cost-effectiveness of a novel treatment for PDR, that is, the injection of aflibercept (Eylea^®, Regeneron, Tarrytown, NY, USA/Bayer Pharma AG, Berlin, Germany), with the NHS standard treatment. The CLARITY trial was led by Professor Sobha Sivaprasad, Moorfields Eye Hospital NHS Foundation Trust (London, UK)/University College London.

The trial demonstrated that aflibercept treatment was superior to standard treatment in clinical outcomes and was associated with fewer adverse effects and higher patient satisfaction. 33,34 The mechanistic substudy to the trial also led to important (and unexpected) insights into the treatment’s mechanism of action. 35 However, the study’s economic evaluation highlighted significant cost implications – an additional cost of £5475 per patient. The high cost of aflibercept treatment alongside concerns about patients failing to seek the necessary follow-up prevents adoption by the NHS. 36 Evidence from the CLARITY trial has, however, informed regulatory decisions and clinical practice abroad. 37

Leading the CLARITY trial has had a lasting impact on Professor Sivaprasad’s research career. She has presented the CLARITY trial’s findings at more than 40 international meetings and the trial continues to be discussed in reviews, conference panel discussions and professional magazines from across the globe. 38–42 The CLARITY trial has also helped to place the UK on the map for ophthalmology research by demonstrating the UK’s capabilities in successfully implementing multicentre ophthalmology studies to a high level of quality. Since the CLARITY trial, Professor Sivaprasad has been the CI for a further 15 trials and was approached by other research groups to be co-applicant on grant proposals as a result of her increased profile. Professor Sivaprasad has also won numerous awards and nominations. For example, she was named ‘Researcher of the Year 2017’ by the NIHR and the Royal College of Ophthalmologists,43 has received the Royal College of Ophthalmologists’ Nettleship Medal for the CLARITY trial as ‘the best piece of original work by a British ophthalmologist’44 (reproduced with permission from The Royal College of Ophthalmologists, 2021) and was appointed to the HTA board for NIHR commissioned calls and as chairperson for a Clinical Study Group of the UK-wide Ophthalmology Clinical Research Strategy. 45

An extended case study is available in Report Supplementary Material 1.

CLARITY, Clinical Efficacy and Mechanistic Evaluation of Aflibercept for Proliferative Diabetic Retinopathy; PDR, proliferative diabetic retinopathy.

Research outcomes

Collaboration on new projects

Around half of the CIs who responded to the survey indicated that they were collaborating on new projects with members of the EME project team, referring to both established and new collaborators that had participated in the EME project [18/37 (49%) and 19/40 (48%), respectively] (Figure 27). Approximately one-third (9/31, 29%) of the respondents reported that they were collaborating on a new project with existing commercial partners and three pointed to interest in the study findings from industry, indicating the potential for future collaboration.

FIGURE 27.

Reported further collaborations with (a) existing team (n = 37); (b) existing industry partners (n = 31); and (c) new collaborators (n = 40). Data labels indicate the number and of share of projects. Source: Technopolis analysis of survey data.

Collaborations and partnerships

Around half of the EME projects reported in Researchfish provided details on collaborations (74/141, 52%). In total, 347 collaborations were reported, ranging from 1 to 28 collaborations per project, with a median of two collaborations. Most collaborations were based on a formal agreement (260/347, 75%). (These figures are likely to represent a mix of collaborations in the EME-funded project itself and collaborations that resulted from the EME-funded research.)

Chief investigators of two-thirds of the projects collaborated with academic institutions/universities (50/74, 68%), whereas approximately one-third each collaborated with the private sector (25/74, 34%), with hospitals (24/74, 32%) and with the public sector (23/74, 31%) (Figure 28).

FIGURE 28.

Collaboration partners by sectors and by share of projects. Data labels indicate the number of projects (n = 74). Source: Technopolis analysis of 2020 Researchfish data.

The vast majority of collaborations involved in-kind contributions (282/347, 81%), whereas only 13% (45/347) involved direct financial contributions. Collaborations with the private sector were more likely to involve direct financial contributions. In contrast, 93% (57/61) of contributions from the public sector were in kind.

Where reported, collaborators were most commonly located in the UK (215/329, 66%), followed by other European countries, especially Germany and Italy (66/329, 20%), and the USA (27/329, 8%).

Access to expertise and infrastructure

The majority of projects made use of external expertise and infrastructure. For example, surveyed CIs reported accessing CRNs (34/46, 74%), NHS resource (mainly staff time) (25/46, 54%) and BRCs (17/46, 37%) (Figure 29). Several CIs (n = 9) elaborated further on the role of research infrastructure, with recruitment being the most commonly reported role (n = 5), followed by data collection (n = 3) and project delivery (n = 3).

FIGURE 29.

Research infrastructures contributing to EME projects. Data labels indicate the number of projects (n = 46). Source: Technopolis analysis of survey data.

In interviews, 26% (6/23) of CIs said that a CRN had been a key enabler of their project, and emphasised the importance of partnering with the ‘right’ CRN, particularly for recruitment. In line with these views, five of the projects achieved their recruitment target. One CI commented:

I think that [recruitment] was always the biggest risk, people would say ‘It’s too difficult, you can’t do it’. That was where the CRN made a massive difference. This was not a trial that could have been done before the network had been put in place.

Another said:

The CRN played a critical role in delivering the trial.

Two interviewees also mentioned that they had made use of BRC (i.e. support from a research nurse) or BRU infrastructure (i.e. use of treatment delivery infrastructure). Other existing infrastructure also provided important support, as one CI commented:

I wouldn’t have been able to get the [EME] grant if I hadn’t had the NIHR infrastructure capital award previously.

However, 11% (5/46) of CIs surveyed described challenges in working with a CRN (e.g. lack of expertise, not helping ‘beyond a few phone calls’ or even adding hurdles to the project).

The majority of CIs (35/45, 78%) reported that their projects had accessed the appropriate expertise and infrastructure. Four CIs elaborated that, although they had the appropriate expertise and infrastructure, they had either encountered challenges with setting up the collaboration or experienced ongoing concerns with expertise and/or infrastructure (e.g. unavailability during certain time periods and a new trial site lacking capabilities).

A smaller share of CIs indicated that their project was missing critical infrastructure (7/45, 16%). A few researchers pointed out that devolved nations are at a disadvantage, as they cannot draw on BRU/BRC infrastructure. This corresponds with a comment from a CI based in England:

I’ve got the benefit of working within a BRC where we’re able to fund early translational work and get pilot data for a grant. So I may be speaking from an unrepresentative position, but probably most of the people who get these types of grants are those types of institutions.

Patient and public involvement

All surveyed CIs (n = 46) and 91% (20/22) of interviewed CIs had engaged in PPI at some stage of the EME project. Of completed projects, 75% (12/16) engaged in PPI during the design phase and 69% (11/16) during project implementation. Among active projects, 65% (30/46) engaged in the design phase and 50% (23/46) in the delivery phase, with the majority involving three or more PPI representatives.

Patient and public involvement is also embedded in the selection of projects: the EME Funding Committee includes two PPI representatives with full voting rights on funding recommendations. One interviewee described the EME programme as advanced in terms of PPI. EME Funding Committee members, including the PPI representatives, do not have sight of project monitoring data, such as annual reviews, and, therefore, do not know if and how PPI is implemented in practice as part of the research. This was considered a missed opportunity for learning.

The majority of surveyed CIs who elaborated on their project’s approach to PPI reported that PPI representatives were members of the TSC or Trial Management Group (25/37, 68%). Most representatives were patients (17/28, 61%), followed by patient representative groups (14/28, 50%) and parents (6/28, 21%). Four CIs reported that they had a PPI representative as a co-applicant.

Patient and public involvement contributed most frequently to the development of patient-facing information (39/45, 87%), the development of the research methodology and approach (25/45, 56%) and the design of the recruitment or retention strategy (21/45, 47%) (Figure 30). Interviewed CIs highlighted the role of PPI in ensuring that the study protocol and intervention are acceptable to patients (n = 9), in shaping of documentation for and communication with patients (n = 4), in disseminating the EME project results (n = 3) and in supporting recruitment for the trial (n = 3, two of whom described the support as ‘invaluable’).

FIGURE 30.

Area of contribution of patient/public involvement. Data labels indicate the number of projects (n = 45). Source: Technopolis analysis of survey data.

Forty-six per cent (21/46) of surveyed CIs felt that PPI had enabled their research ‘to a large extent’ in the project design phase, with 48% (22/46) reporting that it had done so ‘to a limited extent’. In the delivery phase, a lower share of CIs (35%, 16/46) indicated that PPI had enabled the project ‘to a large extent’ and 52% (24/46) suggested ‘to a limited extent’ (potentially reflecting the fact that many projects are still in the early phases). CIs who felt that PPI was an important enabler further explained that the patient perspective had been a key factor behind the success of the project. One CI wrote ‘I cannot emphasise how transforming it has been to have [PPI representatives] work so closely with the team’. Key contributions included shaping the project idea, assisting recruitment and ensuring that information was appropriate for patients. Similarly, four interviewed CIs emphasised that PPI had significantly contributed to shaping the project design, with two CIs providing examples of changes made to the study protocol that would have posed a risk to the ability to recruit patients. In one case, PPI representatives were involved in direct communication with the regulator, setting out the need to consider not only risks, but the scale of the needs to be addressed. The CI felt that this had been important in the MHRA’s decision-making and subsequent approval of the trial.

Chief investigators who felt that PPI was a limited enabler in both the design and the delivery phases of the project explained that this was because of project-specific challenges or that PPI would have a more important role later in the project.

Health and health systems outcomes and impacts

Of the completed EME projects that followed trial methodology (n = 43), seven (16%) have informed clinical guidelines (including one guideline completed but not yet published at the time of the evaluation) and eight have potential to do so in the future based on the nature of their findings. Therefore, up to 35% of EME trials may inform guidance and practice, demonstrating that EME-funded research is generating important evidence for health-care decisions. This applies to projects showing the benefits of an intervention, as well as projects showing that the intervention does not have an effect.

Of the six trials that showed that the intervention had a positive effect on the primary outcome, three have been cited in clinical guidelines or informed regulatory approval of an intervention and a fourth (recently published) trial has the potential to do so in the future. However, although one of these trials demonstrated that the intervention is beneficial, the EME project included an economic analysis that showed a lack of cost-effectiveness (see Box 5). As a result, UK guidance does not recommend use of the intervention, but the evidence has contributed to a successful application for regulatory approval to the US Food and Drug Administration. Another trial has led to substantial investment from industry for further development of the underlying technology.

Of the 35 trials that showed that the tested intervention does not have an effect on the primary outcome, three have informed clinical guidelines and a fourth is expected to do so in the next update of the relevant guideline. Two of these trials showed potential safety issues of treatments currently in clinical use and led to guidelines recommending an alternative. A further seven trials have the potential to inform clinical guidelines and this includes evidence from trials showing that current practice is ineffective (n = 2) or potentially harmful (n = 2), as well as evidence of the (lack of) cost-effectiveness of shifting to a novel treatment approach under current conditions. Another trial provided proof of concept that the platform developed as part of the research can be used to test other agents, and has led to strong commercial interest.

Trials showing that an intervention does not have an effect are recognised as important in investigating potential treatments and yielding research outcomes, such as an enhanced understanding of disease and increased capacity for implementation of trials with a given patient population. As one CI commented:

The negative findings [i.e. findings showing no effect] of the trial were unintended, but at least the clear negative result helped to remove one possible candidate from the table.

The 10 completed/published projects that followed methodologies other than trial methodology also led to several health outcomes:

• Three substudies successfully supported ‘their’ clinical trials, all of which have resulted in changes in clinical guidelines or practice. Two of the trials showed that routine practice was safe, which lent confidence to treatment decisions, and in one case saved costs (as the alternative approach would have been more expensive). The third trial developed an approach that is now routinely used to support surgical procedures (Box 6).

• One project that developed a diagnostic tool has contributed to evidence that led to its uptake in clinical guidelines. The diagnostic test is now routinely offered in the NHS.

• Another study showed that a patient screening approach to inform treatment decisions was more effective than the current standard. Although the evidence was cited in clinical guidelines, a change to current practice was not recommended on the basis of the study’s cost–benefit analysis.

In addition, the CI of an active trial reported that the study had already led to changes in how PHE tests for the condition under investigation, which were expected to be rolled out nationally in the future. Findings from another active trial were cited in a guideline. Impacts on patient and population health may accrue further from the 53% (77/145) of projects that are currently ongoing. The four HRCS health categories accounting for the largest share of active projects are ‘oral and gastrointestinal’ (12/77, 16%), ‘cancer’ (9/77, 12%), ‘reproductive health and childbirth’ (8/77, 10%) and ‘cardiovascular’ (8/77, 10%). This indicates that health benefits may accrue in these areas in the future (while recognising that many factors affect the potential for impact, such as the maturity of the research field).

In Researchfish, CIs reported an ‘influence on policy’ for 36 projects, with a total of 86 instances of policy influence, ranging from 1 to 18 per project, with a median of two. The most common type of influence was ‘membership of a guideline committee’ (17/36, 47%), followed by ‘citation in clinical guidelines’ (11/36, 31%) and ‘influenced training of practitioners or researchers’ (9/36, 25%) (Figure 31).

FIGURE 31.

Type of policy influence by share of projects. Data labels indicate number of projects (n = 36). Reports of ‘citations in clinical guidelines’ were validated through desk research and was confirmed for seven of the 11 reports. No link between three reports of guidelines citations and the corresponding EME awards could be established (e.g. one guideline citation was related to the topic of the EME award, but the publication cited referred to a HTA-funded trial). Only the seven confirmed projects are included in the outcomes analysis. Source: Technopolis analysis of 2020 Researchfish data.

Economic impacts

EME programme design and management

Review process and feedback

Resubmissions were requested for 44 applications at either stage 1 (n = 29) or stage 2 (n = 16) of the review process (one application, rejected at stage 2, had been resubmitted by invitation at both stages). Nearly half (13/29, 45%) of the applications resubmitted at stage 1 proceeded to stage 2 and 28% (8/29) of these were ultimately funded. Of applications resubmitted at stage 2, 69% (11/16) were awarded funding. This indicates that the quality of applications improved as a result of feedback from the EME Funding Committee.

The majority of surveyed CIs (40/45, 89%) reported that the feedback they received from the EME Funding Committee was helpful (Figure 32). Responses were more mixed among interviewees. Of the five CIs who commented on this aspect, two stated the feedback had been helpful, with one explaining:

I think the feedback at each stage is excellent. It really sets out with clear direction to the applicants what they want them to do [. . .]. It’s the best feedback I’ve seen from any grant award body.

Conversely, the other three CIs felt that expertise was missing from the review process or that feedback had been inappropriate. For example, one CI stated:

The review was too prescriptive. I was asked to include another test [in the research project], which I knew I would not be able to recruit for. This was indeed the case.

FIGURE 32.

Share of CIs who reported as helpful support/feedback from (a) the EME programme team at the proposal stage; (b) the EME Funding Committee at the proposal stage; and (c) the EME programme team at the project implementation stage. Source: Technopolis analysis of survey data (n = 45).

Perhaps not surprisingly, unsuccessful applicants were less positive about feedback provided by the EME Funding Committee. Fifty per cent (14/28) of respondents had found the feedback ‘not helpful’. Issues discussed included receiving contradictory feedback (n = 4) (e.g. contradictions between stage 1 and stage 2 feedback, or between reviewer comments and funding committee feedback), receiving too little feedback (n = 3) and a lack of clarity about whether or not the proposed projects fitted the EME programme’s remit (n = 3). Two respondents felt that the feedback reflected a misunderstanding of research needs. Three respondents thought that the application process was extremely burdensome, with a further two respondents reporting that their final stage applications were denied for reasons that should have been flagged to the CI at stage 1 (e.g. research being out of scope).

Evaluation question 1: what value does the EME programme bring to the funding landscape for the development and assessment of health interventions?

The EME programme is a competitive funding scheme, with a success rate of 18.5% over the 2009/10 to 2018/19 period. EME application success rates are comparable to, if slightly lower than, application success rates for the NIHR HTA programme (an average of 19.6% for the EME programme vs. an average of 23.2% for the HTA programme over the 2015/16 to 2018/19 period)61 and similar to those of the NIHR RfPB scheme (an average of 19.5% for the EME programme vs. an average of 20.7% for the RfPB scheme over the 2016/17 to 2018/19 period). 62 The average success rate for EME applications is also broadly in line with success rates of MRC grants, at 20.2% and 23.2%, respectively, for the 2011/12 to 2017/18 period. 63

By 30 September 2018, 145 EME projects were contracted for a total of £176M, with an average project cost of £1.2M. The programme fills a gap in the funding landscape (as agreed by 89% of CIs consulted), bridging the gap between early proof-of-concept and effectiveness studies. Within the funding landscape, the programme sits among other MRC and NIHR schemes and grants provided by charities, mainly the Wellcome Trust, British Heart Foundation and CRUK, and also many smaller disease-specific charities:

• Around three-quarters of projects reported on funding prior to the EME award. Funding sources were mainly UK public funders (58% of projects) and charities/non-profit organisations (46%), with funding from industry supporting a smaller share of projects (17%). Previous support from the NIHR was mainly through BRCs/BRUs and the RfPB scheme. Many EME awards (n = 5) also cited NIHR HTA studies as the underpinning research. Little detail was provided on which MRC schemes supported preceding work and, interestingly, the MRC DPFS was not mentioned.

• Follow-on funding was reported for more than one-third of EME awards (52/141, 37%), with numbers likely to rise as more projects complete. The share of projects funded by each sector remained relatively stable compared with pre-EME award funding, at 63% for UK public funders, 56% for charities/non-profits and 15% for industry. However, the share of projects funded by the NIHR and the MRC increased from the pre- to post-EME award stage, from 23% to 42% for the NIHR and from 19% to 33% for the MRC, whereas funding from other public sources, such as the NHS and CSO Scotland, decreased. Further funding from the NIHR included EME awards (n = 4), RfPB awards (n = 3), HTA awards (n = 2), NIHR Research Professorships (n = 3) and NIHR Clinical Fellowships (n = 2). Further funding from the MRC included two awards each from the MRC DPFS, the Stratified Medicine scheme and the Proximity to Development scheme.

• The total value of follow-on funding amounts to approximately £355M. The NIHR provided £155M, the MRC made available £40M and the Wellcome Trust made available £19M.

• Three of the six EME projects that demonstrated efficacy for the intervention tested reported follow-on funding. Of these projects, two are supported by industry and Wellcome Trust grants, and the third is funded by the NIHR RfPB scheme and a charity grant.

The main alternative sources of funding cited were the NIHR HTA programme, MRC research grants and charity funding, but each was associated with different limitations compared with the EME programme.

Chief investigators considered the EME programme unique in providing support for combined efficacy and mechanistic studies, described as a ‘bridge between the MRC and NIHR’ and an opportunity for clinical researchers and discovery scientists to collaborate. Overlap with other sources of funding was considered limited. However, at least some projects are able to access alternative funding sources/schemes. For example, 25% (7/28) of CIs whose application to the EME programme had been unsuccessful went on to conduct efficacy trials. Four of these projects were supported by the NIHR HTA programme, the JP Moulton Charitable Foundation, the MRC DPFS scheme and the British Heart Foundation. Other CIs continued with mechanistic aspects of the study (11/28, 39%), for example with funding from the Wellcome Trust and CRUK, or a feasibility study, but did not secure funding for an efficacy trial.

Of projects that reported co-funding (n = 62), most were co-funded by the pharmaceutical industry (50%) and/or medical technology and health-care SMEs (37%) and charities (19%). Most co-funding was provided in the form of free or reduced-cost interventions (42/62, 68%) or placebo (19/62, 31%). Eight-seven per cent (n = 54) of these projects quantified the co-funding provided, reporting a total value of £15M.

A need for funding for proof-of-concept studies was identified to bridge the gap between discovery research and the EME scheme. One interviewee also pointed to a gap between research and uptake by industry and the NHS, such as commissioning and ‘getting the solution to the patient’. While DPFS is positioned to act as a feeder for EME, providing funding to support preclinical and early phase clinical studies, none of the CIs indicated that an EME award had been preceded by a DPFS grant. In line with this finding, an evaluation of the MRC’s translational research portfolio found that, of 24 DPFS and MRC Regenerative Medicine Platform research projects, 20 had secured follow-on funding from industry and only three from the NIHR and PHE. 64 Interest from industry in the DPFS portfolio may be stronger because of its focus on new product development, with two-thirds of CIs intending to exploit any IP arising through commercial routes. 65

On the other hand, the NIHR HTA scheme emerged as both a source of follow-on funding from EME studies (n = 2) and a source of evidence and ideas for EME projects (n = 5), demonstrating that research in the later stages of the translational pathway informs research at the earlier stages. Therefore, it is clear that the concept of a linear DPFS–EME–HTA funding pathway does not correspond to actual funding pathways. Nevertheless, the EME programme was described by five CIs as a bridge between MRC grants and the NIHR HTA programme, combining NIHR-orientated efficacy studies and MRC-orientated discovery research via mechanistic study components.

The EME programme brings value to the broader funding landscape by enabling efficacy studies of limited or no commercial interest. Just under half (49%) of the projects in the EME portfolio targeted repurposing of existing interventions and 17% generated evidence to test current practice. Although approximately one-third (34%) of projects supported the development or uptake of new therapies and approaches, many of these did not target commercialisation. For example, some projects tested new technologies under development by overseas companies to determine whether or not they are superior to the current standard of care. Other projects investigated a new approach to stratify patients for a specific therapy. This important evidence is needed to inform regulatory and treatment decisions and is, therefore, important to ensuring high-quality care. Such projects would, however, not be expected to lead to commercial outcomes, such as new IP, licensing agreements and spin-outs, nor to economic impacts, such as increased GVA or job creation (other than for the commercialising company).

Similarly, less than one-third (29%) of CIs of active awards indicated that their projects have the potential to lead to commercial benefits, whereas most interventions tested present limited commercial opportunities. This is also reflected by the fact that half of CIs of active EME projects did not seek industry co-funding, because the intervention tested was not of commercial interest or relevance (e.g. off-patent and trial testing treatment withdrawal) or because it was deemed important to conduct the trial independently from industry. In addition, one interviewee explained that, for interventions of commercial interest, industry may prefer to ‘go it alone’ in the earlier stages of development. Another CI highlighted that development of new therapies would require larger budgets than usually provided by EME awards.

Therefore, the programme mainly funds research that generates evidence to inform existing interventions by testing and/or expanding their use (repurposing). This is also reflected in the origin of EME research ideas, which stemmed mainly from clinical researchers (73%), academic researchers (51%) and health professionals (27%), often based on personal experience or identification of a specific clinical need. Only 4% of project ideas originated with industry partners. The reason for the high share of repurposing studies in the current portfolio is unclear. This could be investigated further, for example by reviewing whether or not applications proposing to develop new therapies and approaches are less successful or by reviewing whether or not fewer applications are received.

The value the EME programme brings to the funding landscape is acknowledged by the wider research community, including key opinion leaders consulted as part of this evaluation, and a recent evaluation of the MRC translational research portfolio, which commented that the scheme ‘had made repurposing an acceptable research area for academic researchers’ (reproduced with permission from the MRC, 2021, personal communication). 66 Similarly, one survey respondent explained that ‘We should vigorously defend EME’s courage in supporting studies that could lead to health gains but not necessarily money making for industry’. Therefore, the EME programme addresses a research need in areas where industry funding is unlikely to be made available. In addition, research targeting repurposing or providing evidence to inform use of existing interventions is likely to provide good value for money when compared with the more costly and higher-risk development of new interventions, and lead to improved efficiencies in the health-care system.

In summary, the EME programme is considered a unique programme in the research funding landscape. The programme provides an important bridge between discovery and effectiveness research. Before the EME programme was initiated, efficacy studies were either funded by industry (if of commercial interest) or were conducted as small investigator-led projects. EME funding enables multicentre efficacy trials while also advancing our understanding of the underlying biology through mechanistic studies. This combination is seen as extremely valuable and sets the EME programme apart from other schemes.

Evaluation question 2: has the EME programme attracted/commissioned research projects in areas of interest, importance and strategic need for UK government, patients, the NHS and other key stakeholders?

The EME programme attracts research projects via two main avenues: (1) researcher-led calls and (2) commissioned calls. Commissioned calls can be specific to the EME programme, preceded by horizon scanning to identify gaps, or can be part of broader cross-NIHR ‘themed calls’.

Efficacy and Mechanism Evaluation projects were funded across nine of the top 10 causes of DALYs in the UK (2017). Commissioned calls are likely to have contributed to this alignment. For example, of the seven projects investigating diabetes-related conditions, which are among the top 10 causes, five were funded through commissioned calls. Furthermore, the share of projects for a range of HRCS health category and activity codes differ between the commissioned and researcher-led research portfolios, indicating that commissioning has an influence on awards made. For example, a larger share of commissioned projects was associated with the category ‘metabolic and endocrine’ (16% vs. 4%), whereas the share of ‘cardiovascular’ projects among researcher-led projects was nearly twice that of commissioned projects (19% vs. 10%). Overall, CIs did not raise any issues with the EME programme’s parallel mechanisms of researcher-led and commissioned calls.

Projects in the EME portfolio address a range of UK health needs and several government priority areas, but do not fully align with the level of health need (expressed in DALYs) encountered. For example, four of the 10 most frequent causes of DALYs are targeted by only one EME project or not at all, and based on their share of UK DALYs, the areas ‘injuries and accidents’ and ‘musculoskeletal’ are under-represented in the EME portfolio, at about one-quarter to half of the UK DALY share. Conversely, the areas ‘reproductive health’, ‘metabolic and endocrine’ and ‘oral and gastrointestinal’ are highly represented in the EME portfolio, with portfolio shares larger than their share of UK DALYs (see Appendix 4 for details of and caveats to this analysis).

Some of the misalignment between the focus of EME projects and the level of UK DALYs or government priorities may be due to a smaller number, or a lower quality, of applications received in certain areas of research. Of the 84 commissioned calls, 39% (n = 33) received only one, two or three applications and 44% (n = 37) did not lead to any funded projects. Calls in some targeted research areas (e.g. COPD, sleep disorders and mental health) have led to few successful applications. The underlying reasons for the small number of awards in certain areas or through certain commissioned calls are unclear. Potential reasons include that a narrowly defined call restricted the pool of potential applicants, that a health category is addressed by relatively few researchers in the UK or that R&D in the field is not at the right stage for efficacy studies. It is also possible that certain types of research are funded through other funding sources and do not need funding through the EME programme, or that the research community is not sufficiently aware of the scheme. Each research area is likely to be associated with its own set of determinants and a closer examination can help to address barriers to progress.

Furthermore, although the largest share of projects was associated with the HRCS research activity code ‘evaluation of treatments and therapeutic interventions’ (87%) and investigated pharmaceutical interventions (58%), projects to explore aetiology, prevention and management of diseases were largely absent from the EME portfolio. It was suggested that the EME Funding Committee may be challenged to cover non-traditional health research areas of interest, which limits funding awarded to more innovative approaches and multidisciplinary research across the medical and social sciences. Discussions with the Economic and Social Research Council (ESRC) on bringing social science into the EME programme could help to address this gap, building on other ESRC funding partnerships with the NIHR and MRC, such as the Dementia Research Initiative and the Tackling Multimorbidity at Scale Fund.

Evaluation question 3: what should the EME programme do more of to achieve greater impact?

The PLM identified intended long-term impacts for the EME programme at both the project and the programme levels. At the project level, impacts relate to the health domain (i.e. health benefits and efficiency gains in the health system) and the commercial domain (i.e. economic benefits from commercial activity). At the programme level, expected impacts include influence on the wider research funding agendas. EQs 13–15 provide detail on project-level outcomes and impacts to date (see Table 1). Here, opportunities for enhancing impacts are explored.

The EME programme is firmly rooted in clinical research, with nearly three-quarters of CIs reporting that the original idea for their research topic was informed by clinical researchers. Interestingly, compared with successful applicants, a larger share of unsuccessful applicants counted patient groups (46% of unsuccessful applications vs. 17% of successful applications) and the literature (32% vs. 11%) among the origins of their research ideas. PPI can provide an important steer towards research areas with high potential for patient benefit and health impact. The finding that ideas that originated from patient groups were less successful, therefore, merits further investigation. For example, whether or not these applications were outside the CI’s usual area of research and, therefore, of lower quality (e.g. if the applicant has less experience with relevant clinical research), whether or not the proposed research was less suited to the remit of the EME programme, or whether or not the issues targeted were considered a lower priority by the EME Funding Committee.

As set out under EQ2 (see Table 1), projects in the EME portfolio address a range of UK health needs and several government priority areas, but do not fully align with the level of health need (expressed in DALYs) encountered. An analysis of why some areas of research that address important health needs are currently under-represented can inform future strategies for closing research gaps.

As set out under EQ1 (see Table 1), a large share of EME projects focuses on repurposing or providing evidence to inform use of existing interventions. This research can lead to health impact and cost savings. However, most of the interventions tested are of limited commercial interest. To enhance the potential for economic impact, selection of EME projects needs to place more emphasis on research with commercial opportunities. However, it should be born in mind that EME programme addresses a funding gap by supporting research in which industry is unlikely to invest.

Key opinion leaders (n = 3) highlighted that the EME programme’s potential for health and/or economic impact could be enhanced through increased communication with other funders. This would also extend the scheme’s influence on the wider research funding agendas. Although high-level strategy fora, such as the UK CRC Experimental Medicine Funders group, were named, some interviewees felt that discussions at the EME programme level would be beneficial. Co-ordination between funders can support portfolio selection by improving alignment of funding decisions, therefore creating synergies and avoiding duplication. For example, representatives from one charity recommended communication, and ideally consultation, before a call relevant to their disease area is commissioned to allow strategic alignment. Closer working could also lead to identification of opportunities for co-funding, such as substudies of other trials [see Evaluation question 13: what benefits for patients and populations (health impacts) have been achieved by EME-funded research and what benefits are likely to arise in the future?], which leverages EME funding and extends the level of support available to CIs (e.g. through charity networks).

The majority (82%) of CIs consulted considered the overall design of the EME programme appropriate for achieving its aims. Programme management and communication were viewed positively, with the EME programme team being described as supportive, communicative and flexible. One issue raised, by 16% of surveyed CIs, is the programme’s ‘slow’ application and contracting processes.

A number of CIs felt that the requirement to submit to the EME journal in addition to publishing results in peer-reviewed journals duplicates effort without benefit to the research community and, therefore, represents an unnecessary burden on the project team. The EME journal is not indexed in major citation databases and linking to such databases would increase the discoverability of EME project outputs and contribute to outcomes and impacts.

Evaluation question 4: were the EME-funded projects well designed?

The majority of EME projects were designed in a way that enabled their full implementation. Of the 145 EME awards, 59 are no longer active (i.e. marked as completed or discontinued, or published main study results). Nine (15%) of these projects encountered issues that affected delivery of the research, rather than scientific issues, such as safety concerns or not passing a milestone:

• Three projects initiated but were stopped early because of issues with their implementation, such as low recruitment rates and low levels of adherence.

• A further six projects were awarded but did not initiate because of implementation issues related to study set-up, drug suppliers and changes in the research environment.

Feedback from the EME Funding Committee helped to improve project plans. Of applications that were resubmitted after incorporating feedback, 43% (19/44) reached a level of quality that led to a positive funding decision. The majority of surveyed CIs (40/45, 89%) described feedback on their applications as helpful (although a few interviewed CIs felt that the feedback had been too prescriptive). On the other hand, several CIs of unsuccessful applications (n = 4) noted that feedback had been contradictory (e.g. between stage 1 and stage 2 of the process).

The main challenges and barriers encountered by CIs were ‘complex and slow contracting processes’ (76%) and the setting-up of study sites (65%). These required substantial investment of time and resource, and often led to delays. Coverage of ETCs, low prioritisation of research and/or lack capacity of NHS staff were recurring difficulties of working within the NHS.

Patient recruitment was highlighted as a challenge by 61% of CIs. Recruitment issues were also evident in that one-fifth (21%) of trials recruited a smaller number of participants than the target number. Four trials were unable to carry out the planned mechanistic studies because of smaller recruitment numbers. Of projects that were awarded but either stopped or discontinued (n = 12), three had been unable to recruit. Feedback at the review stage on recruitment criteria, strategies and timelines could help to uncover potential issues in the project plan and allow these to be addressed from the outset.

Problems were often described as interlinked; for example, one-quarter of CIs reported that delays to contracting, site set-up and/or ethics approval, and low motivation by NHS staff to engage with research, had caused issues in other areas, such as recruitment. Around 20% of CIs also reported delays due to regulatory processes. In particular, CIs with little experience of leading clinical trials and CIs of trials that required setting up of research structures (e.g. working in research-naive areas of clinical research or across different care settings) struggled with the associated administrative burden. Other challenges included some CIs’ inability to fully analyse study data because of issues with access to data held by CTUs (which precludes the use of bespoke analysis tools), changes in the availability of the tested intervention and the time required for the NIHR’s contractual processes.

In hindsight, the majority of CIs (60%) would make changes to their project’s design and implementation. Most needed to extend the study timeline to account for delays due to contracting and site set-up and to allow more time for recruitment. A few CIs (n = 4) would place stronger emphasis on relationships with, and commitment from, partners, such as NHS sites, CTUs or CRNs, in the early project phases to avoid problems later on.

In terms of enablers of research, several CIs emphasised the crucial role of a funded CRF position for trial implementation, highlighting the ability of CRFs to understand the research, work autonomously and liaise authoritatively with third parties (e.g. regulatory agencies and committees). CRNs and BRCs were also mentioned as important enablers of research.

This evaluation does not have sufficient evidence to determine whether or not issues with project implementation could have been identified at the review stage. In some cases, a small pilot study and/or stronger PPI can assist in shaping the design to avoid these barriers.

Evaluation question 5: what have been the outputs of EME-funded research?

Projects reported their main findings in the EME journal (85%) and in other peer-reviewed journals (70%). Beyond the EME journal, the largest number of studies were published in The Lancet (n = 8) and The Lancet Psychiatry (n = 4). In Researchfish, the majority of projects (74%) reported at least one publication, resulting in a total of 671 publications, predominantly journal articles. (However, because Researchfish captures self-reported data, some of these may be only loosely associated with the EME project for which they are listed.)

Efficacy and Mechanism Evaluation-funded research has created outputs that can be used for further research. These include biological sample collections (18 projects); re-useable research tools (15 projects), such as imaging techniques for assessment of disease symptoms (five projects); shareable data sets (seven projects); image collections (three projects); new outcomes-associated markers (four projects); patient stratification approaches (three projects); apps (two projects); and IP (four projects).

Further publications and reusable outputs from other currently active awards can be expected as research projects progress. Accordingly, 37% of CIs of active projects indicated that the research had already led to findings, as yet unpublished (see also Evaluation question 6: what have been the findings of research funded as part of the EME programme?).

The importance of results from studies that demonstrated ‘no effect’ is clearly recognised by CIs. Although one interviewee explained that findings of trials showing ‘no effect’ are unlikely to be published by high-impact scientific journals, the publication record of the EME programme demonstrates that this is not always the case. For example, five studies showing ‘no effect’ with respect to the primary outcomes of the trial led to articles in The Lancet. Most trials (78%) also investigated a mechanism, and this led to a better understanding of the trial findings (both positive and ‘no effect’), providing ‘publishable’ scientific insight, as well as lending confidence to the findings.

Evaluation question 6: what have been the findings of research funded as part of the EME programme?

Around one-third (37%) of EME awards have completed and/or published main findings. Of these projects, 43 employed clinical trials methodology and 10 followed other methodologies.

Of the 43 completed trials, 14% (6/43) showed that the intervention tested had a positive effect, whereas 74% (32/43) of interventions did not have an effect on the primary outcome. Although ‘no effect’ was demonstrated, the latter would avoid further research costs and provide important evidence to inform use of interventions. Illustrating this point, seven of these studies provided strong evidence that current clinical practice is either ineffective or potentially harmful. Findings related to the mechanism of action of the intervention often confirmed the ‘no effect’ conclusion of the trial, lending confidence to the trial outcome.

Findings from four EME trials tested novel approaches to treatment that are being taken forward for further development. Five trials changed assumptions about the mechanisms of action of the intervention and underlying disease mechanisms, and identified markers of genotypes associated with the disease under investigation. Two studies challenged assumptions about the prevalence of the disease under investigation, whereas one-third of studies uncovered strong divergence between current guidelines and practice.

The 10 completed projects that did not follow clinical trials methodology informed the development and/or use of diagnostic technologies and approaches (n = 4), developed a methodology for further research (n = 1) and developed imaging approaches for patient assessment or to support treatment (n = 4). One project showed that a study approach is feasible (but does not seem to have been taken forward).

Twelve studies employed adaptive trial designs. Two of these trials have demonstrated that the design can inform the decision to proceed or stop clinical evaluation, therefore reducing the risk of research waste. A third trial successfully employed an adaptive randomisation algorithm to allocate patients to treatment arms.

In addition, 37% of CIs indicated that their active projects had already led to findings, as yet unpublished, and 14% reported findings related to the mechanism of action of the intervention.

Evaluation question 7: what scientific outcomes and impacts have arisen from the findings of the EME programme?

The findings of EME projects underpin further research. The majority of CIs continue to pursue research on the intervention tested in the EME trial (71% of CIs interviewed). CIs reported that further funding has been secured for more than one-third (38%) of EME projects, amounting to a minimum of £355M (see also Evaluation question 1: what value does the EME programme bring to the funding landscape for the development and assessment of health interventions?). Scientific outcomes also include the formation of new or strengthening of existing collaborations, with around half of CIs of active projects indicating that they were already collaborating on new projects with team members of the EME-funded study (see Evaluation question 11: what has been the broader impact on UK clinical research and clinical research community?). In some cases, where trials showed that the intervention tested had ‘no effect’ on the primary outcome measure (19%), the evidence allowed the decision to terminate the line of investigation and focus on other approaches.

Efficacy and Mechanism Evaluation projects findings are also being taken up by others in the research community. Thirty-eight projects published their main findings in peer-reviewed journals other than the EME journal. By September 2020, these had accumulated 2182 citations in the scientific literature, with some articles gathering > 200 citations. These scientific findings will continue to diffuse. Most articles are available as open-access publications, further increasing the potential for uptake.

Other outputs from EME-funded research continue to be used, supporting scientific progress and outcomes:

• All of the reported biological sample collections (n = 18) are still used by the EME project team. Four collections have been accessed by other research groups and two are being prepared for sharing.

• Seven data sets have been, or are being, shared with other research groups. Two completed studies have combined data sets with those of other trials to conduct meta-analyses (one of which was funded through an EME award).

• The three image collections have been shared with other researcher groups.

At the time of review, only around one-third (37%) of EME-funded projects had completed and/or published their main findings, with more than one-quarter of research papers published in 2019 and 2020. The level of outcomes and impacts from the EME programme can, therefore, be expected to rise as active EME-funded projects complete and research findings are taken up and developed further.

Evaluation question 8: how has performance varied across the EME portfolio in terms of scientific and clinical outputs and outcomes, and why?

At this stage of the EME programme, the performance of projects funded through the different EME workstreams cannot be robustly compared. To date, only eight projects funded through commissioned calls have published the main research findings (including two projects focused on mechanisms of action).

For projects that have achieved outcomes [see Evaluation question 13: what benefits for patients and populations (health impacts) have been achieved by EME-funded research and what benefits are likely to arise in the future?], no clear patterns of differences in performance by health area could be identified, potentially owing to the small number of projects in some areas. Three mechanistic substudies to larger trials funded by other funders, including charities, have resulted in important findings and likely benefited from the scale and access to patients of the main trial. Therefore, these substudies represent good value for money. However, requirements of the EME programme have changed since these awards were made, with programme funding now restricted to substudies of trials supported by the NIHR or the devolved administrations.

Efficacy and Mechanism Evaluation projects draw on existing infrastructure for their implementation, predominantly CRNs (74%), NHS resource (54%) and BRCs (37%). In interviews, CRNs were described by many as key enablers of recruitment (but issues were also reported), whereas BRC/BRUs provided essential infrastructure and supporting research staff. Many CIs mentioned the need for NHS staff time in conducting the studies. External infrastructure is, therefore, necessary for (at least some) EME projects to succeed. The majority of CIs (78%) were able to access the necessary expertise and infrastructure for their projects. A smaller share (n = 7, 16%) reported issues in accessing critical infrastructure, mostly related to administrative hurdles or limited capacity at trial sites (both staff time and level of research expertise). Four CIs elaborated that although they had access to the appropriate expertise and infrastructure, they had encountered challenges with setting up the collaboration or experienced ongoing issues during project implementation (e.g. unavailability of infrastructure during certain time periods and new trial sites lacking capabilities).

Nearly all CIs consulted had engaged in PPI at some project stage. For example, of completed projects, 75% engaged during the design stage, 69% during the implementation of the project and 31% after the research was completed. PPI representatives were commonly members of the TSC or Trial Management Group and, when involved, contributed most often to development of patient-facing information (in 87% of projects with PPI), the development of the research methodology and approach (in 56% of projects with PPI) and design of the recruitment or retention strategy (in 47% of projects with PPI). CIs highlighted that PPI has been important in enabling project delivery by ensuring the study protocol and intervention were acceptable to patients (n = 9), shaping of documentation for, and communication with, patients (n = 4), disseminating the EME project results (n = 3) and supporting recruitment for the trial (n = 3, two of which described the support as ‘invaluable’). Around half of CIs of active projects felt that PPI had enabled research ‘to a large extent’ in the project design phase and one-third of CIs indicated that PPI had enabled project delivery ‘to a large extent’ (the lower share potentially reflects the fact that many active projects are still in their early phases). Five examples of where PPI had been crucial for project implementation emerged, two related to PPI support in patient recruitment, two to addressing issues in the study protocol that would have posed a risk to the ability to recruit patients and one where input from PPI representatives was important in gaining MHRA approval for the trial. PPI is, therefore, an important enabler of EME projects.

Evaluation question 9: to what extent did EME-funded projects involve collaborations with industry, charities and other partners?

Two-thirds (68%) of projects reported collaborations with academic institutions/universities, and approximately one-third each collaborated with the private sector, with hospitals and with the public sector. Most CIs had worked with the research team of the EME award on other projects (i.e. 72% of active projects). Where new collaborators were included, CIs did so mainly to bring specific expertise into the team. In interviews, CIs felt that their projects had involved ‘all the right partners’ and that no expertise was missing.

The level of co-funding for EME projects is an indicator of the level of collaboration with charities and industry. Of the 62 co-funded projects, most were co-funded by the pharmaceutical industry (50%) and/or SMEs in medical technology and health care (37%). These arrangements mainly provided free or reduced-cost access to the interventions tested (68%) and, in the majority of cases, are unlikely to represent true collaboration. However, 11 (18%) projects reported industry co-funding of research costs, which may indicate a more active involvement of the industry partner in the delivery of the research project.

The potential to support collaborative research with industry, particularly SMEs, was a driver for DHSC to invest in the programme as part of the nation’s ‘growth agenda’. Therefore, there was an intention for the EME programme to attract ‘investigator-led’ projects from SMEs, as well as delivering value to companies as co-applicants of clinical studies through providing access to academic and clinical networks. However, to date, there have been no CIs of EME projects from SMEs. Potential reasons for a lack of SME-led projects put forward were that the funding model did not suit SMEs because of long timelines from application to contract, the complex governance arrangements required for running clinical trials (which are beyond the means of most SMEs) and an ‘academic approach’ of the EME programme to applications. In addition, at less than 18% of projects, the level of ‘true’ collaboration with industry is fairly low (e.g. when compared with 42% of projects in the MRC’s portfolio that are directed towards translational research and 35% of projects in the MRC’s non-directed translational portfolio). 67 This is also reflected in the limited commercial opportunities presented by EME projects (see Evaluation question 1: what value does the EME programme bring to the funding landscape for the development and assessment of health interventions?). It is also possible that demand for funding to support industry or industry collaboration is already covered through other funding schemes, such as the Invention for Innovation Programme, which directly funds SMEs to progress early-stage health-care technologies with potential for use in the NHS.

Charities/non-profit organisations co-funded 12 projects (i.e. 19% of all co-funded projects). CIs described that charities were frequently engaged to support PPI, and project management by the charity generally involves an advisory function. In addition, three completed studies were substudies to trials funded by other funders, such as charities (e.g. the British Heart Foundation or CRUK).

Evaluation question 10: how has the EME programme contributed to capacity building and at what levels?

The majority (69%) of CIs reported that the EME-funded research had contributed to training and/or capacity building among research team members and at ‘research-naive’ trial sites. In particular, early-career CIs and CRFs were highlighted as benefiting from support through the programme. In addition to increasing knowledge in their respective research fields, research teams acquired skills underpinning translational research ‘on the job’, such as experience working with regulatory processes and the MHRA (n = 5), working in teams (n = 4), general translational research skills (n = 3) and data/data science capabilities (n = 2).

The EME programme also enables CIs to increase their research profiles and influence in the research ecosystem. Awards and/or recognition for the CI were reported for 60 EME projects, mainly related to invitations to provide a keynote presentation (n = 37) or to take a prestigious and/or advisory position to an external body (n = 24). Twenty-one CIs also received research prizes and reported that EME-funded research had led to a ‘membership of a guideline committee’.

Many CIs (n = 11) emphasised that EME projects had built or strengthened trial capacity and networks across a range of research areas, including different indications, treatment approaches and settings. Three CIs specifically highlighted that the EME project had built, or was building, capacity in research areas in which few or no clinical trials had been conducted previously.

Efficacy and Mechanism Evaluation funding mainly supports projects led by CIs at academic institutions (73%), predominantly from the Russell Group (61%). CIs from Imperial College London led the largest number of studies (9%), followed by CIs based at King’s College London and the University of Birmingham (6.8% and 6.2%, respectively). Therefore, it can be expected that the EME programme has enhanced research capacity at these institutions especially. (Data were available only for lead institutions.)

Evaluation question 11: what has been the broader impact on UK clinical research and clinical research community?

The EME programme has contributed to a network of new research collaborations. Around half of CIs who responded to the survey indicated that they were collaborating on new projects with team members of the EME study, referring to both established and new collaborators. One-third of respondents reported that they were collaborating on a new project with commercial partners.

More broadly, many CIs emphasised that EME projects had built or strengthened networks across a range of areas, including different indications, treatment approaches and settings. For example, CIs of two EME studies reported that these had resulted in the formation of research networks, with one setting up a disease-specific registry. Involvement in EME projects strengthened interest and participation of some research team members that had not been involved in research previously, including clinicians, nurses and hospital administrative staff. Although these effects were reported, no clear ‘centres of excellence’ emerged as a result of the EME programme.

Few alternative sources of funding to the EME programme were identified and each was limited in the extent to which they covered both efficacy and mechanism research. As set out under EQ1 (see Table 1), the scheme’s focus on projects assessing repurposing of existing interventions or current medical practice, accounting for 49% and 17% of awards, respectively, indicates that EME funding is important in enabling the UK clinical research community to conduct research of limited or no commercial interest. As such, the programme focuses on informing the use of interventions, rather than on ‘broadening the pipeline of interventions’.

Evaluation question 12: is there evidence that EME funding and research has influenced the strategies of other funders?

Over the past decade, and following on from the Cooksey report1 and the establishment of the NIHR, public funders and charities (e.g. the British Heart Foundation) have strongly increased their focus on translational research, establishing dedicated translational funding streams, including for later-stage trials. 68 Therefore, the overall funding landscape has diversified. These efforts were complemented by the establishment of the NIHR BRCs to provide infrastructure support. However, as one interviewee from a funding organisation described:

In summary, there are now more funders in this space, and hence the landscape is more complex than 10 years ago. But no, we have not solved the Valley of Death [pull-through of discovery research to commercialisation stages], at least not in its totality.

One funder explained that they are sharing information with the EME programme team to actively align project decisions and. therefore, minimise overlaps and/or pool resources. Similarly, 19% of EME awards received co-funding from charities.

Although no evidence of direct influence of EME funding and research on the strategies of other funders was uncovered, the scheme may have served as a model for supporting clinical trials alongside mechanistic studies. For example, CRUK’s statement of intent for clinical research sets out the ambition ‘We will learn as much as we can from the patients on our trials’, incorporating mechanism studies and sample collection (Cancer Research UK, the world's leading independent cancer charity dedicated to saving lives through research, influence and information. © Cancer Research UK 2021. All rights reserved). 69 In addition, CRUK’s Experimental Medicine Awards are described as funding ‘highly ambitious translational research conducted in association with a clinical trial or well-designed clinical study’ (Cancer Research UK, the world's leading independent cancer charity dedicated to saving lives through research, influence and information. © Cancer Research UK 2021. All rights reserved). 70 However, the visibility of the EME programme in the broader translational research funding landscape is limited. For example, one funding organisation approached for interview declined, as they were not sufficiently aware of the programme. Likewise, one key opinion leader explained that their comments were related to NIHR-funded research more generally, rather than being specific to the EME programme. Therefore, increased engagement with other funders at the EME programme level is needed to increase scheme’s influence on other funders, and leverage investment and complementary expertise.

The NIHR’s focus on PPI in research was described as contributing to a culture change in the medical research community and increasing the acceptance of PPI’s contribution to processes and projects. At the same time, PPI is not a truly representative group of the public or the relevant patient population and uptake of PPI is still uneven across the research community.

Evaluation question 13: what benefits for patients and populations (health impacts) have been achieved by EME-funded research and what benefits are likely to arise in the future?

The intended impact of the EME programme set out in the PLM include the delivery of better-quality care and/or more effective and efficient use of available health-care resources, ultimately resulting in benefits to patient and population health. One pathway to impact is the incorporation of research evidence into guidelines that are then taken up into clinical practice, improving the delivery of health interventions.

Efficacy and Mechanism Evaluation-funded research is generating important evidence that has informed health-care decisions. Of completed EME projects employing trial methodology, one-third (35%) have informed or have the potential to inform clinical guidelines and, therefore, change or confirm health-care practice. Seven (16%) projects have already informed clinical guidelines or regulatory approval decisions and another eight have the potential to do so in the future given the nature of their findings. Trials showing a positive effect of an intervention were more likely to be taken up (e.g. in clinical guidelines) than those that showed ‘no effect’ (50% vs. 11%).

In addition, three clinical trials supported by EME-funded substudies led to changes in clinical guidelines or practice, and one technology developed by an EME project was taken up into clinical guidelines and is now routinely offered in the NHS. A further study showed that a diagnostic approach was more effective than current practice and this evidence was included in a guideline update, but the approach was not recommended because of cost considerations. CIs of active trials also reported uptake of their findings into guidelines and practice (e.g. one study has informed PHE testing).

Therefore, impacts on patient and population health have started to emerge from the EME programme. Six EME projects support the use of new clinical applications, with two described as having entered routine practice. Four projects demonstrated that an intervention currently in use may be harmful (with findings of two informing guidelines). Further scale-up of impacts will depend on take-up into practice across the UK health systems. Health impacts from currently active EME projects can be expected to accrue, for example CIs of ongoing trials reported that the study had already led to changes in how PHE tests for the condition under investigation and findings from another active trial were cited in a guideline. Many active studies focus on the health areas ‘oral and gastrointestinal’ (16%), ‘cancer’ (12%), ‘reproductive health and childbirth’ (10%) and ‘cardiovascular’ (10%), which, therefore, have a higher potential for future health impacts.

Evaluation question 14: what factors led to high or low impact on health and the health system across the EME-funded research?

As set out under EQ13 (see Table 1), an important pathway to impact is the incorporation of research evidence into practice. A research project’s potential for direct impact will depend on a range of factors, including internal factors (e.g. the quality and weight of the evidence generated, as affected by the study’s design, expertise and experience within the study team, and availability of supporting infrastructure) and external factors [e.g. the level of (potentially conflicting) evidence already available, the importance policy-makers, patients and practitioners assign to the problem addressed, the skills and resources available to adjust and deliver a changed/new intervention and the CI’s links to, or participation in, policy discussions and committees]. A range of parameters that may affect the level of impact of EME projects were explored further, using take-up of evidence into health-care guidelines and changes to clinical practice as proxy indicators.

At this stage of the EME programme, 12 projects have informed health-care guidelines or changed clinical practice. Three of these EME projects were substudies of trials funded from other sources. As set out under EQ8 (see Table 1), all three substudies have led to impacts and leveraged the resources of the trials they accompanied. The remaining nine projects span health areas (e.g. HRCS health category codes ‘eye’, ‘reproductive health and childbirth’, ‘respiratory’ and ‘inflammatory and immune system’) and study types (including two novel approaches to treatment/diagnosis, four studies testing/informing current practice and three repurposing studies). Therefore, at this stage of the EME programme, no specific areas or study types emerge as more successful in achieving impact than others.

The lead CIs of the 12 studies that have informed health-care guidelines or changed clinical practice were associated with six universities and one hospital trust. Only Imperial College London was represented more than once (with three project leads). Of the seven institutions, five are part of the cluster of research institutions located in London, Oxford and Cambridge. CIs associated with these institutions led on 42% (22/53) of completed studies and on 83% (10/12) of studies that have informed guidelines or changed clinical practice. Although numbers are small, this suggests that the context of well-funded research-intensive universities may favour success in achieving impact. Access to BRCs or BRUs at the CIs’ institutions and/or to proof-of-concept funding may also have supported their research. In addition, or alternatively, CIs from these institutions may have asked more pertinent research questions or designed better trials.

As set out for EQ8 (see Table 1), it is not possible to compare the performance of EME projects funded through the different EME workstreams. So far, only eight projects funded through commissioned calls have published main research findings. Of these, two have generated evidence that were taken up in clinical guidelines (i.e. a 25% ‘success rate’). However, these numbers are too small to draw robust conclusions.

As described for EQ8, PPI has played an important role in enabling project delivery. The evaluation did not find any evidence for the effect PPI has on the level of health outcomes and impacts (i.e. once the research project has completed).

Evaluation question 15: what socioeconomic impacts has EME-funded research contributed to (in the UK)?

To date, few projects have generated commercial outcomes and revenue. One active project reported that it had contributed to the formation of a spin-out company that secured £35M in series A financing. Four projects may have led to IP, with one project patenting a diagnostic test that is ‘in the process of being licensed to a commercial entity’.

Several EME projects have led, or have the potential to lead, to efficiency gains in the health system, either by demonstrating that a new, more expensive treatment approach does not lead to better outcomes than current practice, or by showing that treatments currently offered in routine practice do not result in any benefit. With few health economic analyses available, these cost savings have not been quantified at the programme level.

There is potential for future commercial impacts and efficacy gains:

• In Researchfish, CIs of eight projects reported follow-on grants or investments from industry based on the EME-funded research, and further research from another EME project is supported by a private health-care provider. It should be noted that these funding decisions may be based on the expertise developed by the research team, rather than the intervention tested.

• CIs of a further five projects reported interest from industry in supporting further development of the intervention.

• A total of 29% of CIs of active projects reported that their research had the potential to yield commercial benefits or efficiency gains in the health service.

Increased focus on projects with commercial potential

The majority of projects funded by the EME programme investigated repurposing of existing interventions or current medical practice (i.e. 49% and 17% of awards, respectively). This research has already contributed to health impact and led to efficiencies and cost savings in the delivery of care. A smaller share of projects focused on new technologies and approaches (i.e. 34% of awards). Among this group are a smaller number of projects with potential for commercial activity and economic impact. Alongside the high risk associated with the development of new technologies, the small number of ‘commercial projects’ funded is likely to have contributed to the fact that the evaluation found few examples of commercial outcomes and emerging economic impact.

The expectation that the EME programme would achieve economic impact from commercial activity was communicated to the evaluation team at the outset of the study. However, there was no indication of the expected of level of commercial outcomes and economic impact.

Recommendation 1a: assess whether or not the level of commercial outcomes and economic impact meets the funders’ expectations for the EME programme

If a higher level of outcomes and impact from the commercial domain is expected, future funding decisions should re-direct the programme’s portfolio more strongly towards progressing new therapies or approaches with commercial potential (i.e. ‘broadening the pipeline of interventions’). This could include a requirement for partnership with industry or other actors in the innovation ecosystem. However, it needs to be borne in mind that, by funding projects of limited commercial interest, the EME programme is likely to address a gap in research funding, supporting work that industry is unlikely to invest in.

The reason for the lower share of studies with commercial potential in the current portfolio is unclear. This should be investigated further (e.g. by reviewing whether or not applications proposing to develop new therapies and approaches are less successful or whether or not fewer applications are received) to inform measures aimed at strengthening this type of research as part of the EME programme.

Recommendation 1b: if a higher level of commercial outcomes is expected, identify reasons for the low level and steer funding towards projects with commercial potential (e.g. through funding criteria and/or project requirements) and communicate these to the research and development community

Recommendation 2: identify areas of research that address important health needs but are currently under-represented in the EME portfolio and investigate reasons for their low representation and implement support measures where appropriate

For areas with small numbers of awards and/or proposals, the underlying reasons should be further examined and, if needed, support measures offered through the EME programme or other MRC or NIHR schemes. Depending on the underlying reason, this could include targeted funding for discovery research to progress a research field to the stage of efficacy studies, and/or capacity building in clinical research targeted to specific research areas. To align the EME Funding Committee’s decisions with the programme’s objectives, committee members need to be orientated accordingly (e.g. through a briefing session).

Recommendation 3: clarify the role of mechanistic studies in the EME programme, potentially ‘optionalising’ this component – the ‘M’ – in the programme’s name

Rather than the current emphasis on ‘efficacy and mechanism’ (as in the name of the funding programme), the EME programme’s remit should be communicated as ‘efficacy with the option to include mechanism’, and ‘mechanism’ should also be an option for larger trials.

Recommendation 4: analyse EME applications data and review scores to understand current funding gaps for proof-of-concept studies and tailor additional funding offers

Recommendation 5: improve engagement and co-ordination with other funders at the EME programme level and explore options for partnership

This recommendation involves alerting and consulting other relevant funders prior to commissioning a call in a specific disease area and considers opportunities for joint working and co-funding.

Recommendation 6a: ensure that recruitment strategies, project timelines and costs set out in applications undergo critical assessment at the review stage, with clear feedback provided to applicants

It should be noted that adjustment of project plans to more appropriate budgets and timelines is likely to increase the level of required project funding.

Recommendation 6b: take further action to support researchers in planning for and addressing common challenges to research project implementation

• The EME programme team should provide support in study set-up and discussions with NHS stakeholders (e.g. with hospital R&D departments and research-naive study sites). This could include broader discussions with the DHSC and NHS to streamline processes (e.g. development of a unified policy for contractual arrangements with study centres and coverage of ETCs).

• Provision of funding prior to contracting with the NIHR should be provided to allow recruitment of a trial manager to take on administrative tasks and trial set-up.

• Funding of CRF positions to support project implementation (and build capacity) should be made available. This could include encouraging co-applications of senior and junior researchers, therefore allowing the junior co-CI to earn accreditation and acknowledgement for the study.

• Access for CIs to advisors with experience in clinical trials and/or regulatory processes, particularly for researchers new to clinical trial implementation, should be made available.

• Advice and development of best practice in setting up research projects to maximise data access and enable full analysis, especially for mechanistic studies, should be established.

• Existing training available from the NIHR to EME award holders (e.g. NIHR Academy or the NIHR/Health Data Research UK Incubator in Health Data Science) should be promoted.

Recommendation 7: examine whether or not EME programme implementation can be optimised

Future studies should examine whether or not EME programme implementation can be optimised in terms of:

• achieving faster turnaround in the application and contracting processes by identifying and removing any bottlenecks (in line with the UK Government’s current goal of reducing research bureaucracy72)

• coverage across disciplines by EME Funding Committee member expertise, especially in key areas of health need and non-traditional medical fields (e.g. behavioural approaches)

• ensuring that feedback on applications is clearly understood by applicants and consistent across review stages 1 and 2 (e.g. by providing continuity of reviewers or, at a minimum, historical context at all stages as an application moves through the process).

Research activity

1. Where did the original idea for the research topic come from? Select all that apply:

   1. Academic researchers.

   2. Clinical researchers.

   3. Patient groups.

   4. Health professionals.

   5. Regulators.

   6. Industry partner.

   7. Literature.

   8. Other.

Please provide further details.

1. How was the preceding (proof-of-concept) study funded? Select all that apply:

   1. Funding from NIHR.

   2. Funding from MRC.

   3. Funding internal to my organisation.

   4. Industry.

   5. Charity.

   6. Other.

Please specify source of funding and funding stream (if applicable).

1. Had you worked with the research team for this project before? [Select one.]

   1. Yes.

   2. No.

If no, how did you identify the co-investigator(s) and why did you include them in the team?

1. Does/did the project involve partnership(s) with industry, charities, (other) health-care professionals and/or international academic partners? [Select one.]

   1. Yes.

   2. No.

If yes, who are the partners and what do they bring to the project?

1. Is/was your grant co-funded from other sources? [Select one.]

   1. Yes.

   2. No.

If yes, please provide a brief description of the origin and nature of support received and the role that this has played in enabling the research project.

1. If your project does/did not receive industry partnership funding, had you sought industry co-funding or in-kind contribution? [Select one.]

   1. Yes.

   2. No.

   3. N/A (we received industry contribution).

If yes, please explain why industry contribution was not provided.

If no, please explain why industry contribution was not sought.

1. What research infrastructure is/was the project drawing on? Select all that apply:

   1. Biomedical Research Centre (BRC).

   2. Biomedical Research Unit (BRU).

   3. Clinical Research Network (CRN).

   4. NHS resource.

   5. MRC-supported research infrastructure.

   6. Charity-supported research infrastructure.

   7. Industry research infrastructure.

   8. Other (please specify).

Please provide a brief description of your answer.

1. In the light of your experiences to date, does/did the project have the right partners and infrastructure? [Select one.]

   1. The expertise and infrastructure accessed by the project is/was appropriate.

   2. Critical expertise is/was missing.

   3. Critical infrastructure is/was missing.

   4. Critical infrastructure and expertise are/were missing.

Please provide further details.

1. Does the project involve any of the following aspects? Select all that apply:

   1. Mechanistic study.

   2. Repurposing of an existing intervention.

   3. Applying novel study design.

   4. Generating/use of digital health data.

   5. Precision medicine approaches.

   6. Regenerative medicine approaches.

   7. Including comorbidities and multimorbidities.

   8. Targeting age-related condition.

Please provide further details.

Patient and public involvement

1. How many patient and public representatives (PPI) are/were involved in the project? [Select one per row.]

Please provide further details.

1. If relevant, what was the frequency of patient and public involvement? [Select one per row.]

Please provide further details.

1. If patients/members of the public are/were involved in the design and/or delivery of the project, which task(s) did they contribute to? Select all that apply:

   1. Developing the research questions.

   2. Developing the research methodology and approach.

   3. Design of the recruitment or retention strategy.

   4. Development of patient-facing information.

   5. Directly approaching/recruiting or retaining study participants.

   6. Dissemination of information and study findings.

   7. Other (please specify).

Please describe task(s) in more detail.

1. In your view, to what extent is/was patient and public involvement (PPI) an important enabler for the project? [Select one per row.]

Please explain your answer.

Challenges encountered and adjustments to project plan.

1. Did you face any of the following challenges in the implementation of the research project? Select all that apply:

   1. No challenges encountered.

   2. Contracting processes.

   3. Difficulty securing ethics approval in the time frame required.

   4. Setting up study sites.

   5. Capacity issues/shortage of trained staff.

   6. Patient recruitment.

   7. Supplier issues.

   8. Lack of infrastructure/equipment.

   9. Other resource issues.

   10. Methodology issues (e.g. related to statistics, data return, loss to follow-up).

   11. Regulatory issues.

   12. The intervention was not safe.

   13. Change in guidelines/practice during research project.

   14. Other (please specify).

Please describe the main challenges encountered.

1. Did you have to adjust any of the following aspects of the project plan after the start of the project? This question does not apply to adaptive trials. Select all that apply:

   1. None, the project plan has aligned closely with plan put forward in the proposal.

   2. Patient recruitment target number.

   3. Study timeline.

   4. Type of data collected/outcome measures.

   5. Method of data collection.

   6. The study team.

   7. Level/frequency/nature of patient and public involvement.

   8. The mechanistic study/explanatory aspects of the project.

   9. Other (please specify).

If changes were made, please describe these and explain how they have helped to address the challenges encountered.

1. In the light of your experiences to date, would you approach the project’s design or implementation differently? [Select one.]

   1. No, I would not make any changes to the project’s design and implementation.

   2. Yes, I would make minor changes to the project’s design and implementation.

   3. Yes, I would make substantial changes to the project’s design and implementation.

If yes, which aspects would you change? Select all that apply:

1. Patient recruitment target number.

2. Study timeline.

3. Preparatory data collection (e.g. in a feasibility study).

4. Type of data collected/outcomes measures.

5. Method of data collection.

6. Recruitment of additional experts to the team.

7. Level/frequency/nature of patient and public involvement.

8. The mechanistic study/explanatory aspects addressed by the project.

9. Other (please specify).

Please outline the main changes you would make and why.

Key findings and project outputs

Project outcomes

Funding landscape

1. Do you think the EME programme fills a gap in the funding landscape, enabling progress of interventions along the translational pathway? [Select one.]

   1. Yes, the EME programme fills a gap in the funding landscape.

   2. To a limited extent, the EME programme contributes to progressing interventions but gaps remain.

   3. No, the EME programme does not fill a gap.

   4. Do not know.

Please explain your answer.

1. What funding programme could you have applied to if the EME scheme had not been available?

2. What sources of further funding are available to continue to develop a promising intervention following on from an EME grant?

Design and management of the EME programme

1. Based on your experience, do you consider the EME programme appropriate in terms of funding amount, project length and scope? [Select one.]

   1. Yes, the EME programme is appropriate for achieving its aim.

   2. No, the EME programme is not appropriate and it needs to change.

   3. Do not know.

Please explain your response.

1. What do you consider the main strengths of the EME programme? What sets it apart from other related funding programmes?

2. What do you consider the main weaknesses of the EME-programme? What improvements would you suggest?

3. Should the EME programme support additional activities to make the research more effective and increase its potential for impact? [Select one.]

   1. Yes.

   2. No.

   3. Do not know.

Please explain what additional activities should be supported and why.

1. Was the management and support provided by the EME programme staff and EME Board (Review Committee) helpful for you (e.g. the call guidelines; communication with the programme team; knowledge and support provided by the EME Board as part of the review process; interactions and advice as part of project monitoring and reporting)?

Please provide further details.

Final comments and close

1. Do you have any other comments about the EME programme or any suggestions to the funders?

About you

1. What was your institution’s name at the time of EME grant application?

2. What was your role at the time of EME grant application?

Your EME proposal

If you have had multiple unsuccessful proposals, please select one for the purpose of this survey where you have the most information available.

1. What was the title of your EME proposal (optional)?

2. Where did the original idea for the research topic come from? Select all that apply:

   1. Academic researchers.

   2. Clinical researchers.

   3. Patient groups.

   4. Health professionals.

   5. Regulators.

   6. Industry partner.

   7. Literature.

   8. Other.

Please provide further detail on your answer.

1. Was the management and support provided by the EME programme team helpful for you during the proposal stage (e.g. the call guidelines; communication with the programme team during the proposal stage; support provided by the secretariat)? [Select one.]

   1. Yes, the management and support provided by the EME programme team was very helpful.

   2. Somewhat, the management and support provided by the EME programme team could have been more helpful.

   3. No, the management and support provided by the EME programme team was not helpful.

   4. I do not recall whether the management and support provided by the EME programme team was helpful.

Please provide further detail.

1. Was the feedback provided by the EME Board helpful for you (e.g. knowledge and support provided by the EME Board and secretariat as part of the review process)? [Select one.]

   1. Yes, the feedback provided by the EME Board was very helpful.

   2. Somewhat, the feedback provided by the EME Board could have been more helpful.

   3. No, the feedback provided by the EME Board was not helpful.

   4. I do not recall whether the feedback provided by the EME Board was helpful.

Please provide further detail.

Your further research activity

1. Did you continue working on the specific research idea/intervention after your proposal to the EME programme was unsuccessful? [Select one.]

   1. I continued working on the research idea and conducted an efficacy trial.

   2. I continued working on aspects of the research idea, such as mechanistic studies, but did not conduct an efficacy trial.

   3. I did not pursue this research idea further, but I am aware that others have.

   4. I did not pursue this research idea further and, as far as I know, it has not been tested elsewhere.

   5. Other.

Please provide further detail on your answer.

1. Did you apply for funding elsewhere for your specific research idea after your proposal to the EME programme was unsuccessful? [Select one.]

   1. I submitted a grant application to another funding programme, and was successful.

   2. I submitted a grant application to another funding programme, but was not successful.

   3. I was funded by industry to continue working on the intervention.

   4. I did not apply for funding elsewhere.

Please provide detail on your answer, including which funding programme you applied for, the level of funding secured and (if applicable) grant reference number.

Research progress and outcomes

1. Has the specific research idea/intervention progressed from the stage it was at when you submitted the EME proposal? [Select one.]

   1. Yes, the intervention has progressed, and was shown to be effective.

   2. Yes, the intervention has progressed, and was shown to not be effective.

   3. No, the intervention has not progressed.

2. If yes, did the research lead to any outputs/outcomes? [Select all that apply.]

   1. Yes, further research on the intervention led to a result that was published in a peer-reviewed journal.

   2. Yes, further research on the intervention led to changes in policy and/or practice.

   3. Yes, further research on the intervention led to commercial revenue (e.g. from spin-outs, licensing).

   4. Yes, further research on the intervention has led to cost savings in the health system.

   5. No, further research on the intervention has not led to any outputs/outcomes.

Please provide further detail on your answer, including web links where available.

1. If no, would the research idea still be relevant to pursue? [Select one.]

   1. Yes, the proposed research would still be relevant.

   2. No, the proposed research would no longer be relevant as a result of recent research findings.

   3. No, the proposed research would no longer be relevant as a result of recent changes in the guidelines and practice.

Please provide further detail on your answer.

1. Please provide any other comments about the EME programme or any suggestions to the funder.

Project background: aim, preparation, team

1. Can you briefly describe the aim of the project, at its outset, and what you hoped to achieve?

   • What was the primary question you sought to answer? What was the health problem the trial sought to address?

   • How did you envisage the findings of the project to be progressed to the point where they are taken up into guidelines and/or NHS practice?

2. Please outline the research that was to be conducted.

   • Did the project involve any mechanistic studies?

   • Did the trial involve any novel approaches? What was the trial methodology?

   • Did the project involve repurposing of an existing intervention?

3. Could you describe how you prepared for the project?

   • What experience/insights did you base the proposed research on?

   • What was the origin of the research topic? Did the original idea come from academic researchers, clinical researchers, patient groups, health professionals, regulators, industry partner or literature – or a combination of these?

   • How did the project relate to your previous work (e.g. continuation of research programme, or ‘new’ to this research area)?

   • How was the preceding study funded, and by whom?

   • Did you identify any risks that could potentially undermine the success of the project? If yes, what were these and how did you mitigate them? (For example, did you conduct a feasibility study? Were there any considerations around background IP?)

   • Did you involve stakeholders [e.g. professional bodies, a registered Clinical Trials Unit (CTUs)] in the design phase of the project (i.e. before submitting the application)? If yes, please describe their role, how you interacted and how this shaped the project design.

4. How was the project team organised?

   • Please describe the project team, and the roles and contributions of team members (e.g. skills, assets, infrastructure).

   • Had you worked with this team before? Please describe the level and nature of previous collaboration.

   • Did you collaborate with/work in partnership with industry, charities, (other) health-care professionals, international academic partners?

     • If yes, what specific role were these partners intended to bring to the project?

     • If no, why not?

   • Did the project receive any co-funding from other sources?

     • If yes, could you describe the co-funding arrangements (who were the co-funders, what were they contributing and at what level)?

     • If no industry co-funding, did you seek industry partnership funding? What was the reason (or reasons) this was not provided?

   • In hindsight, did the project draw on the right partners, or was there anything missing? Did the partners’ actual contributions match the original project plan?

   • Has the project led to further collaboration? Are you working with team members on other projects as a result of the EME-funded research?

5. Were any external resources or infrastructure necessary for the project?

Trial experience and learning

1. Was the project plan adjusted after the start of the project? If yes, how and why?

2. Did you encounter any challenges to project implementation? If yes, what were they?

   • Were (some of) these unexpected? Could they have been anticipated?

   • How did you address them?

   • Could adjustments have been made to the project design to overcome these difficulties/what adjustments did you make?

3. Did you have adequate resources or did you need to return for more funding?

4. How were patients or the public involved in the delivery of research?

   • How many representatives were involved?

   • How often did you involve them?

   • Did patient representatives contribute to specific tasks?

   • How did patient involvement shape the project?

   • How did it affect the implementation of the project?

   • Did it enable the take-up of research findings into policy and practice?

   • How important do you think were these contributions?

5. In hindsight, is there anything you would change about how the project was designed and implemented?

Key findings

1. Could you summarise very briefly the key findings of the project?

   • Did the research provide a ‘conclusive’ answer to the research question?

   • Did the findings allow a decision on whether (or not) to take the research further?

   • What is the wider scientific and clinical significance of the findings?

   • If inconclusive, were there any aspects of the underlying research design that could have been altered to arrive at a conclusive answer (e.g. did the research involve the appropriate patient groups, participant numbers)?

   • If applicable, what were the findings of mechanistic work carried out as part of the project? What is the wider scientific and clinical significance of the findings?

2. Has the project yielded any additional perhaps unexpected findings (including not anticipated at the outset of the project)?

Project outputs

1. I have already read your publication on the main trial results, but were there any other publications directly stemming from the EME-funded research? Any other way you have disseminated the findings and learning?

2. Tools, databases and sample collections, were any new research tools, databases or sample collections developed as part of the EME-funded project? Are these being used?

3. Were there any patents, spin-outs or licensing agreements as a result of the EME-funded project?

   • If yes, could you provide further details?

4. Have you used any advanced trial methodology?

   • If yes, what aspects of the trial were novel? What were the advantages and disadvantages of using this methodology?

   • Has this led to other investigators approaching you for advice?

5. How has the EME-funded research contributed to capacity building?

   • At what levels (clinical research fellows, PhD, clinical investigator, etc.)?

   • What were the skills and knowledge improved as a result (e.g. disease/intervention specific but also related to translational research skills, regulation, industry standards, patient involvement, trial methodology, team working)?

Project outcomes

1. Has the EME-funded work led to further research?

   • Has it facilitated further clinical research and/or development? By your group, or others?

   • Has it informed (or what is their potential to inform) further mechanistic/discovery research?

   • Has it enabled you to secure follow-on funding on the basis of the project findings? If yes, where from, what is the work funded, and at what level?

2. If the EME project has led to further research, what have the follow-on studies achieved?

   • How does progress compare to the expectations you had at the outset of the EME-funded project?

3. Have the project findings informed clinical applications, or do they have the potential to do so?

   • If yes, what are these? Did further development take place based on these project findings?

   • Who are current or future beneficiaries? Has the project increased the patient population who can access and benefit from this application?

   • If no, why not? What are the barriers?

4. Have the project findings informed clinical practice guidelines and/or medical practice, or do they have the potential to do so? If yes, please explain further.

5. Has the project contributed to any efficiency effects inside/outside the NHS (cost savings)? Have any health economic analyses/cost-effectiveness studies been conducted? If yes, could you provide details and references?

6. Did the project lead to any commercial benefits (e.g. for the industry partners or background IP holders)? Has the potential market size for the developed application increased?

7. Were there any other outcomes (including unintended)?

Funding landscape

1. What have been the key changes over the past 10 years of conducting efficacy trials in the UK?

2. At its inception, did the EME programme fill a gap in the funding landscape?

3. Did it bridge the gap from proof of concept to HTA/effectiveness studies and commercialisation/implementation?

4. Has the EME programme’s role changed over time? To what extent does it complement other funding streams today?

5. What funding programme could you have applied to if the EME scheme had not been available? Are there alternative sources of funding for this type of research? How do alternative funding sources differ from the EME scheme?

6. Are there synergies or duplications with other funding programmes?

7. What gaps remain in certain research areas/disciplines or specific types of efficacy research?

8. How was the research underpinning the EME-funded project we discussed earlier funded?

9. Where have you/would you look for funding to further develop a promising intervention following on from an EME grant?

Design and management of the EME programme

1. Is the EME programme an appropriate funding mechanism, in terms of funding amount, project length and scope?

2. Are there aspects of the scheme’s current design and requirements that you consider a barrier to achieving its aims?

3. Are there additional activities the EME programme could have supported that you think would have made the research more effective and increased the potential for impact?

4. Was the management and support provided by the EME programme staff and review committee helpful for you?

Final comments and close

1. Do you have any other comments about the EME programme or any suggestions to the funder?