Models and applications for measuring the impact of health research: Update of a systematic review for the Health Technology Assessment Programme

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 14/72/01. The contractual start date was in June 2014. The draft report began editorial review in May 2015 and was accepted for publication in December 2015. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

James Raftery is a member of the National Institute for Health Research (NIHR) Health Technology Assessment Editorial Board and the NIHR Journals Library Editorial Group. He was previously Director of the Wessex Institute and Head of the NIHR Evaluation, Trials and Studies Co-ordinating Centre (NETSCC). Amanda Blatch-Jones is a senior researcher at NETSCC.

Copyright statement

© Queen’s Printer and Controller of HMSO 2016. This work was produced by Raftery et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Evidence explaining why this research is needed now

Internationally, there has been a growing interest in assessing the impact of programmes of health research, and recent developments in the UK have created a new context for considering impact assessment. Besides the claim that much research is wasteful, other factors include pressure on higher education institutions to demonstrate accountability and value for money, the expansion in routine collection of research impact data and the large-scale assessment of research impact in higher education through case studies in the Research Excellence Framework (REF).

Aim

To review published research studies on tools and approaches to assessing the impact of programmes of health research and, specifically, to update the previous 2007 systematic review funded by the HTA programme. 2

Objective

Our objective was to build on the previous HTA review2 (published in 2007, covering the literature up to 2005) to:

1. identify the range of theoretical models and empirical approaches to measuring impact of health research programmes, and collate findings from studies assessing the impact of multiproject programmes

2. extend the review to examine (1) the conceptual and philosophical assumptions underpinning different models of impact and (2) emerging approaches that might be relevant to the HTA programme, such as studies focusing on monetised benefits and on the impact of new trials on systematic reviews

3. analyse different options for taking impact assessment forward in the National Institute for Health Research (NIHR)/HTA programme, including options for drawing on routinely collected data.

Structure of the report

Chapter 2 describes the methods used for the review, Chapter 3 reports the findings from the updated review, Chapter 4 presents a broader taxonomy of impact models, Chapter 5 provides the findings on the monetary value of the impact of health research, Chapter 6 reports on the impact of trials on systematic reviews, Chapter 7 summarises the impact of trials on discontinuing the use of technologies and Chapters 8 and 9 provide a discussion of the main findings, including options for NIHR/HTA to take research impact assessment forward and draw conclusions from the report, and discuss recommendations for future impact assessment.

Review methods

Given the nature and scope of the reviews included, a range of methods were used to identify the relevant literature:

1. systematic searching of electronic databases

2. hand-searching of selected journals

3. citation tracking of relevant literature

4. literature known to the team (i.e. snowballing)

5. bibliographic searches of other reviews

6. bibliographic searches of references in identified relevant literature.

Search strategies

Although different search strategies were conducted for the different elements, details of the individual search strategies can be found below (see Appendix 1 for full listing of the search strategies used).

Update to the previous review methods

The previous assessment of the impact of the HTA programme2 was informed by a review of the literature on assessing the impact of health research. It found an initial list of approximately 200 papers, which was reduced to a final ‘body of evidence’ of 46 papers: five conceptual/methodology, 23 application and 18 combined conceptual and application (please refer to the original Hanney et al. 2 report for a full list of these references). (In that review, as in the current one, ‘paper’ refers generically to the full range of publications, including reports in the grey literature.) The discussion included an analysis of the strengths, and weaknesses, of the conceptual approaches. The Payback Framework, the most widely used approach, was considered the most appropriate framework to adopt when measuring the impact of the HTA programme, notwithstanding the limited progress made in various empirical studies in identifying the health and economic benefit categories from the framework.

The first question for the updated review was: ‘what conceptual or methodological approaches to assessing the impact of programmes of health research have been developed, and/or applied in empirical studies, since 2005?’. 2

The second question was: ‘what are the quantitative findings from studies (published since 2005) that assessed the impact from multiproject programmes (such as the HTA programme)?’.

Extension of the literature methods

The second stream formed four parts:

1. exploring the conceptual and philosophical assumptions of models of impact

2. monetary value on the impact of health research

3. the impact of randomised trials on systematic reviews

4. the impact of randomised trials on stopping the use of particular technologies.

The methods for each part are discussed below.

Review findings

The number of papers identified though each source is set in Table 1. A total of 513 records were identified, of which 161 were eligible; databases directly identified only 40 of these 161 (see Appendix 3, Table 14, for a brief summary of each of the 161 references) (Figure 1).

FIGURE 1.

Flow diagram of identified studies.

Summary of the literature identified

From the initial searching and application of the inclusion criteria, the number of publications identified this time was approximately three times the 46 included in the ‘body of evidence’ for the 2007 review. 2 Using wider criteria, we ended up with a list of 161.

We classified 51 as conceptual/methodological papers (including reviews), 54 as application papers and 56 as both conceptual and application papers (these are classified and reported in Appendix 3, Table 14, under column ‘Type’). The 51 conceptual and methodological papers not only reflect an increase in the discussion about appropriate frameworks to use but also reflect the wider criteria used in the extension to the update, including some pre-2005 publications. Thus, a simple comparison between the 51 conceptual papers in the update and the five in the previous review would not be appropriate.

The papers come predominantly from four English-speaking nations (Australia, Canada, the UK and the USA), with clusters from the Netherlands and Catalonia/Spain. We also identified an increasing number of health research impact assessment studies from individual low- and middle-income countries, as well as many covering more than one country, including European Union (EU) programmes and international development initiatives.

Some of the studies on this topic are published in the ‘grey literature’, which probably means they are even more likely to be published in local languages than they would be if they were in the peer-reviewed literature. This exacerbates the bias towards a selection of publications from English-speaking nations that arises from the inclusion of publications if they are available only in English.

Appendix 3 (see Table 14) lists the 161 included studies with a brief summary of each. We note basic data such as lead author, year, type of study (method, application, or both) and country. The last item has become more complicated with the increase in the range of studies conducted. We prioritised the location of the research in which impact was assessed rather than the location of the team conducting the impact assessment. Similarly, for reviews or other studies intended to inform the approach taken in a particular country, it is important to identify the location of the commissioner of the review, if different from the team conducting the study. We also recorded the programme/specialism of the research in which impact was assessed, and the conceptual frameworks and methods used to conduct the assessment. A further column covers the impacts examined and a brief account of the findings. The final column offers comments, and quotes, where appropriate, on the strengths and weaknesses of the impact assessment and factors associated with achieving impact.

We also identified a range of papers that were of some interest for the review, but the papers did not sufficiently meet the inclusion criteria (see Appendix 4 for further details of these papers).

The included studies demonstrate that the diversity and complexity of the field has intensified. It has long been recognised that research might be used in many ways, even in relation to just one impact category, such as informing policy-making. 34,35 Within any one impact assessment, there can be many different ways and circumstances in which research from a single programme might be used. Furthermore, as a detailed analysis of one of the case studies described in Wooding et al. 36 illustrated, even a single project or stream of research might make an impact in various different ways, some relying on interaction between the research team and potential users and some through other routes.

The diversity in the approaches is also linked to the different types of research (basic, clinical, health services research, etc.) and fields, the various modes of research funding (responsive, commissioned, core funding, research training), and the diverse purposes and audiences for impact assessments. These are considered at various points in this review.

The 51 conceptual/methodological papers in Table 14 (see Appendix 3) illustrate the diversity. Some of these 51 papers developed new conceptual frameworks and some reviewed empirical studies and used the review to propose new approaches. Others analysed existing frameworks trying to identify the most appropriate frameworks for particular purposes. RAND Europe conducted one of the major streams of such review work. These reviews include background material informing the framework for the Canadian Academy of Health Sciences (CAHS),37 an analysis commissioned by the HEFCE to inform the REF,38 and a review commissioned by the Association of American Medical Colleges. 9

Such reviews represent major advances in the analysis of methods and conceptual frameworks, and each compares a range of approaches. They often focus on a relatively small number of major approaches. Although Guthrie et al. 9 identified 21 frameworks, many are not health specific and they vary in how far the assessment of impact features in the broader research evaluation frameworks.

Our starting position was different, and aimed to complement this stream of review work. We collated and reviewed a much wider range of empirical studies, in addition to the methodological papers. We not only identified the impacts assessed, but also considered the findings from empirical studies, both to learn what they might tell us about approaches to assessing research impact in practice and also to provide a context for the assessment of the second decade of the HTA programme.

In selecting the conceptual frameworks and methods on which to focus, we thought it was important to reflect the diversity in the field as far as possible, but at the same time focus on analysis of approaches likely to be of greatest relevance for assessing the impact of programmes such as the HTA programme.

Conceptual frameworks developed and/or used

We identified a wider range of conceptual frameworks than in the previous review. How the 20 frameworks were used can be seen later (see Table 2). We have grouped the discussion of conceptual frameworks into three main sections. The data are presented in ways that allow analysis from several perspectives. First, we present a historical analysis that helps to identify which frameworks have developed from those included in the 2007 review. Second, we order the frameworks by the level of aggregation at which they can be applied. Having briefly introduced each of the frameworks we then present them in tabular form under headings, such as the methods used, impacts assessed, strengths and weaknesses. Finally, in our analysis comparing the frameworks we locate each one on a figure with two dimensions: categories of impacts assessed and focus/level of aggregation at which the framework has primarily been applied.

The three main groups of frameworks are:

1. Post-2005 application, and further development, of frameworks described in the 2007 review, and reported in the order first reported in 2007 (five frameworks).

2. Additional frameworks or approaches applied to assess the impact of programmes of health research, and mostly developed since 2005 (13 frameworks). (These are broadly ordered according to the focus of the assessment, starting with frameworks that are primarily used to assess the impact from the programmes of research of specific funders, then frameworks that are more relevant for the work of individual researchers and, finally, approaches for the work of centres or research groups.)

3. Recent generic approaches to research impact developed and applied in the UK at a high level of aggregation, namely regular monitoring of impacts [e.g. via researchfish^® (researchfish Ltd, Cambridge, UK)] and the REF (two frameworks or approaches).

Post-2005 applications of frameworks described in the 2007 review

Five are listed as follows:

1. the Payback Framework39

2. monetary value approaches to estimating returns from research (i.e. return on investment, cost–benefit analysis, or estimated cost savings)

3. the approach of the Royal Netherlands Academy of Arts and Sciences (2002)40

4. a combination of the frameworks originally developed in the project funded by the UK’s Economic and Social Research Council (ESRC) on the non-academic impact of socioeconomic research41 and in the Netherlands in 199442 [this became the Social Impact Assessment Methods through the study of Productive Interactions (SIAMPI)]

5. detailed case studies and follow-up analysis, on HTA policy impacts and cost savings: Quebec Council of Health Care Technology assessments (CETS). 43,44

The Payback Framework

The Payback Framework consists of two main elements: a multidimensional categorisation of benefits and a model to organise the assessment of impacts. The five main payback categories reflect the range of benefits from health research, from knowledge production through to the wider social benefits of informing policy development, and improved health and economy. This categorisation, which has evolved, is shown in Box 1.

BOX 1 - Example of the multidimensional categorisation of paybacks of the Payback Framework

1. Knowledge

• Journal articles, conference presentations, books, book chapters and research reports.

2. Benefits to future research and research use

• Better targeting of future research.

• Development of research skills, personnel and overall research capacity.

• A critical capacity to absorb and appropriately utilise existing research, including that from overseas.

• Staff development and educational benefits.

3. Benefits from informing policy and product development

• Improved information bases for political and executive decisions.

• Other political benefits from undertaking research.

• Development of pharmaceutical products and therapeutic techniques.

4. Health and health sector benefits

• Improved health.

• Cost reduction in delivery of existing services.

• Qualitative improvements in the process of delivery.

• Improved equity in service delivery.

5. Broader economic benefits

• Wider economic benefits from commercial exploitation of innovations arising from R&D.

• Economic benefits from a healthy workforce and reduction in working days lost.

Although a detailed account of the various impact categories is available elsewhere,2 key recent aspects of the framework’s evolution relate to headings number 2 and 5 in Box 1.

In the ‘Benefits to future research and research use’ category, the subcategory termed ‘A critical capacity to absorb and appropriately utilise existing research, including that from overseas’ had proven difficult to operationalise in applications of the Payback Framework. However, a more recent evidence synthesis46 incorporated this concept into a wider analysis of the benefits to the health-care performance that might arise when clinicians and organisations engage in research. Although the evidence base is disparate, a range of studies was identified that suggested when clinicians and health-care organisations engaged in research there was a likelihood of improved health-care performance. Identification of the mechanisms through which this occurs contributes to the understanding of how impacts might arise, and increases the validity of some of the findings from payback studies in which researchers claim that research is making an impact on clinical behaviour in their local health-care systems.

In the ‘Broader economic benefits’ category, recent developments emphasise approaches that monetise the health gains per se from research, rather than assessing the economic benefits from research in terms of valuing the gains from a healthy workforce. 26 Nason et al. 47 applied the Payback Framework in a way that highlighted the economic benefits category and identified various subcategories.

The payback model is intended to assist the assessment of impact and is not intended necessarily to be a model of how impact arises. It consists of seven stages and two interfaces between the research system and the wider environment, with feedback and also the level of permeability at the interfaces being key issues: developments do not necessarily flow smoothly, or even at all, from one stage to the next (Figure 2).

FIGURE 2.

The Payback Framework: model for organising the assessment of the outcomes of health research. Reproduced with permission. 48

As noted in the 2007 review,2 although the framework is presented as an ‘input–output model’, it ‘also captures many of the characteristics of earlier models of research utilisation’ such as those of Weiss34 and Kogan and Henkel. 49 The framework recognises that research might be utilised in various ways. It was devised to assess the impact of the Department of Health/NHS programme of research, a programme in which development was informed by Kogan and Henkel’s earlier analysis of the department’s research and development. 49 That analysis had promoted the idea that collaboration between potential users and researchers was important in encouraging the commissioning of research that was more likely to make an impact. Partly, the development of the Payback Framework was a joint enterprise between the Department of Health and the Health Economics Research Group. 50 The inclusion in the updated review of the findings from the application of the framework to the assessment of the first decade of the HTA programme illustrates the context within which the framework seems best suited.

The conceptual framework informs the methods used in an application; hence, documentary analysis, surveys and case study interview schedules are all structured according the framework, which is also used to organise the data analysis and present case studies in a consistent format. The various elements were devised both to reflect and capture the realities of the diverse ways in which impact arises, including as a product of interaction between researchers and potential users at agenda-setting and other stages. The emphasis on examining the state of the knowledge reservoir at the time of research commissioning enables some evidence to be gathered that might help explore issues of attribution, and possibly the counterfactual, because it forces consideration of whatever other work might have been going on in the relevant field.

One of the limitations of the Payback Framework, and various other frameworks, arises because of the focus on single projects as the unit of analysis, when it is often argued that many advances in health care should be attributed to a body of work. This ‘project fallacy’ is widely noted, including by many who apply the framework. In some studies applying the framework, for example to the research funded by Asthma UK,51 the problem was acknowledged in the way in which case studies that started with a focus on a single project were expanded to cover streams of work. Although some studies have been able to apply a version of the framework to demonstrate considerable impact from single studies,52 this has tended to be in particular types of research – in this case, intervention studies.

Some studies applied the framework in new ways, as noted in Table 14 (see Appendix 3). This might lead to welcome innovation, but also to applications that do not recognise the importance of features such as the interfaces between the research system and the wider environment and the desirability of capturing aspects such as the level of interaction prior to research commissioning.

Despite the challenges in application, 27 of our 110 empirical studies published since 20052,36,47,51–74 claim their framework is based either substantially or partly on the Payback Framework (Table 2).

TABLE 2 - Empirical studies using the 20 selected frameworks/approaches

SDO, Service and Delivery Organisation.

Studies in bold indicate that more than one approach substantially informed the approach eventually adopted/developed by the study (in these cases, the other approaches are not shown unless they too are one of the 20 selected frameworks).

Framework/approach (in order presented in text of Chapter 3)	Empirical studies applying the framework or drawing on aspects of it
Payback Framework	Action Medical Research, 2009;54 Anderson, 2006;55 Aymerich et al., 2012;56 Bennett et al., 2013;57 Catalan Agency for Health Technology Assessment and Research, 2006;58 Bunn, 2010;59 Bunn and Kendall, 2011;60 Bunn et al., 2014;61 Cohen et al., 2015;52 Donovan et al., 2014;62 Engel-Cox et al., 2008;63 Expert Panel for Health Directorate of the European Commission’s Research Innovation Directorate General, 2013;53 Guinea et al., 2015;64 Hanney et al., 2007;2 Hanney et al., 2013;51 Kalucy et al., 2009;65 Kwan et al., 2007;66 Longmore, 2014;67 Nason et al., 2011;47 NHS SDO, 2006;68 Oortwijn, 2008;69 Reed et al., 2011;70 RSM McClure Watters et al., 2012;71 Schapper et al., 2012;72 Scott et al., 2011;73 The Madrillon Group, 2011;74 and Wooding et al., 201436
Monetary value	Deloitte Access Economics, 2011;25 Guthrie et al., 2015;27 Johnston et al., 2006;75 MRC, 2013;76 Murphy, 2012;77 and Williams et al., 200878
Royal Netherlands Academy of Arts and Sciences and others	Royal Netherlands Academy of Arts and Sciences 201079
Social impact assessment model through the study of productive interactions	Meijer, 2012;80 and Spaapen et al., 201181
Quebec Council of Health Care Technology’s assessments	Bodeau-Livinec et al., 2006;82 and Zechmeister and Schumacher, 201283
CAHS	Adam et al., 2012;84 Aymerich et al., 2012;56 Cohen et al., 2015;52 Graham et al., 2012;85 Saskatchewan Health Research Foundation, 2013;86 and Solans-Domenèch et al., 201387
Banzi’s research impact model	Laws et al., 2013;88 and Milat et al., 201389
National Institute of Environmental Health Sciences’s logic model	Drew et al., 2013;90 Engel-Cox et al., 2008;63 Liebow et al. 2009;91 Orians et al. 200917
Medical research logic model (Weiss)	Informed various approaches rather than being directly applied
National Institute for Occupational Health and Safety’s logic model	Williams et al., 200992
The Wellcome Trust’s assessment framework	Wellcome Trust, 201493
VINNOVA	Eriksen and Hervik, 200594
Flows of knowledge, expertise and influence	Meagher et al., 200895
Research impact framework	Bunn, 2010;59 Bunn and Kendall, 2011;60 Bunn et al., 2014;61 Caddell et al., 2010;96 Kuruvilla et al., 2007;97 Schapper et al., 2012;72 and Wilson et al., 201098
Becker Medical Library model	Drew et al., 2013;90 and Sainty, 201399
Societal quality score (Leiden University Medical Centre)	Meijer, 2012;80 and Mostert et al., 2010100
Research performance evaluation framework	Schapper et al., 201272
Realist evaluation	Evans et al., 2014;101 and Rycroft-Malone et al., 2013102
Regular monitoring	Drew et al., 2013;90 MRC, 2013;103 MRC, 2013;76 and Wooding et al., 2009104
REF (informed by Research Quality Framework)	Cohen et al., 2015;52 Group of Eight and Australian Technology Network of Universities, 2012;105 and the HEFCE, REF Main Panel A, 2015106

In addition, the Payback Framework also informed the development of several other frameworks, especially the framework from the CAHS. 7 Furthermore, the framework based on the review by Banzi et al. 4 built on both the Payback Framework and the CAHS’s Payback Framework. The Payback Framework also contributed to the development, by Engel-Cox et al. ,63 of the National Institute of Environmental Health Sciences (NIEHS) framework.

Monetary value approaches to estimating returns from research (i.e. return on investment, cost–benefit analysis or estimated cost savings)

These approaches differ in the scope of the impacts that are valued and the valuation method adopted. In particular, since 2007 further methods have been developed that apply a value to, or monetise, the health gain resulting from research. Much of this work assesses the impacts of national portfolios of research, and is thus at a higher level of aggregation than that of a programme of research. Most of the studies of this are, therefore, not included here in Chapter 3, but are described in Chapter 5, which looks specifically at such developments. Nevertheless, three studies25,27,75 from this stream do assess the value of a programme of work and so are included in the update. Of the three, Guthrie et al. 27 and Johnston et al. 75 are the clearest applications of this approach to specific research programmes.

Furthermore, many econometric approaches to assessing research impact do not relate to the impact of specific programmes of research. However, an increasing number of frameworks have been developed that propose ways of collecting data from specific projects or programmes that can be built up to provide a broader picture of economic impacts. For example, Muir et al. 107 developed an approach for measuring the economic benefits from programmes of public research in Australia. Other work includes the development of frameworks by the UK department responsible for science; the Department of Business, Innovation and Skills (BIS) and, earlier, the Department for Innovation, Universities and Science108 developed frameworks under which the department collects data on economic benefits from each research council’s programmes of research, including the MRC. 76 The impacts include patents, spin-offs, intellectual property income; and data collection overlaps with the approach of regular collection of data from the MRC described below (see Regular monitoring or data collection). 76

A further category in the BIS framework is data on the employment of research staff. The classification of such data as a category of impact is part of a wider trend, but is controversial. However, in political jurisdictions, such as Ireland47 or Northern Ireland,6 it might be appropriate to consider the increased employment that comes as a result of local expenditure of public funds leveraging additional research funds from other sources.

To varying degrees the assessment of economic impacts can form part of wider frameworks, including the Payback Framework, as in the two Irish examples above, and the VINNOVA approach described by Eriksen and Hervik94 (see VINNOVA).

The approach of the Royal Netherlands Academy of Arts and Sciences

The report from the Royal Netherlands Academy of Arts and Sciences79 updated the evaluation framework previously used by the academy to assess research, not just impact, at the level of research organisations and groups or programmes. The approach combines self-evaluation and external peer review, including a site visit every 6 years. The report listed a range of specific measures, indicators or more qualitative approaches that might be used in self-evaluation. They included the long-term focus on the societal relevance of research, defined as ‘how research affects specific stakeholders or specific procedures in society (for example, protocols, laws and regulations)’. 79 The report proceeds to give the website for the Evaluating Research in Context (ERiC) project, which is described in Spaapen et al. 109 as being driven partly by the need, and/or opportunity, to develop methods to assist faculty in conducting the self-evaluation required under the assessment system for academic research in the Netherlands.

A combination of the frameworks originally developed in 2000 in the project funded by the UK’s Economic and Social Research Council on the non-academic impact of socioeconomic research and in the Netherlands in 1994 (this became the Social Impact Assessment Methods through the study of Productive Interactions)

In 2000, a team led by Molas-Gallart,41 working on the project funded by the UK’s ESRC on the non-academic impact of socioeconomic research, developed an approach based on the interconnections of three major elements: the types of output expected from research; the channels through which their diffusion to non-academic actors occurs; and the forms of impact. Later the team combined forces with Spaapen, whose early work with Sylvain42 on the societal quality of research had long been influential in the Netherlands, and, collectively, they led the SIAMPI approach. 110 This overlaps also with the development of the SciQuest method by Spaapen et al. 109 that came from the ERiC project described in The approach of the Royal Netherlands Academy of Arts and Sciences.

Its authors described SciQuest as a ‘fourth-generation’ approach to impact assessment. The previous three generations were characterised, they suggested, by measurement (e.g. an unenhanced logic model), description (e.g. the narrative accompanying a logic model) and judgement (e.g. an assessment of whether the impact was socially useful or not). The authors suggested that fourth-generation impact assessment is fundamentally a social, political and value-oriented activity and involves reflexivity on the part of researchers to identify and evaluate their own research goals and key relationships.

SciQuest methodology requires a detailed assessment of the research programme in context and the development of bespoke metrics (both qualitative and quantitative) to assess its interactions, outputs and outcomes. These are then presented in a unique research embedment and performance profile, visualised in a radar chart.

In addition to these two papers,109,110 the study by Meijer80 was partly informed by SIAMPI (see Appendix 3).

Detailed case studies and follow-up analysis on Health Technology Assessment policy impacts and cost savings: Quebec Council of Health Care Technology assessments

In the 2007 review,2 we described a series of studies of the benefits from HTAs conducted by the CETS. 43,44 They conducted case studies based on documentary analysis and interviews, and developed a scoring system for an overall assessment of the impact on policy that went from 0 (no impact) to +++ (major impact). They also assessed the impact on costs. Bodeau-Livinec et al. 82 assessed the impact on policy of 13 HTAs conducted by the French Committee for the Assessment and Dissemination of Technological Innovations. Although they did not explicitly state that they were using a particular conceptual framework, their approach to scoring impact appears to follow the earlier studies of CETS in Quebec.

Zechmeister and Schumacher83 assessed the impact of all HTA reports produced in Austria at the Institute for Technology Assessment and Ludwig Boltzmann Institute for HTA aimed at use before reimbursement decisions were made or decisions for disinvestment. Again, they developed their own methods, but the impact of these HTA reports was analysed partly by descriptive quantitative analysis of administrative data informed by the Quebec studies. 43,44

Additional frameworks or approaches applied to assess the impact of programmes of health research and mostly developed since 2005

Many other conceptual frameworks have been developed to assess the impacts from programmes of health research, mostly since 2005. Some studies have combined several approaches. Below we list 13 frameworks that have also been applied at least once. Some frameworks combine elements of existing frameworks, an approach recommended by Hansen et al. 111 This means that in the list of studies that have applied different conceptual frameworks (see Table 2), there are some inevitable overlaps. Scope exists for different interpretations of exactly how far a specific study does draw on a certain framework. An important consideration in deciding how much detail to give on each framework has been its perceived relevance for a programme such as the HTA programme.

The 13 conceptual frameworks are presented as follows: first, frameworks applicable to programmes that have funded multiple projects; second, frameworks devised for application by individual researchers; third frameworks devised for application to groups of researchers or departments within an institution; and, finally, a generic evaluation approach that has been applied to assess the impact of a new type of funded programmes. Inevitably, it is not this clear-cut and there are some hybrids.

Canadian Academy of Health Sciences

The CAHS established an international panel of experts, chaired by Cyril Frank, to make recommendations on the best way to assess the impact of health research. Its report, Making an Impact: A Preferred Framework and Indicators to Measure Returns on Investment in Research,7 contained a main analysis, supported by a series of appendices by independent experts. The appendices discuss the most appropriate framework for different types of research and are analysed in Table 14 (see Appendix 3). 37,112–114

The CAHS framework was designed to track impacts from research through translation to end use. It also demonstrates how research influences feedback upstream and the potential effect on future research. It aims to capture specific impacts in multiple domains, at multiple levels and for a wide range of audiences. As noted in several of the appendices, it is based on the Buxton and Hanney Payback Framework (see Figure 2). 39 The framework tracks impacts under the following categories, which draw extensively on the Payback Framework: advancing knowledge; capacity building; informing decision-making; health impacts; broader economic; and social impacts. 7,115 The categories from the Payback Framework had already been adopted in Canada by the country’s main public funder of health research, the Canadian Institutes of Health Research, for use in assessing the payback from its research.

The main difference in the categorisation from that in the original Payback Framework is the substitution of ‘informing decision-making’ for ‘informing policy and product development’. The CAHS return on investment version,7 allows the categorisation to include decisions by both policy-makers and individual clinicians in the same category, whereas the Payback Framework distinguishes between policy changes and behavioural changes, and does not specifically include decisions by individual clinicians in the policy category. Therefore, the CAHS framework explicitly includes the collection of data about changes in clinical behaviour as a key impact category, but in studies applying the Payback Framework any assessments that can be made of behavioural changes by clinicians and/or the public in the adoption stage of the model help form the basis for an attempt to assess any health gain.

The CAHS’s logic model framework also builds on a Payback logic model, and combines the five impact categories into the model showing specific areas and target audiences where health research impacts can be found, including the health industry, other industries, government and public information groups. It also recognises that the impacts, such as improvements in health and well-being, can arise in many ways, including through health-care access, prevention, treatment and the determinants of health.

The Canadian Institutes of Health Research divided its research portfolio into four pillars. Pillars I–IV cover the following areas: biomedical; clinical; health services; and social, cultural, environmental and population health. The CAHS team conducted detailed work to identify the impact from the different outputs arising in each of these areas.

The team also developed a menu of 66 indicators that could be collected. It was intended for use across Canada, and has been adopted by the Canadian Institutes of Health Research and in some of the provinces, for example by Alberta Innovates: Health Solutions (AIHS), the main Albertan public funder of health research. AIHS also further developed the framework into a specific version for their organisation and explored how it would be implemented and developed. Implementation had to do with standardising indicators across programmes to track progress to impact. It was developed to improve the organisation’s ability to assess its contributions to health systems impacts, in addition to the contributions of its grantees. 85 The CAHS framework has also been applied in Catalonia by the Catalan Agency for Health Information and Quality. 84

Banzi’s research impact model

Banzi et al. ,4 in a review of the literature on research impact assessment, identified the Payback Framework as the most frequently used approach. They presented the CAHS’s payback approach in detail, including the five payback categories as listed above. Building on the CAHS report, Banzi et al. 4 set out a list of indicators for each domain of impact and a range of methods that could be used in impact assessment. The Banzi research impact model has been used as the organising framework for several detailed studies of programmes of research in Australia.

A number of the applications have suggested ways of trying to address some of the limitations noted in the earlier account of the Payback Framework. For example, the study by Laws et al. 88 applied the Banzi framework to assess the impact of a schools physical activity and nutrition survey in Australia. They found it difficult to attribute impacts to a single piece of research, particularly the longer-term impacts, and wondered whether or not the use of contribution mapping, as proposed by Kok and Schuit may provide an alternative way forward (see Chapter 4 for a description of Kok and Schuit116).

National Institute of Environmental Health Sciences’s logic model

The US NIEHS developed and applied a framework to assess the impact from the research and the researchers it funded. Engel-Cox et al. 63 developed the NIEHS logic framework and identified a range of outcomes by drawing on the Payback Framework and Bozeman’s public value mapping. 117 These outcomes included translation into policy, guidelines, improved allocation of resources, commercial development; new and improved products and processes; the incidence, magnitude and duration of social change; health and social welfare gain and national economic benefit from commercial exploration and a healthy workforce; and environmental quality and sustainability. They added metrics for logic model components. The logic model is complex; in addition to the standard logic model components of inputs, activities, outputs and outcomes (short term, intermediate, long term), there are also four pathways: NIEHS and other government pathways, grantee institutions, business and industry, and community. The model also included the knowledge reservoir and contextual factors (Figure 3).

FIGURE 3.

The NIEHS’s logic model. Reproduced with permission from Environmental Health Perspective. 63

The various pathways allow a broader perspective to be developed than that of individual projects, for example by the grantee institution pathway, and by focusing on streams of research from multiple funders. Challenges identified in the initial case studies included ‘the lack of direct attribution of NIEHS-supported work to many of the outcome measures’. 63 The NIEHS put considerable effort into developing, testing and using the framework. Orians et al. 17 used it as an organising framework for a web-based survey of 1151 asthma researchers who received funding from NIEHS or comparison federal agencies from 1975 to 2005. Although considerable data were gathered, the authors noted that ‘this method does not support attribution of these outcomes to specific research activities nor to specific funding sources’. 17

Furthermore, Liebow et al. 91 were funded to tailor the logic model of the NIEHS’s framework to inputs, outputs and outcomes of the NIEHS asthma portfolio. Data from existing National Institutes of Health databases were used and, in some cases, data matched with that from public data on, for example the US Food and Drug Administration website for the references in new drug applications, plus available bibliometric data and structured review of expert opinion stated in legislative hearings. Considerable progress was made that did not require any direct input form researchers. However, not all the pathways could be used and they found their aim to obtain readily accessible, consistently organised indicator data could not in general be realised.

A further attempt was made to gather data from databases. Drew et al. 90 developed a high-impacts tracking system: ‘an innovative, Web-based application intended to capture and track short-and long-term research outputs and impacts’. It was informed by the stream of work from NIEHS,17,63 but also by the Becker Library approach118 and by the development in the UK of researchfish. The high-impacts tracking system imports much data from existing National Institutes of Health databases of grant information, in addition to text of progress reports and notes of programme officers/managers.

This series of studies demonstrates both a substantial effort to develop an approach to assessing research impacts, and the difficulties encountered. The various attempts at application clearly suggest that the full logic model is difficult and too complex to apply as a whole. Although the stream of work has, nevertheless, had some influence on thinking beyond the NIEHS, apart from the in-house stream of work no further empirical studies were identified as claiming that their framework was based on the NIEHS’s logic model approach.

Medical research logic model (Weiss)

Anthony Weiss analysed ways of assessing health research impact, but, unlike many of the other approaches identified, his analysis was not undertaken in the context of aiming to develop an approach for any specific funding or any research-conducting organisation. He drew on the United Way model119 for measuring programme outcomes to develop a medical research logic model. As with standard logic models it moves from inputs, to activities, outputs, and outcomes: initial, intermediate, long term. He also discussed various approaches that could be used, for example surveys of practitioners to track awareness of research findings; changes in guidelines, and education and training; use of disability-adjusted life-years (DALYs) or quality-adjusted life-years (QALYs) to assess patient benefit. He also analysed a range of dimensions from the outputs, such as publications through to clinician awareness, guidelines, implementation and overall patient well-being. 120

Although this model was not developed for a specific organisation, it does overlap with the emphasis given to logic models in various frameworks and studies, including the W.K. Kellogg logic model. 121 Weiss’s account is included here because it has become quite high profile and is widely cited. It has informed a range of studies rather than being directly applied in empirical studies.

National Institute for Occupational Health and Safety’s logic model

Williams et al. ,92 from the RAND Corporation in the USA, with advice from colleagues in RAND Europe, developed a logic model to assess the impact from the research funded by the National Institute for Occupational Health and Safety (NIOSH). At one level the basic structure of the logic model was a standard approach, as described by Weiss120 and as in the logic model from W.K. Kellogg. 121 Its stages include inputs, activities, outputs, transfer, intermediate customs, intermediate outcomes, final customers, intermediate outcomes and end outcomes.

A novel feature of the NIOSH model was outcome worksheets based on the historical tracing approach,122 which reversed the order ‘articulated in the logic model and essentially places the burden on research programs to trace backward how specific outcomes were generated from research activities’. 92

Research programmes could apply these tools to develop an outcome narrative to demonstrate and communicate impact to the National Academies’ external expert review panels established to meet the requirements of the US Government’s Performance Assessment Rating Tool. Williams et al. 92The report stated that intermediate outcomes include adoption of new technologies; changes in workplace policies, practices, and procedures; changes in the physical environment and organisation of work; and changes in knowledge, attitudes and behaviour of the final customers (i.e. employees, employers). End outcomes include various items related specifically to occupational health, including reduced work-related hazardous exposures, and, in relation to morbidity and mortality, reductions in occupational injuries and in fatalities within a particular disease- or injury-specific area.

The outcome worksheet was primarily designed as a practical tool to help NIOSH researchers think through the causal linkages between specific outcomes and research activities, determine the data needed to provide evidence of impact, and provide an organisational structure for the evidence.

The combination of historical tracing with a logic model is interesting because previously historical tracing has been more associated with identifying the impact made by different types of research (i.e. basic vs. clinical), irrespective of how they were funded, rather than contributing to the analysis of the impact from specific programmes of research.

The Wellcome Trust’s assessment framework

The Wellcome Trust’s assessment framework has six outcome measures and 12 indicators of success. 93 A range of qualitative and quantitative measures are linked to the indicators and are collected annually. A wide range of internal and external sources is drawn on, including end-of-grant forms. The evaluation team leads the information gathering and production of the report with contributions from many staff from across the trust.

‘The Assessment Framework Report predominantly describes outputs and achievements associated with trust activities though, where appropriate, inputs are also included where considered a major Indicator of Progress.’93 To complement the more quantitative and metric-based information contained in volume 1 of the Assessment Framework Report, volume 2 contains a series of research profiles that describe the story of a particular outcome or impact associated with Wellcome Trust funding. The Wellcome Trust research profiles are agreed with the researchers involved and validated by senior trust staff.

Although there is no specific overall framework, it is a comprehensive approach. This is another example of a major funder including impact in the annual collection of data about the work funded. On the one hand, the importance of case studies is highlighted: ‘Case studies and stories have gained increasing currency as tools to support impact evaluation’,93 but, on the other hand, the report described an interest in also moving towards more regular data collection during the life of a project: ‘In future years, as the Trust further integrates its online grant progress reporting system throughout its funding activities . . . it will be easier to provide access to, and updates on grant-associated outputs throughout their lifecycle’. 93

VINNOVA

VINNOVA, the Swedish innovation agency, has been assessing the impact of its research funding for some time. The VINNOVA framework consists of two parts, an ongoing evaluation process and an impact analysis, as described in the review for CAHS by Brutscher et al. 37 The former defines the results and impact of a programme against which it can be evaluated. It allows the collection of data on various indicators. The impact analyses, the main element in the framework, are conducted to study the long-term impact of programmes or portfolios of research. There are various channels through which impacts arise, but each specific impact analysis can take a particular form.

In Table 14 (see Appendix 3) we describe one example: the analysis of the impacts of a long-standing programme of neck injuries research conducted at Chalmers University of Technology. 94 This considered the benefits to society through a cost–benefit analysis, the benefits to companies involved through an assessment of the profits expected in the future as a result of the research and the benefits to the research field through traditional academic approaches of considering the number and quality of articles and doctorates, and peer review of the quality of the institute. Eriksen and Hervik94

The aim has been, as far as possible to quantify the effects in financial terms, or in terms of other physically measurable effects, and to highlight the contribution made by the research from the point of view of the innovation system.

This approach is a hybrid in that it does relate to a stream of research funded by a specific funder, but it is at a single unit.

Flows of knowledge, expertise and influence

Meagher et al. 95 developed the ‘flows of knowledge, expertise and influence’ approach to assess the impact of ESRC-funded projects in the field of psychology research. As part of a major analysis of the ways in which research might make an impact, the authors pointed out that one limitation was that their study was on a collection of responsive-mode projects and while they did have a common funder (i.e. the ESRC), they had not been commissioned to be a ‘programme’. This again makes the example more of a hybrid, and the study is described in more detail in Chapter 4, but this is the only application of the approach that we identified in our search.

Research impact framework

The research impact framework (RIF) was developed at the London School of Hygiene and Tropical Medicine by Kuruvilla et al. ,123 who noted that researchers were increasingly required to describe the impact of their work, for example in grant proposals, project reports, press releases and research assessment exercises for which the researchers would be grouped into a department or unit within an organisation. They also thought that specialised impact assessment studies could be difficult to replicate and may require resources and skills not available to individual researchers. Researchers, they felt, were often hard-pressed to identify and describe research impacts, but ad hoc accounts do not facilitate comparison across time or projects.

A prototype of the framework was used to guide an analysis of the impact of selected research projects at the London School of Hygiene and Tropical Medicine. Additional areas of impact were identified in the process and researchers also provided feedback on which descriptive categories they thought were useful and valid vis-à-vis the nature and impact of their work.

The RIF has four main areas of impact: research-related, policy, service and societal. Within each of these areas, further descriptive categories were identified, as set out in Table 3. According to Kuruvilla et al. ,123 ‘Researchers, while initially sceptical, found that the RIF provided prompts and descriptive categories that helped them systematically identify a range of specific and verifiable impacts related to their work (compared to ad hoc approaches they had previously used).’123

TABLE 3 - Research impact framework123

Research-related impacts	Policy impacts	Service impacts	Societal impacts
Type of problem/knowledge	Level of policy-making	Type of services: health/intersectoral	Knowledge, attitudes and behaviour
Research methods	Type of policy	Evidence-based practice	Health literacy
Publications and papers	Nature of policy impact	Quality of care	Health status
Products, patents and translatability potential	Policy networks	Information systems	Equity and human rights
Research networks	Political capital	Services management	Macroeconomic/related to the economy
Leadership and awards		Cost-containment and cost-effectiveness	Social capital and empowerment
Research management			Culture and art
Communication			Sustainable development outcomes

Although it is multidimensional in similar ways to the Payback Framework, the categories were broadened to cover health literacy, social capital and empowerment, and sustainable development.

Another major feature of the RIF is the intention that it could become a tool that researchers themselves could use to assess the impact of their research. This addresses one of the major concerns about other research impact assessment approaches. However, while the broader categorisation has been used, on its own or in combination, in an increasing number of studies124, we are not aware of any studies that have used it by adopting the self-assessment approach envisaged. Nevertheless, it could be useful to researchers having to prepare for exercises such as the REF in the UK.

The Becker Medical Library’s model/the translational research impact scale

Sarli et al. 118 developed a new approach called the Becker Medical Library model for assessment of research. Its starting point is the logic model of the W.K. Kellogg Foundation,121 ‘which emphasises inputs, activities, outputs, outcomes, and impact measures as a means of evaluating a programme’. 118

For each of a series of main headings, it lists the range of indicators and the evidence for each indicator. The main headings are research outputs knowledge transfer; clinical implementation; and community benefit. The main emphasis is on the indicators for which the data are to be collected, and referring to the website on which the indicators are made available the authors state: ‘Specific databases and resources for each indicator are identified and search tips are provided’. 118 The authors found during the pilot case study that some supporting documentation was not available. In such instances, the authors contacted the policy-makers or relevant others to retrieve the required information.

The Sarli et al. 118 article includes the case study in which the Becker team applied the model, but the Becker model is mainly seen as a tool for self-evaluation, with the suggestion that it ‘may provide a tool for research investigators not only for documenting and quantifying research impact, but also . . . noting potential areas of anticipated impact for funding agencies’. 118 It is generating some interest in the USA, including partially informing the Drew et al. 90 implementation of the NIEHS framework described above, and a UK application from Sainty. 99

More recently, Dembe et al. 124 proposed the translational research impact scale, which is informed not only by a logic model from the W.K. Kellogg Foundation and by the RIF,123 but also by the Becker Medical Library model. 118

The authors identified 79 possible indicators, used in 25 previous articles, and reduced them to 72 through consulting a panel of experts, but further work was being undertaken to develop the requisite measurement processes: ‘Our eventual goal is to develop an aggregate composite score for measuring impact attainment across sites’. 124 However, there is no indication provided about how a valid composite score could ever be devised. Although as far as we are aware an application of it has yet to be reported, from the perspective of our review it usefully illustrates how new models are being built on a combination of existing ones.

Societal quality score

Mostert et al. 100 developed the societal quality score using the theory of communication from Van Ark and Klasen. 125 Audiences are segmented into different target groups that need different approaches. Scientific quality depends on communication with the academic sector and societal quality depends on communication with groups in society; specifically, three groups: lay public, health-care professionals and private sector.

Three types of communication are identified: knowledge production, for example papers, briefings, radio/television services, products; knowledge exchange, for example running courses, giving lectures, participating in guideline development, responding to invitations to advise or give invited lectures (these can be divided into ‘sender to receiver’, ‘mutual exchange’ and ‘receiver to sender’); and knowledge use, for example citation of papers, purchase of products, and earning capacity (i.e. the ability of the research group to attract external funding). Four steps are then listed:

• Step 1: count the relative occurrences of each indicator for each department.

• Step 2: allocate weightings to each indicator (e.g. a television appearance is worth x, a paper is worth y).

• Step 3: multiply 1 by 2 = ‘societal quality’ for each indicator.

• Step 4: the average societal quality for each group is used to get the total societal quality score for each department.

It is a heavily quantitative approach and looks only at process, as the authors say that ultimate societal quality takes a long time to happen and is hard to attribute to a single research group. The approach does not appear to control for the size of the group but seems to be more applicable to research at an institution rather than project level.

Research performance evaluation framework

Schapper et al. 72 describe the research performance evaluation framework used at Murdoch Children’s Research Institute in Australia. It is ‘based on eight key research payback categories’ from the Payback Framework and also draws on the approach described in the RIF. 123

The centre has an annual evaluation overseen by the Performance Evaluation Committee, with a nominee from each of six themes and external member and chairperson. The evaluation ‘seeks to assess quantitatively the direct benefits from research, such as gains in knowledge, health sector benefits, and economic benefits’. 72 Data for the Research performance evaluation are gathered centrally by the research strategy office and verified by the relevant theme. The theme with highest score on a particular measure is awarded maximum points; others are ranked relative to this. Each theme nominates its best three research outcomes over 5 years, and is then interviewed by the research strategy team using detailed questionnaires to gain evidence and verify outcomes. Research outcomes are assessed using a questionnaire based on the RIF. There are three broad categories: knowledge creation; inputs to research; and commercial, clinical and health outcomes. The six major areas of outcomes are development of an intervention; development of new research methods or applications; communication to a broad audience; adoption into practice and development of guidelines and policy; translation into practice; and impact of translation and on health.

Realist evaluation

The final approach described in this subsection, realist evaluation, is a relatively new generic evaluation approach originally developed in the field of social policy. It has been applied to evaluating the impact of the NIHR-funded Collaborations for Leadership in Applied Health Research and Care (CLAHRCs). This evaluation by Rycroft-Malone et al. 102 is described in Chapter 4 [see Co-production models (e.g. multistakeholder research partnerships)]. Realist evaluation may be more widely applicable to other programmes in the NIHR. The realist evaluation approach was also used in the evaluation of public involvement in health research in England. 101

Comparing frameworks

The various analyses of research impact assessment frameworks conducted by RAND Europe involved making a series of detailed comparisons. 9,37,38 These included the scoring of 21 frameworks (e.g. SIAMPI, REF, CAHS/Payback) against 19 characteristics (e.g. formative, comprehensive, quantitative and transparency). 9

Over half of the 20 frameworks we described above were included in one or more of the three comparisons of frameworks noted here. Appendix 5 lists all the frameworks appearing at least once in the main analyses in these reviews, and identifies those we have included in our list of 20 frameworks, those for which we have included a later or alternative version, and those not included, with reasons, but some of these are described in Table 14 (see Appendix 3). The additional ones we have included that were not in the three reviews are generally more recent and have been applied specifically to assess the impact of programmes of health research.

In Table 4 we provide a brief analysis of the 20 frameworks described above. Much of the discussion of strengths and weaknesses focuses on specific aspects of particular frameworks, with more generic analysis in Chapter 4. The table of comparisons is intended to inform our assessment of options in Chapter 8.

Figure 4 locates the various frameworks on two dimensions in an attempt to identify clusters of frameworks that might be attempting to do similar things. One dimension is the type of impact categories assessed. We have abstracted the key impact categories described in the frameworks: multidimensional (i.e. covers a range that can include health gains, economic impacts and policy impacts); economic impacts (value of improved health and GDP); policy impacts (including clinical policies); and communication/interactive processes. The other dimension is the level of aggregation at which the framework has primarily been applied and whether the focus is on programmes of work from funders or on the portfolio of work of individual researchers, groups of researchers or institutions. (We classed the REF as being in the producers of research category because the work assessed was funded by multiple organisations and conducted by institutions and their units, even though the assessment results will then be used to allocate the future funds from the specific funding organisation conducting the assessment, i.e. the HEFCE.) Where the focus is on programmes of funded research, the impact assessment is most likely to gather data from individual studies, but these are then pulled together and reported on at an aggregate programme level. Furthermore, there can be some data gathering about the whole programme.

FIGURE 4.

Twenty key frameworks: prime assessment focus/level and impact categories assessed.

Finally, in this section we draw attention to a very different approach: the balanced scorecard (BSC), which is analysed in the CAHS report. 7 Some studies describe health-care systems that include research as part of a BSC approach to assessing performance of their system,130,131 but it is argued that the approach is not a comprehensive impact assessment of research. 7 If, however, a BSC approach is used to assess health-care organisations, and includes research impact as one of the criteria, this could be a mechanism for encouraging health-care organisations to foster research activity in their facilities.

Methods used in empirical impact assessment studies

Our updated review identified several studies that undertook important analysis of the methods used in research impact evaluation. These include the UK Evaluation Forum,19 the CAHS report7 and the report from RAND for the Association of American Medical Colleges. 9 The last analysed 11 methods or tools used in a range of six major research evaluation frameworks; most relate to the collection of data and others to how data are presented. The authors provided a brief description of each with a suggestion of when and how it is used. The 11 methods/tools were set out in alphabetical order: bibliometrics, cases studies, data mining, data visualisation, document review, economic analysis, interviews, logic models, peer review, site visits and surveys. The review by Boaz et al. 5 of studies assessing the impact of research on policy-making identified 16 methods as having been used, with semistructured interviews, case study analysis and documentary analysis as the three most commonly adopted. Milat et al. 129 reported that typically mixed methods were used, which could include publications and citations analysis, interviews with principal investigators, peer assessment, case studies and documentary analysis.

Our review of 110 empirical studies also found that a wide range of methods were adopted, but in various combinations. Frequently used methods included desk analysis, surveys, interviews and case studies. The full range of methods used in the studies listed can be found in Table 14 (see Appendix 3), and below we note some interesting trends and show how our review provides further evidence on long-standing issues about the range of methods available for impact assessments. In relation to surveys, for example, there are concerns about the burden on researchers of completing them and on the accuracy of the data. The burden is widely viewed as having increased with the introduction of the above annual surveys, notwithstanding the attempts to reduce the burden by enabling the data entered to be attached to a range of grants. This increased burden might result in incomplete data in the response to specific questions within the overall survey, and might also have implications for the willingness of researchers to complete voluntary but bespoke surveys that specific funders might consider commissioning. The survey response rates in the included studies varied enormously. The compliance requirements in a survey such as researchfish result in very high formal response rate, but the rate has also been high in other surveys; for example, it was 87% in a study in Hong Kong. 66 The rate, however, was only 22% in a pilot study assessing the impact of the EU’s international development public health programme,27 but they did use a range of other methods as well.

In terms of the accuracy of the data from surveys of researchers, several studies report that, in general, the findings from users were similar to those from researchers, for example Guinea et al. 64 and Cohen et al. 52 When comparisons have been made between the responses to surveys, and the data gathered in subsequent case studies on the same project, researchers have been found not to routinely exaggerate. 2 Indeed, Gutman et al. 132 found that researchers interviewed claimed a higher level of impact on policy than was reported by researchers in a web survey, although the questions were slightly different. Meagher et al. 95 also reported that while, ‘case studies were crucial in illuminating the nature of policy and practice impacts . . . there were no evident contradictions between results obtained by different methods.’

Doubts have also been expressed as to how much researchers actually know about the impact their research might have made. One trend that might provide some reassurance about this is that some of the studies in Table 14 (see Appendix 3) report relatively small-scale research funding schemes in which much of the claimed impact arises from the adoption of the findings in the researcher’s own health-care unit, where researchers are well-placed to know the impact made. Some examples of this were reported by Caddell et al. 96

A balance must be found between coverage and resources. Several of the reported assessments relied on the programme office and/or impact evaluators gathering the data from databases, for example in the case of the evaluation of the impact from the EU’s public health programmes53 and in one of the NIEHS’s studies. 91 However, in both cases and others there were some doubts about whether or not sufficient data could be collected in this way, but one of the advantages was that it did not place the burden on researchers. Other attempts to increase practicality go in other directions. Individual researchers might be encouraged to construct accounts of the impact from their own work. In particular, Kuruvilla et al. 123 designed the RIF as a do-it-yourself approach, which prompts researchers to systematically think about the impact of their work using descriptive categories. The Becker Medical Library model was also primarily seen as a tool for self-evaluation. 118

Case studies tend to provide a wider and more rounded perspective on how the impact might have arisen and can address attribution. They tend to be resource intensive and usually conducted only selectively. One dilemma is case study selection, for which a purposive approach is often adopted. However, a stratified random selection has been used when applying the Payback Framework,2,36 and a recent study in Australia conducted case studies on all the projects in which the respondents had completed two surveys and an interview, thus avoiding any selection bias. 52 Case studies can, however, be conducted through self-assessment, perhaps based on desk analysis. They can then be evaluated by peers in an approach that seems to be becoming increasingly important and broadly successful. 33,105,106

There are also an increasing number of studies reporting attempts to score case studies. In addition to the examples of scoring of self-assessment described above, this also includes scoring case studies produced by impact assessors,36,52,89 or produced initially by central teams in the institution, including the cases produced for the research performance evaluation framework used at Murdoch Children’s Research Institute in Australia. 72

Whatever the method of data collection, attention has been given in several studies to expected benefits. We excluded studies that solely considered potential impact before research was commissioned, but some studies are considering aspects of ‘expected’ impacts in several ways. Some make a comparison between what was expected from a project and what had been achieved. Examples include studies from the EU studies,53,133 from Catalonia/Spain56,58 and from Australia. 70 Studies can also emphasise what impacts are expected from research that has already been completed, but which had not yet arisen at the time of the impact study: such questions are, for example, often a feature of surveys in studies applying the Payback Framework. This also includes the application of the framework to assess the impact of the funding provided for biomedical research by the annual TV3 Telethon in Catalonia. 58

Attempts are also being made to develop ways to consider the impact of programmes as a whole in addition to the impact that might come from the collation of data on individual projects. This overlaps with consideration of conceptual frameworks, where, for example, we discussed the role of realist evaluation in assessing one of the CLAHRCs,102 but it can also relate to the methods used in other studies. For example, in their assessment of the Austrian HTA programme, Schumacher and Zechmeister134 set out the methods they had used and the issues that could be addressed by each one, including attempts to identify the development of a HTA culture. Rispel and Doherty135 claimed that in their assessment of the impact of the Centre for Health Policy in South Africa, their own experiences gave them an ‘insider–outsider’ perspective, and that a rounded view of the Centre was provided by interviewing people with a predominantly ‘insider’ perspective, and others with an ‘outsider’ perspective.

Finally, in the 2007 report there was speculation regarding whether a conceptual framework was really needed or whether it might be possible just to apply some of the methods. It was claimed, however, that a conceptual framework could be most useful in informing the structure of a range of methods, such as documentary analysis, surveys and case study interviews. This was seen to be the case with the Payback Framework, and has remained so, as illustrated by the both the survey and the semistructured interview schedule included in the article describing the assessment of the impacts from Asthma UK funding. 51 This is also the case for newer frameworks such as the RIF.

Summary findings from multiproject programmes

The findings from the analysis of multiproject programmes reported in the 2007 review provide a context for the current analysis. That review found that the six impact assessment studies that were focused on HTA programmes reported that the number of individual projects making an impact on policy ranged between 70% and 100%. The 10 impact assessment studies that were focused on ‘other health research programmes’, claimed that the number of individual projects making an impact on policy ranged between < 10% and 53%, and the number of projects making an impact on practice ranged between < 10% and 69%. These findings reflected the different roles of the two identified groups of programmes, but there was also considerable diversity within the nature of the programmes within each group.

The study of the impact of the first decade of the NHS HTA programme was reported as the main part of the 2007 report. However, the study was not included in the literature review chapter of that 2007 report because that review included studies published up to a cut-off point of mid-2005, and had been conducted in order to inform the assessment that was undertaken of the NHS HTA programme. Therefore, the findings below, from the survey of the lead researchers conducted as part of the assessment of the HTA programme, were not referred to in the review chapter. They show a similar pattern to that identified in the 2007 review, that is, an even higher level of impact being claimed for the Technology Assessment Reports (TARs) than for the other types of HTA-funded research, which, in the case of trials, are nearer to the research in the ‘other research programmes category’ than they are to appraisals that constitute the work of most HTA programmes (Table 5).

In our current review a collation of the quantitative findings from studies assessing the impact from multiproject programmes (such as the HTA programme) and published since the previous review conducted in 2005 should provide a context for the results from the parallel study being conducted of the impact from the second decade of the HTA programme.

The diversity of circumstances makes it difficult to be certain about which studies to include, but we classified 26 studies as being empirical studies of the impact from multiproject programmes, and a further two studies of the impact from research training have been included because the impact assessment covered the wider impact made by the research conducted in each training award, as well as the impact on the trainees’ subsequent careers (see Table 6 for the included studies). Even for these 28 studies there is considerable diversity in a range of aspects, including:

• types of research and modes of funding of the programmes of research assessment

• timing of impact assessment (some while the programme was still continuing, some conducted years afterwards)

• conceptual frameworks used for assessment (e.g. some ask about impact on policy, including guidelines, and separately ask about impact on practice; but others ask about a combined ‘decision-making’ and have that as an impact category)

• methods used for collecting and presenting data in impact evaluations (e.g. some present percentage of projects claiming each type of impact and some present the total number of examples of each type of impact, making it impossible to tell how many projects are represented by the total number because some projects might have generated more than one example of a particular type of impact).

It is likely that there will be different levels of impact on policy achieved, for example by a programme of responsive mode basic research than by a programme of commissioned HTA research. However, studies assessing impact from research do not necessarily fall into such neat categories because different funders will have a different mix of research in their programmes and portfolios. Therefore, we have listed all 28 studies (Table 6), but do not include the figures for each study for the percentage of project principal investigators claiming to have made various impacts.

All the data for the individual studies are available from Table 14 (see Appendix 3), but here in Table 7 we show the average figures for the 23 of the 26 multiproject programmes in which the data were presented in terms of the number, or percentage, of individual projects claiming to have made an impact in the categories being assessed. Presenting it in this way allows the overall picture from the quantitative analysis of multiproject programmes to be seen, but also allows a commentary to include some data from individual projects, while at the same time describing key features of a particular research programme, including sometimes the context in which it had been conducted. Table 7 presents the averages and the range on each of the following criteria: impact on policy; impact on practice; a combined category, for example policy and clinician impact, or impact on decision-making; and impact in terms of improved care/health gain/patient benefit.

These are considered in turn.

Discussion

The considerable growth of interest in assessing the impact from health research was captured in our review. We identified an increasing number and range of conceptual frameworks being developed and applied, and included 110 new empirical applications (see Appendix 3), in comparison with the 41 reported in the review published in 2007. 2 In particular, we described and compared 20 frameworks or approaches that had been applied since 2005, some of them having also been described in the previous review. Quite a few of the 20 frameworks, and others, built on earlier frameworks, and sometimes combine elements from several. This partly reflects the need to address the various challenges identified as facing attempts to assess the impact from research.

The Payback Framework39 remains the most widely used approach for evaluating the impact of funded research programmes. It has been widely applied, and sometimes adapted and refined, including in the CAHS framework115 and Banzi’s research impact model. 4 Other robust models that show promise in capturing the diverse forms of health and non-health impacts from research include the RIF123 and various approaches to considering the economic impacts of health research. A comparison of the 20 frameworks indicates that while most, if not all, could contribute something to the thinking about options for future assessment of impact by NIHR, some are more likely than others to be relevant for assessing the impact of the bulk of the portfolio.

There is considerable diversity in terms of the impacts measured in the studies examined. Some of them make no attempt to move beyond the assessment of impact on policy to consider whether or not there has been any health gain. Others that adopt a multidimensional categorisation often recognise the desirability of identifying health gains, but, in practice, lack the resources to make much progress in measuring the health gain even in those cases (usually a small minority) where some links can be established between the research being assessed and the eventual health gains. Finally, some studies, at least in a few of the case studies included in an overall assessment, do go on to attempt to assess the health gains that might be at least partially associated with particular research. The variations depend on combinations of (1) the type of research portfolio that is being assessed, for example if it is a commissioned programme; (2) the type of framework being used for the assessment; (3) the resources available; and (4) the actual outcomes from the particular examples of research assessed. The multidimensional categorisation of impacts, and the way it is applied in approaches such as the Payback Framework and CAHS framework, allows considerable flexibility. In each case study, for example, it might be appropriate to take the analysis as far along the categorisation as it is practical to go. So, for some it might be possible to show an impact on clinical policies, such as guidelines or screening policies, and then for a minority of those there might be opportunities to take the analysis further and explore whether or not there is evidence from databases of practice change, screening uptake rates, etc. that could feed into an estimate of possible health gain.

Although interviews, surveys, documentary analysis and cases studies remained the most frequently used methods to apply the models, the range of methods and ways in which they were combined also increased. The purpose behind a particular study often influenced the frameworks and methods adopted. We identified 28 studies that had reported the findings from an assessment of the impact from all the projects in multiproject programmes. We were able to compare the findings from 25 of these studies, and, as in the previous review, they varied markedly in the percentage of projects within each programme that seemed to make an impact on health policy and practice. Generally, the programmes with the highest levels of impact were HTA-type programmes in which the projects were primarily reviews or appraisals that fed directly into policy-making processes. Other programmes in which quite high proportions of projects were seen to be making some impact were ones in which there had been one or more of the following: thorough needs assessments conducted beforehand; frequent interactions with potential users; and the existence of ‘receptor’ bodies that would receive and potentially use the findings. A key conclusion from this is that impacts from such programmes were best assessed by frameworks devised to capture data about the context and interactions related to research programmes.

The consideration of the findings from studies and the role of the different possible frameworks and methods have to take account of the major recent developments in impact assessment described in the chapter, namely the introduction of regular monitoring of impact, for example through researchfish,76 and the major, and largely successful, REF exercise in the UK. 33 Both of these developments mean that any future additional assessment of impact by NIHR will take place in an environment in which there is already considerably more data available about impacts than was ever previously the case. Both developments also demonstrate that impact assessment can be conducted in ways that identify that a wide range of impacts come from health research and, therefore, provide a degree of endorsement of the previous smaller exercises. However, many challenges remain in assessing research impact and further consideration of the most appropriate approaches is highly desirable.

Different philosophical roots: five ‘ideal types’

Different approaches to measuring research impact also rest on different (usually implicit) assumptions about the nature of knowledge. To understand impact and measure it in a valid way, we need to clarify questions of ontology [what is (research) knowledge?], epistemology (how might we come to understand that knowledge), the purpose of scientific inquiry and the mechanism by which research is assumed to link to practice. The philosophical assumptions of different approaches to understanding scientific inquiry are summarised in Table 8.

Traditionally, HTA’s focus has been on experimental studies of drug treatments or surgical interventions from a positivist ‘hard science’ perspective. Mostly outwith HTA’s terms of reference, but within the wider scope of health services research [see, for example, the NIHR CLAHRC programme (www.clahrcprojects.co.uk/), the Wellcome Trust’s ‘Society and Ethics’ programme (https://wellcome.ac.uk/) or some elements of the European Commission’s Horizon 2020 programme (https://ec.europa.eu/programmes/horizon2020/)], are research designs such as collaborative codesign, policy analysis, health systems analysis and organisational case study that are built (variously) on constructivist, critical or performative assumptions (see Table 8). HTA has occasionally commissioned overviews of qualitative research from a constructivist perspective,142,143 systematic reviews with a realist component144 or a systematic review of action research that acknowledged (although it did not prescribe) a critical perspective. 145 However, even considering HTA’s main focus on clinical trials, few, if any, models of research impact assume a direct, linear and unproblematic link between a trial and its subsequent impact. Most begin with a basic logic model and enhance it with an interpretive account of the different relationships, interactions and contextual influences that affect the research–impact link.

In this chapter, we will outline some research approaches and models of research impact that draw on the wider range of philosophical assumptions set out in columns 3–6 in Table 8.

Different readers will have different views on the ‘correct’ or ‘preferred’ approach to research or the measurement of impact. However, it is important to note that these different philosophical positions tend to be linked to very different research topics and questions. Positivist assumptions tend to underpin quantitative and experimental studies (especially the question of if and for whom a particular intervention ‘works’, and what magnitude of benefit can be expected). In contrast, studies with a strong explanatory component (e.g. those that seek to build theory about a complex social intervention) may reject the positivist assumption that there is a transferable ‘effect size’ and focus instead on describing interactions and/or drawing out theoretical mechanisms of change. Studies that are driven by a passionate commitment to improve the lot of a marginalised or underserved group, such as refugees or the homeless, may find a critical perspective (and an action research study design) more appropriate and feasible than a randomised trial.

The literature on impact in health services research is increasingly philosophically diverse, attempting to combine the outputs of (positivist) evidence-based medicine (e.g. quantitative findings on the efficacy of tests and treatments) with a broader (constructivist, realist, performative), epistemology of research utilisation that incorporates various social science disciplines (notably, social psychology, organisational sociology, social policy, and science and technology studies). Some of these extended models sit more comfortably in a social policy paradigm than in implementation science. 146 In the following section (see ‘Logic models’ of research impact: strengths and limitations), we summarise these contrasting philosophical ideal types and suggest how the different models of research impact align with them, and the implications this has for assessing research impact in an increasingly diverse health research system.

Given that many of the models reviewed in this report draw on multiple philosophical assumptions, it is worth introducing a composite philosophical position (not shown in Table 8), which is pragmatism. Ontologically and epistemologically eclectic pragmatism proposes that when combining scientific and practical knowledge (e.g. when attempting to link a body of research with its application in the real world), the relevance of each competing position should be judged in terms of how well it addresses the problematic situation or issue at hand. 147

It should be noted that positivism is the only philosophical position that strongly supports a ‘model’ with ‘tools’ to apply it. Hence the perspectives in other philosophical schools might best be thought of as ‘approaches’ rather than as ‘models’.

‘Logic models’ of research impact: strengths and limitations

Most, although not all, approaches to assessing research impact in the health sciences include some kind of ‘logic model’ – defined as a depiction of the logical (implicitly, causal) relationships between the resources, activities, outputs and outcomes of a programme. However, few, if any, of these approaches assume that the link between research and impact is as linear and direct as the logic model implies. Different approaches enhance the logic model in different ways. In this section we summarise the approaches that include a logic model and review the strengths and limitations of the logic model.

Chapter 3 described a number of widely used approaches from the mainstream health services research (or ‘research on research’) literature, including the Payback Framework and its variants,39 the monetary value approach, the Quebec HTA approach,43,44 the CAHS approach,7 Banzi’s research impact model,4 the NIEHS logic model,63 the medical research logic model,120 the NIOSH logic model,92 the Wellcome Trust’s assessment framework,93 the VINNOVA framework,37 the RIF,123 the Becker Medical Library model,118 the research performance evaluation framework,72 the UK REF and the Australian RQF. 128 All these examples consist partly or wholly of a logic model (although Chapter 3 also includes some examples of constructivist, realist and performative approaches).

Similarly, all the models described in Chapter 5 on the monetary value are essentially logic models, whether ‘top down’ or ‘bottom up’, which link inputs (research funding) with the research process and then outputs and (monetised) impacts. Chapter 6 (the impact of RCTs via their role in systematic reviews and meta-analyses) and Chapter 7 (the impact of RCTs on stopping ineffective interventions) also relate exclusively to logic models.

The strengths of logic models are the way in which the links between inputs, processes, outputs and outcomes are carefully drawn out, and the fact that mediating and moderating variables can be added to the model to account for successes, failures and partial successes. A robust logic model, systematically applied, should produce valid and reliable statements about the relationship between these variables. However, in the real-world application of research evidence, it is widely recognised that not all research impact links can be predicted or reliably quantified. Part of the elegance of many frameworks that include a logic model is the sophistication of the caveats and nuances they accommodate to explain, for example how, why, by whom and influenced by what historical and contextual factors the impact unfolded as it did. In the Payback Framework, for example, the basic logic model is enhanced by a narrative account of factors and influences, including the context within which the research takes place and the interactions between potential users and the researchers. These are widely viewed as key features of the framework.

Although logic models are only ever an approximation of reality, they can be extremely helpful as conceptual tools. They generally cover a range of impacts considered important to stakeholders, and the various elements of the models provide a framework for organising data collection, analysis and data presentation consistently, taking account of inevitable variations between projects or programmes. A robust basic model, while being rigorously applied, can also be refined further in collaboration with the funders of particular studies and other stakeholders (e.g. patient organisations) to ensure that the assessment of impact focuses on the categories important to those stakeholders. Such models can often be applied flexibly in a way that is compatible with the values of the research funders and researchers, who are usually concerned to show that they are contributing to improved health care.

Even when they include such scope for flexibility and caveats, logic models may be criticised by social scientists who question their value for assessing research that is inherently non-linear (e.g. the evaluation of real-world social programmes that follow the non-linear dynamics of complex systems). We list below the alleged downsides of logic models as described by their critics. 5,6,14,95,115,116,148–151

1. Assumption of linear causality: to a greater or lesser extent (e.g. depending on the degree of ‘permeability’ acknowledged by their architects), logic models reflect a deterministic, research-into-practice mind set that is incapable of fully capturing the messiness and non-linearity of the relationship between research, practice and policy. The epistemological assumption behind unenhanced logic models is that with careful measurement and synthesis of input, process and context variables, it is possible to draw meaningful conclusions about the link between a research programme and subsequent impact, and predict comparable impact in the future. Critics say that this assumption is highly questionable in certain circumstances, notably when the context is complex and multiple input variables are rapidly changing.

2. Disciplinary bias: logic models are said to privilege ‘hard’ research, such as trials and epidemiological studies, over ‘soft’ research, such as qualitative or developmental studies, and to valorise easily monetised impacts such as licensing and start-up creation. Furthermore, the quest to measure the measurable in a rational, ‘objective’ way creates perverse incentives to overlook the unmeasurable elements in any research discipline.

3. Temporal bias: different kinds of research achieve impact over different time scales, and hence an overly rigid logic model will miss impacts at extremes of these scales. In general, the longer the time scale, the more diffuse the chain of causation. As Kok and Schuit stated: ‘pathways from research to ”impacts” are very diverse: sometimes short and traceable, but often long, through multiple reservoirs, and via utilization at untraceable times and places’. 116

4. Attribution: the extent to which an impact can be attributed to a particular research project is a matter of judgement. In reality, attribution is often an inexact science in which accuracy attenuates with each stage in the logic model.

5. Additionality: return on investment models are not designed to address whether or not the claimed impact would still have occurred had the research not been done. There is also the related question of opportunity costs – might the research budget have been spent differently, with greater benefit to the public good?

6. Excessive abstraction: according to critics, logic models are elegant and parsimonious in the abstract and convey the impression of rigour through ‘hard’ analytics, but (depending on how rigidly they are applied) they bear little relation to the messier real-world use of knowledge by human actors and how knowledge is interpreted, negotiated and valued by wider society.

7. Impracticality: application of logic models (which tend to be multimethod, multilevel and seek to build a rich picture of the numerous interacting influences on impact) is resource intensive, hence not a practical or affordable option in most situations. Reducing the rich, multimethod case study approach intended by the original authors to a crude inventory based on tick-box surveys and a handful of standardised interviews will not produce valid or reliable data.

8. Ethical issues: the impact of research may be significant and far-reaching but morally questionable (e.g. if it is achieved at the expense of environmental damage) or even harmful (e.g. if it distorts rather than informs decision-making). It has even been argued that the dominance of economic models of impact reflects a sinister development in which key moral questions about the public good (what kind of science is morally right) are downplayed in favour of instrumental practices aimed at a narrow range of utilitarian goals, especially innovation and economic growth.

The validity of all the above arguments will, of course, depend on context and specifics, but, as noted above, the crude and unenhanced logic model that depicts a naive and deterministic relationship between research and impact has long been rejected in favour of more pragmatic and flexible hybrids. As the range and diversity of research expands, the models reviewed in the next section should perhaps be viewed not as substitutes for (or competitors of) more widely used approaches based on logic models, but as complementary approaches that might prove fit for purpose in particular circumstances.

Alternatives to the logic model approach

Different models of research impact are more or less appropriate for different study designs and research themes – a finding that was evident in the previous HTA review. 2 Similarly, the CAHS panel commissioned separate analyses of how which approaches might work best for each of three of the four ‘pillars’ of research used to categorise health research in Canada: pillar II, clinical research;152 pillar III, health services research;113 and pillar IV, population and public health research. 114 (Pillar I, basic biomedical research, was excluded on the grounds that it ‘is the area where most has been said on understanding the impacts of health research’.)

Some Australian studies have also considered this issue. Cohen et al. 52 found that ‘single intervention research studies can and do have concrete and measurable post-research real-world impacts . . . on policy and practice’. 52 This recent study adds to the view that the degree of impact identified and reliably attributed (at least over relatively short time scales) might vary depending on the type or research, and context in which it is conducted and its findings presented. Kalucy et al. 65 used the Payback Framework to assess the impact of primary care research and reported that the ‘logic model’ of the framework worked better for a RCT than it did for an action research study.

Impact assessments are conducted for a variety of audiences (Table 9) and purposes [the ‘four As’ of advocacy (for research), accountability, analysis (of why and to what extent research is effective) and allocation]. 2,9

Guthrie et al. 9 examined six impact frameworks in detail and suggested that all six could be used if the purpose was advocacy or accountability. However, if the purpose were analysis, only three of the six frameworks would be fit and if the purpose were allocation, a different three would be fit. They applied the four As to inform a decision tree for developing a research evaluation framework.

Audiences and purposes clearly overlap to a degree. A major audience is likely to be the organisation funding the original research, who may also be funding the impact assessment. Research funders might hope a single impact assessment will inform both analysis and allocation, but may be unaware that different assessment tools are more or less suited to different purposes.

Bearing in mind the consistent evidence in favour of different models for different types of research, different audiences and different purposes, the next few sections address alternatives to the logic model for assessing research impact. Although many are only marginally relevant to the current remit of the HTA programme, they are nevertheless important components of a broader toolkit.

Constructivist models of impact (developed in social sciences)

As column 3 in Table 8 shows, constructivist (sometimes called interpretivist) research focuses on people’s interpretations of the world. This is important for the study of research impact because policy-makers, clinicians and patients interpret the world they inhabit – and they also interpret research evidence in ways that may not align with researchers’ perspectives.

This misalignment was systematically documented in a detailed ethnographic study undertaken in the 1970s,49 and more recently updated,153 of the interactions between national policy-makers and the university-based researchers from whom they commissioned research (the so-called ‘Rothschild experiment’ in the English Department of Health and Social Security). Kogan and Henkel’s49 landmark study demonstrated a number of key principles that are still relevant to the measurement of research impact today. First, science and government are from different cultural worlds; interaction between them is a ‘contact sport’, in which success depends on sustained linkage and exchange with ‘knowledge brokers’ playing a key role. 154

Second, despite the ubiquity of simple, linear models of research-into-policy, scientific research and the business of government is, in reality, highly complex. Simplistic models fail to capture their important nuances. In particular, science and government are interdependent and mutually shaping, hence even commissioned research does not follow a simple, customer–contractor logic.

Third, research priorities, even in applied fields, are rarely self-evident, partly because different stakeholders view the world differently and have competing vested interests.

Finally, the different perspectives within each ‘multimodal’ group of scientists and users can have major consequences; for example, if the chief scientist took a narrow, positivist and quantitative view of what science should look like, this would limit the scope for university researchers and research users to develop more emergent partnerships to address highly complex real-world problems.

In 1979, sociologist Carol Weiss34 challenged the prevailing assumption that research impact is direct and linear (and hence that it can be meaningfully summarised in logic models). Drawing on empirical studies in the social sciences, she argued that the assumed knowledge-driven mode of impact, along with problem-solving mode (research commissioned directly to solve particular policy problems – as in the Rothschild experiment) were rare, not least because the findings of social science research tend to illuminate the complexity and contingency of phenomena rather than providing simple and universal solutions to them.

Weiss and others have shown that research in the social sciences (which includes much applied health research) is generally characterised by an interactional mode of impact in which researchers and policy-makers, through repeated interaction over time, come to understand each other’s worlds and develop shared goals and approaches. 34 Impact may also occur by other non-linear mechanisms including enlightenment – as Hanney et al. 35 stated, drawing on the work of Thomas,155 ‘the gradual “sedimentation” of insight, theories, concepts and perspectives’, as a result of continuing exposure to research insights and ideas. Finally, research findings may be used by policy-makers symbolically (either politically to support a particular course of action or tactically to delay a decision, perhaps commissioning new research in order to buy political breathing space). 34,35,156

Knowledge in the social sciences has complex properties. It is fluid, context dependent, embodied by individuals and embedded in organisational routines and norms. Hence social scientists are uncomfortable with models of impact that rest heavily on the ‘transfer’ or ‘implementation’ of an assumed fixed body of knowledge. Conversely, they are often keen to explore the processes by which knowledge, which is assumed to take multiple forms, is interpreted, negotiated, transformed and applied in practice, and how context may profoundly affect these processes. 149

Meagher et al. ,95 for example, applied a model with some parallels to the Payback Framework, but resting on interpretivist assumptions and placing more emphasis on processes and activities, to study the impacts of ESRC-funded research in a number of detailed case studies. They found that conceptual (e.g. indirect, enlightenment based) impacts were more common than instrumental (e.g. direct, knowledge driven) ones. They also found that most principal investigators had a naive and linear view of the research–impact link (e.g. few knew about interactive or enlightenment mechanisms or the need for ongoing linkage and exchange with policy-makers). They questioned the value of tracking impacts in the absence of specific activities aimed at facilitating uptake. Indeed, they felt it might be inappropriate to try to measure something that one has not expressly tried to bring about. They commented:

It was extremely difficult to attribute with certainty a particular impact to a particular project’s research findings. It was often more feasible to attach an impact to a particular researcher’s full body of research [. . .] Changes in practice or policy often appear to stem from a general ‘awareness-raising’ or conceptual shift.

p. 17095

The ‘full body of research’ referred to in the above quote has been described by RAND as a ‘research cloud’ and explored using electronic bibliometrics. 157

Brambila et al. 137 used a rare longitudinal case study methodology to demonstrate Weiss’s ‘incremental’ mechanism of impact in a sample of 44 community-based health care projects in Guatemala between 1988 and 2001. Like Meagher et al. ,95 they found few linear impacts directly attributable to single projects. Rather, policy change occurred through ‘a gradual process of information sharing, where researchers influence decision-makers through a continual stream of information rather than a single set of findings’. 137

de Goede et al. 158 developed a three-phase framework for capturing the complexity of research utilisation: (1) describe the research network and the policy network; (2) describe the types of research utilisation (classified as instrumental, conceptual and symbolic); and (3) describe the (reciprocal) interactions between researchers and policy-makers. Barriers to the effective uptake of research may occur at the level of expectation (are policy-makers ‘ready’ for these findings?), transfer (how effectively and appropriately are findings communicated?), acceptance (are findings seen as credible and true?) and interpretation (what value do policy-makers place on them?). Using three detailed case studies, these authors showed, like Meagher et al. ,95 and Kogan and Henkel earlier,49 that most research utilisation was conceptual and that non-uptake could often be explained by a mismatch of world view and problem definition between researchers and policy-makers.

de Jong et al. 159 developed a model for incorporating context into impact assessment (Figure 5), as different fields of inquiry provide very different contexts for research and impact. (This finding resonates with Nicolini et al. 160 on the different nature of knowing in different clinical specialties.)

FIGURE 5.

de Jong et al. ’s framework for assessing impact in context. Reproduced from de Jong SP, van Arensbergen P, Daemen F, van der Meulen B, van den Besselaar P. Evaluation of research in context: an approach and two cases. Res Eval 2011;20:61–72, by permission of Oxford University Press. 159

Step 1 considers ‘research field context’: nature and range of research, how quality is defined, and implications for agenda-setting, collaboration, knowledge dissemination and impact. Steps 2–4 consider missions of the research group, audiences and outputs within this wider context. Contrasting examples of architecture and law illustrate that attempts to assess research impact make little sense without preliminary contextualisation of the field.

A study from Australia based on five case studies in contrasting disciplines found similarly that ‘disciplinary and methodological context matters when it comes to understanding the translation, dissemination, and utilization of academic social research.’161

More recently, Lemay and Sá162 depict research utilisation as having the non-linear dynamics of a complex adaptive system, composed of multiple interacting entities, coevolving, locally adaptive, self-organising, path-dependent and sensitive to initial conditions. They view research users as active problem-solvers and generators of knowledge, not passive receptacles. They propose that impact may (theoretically at least) be modelled using computational techniques such as agent-based modelling, data mining or socionics. Such approaches would require a shift in the policy mind set.

Normative policy development and implementation are about directing, controlling and minimising uncertainty about outcomes. Taking into account the contingent, emergent and unpredictable nature of research use would imply acknowledging and accommodating unpredictable outcomes that might emerge over time.

p. 480162

In a widely cited systematic review of knowledge utilisation, Contandriopoulos et al. 163 depicted knowledge in two essential forms: individual, that is, held in people’s heads and translated (or not) into action by human will and agency (a conception of knowledge that rests largely on positivist assumptions); and collective, that is, socially shared and organisationally embedded (a conception that rests on more constructivist assumptions). They reviewed the mechanisms by which knowledge may become collectivised, including efforts to make it relevant, legitimate and accessible and to take account of the values and priorities of a particular audience. If there is broad agreement on what the problem is and what a solution would look like, arguments can proceed through logic models along the lines of conventional scientific inquiry (e.g. strength of research evidence). If not, the research impact challenge must take account of people’s interpretations, and hence enters the more fluid and subjective realm of political science in which research use is, in Weiss’s taxonomy, instrumental and/or tactical rather than knowledge driven. 34

In summary, whereas the natural sciences can be thought of as trading in more or less universal truths, knowledge in the social sciences (including the study of how individuals interact and how organisations and communities operate) is more fluid, dynamic and value-laden. The uptake and use of knowledge depend heavily on context; impacts may be diffuse, subtle, diverse and unpredictable; and causality tends to be explanatory rather than probabilistic.

Realist models: impact as ‘theory of change’

The studies described in the previous sections applied an interpretivist lens to explore the research–impact link in (mostly) single case studies without making predictions about other cases. A different approach, realist evaluation, uses case study methodology but through abductive theorising about context–mechanism–outcome configurations, seeks to make generalisable statements about what tends to work for whom in what circumstances. A preliminary paper purporting to apply realist methods to the study of impact has been published, although this monograph would more accurately be described as an introduction to realist methodology in general. 164 The principles of the realist approach are summarised below.

Realist evaluation was developed by Pawson and Tilley in the 1990s for the evaluation of ‘what works for whom in what circumstances and how?’165 This early work made the following points.

• Complex interventions (what Pawson and Tilley call ‘social programmes’, e.g. an intervention to encourage people to consult their general practitioner rather than attend the emergency department) are an attempt to create some level of social change.

• These interventions ‘work’ by enabling participants to make different choices.

• Making and sustaining different choices requires a change in a participant’s reasoning (e.g. in their values, beliefs and attitudes or the logic they apply to a particular situation) and/or the resources (e.g. information, skills, material resources, support) they have available to them. This combination of ‘reasoning and resources’ is what enables the intervention to ‘work’ and is known as a ‘mechanism’.

• Complex interventions ‘work’ in different ways for different people because contexts (social, economic, organisational, interpersonal) have different influence on different people, triggering different mechanisms: context + mechanism = outcome.

• As complex interventions work differently in different contexts and through different change mechanisms, programmes cannot simply be replicated from one context to another and automatically achieve the same outcomes (i.e. impacts). Theory-based understandings about ‘what works for whom, in what contexts, and how’ are, however, transferable.

• Therefore, one of the tasks of evaluation is to learn more about ‘what works for whom’, ‘in which contexts particular programmes do and don’t work’, and ‘what mechanisms are triggered by what programmes in what contexts’.

In summary, a realist approach to research impact is centrally concerned with looking at how different research programmes may have different impacts in different settings. Empirical studies applying realist methodology to the evaluation of research impact are currently sparse,166 but the approach is rapidly growing in popularity in the health-care field, and hence we flag it here as a potential (if largely untested) option.

Participatory and critical emancipatory models of impact

One of the most striking developments in UK health services research since the publication of the 2007 HTA report has been the increased emphasis on patient and public involvement as a contributory factor in research impact. 167 User involvement on grant applications and study steering groups is widely promoted and an important criterion against which studies are evaluated. However, this involvement is sometimes under-theorised, depicted as instrumental (aimed at increasing recruitment to clinical trials) and couched in strongly positivistic terminology, with an emphasis on standardisation and formal measurement of variables. One study, for example, was entitled Involving Service Users in Trials: Developing a Standardised Operating Procedure. 168 Another offered a 31-point checklist to ensure uniformity in this aspect of research. 169

An alternative approach to public involvement in research has been presented in the critical social science literature. 170 From this perspective, research impact is centrally concerned with achieving social justice and improving the lives of disadvantaged groups. Action research, sometimes known as community-based participatory research (CBPR), seeks to redress the adverse influence of social determinants of health (e.g. poverty, sex, ethnicity, education/literacy, citizenship status and access to services) through collaborative, multistakeholder activity. 171

A key challenge of CBPR is ensuring that the research process should remain democratic despite imbalances of resources and power, so some tools have been designed to measure the extent of power sharing. White,172 for example, writing in the CBPR literature, distinguishes nominal involvement of the lay public (undertaken to confer legitimacy on a project), instrumental involvement (to improve its delivery and/or efficiency), representative involvement (to avoid creating dependency) and transformative involvement (to enable people to influence their own destiny).

Additional dimensions of the CBPR process may also be assessed to estimate the level of democratic decision-making, such as the extent to which research designs are culturally and logistically appropriate; the extent of measures to develop capacity and capability in stakeholder groups; how, and to what extent, conflicts are managed; and the extent to which mutual trust builds over time. 173

Martin174 classifies involvement of practitioners (e.g. clinicians) in collaborative research on a five-point scale: informant (‘type 1 co-production’ – supplying data for a mode 1 research study but no other involvement); receiver (‘type 2 co-production’ – involved at the end of a mode 1 study to receive the findings, usually on terms set by the researchers); endorser (‘type 3 co-production’ – involved from an early stage to endorse, but not influence, priority setting and research programmes); commissioner (‘type 4 co-production’ – involved from the outset to conceive and initiate studies that are taken forward by researchers); or co-researcher (‘type 5 co-production’ – working democratically alongside researchers at every stage in the research).

Macaulay et al. 171 applied CBPR to health care. They proposed some indicators of whether or not a community–campus partnership was truly democratic, including (1) ‘Were the research goals, objectives and methods negotiated among all partners?’; (2) ‘Were the terms of the community–researcher partnership made explicit and agreed?’; (3) ‘Who evaluated the project and how?’; (4) ‘Where were the data filed and who had control over their subsequent analysis and publication?’; (5) ‘What were the arrangements for resolving disagreements?’; and (6)’ How and to whom were the findings disseminated?’.

In CBPR, partnerships succeed largely through partnership synergy – defined as combining people’s perspectives, resources and skills to ‘create something new and valuable together – a whole that is greater than the sum of its individual parts’. 175 Partnerships are often characterised, at least initially, by conflict, but synergy may increase as cogoverning partners work together, leading to convergence of perspectives by progressive alignment of purpose, values and goals and growth of mutual understanding and respect. 173

Cacari-Stone et al. 176 linked the CBPR approach to policy-making by linking CBPR contexts (political–societal and specific collaborative histories) and partnership processes (e.g. equitable decision-making or leadership) to intermediate research and system or capacity outcomes, and more distally to health outcomes (Figure 6). 176 They depict the policy process as iterative, non-linear and characterised by windows of opportunity. CBPR may influence this both instrumentally (by generating evidence) and interactively (through civic engagement).

FIGURE 6.

Conceptual model for illustrating the link between CBPR and policy-making. Cacari-Stone L, Wallerstein N, Garcia AP, Minkler M. The Promise of Community-Based Participatory Research for Health Equity: A Conceptual Model for Bridging Evidence With Policy. Am J Public Health 2014;104:1615–23, with permission from The Sheridan Press (on behalf of The American Public Health Association). 176

Community-based participatory research depicts sustainability of research impact in synergistic terms as progressive strengthening of the community–campus partnership for further collaborative knowledge production (hence the feedback arrows from the outcomes of one project to the context for the next project in Figure 6).

The literature on CBPR, and on ‘socially engaged’ research more generally, uses the language of critical sociology and critical public health. It is a world away from most clinical research, which remains dominated by the language and logic of epidemiology and RCTs. However, while the RCT is predicated on the positivist assumption that knowledge is fixed and stable and political issues lie beyond the analytic frame (see Table 8), the use of evidence from RCTs requires attention to the policy process and hence to interpretation (see the ‘constructivist’ column in Table 8) and the balance of power (see the ‘critical’ column in Table 8).

In 2011, Kessler and Glasgow177 famously called for a 10-year moratorium on RCTs to allow the health services research community to learn and apply the concepts of a more applied and socially engaged approach. They subsequently drew the disparate paradigms of RCTs and CBPR together in an evidence integration triangle designed to ‘[marry] rigorous design focused on internal validity and theory-driven hypotheses with an increased focus on external validity, contextual considerations, and stakeholder relevance’. 178

Glasgow et al. ’s ‘evidence implementation triangle’ (Figure 7)178 is an example of how the research tradition of knowledge translation has sought to embrace a wider range of paradigms. The triangle comprises an evidence-based intervention or policy (perhaps tested in a RCT), a participatory implementation process (perhaps using CBPR or some other developmental approach), and practical (i.e. feasible, affordable, actionable and sensitive to change) progress measures, all studied with attention to context. Although this is a worthy attempt to link the different cultural worlds of clinical trials and real-world implementation, the depiction of CBPR as an ‘implementation process’ for the findings of mode 1 research spectacularly misses the central emancipatory principle of this approach.

FIGURE 7.

Glasgow et al. ’s evidence implementation triangle. 178 Reprinted from Glasgow RE, Green LW, Taylor MV, Stange KC. An evidence integration triangle for aligning science with policy and practice. Am J Prev Med 2012;42:646–54, with permission from Elsevier. 178

Many writers in social science traditions consider that a focus on ‘translation’ or ‘dissemination’ of pre-existing research knowledge offers only limited purchase on the phenomenon of impact. In this section, we review approaches that focus instead on knowledge production.

Co-production models (e.g. multistakeholder research partnerships)

In 1994, a book entitled The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies introduced a new taxonomy: ‘mode 1 scientific discovery’ and ‘mode 2 knowledge production’. 179 This centred on the practices through which knowledge is produced rather than the mechanisms by which it was subsequently disseminated. The book had much influence in the social sciences, but less so in health-care fields. However, with the emergence of complex organisational forms (such as Academic Health Sciences Networks,180 funded by the NHS, and CLAHRC, funded jointly by the NHS and NIHR181–186) oriented towards the collaborative production of research knowledge between universities, the NHS and other partners, the literature on mode 2 research is increasingly relevant for the UK research community – though perhaps less so for the HTA programme in which the commissioning model remains largely mode 1.

Mode 1 research equates to traditional academic science, led by university researchers and then disseminated to stakeholders beyond higher education. In contrast, mode 2 knowledge is generated within its context of application – what Gibbons et al. 179 refer to as a ‘multi-stakeholder transaction space’ – embracing state, economy, culture and the wider public sphere, as well as academic institutions (of which Academic Health Service Networks and CLAHRCs are good examples). In this ‘transaction space’ (the formal board meetings of AHSNs and CLAHRCs, as well as the informal interactions that go on behind the scenes to keep these shows on the road), problems are identified, questions debated, methodologies developed and outcomes disseminated. There are many players, many experts (of different kinds) and an evolving collective view (though rarely a consensus) on what the questions and challenges are.

Gibbons et al. 179 emphasise that in mode 2 research, a range of theoretical perspectives and practical approaches – including, but not limited to, specialist scientific techniques – is mobilised and managed, often for only a limited period, to address a particular set of problems. The planning, execution, dissemination and implementation of research are not separate and linear phases but interwoven, and the relationship between scientists and research users (industry, policy-makers, citizens, patients) is one of Co-production rather than ‘producer’ and ‘consumer’ or ‘commissioner’ and ‘contractor’. The authors of the mode 2 hypotheses depicted the process as highly reflexive.

The research process can no longer be characterised as an ‘objective’ investigation of the natural (or social) world, or as a cool and reductionist interrogation of arbitrarily defined ‘others’. Instead it has become a dialogic process, an intense (and perhaps endless) ‘conversation’ between research actors and research subjects.

p. 4187

The emergence of complex forms for mode 2 research in health care in the UK is mirrored in other countries, for example the Canadian Community–University Research Alliances,188 the Dutch Academic Collaborative Centres for Public Health189 and the Australian Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork. 190 One recent publication coined the term ‘multi-stakeholder health services research collaborations’ to describe the increasingly complex, intersectoral and networked structure of contemporary medical and health services research. 190 These complex forms should be seen as part of the wider emergence of ‘health research systems’ – organised networks of researchers, knowledge intermediaries, policy-makers and others who provide a context for health sciences research and its uptake and application. 35,191 We include brief mention of them here, as even though the HTA programme is not directly linked to such forms, it must take account of their indirect influence on the UK research landscape.

While logic models tend to centre on demonstrating more or less causal connections in the temporal sequence of inputs (research funding), processes (execution of discrete projects or programmes of research, usually following a predefined protocol), outputs (e.g. publications and presentations) and outcomes (impacts on end-users of research), the study of knowledge production has emphasised the non-linearity, messiness and unpredictability of the collaborative knowledge production process. It has also surfaced the importance of power relations and governance for the success of such partnerships, as each partner brings different expectations and goals and the generation of robust and useful research will inevitably depend on managing these conflicts. Evaluations of multistakeholder research collaborations have consistently found that ‘colliding institutional logics’,192 ‘ambiguous loyalties . . . different interests . . . competing goals’193 and ‘multiple accountabilities’. 189 Hinchcliff et al. 190 distinguish between the sanitised written accounts of multistakeholder interactions (‘draped in the formal collaborative language and procedures prescribed by funding agency protocols’) and the reality in which ‘participants . . . view each other pragmatically as consultants, clients or even competitors, rather than partners’. 190

The analysis of developments such as CLAHRCs has (rightly) drawn on a range of different perspectives, approaches and models. Although they are empirically diverse and philosophically heterogeneous, they have in common recognition that the logic model approach is inadequate to capture the multiple (and bidirectional) interactions occurring within the system. These include:

• approaches based on an in-depth case study that emphasised the formative role of real-time evaluation findings in shaping the emergence of CLAHRC activities185

• approaches based on a complex adaptive systems perspective, including that proposed by the World Health Organization Alliance for Health Policy and Systems Research194 and Shepherd’s model of the evidence ecosystem in health and social care195

• adaptation of complex systems theory that focus specifically on conflict, including Bennet et al. ’s196 metaphor of collaborative entanglement (depicting the conflict-ridden, messy, unpredictable and evolving interactions among stakeholders in contemporary mode 2 activity), explored further by Phipps et al. 197,198

• Kislov et al. ’s199 theorisation of CLAHRCs as communities of practice

• Caldwell and Mays’ application of sociological ‘frame analysis’ (Goffman200) to analyse a CLAHRC from macro (policy context), meso (organisational activities and interactions) and micro (individual behaviour) perspectives182

• Currie et al. ’s201 application of neo-institutional theory to explore the interplay between knowledge brokering and professional power dynamics in CLAHRCs (‘Professional hierarchy means that some change agents or knowledge brokers are accorded greater legitimacy than others’) and question the level of policy commitment to the CLAHRC model

• Rycroft-Malone et al. ’s102,166 adaptation of realist evaluation to look at context–mechanism–outcome configurations in a CLAHRC.

Critics have questioned if, and to what extent, mode 2 knowledge production (in which multiple stakeholders negotiate a research agenda that may not meet scientists’ standards of rigour) counts as science at all. 202 These reservations might be partially assuaged by Van de Ven and Johnson’s paper on engaged scholarship,203 in which they apply the concept of mode 2 to organisational academics working alongside managers. Using the philosophical lens of pragmatism, they argue that mode 2 is essentially a dialectical process of bringing competing perspectives (academic and practical) to bear on a problem – a task they term arbitrage and others have called ‘bricolage’. 204 They explain:

By exploiting multiple perspectives, the robust features of reality become salient and can be distinguished from those features that are merely a function of one particular view or model.

p. 810203

Such an approach is invariably conflict ridden; the key to its success is encouraging task-oriented conflict (which is creative and productive) while managing the potentially destructive influence of interpersonal conflict.

Discussion

In summary, the landscape of applied health research in the UK is changing. A great deal of research remains unambiguously university led, university based and funded (in either ‘responsive’ or ‘commissioned’ mode) from research councils, NIHR and the Department of Health (e.g. via the HTA programme). However, such mode 1 research (in which impact can often be effectively evaluated through logic models) increasingly coexists with mode 2 programmes of activity that are jointly led and governed by multiple (university and non-university) stakeholders, organised in a more neutral transaction space and characterised by organic, non-linear dynamics and focus on the ‘production’ rather than ‘translation’ of knowledge.

These findings are discussed further in Chapter 8.

Introduction

The economic impacts from medical research form a subset of many of the logic models presented in Chapter 3. A section of the literature has addressed the specific issues relating to undertaking exercises to determine economic impacts or the returns on investment from medical research and development spending. Previous reviews of the literature, which form a starting point for this review, have highlighted work that has been done to advance the field. 2,205 The methods used to assess these impacts or returns on investment are born from the economic evaluation literature, and the difference in approaches lies largely in the scope of the cost and benefits assessed, and the valuation methods for seemingly non-monetary components of the impact. As stated in Chapter 2, the purpose of this review was, using Buxton et al. 28 as a starting point, to identify studies since 2004 that have used any methods to attempt to value (in monetary terms) the benefits (health and cost savings) of a body of health research and link that with an investment in the body of research. Articles were included only if they contained a component that attempted to value the impact of research and development investment on population health.

The article in the Bulletin of the World Health Organization by Buxton et al. 28 attempted to learn from previous studies that had estimated ‘monetary values for the societal benefits obtained from health research, especially those studies that have attempted to link (and value) benefits to a specific society from a specified (and costed) body of research’.

The authors characterised the identified methods into four categories:

1. valuing direct cost savings to the health-care system

2. valuing benefits to the economy from a healthy workforce

3. valuing benefits to the economy from commercial development

4. measuring the intrinsic value to society from health gain.

Studies were identified that had considered the benefit of medical research and development as direct cost savings to the health-care system, brought about by a reduced number of people requiring treatment or reductions in per patient treatment costs. This approach had been predominant in estimating the benefits of vaccination research, which had the potential to eradicate subsequent disease and associated treatment costs. 205–208 Cost savings could be included as part of cost–benefit analysis, but these studies did not always link this to an investment period or country-specific research.

One of the earliest studies to attempt to calculate a rate of return from medical research was conducted by Mushkin and Landefeld. 209 A human capital approach (equating the value of life to market values, i.e. wages) was used to value gains from US biomedical research, characterised by a healthier workforce. The limitations of such an approach were acknowledged by the authors and others28,209,210 and tend to overstate benefits when lost labour can be replaced, while understating benefits for those sections of the population not of working age.

Buxton et al. 28 drew largely on a review conducted by Salter and Martin,211 which explored the commercial economic benefits from basic research. Salter and Martin noted progress made by Mansfield212,213 that estimated a worldwide social rate of return (benefits accrued to the whole of society, as opposed to one firm or funders of one project) of 28% for research undertaken 1975–78. Studies have also demonstrated the economic benefits of medical research through industrial applications to other industries. 214

An emerging field highlighted by a number of studies in the Buxton et al. 28 review had measured the intrinsic value of health gains brought about by research and development. A US initiative of the Mary Woodard Lasker Charitable Trust, Funding First,215 produced a series of papers that formed a subsequent book. 216 An informal approach used willingness-to-pay methods to value the increased longevity of life experienced by the US population, attributing a fraction of these gains to medical research. The results suggested ‘exceptional returns’ of nearly 20 times the investment in US medical research. This type of analysis was performed in a more systematic fashion in an Australian study, taking a similar ‘top-down’ approach to valuing health gains, to produce an estimate of the annual rate of return to investment in research and development. 217 They estimated a favourable benefit-to-cost ratio of 2.40 (i.e. AUS$1 invested creates an additional AUS$1.40 benefit); however, this work has been subsequently criticised because the time for investment in medical research to produce health gains was not considered. 218

Buxton et al. 28 noted that there is significant scope for these methods to be extended and refined to allow more robust estimation of the economic benefits from medical research. In particular, a widely acknowledged central challenge that must be addressed in this kind of analysis relates to the attribution problem; the relationship between investment in research and health outcomes. 15,22,28,115 This manifests itself as several related issues regarding the contribution of health research in improving health outcomes and what would have happened without research, that is the unobservable counterfactual. Assumptions must be made regarding the share of health gains attributable to health research, and given there is an international pool of health research, the contribution of any particular country to particular health gains. Finally, assumptions must be made regarding the temporal relationship between a period of investment and a period of health gains. Different approaches face somewhat different problems in dealing with attribution, but methods have continued to be developed to address these issues.

Review findings

The search of databases produced 413 articles, which were initially screened by a reviewer by title (Figure 8). After initial screening and deduplication, 102 articles were screened by abstract. Seventeen articles were reviewed in full, with five included. 22,26,75,218,219 Two of these articles were included in the main literature review. 26,75 One additional report that was not picked up by the supplementary search was included from the main literature review. 25 One additional article and one report known to the authors was also added. 24,220 One in-press article that the authors kindly gave us access to was also included. 27 In total, the review produced nine articles/reports. The studies and methods of assessing return on investment, that included a component that attached a monetary value to health gains, are summarised in Table 16 (see Appendix 6).

FIGURE 8.

Flow diagram of included studies.

The nine studies identified in the review can be split into two categories in terms of how health gains are measured: those taking a top-down approach and those taking a bottom-up approach. There are several other important issues that must be addressed in quantifying the returns; this simple taxonomy allows us to explore the different methodologies. Figure 9 depicts the basic methodologies.

FIGURE 9.

Approaches to identifying health gains from research. R&D, research and development.

Studies have been compared on a number of key facets of the analysis and assumptions that have to be made regarding measuring and valuing net health gains and how to attribute a proportion of health gains to a body of research as follows: How were health gains measured? How were health gains valued? Were health gains ex post or ex ante? Were the costs of delivery accounted for? Was the lag between investment and health gain considered? How was the attribution problem addressed?

Discussion

There have been contributions to the literature that estimate the impacts of health research using methods to attach a monetary value to health gains. Approaches have attempted to estimate the resultant health gains from investment in bodies of research, and, in doing so, must deal with several problems relating to attributing health gains to particular investments. Techniques that attempt to deal with the problems of attribution have been established. However, authors have acknowledged a simplification of the relationship that is required and the reliance on a logic model view of research impacts. Some of these contributions also consider non-health sector benefits falling on the wider economy, although the scope of the benefits considered often differs, as does the valuation.

Only a few studies specifically considered programmes of health research. 25,27,75 Guthrie et al. 27 estimated the gains of the NIHR HTA programme, but made cautious conclusions on the returns based on hypothetical uptake of a subset of HTA-funded research. 27 Clearly there is scope for these types of methods to be applied to estimate returns from programmes such as NIHR HTA, but several additional considerations need to be taken into account. Conversely, there are advantages to having a well-defined unit of analysis.

It would appear that assessing monetised impact at a programme level is conducive to the bottom-up approach, when the set of interventions is well defined and the task of identifying those that are ‘important’ could be avoided. To an extent, data feasibility issues that limit the bottom-up approach should be mitigated by programmes such as NIHR HTA in that most of its research includes cost–utility estimates. However, issues of scale are present if the number of studies undertaken by a programme is large, such as in the NIHR HTA programme. This might be mitigated to an extent by the need to consider only those trials that showed a significant effect, but this makes a bold assumption about the nature of evidence being used in clinical practice.

When attempting to measure health gains from a programme of research, the attribution problem manifests itself as an added layer of uncertainty regarding the proportion of total health gains that should be attributed to the specific programme. Using a top-down approach, Deloitte Access Economics25 dealt with this by using the percentage of total citations in clinical sciences that were studies funded by the programme as a proxy. This additional attribution problem is not circumvented by the bottom-up approach and a consideration must be still be made. The view taken in Johnston et al. 75 and Guthrie et al. ,27 that National Institute of Neurological Disorders and Stroke’s trials and HTA studies are definitive in terms of changes in uptake, is insufficient, especially in developed countries with multiple funding streams and complex and evolved research ecosystems. The use of the weights attached to particular RCTs in meta-analyses could provide a more systematic way of considering the relative impact of different clinical research supported by multiple funders. Regarding the proportion of health gains that should be attributed to world research and development, this could be used as an intervention-specific replacement for guideline analysis or used as an adjunct. Although time lags must be included, it is not clear how best to estimate these.

An additional problem for programmes that fund only clinical research is dealing with the role of basic research in health gains. This is unclear and constitutes a major potential limitation in these methods. No study looking at a programme has yet encompassed these kinds of considerations into the approach dealing with attribution of health gains to a programme, using either top-down or bottom-up methods.

Although it would clearly be possible to estimate the returns on investment from the NIHR HTA programme, significant challenges remain.

Introduction

This chapter considers how the impact of a randomised trial might be measured in terms of its contribution to a subsequent systematic review. After an introduction, the results of the literature search are presented. The results of a search of The Cochrane Library for systematic reviews that include randomised trials funded by the HTA programme are then presented and the implications of these findings are discussed.

The HTA programme aims to fund research on technologies that matter to the NHS. In doing so, it funds mainly systematic reviews and randomised trials. Given the widely used hierarchy of evidence, which privileges randomised trials, most systematic reviews are composed of only randomised trials. Meta-analysis provides a synthesised estimate of the effectiveness of the technology. Results from systematic reviews are seen as more robust than those from particular trials and are the preferred basis of decisions by NICE and similar bodies. The two relate in that new trials are rarely funded without a prior systematic review about what current evidence exists. Many of these identify weaknesses in the body of research and recommend new research, usually randomised trials.

Systematic reviews can also provide a way of retrospectively assessing the impact that such trials made to the evidence on the clinical effectiveness and cost-effectiveness of the technology involved. Systematic reviews and their linked meta-analyses provide quantitative measures of the quality and impact of individual trials. Trials are assessed for quality to be included. The included trials indicate not only how many have reported but also the contribution (weight) of each. The resulting synthesised effect size may be statistically significant even though none of the contributing trials was, or, despite some trials showing statistically significant differences, the synthesised effect may not be statistically significant.

Although the HTA programme uses systematic reviews to identify topics for randomised trials, it does not review the impact that those trials have on any subsequent systematic reviews. To some extent it relies on this happening automatically, as the programme partly funds the Cochrane Collaboration, which has pioneered impartial, thorough systematic reviews that are in the public domain. However, the HTA programme keeps no record on the extent to which relevant systematic reviews include trials it has funded (whether new or updates). Updating systematic reviews varies widely, reflecting developments in science. Some HTA journal reports include updated systematic reviews by the triallists and others take a long time to be updated. Some updates adopt slightly different, narrower foci. No guarantee exists that a trial funded by the programme in response to an earlier systematic review will be included in that review’s update, if and when it is updated.

The HTA programme’s early adoption of the process whereby systematic reviews preceded new trials implied a cycle in which new RCTs would contribute to the updated meta-analysis so that uncertainty was reduced if not eliminated. Although methods of measuring such progress were not available when the programme started, this chapter explores if that has remained the case.

Many of the early projects commissioned by HTA were systematic reviews, all recommending further research (usually RCTs), some of which the programme went on to commission and fund. Of 109 projects that included a randomised trial to March 2011, 62 (57%) were recorded as quoting a prior systematic review in its monograph report. 228 Most [109/125 (87%)] of the RCTs funded by the HTA programme were superiority, not equivalence (non-inferiority), trials. Superiority trials also made up the bulk of systematic reviews.

Owing to the time lag between identification of topic and publication in HTA Journal, almost all trials published to 2011 had been commissioned pre-2006 when researcher-initiated bids began to be funded. Therefore, almost all were on topics identified and prioritised by the programme. For RCTs that are ‘commissioned’ (i.e. defined) by HTA, the research question, the existing evidence and the need for a trial are specified.

To explore the scope for measuring the impact of randomised trials through their contribution to systematic reviews, one needs to know what the literature offers in terms of methods and the extent to which systematic reviews that include trials funded by the HTA programme exist.

Literature searches

The methods used in the literature search were, as outlined in Chapter 2, supplemented by the team’s knowledge. The search of The Cochrane Library of Systematic Reviews was based on the relevant technology and disease. Reviews that included both the technology and disease were searched to locate randomised trials funded by the HTA programme, using the lead author as the link. For each review including a trial funded by the programme, the following items were extracted: technologies compared, whether or not the difference was statistically significant and the weight attached to the trial.

Results

The results of the literature search were divided into descriptive studies, those that explored the use of systematic reviews in designing trials, those regarding VOI [and its York/Patient-Centered Outcomes Research Institute (PCORI) variant] and those assessing systematic reviews in relation to their use in clinical guidelines.

Discussion

The chapter has shown that the HTA programme contributes only one side of the loop linking systematic reviews and trials. Although it usually carries out a systematic review before funding a trial, it does not follow up with any analysis of the contribution that trial makes to any subsequent systematic review.

The York/PCORI approach used in 2015 by the HTA programme for deciding when to fund trials is important and deserves to be evaluated. However, without subsequent analysis of the trials funded, this will not help with assessing the contribution any trial funded makes to the body of evidence.

The GRADE tool offers a way to assess how complete the evidence in systematic reviews is for the purpose of drawing up clinical guidelines. This may offer a way for the HTA programme to assess the extent to which a particular trial has contributed and if another is required.

The finding that only around one-quarter of HTA-funded trials are included in a subsequent systematic review shows that a contribution by a trial to systematic reviews cannot be assumed. Furthermore, the fact that around half of the Cochrane systematic reviews to which a trial contributes do not demonstrate superiority prompts questions as to how to assess the value of these.

Introduction

This chapter was prompted by the suggestion that the impact of some RCTs funded by the HTA programme were mainly concerned with stopping new technologies. 30 The two cases cited were water softeners for eczema and larvae for wound healing. 31,32 Both were novel technologies, funded by the HTA programme. Both were negative in the sense that neither demonstrated that the technology added value to current treatments. Subsequently, neither technology appears to have been used much by the NHS, although no formal evaluation has been carried out.

To discredit or stop the diffusion of a technology, a randomised trial must be definitive; being one of the first and biggest in the field can do this. The water softener trial was both the first and biggest, whereas the larvae healing trial was the biggest but not the first.

This chapter explores the related literature and briefly reviews the HTA portfolio of randomised trials in terms of how many were ‘first in class’. In brief, it finds little of relevance in the literature on trials stopping technologies. It found that, of the 121 eligible trials published up to 2011, 30 could be considered first in class. A further 10 projects had three or fewer prior randomised trials (see Table 10 for these references). 228

Literature searches

Two searches were carried out, the terms of which are provided in Chapter 2. The first focused on terms such as ‘randomised controlled trials’ and ‘disinvestment/rationing’ and the second widened the terms to include The Lancet article,30 which included the water softening and larvae cases. Between them they identified 78 articles, of which 38 were deemed relevant. Of these:

• 16 were descriptive or discursive83,301–315

• six were on programme budgeting and marginal analysis316–321

• six were surveys of attitudes to disinvestment302,322–326

• five related to NICE327–331

• one analysed a RCT in terms of discontinuation of a particular drug. 332

Four systematic reviews were identified, three of which were concerned with disinvestment. Leggatt et al. 333 found that eight countries had some evidence of such work but no clear findings. Polisena et al. ’s334 review of case studies of disinvestment found 14 cases, with programme budgeting marginal analysis used in six. Neilsen et al. 335 reviewed HTA in European countries, indicating the priority many attached to disinvestment. The final systematic review, by Watt et al. ,336 reviewed the use of safety and effectiveness in a single service, assisted reproduction.

Points arising from the descriptive papers included: the asymmetry between existing and new services in terms of assessment and scope for disinvestment,318 and the importance of safety concerns in decisions to stop use of particular technologies. 336 Stepped-wedge trials were advocated as a means of evaluating new technologies before/during implementation. 304

Overall, with the exception of the advocacy of the stepped-wedge design, the literature review offered little on the scope for using trials to prevent the diffusion of new technologies. The literature was focused on disinvestment and was concerned mainly with the removal of older, less effective technologies.

First in class

A term usually referring to the first pharmaceutical in a class to be authorised for marketing, the term acquires a gradation of meanings applied to HTA programme trials. The interventions are often complex packages, parts of which differ from trial to trial. Furthermore, such interventions can often be provided in different settings (hospital or community) or by different people (doctor or nurse). All those randomised trials with a reasonable claim to be first in their class, based on the background provided in each HTA journal, were included.

Forty-four out of a total of 121 trials involving a RCT that did not have a preceding systematic review were identified from the metadata database228 and examined in terms of preceding evidence. Twenty-two could be considered ‘first in class’. Systematic reviews that had a weight of 100% were added to these. As discussed in Chapter 6, eight fell into this category. Therefore, 30 out of 121 (around one-quarter) eligible trials could be considered first in class.

Few of these were novel technologies such as water softeners or larvae. The nearest to a new technology was a befriending service for people with dementia. 337 Many interventions were new only in the sense of being the variants of existing procedures. In surgery, these included different methods for performing tonsillectomy290 and for reconstructive surgery for limb ischaemia. 338 Some were the first in a particular setting, mainly in general practice, while others were new diagnostic tests,339 referrals for pain and for lumbar spine radiography,340,341 and screening for atrial fibrillation. 258

A further 10 had three or fewer preceding randomised trials. These were usually small trials, most of which were deemed of poor quality.

Discussion

The role of randomised trials in stopping the diffusion of particular technologies has received almost no attention in the literature. To have scope to prevent diffusion, any such trial needs to be definitive: this implies being either the first or the largest trial. Given the paucity of literature, a preliminary search was carried out to establish how many HTA trials could be considered first in class. This search showed that around 30 trials funded by the HTA programme could be titled ‘first in class’. However, many of these involved small variations on existing treatments. Very few were studies of completely new types of technology. The programme may wish to consider identifying trials that might be considered first in class and on the extent to which they can be considered definitive.

Updated systematic review

This updated systematic review identified and described 161 publications on research impact (see Appendix 3, Table 14). Most of these were published since the previous original review that included studies published up to 2005. 2 The earlier review2 described 46 publications from the period from 1990 to 2005 and hence the number of studies has grown, as has the range of approaches used (please refer to the original Hanney et al. 2 report for a full list of these references).

Continuities exist between this and the previous review. 2 Some older frameworks continued to be used, but with variations, particularly the Payback Framework and the monetary valuation of benefits. Additional frameworks have been developed and applied, often drawing on and combining earlier approaches. Major work building on previous frameworks was conducted by the CAHS,7 summarised in Frank and Nason. 115 This was further developed and built into the performance measurement system by the Alberta health research system. 85

The scale of application varied enormously, with some frameworks using small-scale approaches designed for use by individual researchers, such as the RIF developed by Kuruvilla et al. 97 At the other end of the spectrum national schemes continue in the Netherlands,79 and have developed with the UK’s REF. 106 This reflects the growing interest by research funders from national governments through to small funding organisations. For example, the Addenbrooke’s Charitable Trust commissioned a study to assess the impact of its fellowship scheme, which invested less than £1M. 67

Much of the literature comes from the English-speaking nations of Australia, Canada, the UK and the USA, perhaps partly reflecting the search restriction to English on pragmatic grounds. However, we also include ongoing work conducted in many other countries, including the Netherlands and Austria, and published in English. We also identified assessments of the impact of health research in international development programmes, including programmes of research training.

Some core methods continue to be widely used: interviews, surveys, documentary analysis and case studies. Case studies are of interest for several reasons. They have confirmed responses to surveys suggesting that researchers do not generally exaggerate the impact they report in surveys. 2,51,52,62,95 Case studies have enabled the capture of the wider impacts from research training programmes, such as that funded by the National Institutes of Health Fogarty International Centre. 57 Case studies have also incorporated qualitative findings in mixed-method approaches such as that adopted by the Wellcome Trust. 93 Finally, although the 2014 REF and the EIA exercises were based on the outputs of groups of researchers rather than funded programmes, they too demonstrated the advantages of using case studies. Those conducting the REF warned of the danger of excluding case studies in favour of metrics in future such exercises. 33

We also noted attempts to develop the methods used, owing, in part, to the recognition of limitations with the existing methods. One limitation frequently noted is that most of the studies have been funded by the same organisations that funded the original research. Potential for conflicts of interest can also arise owing to the involvement in impact assessments of members of the research organisation whose research is being assessed.

The impact assessment studies reported in our review (see Table 14, Appendix 3) generally start with specific pieces of research and work forwards to identify impacts. Many of them suggest that research has made a larger impact on policy than is often reported in reviews of studies that start with policy-making and work backwards to consider how far research might have impacted on those policies. 13 This might reinforce the suggestion that the best way to identify research impacts is to start from a body of research and work forwards to trace any impact that it might have had. 2 Tracing forwards provides a different focus from studies that start from impacts, such as on policy, and work backwards. For example, a researcher interested in their research being used might have better recall about how, and to whom, the findings were disseminated than a busy policy-maker would have of which research influenced him or her. The growing portfolio of case studies supports the claims that a range of impacts can be identified as coming from research. Sensitivity analyses have generally indicated the robustness of these estimates. On balance, differences in method rather than conflicts of interest seem to account for the high impact reported in some studies included in our review.

Not all types of programmes have the same potential for delivering impact. There is considerable evidence that programmes based on a prior assessment of needs by funders, or which have been commissioned by an authoritative ‘receptor body’, are likely to have more impact than programmes driven by researchers’ interests. This was, of course, the rationale for establishing the HTA programme in which the commissioning of research was linked to the needs of the NHS. 2,49 It is also seen internationally. This suggests that the frameworks to assess the impacts from these programmes should take account of the context within which the research occurs.

Many studies indicate that the impacts of research are often slow and incremental, and that many factors impinge on potential users. 137 Some highlight the importance of personal interactions between actors to achieve impact. We have shown that the programmes with the strongest interactions are among those reporting highest levels of impact. Does this suggest that a greater emphasis on interactions might be having some successes? Some of the studies described might indicate this is the case. 66,89,132,136–138,141 More detailed assessments might play a useful role in understanding the processes involved. However, initiatives informed by concepts such as linkage and exchange342,343 do not necessarily make an impact if, for example, there is a high turnover among the government officials. 344 Iteration at an organisational level may be required for interactions to be successful. Impact assessment requires attention to scale and context. The validity and reliability of different data sources also needs to be assured.

Taxonomy of approaches

The extended review of the social sciences literature revealed a wide variety of approaches which reflected a range of philosophical positions: positivist (e.g. the unenhanced logic model), constructionist (interpretative and interactionist), realist (context–mechanism–outcome–impact models), critical (participatory models) and performative (or Co-production models). However, these ‘ideal type’ positions are often used eclectically in combination. For example, the Payback approach is a hybrid, including a central logic model but enhanced by a focus on case study methods, reflecting the importance of an interpretative narrative and paying attention to context.

Approaches that emphasise ‘co-production’ of research findings are becoming more popular, particularly in Europe. These approaches may be less appropriate for the HTA programme than for some types of NIHR programmes such as CLAHRCs and Academic Health Science Networks.

Monetary value on the impact of health research

The literature on the economic returns from medical research has also expanded since 2005. Health gains have been monetised by putting a value on each QALY gained, net of the cost of delivery to produce estimates of net monetary benefit and linked to specific investments in research. Some of these analyses have also considered wider non-health benefits from research investments. These analyses have been performed at various levels from programmes to specific diseases. All these analyses required major assumptions relating to attribution, first of the contribution of research to health over specified time periods and then attribution by country. Each analysis has provided its own challenges in finding estimates to meet these assumptions.

A new method for identifying health gains has used a ‘bottom-up’ as opposed to a ‘top-down’ approach, building the totality of health gain based on individual research-led interventions. 22,26,27,75,219 Most of these assessed health gains ex post. The ‘top-down’ approach that formed much of the earlier work in the field has also continued to develop. 24,25,218,220

The bottom-up approach has produced more conservative returns and helps deal with some of the research/health attribution problems specific to the top-down approach, in that only research-driven interventions need be included in the analysis. In the latter, it is clear that estimates rely heavily on the fraction of health gains that are attributed to research and development, for which robust estimates do not exist. Although the bottom-up approach may be better able to address this attribution problem, it struggles with the issue of scale. This has often narrowed its scope because of the richness and level of data required.

Most of the identified studies acknowledge the time lag between the investment and health gains but based on different assumptions. The most robust estimates have been informed by guideline analysis,22,26,219 but even these share uncertainties about time lags and implementation.

Assessing the impact of trials on systematic reviews

The HTA programme contributes one side of the loop, from a systematic review to the need for new trials but not vice versa. The impact of a new trial on the subsequent systematic review is not routinely examined. Only by completing this side of the loop can it be established if further research is required. Clinical guidelines are the main ‘customer’ for systematic reviews. Both the York/PCORI (which is being used by the programme) and the GRADE toolkit consider systematic reviews and guidelines to assess if further research is required.

However, it remains the case that many trials funded by the HTA programme do not show statistically significant differences in their primary outcomes. When included in systematic reviews, these often also fail to show superiority of the intervention in question. Non-superiority, however, cannot be interpreted statistically as non-inferiority. A preliminary search showed that around only one-quarter of trials funded by the HTA programme were included in subsequent systematic reviewed by Cochrane; of these reviews, around half showed superiority. For the rest, the 95% confidence interval included a measure of no difference.

Although each trial has value in contributing to a meta-analysis, the policy implications are unclear when the meta-analysis fails to indicate superiority of any compared intervention. Work is required to assess what further research might indicate if non-superiority can reasonably be taken as indicating non-inferiority. The implications can be considerable. For instance, if two interventions had no difference in efficacy, a choice might be offered if the cost was similar. If the costs are materially different, the less costly might be preferred. More generally, funding trials and systematic reviews on the basis of hypothesised differences but being unwilling (or unable) to draw conclusions when such differences are not proven amounts to waste.

Cost-effectiveness modelling, such as that funded by the HTA programme, to support NICE’s technology appraisal programme, goes beyond systematic reviews to synthesise data on costs and effectiveness often projected to cover patients’ lifetimes. Such models can have a major impact on decisions made by NICE. Although we have not considered these models, we believe that assessment of their impact should be included in future.

Impact of randomised trials on stopping the use of particular health technologies

Although disinvestment has attracted considerable interest in the literature, it has been largely concerned with existing rather than new technologies. The literature provides no instances of a randomised trial playing a major role in stopping the diffusion of a new technology. A few trials funded by the HTA programme related to water softeners and larval healing may have played such a role. 31,32 One was first in class and both were the biggest trials of each topic. Around one-quarter of trials funded by the HTA programme can be considered first in class, but many of these were concerned with small changes in existing treatments. The term ‘first in class’ is usually used in discussing pharmaceuticals where ‘class’ may be defined more rigorously than for other interventions.

Limitations of the Health Technology Assessment review

In conducting this review we faced various challenges, and, although we attempted to address them, some limitations remain. The precise inclusion/exclusion criteria were difficult to define and apply, especially around issues such as what is meant by a ‘programme’ of research. For many programmes from specific funders they were, of course, clear-cut. However, the boundaries were less clear in relation to the collective work supported by multiple funders and conducted by a group of individual researchers at one end of the spectrum, and a large part of the research system at the other.

The nature of the review was also complex because we wished to not only update the earlier 2007 review,2 but also extend the focus of the review to include a wider range of analysis and address issues of particular (policy) relevance to the HTA programme. Therefore, inevitably, there were some issues in which the boundaries became fuzzy. At the same time, we wanted to maintain a degree of coherence by presenting just one main list of included studies before delving into the more specialist areas covered by Chapters 5–7. In addition, to ensure as wide a range of papers as possible was included in the main list, we drew on the knowledge of the team and snowballing to a considerable extent. These methods provided over one-quarter of the papers (43/161 included studies), a higher proportion than for many reviews. Although this might create some difficulties for any replication of the review, and the diversity of papers did present challenges for the analysis, the importance of these limitations was reduced because there was no possibility that a meta-analysis could have been meaningfully conducted in a field such as this. Nevertheless, the scope of the review remained challenging, especially in view of the resource constraints within which we operated.

Finally, as discussed below (see Options for the National Institute for Health Research/Health Technology Assessment to take research impact assessment forward), we were aware that the whole field has become increasingly controversial in recent years. Furthermore, there have been major developments at the system-wide level that, while they have not been fully evaluated, are having an increasing impact on what assessment might be undertaken at the level of specific funded programmes.

Options for the National Institute for Health Research/Health Technology Assessment to take research impact assessment forward

Options for the National Institute for Health Research/Health Technology Assessment for health research impact and research recommendations

One option for the HTA programme is to plan how best to meet the data requirements of future impact assessments, both by the programme but also in external assessments such as any future REF exercises. The data on the management of the HTA programme, held by NIHR Evaluation, Trials and Studies Coordinating Centre, focus on administration and management of the programme. Development of its electronic management information system has been slow and uneven. The programme has invested in projects that quality assures the data, and enables comparisons and trends to be analysed,228 but these have applied only to the randomised trials. The likely data requirements of future assessments of impact and of the REF need to be planned for and included either in management information systems or in special projects.

Contributions of authors

James Raftery (Professor of HTA) was the chief investigator, originated the proposal, was project lead on Chapters 6 and 7, assisted with data extraction for Chapter 3 and prepared the report for publication.

Steve Hanney (Professorial Research Fellow) was a member of the research team, contributed to the development of the proposal, was project lead on Chapter 3, including literature searching, data extraction and write-up, and contributed and reviewed drafts of the report.

Trish Greenhalgh (Professor of Primary Care Health Sciences) was a member of the research team, contributed to the development of the proposal, was project lead on Chapter 4, including literature searching and write-up, assisted with data extraction for Chapter 3 and contributed and reviewed drafts of the report.

Matthew Glover (Research Fellow) was a member of the research team, was project lead on Chapter 5, assisted with data extraction for Chapter 3 and contributed and reviewed drafts of the report.

Amanda Blatch-Jones (Senior Research Fellow) was a member of the research team, contributed to the development of the proposal, collaborated on Chapter 3 with Steve Hanney including literature searching and data extraction, contributed and reviewed drafts of the report, worked closely with James Raftery to keep the project running and prepared the report for publication.

Data sharing statement

All available data, in addition to what is provided in the appendices, can be obtained from the corresponding author.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health.

Search strategy for the update to the 2007 systematic review (impact of Health Technology Assessment research)

Search date: August 2014.

Search strategy for monetary value on the impact of research

Search date: October 2014.

Search strategy for randomised trials impact on systematic reviews

Search date: October 2014.

Search strategy for impact of randomised trials on stopping health technologies