The impact of Cochrane Reviews: a mixed method evaluation of outputs from Cochrane review groups supported by the NIHR

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 12/203/03. The contractual start date was in April 2013. The draft report began editorial review in September 2013 and was accepted for publication in November 2014. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Frances Bunn and Daksha Trivedi are editors with the Cochrane Injuries Group, Phil Alderson is an employee of the National Institute for Health and Care Excellence (NICE) and was employed by the UK Cochrane Centre (UKCC) from 1998 to 2004 and was seconded part time to the UKCC from May 2013 to March 2014, Frances Bunn, Daksha Trivedi, Phil Alderson and Steve Iliffe are all authors on Cochrane Reviews and Phil Alderson has also published papers about Cochrane Reviews.

Copyright statement

© Queen’s Printer and Controller of HMSO 2015. This work was produced by Bunn et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

The role of systematic reviews in evidence-informed decision-making

The development of methods for the synthesis of research has been a key driver in the move towards evidence-informed policy and practice. Although a number of terms have been used for such syntheses, the most widely used and understood is systematic review. Systematic reviews have several advantages over other types of research that have led to them being regarded as particularly important tools for decision-makers. Systematic reviews take precedence over other types of research in many hierarchies of evidence, as it inherently makes sense for decisions to be based on the totality of evidence rather than a single study. 6,7 Moreover, they can generally be conducted more quickly than new primary research and, as a result, may be attractive to policy-makers required to make a rapid response to a new policy issue. 8 Despite such arguments in favour of using systematic reviews to inform decision-making, it has been suggested that they have not had an impact on policy and practice in the way one might expect. Indeed, there is evidence to suggest that the diffusion and use of evidence across the NHS is generally poor. 4

Cochrane

One organisation involved in producing systematic reviews is Cochrane (www.cochrane.org/about-us). Established in the early 1990s, Cochrane is a global independent organisation that has the aim of promoting evidence-informed decision-making through the production of systematic reviews. Reviews are produced by Cochrane Review Groups (CRGs), which are made up of people who prepare, maintain and update the Cochrane Reviews and people who support them in this process. Each group has an editorial base, a small team of people that supports the production of Cochrane Reviews. Groups focus on a particular area of health, and review topics are identified by review authors or through prioritisation processes at editorial bases.

Cochrane is a not-for-profit organisation funded by a variety of sources including governments, universities, hospital trusts and charities. Cochrane systematic reviews should be well placed to influence policy-makers, practitioners and researchers, as they are generally acknowledged to be comprehensive and rigorous summaries of the best available evidence on a given topic. Moreover, they are periodically updated in the light of new evidence and there is increasing interest in the dissemination and impact of Cochrane Review findings. 9

Evidence-informed decision-making

It has long been recognised that the relationship between research and policy or practice is a complex one10 and that research may not always have the impact that researchers desire. 11 One reason for this is that research evidence is only one factor in shaping policy and practice. Decision-makers are subject to many different influences including political imperatives, the media, non-research evidence and powerful lobbying groups such as industry. 7,12 In addition, the usefulness of systematic reviews for aiding policy-makers in the decision-making process has come into question, with commentators suggesting a number of factors that might reduce their utility. These include a lack of good-quality primary research for synthesis, a tendency for reviewers to focus on randomised controlled trials (RCTs) and controlled evaluations at the expense of other types of research, and inadequate evaluation of complex interventions with little recognition of the importance of contextual factors. 8,13,14 Moreover, there are significant challenges associated with conceptualising impact and identifying the extent to which systematic reviews are used to inform decision-making. 15

Defining research impact

A variety of terms have been used to describe the impact of research on policy and practice. These include research impact, influence, outcomes, benefit, payback, translation, transfer, uptake and utilisation. 16,17 Research can be used either directly in decision-making related to policy and practice or indirectly by contributing to the formulation of values, knowledge and debate. Commentators have pointed out that there is a key distinction to be made between conceptual use, which brings about changes in levels of understanding, knowledge and attitude, symbolic use which can lead to the mobilisation of support, and instrumental, or direct use, which results in changes in practice and policy making. 10,18–20 Indeed, ‘research impact forms a continuum, from raising awareness of findings, through knowledge and understanding of their implications, to changes in behaviour’. 21

Reason for conducting this study

The National Institute for Health Research (NIHR) systematic review programme currently supports 20 CRGs based in academic or health institutions in the UK. These groups cover a broad range of health-care areas and produce almost half of all Cochrane Reviews, publishing around 200 new reviews each year, as well as bringing a similar number of existing reviews up to date. 22 It is important that this funding represents value for money and that the reviews produced by these groups be useful for practitioners, policy-makers, service users and members of the public. One way in which their value might be judged is by the impact that the reviews produced by NIHR-funded CRGs have, or potentially have, on policy and practice, and on future research. However, although it is acknowledged that CRGs produce high-quality systematic reviews,23–25 to date there is a lack of information about the impacts of Cochrane Reviews. Moreover, it is important to understand how reviews are currently used in order to develop appropriate strategies for knowledge transfer and exchange. This study aimed to enhance our understanding of how Cochrane Reviews impact on policy and practice and to inform the development of methods for evaluating the impact of systematic reviews and future strategies for dissemination and knowledge transfer.

Aims and objectives

The overall aim was to identify the impacts and likely impacts on health care, patient outcomes and value for money of Cochrane Reviews published by NIHR-funded CRGs between the years 2007 and 2011 (time period set by funders). The research objectives were to identify and describe the impacts of Cochrane Reviews in terms of evidence of direct effect on clinical practice; their inclusion in, or use for, the preparation of national or international clinical guidance, such as guidance published by the National Institute for Health and Care Excellence (NICE); their likely influence on clinical practice directly (i.e. without or before incorporation into national clinical guidance); and their identification of important gaps in knowledge and possible influence on the conduct of new primary research studies. The research questions are:

1. Have systematic reviews produced by NIHR-funded CRGs during 2007–11 had a direct effect on clinical practice?

2. Have systematic reviews produced by NIHR-funded CRGs during 2007–11 had a direct effect on NHS organisation and delivery?

3. To what extent have reviews produced by NIHR-funded CRGs during 2007–11 been included in clinical guidance, such as that produced by the NICE?

4. To what extent are reviews produced by NIHR-funded CRGs used in the preparation of NICE guidance?

5. What evidence is there that systematic reviews produced by NIHR-funded CRGs during 2007–11 are likely to change future clinical practice?

6. What influence have systematic reviews produced by NIHR-funded CRGs during 2007–11 had on the conduct of new primary research studies?

7. What are the barriers and facilitators to Cochrane systematic reviews impacting on policy, practice and future primary research?

Structure of the report

Chapter 1 describes the background and rationale for the study, Chapter 2 describes the conceptual approach adopted and the framework used to structure the evaluation and Chapter 3 details the methods used for the questionnaire surveys, bibliometrics and documentary review. The findings of the questionnaire surveys, bibliometrics and documentary review are presented in Chapters 4 and 5, with Chapter 4 including details of the findings relating to the overall impact of NIHR-funded CRGs and Chapter 5 focusing on the findings relating to the impacts of a representative sample of Cochrane Reviews produced by the NIHR-funded groups. Chapter 6 reports the methods and findings of the qualitative interviews. Chapter 7 summarises the study findings and looks at their implications. This includes a summary of the results of the impact evaluation, an analysis of how the results contribute to our knowledge of barriers and facilitators, a discussion of the strengths and limitations of the study and a consideration of the implications of the findings.

Framework

The use of a framework for structuring assessments of impact has been recommended, as it can help organise inquiry30 and allow for easier comparison across reviews. 31 We structured our data collection and analysis using a framework that combined elements from two existing frameworks, the Health Economics Research Group (HERG) framework for assessing health research payback26,32,33 and The Research Impact Framework developed by Kuruvilla et al. 27 The HERG framework consists of a multidimensional categorisation of the benefits, or payback, from health research33 and includes five main categories: (1) knowledge production; (2) research targeting; (3) capacity building and absorption; (4) informing policy and product development; and (5) health benefits and broader economic benefits. The rationale for using this framework is that it is the most commonly used framework in the evaluation of health research impact,34 it is well described in the literature and there are a number of publications detailing suggested methods for conducting evaluations. In addition, although it was not developed specifically for systematic reviews, it has been used to assess their impact. 35

As previously stated, health benefits and broader economic benefits of research are hard to measure and are largely beyond the remit of this evaluation. 29 Therefore, we used a framework that combines elements of the HERG framework (knowledge production, research targeting, informing policy and product development) with elements (impact on practice/services) from The Research Impact Framework developed by Kuruvilla et al. 27 The latter is a conceptual framework that uses a standardised way of describing a wide range of potential areas of health research impact. 27,36 It was created by identifying potential areas of health research impact and draws on a number of other models including the payback model of health research benefits previously described26 and Lavis’s knowledge transfer approach to assessing the impact of research. 37 The framework we used for this evaluation, including main and subcategories, can be seen in Table 1.

Measuring research impact

There is no single standard approach to measuring impact, and a variety of evaluative methods exist including bibliometrics, documentary analysis, semistructured interviews, case studies, panel review, surveys and network analysis. 17,34 The methods most frequently suggested for analysing the impact of research are bibliometrics, documentary review and interviews. 17,39 There are advantages and disadvantages of each method and it is generally recommended that a variety of sources be used in evaluations of research impact. 29,37 In the light of these considerations we used a mixture of bibliometrics, documentary analysis, questionnaire surveys and interviews. These methods are chosen because they were considered appropriate for determining and comparing the impact of reviews published by 20 CRGs on a variety of topics and over a 5-year period. They also enabled richer data to be gathered and allowed for triangulation. Moreover, these methods enabled us to track backwards from policy documents (WP 1) and track forward from specific systematic reviews (WP 2). These methods are discussed in greater detail in the sections following.

Introduction

There were three WPs, with WPs 1 and 2 being conducted in parallel. The aim of WP 1 was to obtain a general overview of the impact of CRG outputs and the aim of WP 2 was to undertake a more detailed exploration of the impacts of a representative sample of Cochrane Reviews published by the NIHR-funded CRGs. In WP 3 we synthesised the findings from WPs 1 and 2.

There are currently 20 CRGs that receive support from the NIHR Health Technology Assessment programme. They are the following: Airways; Bone Joint and Muscle Trauma; Cystic Fibrosis and Genetic Disorders; Dementia and Cognitive Improvement; Depression, Anxiety and Neurosis; Ear, Nose and Throat; Epilepsy; Eyes and Vision; Gynaecological Cancer; Heart; Incontinence; Injuries; Neuromuscular; Oral Health; Pain, Palliative and Supportive Care; Pregnancy and Childbirth; Schizophrenia; Skin; Tobacco Addiction; and Wounds. The evaluation focused on outputs, in the form of systematic reviews, published by the CRGs between 2007 and 2011. This time frame was stipulated by NIHR in the original project brief. We included reviews that had either been first published or updated during 2007–11. To ensure that the analyses focused on outputs published during the specified years, review titles were crosschecked against details in The Cochrane Library and a master list of reviews provided by Wiley, the publisher of The Cochrane Library. These data from Wiley included details of the year and issue of The Cochrane Library when reviews were first published and any subsequent updates.

Overview of the research plan

The methods are outlined here and then described in more detail later in the chapter. For details of the methods used for the semistructured interviews, see Chapter 6. A diagrammatic summary of the study can be seen in Figure 1.

FIGURE 1.

Diagrammatic summary of methods. SIGN, Scottish Intercollegiate Guidelines Network; WHO, World Health Organization.

Methods for overview of impact of Cochrane Review Group outputs (2007–11)

Methods for evaluation of impact of representative sample of Cochrane Reviews (2007–11)

In WP 2 we undertook further analysis on a representative sample of Cochrane Reviews published in the last 5 years.

Methods for synthesis of work packages 1 and 2 and consensus meeting (work package 3)

In WP 3 we synthesised findings from WPs 1 and 2. Synthesis was guided by our overarching framework and took into consideration ideas relating to types of impact (e.g. instrumental/conceptual/symbolic). Preliminary findings were discussed at a consensus meeting involving the research team, a representative from the National Clinical Guideline Centre at the Royal College Physicians and members of the University of Hertfordshire Public Involvement in Research Group (PIRG). The purpose of this meeting was to assist in the process of conceptualising and identifying impact and to help make judgements about impacts. Examples of impact, including a review synopsis and any evidence relating to impact, were used as a basis for discussion. This process was considered important in order that we made appropriate judgements about potential or likely impacts and to ensure that impact was not overstated.

Results of Cochrane Review Group questionnaires

In total, 17 of the 20 CRGs returned the questionnaire, although not all had completed all sections. Reasons for not completing all sections of the questionnaire were because the CRG did not have any examples for that section, because it did not have any additional information to that already provided in their annual reports and because it did not have time to fill out all sections. Three declined to fill out the questionnaire. Reasons for not returning the questionnaire included the following: the group felt that the evaluation should be done externally without input from them, the group did not have time to fill out the questionnaire and the group felt there was no additional information to add to that already provided to NIHR in their annual reports. More details can be seen in Table 2.

Research targeting

Cochrane Review Groups were asked if they were aware of any reviews first published by their group between 2007 and 2011 that had generated subsequent research (e.g. contributed to successful grant applications for future primary research). Of the 13 groups that responded to this question, 12 (92%) said yes and one (8%) said no. Of the groups that said yes, 12 provided some form of supporting evidence (some of them providing several examples) and 11 gave details of who had funded the research. All of the 11 groups that gave details of the funding body cited at least one primary study funded by NIHR. Other funders included government organisations in other countries, local funders and charities. A summary of all data on the impact on primary research can be seen in Table 3. Data in Table 3 are collated from CRG and author questionnaires and documentary review.

Informing policy development

Seventeen groups provided us with information about potential impacts on policy, mostly in the form of guidelines which cited a review produced by their CRG. Additional information on inclusion in guidelines was collated from the CRG annual reports, from data compiled by the UKCC, and from documentary review and database searches. All of the CRGs had produced reviews which had affected policy-making in the form of guidelines or guidance. Across the 20 CRGs there were 722 citations in 248 guidelines, with 481 systematic reviews being cited at least once. Cochrane Reviews produced by 13 CRGs had been cited in 30 different NICE guidelines, and reviews from 12 CRGs had been cited in 23 sets of guidance developed by SIGN. Information on inclusion in guidelines (by CRG) is summarised in Table 4. This shows the total number of reviews cited in guidance, the number of guidelines and the level of the policy. The level of policy is categorised as international, national and local, with national meaning guidance produced by a national body in any country (not just the UK) (Table 5 has further details). A full list of guidelines and a list by CRG is available on request.

Nature of policy impact

The conceptual approach adopted for this evaluation includes an assessment of the nature of the policy impact based on the categories devised by Weiss. 53 This attempts to distinguish between instrumental or direct impact and conceptual or symbolic use. In this case there was evidence that a number of Cochrane Reviews had some direct or instrumental impact in that they were used to inform practice guidelines. However, in this part of the evaluation we assessed only whether or not the reviews were cited by guidelines and did not look in detail at how reviews were used or if guidelines included recommendations that were in agreement with the Cochrane Review conclusions. In addition, most guidelines are based on a number of different publications and it is not easy to determine the contribution of individual reviews. For a further exploration of how GDs use Cochrane Reviews, see Chapter 6.

Impact on practice and services

In the questionnaire we asked CRGs if they were able to give us examples of ways in which reviews produced by their CRGs had influenced practice and services. Eight groups provided examples of changes to behaviour and six gave examples of how their reviews had led to health, health-service or economic benefits. Impacts on behaviour included changes through the use of evidence-based guidance and impacts on prescribing behaviour or the use of new technologies. Benefits to the health service included safer prescribing, a reduction in inappropriate prescribing and the identification of new effective drugs or treatments. We discounted some of the examples we were given, either because the impact was judged to be outside the time frame for this evaluation or because no supporting evidence was provided and we were unable to verify the impact in any way.

Examples of impact judged to relate to reviews first published or updated between the years 2007 and 2011 include the following:

• A review on support surfaces for pressure ulcer prevention54 was used to inform guidance on purchasing within the NHS. 55

• Reviews on long-acting beta-antagonists in asthma56–58 may have led to safer prescribing of these drugs for people with asthma (www.fda.gov/Drugs/DrugSafety/PostmarketDrugSafetyInformationforPatientsandProviders/ucm200776.htm).

• A review on colloids versus crystalloids for fluid resuscitation59,60 may have influenced calls to stop starch use within the NHS, a decision that has the potential to save both lives and money. 61

• An updated review on antiviral treatment for Bell’s palsy62 may have contributed to changes in practice and a reduction of prescriptions of antiviral drugs for Bell’s palsy (http://cks.nice.org.uk/bells-palsy).

• A review on antifibrinolytic drugs for trauma patients63,64 led to follow-on research which influenced the decision by the Medicines Innovation Scheme to fast-track tranexamic acid for use in the NHS. Ambulance crews throughout the NHS now administer tranexamic acid to bleeding trauma patients (www.swast.nhs.uk/txa.htm).

In many instances there was a lack of specific evidence to support the claims made in the CRG questionnaires. Even with the additional searches carried out by the research team it was often difficult to verify these claims and find a clear link between a review and an outcome. Moreover, attributing particular behaviour changes, health benefits or costs saving to a particular systematic review (or reviews) is difficult. Generally, new research adds to an existing pool of knowledge65 and many research projects may lie behind a specific advance in health care. 66 Therefore, many of the examples we were given by CRGs have to be considered likely or potential impacts rather than proven impacts.

Despite difficulties verifiying impacts, it is apparent that many Cochrane Reviews have the potential to affect practice and policy and that some produce findings that could potentially lead to cost savings and health-service benefits. Some of these potential benefits are highlighted in the Cochrane quality and productivity topics. These documents are developed as part of the NICE QIPP. They are based on the ‘implications for practice’ section in Cochrane systematic reviews and focus on interventions that lack evidence of change, those with strong evidence for ineffectiveness and those in which risks outweigh benefits. Topics are evaluated only if they reflect a current gap in NICE guidance. 67 We found 19 relevant quality and productivity reports based on reviews from nine different CRGs. Potential benefits identified by NICE included economic benefits through cash savings or the release of cash, improvements in clinical quality, reduction in the use of unproven or unnecessary procedures, and improvements in patient and carer experiences. Table 6 has details of the reviews on which these are based and the potential cost savings and health-service benefits.

Results of author questionnaire

Three reviews per CRG were selected for further evaluation. One review per group was chosen randomly (n = 20) and two on the basis that they were more likely to have had an impact (n = 40). Where possible, the latter were selected from examples provided by the CRGs. Four CRGs did not provide information on impact and reviews for these CRGs were selected by the researchers. Nine reviews were updates and the rest were new reviews. Thirteen were published in 2007, 23 in 2008, 12 in 2009 and six each in 2010 and 2011. Details of these reviews, including citation, country of first author and whether they were a new review or an updated review, can be seen in Appendix 4. In a couple of instances the review initially selected had to be changed for a different review. This was either because the review had subsequently been allocated to a different CRG68 or because the review had been withdrawn from The Cochrane Library. 69

In total, 29 out of 60 authors (48%) returned their questionnaire; 16 were returned after the initial mailing, five after a reminder from the research team and a further eight after they were contacted by the managing editor of the CRG concerned. Thirteen questionnaires were returned by authors based in the UK (out of a possible 34) and 16 (out of a possible 26) from authors outside the UK. The numbers of questionnaires returned for each CRG can be seen in Table 7.

Research targeting

Authors were asked if their review had informed the development of future research, either by any of the review authors themselves or by other researchers. The responses can be seen in Table 8. Supporting evidence included information such as a reference to the study, the name of the funding body or the name of the study. Where the funder was not provided, we undertook internet searches for further details. Research had been funded by a variety of organisations in the UK and elsewhere; this included governmental bodies such as NIHR, charities, universities and industry.

Informing policy development

Authors were asked if they thought their review had influenced health policy- or decision-making at any level of the health service, including international, national, regional, local trust or unit, professional, administrative or managerial. They were also asked if they were aware of any potential future impact, for example if the review was being used in guidelines currently under development. Results can be seen in Table 9. In general, the supporting evidence provided related to the review having been cited in or used to develop some form of clinical or practice guideline.

Impact on practice and health services

Authors were asked if they thought the findings from their review had already led to changes, either directly or through the application of research-informed policies in the behaviour of health-care professionals or providers, health-care managers, health-service users or the wider public. Responses can be seen in Table 10. More respondents gave examples of changes to the behaviour of health professionals (37.9%) than changes to the behaviour of health-care managers (20.7%).

Authors were also asked if they thought the findings from their review had led to any health, health-service or economic benefits. This included costs reductions in existing services, improvements in the process of service delivery, increased effectiveness of services (e.g. through increased health), greater equity (e.g. through the improved allocation of resources) or economic benefits arising from a healthier workforce. Responses can be seen in Table 11.

The majority of respondents reported that they did not know if the review had led to health-service or economic benefits. For those who said yes, the examples of impact related to changes to practice (e.g. a reduction in treatments or technologies not proven to be beneficial) or cost savings through the health service no longer having to pay for a treatment not proven to be beneficial. However, in most cases the supporting evidence provided was anecdotal or difficult to substantiate.

Results of bibliometric analysis

Results of documentary review

Review downloads

We looked at information on the number of times that the reviews were downloaded between 2007 and 2011. Data on the number of downloads was provided by Wiley, the publisher of The Cochrane Library. It was able to supply us with the number of full-text downloads for all years and abstract downloads for 2009–11. The number of downloads varied considerably between reviews. Of the sample of 60 reviews, the 10 that were downloaded most frequently (full text and abstract) can be seen in Figures 3–5. These figures give an indication of the impact of reviews within the research and practice communities and show how downloads for reviews have increased over the 5-year period.

FIGURE 3.

Top 10 downloads from The Cochrane Library in 2009. Identifying codes for reviews can be seen in Appendix 4. a, Top 10 full-text downloads from 2009. b, Top 10 abstract downloads from 2009. This figure has been reproduced from figure 3 © Bunn et al. The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research Syst Rev 2014;3:125,38 under Creative Commons Licence 4.0, https://creativecommons.org/licenses/by/4.0/.

FIGURE 4.

Top 10 downloads from The Cochrane Library in 2010. Identifying codes for reviews can be seen in Appendix 4. a, Top 10 full-text downloads from 2010. b, Top 10 abstract downloads from 2010. This figure has been reproduced from figure 3 © Bunn et al. The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research Syst Rev 2014;3:125,38 under Creative Commons Licence 4.0, https://creativecommons.org/licenses/by/4.0/.

FIGURE 5.

Top 10 downloads from The Cochrane Library in 2011. Identifying codes for reviews can be seen in Appendix 4. a, Top 10 full-text downloads from 2011. b, Top 10 abstract downloads from 2011. This figure has been reproduced from figure 3 © Bunn et al. The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research Syst Rev 2014;3:125,38 under Creative Commons Licence 4.0, https://creativecommons.org/licenses/by/4.0/.

Summary of impact

Overall, 40 of the 60 reviews had been cited in some form of clinical guidance and 15 had influenced further primary research. There were 12 examples of impact on practice or services but not all of these were verified. A summary of the main impacts can be seen in Appendix 5. The data in Appendix 5 are collated from the questionnaires, citation analyses, documentary review and internet searches and show the impact of the reviews in terms of knowledge production, research targeting, informing policy and impact on practice and services.

Introduction

Clinical guidelines have been defined as ‘an attempt to distil a large body of medical expertise into a convenient, readily usable format’76 and as ‘systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances’. 77 A number of potential health- and service-related benefits of clinical practice guidelines have been identified. 77 This includes improvement in the quality of health care, a reduction in variation in service delivery among different providers and geographical regions, better equity and improved efficiency. Although the methods and analyses described in previous chapters enabled us to ascertain if Cochrane Reviews have been cited in guidance, it did not necessarily tell us the role Cochrane Reviews play in the development of guidance, for example whether they are just used as supporting evidence or they were instrumental in informing guidance. Therefore, in order to gain a greater understanding of the role that Cochrane Reviews play in the development of guidance, we undertook semistructured telephone interviews with key informants at national and international bodies involved in the development of guidance.

Methods

Analysis

Owing to time limitations, and because we had a relatively small number of transcripts for analysis, we undertook the qualitative data analysis by hand. In order to elicit key features of GDs’ experiences of using Cochrane Reviews, we used thematic content analysis. 79 To ensure a degree of inter-rater reliability and transparency, two authors (of AM, FB and DT) independently read and coded each transcript. From this a list of initial codes and themes were created which were then further refined after discussion with the wider project team. 80

Findings

Altogether we interviewed eight participants: four from NICE (or NICE collaborating centres), two from SIGN and two from WHO. Out of the eight participants, five were female and three were male. More details of the participants and their roles can be found in Table 13.

Participants have been assigned a number, which is linked to any quotes in the text. In order to preserve interviewees’ anonymity these numbers were assigned randomly.

The analysis resulted in six overarching themes and a number of subthemes relating to the views and experiences of GDs and their use of Cochrane Reviews. The overarching themes are:

• the process of using Cochrane Reviews in the development of guidance

• the quality of Cochrane Reviews

• culture and approaches

• up-to-date evidence

• methodological issues

• collaboration and communication.

These themes and subthemes can be seen in Box 1, and are described in more detail in the text.

Theme 1: the process of using a Cochrane Review in the development of guidance

It was clear that Cochrane Reviews were used at a number of different stages of the guideline development process. They were often used early in the process, for example to scope review questions and assess the strength of existing evidence.

Normally at the guideline group meetings I might present the findings [of Cochrane Review] . . . to give them an introduction into what kind of research is out there already.

Participant 4

All of the GDs we spoke to said that they would search for Cochrane Reviews as part of the guideline process, with searching for Cochrane Reviews often seen as a priority.

If you find a couple of good Cochrane Reviews you think oh thank heavens, you know Cochrane have done it. So we would use those first and foremost.

Participant 8

If suitable Cochrane Reviews existed they would be used as they were or built on and updated.

We would build on an existing review if it directly addresses the review question and then we might just update the search to look for some new evidence.

Participant 5

However, GDs were not always able to use existing reviews because they needed updating or did not focus on the same questions or outcomes as the guidance under development.

If it fits the review protocol perfectly, we will just use the review as a review . . . Obviously, it’s quite rare that that happens.

Participant 7

[T]hey [Cochrane reviewers] perceive a particular question from more of an academic ground while within the guideline development group they tend to want to find answers that are more applicable on the ground, in clinical practice.

Participant 7

However, even when GDs were not able to use the whole Cochrane Review, it appeared that they often built on existing reviews or used component parts of the review, such as the search strategy, evidence tables, quality assessments or analyses.

Theme 2: quality of Cochrane Reviews

Views on the quality of Cochrane Reviews varied between interviewees but on the whole Cochrane Reviews were seen as of high quality and robust. The transparency of Cochrane Reviews was seen as an advantage in that it made the reviews easy to interpret and replicate.

Well I think they’re kind of gold dust, you know, I’m always really pleased when I find one because I think they’re good quality but they’re also laid out in a very systematic way.

Participant 8

It’s, you know, a very reliable, trustworthy brand that people know about.

Participant 1

However, there were some concerns about review quality, with the suggestion that not all Cochrane Reviews were of the same quality and that older reviews were sometimes of poorer quality than newer ones.

I think the majority of Cochrane Reviews are better than the normal systematic reviews published out there but within Cochrane Reviews you do find quite poor-quality Cochrane Reviews.

Participant 7

Theme 3: culture and approaches

It was clear that there were many similarities in the culture and approaches of Cochrane and the GDs. The GDs took a similar approach to that of Cochrane reviewers towards evaluating and interpreting evidence and both seem to prioritise systematic reviews and RCTs as gold standard evidence.

I think their [Cochrane] processes are quite similar to the processes that we use, so if I know that there’s a Cochrane Review published I’ll definitely go to that as my first line.

Participant 4

Guideline developers, however, sometimes included broader types of evidence than Cochrane Reviews (e.g. including observational studies), and, because of the time frame in which guidelines are produced, they sometimes had to make more pragmatic decisions about the review process (e.g. doing only a percentage of double data extraction and critical appraisal in contrast with the Cochrane standard of independent double data extraction).

There were also perceived differences in the way review questions were developed, with interviewees reporting that they felt Cochrane Reviews sometimes took too narrow a focus to be useful for guideline development.

There are people [referring to Cochrane reviewers] who will take a kind of broad view and put a lot of interventions together, and then do kind of analyses comparing among them. And that’s really the strategy that you need for a guideline. You’re interested in choosing among a set of treatment alternatives, knowing whether one treatment is better than nothing. And then there are a fair number of Cochrane Reviews that look at one tiny little thing . . . well, it’s difficult for us to use that sort of thing in the guideline.

Participant 3

Another participant acknowledged that, although Cochrane Reviews with a narrow focus could be a problem:

Cochrane is moving more towards comparative effectiveness reviews.

Participant 2

Theme 4: up-to-date evidence

A recurring theme throughout the interviews was around the need for up-to-date evidence. As one participant put it, their first concern about Cochrane Reviews was:

Is the right question there and in the library, and is it up to date?

Participant 2

They went on to say that the likelihood of this was rather ‘hit and miss’. In fast-moving fields Cochrane Reviews could become out of date quickly.

The growth in this field is so rapid that, you know, if a Cochrane Review is 2 years out of date, we’re going to end up redoing the searches and having to add studies to the analysis and things.

Participant 3

Guideline developers also felt that it took too long for Cochrane Reviews to be updated. Factors that were felt to contribute to this delay in updates included a lack of resources for updating, reviewer delay and slow editorial processes on the part of CRGs. Out-of-date reviews appeared to be less of an issue for GDs at WHO, as they had close links with several CRGs and had the resources to pay Cochrane groups to update their reviews.

Theme 5: methodological issues

It was clear that the methods used for Cochrane Reviews were well respected by GDs. Despite this there were instances when GDs felt that the wrong statistical approach had been used in the Cochrane Review or that there was a lack of information about the way statistical calculations had been performed:

Sometimes I wasn’t clear about what correlations they used to adjust the standard deviations and I couldn’t find it inside the text.

Participant 4

There was a suggestion that policy-makers might require different review approaches (such as overviews and network meta-analyses) from that of the traditional Cochrane Review. It was also felt that GDs would benefit from the development of ways of sharing data.

I would like to see us get to a point where we’re sharing data. Where, you know, we’re entering information in a way that means, when somebody has extracted 50 studies of drugs for schizophrenia, if we want to look at another question that isn’t answered in the way that they’ve analysed the data, we can go and just reanalyse the data.

Participant 3

Theme 6: collaboration and communication

A recurring theme throughout the interviews was the importance of good communication between guideline development and Cochrane authors and CRGs. Although GDs had positive experiences of collaboration with CRGs, this was not always the case and there were instances where GDs had found contacting and collaborating with CRGs or review authors difficult.

I think it’s more about, you know, the communication and collaboration, so, you know, how can we share the information, I think that’s the problem at the moment, and it’s extremely difficult when you have so many different Cochrane groups and they all work differently, so, trying to establish that kind of relationship is quite difficult.

Participant 7

It was clear, however, that GDs saw the importance of collaborating more closely with CRGs and some were attempting to do so. For example, one interviewee spoke of the efforts they were making to build better communication with CRGs into their processes:

We would send out information to the Cochrane Review Group that has done the reviews, our review questions and the reviews that we think would be appropriate to try and get some feedback about the, um, whether any of those reviews are being updated or whether there’s any new reviews that we don’t know about in the pipeline. And that’s something that we’re currently in the process of doing for a new guideline that we’re working on.

Participant 5

Several participants referred to issues around ownership of reviews or adequate recognition for Cochrane authors as a barrier to better collaboration.

I think there’s a concern, you know, on the Cochrane side, about how their work will be used and we need to find a way that, you know, Cochrane are confident that their work is being cited properly and represented accurately.

Participant 3

Barriers and facilitators to the use of Cochrane Reviews

It is clear from the data presented that there are a number of barriers and facilitators to the use of Cochrane Reviews in the development of guidance. These relate to the process of guideline development, the quality and methods used for Cochrane Reviews, the culture and approaches of Cochrane and GDs and collaboration and communication between review authors or CRGs and GDs. Some of the main barriers and facilitators identified from our analyses are summarised in Table 14.

Summary of findings

In total, 85% of CRGs and 48% of review authors returned questionnaires. There was evidence that all the CRGs had published reviews that had some impact. The clearest impacts were on research targeting and health-care policy, with less evidence of direct impact on clinical practice and the organisation and delivery of NHS services. Although there was evidence to suggest that some of our sample of 60 reviews had had a significant impact on research, health-care policy or clinical impact, others appeared to have had little or no impact on the research and practice communities. The reviews that demonstrated the clearest evidence of impact were those that had been updated during 2007–11 rather than first published during that time. 60,71,81

Evidence of impact varied between CRGs and between reviews. These differences between CRGs might be attributable to a number of factors, including the scope or speciality of the group, the type and number of outputs, methods used for dissemination and knowledge transfer and the extent to which CRGs routinely collected data on impact. Amongst the sample of 60 reviews, variation might be accounted for by the relevance of the review question and findings to decision-makers, the date of the publication, the strength of the evidence or the strategies used for disseminating the findings.

Limitations

Evaluating the impact of research is complex and difficult and there are a number of methodological issues that might have a bearing on the validity of the results of this study. It is acknowledged that knowledge-production activities related to health research are easier to distinguish than impact on policy or health gain. 34 Moreover, there is a lack of agreed instruments or methods for determining impact. 34 Although health benefits and broader economic benefits may be viewed as the real payback from health research, these are hard to measure, as it is difficult to attribute particular health gains to specific pieces of research. 29 Indeed, although we made some inferences about the wider health and economic benefits of Cochrane Reviews, these are difficult to prove. This evaluation focused on the impacts of published outputs only. It is likely that funding to CRGs results in other forms of payback that are not identified in this study; for example, benefits in terms of research capacity building, methodological development and the promotion of evidence-informed decision-making.

Our approach included the use of citation analysis, with higher citation counts taken as an indicator of greater impact within the research and practice communities. It has been argued that such techniques focus on the quantity of research outputs instead of outcomes or impact. 17 In addition, it may take several months or even years for a work to be first cited and there are questions about the reliability of citation data for Cochrane Reviews. However, we used a mixed-methods approach, including documentary review, questionnaires with CRGs and review authors, and interviews with GDs, which allowed us to get a more comprehensive picture of impact. We also supplemented traditional citation counts with the use of alternative metrics which provided an indication of the amount of attention a review had received in social media.

We examined the impact of three reviews per CRG. Although we attempted to include a representative sample of reviews, by selecting some randomly and some on the basis of impact, our sample is small and it is possible that the reviews we chose are not typical of the outputs of those CRGs. There may also be better examples of impact. Only 29 authors (48%) responded to the survey, which raises further questions about the generalisability of the findings. However, this response rate compares favourably with a previous survey of Cochrane authors in which 26% of contact authors responded. 89 Moreover, although we identified a considerable number of guidelines that had cited Cochrane Reviews, it is unlikely that this list is exhaustive. In particular, we may not have identified local guidelines, non-UK national guidelines or guidelines published in languages other than English.

Much of the data presented in this report are stratified by CRG. However, the varied nature of review topics and differences in the number of questionnaires returned for each group and the length of time since publication mean that it may not be appropriate to make direct comparisons between reviews or between CRGs. It is likely that reviews published more recently may not have had sufficient time to generate citations or impact on policy or practice. It takes a number of years for citation counts to peak, typically 3 or 4 years post publication, and a window of 5 years has been suggested as most appropriate for research assessment. 90,91 Indeed, it may take up to 10 years for the full impact of research to be apparent. 92

This was a retrospective evaluation of impact and as such may be a greater risk of recall bias than one in which data is collected prospectively. Moreover, as many of the data were collected from those most closely associated with the reviews, authors and editorial bases, there is a possibility that respondents might have inflated the impact of their work. In order to prevent this, the research team critically reviewed the information provided, sought evidence to verify impacts and checked identified impacts were related to an output produced in the time frame of interest (2007–11).

As we interviewed only eight GDs, all of whom came from leading guideline development agencies, it is possible that the findings may not be directly transferable to other organisations. Moreover, the majority of the interviewees were based in the UK and their views and experiences may not be the same as those of GDs in other countries. Despite this, consistent and recurring themes emerged from the interviews and the views expressed are concordant with literature in this area. 78

Implications of the findings

The results of the impact analysis presented in this study provides evidence that some Cochrane Reviews produced by NIHR-funded CRGs have had an impact on clinical practice, clinical guidance and the conduct of new primary research. Although we found limited evidence that Cochrane Reviews had had a direct effect on clinical practice, there is evidence that they have an indirect impact on health-care and patient outcomes through their role in informing further primary research and clinical guidance. Although the implementation of NICE guidance has been shown to be variable,93,94 there is evidence that clinical guidelines can be effective in changing the process and outcome of care. 95–99

Similarly to other evaluations of the impact of health-care research, we have found that impacts on policy and research are more easily discernible than those on clinical practice and that there are significant difficulties associated with determining the impact of specific pieces of research. 34 The focus of this evaluation was on the outputs of 20 CRGs covering a broad range of health-care topics. It is possible that it would be easier to determine impact on clinical practice and the behaviour of health-care providers if such evaluations had a narrower focus. This might allow the use of more qualitative and quantitative methods targeted at specific groups of health-care providers.

Although the interviews suggested that Cochrane Reviews are embedded in the guideline development process, there were a number of barriers to their use, including having too narrow a focus or being out of date. Both formal and informal networks are needed to facilitate the transfer of research knowledge to decision-makers100 and improved collaboration and communication between GDs and CRGs might facilitate the use of Cochrane Reviews in the development of guidelines.

Previous research has identified a number of potential facilitators for increasing the impact of systematic reviews on the policy-making process. This includes involving policy-makers in the review process, making reviews relevant to local settings and contexts, collaboration between researchers and policy-makers and disseminating the results of systematic reviews in user-friendly formats. 78 Tailored messages may be an effective strategy for promoting evidence-informed decision-making,101,102 as is identification of the audience for whom the review is planned. 37 This might include the general public, patients, clinicians, managers and public policy-makers. Such considerations will have a bearing on the way the question is formulated and the review undertaken. Considerations of applicability to local context and the inclusion of diverse study types may also increase the usefulness or reviews to policy-makers. 8,84 However, strategies for increasing research impact may require considerable investment on the part of the researcher in terms of time, resources and the development of new skills. 11,103 Despite a growing body of work on increasing research impact, the benefit of such developments is not established and further work in this area is needed. 78,85,101

Conclusions

The last few decades have seen a growing emphasis on evidence-informed decision-making in health care, with research syntheses such as systematic reviews seen as key sources of evidence. In addition, more recently, there has been burgeoning interest in the way in which research is used; researchers are increasingly expected to demonstrate that their work has contributed to society in some way, whether this is through an impact on the economy, quality of life or public policy. This study brings these two trends together in an exploration of the impact of systematic reviews on health-care and patient outcomes.

We identified a number of impacts and likely impacts of Cochrane Reviews. The clearest impacts of Cochrane Reviews are on health-care policy, particularly in the form of clinical guidelines. Interviews with GDs suggest that Cochrane Reviews are embedded in the guideline development process. However, reviews with a narrow focus or that are out of date are of less use to decision-makers, and improved communication between CRGs, review authors and GDs might facilitate their impact on the development of guidance. We found less evidence of a direct impact on clinical practice and the organisation and delivery of NHS services.

An overall aim of the study was to provide information on whether or not Cochrane outputs represent value for money. Methods to quantify and measure value for money remain elusive. Cochrane Reviews influence guidelines, the implementation of which might lead to better outcomes for patients and potential cash savings or economic benefits. However, the extent to which guidelines are implemented and the outcomes of implementation are difficult to measure. What this report does offer is some sense of the size of the knowledge cascade and the way that Cochrane Reviews are contributing to the overall body of evidence.

This study provides some evidence that Cochrane systematic reviews can contribute to evidence-informed decision-making. However, the production of a rigorous review of the evidence is not in itself enough to guarantee impact and there is a growing acceptance that active strategies are needed to facilitate the impact of research, including systematic reviews, on policy and practice. Strategies for knowledge transfer and dissemination are seen by many as an essential part of the review process. It is clear, however, that these strategies for increasing research impact require considerable investment in terms of time, resources and the development of new skills. Despite a growing body of work on increasing research impact, the benefit of such developments is not established and further work in this area is needed. Finally, although it is important for researchers to consider how they might increase the influence of their work, such impacts are difficult to measure. Questions remain about how we define and measure impact and more work is needed to develop suitable methods for impact analysis.

Contributions of authors

Frances Bunn, Daksha Trivedi, Phil Alderson and Steve Iliffe wrote the protocol.

Frances Bunn, Daksha Trivedi, Laura Hamilton, Alice Martin and Emma Pinkney analysed the data.

Frances Bunn wrote the first draft of the report.

Phil Alderson, Steve Iliffe and Daksha Trivedi contributed to the writing of the report.

All authors critically reviewed the manuscript and agreed the final version.

Publication

Bunn F, Trivedi D, Alderson P, Hamilton L, Martin A, Iliffe S. The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research. Syst Rev 2014;3:125.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health.