Effects of a demand-led evidence briefing service on the uptake and use of research evidence by commissioners of health services: a controlled before-and-after study

Article history

The research reported in this issue of the journal was funded by the HS&DR programme or one of its preceding programmes as project number 12/5002/18. The contractual start date was in October 2013. The final report began editorial review in February 2016 and was accepted for publication in July 2016. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

none

Disclaimers

This report contains transcripts of interviews conducted in the course of the research and contains language that may offend some readers.

Copyright statement

© Queen’s Printer and Controller of HMSO 2017. This work was produced by Wilson et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Primary research question

Does access to a demand-led evidence briefing service improve uptake and use of research evidence by NHS commissioners compared with less intensive and less targeted alternatives?

Secondary research questions

Do evidence briefings tailored to specific local contexts inform decision-making in other CCGs?

Does contact between researchers and NHS commissioners increase use of research evidence?

This was a controlled before-and-after study involving CCGs in the North of England. The original protocol is available online (see www.nets.nihr.ac.uk/projects/hsdr/12500218) and has also been published in the journal Implementation Science. 32 There were three phases:

1. phase 1 – pre intervention: recruitment and collection of baseline outcome data (survey)

2. phase 2 – intervention: delivery of study interventions

3. phase 3 – post intervention: collection of outcome measures (survey) and qualitative process evaluation data (interviews, observations and documentary analyses).

Setting, participants and recruitment

Nine CCGs from one geographical area in the north of England were the original focus of this study. The recruitment process is presented as a flow diagram in Figure 1.

FIGURE 1.

Flow diagram of CCG recruitment.

When designing the study, we had anticipated that we would invite nine or ten CCGs from the geographical area based on the 2012/13 primary care trust (PCT) cluster arrangements. By the start of the study, some consolidation in the proposed commissioning arrangements had occurred in the transition from PCTs to CCGs and so the Accountable Officers of the resulting seven CCGs were contacted, told the nature of the study and invited to participate. Of these, six agreed to participate. One CCG declined, intimating that it could not participate in any intervention. No CCG asked for financial reimbursement for taking part in the study.

Clinical Commissioning Groups that agreed to participate were asked to provide details of all governing body and executive members, clinical leads and any other individuals deemed as being involved in commissioning decision-making processes. These individuals were then contacted by the evaluation team and informed of the study aims.

We had originally intended to randomly allocate CCGs to interventions. However, a combination of expressed preferences (one CCG indicated that it would like to be a ‘control’) and the prospect of further consolidation in commissioning arrangements meant that this was not feasible. Taking these factors into account, two CCGs were allocated to receive on-demand access to the evidence briefing service, three coterminous CCGs (who were likely to merge) received on-demand access to advice and support from the CRD team and one to a ‘standard service’ control arm.

After the initial allocation, we were approached by a research lead from a CCG in a neighbouring geographical area who had heard about the study and indicated that he and colleagues in other CCGs were also keen to participate.

The research team then had discussions with representatives of five CCGs at two research collaborative meetings. At these meetings, we explained that any CCGs willing to participate would be recruited as ‘standard service’ controls, but would be offered the opportunity to receive on-demand access to the CRD evidence briefing service after the follow-up phase was complete. Three CCGs agreed to participate. A fourth CCG initially agreed to participate but failed to provide contact details for any personnel involved in commissioning processes, despite repeated requests from the research team to do so. As we would therefore be unable to collect baseline data, rather than delay the start of the intervention phase, the team informed the CCG that it would have to be excluded from the study.

Characteristics of participating Clinical Commissioning Groups

In total, nine CCGs agreed to participate and were able to provide contact details for personnel involved in commissioning processes.

Baseline and follow-up assessment

We collected data for our two primary outcome measures (perceived organisational capacity to use research evidence and reported research use) at baseline (phase 1) and again 12 months after the intervention period was completed (phase 3).

Main study Clinical Commissioning Groups

The survey instrument (see Appendix 1) was the means by which we collected these data. It was designed to collect four sets of information that assess the organisations’ ability to acquire, assess, adapt and apply research evidence to support decision-making. Section 1 was based on a tool originally devised by the Canadian Health Services Research Foundation33,34 and then modified by the SUPPORT (SUPporting Policy relevant Reviews and Trials) Collaboration. 35 The SUPPORT Collaboration included additional domains designed to assess the extent to which the general organisational environment supported the linking of research to action;36 specifically the production of research, efforts to communicate research findings (‘push’), and efforts to facilitate the use of research findings (‘user pull’).

Section 2 was based on a modified version of a tool designed to be administered as part of a planned trial evaluating the effects of an evidence service specifically designed to support health system policy-makers in finding and using research evidence. 37,38 This Canadian tool was itself based on the theory of planned behaviour, a widely used theoretical framework for understanding and predicting behaviours. 39 We used this to assess the intentions of individual CCG staff to use research evidence in their decision-making. The theory of planned behaviour is useful for examining intentions and behaviours of CCG decision-makers as it provides a (validated) model of how the social action involved in using research is shaped by three key variables: attitudes (i.e. beliefs and judgments), subjective norms (i.e. normative beliefs and judgments about those beliefs) and perceived behavioural control (i.e. the perceived ability to enact the behaviour). These three variables drive intentions to behave, which in turn shape future behaviour. 40–42 Lavis et al. 37 and Wilson et al. 38 highlight a compelling rationale for the utility of the theory of planned behaviour as an explanatory framework for at least some of the variability (in the influence on intentions and behaviour) in health care professionals and – in theory – policy-makers:

• About 39% of the variance in intention and about 27% of the variance in behaviour can be explained by theory of planned behaviour constructs.

• Producing valid and reliable measures of key theory of planned behaviour constructs for use with health-care professionals is feasible.

• The proportion of the variance in health-care professionals’ behaviour explained by their intentions was similar in magnitude to that found in the broader literature.

• The agency relationship – between health-care professionals and patients – is not dissimilar to the agency relationship between policy-makers and others.

It was clear from preliminary discussions and our previous contact with CCG decision-makers that they were aware of the desirability of using research and often expressed an intention to use research (indeed, this was one of the principal drivers for our research), but that other mediating factors impacted on their ability to enact these intentions. Using the theory of planned behaviour allowed us to model an important proportion of at least some of the drivers for any eventual behaviour reported or observed.

Section 3 was designed to evaluate the changes to the nature of the (proposed) interactions, both within the participating sites and between commissioners and researchers. Participants are asked how much contact they have had with researchers in their job (quantity), and how successful the interaction (quality) had been, using an existing modified measure. 43 This section included questions regarding the extent to which the interactions were perceived as friendly and co-operative, and as helping to achieve the goals of both managers and researchers. The extent to which those involved in the interaction are perceived as being on an equal footing, without either group dominating, and the extent to which the contact is perceived as being supported by the CCGs, and the NHS more generally, was examined. Participants were also asked to indicate the extent to which their status as a NHS manager/lead is important to them (in-group identification) and to what extent they see themselves and researchers as part of one overarching group committed to achieving the same things (superordinate identification). In addition, we included measures of perceptions of researchers in general using a generalised intergroup attitude scale. 44

Section 4 captured information on individual respondent characteristics, which was collected to help understand any variation in responses.

The language used in all sections was adapted to match the NHS commissioning context and readability was first piloted with the study advisory group. The sections were ordered by importance beginning with the primary outcome measure, the organisational use of evidence. The instrument was then piloted to assess ease of completion, time to complete, appropriateness of language and face validity with a small group of commissioning staff from outside the study setting. Feedback suggested that the questionnaire was comprehensive but feasible, especially as its administration would be solicited rather than unsolicited. As a result of the feedback and in anticipation of some fall in responses as a result of fatigue, we deliberately chose to prioritise the primary outcome measure as the first section on the questionnaire.

National survey of Clinical Commissioning Groups

A second survey instrument that included only the questions from Section 1 in the main case site survey was used to collect data from other CCGs across England. This was delivered at baseline and then again post intervention.

Survey administration: main sites

Each participating CCG supplied a list of names and e-mail addresses for potential respondents. These were checked by a member of the evaluation team and where inaccurate or missing details were identified, these were sourced and corrected. Survey instruments were sent by personalised e-mail to identified participants via an embedded URL. The online questionnaire was hosted by the Survey Monkey website (www.surveymonkey.com). Reminder e-mails were sent out to non-respondents at 2, 3 and 4 weeks. A paper version of the questionnaire was also posted out and telephone call reminders were made by the research team. In addition, the named contact in each CCG sent an e-mail to all their colleagues, encouraging completion.

Survey administration: national Clinical Commissioning Groups

As CCGs were new and evolving entities at the time of the study, we needed to be able to determine if any changes viewed from baseline were linked to the intervention(s) and were not just a consequence of the development of the CCG(s) over the course of the study. To guard against this maturation effect/bias, and to test the generalisability of findings, we administered the instrument to all English CCGs to assess their organisational ability to acquire, assess, adapt and apply research evidence to support decision-making. The most senior manager (chief operating officer or chief clinical officer) of each CCG was contacted and asked to complete the instrument on behalf of their organisation. For the national survey we used publicly available information (NHS England and CCG websites) supplemented by telephone calls to CCG headquarters to construct our sampling frame consisting of every CCG in England.

Interventions

Participating CCGs received one of three interventions aimed at supporting the use of research evidence in their decision-making.

1. Intervention A: contact plus responsive push of tailored evidence.

2. Intervention B: contact plus an unsolicited push of non-tailored evidence.

3. Intervention C: unsolicited push of non-tailored evidence (‘standard service’).

Quantitative analysis

The primary analysis measured the impact of study interventions on two main outcomes (perceived organisational capacity to use research evidence and reported research use) at two time points: baseline (pre intervention) and 1 year later (post intervention). The key dependent variable was CCG-perceived organisational capacity to use research evidence in their decision-making as measured by Section 1 of the survey instrument (see Appendix 1). We also measured the impact of interventions on our second main outcome of perceived research use (see Appendix 1, Section 3) and CCG members’ intentions to use research (see Appendix 1, Section 2). These were treated as continuous variables and for each we calculated the overall mean score, any subscale means, related standard deviations and 95% confidence intervals (CIs) at two time points pre and post intervention.

Secondary analysis assessed any relationships between the model of evidence briefing service (intervention) received and three further continuous independent variables measuring individual demographic characteristics (e.g. job role, clinical or other qualifications) and the quality and frequency of contact with researchers on the two outcome measures.

In our original protocol we (rather optimistically) held out the possibility that the data might allow for a more complex multivariate analysis, which would take into account clustering effects associated with CCGs or NHS Regions. There were insufficient data of sufficient quality to allow for such an analysis. When measures were non-normal, we transformed the data (logarithmically) where necessary and possible. Analysis was undertaken using IBM Statistical Product and Service Solutions (SPSS) Statistics, version 22.0 (IBM Corporation, Armonk, NY, USA) and Stata statistical software version 14 (StataCorp LP, College Station, TX, USA).

We undertook a number of statistical comparisons:

Chi-squared tests of independence were performed to examine the relation between the model of evidence briefing service received and the biographical characteristics of respondents.

To examine the hypothesis that CCGs would differ in their capacity to acquire, assess, adapt and apply research evidence to support decision-making as a result of receiving one of the interventions, we undertook a factorial analysis of variance [(ANOVA) SPSS, version 22.0, general linear model procedure], comparing the main effect of a single independent variable (CCG status) on a dependent variable (capacity to acquire, assess, adapt and apply research evidence to support decision-making) ignoring all other independent variables (i.e. the effect ignoring the potential for confounding from other independent factors). Thus, we assessed the main effects of time and intervention received and the interaction effect (effects of all independent variables on a dependent variable) of both time elapsed and of the intervention on domain scores. Thus, we had one independent variable (the type of intervention) and one repeated measures variable [the total score and domain subscore(s) at baseline and 1 year later].

To examine the hypothesis that the intervention would impact on CCG’s collective intention to use research evidence for decision-making, a factorial ANOVA using the SPSS (version 22.0) general linear model repeated measures procedure was conducted to compare the main effects of time and evidence briefing service received and the interaction effect of time and evidence briefing on intention to use research evidence (using a measure derived from the theory of planned behaviour – see the ‘intention’ component of the study questionnaire, Qs 41–43). As the theory of planned behaviour (in the context of this study) predicts that intention to use research evidence for decision-making will be positively correlated with attitude, group norms and perceived behavioural control in the CCGs according to the intervention it received, we also examined the main effects of time and evidence briefing service received and the interaction effect of time and evidence briefing on these variables.

To examine the effects of (1) perceived contact and (2) the amount of perceived contact with the evidence briefing service, (3) institutional support for research, (4) a sense of being equal partners during contact, (5) common in-group identity, (6) achievement of goals and (7) perceptions of researchers generally, we undertook a mixed 3 (intervention: A vs. B vs. C) × 2 (time: baseline vs. outcome) ANOVA using SPSS (version 22.0), with the intervention as a between-subjects independent variable, and repeated measures on the second factor, time.

Missing data

Missing data and attrition between baseline and 1-year follow-up were issues. Although only ≈16–20% of questionnaires had missing data at baseline, at follow-up more than half the responses were missing or incomplete. As analysing only the data for which we had complete responses would have led to potentially biased results,45 and as anticipated at bid and protocol stages, the use of multiple imputation techniques was required. 46 SPSS multiple imputation processes were used. We assumed that data were missing at random (visual comparison of original versus imputed data and significance testing of response and non-response data impact on outcome variables – see Chapter 4). Five imputed data sets were created and the data imputed were the dependent variables of the capacity score derived from Section 1 of the survey instrument, theory of planned behaviour variables and the measures of perceived quality and quantity of contact with researchers.

We used guidance on interpreting effect sizes in before-and-after studies to examine the clinical/policy significance of any changes. 47

Blinding

Baseline and follow-up assessments and the qualitative aspects of the research were undertaken by a separate evaluation team. The CRD evidence briefing team members were blinded from both baseline and follow-up assessments until after all the data collection was complete. The CRD evidence briefing team were made aware of baseline and follow-up response rates. Participating CCGs were also blinded from baseline and follow-up assessments and analysis.

Qualitative evaluation

To internally (within the context of the local health economy) validate the self-reported data collected in phases 1 and 3 and to explore the decision-making processes within each case site, we collected qualitative data. This was also an opportunity to explore CCGs’ experiences of working with the CRD intervention team and to feed back directly on the service it received. The qualitative data collected via observations and interviews were used to address the following questions:

1. What do commissioners consider to be ‘evidence’?

2. How is research evidence used in the commissioning decision-making processes in CCGs?

3. What is the perceived impact of a demand-led evidence briefing service on organisational use of research evidence?

4. What were commissioners’ experiences of the evidence briefing service?

5. How could the evidence briefing service be improved?

Change to protocol

Part of our original plan (outlined in the study protocol) was to collect and analyse documentary evidence of the use of evidence in decision-making using executive and governing body meeting agendas, minutes and associated documents. This component aimed to capture reported actual use of research evidence in decision-making, whereas our primary outcome measures focused on intention to do so. This was to be supplemented with interviews to explore perceived use of evidence and any unanticipated consequences.

Early in the intervention phase, it became apparent that this approach may not be feasible. With a few exceptions, we found a lack of recorded evidence of research use (a finding in itself), as executive and governing body meetings were mainly used to ratify recommendations and so would not tell us anything about sources or processes. With research use and decisions occurring elsewhere and often involving informal processes, we decided to undertake four case studies to explore use of research evidence in decision-making in the intervention sites. The case studies were three case site CCGs and one commissioning topic involving all CCGs across the region (low-value interventions). The Project Advisory Group approved this change in October 2014. Within the case studies, three types of data were collected: documents, observations and interviews.

Analysis and data integration

This was a mixed-methods study using a sequential explanatory strategy. Initial integration was of the three forms of qualitative data. Data from interview, observation and documentary analysis were uploaded into analysis software and combined to generate a descriptive account of the use of evidence in decision-making within each case. The primary point of data integration was the analysis stage in which themes generated by qualitative analysis were used to help us to understand variation in quantitative outcomes.

Qualitative analysis

Analysis was by constant comparison and used the qualitative data analysis software package NVivo, version 10 (QSR International Pty Ltd, Melbourne, VIC, Australia) to organise and manage the data. Our analytical approach was both deductive (developing themes from the research questions and survey instruments employed) and inductive (new themes emerging from the accounts of key informants). This process was iterative, the researcher returned to the original data several times, reviewing codes and revising each case-study narrative. During this process, data were integrated in three ways. First, interviews were categorised according to the intervention received and differences in the themes generated by each interview were compared and contrasted across each case. Once all data had been collected, one researcher (KF) developed a coding framework based on initial readings of the interview data without grouping by case. Cases were coded systematically with categories emerging from the data itself as well as from the research questions and theories and research literature relating to evidence-informed decision-making. These categories were reviewed by members of the research team (CT, ML, PW and KF) in order to focus the next iteration of coding. KF then reviewed and recoded all transcripts grouped as case studies. At the same time KF conducted text searches of all documentation and observation notes (text searches and manual review of observation notes) to understand the role of evidence obtained from research in decision-making. Identified terms were examined individually to understand the textual context of its use. Finally, themes generated by interviews were compared with those arising from documentary evidence to identify any conflict or consistency between local perceptions of the use of evidence and recorded use of evidence. Analysis of each type of data was integrated into case summaries for each of the three CCG case studies. For example, evidence of the use of research in documentation was used to explore support or refute descriptions given in interviews. Transcripts were randomly selected for review by CT to identify additional themes and to challenge conclusions made by KF.

Summaries describing the characteristics of each case and the local health economy were developed by two researchers (ML and LB). These were used to set the context of the case study and to inform discussion. Some themes were identified in advance from the research questions and theories and research literature relating to evidence-informed decision-making, others emerged from the data during analysis. The researcher was also alerted to concepts and themes while observing meetings during the intervention period. These were explored or reignited during the interview analysis period. The researcher sought confirmation or deviation from these concepts in transcripts and by revisiting notes from observations. Case summaries were developed that drew on data from all sources. Once these had been created, KF returned to the original data to identify any deviation from the narratives created. The second point of data integration was the analysis stage in which themes generated by qualitative analysis were used to help us to understand variation in quantitative outcomes.

Ethics and governance

This study was granted research ethics permission by the Department of Health Sciences, University of York Research Ethics Board. Appropriate research governance approval was also obtained.

Organisation-level consent granting permission to contact staff was obtained from each participating CCG. Individual participants had the opportunity to discuss any aspect of the study and their involvement in it with the research team at any stage of the study. Completion of questionnaires was anonymised and CCGs were informed of response rates but not of individuals’ participation. Interview participants and those present at observed meetings gave informed consent to their participation. None of the interventions involved any direct risks or burdens to the CCGs involved.

Patient and public involvement

The primary focus of this study was interventions targeted at NHS staff undertaking core roles within CCGs, so the active involvement of the public or service users in the design of this project was not sought. Patient and public involvement was provided through lay representation on the Project Advisory Group and through the development of the Plain English summary. We also committed to produce a summary of our findings in plain English and to ensure that these are shared with lay members of governing bodies in all of the participating CCGs.

Introducing the service

For the five participating CCGs allocated to receive contact via interventions A and B, we offered to come and explain the nature of the evidence briefing service and the aims of the study at the next available executive team meeting. Three of the five CCGs accepted the offer. Face-to-face meetings were arranged with representatives of the remaining two CCGs (who were also two of the three CCGs likely to merge). At each meeting, we outlined the aims of the study and highlighted the free advice and support for evidence-informed commissioning being made available from the CRD. Specifically, we offered help on clarifying issues, formulating questions and advice on how to make best use of the research evidence relevant to the commissioning challenges it faced. Recent work on telehealth undertaken for a CCG outside the study setting was used to illustrate how the evidence briefing process worked and what the CCG could expect in terms of a response to any questions it raised. At the meetings, we emphasised that participation in the study would help the CCG to fulfil its statutory duties under the Health and Social Care Act 2012,3 but also stressed that as this was a demand-led service; the extent to which the CCG engaged with the service was entirely at its own discretion.

After each meeting, a personalised e-mail was sent to all Executive Team members, clinical leads and commissioning managers within the CCG restating the aims of the study and the nature of the offer from the CRD.

For co-ordination purposes we suggested that each CCG nominate a senior person who we could liaise with and could act as the conduit for all CCG requests. Once named contacts were identified, they were invited to discuss areas of interest with their colleagues and get in touch and discuss their needs with the evidence briefing team. Each named contact was then met individually face to face to discuss the evidence briefing process, the nature of support being offered and to identify any initial CCG priorities.

Producing the evidence briefings

The process for producing evidence briefings followed that developed as part of the TRiP-LaB (Translating Research into Practice in Leeds and Bradford) theme of the NIHR CLAHRC for Leeds, York and Bradford and by the CRD as part of its core contract under the NIHR Systematic Reviews Programme. 30

On receipt of each request, an attempt was made to define the research question to be addressed in terms of population, intervention, comparator and outcome. 49 This was done via discussion with the named contact and/or the individual(s) making the request. Discussions rarely involved more than three named individuals as decision-making processes were found to be largely informal and rarely involved minuted meetings or gatherings of CCG staff. Most interactions around priority topics and questions were either telephone or e-mail based (> 500 e-mails relating to the formulation of questions and the production and dissemination of briefings were received or sent over the course of the study). Relevant contextual information, and, in particular, the background to the request being made, were also sought from the individuals making the request.

In some instances, interest in a topic was identified but a specific research question could not be framed initially. In such cases, we produced evidence notes, which aimed to provide a quick scope of the available evidence in the area. This then helped to frame the question(s) to be explored by subsequent, more focused, evidence briefings.

Identifying the content

As with our earlier developmental work,30 the evidence briefings were based on existing sources of synthesised, quality-assessed evidence and applied to the local context. Searches for relevant systematic reviews and economic evaluations were performed by the researchers responsible for each briefing. The core sources searched for evidence were:

• Database of Abstracts of Reviews of Effects (DARE)

• NHS Economic Evaluation Database (NHS EED)

• Health Technology Assessment (HTA) database

• International Prospective Register of Systematic Reviews (PROSPERO)

• Cochrane Database of Systematic Reviews (CDSR).

During the course of the study, NIHR funding for the production of two databases, DARE and NHS EED, ceased. The CRD continued to conduct weekly searches, systematic reviews and economic evaluations until the end of December 2015. From January 2015 onwards, when searching for systematic reviews, the briefing researchers undertook additional searches of PubMed using the ‘Review’ filter and NHS Evidence using the ‘Systematic review’ filter.

For topics that were likely to be impacted by national guidance, we searched the National Institute for Health and Care Excellence (NICE) website. Additional sources were also searched for relevant policy reports and for other grey literature. These included the websites of The King’s Fund, Nuffield Trust, Health Foundation, Nesta, NHS England and the NIHR Journals Library. If systematic review evidence was limited, recent primary studies (published from 2010 onwards) were identified by searches of PubMed.

Data extraction and quality assessment

We stored the literature search results in a reference management database [EndNote (Thomson Reuters, CA, USA)]. One researcher screened all titles and abstracts obtained through the searches for potentially relevant content. Two researchers then independently made decisions on content most relevant to the questions to be addressed. Once selected, data were extracted into summary tables by one researcher and checked by another. Throughout this process discrepancies were resolved by consensus or where necessary by recourse to a third researcher.

Systematic reviews and economic evaluations included in DARE and NHS EED meet basic criteria for quality and a significant number have been critically appraised in a structured abstract. Where a critical abstract was not available, or was identified through other sources, we applied the well-established CRD critical appraisal processes for DARE and NHS EED (see www.crd.york.ac.uk/crdweb/HomePage.asp). For systematic reviews, the specific aspects assessed were the adequacy of the search; assessment of risk of bias of included studies; whether or not study quality was taken into account in the analysis and differences between studies accounted for; any investigation of statistical heterogeneity; whether or not the review conclusions were justified. When systematic review evidence was limited and primary research was identified, quality was assessed using the appropriate Critical Appraisal Skills Programme tool for the study design. 50 We included only evidence from primary studies that were judged to be well conducted. Quality assessments were performed by one researcher and checked by a second; discrepancies were resolved by consensus or recourse to a third researcher where necessary.

Presentation and dissemination

The presentational format for evidence briefings was based on our previous experience producing the renowned Effective Health Care and Effectiveness Matters series of bulletins (www.york.ac.uk/crd/publications/archive/)51,52 and from CRD guidance on disseminating the findings of systematic reviews. 49 With the exception of the independent appraisal of the evidence underpinning the proposed policies for musculoskeletal (MSK) procedures, evidence briefings took the following format:

• front page bullet point summary of key actionable messages

• background section describing the topic and the local context

• evidence of effectiveness: a summary of systematic review findings (or primary studies if necessary); critical appraisal of the strength of the evidence; assessment of generalisability

• evidence of cost-effectiveness: summary of economic evaluations and their findings; critical appraisal

• implementation considerations based on the evidence, for example, implications for service delivery, patient and process outcomes, and health equity

• references.

Evidence briefings were formatted using InDesign (Adobe, San Jose, CA, USA) desktop publishing software and were reviewed and edited by a second researcher and the principal investigator before being approved for circulation. Once approved, evidence briefings (and evidence notes) were e-mailed as an attachment to the named contact and to the individual(s) who made the initial request. The e-mail included the headline messages from the briefing, a request to circulate and an offer both to discuss the findings further (either by telephone or face to face) and to respond to any questions or clarifications that readers may have. Each evidence briefing was also e-mailed to the named contacts at other CCGs using the same format.

Each evidence briefing was published (with metadata) on the CRD website, and a record added to the HTA database. HTA database records contain full bibliographic details, hyperlinks and contact information for the organisation publishing the report. Indexing on the HTA database increased the likelihood that anyone searching for related terms on linked platforms such as The Cochrane Library, NHS Evidence, TRiP (Turning Research into Practice) Database and The Knowledge Network of NHS Scotland would identify any relevant evidence briefing as part of their search.

Questions addressed by the evidence briefing service

Over the course of the study we addressed 24 questions raised by the participating CCGs, 17 of which were addressed during the intervention phase (see Table 1). The majority of requests were focused on options for the delivery and organisation of a range of services and way of working rather than on the effects of individual interventions. Vignettes for each topic addressed are presented in Appendix 3. The evidence briefings are available at www.york.ac.uk/crd/publications/evidence-briefings/ (accessed 9 June 2016).

Types of evidence use

Requests for evidence briefings from the CCGs served different purposes. Four broad categories of research use have been proposed. 8,53,54 Conceptual use is when new ideas or understanding are provided and, although not acted on in direct and immediate ways, these influence thinking towards options for change. Instrumental use is when evidence directly informs a discrete yes/no, should we invest/disinvest decision-making process. Symbolic (or tactical) use refers to those instances in which research evidence is used to justify or lend weight to pre-existing intentions and actions. The final category is imposed when there are organisational, legislative or funding requirements that research be used.

For each evidence briefing and note produced, we employed these categories to classify the underlying purpose driving the type of research use. Although our interpretation is subjective, the classification presented in Table 1 is derived from a consensus-based approach. Most of the requests we received were categorised as conceptual. These were not directly linked to discrete decisions or actions but were requested to provide new understanding about possible options for future actions. Symbolic drivers for evidence requests included a pre-existing decision to close a walk-in centre, a successful challenge fund bid to implement self-care and decisions to implement GP telephone consultations. Questions categorised as instrumental related to explicit disinvestment or investment decisions. There were no instances that we considered to represent an imposed use of research.

In addition to the evidence briefings and notes, we also circulated other CRD-generated content known to be of relevance and interest to participating CCGs. Effectiveness Matters is a short, four-page summary of research evidence about the effects of important interventions for practitioners and decision-makers in the NHS.

During the study period, a number of these bulletins were produced by the CRD in collaboration with the Improvement Academy of the Yorkshire and Humber Academic Health Science Network [www.york.ac.uk/crd/publications/effectiveness-matters (accessed 9 June 2016)]. When topics aligned with the stated areas of interest of the intervention CCGs, relevant issues of Effectiveness Matters were circulated to the named contacts for onward dissemination within the CCG. The issues of Effectiveness Matters that were circulated were as follows:

• Dementia carers: evidence about ways of providing information, support and services to meet the needs of carers for people with dementia (May 2014)

• Preventing pressure ulcers in hospital and community care settings (October 2014)

• Preventing falls in hospital and community care settings (October 2014)

• Recognising and managing frailty in primary care (January 2015)

• Acute kidney injury: introducing the 5 ‘R’s approach (December 2015).

Other questions raised but not addressed

Other topics of interest were raised by CCGs around the beginning of the intervention period but were not addressed as individual evidence briefings or notes. Some were not pursued, as CCGs deemed other topics to be of higher priority, whereas others were constituent parts of other published or planned briefings. Details of questions raised are presented in Table 2.

Training

The evidence briefing team offered to provide training on how to acquire, assess, adapt and apply synthesised evidence to those CCGs receiving interventions A and B. The formal offer was made at baseline to named contacts and separately to all staff. Separate informal offers were made to named contacts throughout the course of the intervention. Two CCGs (A2 and B3) did express interest in receiving training on identifying and using research evidence but were then unable to respond to requests to suggest dates and times for the training to take place. As such, the team instead opted to devise and circulate a two-page guide for commissioners on using evidence to support decision-making, based on the process for developing evidence briefings (see Appendix 5). The guide was circulated in April 2015 and made available on the CRD website at www.york.ac.uk/media/crd/Process%20flowchart_expanded%20FINAL.pdf (accessed 9 June 2016).

What follows are three exemplars of the types of conceptual, instrumental and symbolic research use experienced during the study.

Conceptual use of research evidence: social prescribing

In October 2014 we were approached by the A2 CCG and asked if we could provide evidence on the effectiveness of social-prescribing schemes. The CCG provided a short briefing report, which outlined that it was considering introducing a pilot scheme in one locality to improve the health and well-being of people with long-term conditions. It was envisaged that people would be referred to community-based services that would complement traditional medical interventions and that these would help people to manage their conditions better by learning new skills in self-management and avoid costly interventions in specialist care in the longer term. The CCG recognised that partnership with the voluntary sector could provide increased choice and value for money and that services could be more closely tailored to the needs of the community.

The CCG was particularly interested in any evidence that social prescribing reduced primary, secondary and community care workloads and service utilisation and in any evidence of cost-effectiveness (a specific request for cost per quality-adjusted life-year gained). After visiting a few high-profile schemes, it was concerned that its plans to introduce small-scale social prescribing on a ‘shoestring’ might not be effective and/or sustainable. It was hoped that a review of evidence would help to justify the small investment needed to get the scheme off the ground and would help to ensure that this would be a service investment that would pay back in the longer term.

A quick (10-day) response was requested, so we opted to compile a short evidence note rather than a full briefing. We searched the DARE, NHS EED and CDSR databases for relevant systematic reviews and economic evaluations. These initial searches revealed little relevant evidence, so we also conducted quick searches of MEDLINE, Applied Social Sciences Index and Abstracts (ASSIA), Social Policy and Practice, NICE, Social Care Institute for Excellence (SCIE) and NHS Evidence to locate details of any relevant guidance, case studies or service evaluations.

Overall, we found little supporting evidence to inform the commissioning of a social-prescribing programme. The identified evidence was characterised by brief descriptions of small-scale projects and failed to provide sufficient detail to judge either success or value for money. Rigorous evaluation of the cost-effectiveness of social-prescribing schemes was also lacking.

On feeding back these findings to the CCG, it highlighted that it knew of the existence of an evaluation of a scheme that it had visited but that was not included in the evidence note. The evaluation was of interest as the scheme had showed areas of improvement and possible savings. We explained that, as we had not yet searched for grey literature, the report had not been identified but that we would appraise it separately. Although detailed, the evaluation presented a number of significant limiting factors, which were in line with the overall findings of the evidence note. Specifically, the uncontrolled before-and-after evaluation failed to address the counterfactual potential confounders and the issue of regression to the mean. The report also lacked detail about the type of patients included in the analysis – what conditions they had, what interventions they received. The CCG indicated that it found the additional information ‘very helpful’ and noted the absence of evidence. The CCG opted to proceed with developing a pilot social-prescribing programme in conjunction with the local authority.

Before circulating the findings more widely, we decided to convert the evidence note into an evidence briefing. As we were aware that we had missed an evaluation, we conducted updates of our initial searches and undertook systematic searches for individual studies and for grey literature. Plans to convert this work into a systematic review were also registered with PROSPERO (CRD42015023501). The evidence briefing was circulated to all participating CCGs in March 2015.

Once publicly available, the evidence briefing generated media interest with the briefings headline message of a lack of evidence featuring in the Guardian newspaper. The team also received a number of enquiries from CCGs and Health and Wellbeing Boards located elsewhere in England and Scotland. All of the enquiries focused on evaluation and asked, given the absence of evidence, how should the effects of social-prescribing schemes be evaluated in ways that would improve the existing evidence base?

In July 2015, we were contacted by members of the public health team in a local council in the geographical area in which the study was based. They had been asked to summarise the latest evidence related to social prescribing and indicated that this was an area that the council in conjunction with B1 CCG was keen to explore further. They had found the evidence briefing through a search of NHS Evidence and were unaware that the briefing had previously been circulated to contacts within B1 CCG or of the plans of A2 CCG to develop a pilot programme.

They asked if we had any plans to update and if we would be willing to present the evidence base around social prescribing at a workshop being held for the Health and Wellbeing Board. The workshop took place in November 2015 and brought together local councillors, NHS organisations, third-sector agencies and representatives from social-prescribing schemes to explore if and how best to take social prescribing forward in the area. Our contribution was to present on ways to improve the measurement and evaluation of social-prescribing schemes. The Health and Wellbeing Board indicated that it intended to proceed with developing plans for a pilot social-prescribing programme in 2016.

Instrumental use of research evidence: low-value interventions

In early discussions with the named contact for the B3 CCG about their priority areas, low-value interventions were identified as a major area of interest for all CCGs in the region.

Low-value interventions are those treatments that are considered to be of no or low clinical benefit or that are not cost-effective compared with treatment alternatives. A region-wide list of low-value procedures was established in 2010. The aim of the value-based commissioning policy (VBCP) list was to provide local GPs with clear criteria for funding and referral, and to ensure that policies were applied consistently across all PCTs in the region. Each PCT had run its own individual funding request (IFR) process to handle any requests (on exceptionality grounds) that fell outside the commissioning criteria. The regional VBCP list included 39 procedures and was last reviewed and updated in 2012. This last update pre dated the transition of commissioning arrangements from PCTs to CCGs.

The initial exchange of e-mails focused on the practical challenges in identifying and implementing low-value policies, and a paper on the experiences from a NICE and Cochrane project was circulated. 55 The named contact revealed that they were about to start heading up a project group representing a cluster of seven CCGs to try to look at procedures of limited clinical value being undertaken in secondary care. The aim of the collective work was to consider the inclusion of a wider range of procedures on the regional VBCP list. The group also planned to assess the usability of data monitoring reports and to look at how IFR policies were being implemented with a view to developing strategies to ensure more effective implementation of policies. The CRD intervention team was invited to attend the meetings.

As a first contribution, we offered to identify any further policies that might be considered going forward. A systematic search of the NICE Do Not Do Database and Cochrane Quality and Productivity topic reports was conducted and yielded a list of 36 potential topics for consideration. These were presented to the Group in a summary form in July 2014.

At the September meeting of the VBCP Implementation Group, the Lead for the A2 CCG presented a MSK resource pack they had compiled. They indicated that it had been developed as part of a cost-saving exercise and were anticipating that its implementation would reduce the number of referrals as well as ensure appropriateness of referrals from GPs. The resource pack included 16 policies, 14 of which were not included in the regional VBCP list. The pack had been compiled from existing policies identified at other CCGs across the country. It was also ‘sense checked’ by a consultant in public health who had been involved in the compilation of the original regional VBCP list. Discussions had taken place with colleagues in the local provider trust who, it was reported, had not expressed any concerns about the proposed polices.

The A2 CCG had already asked its member practices to implement the new MSK procedures in addition to the existing 39 procedures. However, a regional web-based system to manage IFRs from GPs in all CCGs had recently been introduced. This meant that any IFRs derived from this new list would have to be processed separately. GPs making IFRs from the new list were being asked to complete a paper-based checklist and incorporate this into the patient notes and referral request. The A2 CCG therefore hoped that all the other local CCGs would adopt the MSK policies and that these would become incorporated into the regional VBCP list and the web-based system. It asked if this could be taken forward for consideration by individual CCGs and if an indicative stance could be provided at the next meeting.

There was some discussion about the provenance of the resource pack and the lack of involvement of other CCGs in its development. To assist the deliberation process, the CRD intervention team offered to undertake an independent and systematic appraisal of the evidence underpinning the proposed policies for MSK procedures. It was agreed that a preliminary assessment would be presented at the next meeting.

As no established local process appeared to be in place for assessing the evidence for proposed policies, the CRD team devised a simple process to appraise the 14 policies not included in the regional VBCP list. Figure 2 illustrates the process. We searched DARE, HTA and CDSR for potentially relevant systematic reviews and conducted web searches to identify relevant NICE or national specialty guidance. Taking each policy in turn we asked the following questions:

• Is the proposed policy or change based on NICE or nationally recognised specialty guidance?

• Is the guidance up to date?

• Does the wording of the proposed policy or change match current evidence?

FIGURE 2.

Using evidence to support commissioning decisions.

A one-page summary of the CRD preliminary assessments was presented to the VBCP Implementation Group at their next meeting. For clarity, we used a traffic light system to indicate the extent to which each proposed policy was supported by guidance and/or evidence from systematic reviews. Nine policies were rated green. This rating indicated that these were supported by national guidance recommendations and/or good-quality evidence from systematic reviews. Five policies were rated amber. An amber rating indicated that there was no explicit national guideline recommendation but that proposed policy reflected current evidence (low- or moderate-quality evidence). None of the proposed policies received a red rating. A red rating would have been used for any proposed policy that contradicted national guidance and/or was not supported by evidence from good-quality systematic reviews.

The preliminary assessments were well received by the CCG and the ‘York review’ was deemed to have provided reassurance. There appeared to be a consensus that moves to include these policies should be part of the natural progression of the regional VBCP list. The CCG also felt that the inclusion of the additional policies could potentially assist in waiting list management.

Individual CCGs would need to ratify the revised VBCP list. At the meeting there was recognition that the project group represented only seven of the 14 CCGs that would need to ratify the revised list if it was to be adopted. As a similar implementation group existed for the other seven CCGs, individual CCG ratification would be sought via that panel. The proposed revisions would also be sent to a geographically distant CCG that had been invited to join the group but could not attend. The team prepared a briefing including the traffic light indications and process flow diagram for circulation with the revised VBCP list.

At this point, moves to merge the two project groups into one also began. This revised grouping met in April 2015. The sign-off for the incorporation of the proposed policies into the regional VBCP list of commissioning intentions for 2016 occurred at the October 2015 meeting.

Symbolic use of research evidence: self-care

In 2013, the National Collaboration for Integrated Care and Support committed to support a number of local integration pioneers that would test new models of commissioning, organising and delivering integrated care and support at scale and pace. 56

Clinical Commissioning Group A1 was part of a successful pioneer bid that planned to implement a comprehensive 5-year programme promoting self-care through all health, care and community services. The local programme described as complex and transformational, aims to deliver self-care as the accepted norm of practice across the whole system. The programme has three objectives: (1) to shift the culture from helping the public to helping the public to help themselves, (2) to help staff to support local people’s ability to better manage their long-term conditions and day-to-day lives and (3) to reduce over-reliance on statutory services. Four key performance indicators were to provide a focus for measuring the impact of the programme. These were:

1. proportion of people who use services who have control over their daily life

2. proportion of people feeling supported to manage their condition

3. proportion of pregnant woman smoking at time of delivery

4. unplanned hospitalisation for chronic ambulatory care sensitive conditions.

The programme was in its early stages when the intervention phase started and the A1 named contact asked for help in ‘providing evidence to support or contradict some of the assumptions we are making’ and whether or not the team could provide a ‘quick and dirty’ appraisal of the evidence relating to the following questions as ‘quickly and as briefly’ as possible?

• Does self-care improve well-being?

• Does self-care improve health outcomes?

• Does self-care reduce demand on unplanned health services?

• Does self-care reduce demand on unplanned social services?

• What are the strategies for encouraging self-care among staff and the public?

Answers to the first four questions were deemed to be most crucial to help build up a case for change and to develop a vision. The CCG also highlighted a King’s Fund report that it felt would give an idea of what it was trying to achieve by involving social care and the third-sector agencies as well. 57

It also forwarded a Nesta report, ‘The Business Case for People Powered Health’,58 that it thought would probably help the evidence appraisal.

An initial search of DARE, NHS EED, HTA and CDSR and the websites of the Health Foundation, The King’s Fund and Nesta revealed a large number of potentially relevant reports, systematic reviews and economic evaluations. As a quick (10-day) response was requested, we opted to compile a short evidence note based on an appraisal of two overview reports, one from the Health Foundation and one from the Nesta report.

Our two-page summary highlighted that both reports made some attempt to systematically identify and appraise relevant evidence, although neither adopted a very rigorous or reproducible approach. Overall, it seems that there is reasonable evidence that self-management support and related interventions can produce improvements in outcomes for patients with long-term conditions (including most of the outcomes specified by the CCG). However, there is a lot of uncertainty around the magnitude of benefits, the cost of interventions, and which patient/population groups would benefit most. Detail about specific self-management interventions, their delivery settings and what was actually implemented was also lacking.

We suggested that rather than adopting a whole-systems approach to self-care from the outset, it may be more beneficial to focus on the groups and conditions that would benefit most. Priority should then be given to identifying the self-care interventions most likely to be effective in these groups and to considering ways of overcoming the documented barriers to implementation. The offer was made to discuss how best to further interrogate the evidence base once the CCG and Operating Group had had an opportunity to digest and discuss this initial sift.

Although the initial request can be viewed as using evidence symbolically to lend weight to an existing course of action, much of what followed in terms of request was more conceptual in nature. The CCG made a series of requests, which, although not directly linked to discrete decisions or actions, could be seen to influence their understanding and thinking on how best to deliver self-care.

We were aware that self-care for chronic obstructive pulmonary disease (COPD) was an area of interest for other participating CCGs (B1–3). Following on from this initial sift and further discussions with the A1 named contact about priorities, we offered to produce an evidence briefing focused on self-care support for people with COPD (separate work on loneliness and social isolation also emerged from these discussions). We suggested a scope that looked at multicomponent interventions (including elements such as education, telephone support and action plans) and pulmonary rehabilitation. At a face-to-face meeting this scope was agreed. The A1 named contact also asked that, rather than using our current briefing format, could we instead consider a simpler summary format to aid group discussion. An infographic or pictorial representation (i.e. smiley faces) was requested, but we offered instead to produce a clearer one-page evidence summary as part of the briefing.

The evidence briefing was circulated to all participating CCGs in July 2014 with a headline message that there was consistent evidence that multicomponent interventions reduce respiratory-related hospital admissions and improve quality of life for people with COPD.

In November 2014, the A1 CCG named contact got in touch again to say that the Pioneer Programme had been generating interest among staff and the public in self-management through a series of workshops titled ‘Changing the Conversation’. He stressed that the programme had a wider focus than the management of long-term conditions and was aiming to encompass a spectrum of activities from promoting healthy lifestyles, to expert patient/shared decision-making through to self-care interventions that could be stratified according to population group. The named contact said that he was struggling to formulate a clear research question and was not sure what to do next in terms of giving patients the confidence and skills to effectively self-manage their physical, mental and social health issues. The CCG was aware of expert patient courses, but said that it could really do with an evidence-based steer on what sorts of programmes or structures it might commission to help people to manage their own care. The named contact stressed that the CCG wanted to avoid investing badly, but was especially interested in anything that could be ‘community led’.

One topic, the effectiveness of mobile telephone applications (apps), was identified as urgent, as the programme group were considering whether or not to commission an app and were meeting with developers the following week. We produced an evidence note based on three systematic reviews and one rapid scope of the literature. Our one-page summary highlighted that despite growing popularity and availability, much of the available evidence is small scale and focuses on development, user testing and feasibility, and that evidence is lacking on the effects of mobile telephone apps on health-related outcomes. After the meeting with the developers, the A1 named contact indicated that they had decided not to pursue mobile telephone apps as an intervention option.

After an initial sift of the evidence base, we suggested that rather than producing one large briefing, we would offer to scope the available evidence under the following broad themes:

• self-management support for long-term conditions

• provision of education and supportive (lay-led) interventions to increase patients’ skills and confidence in managing their own health

• self-care interventions targeting frail elderly populations

• self-care interventions generally

• interventions that promote shared decision-making.

The A1 CCG named contact indicated that the staff were planning to have a brainstorming session to consider all the interventions possible. He said that they had learned from our advice regarding mobile telephone apps and were going to consider how to target certain interventions to certain populations or parts of the system. He also mentioned that they had been running workshops to introduce the concept of supported self-management to GP staff and then introduce some of the skills needed by staff to promote it. They were considering using action learning sets of keen staff who wish to implement their learning and need support in doing so and thus would be interested in the evidence for this approach (or for others) that would help create an environment and culture that supports self-care.

The first briefing we produced in December 2014 focused on lay-led education programmes and was based on two systematic reviews and a scoping review. We highlighted that the evidence suggests lay-led self-care education leads to small, short-term improvements in self-efficacy, self-rated health, cognitive symptom management and frequency of exercise, but that there was no evidence for improved health-related quality of life, or reduced primary care and emergency department visits. On the last point we were asked to clarify if this meant that no evidence of effect or an absence of evidence. We clarified that it was the former, but with the caveat that participants were relatively healthy/well managed at the outset, so it was possible that differences at 6–12 months would be less likely.

Supplementary comments were made in relation to the suggestion that men may want different things to women and that the programme should consider this, or it may inadvertently widen health inequalities. The public health consultant thought that it was really important to ensure that the programme does not widen the inequality gap, although she did not think that it was targeting according to need currently.

The nature of the study participants was also considered. The public health consultant noted that the underlying message appears to be one of careful targeting. The recruits to the studies were already self-reporting as being in good health – and so she asked if those who reported less good health benefited more or less. She was keen to ensure that they did not just end up recruiting the worried well. The limited but potentially positive evidence in relation to health champions and similar roles was also noted and it was mentioned that this was part of the Every Contact is a Health Improvement Contact work from front-line staff and being evaluated as part of an AHSN bid and that there may be some potential to scale.

We mentioned that we have not been able to identify any relevant evidence on action learning sets but have signposted a ‘how to’ toolkit from the Faculty of Public Health which may be helpful. The named contact asked if we could revisit action learning sets when we looked at staff-orientated interventions.

The named contact also mentioned that they were likely to adopt some interventions when there was no evidence one way or the other and could the group call on our help in evaluating them? We said that we could provide advice on what to measure and we may want to have a separate meeting focused on that.

The next Evidence Note produced in January 2015 focused on interventions to promote shared decision-making. Based on five systematic reviews and one overview of reviews, it suggested that, if tailored, appropriately shared decision-making can have beneficial effects on patient-centred outcomes. We offered to look at some of the more successful interventions in more depth if the group wished.

We started work on an evidence briefing on interventions to support self-management in people with long-term conditions, and were asked to present key messages at a forthcoming Programme Operating Group development session. Two NIHR HSDR-funded reviews formed the basis of the briefing. 59,60 The Reducing Care Utilisation through Self-management Interventions (RECURSIVE) review focused on the effect of self-management on health services utilisation and costs; the Practical systematic Review of Self-Management Support for long-term conditions (PRISMS) review summarised the key components of self-management and looked at issues around implementation. The named contact circulated the PRISMS review to the group while the CRD team were preparing the briefing.

At the meeting, further clarification around who does and does not engage with self-management programmes was requested as well as more detail on the barriers to and facilitators of patient participation. Evidence relating to self-management in a social care context was also sought. The presentation with its long-term conditions focus also generated quite a bit of discussion/concern within the wider programme group around whether or not there was to be a reassessment of the scope of the Pioneer project. Was it now going to focus solely on long-term conditions, or would the goal remain self-care across the full spectrum of public experiences from those who have limited contact with services to those with regular and increasing contact? It was emphasised that the project remained committed to the latter, while also recognising that there is potential overlap with other (e.g. Change4Life)61 initiatives.

A marketing company also presented ideas for the Pioneer project at the meeting and the A1 named contact asked about the effectiveness of public health mass media campaigns. Post event, the CRD researcher responded via e-mail to say that much of the evidence relates to smoking cessation-type interventions and, although there appears to be reasonable evidence for raising awareness, it was decidedly mixed for changing individual behaviours. At this meeting the named contact again mentioned how the CCG much preferred the shorter format for briefings.

Prior to a ‘shaping self-care’ event in March, the research team circulated the final self-care evidence briefing on patient-centred consultations, which we were informed were being increasingly advocated by NHS England. We highlighted that there is consistent evidence that most interventions promoting patient-centred approaches lead to improvements in the patient-centeredness of consultations and that investment in training and skills development for health professionals appears key.

After the intervention phase was complete, a local public health consultant informed the CRD team that he had in September 2015 been asked to revisit and summarise all the self-care briefings produced for the CCG. They did this in an informal presentation to around 20 colleagues from the CCG and Programme Operating Group. The presentation highlighted the key messages from each briefing and included pictorial representations for value for money, reduced admissions to emergency care, patient satisfaction, reduction in inequalities and the quality of evidence. Four areas for future focus were also proposed, namely (1) the staff culture, (2) patient choice, (3) collaborative action planning and (4) further exploration of COPD self-care intervention options.

Response rates

In total, 123 questionnaires were returned at baseline (A = 37; B = 54; C = 32), giving a response rate of 68%. Of these, 101 were completed, 13 were deemed to be incomplete (one section or fewer completed) and nine were from individuals declining to participate or indicating that they had departed the CCG.

At the 1-year follow-up, 76 questionnaires were returned (A = 23; B = 28; C = 25), giving a response rate of 44%. Of these, 71 were completed, two were deemed to be incomplete (one section or fewer completed) and three were from individuals declining to participate or indicating that they had departed the CCG.

Characteristics of respondents

Survey respondents reported holding a range of roles within the CCGs (Table 3). Most respondents were highly qualified, but only a minority reported having had prior experience in commissioning or undertaking research (Table 4). Sites with a lower response rate had a higher proportion of clinically qualified respondents [× 2 (2, n = 53) = 6.15; p = 0.05], but except for this difference, there were no significant differences in the characteristics of respondents receiving the three interventions.

Missing data

The proportions of missing data at baseline and follow-up for the variables measuring capacity to use research in decision-making and intention to use behaviour are presented in Table 5. Individuals with missing data did not differ significantly in scores from those for whom complete data existed.

Original and imputed means for the main variables used in the analysis of capacity for and intention to use are presented in Table 6. As can be seen, the original and imputed means are similar. ANOVA of means in the original and imputed datasets reveal no significant differences.

Benchmarking against the national picture

Section 1 of the survey instrument was used to collect benchmarking data from other CCGs across England (see Chapter 2 and Appendix 1). The most senior manager (Chief Operating Officer or Chief Clinical Officer) of each CCG was contacted and asked to consult with colleagues and complete the instrument on behalf of their CCG. At baseline we received usable responses from 79 CCGs (a response rate of 39%) and 1 year later at follow-up, we received usable responses from 31 CCGs (a response rate of 15%).

Table 7 illustrates that mean total scores for CCGs were in the area of some capacity to make use of research, but unlikely to be well equipped to do so or to do so often. The total score (overall capacity) increased marginally but not significantly over the year. With the exception of the ability to ‘adapt’ research through summarising in a more user-friendly way [baseline Canadian Health Services Research Foundation (CHSRF) M = 3.07, SD = 0.65; 1 year later M = 3.57, SD = 0.58; t(13) = –2.7; p = 0.02] no other significant differences were observed. However, it is important to be cautious in interpreting this,62 as our 1-year follow-up rates were very low and a difference of this magnitude is unlikely to be behaviourally significant.

Did the evidence briefing service improve Clinical Commissioning Groups’ ability to acquire, assess, adapt and apply research evidence to support decision-making?

We examined the hypothesis (Table 8) that CCGs would differ in their capacity to acquire, assess, adapt and apply research evidence to support decision-making as a result of receiving one of the interventions.

Overall capacity to acquire, assess, adapt and apply research evidence to support decision-making

The total capacity to acquire, assess, adapt and apply research evidence to support decision-making appeared to improve slightly over time, both among our national survey (see Table 8) and irrespective of the presence of any intervention (see Table 9 and Figure 3). The main effect of time in the factorial ANOVA yielded an F-ratio of F(1,127) = 4.49; p < 0.05, ηp2 0.034, indicating a significant difference over time in all three groups of CCGs’ total capacity to acquire, assess, adapt and apply research evidence to support decision-making. The main effect of the evidence briefing service received yielded an F-ratio of F(2,127) = 0.77; p > 0.5, ηp2 0.012. The interaction of time and the intervention was also not significant yielding an F-ratio of F(2,127) = 0.213; p > 0.05, ηp2 0.003. Exposure to the intervention had no significant effect on perceived CCG capacity.

FIGURE 3.

Total capacity to acquire, assess, adapt and apply research evidence to support decision-making. CHSRF, Canadian Health Services Research Foundation.

Impact of the evidence briefing service on Clinical Commissioning Groups’ capacity to acquire, assess, adapt and apply research evidence to support decision-making

Although there was no summary effect on capacity, we nonetheless hypothesised that the interventions may have had differential effects on different aspects of capacity.

Acquiring: capacity to look for research in the right places

Clinical Commissioning Groups’ perceived capacity to look for research in the right places appeared to improve over time (Figure 4). The main effect of time yielded an F-ratio of F(1,127) = 4.76 (p < 0.05, ηp2 0.036), indicating a statistically significant improvement over time irrespective of any intervention. The main effect of the evidence briefing service received yielded an F-ratio of F(2,127) = 0.09 (p > 0.5, ηp2 0.01). The interaction of time and evidence briefing service was also not significant, yielding an F-ratio of F(2,127) = 0.05 (p > 0.05, ηp2 0.01), indicating no statistically significant benefit from the form of intervention received.

FIGURE 4.

‘Acquire’: capacity to look for research in the right places to support decision-making. CHSRF, Canadian Health Services Research Foundation.

Assessing: capacity to tell if research is relevant and applicable

There was an apparent increase in the capacity to determine relevance of research across the intervention groups (Figure 5). The main effect of time yielded an F-ratio of F(1,127) = 7.62 (p < 0.05, ηp2 0.06), indicating that all three groups of CCGs had a statistically significant improvement in their perceived ability to acquire research evidence relevant to decision-making. The main effect of the evidence briefing service received yielded an F-ratio of F(2,127) = 0.9 (p > 0.5, ηp2 0.01). The interaction of time and the evidence briefing service was also not significant, yielding an F-ratio of F(2,127) = 0.82 (p > 0.05, ηp2 0.01), indicating that the intervention had not contributed to CCGs’ perceived improvement in their ability to identify relevant research.

FIGURE 5.

‘Assess’: capacity to tell if research is relevant and applicable to support decision-making. CHSRF, Canadian Health Services Research Foundation.

Adapt: capacity for summarising results in a user-friendly way

There appeared to be a small increase in the capacity of CCGs to summarise research findings and adapt them to decision-making (Figure 6). The main effect of time in ANOVA yielded an F-ratio of F(1,127) = 5.46 (p < 0.05, ηp2 0.04), indicating that this improvement was statistically significant. The main effect of the evidence briefing service received yielded an F-ratio of F(2,127) = 2.62 (p > 0.5, ηp2 0.04). The interaction of time and the evidence briefing service was also not significant, yielding an F-ratio of F(2,127) = 0.52 (p > 0.05, ηp2 0.01), indicating that the evidence briefing service had not contributed to CCGs’ perceived improvement in their ability to summarise and adapt research to their own decisions.

FIGURE 6.

‘Adapt’: capacity for summarising results in a user-friendly way to support decision-making (means and 95% CIs, pre and post intervention). CHSRF, Canadian Health Services Research Foundation.

Did the evidence briefing service improve Clinical Commissioning Groups’ intentions to use research evidence to support decision-making?

As with the effect of the evidence briefing service on capacity to use research for decision-making, we also wanted to examine the effect on CCG’s collective intention to use research evidence for decision-making.

Attitude towards use of research in decision-making was the strongest of these dimensions, and perceived behavioural control the weakest (Table 9). All intervention groups had apparent small and non-statistically significant declines in almost all of these theory of planned behaviour dimensions from baseline to follow-up.

Attitudes

All CCGS, regardless of intervention received, appeared slightly less positive towards using research in their decision-making (Figure 7). The main effect of time yielded an F-ratio of F(1,127) = 4.28 (p < 0.05, ηp2 0.01), indicating a statistically significant difference over time in each of the three groups of CCGs’ attitudes towards research evidence to support decision-making. The main effect of the evidence briefing service received yielded an F-ratio of F(2,127) = 1.55 (p > 0.5, ηp2 0.02). The interaction of time and the evidence briefing service was also not significant, yielding an F-ratio of F(2,127) = 0.72 (p > 0.05, ηp2 0.01). These together confirm the initial impression of a decline in attitude towards use of research.

FIGURE 7.

Theory of planned behaviour: CCGs’ attitudes towards research (means and 95% CIs, pre and post intervention). TPB, theory of planned behaviour.

How do the Clinical Commissioning Groups view their contact with research and researchers?

At both baseline and follow-up, participants receiving all three interventions were asked questions that assessed the quality and quantity of contact with researchers, specifically:

• perceptions of equal status between CCG members and the researchers they encounter

• Clinical Commissioning Group support for that contact (both important aspects of contact quality)

• Clinical Commissioning Groups and researchers seeing themselves as part of an overarching group with common goals

• researchers in general

• whether or not contact with researchers is equally useful for both parties, more important for researchers, or for CCGs.

These measures were used to assess whether or not the evidence briefing service improves contact between CCGs and researchers and/or results in more positive perceptions of researchers in general. Strength of identification as a CCG was also assessed, as a potential moderator of any impact of the intervention, but the small sample size makes it difficult to analyse or interpret this measure as a moderator.

Clinical Commissioning Groups’ professional goals

We examined the hypothesis that the evidence briefing service would improve the perception that communication with researchers helps CCGs to achieve their professional goals. The interaction between intervention and time approached significance [F(1,47) = 2.99; p = 0.06].

Post hoc analyses demonstrate that communication with researchers is perceived as more valuable in achieving CCG goals at outcome (M = 4.87) than at baseline (M = 2.40) in those receiving intervention C [F(1,14) = 12.08; p = 0.48] [and intervention B follow-up: M = 5.05; baseline: M = 2.38; F(1,20) = 25.60; p = 0.0005]. In contrast, there was no change in attitude towards researchers between baseline (M = 4.29) and outcome (M = 4.85) with those receiving the evidence briefing service (intervention A) [F(1,13) = 0.59; p = 0.48]. In summary, the hypothesis was not upheld: intervention did not increase the perception that communication with researchers helps CCGs to achieve their professional goals.

Researchers’ professional goals

We examined the hypothesis that the evidence briefing service would improve managers’ perception that communication with CCGs helps researchers to achieve their professional goals. The interaction between intervention and time was not significant [F(1,47) = 2.45; p = 0.10], indicating that the intervention had no statistically significant impact on achieving their professional goals.

Clinical Commissioning Group and researcher goals

We examined the hypothesis that the evidence briefing service would improve managers’ perception that communication between the two parties helps both researchers and CCGs to achieve their professional goals. The interaction between intervention and time was not significant [F(1,47) = 2.37; p = 0.11], indicating that the intervention had no statistically significant impact on achieving common goals.

Summary

Exposure to the evidence briefing service did not increase perceptions that communication between CCGs and researchers helped CCGs achieve professional goals (or indeed, researchers’ goals). Nor did it lead to increases in what were already positive perceptions of researchers in general. Exposure to the evidence briefing service did not increase perceptions of the quality or quantity of contact between CCGs and researchers.

Table 13 presents a summary of the results of all the hypotheses tested in this chapter.

Exploring uptake and use of evidence in decision-making in the A1 Clinical Commissioning Group

Exploring uptake and use of evidence in decision-making in Clinical Commissioning Group A2

Exploring uptake and use of evidence in decision-making in the B1 Clinical Commissioning Group

Statement of principal findings

Over the course of the study the evidence briefing service addressed 24 topics raised by participating CCGs (see Chapter 3). Requests for evidence briefings served different purposes. The majority of requests were focused on options for the delivery and organisation of a range of services and possible interventions to support the self-management of long-term conditions. Most of the requests could be categorised as conceptual, not directly linked to discrete decisions or actions but often intended to gain knowledge and awareness of possible options for future actions. Symbolic drivers of use of research (i.e. to justify or support pre-existing intentions or actions) were less frequent and included a pre-existing decision to close a walk-in centre and to lend weight to a major initiative to promote self-care already under way. Instrumental use was linked to explicit disinvestment processes. There were no instances in which requests for evidence could be viewed as representing an imposed use of research.

Our primary research question asked whether or not access to a demand-led evidence briefing service would improve the uptake and use of research evidence by NHS commissioners, compared with less intensive and less targeted alternatives. In terms of the primary outcome measure, the evidence briefing service was not associated with increases in CCG capacity to acquire, assess, adapt and apply research evidence to support decision-making.

Regardless of the intervention received, at baseline participating CCGs indicated that they lacked a consistent approach to their research-seeking behaviours and their capacity to acquire research remained the same at follow-up. At baseline, CCGs were non-committal (neither agreeing nor disagreeing) about whether or not they had the capacity to assess the quality, reliability and applicability of research for use in decision-making. This perception remained unchanged at follow-up. There was also no change between baseline and follow-up on perceptions of CCGs’ capacity to adapt and summarise research results for use in decision-making; there was neither agreement nor disagreement that CCGs had the capacity to do so. Finally, individuals’ perceptions that their CCG did not have systems and processes in place to apply research routinely also remained unchanged.

A secondary research question sought to establish whether or not contact between researchers and NHS commissioners would increase use of research evidence. Exposure to the evidence briefing service did not increase perceptions of the quality or quantity of contact between CCGs and researchers, nor did it lead to perceived improvements in institutional (CCG) support for contact between commissioners and researchers. Exposure did not increase perceptions that communication between CCGs and researchers helped commissioners to achieve professional goals, nor did it increase what were already positive perceptions of researchers in general.

Exposure to the evidence briefing service did not appear to have any impact on individuals’ intentions to use research evidence in decision-making or their perceptions of a shift in collective CCG norms towards the use of research for decision-making. Regardless of intervention received, these measures were positively orientated at baseline and were sustained at follow-up.

Our final secondary research question asked whether or not evidence briefings tailored to specific local contexts could inform decision-making in other CCGs. Our ability to answer this question was undermined by a lack of recorded documentary evidence of research use (a finding in itself) across participating sites. With a few exceptions, most discussions between commissioners and the evidence briefing team were informal and rarely involved minuted meetings or formal gatherings of CCG staff. This lack of a visible audit trail or indeed for the onward distribution and cascade of generated outputs makes us dependent on self-report and/or observed use for impact. Therefore, it is difficult to determine the extent to which the evidence briefings produced had wider value across participating CCGs and in those outside the study.

Strengths and limitations

This quantitative component of the evaluation was in many ways the most challenging aspect of the study. We were reliant on the quality of the sampling frames provided by (1) CCG cases themselves and (2) nationally, in the form of contact data for each CCG. We found that information provided by CCGs, and especially that sourced for the national benchmarking component of the study, was sometimes inaccurate (spelling mistakes in e-mail addresses and surnames), incomplete (absent e-mail addresses or contact numbers) or included individuals who no longer worked at a CCG. As such, each CCG had to be contacted to obtain, check and recheck the contact details of staff provided. In a related limitation, we asked individuals to complete the national survey on behalf of their CCG and in consultation with colleagues, but, in a rapidly changing landscape, we cannot rule out the possibility that different individuals completed the survey at baseline and follow-up.

The respective baseline and follow-up response rates of 68% and 44% are not unreasonable given the number of competing requests for information CCGs are routinely faced with. For example, our response rates compare favourably with annual surveys conducted by the Health Foundation and The King’s Fund over the same time period,71,72 and with a contemporaneous Canadian randomised evaluation of the effects of an evidence service on policy-makers’ use of research evidence that failed to recruit. 37,73 However, we acknowledge that we experienced considerable attrition in the percentage of participants in our study who completed both baseline and follow-up surveys. In the study case sites the percentage of individuals completing both surveys ranged from ≈60% for those receiving intervention A to ≈30% in the CCGs who were allocated to receive intervention C, the non-responsive version of the service. As the turnover of staff employed at participating CCGs was relatively stable over the course of the study, there may be a degree of selection bias apparent in our study.

We utilised an 87-item questionnaire to collect data relevant to the primary outcome and, although all responses were on short scales (none required any written responses), piloting estimated that it would take participants up to 45 minutes to complete. We employed a range of factors to increase the odds of response including pre-notification, follow-up contact, online and postal formats, reminder copies, mention of an obligation to respond and university sponsorship. 74 However, we are aware that both shorter questionnaires and financial incentives are also associated with increased response rates. 74 In this instance, it may be that the perceived return for time invested of access to a funded evidence briefing service either immediately or after the intervention phase was complete (the offer made to participants in the ‘control’ standard service intervention C) was deemed inadequate compensation by some participants. The CCGs allocated to intervention C had expressed initial enthusiasm for participation. However, the lack of any immediate return from, or a sufficient relationship with, the evidence briefing service over the course of the study may go some way to explaining why CCGs allocated to the ‘control’ intervention C had the lowest response rate.

Survey length may also have contributed to the lack of completeness in the data collected. This lack of completeness necessitated the use of multiple imputation to strengthen analysis. 45,46 In line with best practice in multiple imputation, comparison with the non-imputed data revealed similar means and distributions.

Taken together, these limitations mean that we have been cautious in our interpretation of any apparent impact of the evidence briefing service on the primary outcome measures. Indeed, we have been careful to avoid the pitfalls of p-values in assessing whether this study provides evidence ‘for’ or ‘against’ rejection of the null hypothesis. 62 Although the statistical tests applied have generated some apparent statistical differences, beyond those that we would have expected to see by chance, our approach to interpretation has, we think, injected appropriate caution in interpreting the real-world significance of what was observed. Although not explicitly stated in the original protocol, it would be reasonable to consider a shift of at least one point on any Likert scale as indicative of impact. So although, for example, we observed a statistically significant decline in attitudes towards research use at follow-up, the magnitude of this shift (no shift on the scale) is unlikely to be behaviourally significant. The benchmark of a national sample of non-intervention CCGs also helps assess the theoretical significance of what was observed. The fact that CCGs receiving the ‘control’ standard service intervention C and the national benchmarking sample have all ‘improved’ (capacity) suggests a degree of maturation and perhaps something of a ‘rising tide’75 phenomenon at play. In other words, a CCG may be making negligible gains in capacity as it becomes more established over time.

An original aim was to employ documentary analysis to identify and understand the ways in which briefings generated by the service were taken up and considered in the decision-making processes of each participating CCG. Our development work undertaken as part of the NIHR CLAHRC for Leeds, York and Bradford (admittedly with PCTs) had suggested that this would be a feasible approach to take. However, early analysis undertaken to trace evidence briefings generated in the intervention phase revealed, with few exceptions, a lack of recorded evidence of use. Most discussions between contacts in CCGs and the evidence briefing team were informal and rarely involved minuted meetings or formal gatherings of CCG staff. Indeed, we were often responding to requests from one, two or three named individuals who would be leading a piece of work or clinical area on behalf of the CCG as a whole. As such, analysis of records supporting the more formal executive and governing body meetings provided little information about sources used or about the decision-making process itself. The ‘unseen and informal spaces’76 of decision-making processes, the small numbers of staff involved and the reality that no audit trail existed for sources used during these processes meant that there was little or no ‘traceability’77 of use of evidence briefings at an organisational level. A similar lack of traceability exists for the dissemination of evidence briefings to other participating CCGs. We know when and to whom content was distributed, but are reliant on self-report and so we know little of what happened or how content was used (if at all) thereafter. Our experience aligns well with others who have faced similar challenges in identifying whether or not systematic reviews are used and the extent to which they add value to decision-making processes in public health. 77

Delivery of the evidence briefing service

In this study, we sought to make best use of outputs from the NIHR Systematic Reviews Programme and specifically those in relation to the CRD’s core work programme. The CRD core funding supported the provision of the DARE, NHS EED, and HTA and PROSPERO databases. As mentioned in Chapter 3, NIHR funding for the CRD’s core work programme ceased during the course of the study and with it so did the availability of a continuously updated single source for systematic reviews and economic evaluations. The ability to acquire and assess research-based knowledge of this type can be a significant undertaking and although systematic reviews continue to be indexed on a variety of database platforms, no such resource now exists for economic evaluations. Although the evidence briefing team were able to utilise existing CRD search and retrieval capacity to ensure the delivery of study commitments, the lack of a continuously updated single resource to draw on does have funding implications for future service provision of this type. It is worth noting that not all questions could be addressed through existing systematic reviews. A feature of many of the outputs produced was an absence of synthesised evidence; this was particularly the case for those that focused on summarising evidence for proposed new models of care. As such, search and retrieval activity was actually greater for topics where we sought to establish ‘known unknowns’ than for those topics with a larger and already synthesised evidence base.

When we conceived the evidence briefing service, the evidence-informed rationale was that addressing real decisions or problems in collaboration with those directly affected should mean that research evidence was more likely to be used and inform decision-making. Then, in the NIHR CLAHRC for Leeds, York and Bradford, the service was an adjunct to a larger implementation programme of research. In this study, the evidence briefing service as constituted represented a resource-intensive intervention. From the outset, we sought to add insight as to how much added value the service would offer over alternative or more basic approaches. Although no costs associated with searching, information support and document retrieval or with publication and dissemination of the evidence briefings were included in the study application, 1.5 full-time equivalent experienced researchers and a significant proportion of the principal investigator’s time were committed to its delivery. There was sustained engagement with the service by individuals in the CCGs receiving intervention A and because we employed a degree of flexibility in the service delivered (employing a combination of full evidence briefings and shorter more exploratory evidence notes in response to questions raised) we were able to deliver a number of outputs beyond the estimate made in our original application. However, the nature of requests we received were largely conceptual and the impact of evidence briefings on more explicit instrumental decision-making processes was limited. Although we recognise that conceptual use of research is an entirely appropriate and necessary goal in itself, we question whether or not supporting conceptual use represents a sufficient level of impact to justify a resource-intensive intervention of this type.

Reflections on delivery

It has long been understood that real-world decision-making reflects a complex interaction between economic, political and social factors, different sources of knowledge (of which research evidence is one) and the beliefs and ideologies of those making the decisions. 10,11,78 This study has provided further insight as to how and where services packaging evidence derived from systematic reviews may most efficiently be deployed to impact on decision-making processes in a commissioning context.

Work undertaken to support decisions around the inclusion of 14 MSK procedures designated as low value into a regional list of interventions that CCGs will not normally fund had the most traceable impact on decision-making. Participating CCGs appeared to value the transparency that the evidence briefing service brought to the process. The existing regional value-based policy list predated the creation of CCGs but the process for assessing the evidence for new policies did not appear to have been transferred into the new system. Indeed, the proposed MSK polices had been compiled by one CCG using a ‘copy, paste and adapt’48 approach from existing policies identified at other CCGs across the country. This led other group members to question their provenance. The offer to undertake an independent and systematic appraisal of the evidence assisted the collective deliberation process, not least by providing reassurance to the representatives of the other CCGs.

Although our intention was that the evidence briefing activity was demand led, there is a consistent message from CCG informants that they would have valued more of the systematic and transparent push approach employed in the low-value work to identify interventions and ways of working that should be funded or not funded. However, we also need to recognise that the nature of decision-making and the processes employed in the context of these low-value policies was very different to those experienced elsewhere in the study. The low-value policy work represented a meso (regional)-level process with CCGs coming together to make decisions collectively. This process had a clear objective, namely to establish clear region-wide policies across CCGs relating to interventions of no or low clinical benefit; a process that needed to be both transparent and defendable. Further clarification on how best to identify and support this type of meso-level commissioning activity may be warranted.

Most other requests from CCGs could be categorised as conceptual, that is, not directly linked to discrete decisions or actions, but intended to provide knowledge and awareness of possible options. The issues raised were iterative and evolving in nature without obvious end points or decisions,79 and our role provided knowledge and awareness of possible options for future actions. This is perhaps best exemplified by the work undertaken around interventions to support the implementation of self-management. Our experience mirrored earlier accounts describing commissioning services for people with long-term conditions as a long drawn-out process. 80 The process of producing the series of related briefings involved a range of discussions and activities with a range of individuals and stakeholders both within and outside the CCG. The time and effort involved appeared to be disproportionate to the likely impact on the local commissioning decisions we sought to support. Even after the intervention phase was complete, deliberations on how best to act were still ongoing and needed additional input from a trusted local source (a senior member of the local public health team) to summarise and contextualise the already summarised information. It is likely that many of the self-management issues that we were asked to address would have been salient and relevant to CCGs in other settings across the country. And it could be argued that this would apply to most of the briefings produced as part of this study. Passive dissemination of the social prescribing briefing has generated considerable interest from CCGs and Health and Wellbeing Boards outside the study. Given the absence of evidence of effect, further advice on how to evaluate has been sought from those either currently providing or considering introducing social prescribing programmes.

Given the large resource requirement and the particularity of process and unpredictable timing of decision-making in individual commissioning organisations, it may be better to invest far more in identifying commissioning priorities and uncertainties from key informants with local credibility. These could then be serviced by a centralised evidence synthesis service and less costly targeted dissemination strategies could be used to raise awareness among what appear to be receptive commissioning audiences’ options or actions in key audiences. The cases examined here suggest that this would include those members of local public health teams supporting CCGs. Targeted dissemination (similar to the approaches the CRD previously employed with the Effective Health Care and Effectiveness Matters series of bulletins) could deliver similar impacts. 15,81 Indeed, passive dissemination of the social prescribing briefing has generated considerable interest from CCGs and Health and Wellbeing Boards outside the study and the evidence briefing team have been asked for further advice on a number of these decision-making processes. Taken together, this may suggest that resource-intensive approaches to providing evidence are best employed to support instrumental decisions occurring at a meso level where impact is likely to be proportionately greater. This would also be consistent with informants’ requests for more ‘supply-side’ push (researcher-led distribution of research) alongside the demand-led (pull) access they received. The potential for impact from the targeting of tailored messages and topics at specific audiences may be of interest to the NIHR Dissemination Centre and merits further investigation.

Implications for research use

If meso-level activity may represent the best focus for resource-intensive services, we still need to consider how to systematise research use among individual CCGs. The Supporting Policy In health with Research: an Intervention Trial (SPIRIT) Action Framework (published after the intervention phase of this study was completed) hypothesises that a catalyst is required for the use of research, the response to which is determined by the capacity of the organisation to engage with available research. 82 Where there is sufficient capacity (the value placed on research, the tools and systems the organisation has to support research engagement; and the skills and knowledge of staff), a series of research engagement actions might occur that facilitate research use. The SPIRIT Action Framework82 predicts that the greater the organisational capacity, the more research engagement actions (accessing and appraising research, generating new research and interacting with researchers) will occur, which will in turn result in a greater use of research evidence.

Using the SPIRIT Action Framework to reflect on this study, we had catalysts and engagement opportunities (around the questions raised and the briefings produced), but the service as constituted did little to enhance the capacity of the organisation to use research routinely. Both baseline and follow-up data suggest that commissioners are well intentioned ad hoc users of research evidence and that they work in a setting where there is a lack of systems and processes to do this routinely. CCG informants also indicated the potential for confirmation bias in their evidence-seeking behaviours and the challenges of being confronted with an absence of reliable evidence for policies or options that they were pursuing. This suggests a knowledge and skills gap that this study has not addressed. The evidence briefing team offered training on how to acquire, assess, adapt and apply evidence to CCGs receiving intervention A or intervention B (which could have addressed these knowledge and skills gaps), but this offer was not taken up. Rather than making training a demand-led ‘offer’ it may have been better to identify the capacity for research use of each CCG at the outset, and develop a corresponding offer to each organisation that included training relevant to their current state. At the very least, this study has highlighted the importance of building organisational capacity as a component of evidence use, an area that appears to be under-researched. 83 SPIRIT is informing an ongoing evaluation of a multifaceted programme to build organisational capacity for the use of research evidence in policy and programme development in Australia. 84 Findings from this will help shed light on the value of the Framework to develop and test other interventions to build organisational capacity to use research.

Public health specialists have traditionally supported and facilitated the use of research evidence in a commissioning context. 19,20,48 Throughout this study we observed that despite its relocation, the public health specialist remained accessed and engaged with by CCGs despite being no longer central to decision-making processes. Some senior staff in participating CCGs had much prior experience of support from public health teams under previous commissioning arrangements. As the interventions followed soon after the preceding arrangements had ceased, it is perhaps no surprise that the CCG commissioning staff made use of the service offered by the CRD. Nevertheless, all the CCGs continued to place value on the knowledge and expertise of trusted ‘critical friends’85 in the shape of public health consultants. They provided a bridge between the old and new commissioning arrangements and brought valuable insights and networks from beyond the boundaries of the CCG. Although we often observed commissioners looking out and undertaking fact-finding trips to see what other CCGs around the country were doing, the same individuals were often unaware that colleagues in adjacent areas were undertaking similar work or grappling with similar questions. Public health specialists were the individuals viewed as most likely to fill this local knowledge gap and to mitigate against a general dissatisfaction with the knowledge-sharing capabilities of the formal commissioning support arrangements. Whether fair or otherwise, there was a general perception among CCG informants that the CSU lacked the necessary infrastructure and/or expertise to efficiently acquire, assess and adapt research for use in decision-making. The one-to-one transactional arrangements the CSU had with CCGs were themselves viewed as a barrier to wider knowledge sharing across the region. This danger of ‘network closure’ undermining local knowledge sharing and historically trusted relationships has been anticipated previously. 86

Wye et al. 48 have argued that researchers need to build relationships and engage with commissioners locally using commissioners’ preferred methods of conversations and stories, to find out what is wanted and how best to deliver it. In this study we had fewer face-to-face engagement opportunities than originally anticipated; this was despite case informants indicating that they would have liked more. We consistently offered to discuss priority areas and the key messages and implications arising from evidence briefings face to face, but in many instances participants found it easier to have a quick telephone or e-mail discussion with the CRD team. Our geographical distance from the intervention sites may have influenced the mode of interaction and communication, and in turn reduced the type of contact perhaps necessary to facilitate an increased use of research evidence on the part of commissioners. Although we do not discourage the cultivation of face-to-face relationships, the reality of the decision-making process is that any engagement is resource intensive and so researchers need to carefully consider how best to target those interactions that will deliver the best return. Even with proximity, somebody needs to be around or ‘in the room’ when ideas first germinate, to spot the potential catalysts to research use and to question what is the evidence for this? Why do we want to pursue this course of action? Given this, Wye et al. ’s48 suggestion that researchers cultivate relationships with local public health teams could represent the intermediary channel through which use of research by individual CCGs can be influenced. Public health staff are more likely to be ‘in the room’ and have the necessary skills and local networks to facilitate knowledge sharing within and across the commissioning landscape. The current emphasis on innovation and the development of new models of health and social care is favouring coproduction approaches to the design, commissioning and delivery of services. This shift may strengthen the intermediary role of public health. But, if this intermediary role is to be sustained, public health specialists will need to be supported and resourced to return to playing a more central role in commissioning.

Alongside capacity building and engagement, macro-level intervention is also needed to enhance research use at the level of the individual CCG. The Health and Social Care Act3 mandates CCGs in the exercise of their functions, to promote innovation in the provision of health services, promote integration and to make use (in the health service) of evidence obtained from research. Infrastructure to support the statutory duty to drive innovation at scale is under way. Fifty ‘vanguard’ sites are supported by a £200M transformation fund from NHS England. A similar commitment of significant resources has also been made via the Better Care Fund and the Prime Minister’s Challenge Fund; providing further impetus to innovation and integration between health and social care. However, whereas the current policy climate explicitly incentivises innovation and integration, there is no equivalent incentive for finding and applying research to support the many decisions required to turn this vision into a reality. The CCG Assurance Framework87 focuses on leadership, financial and performance management, planning and delegated functions, but contains no specific metrics on whether or not CCGs are fulfilling their statutory duties in respect of use of evidence obtained from research.

During the course of this study we were given the opportunity to suggest wording to sharpen that existing in the CCG Assurance Framework87 on the use of evidence derived from research. We suggested the wording ‘each CCG must, in the exercise of its functions, demonstrate the ability to acquire, assess, adapt and apply evidence obtained from research in health-service decision-making.’ This has now been incorporated into appendix 2 of the Assurance Framework operating manual. 87 However, whereas it is stipulated that CCGs must have a plan in place to address their duties in relation to promoting and supporting the conduct of research, there are no similar explicit requirements relating to the use of evidence obtained from research. If we are serious about shifting CCGs from being well intentioned but inconsistent users of research evidence, then a more explicit set of requirements may be necessary. Ideally, the incentive structure that exists for health-service innovation and integration may need to be replicated to support CCGs’ fulfilment of their statutory duties in respect of use of research under the Health and Social Care Act 2012. 3 Without this, the current ad hoc engagement with research is likely to remain.

In the current financial climate, disinvestment decisions relating to interventions of no or low clinical value are likely to remain high on the commissioning agenda. In this study, we witnessed the development of collaborative processes for considering disinvestment at the local level. A lack of organisational memory about the processes previously in place with earlier commissioning arrangements was also apparent. Despite this, practical challenges in identifying and contextualising research evidence to inform these processes remain. 55 Unlike the rigorous processes in place to inform the NICE guidance on the use of new and existing medicines, no similarly resourced infrastructure exists to support disinvestment decisions. 7 Although NICE makes ‘do not do’ recommendations publicly available, we found low awareness of these among commissioners and a notable lack of skills to systematically and transparently identify other relevant evidence that could inform disinvestment decisions. The NIHR already funds infrastructure with the skills necessary to support disinvestment activity at a local level. This includes the NIHR CLAHRCs, rapid evidence centres and HTA groups. More proactive and targeted dissemination of low-value recommendations combined where necessary with synthesis using standardised methods could enhance the ability of local commissioners to identify and then generate local policies on interventions of no or low clinical benefit.

Recommendations for research

We are conscious that our findings relate to a specific decision-making context and setting and have been generated at a time when the commissioning arrangements are rapidly evolving. Given this, further comparative evaluation and clarification of the role and value of similar demand-led evidence briefing services in other contexts and settings may be warranted. The SPIRIT Action Framework may provide a guide on which the evaluation of any future services seeking to increase the use of research in policy can be based.

Our study has revealed commissioners to be well intentioned but lacking the necessary skills and infrastructure to make use of research evidence routinely. Further research is required on the effects of interventions and strategies to build individual and organisational capacity to use research. Exploration and clarification of the potential for macro-level intervention to incentivise research use is also warranted.

Disinvestment decisions relating to interventions of no or low clinical value remain high on the commissioning agenda. No established process appears to be in place for assessing research evidence to inform the generation of local policies. Rather than have local settings developing their own distinct approaches it would seem sensible if a country-wide approach was taken to identify and then summarise the evidence for interventions of no or low clinical value. Methodological research is therefore required to establish an optimal, transparent and standardised approach that identifies and contextualises research evidence that can then be used to inform local decision-making processes.

Our study suggests that resource-intensive approaches to providing evidence may best be employed to support instrumental decision-making at a meso level. Otherwise, less resource-intensive approaches to delivering optimally packaged systematic review-derived findings should be pursued. We know that passive dissemination can represent better value in some contexts and settings, particularly when there is a single clear message and/or when the topic is known to align with known commissioning priorities and or uncertainties. Many research agencies fund or undertake engagement activities and have invested in a range of communication channels. How best to harness ‘supply-side’ infrastructure to deliver effective targeted communications remains unclear. As such, there is considerable scope for comparative evaluation of the impact of different active and targeted dissemination strategies on the uptake and use of research by commissioners and other key stakeholders.

Contributions of authors

Paul M Wilson conceived the study and was responsible for its overall direction, contributed to research design and led the evidence briefing team and the production of evidence briefings and the final report.

Kate Farley contributed to the evaluation component of the study and to research design, administered pre- and post-intervention questionnaires and managed data, conducted qualitative interviews, led the qualitative analysis and contributed to the production of the final report.

Liz Bickerdike was a member of the evidence briefing team and contributed to research design, the production of evidence briefings and the final report.

Alison Booth was a member of the evidence briefing team (replaced Duncan Chambers) and contributed to the production of evidence briefings and the final report.

Duncan Chambers was a member of the evidence briefing team and contributed to research design, the production of evidence briefings and the final report; he left the study to take up a full-time post in June 2014.

Mark Lambert provided advice to both the evaluation and intervention component throughout the study, and contributed to research design and the production of the final report.

Carl Thompson led the evaluation component of the study and contributed to research design, conducted statistical analysis of pre- and post-intervention questionnaires, conducted qualitative interviews and contributed to qualitative analysis interpretation and the production of the final report.

Rhiannon Turner contributed to the evaluation component of the study and to research design, statistical analysis of pre- and post-intervention questionnaires and the production of the final report.

Ian S Watt provided advice to both the evaluation and intervention components throughout the study and contributed to research design and the production of the final report.

Publication

Wilson PM, Farley K, Thompson C, Chambers D, Bickerdike L, Watt IS, et al. Effects of a demand-led evidence briefing service on the uptake and use of research evidence by commissioners of health services: protocol for a controlled before and after study. Implement Sci 2015;10:7.

Data sharing statement

All available data can be obtained from the corresponding author. All data will be shared in a way that safeguards the confidentiality and anonymity of respondents.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health.

Enhancing access in primary care settings flow chart

Social prescribing flow chart

Care-planning flow chart

Supporting self-management flow chart

Unplanned admissions from care homes flow chart

Lay-led self-care education flow chart

Self-care for chronic obstructive pulmonary disease flow chart

Loneliness and social isolation flow chart

Evidence to inform urgent and emergency care systems flow chart

Consolidating urgent care services flow chart