Improving the capabilities of NHS organisations to use evidence: a qualitative study of redesign projects in Clinical Commissioning Groups

Article history

The research reported in this issue of the journal was funded by the HS&DR programme or one of its preceding programmes as project number 12/5002/20. The contractual start date was in September 2013. The final report began editorial review in June 2016 and was accepted for publication in January 2017. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

John Powell works part-time (0.5 whole-time equivalent) for the National Institute for Health and Care Excellence as a Consultant Clinical Adviser in the Centre for Health Technology Evaluation. John Powell is a member of the National Institute for Health Research Health Technology Assessment and Efficacy and Mechanism Evaluation Editorial Board and was previously a member of the NIHR Journals Library Editorial Group. David Sharp was a NHS employee at the beginning of the project, and the NHS is an organisation that funds and uses research. David Sharp currently works at Optum Healthcare, an organisation that sells commercial services to the NHS.

Disclaimer

This report contains transcripts of interviews conducted in the course of the research and contains language that may offend some readers.

Copyright statement

© Queen’s Printer and Controller of HMSO 2017. This work was produced by Swan et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Introduction to the research

Sustained and successful innovation in health-care services, driven by the best available evidence, is critical to the survival of England’s NHS. 5,13 A major provider of authoritative evidence is the National Institute for Health and Care Excellence (NICE). NICE synthesises and publishes authoritative evidence that is widely considered to be ‘gold standard’. New legislation places NICE at the heart of reforms in health and social care to ensure that patients receive high-quality and cost-effective care. 14 At the same time, more stringent measures to accelerate the uptake and implementation of NICE recommendations are being proposed. Expectations are higher than ever that this evidence will drive innovation as well as lower the costs of service delivery in the NHS.

Much of the responsibility for innovation in the NHS now lies with Clinical Commissioning Groups (CCGs), which are tasked with the design and procurement of health-care services. Multiple evidence-based products are available to commissioners in making informed decisions, including NICE guidance (e.g. guidelines) and quality standards and metrics [e.g. Quality and Outcomes Framework (QOF), CCG Outcomes Indicator Set].

However, fragmentation and variation in the uptake of evidence in health-care decision-making is widely noted. 2,3,15 It is clear that the (abundant) supply of evidence-based products does not always connect to the needs of health-care decision-makers. 1,16 This is especially so when dealing with the kinds of complex, time-dependent and political decision situations faced by health-care managers. 16,17 Walshe and Rundall17 aptly note that the:

constrained, contested, and political nature of many managerial decisions, [make] it difficult for managers to apply research evidence even when it is available.

p. 44517

Thus, a major challenge for NHS commissioning groups is to proactively and strategically consider how they can be better equipped to take hold of, and use, evidence in decisions. A key challenge then, lies in ‘building high learning capacity and appropriate core competencies in NHS organisations, rather than relying on a technological fix’. 5

The imperative for NHS organisations to innovate sits alongside the need to make savings, often by adept commissioning. These objectives can be met only by improving existing services. The need for commissioning managers to use sound evidence is, therefore, greater than ever. 1 In the light of this, organisational capabilities to use evidence in their pursuit of service redesign are especially important and need to be carefully nurtured. Our study, thus, focuses on capabilities to use evidence in service redesign initiatives led by commissioning groups. It seeks to provide an important practical resource for the frontline managers and the clinical leaders of commissioning organisations who will be responsible for making some tough decisions over the coming years. The focus of our work, then, is on improving NHS commissioning organisations’ capabilities to use evidence in decision-making.

Aims and objectives

As stated in our original protocol, the aims of the research were to identify the capabilities (organisational and managerial) that NHS CCGs need to become better users of authoritative evidence. Given our focus on evidence use among social groups in organised settings, we take a knowledge-based view of organisations. 18 As Grant19 comments:

. . . organizational capability, defined as a firm’s ability to perform repeatedly a productive task which relates either directly or indirectly to a firm’s capacity for creating value through effecting the transformation of inputs into outputs.

With this in mind, the objectives of our research were to:

1. Study how authoritative evidence travels into, and is used by, different CCGs (by documenting ‘evidence journeys’).

2. Compare and contrast the travel of evidence across CCGs to understand how and why variations in evidence use occur in practice.

3. Create a toolkit to guide CCGs to better use evidence. Specifically, the tool will help CCGs to assess and improve their capabilities to use evidence. It will be built on process models of how evidence travels in the real world, rather than on abstract accounts.

Our aims and objectives have not changed, but it was apparent from our initial literature search and fieldwork that to understand authoritative evidence use we also needed to consider how commissioners use other forms of evidence too. We had also intended to examine ‘disinvestment’ and ‘investment’ decisions, but it became apparent that, in reality, these are two sides of the same coin. That is, commissioners’ decisions about what not to do occur only when deciding what to do in service design. Hence, the two cannot be practically disentangled when looking at the ways evidence travels in practice, at least in this context.

In the following sections, we outline how our research contributes to the NIHR brief and describe the commissioning landscape as our research context. We then outline our theoretical approach, drawing from ‘knowledge mobilisation’ and evidence-based management (EBMgmt) health-care literatures. 20 Although much can be learned about the ways in which evidence is used in the NHS from these literatures, we conclude that further work is needed to articulate how organisations can improve their capabilities to use evidence more effectively. 1

Response to the National Institute for Health Research Health Services and Delivery Research commissioning brief

This study was funded in response to the NIHR Health Services and Delivery Research (HSDR) programme’s call for ‘research to improve knowledge transfer and innovation in healthcare delivery and organisation’ (HSDR call number 12/5002). The call builds on the Cooksey report21 particularly the observed ‘T2 gap’ between the research output and transfer into practice. The call emphasised research that generates actionable findings to help the NHS make better use of evidence. The three main themes were:

1. research on ‘pull’ by managers (increasing the receptivity and research/innovation capacity of organisations and individual managers)

2. research on ‘push’ by researchers (making the research product/innovation more usable)

3. research on knowledge linkage and exchange (bringing service decision-makers and researchers/industry together in different ways).

The NIHR call focused specifically on the evidence used by organisations and managers to inform decisions about services, rather than on evidence informing clinical interactions, into which there had already been substantial research. 22 Notably, the call sought to build on previous work on ‘knowledge translation’, for example by the Canadian Institute of Health Research, to improve the translational work of NHS management and organisations. The call was premised on the idea that knowledge translation includes a diverse array of activities and should be understood as highly complex, not only in technical, but also in social, organisational and political terms. 6

Research context: the shifting commissioning landscape (2013–16)

Our study began in October 2013, just 6 months after CCGs took over from PCTs. CCGs are part of the government’s attempt to improve health services by bringing the commissioning closer to the community. CCGs differ from PCTs in size and composition. CCGs are much larger and are led by general practitioners (GPs), whereas health-care managers ran PCTs. Commissioning GPs must work alongside managers, consultants, nurses, patients and other experts to manage resources. These groups work together to plan, contract, monitor and transform health care. 23 NHS reforms have sought to create commissioning supported (but not led by) external commercial and not-for-profit providers, changing the relationship between managers and clinicians. Evidence use in commissioning has no doubt been influenced by new modes of working between CCGs and external providers. 1 This shift means there is a need to identify opportunities for learning with regard to developing and sustaining capabilities to use evidence in commissioning.

At last count, 211 CCGs had responsibility for about two-thirds of the NHS budget between them. CCGs can commission any services as long as they meet NHS standards and local needs. Importantly, the Health and Social Care Act 201224 compels CCGs to use evidence to assure the quality of commissioned services. Such evidence is, for the most part, produced by NICE in the form of ‘authoritative guidance, standards, and information’. 25 NICE guidance is considered important so that patient experiences are improved and costs are reduced. 2 However, as above, the uptake of NICE guidance and products has been patchy in commissioning, and recent NIHR research suggests that this shows no sign of changing soon. 1

A key difference around evidence use from these reforms is that some support functions that were formerly ‘internal’ are now ‘external’. 1 Data production, management and analysis is now the remit of commissioning support units (CSUs), whereas PCTs often had dedicated information support managers. These ‘external’ organisations want to support commissioners to make better informed choices. However, a recent qualitative study suggests that this is proving extremely problematic. 1 Wye et al. 1 conclude that:

One consequence of . . . competing organisations . . . was to curtail freely exchanged knowledge transfer . . . organisational boundaries not only frustrated knowledge exchange but also established substantial barriers to the NHS clients’ scope for strengthening commissioning skills . . .

p. xxvii1

This suggests that, although commissioning decisions are now taken by GPs at ‘the coal face’ of service delivery, the ways in which evidence reaches decision-making in commissioning is just as (if not more) varied now as it was in the past.

Theoretical approach: knowledge mobilisation

The so-called ‘gap’ between creating knowledge and using it in practice remains a persistent problem. 23 Policy initiatives aimed at closing the gap include Academic Health Sciences Centres, Biomedical Research Centres and Units, Genetics Knowledge Parks and Collaborations for Leadership in Applied Health Research and Care (CLAHRCs), and new programmes of research (such as this one). NICE was renamed in 2013, following the Health and Social Care Act, in part to emphasis its role in closing the gap. Such initiatives are reinforced by the social and economic costs of not using evidence. 25

However, outcomes have been mixed,5 and it seems that the underpinning assumptions about knowledge may be, in part at least, to blame. 20 Problems still tend to be framed in terms of transferring knowledge in a linear sequence from producer to user. 21 This view treats knowledge as objective facts (e.g. ‘best evidence’) established by scientists and scientific methods, synthesised and captured (e.g. in NICE guidelines) and transferred to users who will pick it up and use it. The limitations of this transfer view as a means of understanding evidence in health care have been well rehearsed (e.g. see Swan et al. 12 for details). While we do not wish to rehash these here, most converge on the observation that health systems often fail to optimally use the substantive body of research evidence available to them. 26

First, linear models depict decision-making as rational process, whereby knowledge travels as objective ‘facts’ from producers to users who rationally decide based on the best information. This objectifies knowledge and divorces it from context and so greatly underestimates its fundamentally social nature and its basis in practice. As critics note, ‘knowing’ is inextricably tied to ‘doing’, and it is divisions of practice (e.g. professional, occupational and organisational) that create divisions of knowledge, and power, in organisations. 15,27,28

Second, health-care settings comprise multiple professional groups, partaking in different practices, with naturally different interests. Experts (e.g. GPs, consultants, nurses and patients) usually seek to use evidence that fits their own ‘epistemic stances’, frames and values. This means that what counts as ‘best’ knowledge and evidence is often contested. 6

Third, the transfer view sees knowledge as an object (e.g. a piece of evidence), divorces it from its context of production and its use (e.g. how things become evidential), and ultimately sees evidence users as merely ‘information sponges’. 29 The knowledge transfer view fails to adequately address the point that mobilising knowledge across boundaries created by different kinds of practice is always a political process. 5 As Crilly et al. 4 note:

The importance of power contests among occupational groups in health systems makes it appropriate to temper positivistic and purely technical approaches to knowledge management with scepticism.

p. 74

Despite these criticisms, the idea of ‘knowledge transfer’ remains pervasive among policy, practitioner and academic groups, with many still attempting to resolve the ‘gap’ between evidence and practice. Indeed, the very notion of a ‘gap’ separates the production and supply of knowledge (or push) from its use (or pull), and places the root of the problem in the hands of ‘users’ who fail to ‘take up’ evidence. The premise is that, if we communicate ‘good’ evidence well, the ‘user problem’ will be resolved. Even the well-informed NIHR programme brief that funded our study called for ‘push’ and ‘pull’ research. This view also predominates the evidence-based medicine (EBM) movement (with its emphasis on ‘the hierarchy of evidence’) and has also been incorporated into streams of research on EBMgmt and implementation science. 1,30 Recently, Greenhalgh et al. 30 argued for a return to ‘real’ evidence-based decision-making, whereby ‘those who produce and summarise research evidence must attend more closely to the needs of those who might use it’. 30

Along with others, we seek to go beyond linear views, which we see as particularly ill-suited to understanding managerial work in health care. 31 When problems are multifaceted, complex and ambiguous, relevant knowledge is not produced in a linear way but, instead, is co-produced through interactions among participating actors. 2 For example, CCGs must account for a multitude of considerations when deciding how to design a service, including quality of care, cost of service delivery and capacity to deliver. Such decisions invoke multiple forms of knowledge and expertise: science alone is not the way to effective decision-making. 31 As Wright et al. 16 put it:

Situated expertise – which is underpinned, in part, by personal experience and judgment – is needed in the handling, adaptation and communication of this evidence . . . evidence does not speak for itself, and neither does it allow decision processes to be enacted without context-sensitive judgement.

p. 17516

We adopt, then, a ‘knowledge mobilisation’ approach, embracing situated and social processes. 20 We pay attention to how evidence is circulated, transformed and arrived at by decision-makers. We define knowledge mobilisation as ‘a proactive process that involves efforts to transform practice through the circulation of knowledge within and across domains’. 12 We share with others the view that evidence is not only passed around as objective facts, but that things become evidential as a result of practice. 9 This approach is underpinned by the following theoretical premises adapted from Swan et al. :12

• Knowledge is social and context specific; evidence exists in midlines, communities of practice and other collectives that sustain, legitimise and transform it.

• Knowledge claims depend on people’s epistemic and interpretive frames; what counts as evidence may be contested across professional and occupational groups.

• At any point certain actors or collectives purposefully pursue knowledge mobilisation; understanding this is, therefore, fundamental to understanding evidence use.

• Agents produce knowledge across multiple contexts, or ‘domains of action’; how evidence is used in one context (e.g. in the practices of CCGs) is nested within, and shaped by, what happens in another (e.g. in the practices of NICE).

• Knowledge mobilisation results from complex interactions under specific conditions (often referred to as institutional context); context matters to evidence use.

• Social relationships and networks (formal and informal) matter to the mobilisation of knowledge; who brokers evidence is important to its use. 32

• Politics, legitimacy and interests inform knowledge mobilisation; evidence does not speaks for itself, and what counts as evidence may change over time. 3,5

• Knowledge does not travel untouched through social interactions; evidence is modified at the point of decision-making.

• Knowledge mobilisation is a transformation process; the same evidence (e.g. NICE guidelines) can be used differently in different organisational contexts. 33

The consequence of adopting this approach goes beyond mere theoretical premises, however; it has concrete implications for the way the ‘gap’ between research and practice is dealt with. In brief, knowledge transfer deals with the robustness and diffusion of evidence, while knowledge mobilisation considers social practice differences that cause different ways of knowing and doing. 12 In knowledge transfer, the gap is closed by creating more research and knowledge in a more timely fashion, upskilling stakeholders to use this evidence and encouraging research cultures in organisations. Conversely, knowledge mobilisation addresses the gap by facilitating and leveraging social processes to circulate knowledge, connecting across boundaries, understanding the role of artefacts and recognising the realities of context.

Knowledge mobilisation and evidence-based management

Evidence-based management literature gained importance in the mid-2000s, having stemmed from the EBM movement. 3 Given its origins, it is unsurprising that the EBMgmt has become particularly salient in health care. Debates about EBMgmt relate to definition and, indeed, whether or not there is good evidence for EBMgmt. 7,34 Some take a rationalistic approach, seeing EBMgmt, similar to its forerunner EBM, as a practice of ‘using scientific knowledge to inform the judgment of managers and the process of decision-making in organisations’. 35 The underlying premises are more akin to a knowledge transfer model, and the key challenge is to ensure the uptake of best scientific knowledge by managers to optimise the rationality of the choices they make. This traditional approach to EBMgmt is criticised for:

1. privileging scientific facts as the objective ‘truth’, which fails to account for the contested nature of knowledge16,31,36

2. underestimating the complexity of management (vis-à-vis professional disciplines such as medicine) and the incompleteness of scientific evidence alone in arriving at management decisions in politically-charged situations17,37

3. lacking support from empirical research on evidence use, which notes the importance of social context in determining what forms of knowledge and information become ‘evidential’, and when, in decision-making2,38

4. losing the importance of situated expertise and judgement of the decision-maker. 10,16,39

Others take a view akin to knowledge mobilisation, recognising that ‘organisational realities seldom map unproblematically onto their idealised “evidence-based” representations’. 33 Furthermore, case study research on evidence use in health-care management suggests that, although practitioners often value scientific evidence [such as NICE guidelines and randomised controlled trial (RCT) data], the way in which they actually use evidence in practice is very different. 32 The complexity of management decisions and organisational context means, also, that the outcomes of evidence use are hard to predict. 40 Walshe and Rundall17 highlight that EBMgmt is challenged by the ‘constrained, contested, and political nature of many managerial decisions, it may be difficult for managers to apply research evidence even when it is available’. 17 These studies reveal that decision-makers, via co-production, mobilise evidence with others. 2 Arndt and Bigelow thus see evidence as ‘an artefact of the social processes that lead to its creation’. 40 That is, things become evidential during, not prior to, decision-making. As Crilly et al. 4 put it:

The original model of strict hierarchies of evidence within clinical practice, transmitted in a linear and rational manner, has been supplanted by a more human and interactive model.

p. 384

This stream of research tells us quite a lot about what types of evidence are likely to be used, and also a bit about when they are likely to be used, in commissioning decisions.

What counts as evidence?

Traditionally, EBM was seen as a means to ensure effective clinical interactions between health-care professionals, patients, families and carers. 41 Decisions were to be based on authoritative, universally applicable evidence from scientific research, including RCTs and evaluation studies. 42 The internal validity associated with scientific research was seen to justify causal links between evidence and clinical behaviour. 43,44 Indeed, evidence champions argued that ‘rational and systematic application of science [brings] about effective, efficient, and accountable practice’. 45 NICE has major responsibility for collating and disseminating this type of evidence in the UK. 13 It is now expected that this type of evidence will be used to ‘organise, structure, deliver, and finance’41 health-care service provision, too.

Although authoritative evidence is still emphasised (e.g. in the ‘hierarchy of evidence’), the role of other evidences is also recognised. In one study, Gabbay and Le May22 showed that primary care clinicians rarely accessed or used explicit research-based evidence but relied on ‘mindlines’, which were informed by colleagues’ experiences and their interactions with each other and with the knowledge, mostly tacit, of opinion leaders, patients and other experts. Indeed, it has been said that there is ‘little compelling support that scientific evidence is treated differently to other types of information’. 43 As Crilly et al. note, ‘even in the medical arena, which draws on experimental, replicable and ostensibly generalisable knowledge, the notion of a hierarchical evidence is contentious’. 4 These findings are also echoed in more recent studies of evidence use commissioning. Wye et al. 1 conducted a study (parallel to ours) of knowledge exchange between commissioners and external providers. They found that commissioners sought information from many sources to build a case for action, defend their position and navigate their proposals through the system. Information was clustered into four primary types:

1. ‘people-based’ sources (relationships, ‘whole-picture’ views, experience, contracting and procurement expertise and information on finance, budgets and performance)

2. ‘organisation-based’ sources (NICE guidelines, Department of Health information such as National Outcome Frameworks and information from NHS Improving Quality, Public Health, CSUs, ‘think tanks’, such as The King’s Fund, Royal Colleges and other providers)

3. ‘tool-based’ sources (electronic software tools, national benchmarking, national and local dashboards, and project management tools)

4. ‘research-based’ sources (journals, search engines, universities, Cochrane reviews, regional networks, such as CLAHRCs, Academic Health Science Networks and electronic newsletters).

Wye et al. 1 note, however, that access to data in their study was obtained mainly through external providers; given the timing (in the transition period from PCTs to CCGs), commissioning organisations proved more difficult to enlist. This might have provided a somewhat broader picture of sources of evidence (especially authoritative sources) than those actually used by commissioners. 1 Even so, the study suggested that, for clinical commissioning, local clinical knowledge from GPs about service provision was prioritised: ‘Local information often trumped generalised research-based knowledge or information from other localities’. 1 Most often, this evidence was acquired interpersonally through ‘conversations and stories, as oral methods were fast and flexible, which suited the changing world of commissioning’. 1

Our previous study used a more direct, detailed ethnography of commissioners’ evidence use in redesigning services. 2 Case studies of four PCTs showed the need to source information creatively so that it was fit for purpose. Evidence use entailed a ‘heterarchy’ rather than a hierarchy of evidence, with local and interpersonal practical knowledge and information being coupled with other scientific sources. Evidence did not ‘speak itself’ but needed to be advocated by experts or authorities. Interestingly, a wider sample of commissioning managers suggested that ‘the single most important source of evidence was examples of best practice from other organisations’. 2 Harvey et al. 46 also describe different kinds of evidence, particularly ‘theoretical’ (how and why something works) ‘empirical’ (efficacy) and ‘experiential’ (personal experiences) evidence. They note that the ‘. . . complexity and context-dependent nature of implementation [makes it] impractical to prioritise one type . . . over others’. 46 This work raises questions about what counts as evidence in CCG decision-making, and so this will be an initial focus of our study. 46

This literature suggests that, in addition to NICE, commissioning stakeholders may mobilise a range of other evidences in their decision-making. Thus, we respond to calls to ‘broaden and deepen our understanding of what counts as “evidence” and which types of evidence are best used to inform differing aspects of clinical decision making’. 47

When is evidence used?

To better understand when evidence is used, we draw from studies of innovation processes that recognise complex problems and developing new practical means of dealing with them. 48 Rogers’ very widely cited work refers to three broad stages of innovation: ‘initiation’, ‘adoption/decision’ and ‘implementation’. 49 Responding to criticism that ‘stages’ are an overly linear and rationalised depiction of innovation processes, Clark et al. 50 developed, instead, their ‘Decision Episode Framework’. This suggests that innovation processes unfold in a series of recursive and overlapping ‘decision episodes’. 50,51 These entail the awareness of new ideas to formulate a problem (‘agenda formation’), designing, sorting and selecting potential solutions (‘selection’), introducing these (‘implementation’), and, finally, embedding them into organisation routines (‘routinisation’). It is important to recognise that these are seen as episodes of work, with feedback loops between them. For example, at any point during the selection or even the implementation of a solution, new problems may arise that require the reformulation of the initial problem (agenda formation).

There are different concerns across episodes, from a focus on matters at stake, to practical design of usable solutions, to outcomes and issues of sustainability. Bledow et al. 52 see innovation as resulting from a dissatisfaction with old ideas (‘thesis’), which, in turn, motivates new ideas (‘antithesis’). Synthesis between ideas creates innovation.

Kyratsis et al. 9 usefully applied a process approach, albeit in a different context: a study of the use of evidence in decisions about technology adoption in nine acute care organisations. They found that the practitioners involved (clinicians, nurses and non-clinical managers) held diverse ‘evidence templates’ when seeking and using evidence. These cognitive templates stemmed from the different practical experiences, professional identities and training of these groups, resulting in different ways of defining and making sense of what constituted acceptable and credible evidence in decisions. All groups drew on a ‘practice-based’ template, using experiential practical knowledge as evidential in their decisions. However, whereas clinicians drew mainly from a ‘biomedical-scientific’ template, prioritising science-based, peer-reviewed and published evidence, non-clinical managers drew more from a ‘rational-policy’ template, preferring evidence based on cost, productivity and fit with policy imperatives. Nurses, in contrast, drew from all three templates. Importantly, different evidence was valued at different points of the process, depending on who the key actors were at the time and how they made sense of the problems within their professional domain.

. . . local trials, and the need for ‘pragmatic evidence’, were deemed important by decision-makers in the early stages of innovation adoption. For innovations (and especially those that require changes in practices or processes), creating an evidence base will require agreement about what is regarded as a legitimate epistemological basis for verifying and validating evidence and relevant knowledge.

p. 1369

Evidence use is not, then, a one-off event. It reflects the priorities present at a particular point in time. It is entirely plausible that different evidences will have differing utility and weight across episodes of innovation work. 52 For example, RCTs may be used as evidence when identifying solutions to a problem, whereas financial data may be used to understand feasibility.

Certain types of evidence relevant in one episode may also be questioned or disregarded in another. Recognising this, we use a processual approach in our own methods and analysis,53 tracing evidence use in service redesign over decision episodes. As Wright et al. observe, ‘deeper insights are uncovered when decisions are investigated in ways that “open up” the role of the decision-maker and of the context in the processes leading to the commitment to action’. 10 Keeping with a knowledge mobilisation approach allows us to understand forms of agency that may make decisions evidence based. It can be expected that multiple evidences will be used and that different capabilities may apply across different episodes of commissioning work in service redesign.

How is evidence used: capabilities for evidence use

Although the literature says much about what evidence is used and when it is used, how it is used is less clear. It is important to understand, also, how evidence comes to be used in practice. In other words, what capabilities are required to actually use evidence? Existing research, noted above, points to the fundamental importance of understanding the social and political processes underpinning evidence-based decision-making. Much of the literature from both the UK and Canada points to the social discursive aspects of evidence use. 54

However, questions remain unanswered about ‘the detailed process by which ideas are captured from outside, circulated internally, adapted, reframed, implemented and routinised in a service organisation’. 30 Crilly et al. conclude that ‘NHS Boards should take a clear view on organisational design elements needed to support knowledge mobilisation’. 4 These elements include ensuring that capabilities (including appropriate work practices and enabling organisational conditions) are in place. A few insights can be found in the literature about these capabilities.

First, evidence-based decision-making involves mobilising knowledge across diverse experts, including patients. Recent health-care research shows that stakeholders bring their own values and sense-making to any decision, which informs how evidence is sought out, shared and applied in transforming services. 1 Professional differences influence the use of evidence, too, with McGivern et al. 11 showing clinical consultants as having the ‘power, knowledge, and self-confidence’ to critique NICE guidelines. Other clinicians (e.g. GPs) in this study were shown to ‘not have the time’ and were too ‘overwhelmed’ by the range of guidelines. Instead, these stakeholders worked to balance research findings with local needs. In that study, management saw NICE guidelines as a means for stakeholders to reach consensus and avoid negative consequences from regulatory bodies. 11

Second, just having appropriate experts and evidence present does not guarantee evidence-based decisions. Actors also pursue particular purposes, with varying degrees of influence (they have agency, in other words). The issue of who searches for, synthesises and presents evidence is, therefore, important. 7 When this happens may also be important, as different priorities beckon at different points (ensuring sufficient evidence to solve a problem, identify solutions in our context, etc.). ‘Knowledge brokers’, or individuals who make knowledge embedded in one community available to another, may enable such evidence sharing. 6

Third, research points to the subjective aspects of evidence use and the importance of human judgement. 5 Harvey et al. 46 describe evidence ‘as forms of knowledge seen as “credible”’, while Freeman and Sweeney55 suggest that use relies on perceived ‘relevance’. In a study of hospital organisations, Jones and Exworthy56 describe how different ‘frames of reference’ shape perceived relevance. They show that framing an issue in terms of clinical necessity engages practitioners with change. Frames may be strengthened through co-construction and discussion. Champions or opinion leaders may also communicate certain evidences and push them into dominance. 57

Broader insights can be gained from organisation studies. Organisational structures (e.g. networks) underpinned by trust-based relationships are said to promote the ‘open sharing’ required to innovate. 58 Crilly et al. argue that ‘Relationships trump organisational design . . . Connective ability of individuals is more important than organisational structure when it comes to making organisations effective’. 4 Another influential stream of research is ‘resource-based theory’ (and, its successor, the knowledge-based view), which points to the importance of resources, especially knowledge, in improvement. 59 It is argued that ability to integrate, build and reconfigure internal and external competences to address rapidly changing environments informs competitiveness.

The work outlined to this point sensitises us to the kinds of capabilities NHS organisations and managers may need to use evidence. Indeed, in one of the few studies of capabilities for evidence use in commissioning, Swan et al. 2 show that evidence use in PCTs entails timely evidence use, managing interfaces across the project and proactively building and managing relationships. By better understanding what counts as evidence, when it counts and how it comes to count as evidence, we can begin to develop capabilities to enable evidence use in decision-making. As Ferlie et al. 5 note, the imperative for mobilising knowledge and evidence in NHS organisations lies not in producing ever more information, but in ‘building high learning capacity and appropriate core competencies in NHS organisations’. 5 We seek to understand capabilities for using different types of evidence in the decision-making that occurs across episodes of work involved in CCG innovation.

How is evidence produced?

A knowledge mobilisation approach reminds us that the use of evidence in one domain (e.g. commissioning) is intertwined (or ‘nested’) with the machineries of its production in another. 12 We thus consider, also, the ways in which evidence is produced by considering ‘inscribed meanings’. 60 This research suggests that a ‘programme of action’ or ‘script’ is inscribed in all artefacts. 60–62 Inscribed meanings constitute a sort of instruction manual for the artefact. Designers of artefacts take a view of how it should be used in order to perform its expected function. Akrich et al. describe how the large part of the work of producers involves ‘defining the characteristics of their objects [and] making hypotheses about . . . the world into which the object is inserted’. 60

Inscribed actions exist in everyday objects such as door handles, which carry the programme of action ‘pull the door’. Actions are also inscribed on the internet, where readers of underlined blue text are invited to click and follow related links. In similar ways, evidence is an inscribed artefact. The inscribed use of evidences must be translated into practice through a relational and negotiated process that requires work, time and motivation. 12 Encounters between scripts and users can broadly result in three possible scenarios:

1. Scripts may persuade actors to play the proposed roles through training, coercion or discipline. 63 If scripts differ from established actions and actors subscribe to them, the artefact produces change. For example, a NICE guideline highlighting risks of current methods may change behaviour.

2. More often, users try to descript artefacts and modify actions to fit what they already do. 60 Users try to adapt the artefact to their ‘tastes, competences, motives, aspirations, [and] political prejudices’ through reinvention. 60 It has been suggested that ‘if potential adopters can adapt, refine, or otherwise modify the innovation to suit their own needs, it will be adopted more easily’. 64 For example, NICE guidelines may be disassembled and used selectively when they fit existing priorities and work patterns.

3. Reinvention has limits, however, which may cause users to ignore, disregard or resist the artefact. The third scenario is, therefore, one of non-use. Coercion can be applied, but users may find ways to circumvent the impositions of the artefact. They may, for example, revert to workarounds or utilise artefacts in a ceremonial way only. For example, NICE guidelines may be ignored or their appropriateness for a particular population may be questioned.

Previous studies suggest that scenarios 2 and 3 often apply in health. 63,64 We set out to explore what sort of evidence users are inscribed in evidence production in order to develop further insights into the use of authoritative evidence in commissioning.

Conceptual framework

Figure 1 summarises the conceptual framework for our study. It links capabilities to use evidence to evidence production (e.g. NICE), decision episodes and outcomes. The framework provides a holistic approach to explaining evidence use and how this links to variation in process outcomes. It builds on three principles (from the review above).

FIGURE 1.

Conceptual framework for the research.

1. It is processual. 53 It takes on board the point that mobilising evidence entails dynamic practices (e.g. sourcing, interpreting and creating evidence) that unfold over time in an iterative, episodic way (hence the double-headed arrows and decision episodes in Figure 1). As per previous work on organisational change, the journey is conceived as an often uncertain process to reflect the ongoing realities of commissioning management decisions, which are far from linear or ‘rational’. 65 It is also more practical because it reflects the situations that managers face, rather than a more idealised version of events. Hence, it helps to provide a direct link from research findings to actionable recommendations for improvement.

2. Our model is underpinned by a knowledge mobilisation approach that sees evidence use as a social and practical accomplishment requiring capabilities: that is, knowing how and when, not just what. Knowledge mobilisation theorising makes us understand evidence use as a skilful performance, situated in context, in which agency, practices and processes matter. 12 We see capabilities as the practices and enabling conditions needed to pursue and accomplish a particular purpose (evidence use in this case), rather than as static variables, or fixed characteristics of individuals and organisations. Capabilities can be applied at the organisational (e.g. resource management) and group level (e.g. role structures).

3. The model will help us to explain variations in evidence use by comparing similar cases, as other authors have done. 66 We will compare and contrast evidence journeys across different CCGs dealing with similar kinds of redesign. We examine, in particular, if (hence the question mark in Figure 1) and how authoritative evidence (e.g. NICE) makes its way into CCGs, along with other forms of evidence. As per our protocol, ‘outcomes’ are to be assessed qualitatively, including evidence use (the extent to which a range of evidences were sourced, interpreted and engaged with), stakeholder experience (the extent to which stakeholders were satisfied or dissatisfied with the process) and objectives met (the extent to which original objectives were met and, if not, whether or not change to them was justified).

Chapter summary

In sum, this research provides an in-depth examination of the use of evidence across NHS organisations (CCGs) in order to identify capabilities for relatively more successful outcomes. As our review of the literature indicates, authoritative evidence never stands alone but must be judiciously combined with other forms of evidence. Viewing evidence use as a process of knowledge mobilisation provides a more realistic (and, therefore, more useful) account. The study develops a process model that articulates the dynamics of evidence use, without losing sight of local work practices, power dynamics and context. This allows us to begin to generate relevant ideas and tools for improvement in practice.

Research on evidence ‘pull’

As described in the introduction, our research responds to the NIHR commissioning brief by aiming, primarily, to understand so-called research ‘pull’ by managers by studying evidence journeys in practice (as evidence travels through CCGs). We contribute (as a secondary aim) to the understanding of ‘push’, by considering, also, the start of the journey (i.e. how certain evidence products are made by producers). The following objectives have been operationalised to achieve these aims:

• to study how authoritative evidence travels in, and is used by, different CCGs, that is, to document ‘evidence journeys’

• to compare and contrast the travel of evidence across CCGs and analyse comparatively ‘evidence journeys’

• to create a process model, a roadmap and a toolkit that will guide CCGs in becoming better users of evidence.

Research on evidence ‘push’

Our original protocol aimed to trace the evidence journey in CCG commissioning, including the place at which evidence ‘enters’ the journey, as guidelines and other forms of evidence-based products. Here, we conducted a further small-scale exploratory study in one large guidance-producing organisation, in which we considered the ways in which projected users are inscribed in published evidence. Our objectives were to understand:

• the user image inscribed in published evidence products

• how an evidence producer believes products will be used in general, and by CCGs.

Over the course of 6 months, during 2015–16, we carried out eight interviews, spoke less formally with other evidence producers and consulted a number of documents, some publicly available and some internal. Informants included project managers and clinical staff involved in the production of the evidence (n = 4); and staff involved in communication, marketing and other customer-facing activities (n = 4). We also observed one visit of an evidence producer to a CCG. We asked questions about the following in semistructured interviews:

1. how the design and production of evidence (e.g. guidance, advice, quality standards) is organised and the nature of the products

2. how products are made available to users and how this has changed over the years

3. a scenario question (according to critical incident interview),78 in which interviewees were asked to focus on a specific product (guidance, quality standard or other) and describe how it is actually used, with examples.

All interviewees gave informed consent. Interviews were recorded and transcribed verbatim. The same inductive coding method was used as described above for identifying capabilities.

Summary of research methods

We conducted two phases of research to understand evidence ‘push and pull’. The findings were shared with case study participants via feedback and with our SSAP by e-mails and meetings. In the next chapters, we present findings from phase 1. We describe what evidences are used (see Chapter 3), and then capabilities for use (see Chapter 4). Building on this in Chapter 5, we show evidences and capabilities in practice through four analytically-structured cases (two on diabetes services and two on MSK services). The remaining cases are shown in Appendices 1–4.

Universal evidence

Information produced by formalised or scientific investigation in institutions or organisations that are deemed to have legitimate authority (e.g. universities, NICE and national bodies) is commonly treated as evidence. 1 This type of evidence is produced systematically through scientific method (e.g. RCTs and NICE guidelines) and/or standardised nationally (e.g. QOF data), so it is considered robust, rigorous, widely applicable and authoritative. Because this type of evidence is expected to be used widely, it has been termed ‘universal’. 2 We think of universal evidence as produced from the ‘top down’. Universal evidence includes NICE guidance, but also NHS reports, public health data, Right Care guidance and academic papers, among others. Previous studies showed the importance of this type of evidence in decision-making in commissioning work. 1,2

Stakeholders also described mobilising universal evidence to understand ‘good practice, best practice’ (Chelsea CCG). They were aware, for example, of ‘what’s being proposed in the Five Year Forward [View]’ (Seaport CCG), the NHS-produced document setting out a vision of ‘GPs, nurses, community health services, and hospital specialists [providing] integrated out-of-hospital care’. 79 A consultant at Seaport CCG described using NICE to inform service design, in this case because it mandates that antenatal diabetes has ‘a midwife, an obstetrician, a consultant, and a scanning machine’. Evidence from academia was also used, including ‘studies about how do you improve service provision, about impact . . . tools’ (Coalfield CCG). Interestingly, some kinds of universal evidence appeared to be more prominent than others. Cochrane reviews were not, for example, referred to in any case (at least directly), whereas NICE guidance was referred to rather more often.

As in other studies, the rigour of universal evidence was understood, but its usability and relevance was questioned. NICE guidelines in particular were problematic, with stakeholders seeing them as ‘very long . . . it’s like reading a book!’ (Horsetown CCG). 1 Furthermore, in order to be widely applicable, this kind of evidence had, by definition, to be abstracted from any particular context, which often left commissioners puzzling over how to apply it in their own setting. At Rutterford CCG, for example, stakeholders described how ‘on top of all of that [NICE] guidance, you have to put in what is going on locally . . . and what is available’.

Local evidence

Information produced and collected locally within CCGs or in their immediate locale was also, and very often, treated as evidence. 1,2 This information is wide ranging, including patient profiles, local activity and performance data, finance data, contracting models and monitoring outcomes. It may also include externalised observations and experiences of stakeholders. Unlike universal evidence, this type of evidence is seen as more readily ‘to hand’ for those involved in commissioning decisions and, therefore, ‘useful’.

Across all cases, local evidence featured heavily in the conversations among commissioning groups. It was used to show problems, for example ‘a 41% increase in hip and knee replacements’ (Horsetown CCG) or ‘you serve only 10% of the population [in the hospital]’ (Seaport CCG). Local evidence was also used to identify solutions to problems, for example ‘The CCG have done a piece [of research] that sort of says . . . 70% of activity can be stripped out [from the hospital]’ (Greenland CCG). Alongside these ‘hard’ numerical data, stakeholders also relied on softer tacit knowledge about the local area: ‘a lot of patients . . . being followed up in secondary care, which is not necessarily where they should be’ (Greenland CCG).

Evidence collected in the local area gives meaning in context, and so it is not surprising that is has been shown as important in the broader literature, too. 1,2 Although readily available at a relatively low cost (in terms of search time and effort, at least), this type of evidence is not subject to the same scrutiny as universal evidence, and so issues around its validity could often arise. 80

Expertise-based evidence

Information based on the experiences and expertise of individuals could also be treated as evidence. Expertise-based evidence can be sourced from health-care providers, colleagues, patients, families, carers and many other groups. 1,2 Stakeholders in our cases used expertise-based evidence to form agendas. At Seaport CCG, for example, the consultant describes how ‘the idea [for the redesign] came from . . . a group of us registrars’ because ‘we always used to talk about it, saying it isn’t quite right the way we’re doing it’. Expertise-based evidence was also used to identify solutions. At Horsetown CCG, for example, commissioners chose community-based solutions because they had ‘set up things in the community before [and seen] the cost reduction . . . the better patient experience’.

Expertise-based evidence is personal, embodied and value-laden. It is often transferred through narratives. It possesses few characteristics of scientific rigour; this evidence is very relevant to context. We noted that individuals with clinical backgrounds who were considered charismatic and were respected tended to be considered holders of ‘expertise’. Although anecdotal, our observation reflects the study by Wright et al. 10 how one individual (‘Dr Clancy’) influenced service redesign in an Australian emergency department.

Trans-local evidence

Examples from elsewhere were also treated as evidence. 1,23 We call this ‘trans-local’ evidence because information is created in one place and used in another. We do not call it ‘best practice’, because evidence can be trans-local without being validated as ‘good’ or ‘best’. In our cases, trans-local evidence was often applied before efficacy evaluations. Rather, the fact that changes were made in other places seemed to have legitimising properties. The GP at Greenland CCG, for example, described choosing to mimic Seaport CCG’s diabetes model. When asked about this decision, the GP at Greenland CCG did not seem entirely certain about efficacy, describing, ‘I suppose we haven’t really got long-term evidence’. The arrival of trans-local evidence in new sites is also far from systematic. Instead, translation through stories or examples, ad hoc gatherings or accidental encounters (e.g. conferences or meetings) seems important. Greenland CCG mimicked Seaport CCG because a GP in the former had heard a talk from a consultant at the latter and ‘felt’ that Seaport’s was the right solution. Chelsea CCG mimicked Seaport CCG because their consultant had seen ‘first hand that it works’. Interestingly, we saw relationships among all categories of evidence. Local and universal evidences were often used together in sense-making activities. Stakeholders understood local evidence (e.g. about patient waiting times) in reference to NHS standards, which are based on universal evidence. In the case of Seaport CCG, for example, local evidence showed that patients were waiting 18 months for an appointment, when the maximum NHS waiting time is 18 weeks. At Greenfield CCG, the transformation manager described how ‘amputation rates were quite high’ (local evidence), especially ‘when compared nationally’ (universal evidence). Stakeholders at Coalfield CCG also described making changes before they had been published (universal evidence) because they knew that certain recommendations were in the pipeline and they felt that those changes were appropriate (expertise-based evidence). It is important, therefore, to recognise that different categories of evidence are relevant to redesign work, but that these are often used, by necessity, in tandem in the practice of decision-making. Hence, in relation to the use of NICE guidelines in commissioning practice, this is not simply a matter of accessing or adopting relevant guidelines in a timely manner, but also requires skilful combinations of different forms of evidence in order to make them actionable in a local context. The four categories of evidence are summarised in Table 5.

Chapter summary

In this chapter, we discussed the evidences used by stakeholders, identified as ‘universal’, ‘local’, ‘expertise-based’ and ‘trans-local’ evidences. Evidences were identified deductively from existing literature and also inductively from our raw data. Universal, local and expertise-based evidences have been discussed, albeit under different labels, in previous studies. Trans-local evidence has also been discussed, to an extent, in previous work. Much of the existing literature has described ‘examples from elsewhere’ or ‘best practice examples’ as evidence. Although similar, we use the label ‘trans-local’ to better represent the fact that this type of evidence must be sourced from a locality different from that of its creation. We also avoid ‘best practice’ because, as will be shown in the cases, rigour, evaluation or efficacy were not always considered in using this type of evidence.

Issues around the use of these evidences were clear, especially in the mobilisation of authoritative NICE evidence. In the next chapter we consider capabilities for evidence use that might alleviate these issues.

Sourcing and evaluating evidence

Clinical Commissioning Groups are expected to use evidence to assure the quality of services. Yet they have received little guidance on how to achieve evidence use. 41,81 The quality and quantity of evidence relevant to health-care contexts is one factor that can undermine its use. It has been estimated that as many as 75 trials and 11 systematic reviews are published daily. 82 Stakeholders in our cases noted that using universal evidence ‘became this minefield because . . . how do you synthesise all this so that you’d end up with something that is succinct so that you can deal with it?’ (Rutterford CCG). Even if evidences can be sourced, application can still be problematic. Stakeholders reported that it was difficult to apply evidence to context, especially ‘NICE guidelines . . . they’re very long . . . it’s like reading a book!’ (Horsetown CCG). The quality of available evidence was also often an issue, as the GP below notes:

[The hospital] thought they had 2000 diabetics [but] it looks like . . . about 500. But they didn’t know, they had no idea at all. If I’d have said to them you’re looking after 8000 they’ll have believed me. Or if I had said it was 150 they would have thought well, we seem very busy but they don’t know. I am astonished, absolutely astonished at the quality of data that comes out of the hospitals . . .

Greenland CCG

Efforts to source and evaluate relevant evidences must be made if CCGs are to conduct evidence-based service design. As noted, ‘evidence does not speak for itself, but needs to be mobilised’. 2 Although mobilising evidence may seem like a fundamental practice, efforts differed across CCGs. In Rutterford CCG, for example, ‘task and finish’ groups were established specifically to achieve evidence use. Subgroups were asked to report if there was:

Any evidence . . . any models of care . . . any best practice . . . [anything] published in the literature that we might want to adopt and change . . .

In other CCGs, efforts towards sourcing and evaluating evidence were weaker and more problematic. At Horsetown CCG, for example, stakeholders reported that they had brought evidence forward but ‘then nothing happened’. At Greenland CCG, the evidence search and evaluation was limited because the outcome had already been decided. Because the solution was predetermined, stakeholders ‘had not considered a lot of options . . . It was just going to go for the community-based team’.

Existing literature (see Chapter 1) also shows that evidence use is time and task dependent. 2 Redesigns involve episodes in which different tasks must be accomplished. During agenda formation episodes, stakeholders may have to scope evidence about demand, while later they may need evidence about efficacy and evaluation. Therefore, it is unlikely that narrowly sourcing evidence early on (as in Greenland CCG) will be sufficient. Instead, stakeholders may need to be responsive to evidence demands across the process. Given the challenges of evidence use, the importance of evidence use in CCG work and the variation shown across our CCGs, our first capability for evidence use is ‘sourcing and evaluating evidence’.

Engaging experts

In addition to sourcing and evaluating, experts must be engaged to effectively apply evidence. By ‘experts’, we mean individuals with relevant education or qualifications (e.g. doctors and consultants) whose expertise is embedded, localised and invested in practice. 27 We also mean non-professionals such as health service users (e.g. patients, families and carers). Indeed, this group is key to the NHS mantra ‘no decision about me, without me’. 83 Both groups of experts can offer expertise-based evidence to redesign projects.

Studies have also shown that different types of evidence are used across professions. 23 Consultants, for example, rely more on universal evidence, whereas nurses and doctors are more familiar with local evidence. 11 It is, therefore, important to have multiple stakeholders engaged who can contribute, discuss, interpret and critique a range of evidences (e.g. local, trans-local and universal). In our cases, engaging experts was also a way to ensure a smooth redesign, because ‘you can’t just simply say well, we’re changing the service as of tomorrow and expect everybody to comply’ (Coalfield CCG).

In the majority of our cases, efforts to engage experts were paramount from the outset of the project. These efforts ranged from ‘sending an e-mail round to everyone’ at Seaport CCG, to contacting individuals in person at Coalfield:

Before every board meeting I spoke to every board member . . . sticking my head round the door while [saying] do you get it? Do you understand? Have you got any questions? Is there anything worrying you? It’s literally simple as that. And they’d go that’s fine, get out, for God’s sake. In the end people get bored, go away . . . we support you, we’re with you . . .

Senior commissioning manager

In some cases, expert engagement among professionals was limited. In most of the cases, patients’ expertise was invoked in a rather ‘afterthought to the main event’ type of fashion. At Greenland CCG, for example, stakeholders described how they did not really feel part of the redesign project. In the next quotation, a stakeholder describes how the project ‘belongs’ to the lead GP:

This is [his] project . . . we’re not really involved – we are in the sense that we’re involved to ask our opinions in terms of information from our diabetes patients but we’re not actively involved with the day-to-day . . . if you think about, if you’ve got only say five or six GPs, you know, that’s a lot of areas that you’re leading on.

Because expert engagement is a means to promote and encourage the use of different types of evidence, and to ensure a smoother redesign processes at CCGs, our second capability for evidence use is ‘engaging experts’ in the redesign.

Effective framing

Effective framing appeared as one means of engaging stakeholders in redesign work. 56 Frames are ‘organising principles that govern the meaning we assign to social events’. 84 Framing allows groups to construct and negotiate meaning. 57 A strong frame resonates with actors and may increase engagement. A weak frame is uncertain and can mean resistance, disinterest and incongruence. 57 It can be expected, then, that stakeholders may be more engaged when a frame resonates with them. In our cases, problems were framed in terms of ‘capacity’, ‘care’ and ‘cost’. Patient care was not good enough, which was often caused by capacity issues, and costs were too high. These issues are all interlinked: capacity issues increase waiting lists, which exacerbates illness so that more complicated care is needed. Care, costs and capacity provide a broad frame that can be understood by different stakeholders. Consultants, for example, may be concerned by capacity, whereas commissioners may be concerned with cost. In many CCGs, this broad framing was applied. The accountable officer at Rutterford CCG, for example, describes how diabetes care was ‘a poor use of resources, not great value for money, and [having] not great outcomes for patients’. In other CCGs, problems were framed more narrowly. In Horsetown CCG, for example, the problem was framed almost entirely in terms of cost: ‘MSK is a high-cost, high-volume activity . . . we were conscious that a lot of people were going into a hospital which is very high cost’.

Framing was also important during solution search. Searching for a solution in a way that was congruent with stakeholders’ expectations was important because ‘you can’t just simply say well, we’re changing the service as of tomorrow and expect everybody to comply’ (Horsetown CCG). A ‘blank sheet’ approach was mentioned across cases, which invited stakeholders to contribute ideas about what could and should be done to improve services prior to any formalisation of a particular model of care. Interestingly, ‘blankness’ occurred on a continuum, ranging from ‘not blank enough’ to ‘too blank’. In Greenland and Chelsea CCGs, the sheet was ‘not blank enough’, because the solution had been pre-selected before stakeholders could contribute to the search, such that they felt that their expertise was being excluded. In other CCGs sheets were ‘too blank’, with experts struggling to understand what kind of evidence they might look at and for what purpose. Some CCGs used, rather, a partially blank approach. In Seaport CCG, for example, the redesign leader knew that community-based care would eventually be a part of the outcome, but asked stakeholders to ‘have a discussion how can we do diabetes care better . . . you tell me what works’. Similarly, stakeholders at Coalfield CCG were asked to give evidence about ‘trying to keep people out of hospital’:

One of them said so what do you want us to do? I said I want you to deliver the best possible care for patients for the limited amount of money you’ve got . . . they said how do you want us to do it? I said I don’t care. I said if you want to get a bag of bones and shake them if that makes a difference that’s fine. I said I just want the outcome . . .

Clinical lead, Coalfield CCG

Our findings suggest that weak frames can mean that stakeholders are less likely to contribute to evidence work, owing to the degree of ambiguity involved and difficulties in reconciling evidence pertaining to multiple concerns (cost, care and capacity). On the other hand, framing problems and solutions too tightly, prior to efforts to engage experts, may seal off the search for evidence too soon and could also create problems of stakeholder engagement with the predefined approach. We suggest, then, that it is important to consider not just what problems and solutions are considered, but how they are put forward, framed and communicated in CCG redesigns if evidence use is to be effective. As a result, we identify ‘effective framing’ as a capability for evidence use.

Managing roles and expectations

In engaging experts, it is also important that their roles and expectations are recognised. Certain CCG roles are mandated (e.g. CCG chairperson and accountable officer), but others are more ambiguous. Stakeholders are often brought in to participate in projects without clear governance on what their role is or what they expect from the role. Although it may not be feasible or practical to detail the role of every stakeholder prior to project commencement, this is not to say that roles should not be a considered. In some of our cases, stakeholder roles were very clear. In relation to evidence use, the diabetes consultant at Chelsea CCG had a clear role in bringing trans-local evidence forward because he had worked in a hospital in which the ‘model had already been developed, delivered, evaluated’ and had ‘seen first hand how it works’. Similarly, at Greenland CCG, the CSU project manager was tasked with bringing different evidences forward. As the GP at Greenland CCG describes:

Without [her] we wouldn’t have been able to do this . . . we are doing our day jobs . . . she . . . gets the information on the table . . . on different models.

Roles were not limited to evidence use. At Seaport CCG, for example, one of the GPs described how his role was to act as ‘similar other’ who could justify and legitimise the redesign work to other GPs. He described, for example, how he would say to other GPs:

This affects me [too] . . . I work in the practice . . . I understand . . . I am going to have to do more than I have traditionally done but . . . I can do that because I am supported and we’ve got the right package of support that is available in the community . . .

Roles, however, were not clearly defined and managed in all CCGs. At Shire CCG, it was felt that the ‘wrong’ people were sometimes involved, which was detrimental because ‘unless everybody understands why they’re there and understands what their role is, then potentially you’re not going to get further forward’. In this case, some stakeholders were not ‘engaged in doing that piece of work . . . and just turning up and being there because they’ve been asked to be’. Similarly, at Horsetown CCG, stakeholders had certain expectations of their role but these were not met. One of the academic GPs described how ‘nobody looked at the evidence . . . in the meetings . . . I was hoping . . . because I’m there and I work in research . . . maybe they will ask me . . . they didn’t do that’. If stakeholder roles and expectations are not considered, incongruence and conflict arising from ambiguity of roles may occur. Therefore, considering what stakeholders expect, and what they are expected to do, is important, and we identify ‘managing roles and expectations’ as a fourth capability.

Managing expert collaboration

Our analysis suggests that while engaging experts it is also necessary to manage ongoing collaborations between them as the service redesign process unfolds. This is because interprofessional health-care work is high-stakes and ‘fraught with tension and anxiety’. 85 In commissioning, individual jobs, contracts, issues of governance, compliance and patient care are simultaneously in question. The transformation manager at Rutterford CCG describes: ‘challenges . . . disagreements . . . debates . . . change is frightening . . . it can make you feel a bit insecure’. Stakeholders were well aware of the challenges inherit in this type of work, describing how vested and competing interests mean that having everyone ‘around the table had got that sort of political aspect to it’ (Horsetown CCG). These concerns could prevent stakeholders from ‘properly discussing’, interpreting and critiquing different forms of evidence, Moreover, during the lifetime of these redesign efforts, experts came and went. This meant that ongoing attention to managing collaborations appeared to be very critical. In some CCGs, expert collaboration was managed well. Stakeholders worked to have teams focus on a common ground, in many cases centred on patient care. In Seaport CCG, for example, a nurse described how the consultant leading the redesign work had:

A way of getting people to recognise . . . we’re trying to achieve the same thing . . . good-quality care for patients . . .

Collaboration was managed less successfully in other CCGs. In Greenland CCG, for example, the lead GP made efforts, but he was unsuccessful in managing these collaborations. He was aware that there was ‘anxiety about levels of knowledge and the increasing workload’ and felt that he had done ‘a huge amount of work’ to engage stakeholders and address these issues. He acknowledged, however, that evidence for the redesign effort was still being ignored ‘there are still some practices who say that they don’t know anything about it’. Concerns about collaboration persisted at Horsetown CCG:

[Y]ou suddenly think ‘oh, what are they going to change?’ There was quite a lot of concern over what was it going to mean . . . Technically we’re in competition . . .

Because collaboration between stakeholders can be problematic and can limit their ability to share ideas and discuss evidence openly, we identify the following ‘managing expert collaboration’ as our fifth, and final, capability for evidence use.

Chapter summary

In this chapter, we identified capabilities for evidence use: ‘sourcing and evaluating evidence’, ‘engaging experts’, ‘effective framing’, ‘managing roles and expectations’ and ‘managing expert collaboration’. Figure 2 shows capabilities for evidence use (dark green box), why they are important (light green box) and an example from the data (white box).

FIGURE 2.

Capabilities for evidence use.

Introduction to case studies

In what follows, four case studies – Seaport, Greenland, Horsetown and Stopton CCGs – are presented. These four cases were selected as exemplars of our finding across cases. Owing to constraints on space in the main body of this report, the remaining four case studies – Rutterford, Chelsea, Coalfield and Shire CCGs – are presented in appendices. Developing these case studies addresses our first objective of studying how evidence is used in CCGs. These case studies also lay the groundwork for comparing evidence use across cases (objective 2) and creating a process model (objective 3).

Each case study begins with an overview of the redesign project and of the study participants. This overview is followed by a chronology of the main events. Decision episodes are applied to structure our cases. 32,51 Stakeholders across cases described how redesigns usually began when a problem was recognised. Akin to Robertson et al. ,86 we label this episode ‘agenda formation’. After this, work to generate a solution tended to commence. Again, we replicate Robertson et al. 86 and label this episode ‘solution selection’. Next, stakeholders described how (and if) they made changes to practice. 86 Other outputs included reports and future plans. We label this episode ‘process outcomes’. These episodes structure the cases described throughout this report. We identify in the cases key themes related to (1) evidence use and (2) capabilities for use so that these can be drawn out in cross-case comparisons. For ease of reference, we indicate evidences used with bold text, and indicate capabilities with bold italic text.

We also summarise whether or not redesign projects were ‘successful’ at the end of the case study. As seen, we adopt three broad criteria to assess this: (1) evidence used effectively, (2) satisfied stakeholders and (3) objectives met. Process maps of the case are also drawn53 and will be presented at the end of each individual case study.

The findings from Seaport CCG are presented first in detail to familiarise readers with our approach to data analysis.

Seaport Clinical Commissioning Group

Seaport summary of findings

Local, universal and expertise-based evidence was used to frame care and capacity issues related to diabetes services at Seaport CCG. A number of experts engaged with these evidences (e.g. GP, diabetes nurses and consultant). Led by the clinical consultant, these stakeholders began to work to select a solution. Although not everyone was enthusiastic to begin with, the consultant framed the solution search in terms of a blank sheet, which meant the multiple stakeholders selected and evaluated universal, local and expertise-based evidences. From this, stakeholders agreed a new pathway in which the majority of diabetes care would be moved to the community setting. They also agreed a small amount of diabetes care that would remain in the hospital.

Process mapping: explanation

Process maps are split into two levels, with the top level containing evidence (shaded in white) and the bottom level containing capabilities (shaded grey). Evidences are shown in differently bordered boxes as follows: local evidence is in a box with a solid line and represented by (L); universal evidence is in a box with a dotted line and represented by (U); expertise-based evidence is in a box with double lines and represented by (E); and trans-local evidence is in a box with dashed lines and represented by (TL). When more than one evidence is used, a double box is presented with both evidence letters included.

The capabilities are shown in circles and labelled as follows: engaging experts (EE), managing expert collaboration (MEC), effective framing (EF), sourcing and evaluating evidence (SEE) and managing role expectations (MRE). If outcomes were achieved (effective evidence use, changes in practice and satisfied stakeholders), these are indicated in hexagonal shapes.

Process maps should be read left to right, following the arrows. Agenda formation, solution selection and process outcome episodes are indicated along the bottom. The rotated triangles indicate barriers to progress. The process map symbols are summarised in a legend in Figure 3 and the redesign process at Seaport CCG is shown in Figure 4.

FIGURE 3.

Process legend.

FIGURE 4.

Evidence use process at Seaport CCG.

Greenland Clinical Commissioning Group

Greenland summary of findings

Local and universal evidence was used to frame issues with patient care. The GP lead and the CSU manager decided to mimic Seaport CCG’s redesign (trans-local evidence). Efforts were made to engage experts in the solution search. The CSU manager had a role in selecting the evidence. There were efforts to manage fears and anxieties, but not everyone felt included. After nearly a year, we managed to conduct follow-up interviews in which we learned that the project had been ‘put on hold’. This resulted from the narrow stakeholder engagement and difficulties in searching and evaluating evidence. There were also questions about the evidence used to frame the problem. The redesign process for this CCG is summarised in Figure 5. Guidance for reading process maps is shown in Figure 3.

FIGURE 5.

Redesign process at Greenland CCG.

Horsetown Clinical Commissioning Group

Horsetown summary of findings

Horsetown CCG sought to identify how community-based services could be provided, framing this in terms of local (e.g. increases in procedures) and universal (e.g. more surgery than other places) evidence. Work was done to engage experts in CRG, who, in turn, questioned the evidence validity. CRG members were tasked with evaluating a variety of local (e.g. referrals to triage) and universal (e.g. START back pathways). This evidence search was narrowly focused on community-based options. The CRG reported that the evidence was difficult to interpret. Issues with expert collaboration and role expectations also problematised their work. A commissioner suddenly developed a new pathway, which was met with several criticisms. The pathway was redrawn (using trans-local evidence). At our last contact we were told that more work would be needed (e.g. defining MSK) to take the pathway forward. The redesign process for this CCG is summarised in Figure 6. Guidance for reading process maps is shown in Figure 3.

FIGURE 6.

Redesign process at Horsetown CCG.

Stopton Clinical Commissioning Group

Stopton summary of findings

Stopton CCG sought to pilot a new MSK triage. It used local and universal evidence to frame this as a care issue and did a lot of work to engage experts and manage collaboration. There were also clear roles, especially with the clinical assessment team lead sourcing and evaluating the evidences. The team used trans-local evidence to identify how the new pilot would run, and, bar some minor challenges, they seemed to be on board with the pilot. The transformation manager saw this as resulting from the fact that there was no real risk involved. The team ran the pilot and collected evaluation data. At our last contact, they had presented this to the SDC, which had requested an additional audit.

The redesign process for this CCG is summarised in Figure 7. Guidance for reading process maps is shown in Figure 3.

FIGURE 7.

Redesign process at Stopton CCG.

Overview

The second objective of our research was to compare and contrast across cases in order to gain insights into capabilities for evidence use by commissioning organisations. In doing so, we endeavoured, also, to understand qualitatively how and why variations in evidence use occur in practice. We began this with our coding exercise (see Chapter 2), which used the whole data set to identify what forms of evidence were being used by CCGs and which kinds of capabilities actually mattered to innovation (the service redesign) process. We have presented these for each of our cases above (and in Appendices 1–4). In this chapter, we extend this analysis by directly comparing patterns of evidence use across the eight cases. We then consider the presence and absence of capabilities across cases, before summarising the outcomes. Cross-case comparisons are summarised in Table 6.

Evidence use

As shown, local, universal, trans-local and expertise-based evidences are used together in redesign work. Evidence use is shown in the process maps. Rather than one type of evidence being more ‘authoritative’, with regard to carrying more weight and legitimacy, different evidences are used, often in tandem, in decisions. During the agenda formation and solution selection episodes, multiple forms of evidence were apparent in every case. Local evidence was often judged against universal evidence. At Greenland CCG, for example, local and universal evidences to show ‘amputation rates were quite high . . . when compared nationally’ (transformation manager). In some cases, expertise-based and trans-local evidences were also used. When describing process outcomes episodes, stakeholders made rather fewer references to evidence use. This finding should be considered with caution because only just over half of our CCGs (n = 5) actually began to implement changes. Many stakeholders also described plans to implement or pilot new models and then evaluate efficacy, so that process outcome episodes become more about creating new evidence than about implementation per se. This reflects the open-ended nature of the decision processes in this context; projects to improve services were rarely ‘finished’ but were ongoing endeavours in which work generated new forms of evidence and judgements that changed, or sometimes halted temporarily, redesign efforts.

This open-endedness of management decisions raises challenging questions about how to assess and understand variation ‘decision outcomes’, in our case in terms of whether or not they are evidence based. In other words, if ‘outcomes’ are continuously emerging, then decision-making is never quite complete, which makes it difficult to judge the efficacy of evidence use. We can, however, compare cases in terms of the processes themselves and see patterns in the extent to which (in the time frame observed) these are relatively smooth (or not) in so far as they allow actors to take action and proceed to the next step. Table 6 provides a summary of evidence use across cases and across episodes. In sum, the patterns of evidence use are as follows:

• Because it was used during agenda formation and solution selection in all eight CCGs, it can be said that local evidence plays an important role during initial redesign episodes. Of the five CCGs that managed to actually implement their service redesign, local evidence was mentioned in three. This was largely in relation to conducting evaluation (e.g. Chelsea CCG) or understanding local needs in more detail (e.g. Shire CCG, Stopton CCG).

• Universal evidence was used during agenda formation in seven CCGs. Stopton CCG was the only CCG not to explicitly use universal evidence in this episode. Universal evidence was also used during solution selection in all eight CCGs. Therefore, universal evidence is equally important during initial redesign episodes. Chelsea CCG was the only one to refer to universal evidence during process outcome episodes.

• In all CCGs, bar Shire, expertise-based evidence was explicitly used in solution selection episodes. Interestingly, expertise-based evidence was relied on in just two CCGs in agenda formation and process outcome episodes. This evidence was used to identify the problem (Seaport CCG) and to push a better way of doing things forward (Chelsea CCG). Expertise was then used to guide practice changes at both CCGs. Both Seaport and Chelsea CCGs were implementing community-based diabetes care (with Chelsea explicitly mimicking Seaport). Both were also led by a diabetes consultant, which may explain the emphasis on expertise-based evidence in these contexts.

• Last, trans-local evidence was used during agenda formation at Chelsea CCG, during solution selection at Greenland and Chelsea CCGs, and during process outcomes Coalfield and Horsetown CCGs. The reasons for using trans-local evidence varied, however. At Coalfield CCG trans-local evidence was used in terms of evaluating outcomes using examples from other countries, whereas at Horsetown this evidence was used to justify the new model.

Local and universal evidences are most commonly used, perhaps because CCGs are under pressure to implement evidence based (usually meaning universal evidence) in context. Whereas previous research suggests that local ‘trumps’ universal evidence, our findings suggest that the two operate together. We saw many instances in which local evidence is judged against a backdrop of universal evidence (albeit not always formally presented) and becomes ‘evidential’ because discrepancies are seen that indicate there is a problem. Thus, our findings suggest that local evidence needs universal evidence in order to be interpreted, and, on the flipside, universal evidence needs local evidence in order to be made to fit the context. Without local evidence and expertise, universal evidence lacks precision and applicability.

Expertise-based evidence was used across episodes but appeared to become particularly important in solution selection. Here, individuals’ experience of ‘what actually works’ played a particularly important role in concretising designs ‘on paper’ into designs ‘in use’. Hence, in those cases where we saw relatively more positive outcomes, we also saw the design being worked through, and championed, by individuals with significant experience and legitimate authority among other groups that were to be relied on to implement (or pilot) the redesign. Trans-local evidence was used across multiple episodes, acting as useful heuristics or short-cuts to redesign efforts. However, although it was often assumed that these were based on ‘good evidence’ (hence the label ‘best practice’), this was not always the case. This suggests that trans-local evidence, although helpful, needs to be interrogated and unpacked, not simply mimicked, or its ‘evidential’ qualities taken at face value, especially as the local conditions (e.g. population demographics, public health economics) vary significantly.

Capabilities for evidence use

Capabilities that enable evidence use in redesign work included ‘sourcing and evaluating evidence’, ‘effective framing’, ‘engaging experts’, ‘managing expert collaboration’ and ‘managing roles and expectations’. We distinguish between the presence of a capability, the failure to apply a capability and the absence of a capability in the data in Table 6. When a capability was present in the data, a tick is used (✓). A dash (–) indicates that the capability was not shown in the data, while a cross (✗) indicates that a capability was not applied. We see the distinction between failure to apply a capability and the absence of it in the data as important, because all capabilities may not be needed for evidence use in certain contexts.

In sum:

• Rutterford was the only CCG in which all five capabilities were shown.

• Seaport and Stopton CCGs showed four of the five capabilities. Here, ‘managing expert collaborations’ was not prevalent in the data. It may not be necessary to manage collaborations if issues do not arise, which would explain the absence of this capability in CCGs where multiple other capabilities were recorded.

• Coalfield CCG also showed four capabilities, bar ‘managing roles and expectations’, which was not recorded in the data, perhaps because the group leading the redesign had been working together over a long period of time. Because of this, roles and expectations may have been established, which would remove the need for them to be managed.

• Chelsea CCG showed three of the five capabilities. ‘Managing expert collaboration’ did not appear in the data, perhaps because the consultant led the project almost single-handedly. As mentioned, our study missed prior discussions, and perhaps disagreements, about community-based care. During our fieldwork such agreements had already been reached. ‘Sourcing and evaluating evidence’ was also not achieved at this CCG, but this may have been less of a concern because they had decided to mimic Seaport CCG.

• Shire CCG also showed three of the five capabilities, but in this case efforts to manage expert collaboration and manage roles and expectations were not successful.

• Both Greenland and Horsetown CCGs showed two of the five capabilities. Both failed to ‘manage collaborations’ or apply ‘effective framing’. Greenland CCGs also failed to engage experts’, while Horsetown CCG failed to ‘manage roles and expectations’.

Certain capabilities also appeared to be particularly important in certain episodes of work. Effective framing, for example, was noted in both agenda formation and solution selection episodes in those cases that had relatively positive outcomes. Expert engagement, where observed, also occurred on early in the process in relatively more successful cases. In most instances, a problem was identified and then redesigns leaders worked to engage other relevant stakeholders in fine-tuning their understanding of the central issues. Sourcing and evaluating evidence, managing expert collaborations, and managing roles and expectations all tended to be important when stakeholders were working to select a solution.

It is important to note that the capabilities we observed, while distinguished analytically, were also linked. For example, the way in which problems and solutions were framed also had implications for engaging experts, managing collaboration and sourcing evidence. Here, a ‘blank sheet’ approach helped to engage active contributions of experts and also helped to widen the search for information, because it did not pre-prescribe a particular course of action. However, using a completely ‘blank sheet’ also had drawbacks, leaving experts puzzled as to quite what their roles were, opening up room for conflict and ongoing disagreement around what actually comprised evidence, and generating inertia. Effective framing (blank but not too blank) appeared, in contrast, to let stakeholders set aside their own interests and values, which are at least partly responsible for failed collaborations, but also to contribute expertise and information focused on a shared goal (e.g. the patient). Our findings suggest, then, that it is not just the presence or absence of a particular capability that matters, but, rather, the ways in which capabilities are used together in particular contexts. As seen, lack of capability in one area (e.g. managing roles and expectations at Coalfield CCG) may not preclude positive outcomes where contextual conditions substitute for it in some way (e.g. in that case by having a long-established group that already had clear role expectations).

Outcomes

As discussed, commissioning decisions are characteristically open-ended and iterative (as, we would argue, are most complex management decisions). Therefore, there is often no clear end (or start) point. Here, a process approach is useful as it reminds us that ‘outcomes’ are always in flux (and can be ‘inputs’, if the focus is to explain the next step). This means that the success or otherwise of ‘outcomes’ is always subjective, being judged relative to where one is (or hoped to be) at that particular point in the process. Nonetheless, we can use qualitative indicators to compare cases in terms of whether or not process outcomes were relatively successful, when judged against actors’ own expectations. We thus assessed outcomes qualitatively in terms of evidence use, objectives met and stakeholder experiences, as shown in Table 6.

Comparing across cases

Objectives were broadly met in Seaport, Chelsea, Coalfield, Rutterford, Stopton and Shire CCGs. Of these, Seaport, Chelsea and Coalfield CCGs appeared to have the best outcomes. They met their objectives, we found few issues with evidence use, and there were broadly satisfied stakeholders. These outcomes occurred even when certain capabilities were not recorded [Seaport CCG (managing expert collaboration) and Coalfield CCG (managing roles and expectations)], suggesting that capabilities may not always be necessary.

Rutterford CCG demonstrated all five capabilities. Here, issues with evidence use were reported but overcome. Similarly, Stopton CCG showed four capabilities, but there were issues with evidence use. This case is slightly different from the others because the issues with evidence use arose from data it had collected to evaluate its own triage. In both of these CCGs, however, objectives were met and stakeholders were satisfied. Shire CCG showed only three capabilities, but still achieved its objectives and used evidence effectively. Stakeholders were largely dissatisfied, however, especially as the project was put to external tender.

Greenland and Horsetown CCGs experienced the worst outcomes; both failed to meet objectives, use evidence or satisfy stakeholders. Both lacked framing and managing collaboration capabilities. Greenland CCG failed to engage experts, while Horsetown CCG failed to search for and evaluate evidence. Thus, although redesign projects may certainly fail for all sorts of reasons, our data indicate that CCGs with fewer capabilities also experienced worse outcomes with regard to the redesign process overall. These data must be treated with caution, of course, as larger studies would be needed to evaluate the strength of any links between the two factors. Two main propositions can be drawn from these findings:

1. There is an inverse relationship between applying evidence use capabilities and redesign outcomes. That is, CCGs with fewer capabilities report poorer outcomes.

2. Context matters. The absence of some capabilities to use evidence did not guarantee poor service redesign outcomes. Rather, using capabilities as appropriate to the particular context seemed more important with regard to outcomes.

We conclude, then, that a tightly prescriptive ‘one-size-fits-all’ approach to capabilities for evidence use in redesign work is not appropriate. Instead, consideration is needed of what capabilities are required in given particular contextual contingencies.

Introduction

In this chapter, we report work on our development of a toolkit aimed at helping commissioning organisations improve capabilities to use evidence. To this point our research has identified four categories of evidence and particular capabilities to use evidence (Table 7).

We see these capabilities as being able to be nurtured, both at the level of the commissioning teams working on redesign projects and at the level of the organisation. Accordingly, the design of the toolkit takes two target audiences into account: (1) those responsible for the overall functioning and management of commissioning organisations (usually director level staff), and (2) those managing/leading local redesign projects (usually middle-level commissioning leads and the teams they assemble). We develop in our toolkit a structured set of questions aimed at each of these groups. The intention is not to prescribe a particular course of action, but to allow each of these groups to (1) reflect on how well they use evidence and (2) identify opportunities to improve their own capabilities to use evidence.

The toolkit may also allow practitioners to track the development and maturity of their own capabilities over time (e.g. during different stages of a project). Owing to its interactive features, the toolkit will prompt practitioners to think critically about the role of evidence in redesign projects. Our toolkit is centred on the five capabilities identified from the CCG data (see Chapter 4): sourcing and evaluating evidence, engaging experts, effective framing, managing roles and expectations, and managing expert collaboration.

Designing the toolkit

We developed our toolkit using feedback from stakeholders and examples in the existing data. In terms of stakeholders, we held two workshops to elicit feedback from individuals external to the research team. The first workshop took place on 1 March 2016, in which a small group of SSAP, including patient representatives, participated. We made a short presentation outlining the aims and objectives of the toolkit, including a set of questions targeted at different toolkit users. Stakeholders suggested we should make the following changes:

• remove the word ‘diagnostic’ from the toolkit, as this may have misleading connotations

• target different types of users within the toolkit (e.g. director-level CCG managers and redesign project leads) because these individuals will have different interactions and understandings of evidence and capabilities

• simplify and reword the language used in some questions to increase usability

• include the criterion ‘assessing local applicability and relevance of evidence’ to make clear the purpose of the toolkit.

On the basis of this feedback, other offline comments and emergent findings regarding the capabilities, we revamped the toolkit questionnaire. In addition to using stakeholder feedback, we looked at examples of toolkits developed by academics for practitioner audiences, including the normalisation process toolkit (http://normalizationprocess.org). This toolkit includes a number of user-friendly features, such as a radar plot, which visualises answers and enables users to understand potential strengths and weaknesses (in that context, implementation of new technology). The radar plot can also be used to visualise ‘performance’ against specific capabilities (e.g. showing high ‘expert engagement’ vs. low ‘effective framing’). We decided to include a radar plot in our toolkit to enable users to get a sense of how well prepared their organisations and teams are to use evidence. A consequence of this decision is that item wording needs to elicit answers that are not simply ‘yes’/’no’ or open-ended, but reflect different levels of capability (e.g. Likert-scale types of questions). We therefore took particular care about how items were phrased to make the production of a radar plot possible.

Against this background, we presented an updated toolkit at our second national workshop (18 May 2016). This was a much larger workshop, comprising 46 participants with backgrounds in a number of NHS organisations, including CLAHRCs, Academic Health Science Networks, NHS commissioning policy units, CSUs and foundation trusts. We presented the findings of our research and then used interactive exercises to elicit participants’ feedback about the toolkit. After we made a short presentation describing the toolkit, the workshop participants were split into five groups. Each group was asked to examine three or four questions from the toolkit for approximately 20 minutes and offer their feedback on these. We also supplied each group with sticky notes and asked them to give comments on question sets. For example, one group was asked to comment on the set of questions concerning ‘managing expert collaboration’, while another was asked to debate and discuss the appropriateness and practical usefulness of the set of questions concerning ‘engaging the expert’.

Owing to the interactive nature of this workshop, the feedback we gathered was very rich and immensely helpful. The workshop participants made specific and constructive comments on all aspects of the toolkit. The participant comments and suggested questions per capability are summarised in Table 8 and were used to inform the final toolkit development. Feedback informed significant changes, and the resulting toolkit is presented in Appendix 5. This toolkit would be accompanied by a summary table that defines different types of evidence and different capabilities with concrete examples.

Chapter summary

In this chapter, we described our toolkit for improving evidence use in commissioning. The development of this toolkit was shaped by our research findings, by the literature and by feedback from our SSAP and workshop participants. We have described the importance of appraising capabilities for evidence use, both within the team and at the organisational level. We have selected a radar plot tool to enable easy interpretation among users. It is important to note that this is just an overview of our toolkit. We also secured an Economic and Social Research Council Impact Acceleration Grant to refine the design and develop this toolkit (e.g. wording) into a user-friendly online learning resource. Work (6 months) began on this resource immediately after the completion of the current NIHR project. The resource ensures that our ‘actionable findings’ have the best chance of having an impact on practice. The work was being carried out in collaboration with Health Education England, the West Midlands CLAHRC and local AHNs. The resource is available at www2.warwick.ac.uk/fac/soc/wbs/research/ikon/commissioning/.

Discourses of use and users

During our interviews and visits it became increasingly clear that the meaning behind key terms used – such as ‘adoption’, ‘implementation’ and ‘impact’ – and the ways in which these were being operationalised varied across the producer organisation. These observations are important because previous research suggests that complex organisations are unlikely to constitute totally coherent entities. More likely, they may host a number of slightly different organisational ‘logics’ or subcultures. The result is a situation of institutional pluralism and culture complexity, which generates specific challenges for management. 87,88

From our initial interviews and observations, it appeared that different parts of the organisation operated slightly different visions and projections on how their products would be used in practice, and on what concerned commissioning groups. Most of the time, such visions were not openly discussed (although there were notable exceptions). They were, however, reflected in the different ways members of the organisation talked about how they could sustain the take-up and utilisation of evidence products and other forms of guidance. At least three of these local discourses of use and users were encountered in our study. We call them here the discourse of production, the discourse of audience feedback and the discourse of implementation consultancy and direct marketing. We use the term ‘discourse’ to reflect the fact that these broad views defined different ways of talking, choices of words and ideas, but also practical beliefs, predispositions and specific practices around how you deal with use and users. 89 It is important to reiterate that these three discourses emerged from a limited sample of interviewees (and observations), and all findings from this exploratory study would need further work to be developed, corroborated and possibly refined.

What real users look like

Before concluding, it is worth reporting a brief extract of our observation in the field, this time turning our attention to the actual conduct of the users. This is because, in spite of our extremely limited sample, we encountered a user whose orientation was very different from that mentioned during our conversation. The extract is from a visit to a CCG by one of the implementation consultants who we shadowed in the field (names have been changed for anonymity):

Mrs Shore arrives with Mr Rainer, a junior person from the governance unit. Mr Rainer will remain rather silent during the meeting while Mrs Shore unremittingly tells the implementation consultant all the amazing things they have done. Mrs Shore starts the conversation explaining that she is moving to larger practice and leaving the CCG. She will work on medicine management but also do some practice development. After a short attempt by the implementation consultant to present his organisation’s offering, Mrs Shore launches into what a long praise of the work of her CCG in relation to the use of technology to support evidence based change of service provision. She speaks using highly technical language, mentioning acronyms, showing thorough knowledge of evidence-based products and guidelines. For long stretches the implementation consultant listens to Mrs Shore’s enthusiastic account. Mrs Shore starts by explaining the work they have done to use the EMIS [Egton Medical Infomation Systems] system to identify patients at risk. She explains that they modify the EMIS so that when certain input suggests people at risk, this is flagged automatically to the GP. The implementation consultant is able to chip in briefly and add that ‘we have a guideline for this . . . it includes a scenario . . . I did a session with another CCG nearby’. Unperturbed, Mrs Shore explains how recent guidance on drug safety was also included in the EMIS system (in the process she ‘recites’ the content of the guidance). Mrs Shore continues by affirming that she is a big fan of systematisation: ‘we have an automatic audit . . . we have a programme . . . we do it already’. She explains that whenever they extract data they compare with ‘previous studies’ and then put in place actions to educate their clients. She gives an example of how they worked with some of the care homes in the area where certain indicators were not as good. She states that ‘our data are brilliant’. All the implementation consultant can do is to nod and say ‘good stuff’.

Mrs Shore appears to be quite a unique user who does not fit with the implementation consultant’s a priori expectations. She is not resisting the use of evidence; quite the contrary, she endorses it to point of being able to recite pieces of guidance. Yet she is keen to make evidence-based guidance ‘disappear’ for frontline clinicians by embedding it in the local general practice records system. Although her interaction with the implementation consultant is not as one would have expected (she is telling things to the consultant, rather than the other way around), this is motivated by a desire to demonstrate (and showcase) her CCG’s capacity and skill. The implementation consultant is granted the type of reception one would give to a highly ranked authority.

We mention this encounter because Mrs Shore was a surprise to us and, more importantly, because people like Mrs Shore were never even remotely imagined for us during our conversations with members of the evidence-producing organisation. Our hypothesis, which would, of course, require further work to be corroborated, is that a user of this type could hardly have been inscribed in any evidence product, as she did not enter the collective imagination of the evidence producers. Mrs Shore reminds us that the only way to design products around real end-users is to go out there and see what they do (something that the producer organisation is starting to do).

Chapter summary

We set out to understand how teams at one organisation that produces evidence-based products think their products will be used in general, and by NHS commissioning organisations in particular. Owing to the very small sample and the limited duration of our engagement, we are in a position to offer only a few brief considerations to inform future research.

First, different (in this case, three) discourses of use and users were detected between teams in the same evidence-producer. As we suggested above, this can be expected given the type of organisation, and the presence of complexity and pluralism is neither good nor bad. Pluralism can be a source of innovation or internal conflict. Diverse discourses and subcultures can co-exist as long as they are brought to the surface and managed at a strategic level.

Second, our exploratory study suggests possible historical parallels with other organisations in the private sector that, over the years, have transitioned from seeing themselves (and acting) as makers of products to providers of services. A notable example is IBM, an organisation that over two decades went through a remarkable transformation from a struggling seller of hardware to a successful broad-range provider of solutions. 90 Our study raises the question of to what extent producers of evidence-based products in health care are going through the same process, and whether or not what we observed was the first trace of this process happening. The industrial context is quite different, of course, but comparisons with historic cases may be a useful means by which to reflect on one’s own identity and try to manage such processes.

Third, and finally, our exploratory study raises questions about the potential benefits of a more explicit and broad engagement of evidence producers with what is loosely described as ‘customer-centric design’ or ‘design thinking’. Brown and Wyatt91 refer to design thinking as an approach to innovation that draws from the designer’s toolkit to integrate the needs of people, the possibilities of technology and the requirements for business success. The approach is supported by a growing set of techniques and engagement modes, some of which we recognised to an extent within this evidence producer. Design thinking puts understanding of what users do, where they are and their active, direct involvement at the core of the development of new product and services. In this way, implementation issues can be designed out rather than grappled with at the ‘end of the journey’ of evidence, and users such as Mrs Shore can become assets to the circulation of products rather than outliers.

Mobilising evidence

The use of ‘authoritative’ evidence (from NICE) is a central consideration of our study, but it was clear that this needed to be coupled with other forms of evidence, identified as local, expertise-based and trans-local. Although different labels can be applied, the categories of universal, local and expertise-based evidence mirror those found in previous work. 23 Rigorous scientific evidence (which we term ‘universal’), evidence from within the local area (which we term ‘local’) and evidence from knowledgeable stakeholders (which we term ‘expertise based’) have been repeatedly shown as important in the literature. The category of trans-local evidence (i.e. evidence created in one place but used in another) is new. Although this resembles what other studies have labelled ‘best practice’ examples,1 our cases indicate that this evidence had often not been actually demonstrated as ‘best practice’ in its original setting before it was adopted and applied in our cases. Hence we think that ‘trans-local’ is a better label, one that also acts as a caution to the assumption that examples that have been tried elsewhere are actually ‘tried and tested’.

Although there were clear differences between the four categories of evidence, in reality they were used in tandem. We showed, for example, that stakeholders do rely on universal evidence (e.g. guidelines on amputations) to understand local evidence and performance data. Conversely, universal evidence needed local evidence to make them applicable to the context of adoption. Evidences are also coupled in more indirect ways. In terms of expertise-based evidence, for example, stakeholders may be asked directly for their opinions. If this is not the case, their interpretations of, and reactions to, other evidences (e.g. local and universal) are likely to be informed by their own expertise anyway. In some cases, experts questioned even NICE guidelines with legitimate authority among their peers. Thus, categories of evidence need to be understood, not as mutually exclusive (e.g. with one ‘trumping’ the other)1 but as tightly coupled, being used interdependently for sense-making.

In addition to understanding what evidence is used, our qualitative research design provided rich insights about these evidences. We know, for example, that commissioning stakeholders, in many instances, were less concerned with the properties of evidence and more concerned with accessibility, relevance and applicability to context. Trans-local evidence, which served as a shortcut to solutions in many instances, may in itself be testament to the reality of evidence use in this context. Indeed, stakeholders who opted to mobilise this type of evidence were often swayed more by the fact that certain CCGs had managed, in fact, to implement a change, than by the efficacy or outcomes of change. Indeed, there was rarely ‘any long-term evidence’ available (e.g. Greenland CCG). Choosing a solution because it has been applied elsewhere, without clear knowledge of efficacy, is likely symptomatic of the highly politicised, time-pressured environments in which health-care managers work.

It is interesting that these evidences have been shown as important in many studies, but issues with mobilisation still persist. 3,5,12 The inherent challenges of mobilising NICE guidelines, for example, have been described by our stakeholders and in the wider literature. 92 Whereas some kinds of universal were mobilised (e.g. NICE evidence, Five Year Forward View, QOF data), it was apparent that other kinds of universal evidence – Cochrane reviews, for example – were never referred to in our data.

Our case studies also offered insights into the challenges of mobilising evidence in other forms. Local evidence, for example, was often difficult to access and interpret. Routes for accessing this kind of evidence were often quite ad hoc and interpersonal, and so depended on the right experts being involved at the right time. Variations in recording and coding data (e.g. between primary and acute care) also posed problems for using local evidence in several CCGs (e.g. Greenland CCG). Structural issues related to the use of different systems to store local evidence also problematised the use of this evidence (e.g. Chelsea CCG). Similarly, the mobilisation of trans-local evidence seemed to be somewhat of a lottery. Although there are centralised sources of ‘best practice’ examples (e.g. Right Care Casebooks), our stakeholders did not describe using them. Instead, stakeholders were informed through more random avenues, including attendance at presentations (e.g. Greenland CCG) and experiences in other settings (e.g. Chelsea CCG).

These prolonged issues with evidence mobilisation may lead to variation and fragmentation in evidence-based decisions. 93,94 One answer is to add to existing calls for CCGs to be supported to use evidence. We know, however, that commissioning stakeholders are not always keen to use supports (e.g. CSUs), because they are costly, and also because they often lack local knowledge and relationships (relative to practising clinicians). 95 By identifying capabilities, in contrast, we show how evidence use can be enhanced from within the redesign group and the commissioning organisation itself. Engaging the right experts (including patient groups) in agenda formation, for example, is seen to ensure access to a wider range of evidences. 11,23 It helps, also, to ensure that stakeholders who will be affected by the change are represented in commissioning design work, thereby increasing the likelihood that a change will be accepted and actually implemented. Collaborations among experts need to be actively managed, however, and the evidence that they ‘bring to the table’ needs to be judiciously evaluated.

Before considering implications for improving capabilities, there is one caveat to our findings. Our SSAP has, quite rightly, advised that we consider how our research design may have influenced our findings. We chose to study how CCGs redesign diabetes or MSK services. This design meant that our research captured large-scale/-scope processes of change, where clinical need, cost, patient satisfaction and staff satisfaction, as well as a host of other factors, have to be considered. With projects of this scope, a multitude of evidences become relevant. If we had, instead, focused more narrowly (e.g. on the redesign of a particular part of the patient pathway), a different pattern of evidence use may have been observed. In this case, our SSAP suggested that universal evidence related to clinical guidance or best practice (e.g. NICE) may have been at the forefront of decision-making. With this in mind, we situate our findings as an overview of evidence use in redesign work, and encourage further, perhaps more targeted, investigations of the topic in future research.

Capabilities to use evidence

The main purpose of our study was to understand capabilities that enable evidence use, that is, practices and conditions that allowed decision-makers to have a solid evidence base for their decisions. Note that, here, our focus was on capabilities to use evidence, in particular, not on capabilities to manage change processes, in general. Although much is known about managing change in health care, including the challenges of dealing with complex, so-called ‘wicked problems’, there has been little focus on specific capabilities to use evidence. 96 Evidence use is only one aspect of any change process, however, so links between process outcomes and capabilities to use evidence are not direct and need to be treated with caution.

This said, we identified five capabilities that appeared in our cases to encourage the use of different types of evidence in decision-making: ‘sourcing and evaluating evidence’, ‘effective framing’, ‘engaging experts’, ‘managing expert collaboration’ and ‘managing roles and expectations’. We have discussed the relevance of each of these capabilities to evidence use, and also the interdependencies between them, in the previous sections (e.g. see Chapter 6). To our knowledge, identifying and collating capabilities for evidence use by commissioning groups conducting redesign work is a unique contribution of our work.

Interestingly, these capabilities reflect broader health-care management literature. Scholars have long argued that evidence is contested, negotiated and socially legitimated. 17 Far from being objective, evidence use is a highly political, tense and anxious process. 85 Reflecting this, we showed that managing expert collaboration and expectations is an important capability for evidence use. Similarly, the literature suggests that who presents evidence and how evidence is presented informs use. 7,46 The perceived relevance of evidence is also important. 46 Our cases showed the importance of framing evidence to resonate with stakeholders (e.g. the blank sheet approach). Overlap between our findings and the literature suggests that these capabilities are an avenue through which more general improvements in health-care management can be made, including enabling evidence-based decision-making.

Although our five capabilities were identified across cases, we do not claim to have created an exhaustive list. It is likely that other capabilities for evidence use in CCGs exist, and we encourage further, larger-scale studies to explore these. In addition to identifying new capabilities, more research to examine whether or not those shown in the present study are generalisable to other situations of health-care management would be beneficial, as would closer examination of the different ways in which capabilities can be developed in practice. We showed, for example, that engaging experts is a capability for evidence use. While commissioners made reference to the often informal ways in which engagement can be achieved (e.g. ‘sticking your head round the door before every meeting’), going forward, researchers may want to consider exactly how these capabilities can be developed.

Finally, research might also consider how capabilities vary across project contexts. We have already seen that not all capabilities are always relevant, and we are conscious that our design will have influenced the capabilities identified. CCG redesign work requires individuals to work across boundaries, and so it is not surprising that capabilities related to engagement and collaboration management were so frequently noted in our data. Had we chosen a less complex management situation (e.g. evidence use within a homogenous professional group,97 it is possible that other capabilities would have been as important. That said, existing literature tells us that health-care management problems are typically very complex, so our choice of empirical setting should mean that the findings have broad relevance.

The cross-case comparisons allowed indicated that there does seem to be some relationship between capabilities to use evidence and process outcomes, albeit indirect (as noted above). Put simply, we saw that CCGs with the narrowest range of capabilities also had the worst outcomes. Importantly, however, our data suggest that the frequency of capabilities was not directly proportional to successful outcomes. Rather, applying capabilities as appropriate in context may have a more positive impact. Just as evidence use may vary across CCG contexts, so too might conditions and practices that enable evidence use. This suggests that, instead of a broad-brush, prescriptive approach to improving capabilities, it is more fruitful to support commissioning groups in understanding their own capabilities for evidence use so that they can consider whether or not the range of capabilities they are deploying is appropriate for their context, or whether or not there are gaps. The ability, at the meta-level, to identify and apply capabilities when needed is an important practical application of our findings. The toolkit we have begun to develop from the findings is just one step in supporting CCGs to do just that.

A roadmap of capabilities for evidence use in Clinical Commissioning Groups

Our research findings to this point attest to the highly iterative nature of the evidence journey in NHS organisations and to the messy and fluid ways in which various forms of information come to be used as evidence. We have shown how multiple kinds of evidence are coupled together across different episodes of the innovation process. This means that improving the ability of authoritative evidence (e.g. NICE guidelines) to actually change practice necessarily means improving its use alongside other forms of evidence in local contexts. These different forms of evidence are shown in our ‘roadmap’ for evidence use in Figure 8. Although these evidences have definitive characteristics, they contribute to redesign decisions when used in tandem.

FIGURE 8.

Capabilities roadmap.

We also sought to understand the capabilities (practices and organisational conditions) that enable evidence to be deployed in context. Just like the evidence discussed in the previous section, capabilities also come in many shapes and forms, and were often about managing and understanding social relationships and politics among redesign stakeholders, reflecting the highly politicised nature of EBMgmt in health care. We also observed interesting patterns in terms of the way in which these capabilities to use evidence mediated outcomes of redesign work. At CCGs where fewer capabilities were recorded, worse outcomes were also shown. However, it is not simply the case that more is always better. Rather, the importance of each capability to successful outcomes depends on the context. In some of our cases redesigns were quite successful, even though certain capabilities were not present. It can also be assumed there is a hierarchy, or, at the very least, an interdependence, across capabilities. Framing, for example, would probably have little impact on evidence-based redesign work without engaging experts.

We conclude that, although there is no ‘silver bullet’ for evidence-based commissioning, support to improve awareness of different kinds of evidence and capabilities would be beneficial. In addition, we have shown varied understandings of evidence use and users among those involved in producing and publishing evidence-based products. These understandings are inscribed in the authoritative evidence produced by these organisations and may also need fine-tuning in order to support the translation of evidence in practice.

Figure 8 draws these findings together as a ‘roadmap for evidence use’. We construct this from our original conceptual framework (drawn from existing literature) in order to show how our findings build on the existing corpus of knowledge and how they developed through our research project. This roadmap shows that:

• Multiple evidences are used alongside each other throughout the redesign process. No one evidence is given precedence over another, but used together they aid commissioning groups in decision-making.

• There are challenges and barriers to evidence use in health care, but certain conditions and practices across the organisation and team can enable commissioning organisations to make more evidence-based decisions. Observations can also be made about the timing and relationships between capabilities. Engaging experts and effective framing, for example, are particularly important during agenda formation work. These capabilities have a mutual dependence, too, with expert engagement itself often increased through effective framing, while framing efforts may be redundant if experts are not present.

• Importantly, the presence of these capabilities to use evidence appeared to mediate project outcomes. Specifically, in commissioning groups where fewer capabilities were shown overall, worse outcomes were reported.

To stay true to the nature of our findings, our roadmap, drawn schematically in Figure 8, is non-linear in nature. Metaphorically, it is perhaps best understood not as a single road, but as a ‘spaghetti junction’, where dynamic capabilities for evidence use are combined in practical settings.

Toolkit: improving capabilities to use evidence

We encourage the application of our findings in practice through the creation of a toolkit (see Chapter 7) and will develop this into a user-friendly and accessible format after the completion of the current project. Specifically, we have secured an Economic and Social Research Council Impact Acceleration Grant that will enable us to create a user-friendly, online learning resource that incorporates the main findings, as well as illustrative case stories, examples, links to other resources and the toolkit. We will consult further with key stakeholders on this impact work, initially through our collaboration in the project with Health Education England, and the West Midlands CLAHRC that is supporting us in this work. We will also be sure to avoid the duplication of existing supports and networks, instead aiming to link to these to ensure that the toolkit is as widely accessed as possible. The Economic and Social Research Council Impact Acceleration Grant is an important outcome of this project, as it will help to ensure that our findings have the best chance of actually impacting on practice.

Considerations and future directions

Although several important insights into EBMgmt in CCGs have been gained, there are of course further considerations to be taken into account. We have already discussed some of these, including the need for further research on the contextual factors shaping evidence use and evidence use in other health-care management settings. It is important, also, to consider how we conceptualise ‘authoritative’ evidence. At the outset of this project we defined guidance produced by NICE as a highly important, albeit not the only, source of authoritative evidence for CCGs. Although NICE guidance was certainly used in our cases, it was rarely given special acknowledgements or consideration. Instead, it was used in tandem with other, usually considered less ‘authoritative’, kinds of evidence.

Whereas health-care research tends to assign authoritativeness according to the attributes of evidence sources, the Oxford English Dictionary defines authoritative as ‘able to be trusted or reliable’. 98 This latter definition seems closer to the realities of health-care management, where authoritativeness is assigned not through modes of creation, but through an ability to mobilise evidence in making judicious decisions at the right time. Although we are not the first to notice, our observations reiterate the importance of reconsidering our conceptualisation of ‘authoritative’ evidence so that the realities of evidence-based health-care management are seriously noted. 2,22

Some other considerations arising from our findings relate to temporal aspects and generalisability of findings. Throughout this report we have discussed innovation with regard to service redesign, in which we expected to follow the use of evidence from inception through to changes in practice. Redesigns were, however, often very slow, iterative and interrupted. Some cases did not make changes within the time frame of our study, as they had anticipated. Other cases did implement changes, but re-evaluation for future work was almost instantaneous. Rather than following a process with distinct starting and finishing points, our research steps into an ongoing stream of redesign work. Unfortunately, the time required to implement change and our study time frame did not coincide so certain episodes of work (e.g. ‘routinisation’) were not captured. A longer study period may have enabled this, although our time frame of 30 months is already quite considerable. Instead, future research might target certain episodes of work to understand evidence use within them.

Another point to consider is the generalisability of our findings. We chose to sample eight CCGs in order to maximise opportunities to explain variation, while attending to the contextual and processual dimensions of redesign work. Doing so allowed us to achieve rich, qualitative insights into evidence-based redesign work. Although we sampled CCGs operating in different parts of the country and serving different populations (see Chapter 2), the generalisability of our findings is, of course, limited by our empirical setting. In future work, generalisability could potentially be achieved by using a bigger sample and a more quantitative approach. Doing so might enable us to understand statistically how outcomes vary according to evidence use and capabilities.

Finally, fieldwork was conducted at a time of significant reorganisation of NHS commissioning. Given the political context, more reorganisations are likely to occur in the future. This, of course, may raise questions about whether or not, and to what extent, our findings will still be relevant. To answer these concerns, we have focused on the process of commissioning, not on the specific organisation and governance of CCGs per se. Given that evidence-based commissioning processes will always be required in the NHS, we believe that our findings will be sustainable, even if further structural change takes place.

Summary of findings

We studied redesign processes in eight CCGs where four categories of evidence – local, universal, expertise-based and trans-local – were used, often in tandem, to make sense of tasks. Evidence use was often challenging, but certain practices and conditions were shown to be enabling, including sourcing and evaluating evidence, effective framing, engaging experts, managing expert collaboration and managing roles and expectations.

Implications for practice

Our findings showed clear implications for commissioning in practice, especially with regard to the use of evidence, the importance of social capabilities and support for decision-makers. These can be summarised as follows:

1. Findings show that evidence is not just accepted as relevant or reliable. Various pieces of information are brought forward and contested or applied. There were issues of perceived relevance (universal evidence), quality (local evidence), quantification and evaluation (expertise-based evidence) and access (trans-local evidence). Support to understand and address these issues might enable better evidence use in practice, which is important given the interdependence across the four categories that was shown in our findings.

2. Although redesign projects are complex, certain social conditions and practices were shown to enable evidence use. Although some capabilities identified in our study relate explicitly to evidence (e.g. sourcing and evaluating evidence), many related much more to managing social interests, conflicts and interactions (e.g. engaging experts and managing expert collaborations). Rather than technical capabilities (e.g. data searching and quality appraisals), our findings suggest that developing and supporting these social capabilities is key to evidence-based CCG redesign work in practice.

3. Cross-case comparisons suggest that successful redesign projects – in terms of evidence use, stakeholder satisfaction and meeting objectives – are informed by capabilities being in place as needed. Quantity, however, is not a measure of quality, and not all capabilities are necessary in all contexts. Managing collaborations may be less necessary, for example, when project teams have worked together and already have established norms and role expectations. It is important, then, not only to support the development of social capabilities but also to empower stakeholders to recognise when and where these are necessary.

4. Supporting health-care managers and commissioning staff to recognise, access and evaluate evidence drawing on their own capabilities relevant to the particular task and context might be an effective way of enabling evidence-based decisions. As NHS organisations are frequently restructured, this may be more sustainable approach than one that prescribes that certain evidences or certain sets of capabilities need to be utilised.

5. Much is to be gained if evidence providers consistently question themselves with regard to which model of user they inscribe, intentionally or otherwise, in their guidelines and other products. The exploratory study also suggests that potential benefits might be derived from a more explicit and broad engagement of evidence producers with the practices and techniques of ‘customer-centric design’ and ‘design thinking’.

Research recommendations

1. First and foremost, our research ‘begins with the needs of knowledge users’ as recommended by Jacobson et al. 99 We documented journeys of evidence through realistic accounts of redesign work to provide a detailed account that has been missing from the literature to this point. Although we see these insights as critical to achieving evidence-based health care, they are not likely to be universal across contexts. Further research is, therefore, needed to understand the nuances and caveats in difference contexts so that commonalities and differences can be identified.

2. We expected universal evidence to be an antecedent of redesign work, given its place in the evidence hierarchy and in studies of evidence-based health care to date. 29 We found, however, that universal evidence often set parameters around projects and is used in tandem with other types of evidence. This finding demonstrates how evidence is actually treated in practice. 2,22 It also builds on existing knowledge by showing that certain types of evidence do not ‘trump’ others;1 instead, they are often used in tandem in sense-making.

3. Much of the research on evidence-based health care has suggested structural or technological improvements (e.g. knowledge management systems). 26 We respond to work by scholars such as Ferlie et al. ,100 who called for a move, instead, towards ‘core competences’. We identified five capabilities that seemed to support better evidence use. Future research might build on this to consider, for example, how exactly capabilities come into existence and are maintained in commissioning settings. Future research might also more systematically measure the extent to which they are present and the impact of these on commissioning effectiveness. Our research provides a snapshot of this impact and should be used as a foundation for more in-depth analysis.

4. Cross-case comparisons suggest that certain capabilities impact on evidence use, stakeholder satisfaction and meeting objectives. Going forward, more nuanced insights might be gained from looking exactly at what parts of the process impacted stakeholder appraisals, or if capabilities had more or less impact on the ability of teams to meet certain objectives. Other outcomes could also be considered (e.g. timeliness and patient experience).

Contributions of authors

Jacqueline Swan (Professor, Organisational Behaviour) acted as the Business School Lead and was responsible for the overall project co-ordination and management, project design, and supervision of research fellows and administrative/clerical staff. She was also involved in data collection, analysis and dissemination events, co-ordinating the SSAP, report writing and publications.

Emmanouil Gkeredakis (Assistant Professor of Information Systems) was the Lead Research Fellow. He led on the qualitative research design, data collection and analysis. Emmanouil left the project to take up his Professorship at the Business School but continued to allocate 10% of his time to the project.

Rachel M Manning (Research Fellow, Warwick Business School) was the Second Research Fellow, replacing Emmanouil Gkeredakis on the project. She led on data collection, analysis, dissemination activities, report writing, workshop organisation and publications.

Davide Nicolini (Professor, Organisation Studies) led on ethics approval for overall project and was involved in project design, data collection, analysis and write-up.

David Sharp (Senior Vice President Growth Strategy, Optum International) began the project as a Director of NHS England and provided advice on project design, case selection, dissemination and analysis.

John Powell (Professor, Public Health Medicine) was involved in project design, and contributed to the interpretation of data and the final report.

Data sharing statement

Data that have been anonymised for confidentiality can be obtained from the corresponding author on request.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health.

Case study overview

We first contacted a GP professor in diabetes at Rutterford CCG. Through him, we met stakeholders who had been involved in the diabetes redesign. We spoke with a diabetes nurse consultant and a CCG transformation manager. We later followed up with the diabetes nurse consultant and interviewed the chief accountable officer at the CCG. At the time of our visit, the CCG had spent the previous year developing recommendations on ‘delivering the best possible care’. Increased numbers of patients with diabetes, increased complications and increased spend were primary concerns in the CCG. The CCG had secured some funding that could be used to buffer any upfront losses from changes to the diabetes pathway. They also had a diabetes education programme in place for upskilling community staff. With these resources in place, a transformation team was put in place. They faced some challenges, which were mostly overcome by the creation of subgroups. These groups collected, evaluated and reported on evidences related to their specific disciplines. Each group made recommendations about the pathway that were then collated into one overall report. The team decided to improve care ‘downstream’, that is, at the beginning of the pathway, in order to prevent complications, improve care and, ultimately, save money. At our last point of contact, which was approximately 2 years after the first interviews, around one-third of practices had attended the education programme, a specification for new services had been written, and they were in the process of evaluating referral data to understand the impact of their redesign work.

Agenda formation

Stakeholders at Rutterford CCG broadly framed problems with care, capacity and cost using local and universal evidences. The diabetes nurse consultant described how there had been a ‘spurt in chronic diabetes’ that meant ‘to even offer basic care service in the future’ changes in the pathway were necessary. Current services were seen as ‘very acute centric’, with the transformation manager describing how they had ‘seven diabetology consultants and even if we appointed another 70 we still wouldn’t have enough to . . . see absolutely everybody’. As a result, ‘11–12% of the NHS budget [was being spent] on complications’ (report, 2013).

Duplication of work was also problematic because it contradicted the universal evidence. This violation was understood in terms of local evidence that showed that they had ‘two foot clinics [that] both happen on the same day’, which violates NICE guidance that patients should have ‘access to a multidisciplinary team within 24 hours’. Local evidence also showed that there was variation in care across clinics, with one operating in line with national guidance (around one patient in every six was seen again), while the other had a ‘follow-up ratio of 1 to 10’ (transformation manager). Overall, these evidences suggested that current services were a ‘poor use of resources, not great value for money and [having] not great outcomes for patients’ (accountable officer).

Recognising these issues, the diabetes nurse described how they ‘formulat[ed] a transformation steering group to understand actually who is likely to be able to make decisions to mobilise this plan [to address the issues]’. She wanted this group to engage experts and offer them an ‘opportunity to . . . get together and really think about what diabetes care in [the area] should look like’ (diabetes nurse). The transformation manager saw the group as ensuring that ‘everybody had a voice . . . was listened to’ and that ‘all the right . . . all of the stakeholders’ were involved. There was broad involvement in the group (there were upwards of 35 in one meeting alone; diabetes nurse).

Selecting a solution

Having established the groups, the transformation manager endeavoured to develop a shared frame on the search for a solution. The nurse commented that they wanted to bring ‘about change that people agreed to and that actually happen, rather than people coming up with ideas but it never going anywhere’. Stakeholders were asked to use expertise-based evidence take a ‘step back [and think] . . . what would [the service] look like if we had a blank piece of paper’:

. . . we tried not to look at this is what we’ve got, this is how we need to change it because that keeps you a bit stuck . . . so we did very much a ‘if we could do anything what would it look like?’ Obviously you can’t do anything, there are restrictions in terms of finance and people and bodies. But in the beginning, we did a lot of visioning and a lot of blue sky thinking and a lot of ‘what would we really want it to look like?’

Diabetes nurse

The diabetes nurse commented that these discussions took place at a time when there was a lot of ‘talk around the ‘left shift’, that is . . . moving non-acute services from the acute trusts back to primary care’. She referred to the universal evidence, including NHS ‘Better Care Together’ (www.bettercaretogether.co.uk) and Monitor’s ‘Care Closer To Home’ (www.gov.uk/guidance/moving-healthcare-closer-to-home). Each of these documents focused on ‘the repatriation of people from secondary care into primary care, reducing hospital admissions, length of stay’ (diabetes nurse). Because of this, the solution search used a blank sheet, but it was not completely blank.

Two factors in the broader landscape were noted as critical to moving beyond agenda formation: funding and education. A transformation fund to support service redesign planning was won from ‘a joint bid between the CCGs and the [hospital]’ (report). They were awarded £1M to ‘do a full pathway review of diabetes care in the region’ (transformation manager), which protected them against financial penalties. Because of this, the work was ‘de-risked’:

The acute trust potentially [could] lose out on income which means that we had to be able to financially compensate them for at least for year 1 . . . if they’re going to lose a couple of million pounds it has a significant impact . . .

Transformation manager

The transformation fund was described as giving ‘extra impetus’ to an education programme for primary care that had already been created in the CCG (transformation manager). Educating primary care stakeholders was seen as vital to avoid putting patients in ‘a risk situation’ (transformation manager). Education was spread ‘right from health-care assistants, right the way up to practice’ so that ‘they feel valued’ (diabetes nurse). These supports meant the group was able to think about ‘how that change might be managed’ (diabetes nurse). The diabetes nurse saw education as a way to strengthen the redesign because:

. . . if people don’t feel competent and skilled to look after [a new way of providing care] it fails and it doesn’t work . . . the people that are doing the caring have the skills and the expertise and competency to do that. If you’ve got that it’s much more likely to succeed . . .

Three main ‘challenges . . . disagreements . . . debates’ arose. First, ‘change is frightening . . . it can make you feel a bit insecure’ and so some stakeholders were hesitant to participate (diabetes nurse). Second, stakeholders ‘at the beginning . . . [had] very fixed ideas about what they wanted to see at the end of the project’. This limits the scope of the evidence search and precludes them from contributing fully to the blank sheet (transformation manager). Last, professional boundaries were a challenge, with secondary care not convinced that primary care could take care of patients (diabetes nurse). Conversely, primary care saw secondary care as ‘holding on’ to patients longer than necessary:

[Secondary care] feel that they are probably the only people who can look after anybody with a foot ulcer. So they will hang on to that person . . . until such time as that ulcer is completely cured and then … maybe we could just hang onto you a bit longer . . . patients essentially went into a foot clinic and stayed there . . .

Diabetes nurse

Despite these challenges, the transformation manager described how she continued to push for the use of a ‘blank sheet’ because she felt that ‘unless we understand the whole way across the pathway then you can’t deal with a particular part of the pathway unless you can see the whole picture’. A variety of approaches was taken to manage expert collaboration and overcome challenges. Among these, informal approaches such as ‘having a chat, taking them out for a meal, having a walk with them’ (accountable officer) were noted as useful. Another approach was to frame the problem so that ‘everybody start[ed] from the point of view of the person with diabetes and what they needed and what would be best for them and how that might be best provided’ (transformation manager).

The most impactful approach to managing expectations, however, was the formation of ‘task and finish’ subgroups. These subgroups were tasked with identifying ‘gaps in provision and making recommendations’ for improvement (report, 2013). They had to ‘map out exactly what everyone provided [bringing] . . . everything onto the table’ (diabetes nurse). Each subgroup was to consider ‘what are we good at, what do we do well, what aren’t we good at, what might be the reasons for that’. They had to look at local evidence (e.g. patient numbers) and utilise expertise-based evidence to identify potential areas of improvement. The groups were also asked ‘is there any evidence, is there any models of care elsewhere, are there any best practice . . . that’s been published in the literature that we might want to adopt and change’ (i.e. universal evidence; transformation manager). The following quotation illustrates an example of the work one subgroup:

We took antenatal care we looked at, you know, clinical outcomes, you know, what sort of levels of care people had when they came into antenatal, were they referred early, were they referred late, you know, were they planned pregnancies, unplanned pregnancies, what were their care during pregnancy, what was their outcome.

Diabetes nurse

The creation of subgroups was a considerable step in managing expert collaboration because stakeholders were asked to identify, evaluate and present evidence from within their field and alongside their peers. Thus, issues around power and interprofessional working are removed so that work can be completed.

Although evidence use seemed to benefit from subgroups, challenges remained. Subgroups had problems with evidence quality. The diabetes nurse described that while ‘there has always been . . . a culture of looking at our data and . . . auditing the service . . . local evidence . . . [was] the one thing that was actually really, really difficult to get’.

Even when evidence could be accessed it was somewhat ambiguous and even outdated:

In secondary care diabetes was either [labelled] under diabetes, integrated medicine, specialty medicine or diabetes and endocrinology. So the whole idea of untangling some of that was incredibly difficult . . .

Diabetes nurse

Working with data that was more than a year old so it wasn’t particularly helpful . . . we tried to do was understand . . . what was the current activity, and then secondly what was our current spend. But diabetes gets wrapped into other conditions. So even if we’re doing something like ‘what’s our drugs spend?’ it became a best guess. Because . . . if the patient had diabetes they were likely to be on the following combinations of therapies so do you count them or do you just count the diabetes ones? Diabetes in the secondary care setting is, it’s just so tied up with loads of other things . . .

Transformation manager

Stakeholders also noted that there was ‘too much’ universal evidence, which often ‘became this minefield because . . . how do you synthesise all this so that you’d end up with something that is succinct so that you can deal with it?’ (transformation manager). Contextualising universal evidence was also problematic, with the diabetes nurse describing it as ‘useful where it exists’, but:

. . . sometimes you get some clinical disagreement with the decisions that NICE has made and you get some local needs that are very specific. So you need to provide something different . . . some areas of the city have got a very young population so there are issues around type 1 diabetes . . . we’ve got lots of Asian people where English isn’t the first language. So you have to, on top of all of that guidance, you have to put in what is going on locally and what isn’t and you know, and what is available. Because, sadly, the same resource isn’t available everywhere . . . so the model has to be realistic into what is available.

Diabetes nurse

Despite these barriers, the transformation manager described how there was good engagement among stakeholders. There was a kind of common-sense ‘buy in’ that meant that stakeholders persisted with their work, even when there were difficulties. The transformation manager reflected that stakeholders really wanted:

. . . to buy into this . . . they saw the vision we want to get to a foot clinic every single day of the week, Monday to Friday . . . they knew fundamentally that was going to offer a better service . . .

Process outcomes

Each subgroup delivered a pathway, and these pathways were then merged into a larger pathway. This new, larger pathway was presented to the overall transformation group. In the larger pathway, stakeholders agreed on the aspects of diabetes care that would remain in the hospital and those that would be moved to the community. The team released a report documenting the new pathway. They also wrote a service specification and attained a firm commitment to a diabetes education programme. As of December 2015, the diabetes nurse reported that the target set for attendance at the education programme had been met with sustained interest and increasing numbers of practices in attendance. It was felt that this programme had caused changes to how people worked by giving them more confidence and knowledge. They had also identified ‘key performance indicators and objectives for the project’. They had also got ‘a number of practices . . . up to enhanced level . . . we’ve got 23 of the 63 practices at enhanced level’ (diabetes nurse), meaning that they would be competent to provide community-based diabetes care. At this point statisticians at the CLAHRC were looking into referral data to establish the impact of the redesign work.

Stakeholders also made reference to the issues of doing and continuing innovative redesign work in the NHS more generally. One questioned the need to have separate diabetes projects across CCGs, suggesting that ‘the NHS is very bad at leveraging best practice and taking what aids that and implement it. They’re more likely to say actually this is my patch and I’ll do it my way, go away type stuff’ (accountable officer), In terms of sustaining innovation, the need for ongoing evaluation was highlighted:

Just because you’ve transformed a service doesn’t mean it’s OK . . . there should be a rolling program of review . . . what we’re going to change this year, what is the long term planning in terms of year 1 . . . 2 . . . 3 . . . I don’t think that has been built in . . . And then things get stale don’t they, and then you have to start again. Whereas I think it would be much better to . . . continually looking and evaluating. But we have, you know, we look for other funding for things that happen.

Diabetes nurse

Summary of findings

Local and universal evidences were used to broadly frame problems with care, capacity and costs. Efforts were made to engage multiple experts during the entire process and involvement in the group was broad. Redesign leaders managed to develop a shared frame on the solution search by using a blank sheet approach. It is important to note that this sheet was not completely blank because stakeholders had an idea that they wanted to move care to the community based on universal evidence talk around the ‘left shift’. A variety of approaches was taken to manage expert collaboration; the most impactful approach to tackling the barriers was the formation of ‘task and finish’ subgroups. These groups looked at local evidence (e.g. patient numbers) and used expertise-based evidence to identify potential areas of improvement. The groups were also asked to look at universal evidence. Although they faced several barriers during the redesign, stakeholders managed to produce a report as per their brief. Attendance at the education programme was sustained, meaning that many practices were competent to provide community-based diabetes care. Statisticians are currently looking into referral data to establish the impact of the redesign work. Overall, evidence use was problematic at Rutterford CCG, but it overcame this to achieve its objectives. Stakeholders also seemed to be satisfied with the process. The redesign process for this CCG is summarised in Figure 9. Guidance for reading process maps is shown in Figure 3.

FIGURE 9.

Redesign process at Rutterford CCG.

Case study overview

Chelsea was one of four CCGs in a trust serving ‘a population of nearly a million’ (report). The trust was concerned with diabetes care and decided that all four CCGs should improve their diabetes care. Participants at Chelsea included a contracting manager, a diabetes consultant, a commissioning manager and a patient representative from a leading charity. The consultant described how each CCG had ‘differing ideas, differing needs. Population profile is entirely different in all four’ (diabetes consultant), but the trust had embraced a model in which patients ‘who can be managed in the community will be sent off to the community to be looked after by the community nurse and the GPs for whom support in various different forms will be provided’ (diabetes consultant). Chelsea was the first of the four to have to ‘a reasonable extent . . . done . . . gone to the first phase where we’re actually discharging people now’ (diabetes consultant). During interviews, stakeholders described how knowing what the new pathway would look like from the outset and so their work was about making evidence from elsewhere relevant to context. The diabetes consultant and performance team were largely responsible for presenting evidence to a steering group. Although it had some faced barriers around communication and finance, the CCG managed to implement changes. At our last contact, the CCG planned to evaluate the impact of the redesign. The remaining three CCGs were still in pre-contemplation stages of change.

Agenda formation

The trust in which Chelsea belongs chose to adopt a model that involved moving patients to the community. This model had been developed at Seaport CCG and was seen as a means to improve care provision by reducing demands on hospital capacity. The diabetes consultant described how ‘philosophy as just the same [as Seaport CCG’s model]’ meaning that trans-local evidence was fundamental in agenda formation. The consultant also had a role in bringing expertise-based evidence because he had worked in a hospital where the ‘model been developed, delivered, evaluated’ and ‘seen first-hand how it works’. He also brought universal evidence, describing, for example, to ‘a paper, I can’t remember where, but there is a paper’ showing model efficacy. He also cited public health data showing ‘diabetes prevalence is slightly higher than the England average’.

The performance team also had a role in bridging ‘the gap between information and the team, to allow [them] to get the information . . . so that we’ve got the data that we can do things’. The performance team was largely responsible for providing local evidence around ‘activity and finance . . . waiting list data’, which was used to show how in ‘the existing system . . . patients are due an appointment in 9 months . . . were just seen just last month’ (diabetes consultant). This delay was caused by long waiting lists, which, in turn, left complex patients needing emergency care. Their emergency care often overlapped with scheduled visits, causing a duplication of work. In the new model, however, it was expected that ‘the complex needs patients who actually need the most will get it on time’ (consultant).

Selecting a solution

Although a solution had been identified from the outset, experts were engaged in a steering group with ‘representation from CCGs, from the trust, from the intermediate care services’. The steering group met ‘every six weeks [to discuss] how we are going to deliver this, what is the next step and how will the patients be discharged’. Over a relatively short period (18 months) the group agreed which diabetes services would be left in the hospital and which would be moved to the community. The services that were left in the hospital were the same as those that were left in the hospital at Seaport, meaning that trans-local evidence played a major role in this episode of work. Expertise-based and universal evidences were also used in tandem during this episode. The contracting manager, for example, described how there was an ‘assumption that the NICE guidelines and those sorts of things are being met and that it will be the responsibility of the lead consultant to ensure that everything is being [met]’ (contracting manager).

Although trans-local evidence meant that stakeholders at this CCG were largely mimicking Seaport, structural conditions acted as barriers to their progress. Communication was the first major barrier mentioned in talk. The diabetes consultant described how the steering group had wanted to develop a computer-based referral system so that he did not have to ‘free up an hour of [his] time for GPs to call . . . [to talk about] discharged patients’. The computer-based communication was not feasible, however, because ‘half of [the practices] are still on a system called System One and half of them are on IBIS and they don’t talk to each other’. Other communication issues came to light when the team were trying to get the trans-local evidence contracted into practice. The contracting manager described how she often had: ‘no idea who I need to speak to, you know . . . it makes life very difficult because every trust is set out differently as well so you never know who’s best to ring’.

Other barriers to implementing the trans-local evidence were financial in nature. Although the trust had been provided ‘about four diabetes nurses’, the CCG still did not have the money to backfill them so that they could go on upskilling days (as had been done in Seaport). Because of this, Chelsea was yet to develop ‘locally enhanced services’ (as Seaport and Rutterford CCGs had). Instead, much of the responsibility for supporting community-based diabetes services was left with the consultant:

The CCG is funding the practice nurses who want to do it to do [education] modules . . . it’s all fully funded [but] there has not been takers for those practices . . . Because they can’t release the people. Because there is, yeah, somebody has to backfill them . . .

Diabetes consultant

Finance was also a challenge because consultants were ‘paid on the number of patients’ seen. This arrangement meant that once the ‘numbers go down’ (as patients are referred to the community), consultants would potentially be at a loss. At our meeting there had been no reassurances put in place, but the diabetes consultant said that there was ‘plenty for him [and other consultants] to do’ and so he was certain that the CCG would find the money to keep him employed. Despite this uncertainty, the consultant had decided to take a risk and let support for the community-based model come ‘out of [his] own time, I’ve just done it’.

Process outcomes

Relative to other CCGs, the redesign work at Chelsea was quite straightforward. It decided to move care into the community, mimicking, almost exactly, the pathway that had been developed in Seaport. Chelsea of course faced some challenges, particularly around finance and communication. Overall, however, Chelsea had agreed a new pathway and had ‘gone to the first phase where we’re actually discharging people’ (diabetes consultant). The consultant described how the steering group continued to hold a discharge meeting in which he used expertise-based evidence while going through a ‘list to identify say 30 patients to be discharged . . . the panel will agree and then the patients will be discharged back to the community with a letter to the GP, with a letter to the patient to say your care is going’.

An evaluation of the changed services was also planned and would be completed by ‘a separate quality team [who have] have a quality review meeting on a quarterly basis with the commissioners . . . and the support unit’ (contracting manager). The evaluation would be based largely on local evidence about ‘follow-ups . . . making sure that they are seen on time, time to first wait’. These evaluations would then be contrasted to outcome markers produced by creators of universal evidence (e.g. the Public Health Observatory) using universal tools (e.g. the Dove Tool).

Overall, Chelsea CCG had met its objectives and stakeholders were largely satisfied. Perhaps because it had decided to mimic Seaport CCG prior to our fieldwork, no major issues with evidence use were recorded in our data either.

Chelsea summary of findings

Chelsea CCG was one of four CCGs in a trust that had been instructed to consider diabetes service provision. Diabetes had been a concern for over 3 years in the area. Once it was given the go ahead, it took just 18 months to implement a new model of diabetes care in practice. This model was based, almost exclusively, on trans-local evidence from Seaport CCG. This meant that the problem and solution search were framed quite narrowly, although role expectations about the redesign were quite clear from the offset. There was a lot of work around expert engagement, especially with the steering group. The group used local, universal and expertise-based evidence to frame issues around capacity for diabetes care during agenda formation episodes. Indeed, the diabetes consultant and the performance team had very clear role structures as actors who brought relevant evidence forward. Although they had faced barriers around communication and finance, they pressed forward with the redesign. At our last point of contact, the CCG had plans to evaluate the impact of its redesign using universal-evidence based tools alongside local evidence about patient experiences of care (e.g. follow-ups, time to first wait). The redesign process for this CCG is summarised in Figure 10. Guidance for reading process maps is shown in Figure 3.

FIGURE 10.

Redesign process at Chelsea CCG.

Case study overview

We first contacted a senior commissioning manager at Coalfield CCG. This individual had played a significant role in the redesign work. He also referred us to other stakeholders who had been involved, including the clinical lead for MSK, the clinical director, the CCG chairperson and a support manager patient engagement. At the time of our visit, this core group had been working together for several years to incrementally change different parts of the MSK pathway (e.g. respite services). They had reached a point, however, at which they felt that the only way ‘to move forward was through whole-scale integration’ of services (senior commissioning manager), and so this is the focus of our case study. At the time of our first meeting, Coalfield had designed and agreed a more integrated MSK pathway and it was about to ‘go live’ (i.e. be implemented in practice). We followed up with the senior commissioning manager on three additional occasions after these first interviews to capture implementation progress. Although there were some challenges, the new pathway was implemented and, at our last contact, stakeholders were working to evaluate its efficacy.

Agenda formation

The senior commissioning manager, the clinical lead for MSK and the clinical director drove the redesign project at Coalfield. These stakeholders effectively framed a more integrated MSK pathway as a means to improve patient care. The need for the new pathway was based on local evidence, with the senior commissioning manager describing how patients were ‘going to the wrong ology at the wrong time’, which meant that they were not getting the right care. He also provided a narrative example, describing one patient who was employed as a steel-grinder, but was approaching retirement. This patient was suffering joint pain and, in the current pathway, would be offered an operation. However, the operation would mean a significant period of recovery that would ultimately jeopardise the patient’s employment. In the new pathway, however, patient need would be the main concern and so, in the case of the steel-grinder, services to maintain his condition (e.g. physiotherapy) would be offered first. Once the patient reached retirement age, other treatment options could be considered, if necessary.

Local evidence pertaining to contracting arrangements was also used to frame the problem. The group felt that the current ‘payment by results system’ was not good for ‘the outcomes of the patients’ (clinical director). They wanted to adopt a form of contracting that recommended by the NHS (universal evidence) known as the capitated outcomes-based incentivised care (COBIC) contract (www.cobic.co.uk). COBIC is a block contract based on patient outcomes, rather than on the number of patients seen (clinical lead). The group decided to pursue this contracting model because it means that ‘behaviour is focused on what’s best for the patients’ (support manager).

The commissioning manager described ‘just trying to keep people out of hospital, lowest intervention, less intensive interventions’ as a means to achieve better patient care. As mentioned in the previous cases, community-based care is informed largely by universal evidence.

Selecting a solution

Work began on the redesign when the group ‘sat down and chewed it over . . . spoke to a few people and mulled it about’ (commissioning manager). As in other CCGs, the group decided to frame the solution search in terms of a ‘blank sheet’ in order to engage stakeholders. Again, however, this sheet was only partly blank given the focus on reducing hospital-based care. The clinical lead saw the blank sheet as important because it engages stakeholders, allowing them to think that ‘actually I’m a bit more in control of my own destiny here’.

Engagement was not straightforward, however, and the first meeting was described as ‘hilarious’ because ‘orthopods hated podiatrists, who hated the physios and they all sat with their back to each other . . . in silence for about 5 minutes’. There was resistance between groups until:

. . . one of them said so what do you want us to do? I said I want you to deliver the best possible care for my patients for the limited amount of money you’ve got. And they said how do you want us to do it? I said I don’t care. I said if you want to get a bag of bones and shake them if that makes a difference that’s fine. I said I just want the outcome . . .

Clinical lead

The commissioning manager was understanding of these issues, however, describing how stakeholders ‘didn’t know each other, they’d never met before . . . didn’t know what clinical skills and options the others could provide’. He described efforts to manage engagement to ensure that they ‘came round and started to have conversations’. First, he felt that because they had done ‘lots and lots of MSK commissioning’ they would have ‘a lot of credibility [because stakeholders] knew [they had] done some big things . . . knew [they’d] done commissioning that worked well’. Other efforts to manage engagement included creating a project website, which the clinical lead saw as ‘something . . . that really pulls people together’. Ultimately, expert engagement was seen as:

. . . the stuff that means your project survives when it gets to a rocky bit . . . a bit cold and a bit difficult . . .

Clinical lead

Efforts to engage patient experts were also notable. Meetings were held with patients, carers and the public so that pathways could be ‘coproduced’. The commissioning manager described how not having ‘local people who completely believe in what you’re trying to do [means] you can’t get anywhere’ The clinical director described how patients can be used as experts, and also be used to manage expert collaboration. As in other CCGs, patients can force stakeholders with divergent ideas or competing interests to reach agreement:

The value of . . . involving the patients . . . right from the word go because they bring a sort of sense of credibility is not the right word but it’s a sense of realism into it right from the outset that you can’t argue with . . . Sometimes you get into little sort of power plays between specialists in the area but actually if you have the view of the patients involved they cut through all that crap . . .

Commissioning manager

During this episode stakeholders sourced and evaluated a mix of universal and expertise-based evidence. The commissioning manager described using universal evidence, including ‘national evidence . . . international evidence’, especially NICE, in relation to ‘good practice, best practice’. This evidence gave ‘a bit of assurance that [stakeholders] are not going to do something whacky’ (clinical lead). Stakeholders also described having to use expertise-base evidence to understand and evaluate universal evidence:

We’re working with expert clinicians . . . many of which have actually written the papers. So they really can understand and make very clear to each other . . . and also argue the case with each other . . .

Senior commissioning manager

Expertise-based evidence was not only important to interpret universal evidence, but also critical to making changes. The clinical director described the MSK screening tool that they wanted to use. It had been produced by academics, but had not been widely trialled before being selected for use at Coalfield. As the clinical director described:

We actually started the changes before the evidence was published so we knew the evidence was coming and you know, we did . . . so we were quite brave really. We sort of thought it’s the right thing to do, let’s do it . . .

They also drew on local evidence to identify a solution. They discussed, for example, how they had a number of facilities in the community (e.g. gyms and sports halls) that were currently underused. They agreed that these spaces could be used for prevention services. So, for example, in these spaces ‘if you need to have your hip replaced would you . . . do your pre-physio [and] your physio afterwards [to] get yourself fit before you have the operation’.

Process outcomes

After several meetings, a new pathway was agreed among stakeholders. The senior commissioning manager described how secondary care stakeholders were ‘saying this [the pathway] fantastic because this is what we’re trained to do . . . We trained in medical school to look after, to help and support people. They were all incredibly frustrated by the systems they currently work in, the silos, the fragmentation’. He also described how the ‘finance director . . . the chief exec . . . the chairman . . . the medical director’ being on board. Although stakeholders were happy with the pathway, the senior commissioning manager still did a ‘lot of leg work’ to ensure expert engagement so that it would be actually implemented in practice:

Before every board meeting I spoke to every board member . . . sticking my head round the door while [saying] do you get it? Do you understand? Have you got any questions? Is there anything worrying you? It’s literally simple as that. And they’d go that’s fine, get out, for God’s sake. In the end people get bored, go away . . . we support you, we’re with you . . .

There were some barriers in implementing the pathway, especially resistance to change. The clinical lead described how ‘the NHS doesn’t have the appetite for innovation that exists in the private sector because the private sector have to make a profit’. Because of this, there was resistance to changing the current pathway, even though it was not working well:

No one has bankrupted each other to be totally blunt. The system has managed. It’s been a complicated way of doing it but . . . the health system as an economy has survived. So . . . to unpick something they don’t like but on the other hand which is relatively low risk and they’ve learned to manage . . . was hugely challenging.

Commissioning manager

To overcome resistance, the clinical lead described how they ‘threatened to stick [the new pathway] out to tender to get [resistant parties] to play ball’. Forcing co-operation in this manner was complicated by governance because it meant that they had:

. . . NHS England on [our] necks because we were ‘privatising’ . . . but once we said we want to work with [local trusts] we had Monitor on our backs saying you’ve not put it out to tender . . . we had to jump through lots of hoops to demonstrate why we weren’t putting out to tender . . . So it’s just a system that doesn’t join up . . .

Clinical lead

Despite challenges, the pathway went live several months after our initial interviews. During follow-ups, stakeholders had a strong interest in evaluating the pathway with methods from other countries (trans-local evidence). They were particularly interested in ‘clinical quality, safety . . . patient gain and patient experience’ (clinical lead). The commissioning manager noted that:

The thing I think the NHS what I’ve seen does badly usually is we do the stakeholder bit really well, we do the implementation planning, we kick off the implementation and then we just pull back. Do you know what I mean? And we don’t do the crucial bit then of studying what happens and then making changes . . . we’re always looking to sort of improve things and sort of make changes. But then as soon as we made a change we’re onto the next thing . . .

Overall, the redesign work at Coalfield CCG was successful. It met its objectives and used a variety of evidences, and stakeholders were satisfied.

Coalfield summary of findings

Stakeholders used local and universal evidences to frame problems with MSK services. They were largely concerned with improving patient care experiences, but did also acknowledge costs. Experts were engaged with the redesign from the offset (senior commissioning manager, clinical lead for MSK, clinical director, CCG chairperson and support manager for patient engagement) but they did a lot of work to build and manage further expert collaboration. In the search for a solution, for example, they adopted the blank sheet approach to frame the search for a solution. Again, this sheet was not completely blank because they had an idea that they wanted to move care to the community based on universal evidence. A variety of approaches was taken to manage expert collaboration, including relying on the group’s credibility, creating a website and focusing on the patients. Universal evidence informed many of their decisions in terms of the selected solution, but this was usually understood in the light of local and expertise-based evidences. Although there were some barriers (e.g. resistance to change), the stakeholders worked to continue to engage experts at the implementation episode. When they were not successful, they used the threat of external tender. The new pathway went live several months after our initial interviews, and stakeholders showed a strong interest in evaluating its impact. The redesign process for this CCG is summarised in Figure 11. Guidance for reading process maps is shown in Figure 3.

FIGURE 11.

Redesign process at Coalfield CCG.

Case study overview

We first contacted the commissioning manager and a GPwSI, who described how the MSK services redesign had originated (approximately 10 years ago) with a GP who ‘a didn’t feel as if some of the services that we were providing locally fitted [the] need’ (GPwSI). The original GP had since retired and the GPwSI interviewed for our study essentially had taken over his role. She shared his feeling that they ‘could do it so much better’. At our first meeting, the GPwSI and the commissioning manager described how they began by inviting a small group of stakeholders. This group later developed into bigger working groups. Between them they had designed a new pathway. Because of several challenges, they were forced to put the pathway out to tender. Stakeholders who had been involved in the design of the pathway came together in a consortium and bid for the tender so that external, private providers would not be able to take their contracts. They were successful and, at our last interviews, had just implemented changes in practice. Other stakeholders interviewed at Coalfield CCG included an external consultant GP, a service manager, a clinical therapies lead and the deputy director of planning.

Agenda formation

The commissioning manager and the GPwSI drew on local and universal evidences to frame problems with MSK services in terms of costs, care and capacity. The commissioning manager drew on local evidence to describe how they did not have ‘the physical space . . . the staff and [were unable] to turn [patients] around quickly enough’. She felt that patients were ‘floating around [secondary care] having treatment which they could have had in the community’. Universal evidences from ‘Atlas of Variation’, which is produced by the Department of Health, showed that Shire CCG was ‘outliers in terms of . . . waiting lists [and] spent too much money’ (GPwSI). Both the commissioning manager and the GPwSI felt that ‘a more integrated pathway’ was necessary that would mean that patients were not ‘bounced from provider to provider’ (GPwSI). Addressing capacity issues was expected to improve patient experiences of care.

Selecting a solution

The GPwSI and the commissioning manager began by engaging ‘probably about six or seven’ stakeholders in a small group (commissioning manager). They decided to engage ‘about 20’ additional experts in ‘MSK working groups’. Groups were tasked with sourcing and evaluating local and universal evidence on ‘the activity, the benchmarking’ (commissioning manager). A GP group was also engaged during this episode of the redesign. The GP group was asked to look at local evidence on ‘the primary care pathway side of things’ (commissioning manager). The GPs also used universal evidence from the ‘Map of Medicine’ (commissioning manager). The Map of Medicine is a resource ‘of national pathways that can then be adapted to local needs’ (GPwSI). The commissioning manager described having good engagement in these groups, including ‘patient representatives . . . the acute trust . . . the community trust . . . physios . . . consultants [and] commissioners’. As in other CCGs, however, there were the barriers that come with multiple professionals working together. The commissioning manager describes how these groups were when:

. . . the conversations got slightly, I wouldn’t say more tense, the conversations were never tense just more difficult because we involved more people who all had a slightly different view of what we were trying to do . . .

In addition to the groups, the GPwSI was working with consultants ‘adapting each pathway [in the Map of Medicine] to [the] local needs’. This adaption was informed by local evidence related to ‘what services you had available locally’. Using local evidence they ‘just tweaked the pathway to say actually don’t send it there, send it here instead, and things like that’. They designed a pathway in which more care would be provided in the community setting, with only patients who needed secondary care getting to the consultants. Universal evidence is also likely to have influenced the wish to have ‘secondary care consultants [to] work in the community service and provide mentorship to the clinicians there and also to do sort of sessions every so often’ (commissioning manager).

The GPwSI produced a new pathway and a service specification. The commissioning manager thought that everybody agreed, noting that an onlooker would have ‘gone away going well, they’re all on board with this. They all seem to get it’ She was ‘adamant that this was the right thing to do . . . and . . . to push it forward’.

Despite her optimism, the commissioning manager described how they ‘just could not get all the different parties to come together to do the one pathway’. In our interview she said she was not able to ‘quite put [her] finger on why’ they could not move towards implementation. Throughout our conversation, however, talk indicated several potential barriers. These were later reinforced in interviews with other stakeholders:

First, the GPwSI described risk averseness and unwillingness to change, noting stakeholders as ‘a bit resistant and set in their ways’. She described how the proposed changes would have been ‘the first major service redesign in the local health economy . . . it’s millions and millions and millions of pounds’. She felt that the risks associated with such a big change were ‘at the back of people’s minds’, causing them to have cold feet. However, in our interviews with clinicians (service manager and clinical therapies lead), a different opinion was shared. These stakeholders described how the service specification that they had been given was ‘very brief . . . with minimal information’. They found it very difficult to ‘understand the commissioners’ wants and desires . . . from the information that was there’.

Second, they both commented that implementation may have been stalled by unclear roles. They said that sometimes the ‘wrong’ people were around the table and ‘unless everybody understands why they’re there and understands what their role is, then potentially you’re not going to get further forward’ (commissioning manager). Some stakeholders involved in the solution selection were seen as not particularly ‘engaged in doing that piece of work . . . and just turning up and being there because they’ve been asked to be’ (GPwSI).

Last, they listed finance as a barrier. In the new pathway, patients would be released to community services and ‘people were scared of losing patients’ (commissioning manager). The acute trust were ‘seeing some of their physio coming out, some of their pain coming out, some of their anaesthetics coming out . . . and not getting over that’ (commissioning manager). Although the commissioning manager tried to manage discussions during expert collaboration, she felt stakeholders could not see ‘that there is a different way of maintaining their income’. She described trying to make them understand the new way would be more effective because specialists would not be ‘seeing so many people who . . . have got a sore toe, [they would be] seeing people who need a hip replacement’.

Contrasting this, clinicians commented that the service specification showed an ‘extreme lack of knowledge about what was happening . . . in the current services and . . . going forward . . . [It didn’t have] an activity schedule and . . . a financial schedule’ (service manager).

Several meetings and discussions were held ‘once every 4–6 weeks for easily 2 years, probably longer’ to try to reach agreement about these barriers (GPwSI). They were unsuccessful, however, and the commissioning manager described how she felt ‘enough is enough’ and ‘made the decision that they were going to go to tender for part of the pathway’ that they could not agree on. Tendering was not seen as an ideal for two reasons. First, it ‘costs both providers and commissioners a lot of time and resource . . . the amount of hours that have been put into the tender is ridiculous, absolutely ridiculous’ (commissioning manager). Second, it is not always reliable and ‘in other places [private providers] had a big contract and they failed to provide on that’ (GPwSI).

After ‘to-ing and fro-ing [about] the wording of [tendering] documents . . . it eventually went out and [there were] . . . three main bidders’ (commissioning manager). Two were private and one was a consortium of health providers that had been involved in the redesign process. The consortium recognised that ‘no one current organisation could actually bid on their own’ (clinical service manager). The consortium agreed ‘no one trust would disadvantage another trust’, which made the tender ‘incredibly difficult to deliver . . . [because] If you want to do a true service redesign . . . organisations have to be disadvantaged’ (clinical service manager). They used local evidence to inform agreement, describing for example that the contract was ‘worth roundabout 1.4 million pounds’ to them (deputy director). They also felt that local providers offer the service would ‘improve patient care . . . people [will be] seen in the right way and having a better journey’ (deputy director). The consortium was the successful bidder for the contract.

Process outcomes

Although they reached agreement and won the bid for the new service, the new group still faced barriers. Recruitment of staff to provide the new pathway was difficult because it is NHS policy not to recruit ‘at risk’. This meant that they were not allowed to recruit new staff until the service was up and running. Because of this they ‘had to start by begging and borrowing people from lots of other services’ (clinical therapies lead). Both the clinical therapies lead and the clinical service manager noted that although, on paper, the NHS encourages competiveness, issues like this undermine progress in ways that you would not find ‘in the private sector at all’.

Despite the hurdles, the objectives were met and a new service was opened. In some cases, however, stakeholders were not satisfied. The clinical service manager described how they ‘ended up in a situation where we all still deliver what we were delivering with bits added on but through different contracting arrangements’. At the time of our meeting the service had been live for just 2 weeks, which meant that they were ‘fire fighting and ironing out the crinkles’. They did say that this was ‘less and less every day’ and that they were ‘getting there’. They also seemed optimistic, with ‘lots of great ideas about what we’ll want to do and how we want to deliver it’ (clinical service manager).

Shire summary of findings

Shire CCG’s redesign was against years of work trying to improve MSK services. Problems were framed in terms of costs, care and capacity. Universal and local evidence were used with the understanding that too much care was in the hospital. The commissioning manager and the GPwSI recruited small groups, which were later expanded to working expert groups. These groups were tasked with looking at various evidences (universal, local) about a solution. The GPwSI was also working up a new pathway, external to, but informed by, the groups. Several challenges were presented and discussed at meetings, until they to went for tender. Providers formed a consortium, which eventually won the tender. After the tender was won, several additional barriers were faced, including recruitment and their inability to ‘disadvantage’ each other. The redesign process for this CCG is shown in Figure 12. Guidance for reading process maps is shown in Figure 3.

FIGURE 12.

Redesign process at Shire CCG.

Work practices

This section of the toolkit is aimed at understanding and improving work practices around commissioning teams’ use of evidence.

Understanding how ready your project team is to use evidence

(Target user: lead/members of redesign projects.)

Preamble Successful service redesign projects are those that develop effective, realistic and well-thought proposals/solutions for changing the commissioning of health-care services at a local level. Service redesign teams are more likely to develop such solutions when their decision making processes are well managed and well informed, that is, backed up with evidence, local knowledge and insights of various experts. In our research, we found that decisions need to be based on different kinds of evidence (universal, trans-local, local, expertise-based). We also identified specific capabilities that can contribute to improving the use of evidence in service redesign decision processes. These are summarised in the attached sheet. The following questionnaire aims to help you assess these capabilities in your own setting so that you can identify areas where your project may be more, or less, ready to use evidence to inform effective redesign solutions.

Area 5: managing expert collaboration

This set of questions is about how well different experts may work together to achieve the project’s objectives.

5a. Project members have different goals and objectives for this project.

	1	2	3	4	5
Strongly disagree	⦿	⦿	⦿	⦿	⦿	Strongly agree

5b. It is important for all members to declare any conflicts of interest.

	1	2	3	4	5
Strongly disagree	⦿	⦿	⦿	⦿	⦿	Strongly agree

5c. We have considered how we can get key experts (including patients) to share relevant information with one another, even if it might be politically sensitive.

	1	2	3	4	5
Strongly disagree	⦿	⦿	⦿	⦿	⦿	Strongly agree

5d. We can recognise and handle a situation in which a project team member seeks to influence the decision-making process to serve their specific vested interest.

	1	2	3	4	5
Strongly disagree	⦿	⦿	⦿	⦿	⦿	Strongly agree

Participant answers to the above questions can then be used to generate a radar plot, an example of which is shown in Figure 13.

FIGURE 13.

Example of radar plot for team capabilities to use evidence.

Assessing how favourable the organisational context is for using evidence

(Target user: director/associate director of commissioning organisation.)

Preamble Using evidence effectively to inform decisions is a very localised skill. Middle-level managers, project teams and other stakeholders involved in commissioning need to take specific actions to find and use evidence in decision-making. In our research, we found that specific practices (e.g. identifying and engaging the right experts) enhance the use of evidence in the design and commissioning of health-care services. At the same time, we found that the organisational environment influences commissioning groups’ efforts to use evidence. The following questionnaire aims to help senior managers, such as you, identify areas in which your organisation may provide a more or less favourable context for the use of evidence in commissioning decision-making.

Area 3: organisational memory

3a. My organisation has clear processes in place to capture evidence and learning from past projects.

	1	2	3	4	5
Strongly disagree	⦿	⦿	⦿	⦿	⦿	Strongly agree

3b. In my organisation, before making important decisions, we almost always check how related or similar decisions in the past were taken here and/or check how other organisations have tackled the problem.

	1	2	3	4	5
Strongly disagree	⦿	⦿	⦿	⦿	⦿	Strongly agree

3c. My organisation maintains repositories, which include information about past projects, project outputs and project participants and these are easily accessible.

	1	2	3	4	5
Strongly disagree	⦿	⦿	⦿	⦿	⦿	Strongly agree

Participant answers to the above questions can then be used to generate a radar plot, an example of which is shown in Figure 14.

FIGURE 14.

Example of radar plot for organisational preparedness to use evidence.