Service user engagement and health service reconfiguration: a rapid evidence synthesis

Article history

The research reported here is the product of an HS&DR Evidence Synthesis Centre, contracted to provide rapid evidence syntheses on issues of relevance to the health service, and to inform future HS&DR calls for new research around identified gaps in evidence. Other reviews by the Evidence Synthesis Centres are also available in the HS&DR journal.

The research reported in this issue of the journal was funded by the HS&DR programme as project number 13/05/11. The contractual start date was in March 2014. The final report began editorial review in September 2014 and was accepted for publication in December 2014. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Jane Dalton is a publicly elected governor at York Teaching Hospital NHS Foundation Trust.

Copyright statement

© Queen’s Printer and Controller of HMSO 2015. This work was produced by Dalton et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

General approach

The project was resourced as a rapid evidence synthesis. There is no generally accepted definition of this term and a number of other terms have been used to describe rapid reviews incorporating systematic review methodology modified to various degrees. Our intention was to carry out a review using systematic and transparent methods to identify and appraise relevant evidence and produce a synthesis that goes beyond identifying the main areas of research and listing their findings. However, we foresaw that the process would be less exhaustive and the outputs somewhat less detailed than might be expected from a full systematic review. Added to this, we expected to find limited evidence on the subject in the peer-reviewed primary literature.

Research questions

We sought to address the following five questions:

1. How have patients and the public been engaged in decisions about health service reconfiguration in the past?

2. How has PPI affected decisions about health service reconfiguration?

3. Which types of PPI have had the greatest impact on these decisions?

4. Which methods of PPI are likely to be sustainable/repeatable?

5. How have differing opinions about reconfiguration between patients, the public, clinical experts and other senior decision-makers been negotiated and resolved?

Scope and definitions

The focus of the review is reconfiguration of health service provision in the NHS. We also considered evidence on health services delivered by non-NHS providers (e.g. voluntary sector/private sector) and the joint provision of health and social care where this impacts directly on NHS provision. Where relevant, we considered international evidence from other health systems which are comparable and relevant to the NHS. In addition to England/the UK, the included systematic reviews covered studies conducted worldwide; other research and case studies additionally covered Scotland and Canada.

Reconfiguration includes large-scale system change, such as relocation of hospitals, (re)location of specialist care and changes in provision of urgent/emergency/out-of-hours care. We did not consider small-scale change, for example at hospital ward level or within a general practitioner (GP) practice. Reconfiguration has been defined in the literature as a deliberately induced change of some significance in the distribution of medical, surgical, diagnostic and ancillary specialties that are available in each hospital or other secondary or tertiary acute care unit in locality, region or health-care administrative area. 14

In the literature, the terms ‘engagement’ and ‘involvement’ are often used interchangeably. For the purposes of public involvement in research, INVOLVE (www.invo.org.uk) distinguishes between active involvement of patients or members of the public in research projects and engagement, which provides information and knowledge about research in an accessible way (e.g. through science festivals or open days). This distinction is difficult to sustain in the context of proposals for service reconfiguration where provision of information may (or may not) lead to active involvement. Events such as public meetings or citizens’ juries have elements of both information provision and active contribution of patients or members of the public to developing or modifying (or rejecting) proposals for change. In this review we define patient/public engagement or democratic involvement as including any means of seeking and responding to the views of patients and the wider public at any stage of the process of reconfiguration (including identifying possible options for change). We have not attempted to standardise the various terminology used to indicate service user engagement across the included studies. In our search strategy, other terms included ‘user’ and ‘carer’ engagement and involvement (see Appendix 1). The scope included existing patients, carers and their representative groups; and the general public and their representatives [e.g. local councillors and Members of Parliament (MPs)].

Inclusion and exclusion criteria

We looked for relevant evidence in three main areas:

• Systematic reviews of methods of/approaches to patient/public engagement. We included only reviews that were relevant to PPI in decisions about health service reconfiguration. Reviews of PPI in research were excluded.

• Empirical studies of any design evaluating methods of/approaches to patient/public engagement. Studies that focused on involvement in research were excluded.

• Case studies examining how PPI worked in specific examples of system change in the recent past. We expected that these were more likely to be found in the grey literature than in peer-reviewed publications. Case studies of this kind were likely to provide a biased sample of ‘successful’ rather than typical PPI but were more likely to provide useful data to inform future practice. We also searched for case studies where public involvement failed to produce an agreed way forward or resulted in unintended consequences, using the website of the IRP as a starting point.

The following were excluded:

• ‘emergency’ reconfigurations triggered by failure of a service provider, such as a NHS trust

• consultation/involvement of NHS staff, except as part of a broader consultation where staff and PPI could not be separated

• patient/public representation on bodies where reconfiguration was part of the remit but was not the main focus

• patient/public engagement methods where complaints management was the focus (e.g. patient advice and liaison service and the Healthwatch independent advocacy arm).

Literature search

Study selection, data extraction and quality assessment

Search results were initially screened by a single reviewer to eliminate obviously irrelevant items. Full-text copies were ordered or downloaded for potentially relevant records. Final study selection was carried out by two reviewers independently, with disagreements resolved by discussion or involvement of a third reviewer if necessary.

We used EPPI-Reviewer 4 (Evidence for Policy and Practice Information and Co-ordinating Centre, University of London, London, UK) to record decisions about study selection and for data extraction and quality assessment. We developed separate data extraction forms to record key information for different evidence sources (systematic reviews, case studies and other research). For case studies, data extraction was done in two stages: basic details were extracted for all included case studies; then a number of ‘exemplars’ were selected for more detailed data extraction and analysis. Exemplars were those case studies that provided most detailed and current information about the methods used for patient/public engagement and involvement and/or assessed the impact of engagement/involvement in reconfiguration decisions. Data extraction was performed by one reviewer and checked by a second.

We assessed systematic reviews for methodological quality and reliability using the approach of DARE. We planned to assess published primary research studies using appropriate design-specific tools described in the guidance of the Centre for Reviews and Dissemination for undertaking systematic reviews in health care (2009). 15 Unpublished case studies and non-peer-reviewed reports were not formally assessed for quality (risk of bias) but we sought to identify any instances of more rigorously conducted and fully reported case studies. Issues considered were:

• the extent to which an appropriate diversity of perspectives (e.g. across service user and NHS) was considered in assessing the impact of patient/public engagement

• the extent to which the case study was conducted and reported with transparency

• reflexivity on any specifically adopted perspective, together with adequacy and clarity of reporting on intervention context, methods and impact.

Synthesis

We carried out a narrative synthesis using multiple frameworks to guide our analysis. In addition to the five research questions specified in the study protocol, we considered chronological aspects of reconfiguration decisions in terms of the seven stages specified in the NHS England guidance on planning and delivering service changes (Box 1). 3 Levels of engagement/involvement were assessed where possible, using the version of Arnstein’s ‘ladder of engagement and participation’ presented in the NHS England guidance on transforming participation in health and care (Table 1). 4 We used the available literature to determine the extent to which evidence supported or disagreed with the recent guidance and to highlight areas where the evidence was conflicting or insufficient.

In synthesising the case studies, we focused on those case studies identified as exemplars (those case studies that provided more detail, see Study selection, data extraction and quality assessment). We were particularly interested in identifying case studies with an element of independent evaluation by an organisation not involved in the reconfiguration being examined.

Given the resources available for the project, we planned to focus on only a small number of exemplars. For other case studies, we extracted basic details only and used these studies to supplement the analysis of themes emerging from the exemplar case studies.

Overview

We identified eight systematic reviews conducted between 2002 and 2012 (see Appendix 2). The number of included studies in these reviews ranged from 8 to 344. Study locations included various European countries, the USA, Canada, Australia, New Zealand, Israel and Japan. All reviews included some proportion of UK studies. Four reviews contained a majority of studies located in the UK16–18,21 and two reviews5,19 had a complete focus on the UK setting (Table 2 and Appendix 2). Reading across the reviews, there was some overlap of studies. Owing to resource limitations, further examination of the nature and extent of this overlap was not carried out.

Quality of the reviews

The quality of the eight systematic reviews varied. Seven reported an adequate search and provided study details, and all presented implications for research and/or practice. However, the extent to which review conclusions were reliably supported by the evidence presented (in the traditional sense of critically appraising systematic reviews) was limited by the fact that only two reviews formally assessed the quality of included studies. Seven of the reviews involved mixed evidence sources such as reviews, qualitative and quantitative (largely observational) studies, grey literature and discussion papers.

Types of reconfiguration

Although all eight reviews were related to service reconfiguration, not all framed their objective in these terms. Where reconfiguration was described beyond general terms, review authors referred to priority setting, local planning and policy development, and decisions about health service resource allocation. Service user engagement was explored across a range of specialist services and generic service or policy development. Examples of specific clinical service changes included those which were related to cancer,6,19 mental health,17 women’s health and maternity,16,17 and older peoples’ services. 19 Reviews also examined system-wide change, such as the shaping of primary care and community services;17,18 one review included two studies concentrating on plans for hospital closure. 17 Some reviews adopted a wider remit, capturing more than merely health implications. These particular reviews focused (in addition to health-care services) on proposals for change in areas such as environmental planning, education and housing. 18,20

Engagement methods and who was involved

There was no consistent definition of service user engagement or involvement. Where this was reported, engagement was specified in the review authors’ terms and did not appear substantially linked to any wider conceptual or theoretical framework.

A range of methods was employed in the engagement process. The extent to which methods were explicitly specified varied. Those that were primarily informative in nature included, for example, communication via traditional publicity and the provision of website materials. 16,20 Other methods indicated more active involvement of service users in eliciting feedback by opinion polls and surveys. 17,18,20 Consultation and deliberative methods featured in all reviews, being largely operationalised as collaborative partnerships, citizens’ juries, working groups, consensus conferences and other mechanisms. Where public meetings and community forums were described, without further detail it was often difficult to determine their positioning on the ladder of engagement and participation (see Table 1). Across many reviews, a mixture of methods was used to capture the service user voice. Four reviews discussed the potential sustainability of methods. 6,16,18,20

Across the reviews, service users were frequently described as ‘the public’, although this term tended to be defined loosely and variably. Others engaged in the process were patients, carers, staff, local residents, councillors, MPs and stakeholders (invariably not defined). The engagement of multiple audiences was referred to in many cases.

Impact

Most reviews were broadly agreed on the paucity of evidence of impact in relation to service user engagement and reconfiguration. More robust evaluative research was generally recommended. Many review authors cited the critical influence of contextual variables on successful engagement; one referred in particular to geographic variability. 16 The absence of measurable outcomes was a problem;5,16,21 the lack of independent research was reported to be a considerable limiting factor. 18

Successful engagement was defined variably across the included studies, with many describing impact on processes rather than service reconfiguration per se; for example, changes in service user views about services, organisational culture change with regard to commitment to user engagement, and shifts in learning about future processes represented outcomes in two reviews. 16,17

There was some evidence of impact on service delivery outcomes in terms of changes to service provision5,17 and, in particular for location and access issues,5 priorities integrated into a regional programme and new resources found for services resulting from the activities of citizens’ juries and other community collaborations. 16 One review included two studies that reported a successful challenge to hospital closure, resulting in the proposal being modified or abandoned. 17

Negative consequences of engagement were rarely reported. However, two reviews referred to service users interpreting the engagement process as tokenism,17 and community stakeholders were reported to experience unintended consequences (feeling ostracised) when challenging statutory sector partners. 16

While there was little evidence to support the isolated success of any particular engagement method,19,20 there were positive indications for those characterised as more deliberative in nature and involving face-to-face interactions,6,16,19,20 and for engagement efforts comprising multiple methods. 19 There was mixed support for partnership working, which was seen as central to success in one review6 and as having no systematic relationship with any form of organisational change in another. 21

Tentative success factors in service user engagement appeared to be organisational support for the process; a willingness of users to engage; clarity surrounding the aims of engagement; and adequate resourcing of evaluations. 6,18,19,21

There was little discussion about the potential sustainability of methods. In one review, the institutionalisation of partnerships was seen as a key driver,6 while regional meetings were seen as potentially repeatable in another. 16

Systematic reviews in summary

Reviews were conducted with a reasonable level of attention to methodological rigour. Because of the diversity and nature of the study designs, the quality of the studies included in the reviews was difficult to determine. A variety of health services were studied, and a range of engagement methods (described by various terminologies) adopted. Not all systematic reviews focused completely on health service reconfiguration. Where this was the case, review objectives seemed closely aligned to reconfiguration (e.g. the focus was on priority setting or decisions about resource allocation for future services).

The isolated impact of service user engagement (as distinct from the engagement of staff and other stakeholders) was sometimes difficult to distinguish. Reviews focused largely on the impact of service user engagement on outcomes related to process (e.g. shifts in organisational views about engagement) rather than those related to the impact of engagement on reconfiguration success.

Positive indications were noted from engagement methods that were more deliberative, those involving face-to-face interactions and those comprising multiple methods. Tentative factors leading to successful service user engagement were organisational support, willingness of users to engage, clarity about the aims of engagement and adequate resourcing of evaluations.

Other research

Case studies

Ladder of engagement and participation

The NHS England guidance on transforming participation in health and care uses a ‘ladder of engagement and participation’4 (based on the work of Sherry Arnstein47) to classify different ways in which patients and the public can participate in health (see Table 1). The ladder has five levels: informing, consulting, involving, collaborating and devolving. It is argued that participation becomes more meaningful towards the top of the ladder (devolving). Although there is academic debate about the limitations of this model, in terms of its narrow focus on transfer of power between providers and services,48 it has been widely used in studies of engagement and participation in health.

For the included case studies, we assessed only the levels of engagement reported in those selected as exemplars. Among the six exemplars, the highest level was devolving in one case,7 collaborating in four12,43–45 and involving in one. 9 Thus, these generally well-reported case studies were characterised by relatively high levels of engagement, which would be expected to allow meaningful interaction between participants and NHS decision-makers. This sample of case studies was too small to allow any assessment of whether or not levels of engagement had increased over time.

The levels of engagement in the studies reported in the included systematic reviews were also high. The highest level reported was collaborating (which involved working in partnership with communities and patients on all aspects of a decision) for all except one review. The broad review of user involvement in change management by Crawford et al. was judged to include examples of devolving (placing decision-making in the hands of the community or individuals). 18 The high levels of engagement may partly reflect the broad coverage of the included systematic reviews.

The levels of engagement reported or discussed in studies in the ‘other research’ category were generally lower than in the case studies or reviews. Two reports related to the collaborating level. 15,28 The Scottish Health Council report sought input from public panels and NHS stakeholders on how to enhance public involvement in NHS service change. 42 The other report was an expert opinion report on how NHS managers should seek to frame debates around reconfiguration. 28 As with the systematic reviews, both reports were broad in scope, although much more specifically focused on service change.

Overall, the ‘ladder of engagement’ was of some help in differentiating among studies but its use was based on the assumption that the methods reported provide genuine opportunities for engagement and were not just offered to meet legal or bureaucratic requirements. The extent to which this was true may depend on contextual factors that were difficult to assess from paper reports.

NHS England stages of reconfiguration

The NHS England guidance covers seven stages (Table 5), ranging from ‘setting the strategic context’ through to ‘implementation’, although the boundaries between these are not always clear-cut. Some themes and issues arose at multiple stages of the process. It should be noted at the outset that most of the evidence appeared to adopt the perspective of health system decision-makers responsible for the process of service change, and comments about ‘successful’ engagement or service change should be seen in those terms.

We did not systematically attempt to assess the extent to which stages of the NHS guidance were addressed in the case studies not selected as exemplars. When considering the stages of the NHS guidance addressed in the other research studies, the extent to which attention to specific stages influenced the overall success of the engagement process and other outcomes was unclear.

Table 5 summarises which of the stages were covered by the literature, with a focus on the relevant exemplar case studies.

Research questions addressed

We did not attempt to assess systematically the extent to which our research questions were addressed in the case studies not selected as exemplars. Engagement methods were often not described in sufficient detail to provide meaningful reporting. Where reported, the range of methods did not appear materially different from those identified in the other types of research above. Similarly, the lack of details about methods precluded meaningful analysis with respect to demonstrable impact on service reconfiguration. The measure of impact on decisions about health service reconfiguration, sustainability of engagement methods, and resolving and negotiating differing opinions about reconfiguration were also not sufficiently reported in these studies.

Tables 6–8 summarise the evidence (by type of research) on the extent to which the five research questions were addressed.

Summary of the evidence

Methods of engagement identified in this rapid evidence synthesis were varied in nature and intensity, and generally involved a mixed methods approach. Engagement programmes were conducted across a range of health services with diverse audiences. There was no evidence on the isolated impact of any particular engagement method or collection of methods, and there was little detail about their sustainability.

The impact of engagement was variably measured and demonstrated. Impact was more frequently defined in terms of process measures rather than success or failure of reconfiguration. Key process factors identified were organisation readiness and commitment to service user engagement, clarity of aims and adequate resources. Little was reported on the potential negative impact of service user engagement, but the variable effect of media coverage (which may encourage polarised views) was highlighted, and lessons from past referrals to the IRP may be helpful.

Early engagement can help to contextualise and influence the perceptions of service users. Ongoing face-to-face interaction may be beneficial, and deliberative methods may provide further help to ensure a wide representation of service user voice. Social media may be useful, particularly to engage with young people. It is helpful to form common views between the public and commissioners, but different priorities may present difficulties.

This was a difficult area to research, the evidence was limited and rigorous evaluations were lacking. Much of the evidence was from single case studies, which may have limited generalisability in other settings. The evidence to support methods of service user engagement in health service reconfiguration was inconclusive; we were unable to recommend a particular method to effect a specific outcome. The lack of independent research was noted (case studies identified were likely to be biased towards successful reconfiguration from the perspective of NHS managers), as was the absence of measurable outcomes and clear definitions of successful reconfiguration.

Reflections on the evidence

Much of the evidence presented in this review related to the UK NHS setting. For those embarking on future service user engagement programmes, the exemplars identified may indeed represent what good evidence looks like. Clearly reported objectives, methods and contextual detail, and reflective reporting are key elements to achieve this.

The review uncovered a broad range of engagement methods, from those simple and informative in nature, to more deliberative methods involving the development of shared understanding with service users and identifying solutions through partnership and collaborative working. No single specific method of engagement appeared to be more effective than another; the use of mixed methods of engagement was frequently reported. While there may be an argument for evaluating the collective effect of multiple methods (on the basis that this is reflective of practice), the relative effects of different engagement methods from a cost-effectiveness viewpoint may be a topic warranting further research.

Service user engagement was rarely evaluated in terms of its direct impact on the success or failure of service reconfiguration. Interim and process outcomes were frequently reported (such as changes in service user views about services, organisational culture change with regard to commitment to user engagement, or shifts in learning about future processes). This poses a question for future research about what is really meant by impact in evaluations of service user engagement and health service reconfiguration. Until more clarity is reached about objectives and outcomes, the evidence is likely to remain inconclusive. Future evaluation of interventions is vital.

Although it was not explicitly stated, impact appeared to be context-specific; the relative success of this was set against a backdrop of local dynamics, historical issues and baseline service user expectations. Furthermore, the impact of interventions to involve patients and the public in reconfiguration decisions was likely to be mediated by contextual factors which could vary widely across settings as well as over time. Some of these factors were at least in part predictable and could be used to select appropriate engagement methods. Examples are the nature of the area (such as urban or rural), age structure and socioeconomic characteristics of the local population and the types of services affected by reconfiguration proposals.

Additionally, local responses to configuration proposals can be inherently unpredictable, making it difficult or impossible to discuss alternatives and seek ways to resolve different opinions. Many community groups have opposed loss or downgrading of services in their local hospital; only in one setting (Wyre Forest) did they succeed in getting a representative elected to Parliament and gain control of the local council. 36 This example dates from the early 2000s and the fact that it has not been repeated since suggests the existence of some highly unusual local factors. In another example, a campaign against changes to hospital services in Surrey and Sussex was strengthened by the support of well-known local residents, a factor unlikely to be reproduced in less affluent areas. 37

The exemplars and other case studies included in our review cover a range of different services and geographical settings. An obvious distinction is that between services for the general population (e.g. primary and urgent care), for specific sections of the population (e.g. maternity services) and for people with specific conditions (e.g. specialist mental health services). The balance between engaging with the general public and with patients, patient groups and carers, and hence the methods used, tend to reflect the type of service. The type of setting (e.g. urban, suburban or rural) may also influence the process of engaging with patients and the public, although it is likely that the increasing use of online and social media-based methods will reduce the importance of geographical factors over time. It is also likely that other, more subtle factors influence the appropriateness of using different methods of engagement in different circumstances. A fuller analysis of this type of issue would require a larger sample of well-reported exemplars than we were able to obtain for this review, but this could be a topic for future primary research. A robust conceptual model of the rationale and goals of PPI in reconfiguration decisions would be helpful for this type of research.

In the synthesis, a pragmatic decision was taken to map the evidence against existing policy and guidance. We turned to potential triangulation between the evidence, the NHS stages of reconfiguration and levels of engagement; the extent to which the different sources correlate with each other and offer a consolidated framework to those considering service user engagement in health service reconfiguration. Generally, convergence of concepts from the different sources was difficult to determine, as it was not possible to distinguish with confidence the seven stages and the five levels of engagement in the included evidence. It was clear, however, that the ‘ladder of engagement and participation’ (based on the work of Sherry Arnstein47) provided less theoretical contribution than the NHS England stages. In respect of the ladder, examples of devolved decisions to the community were not well represented in our evidence. One might propose that devolvement might more appropriately apply to smaller-scale service change than to the various larger-scale reconfigurations considered in this review.

Given the nature of the review questions, it was clear that evidence was unlikely to be generalisable in the traditional sense of identifying elements which frame the research questions (populations, interventions, comparators and outcomes). We suggest that generalisability of the evidence to future service reconfigurations may lie in the approach (e.g. using the seven NHS England Stages) to guide the engagement process, rather than attempting to generalise based on the mechanics or the context underpinning that approach.

We focused our search for evidence on studies of direct relevance to PPI in service reconfiguration. Resource constraints meant that we have not systematically reviewed the theoretical and empirical literature on PPI in health care generally (although this was covered to some extent in the systematic reviews we have included). For theoretical frameworks we concentrated on those that have informed current NHS guidance (NHS England’s stages of service change3 and the ‘ladder of engagement and participation’). Arnstein’s ladder dates back to the 1960s and other theoretical frameworks have been developed more recently. For example, Gibson et al. 50 suggested that earlier models of PPI were unable to respond effectively to the current context of declining faith in traditional political structures and processes and the diversity of values, ideologies and social groups trying to make their voices heard. They proposed a four-dimensional framework for analysing the nature of PPI. This framework encourages knowledge accumulation from multiple sources, with emphasis on the development of reasoned, interactive and equitable discussions between lay and professional people. The framework, they argued, could assist the development of new structures and processes that may allow professionals and lay people to work together more productively than most current structures permit. 50

Another critique of current practice in PPI generally related to the issue of power imbalance between patients and members of the public and NHS managers and clinicians. This was the original purpose of Arnstein’s ‘ladder of involvement and participation’, to suggest that most involvement activity was more tokenistic than genuine participation. 47 Harrison and Mort coined the phrase ‘technology of legitimation’, arguing that PPI ‘can be seen as a means by which managerial legitimacy is maintained in the context of an increasingly pluralistic policy arena’. 51 Similar issues were raised by Martin in a study of service user involvement in the establishment of cancer genetics services in England. 52

In view of resource constraints and the service-focused nature of this review, we were unable to address these issues in any depth, but we recognise that it is important for NHS decision-makers to be aware of and reflect on these critiques. However, these issues relate to all types of PPI in health care (and in policy- and decision-making generally) and are not specific to service reconfiguration, which was the main focus of this review.

Reflections on the review process

The topic area had parallels with the evaluation of complex interventions and public health research; engagement was often part of a multicomponent intervention from which it was difficult to isolate individual effects, and the long-term impact was difficult to measure.

Reconfiguration was defined for this review as large-scale system change [e.g. relocation of hospitals, (re)location of specialist care, changes in provision of urgent/emergency/out-of-hours care] as opposed to small-scale change (e.g. at hospital ward level, within a GP practice). It became evident during the study selection process that the distinction between large-scale and small-scale change was not always straightforward, and this was not helped by variable use of terminology to describe service change. Judgements had to be made, and discussions took place between reviewers to reach agreement on inclusion, particularly where articles reported on a mixture of different levels of change, and often as part of an overarching strategy.

Given the rapid nature and limited resources available, limitations were placed on the review process. For example, strict criteria were applied on the reporting aspects of articles to tighten the focus to those most likely to provide useful information. Studies contained within reviews were not extensively followed up, and the overlap of studies across reviews was not examined. For case studies, a number of websites provided external links to other case studies, for example the Scottish Health Council;42 external links were not followed, but instead a signpost was provided, together with a summary of characteristics of those studies.

Of interest for future search strategies in service-facing reviews, two of the exemplar case studies were retrieved through contacts and not by the review search strategy. Contact with experts routinely forms part of traditional searching and this aspect represented a particularly important contribution to the present review.

The variable quality of evidence across the included systematic reviews may be a consequence of the composite nature of that evidence. Mixed methods evidence appeared to be a defining feature of the topic area selected for this review. Assessing the quality of a review based on mixed methods and/or multiple sources presented difficulties in applying traditional assessment criteria based on individual study design. Currently, there is no robust reliability assessment tool for mixed methods reviews.

Early specification of criteria for evaluating the quality of case studies in this review (based on clarity and transparency of reporting) enabled the successful identification of evaluations that might serve as exemplars of good practice. This was a relatively straightforward aspect of the review. Based on our three assessment criteria, the overall quality of these case studies was good (defined as adequate and clear reporting; evidence of reflexivity; and diverse perspectives considered). However, given that good reporting was an inclusion criterion for case studies, selection bias was a possibility in this review.

Key conclusions from the project

The overall quality of evidence in our review is mixed. The included systematic reviews comprised multiple types of evidence and study designs, and most reviews did not report any quality assessment of included studies. A number of potentially helpful case studies were found, but the quality of reporting was generally poor. Six case studies were selected as exemplars on the basis of clearly reported evaluations, and these provided focus for our analysis. One of the key findings from this review is that meaningful engagement is hard to achieve, and research to evaluate its impact is difficult to conduct.

Nevertheless, our review demonstrated that it was possible to address these difficulties. Great advances have been made to set out frameworks for engaging users in discussions and decisions about service reconfigurations. We have found a number of exemplar case studies which show that meaningful engagement can be achieved. Moreover, although the evidence base was not large, we have found studies that were able to provide insight into the value of user engagement and its impact on shaping service reconfiguration. In particular, this review succeeded in clarifying some of the factors associated with positive service user engagement.

Patients and the public can be engaged through a wide variety of methods ranging from public meetings and distribution of information to the use of social media. People can be engaged as individuals, in small groups and in larger groups. In selecting which methods to employ locally, decision-makers will need to take into account the nature of the local population and of the proposed service changes. In general, engagement was most likely to be successful when the process started at an early stage of planning service change, offered opportunities for genuine interaction and was led and supported by clinicians involved in delivering the relevant services. Interactive methods involving small groups, such as citizens’ juries, could be very successful, although there may be difficulties in recruiting genuinely representative samples.

Our review also highlighted the importance of engaging with public representatives, in England primarily in the form of local authority scrutiny committees. The committees are important because of their power to refer disputed reconfiguration proposals to the IRP, an outcome that NHS decision-makers should seek to avoid. The IRP’s summary of lessons from its reviews1 was an important resource, as was the availability of informal advice from the IRP.

The IRP report noted that problems often arose because NHS decision-makers paid insufficient attention to issues considered important to the public. There was evidence from case studies45 and public opinion research22–25 that many people were unwilling to accept longer journey times in return for the promise of better-quality care at specialised centres. Since reconfiguration often involved centralisation of services, tackling such issues may warrant consideration by the leadership of the NHS at the national and local levels. There were potentially divergent issues to consider across other challenges to the NHS, such as decentralisation of services across several locations or moving services from one location to another.

Drawing on the included systematic reviews and exemplar case studies, our review broadly addressed the first two research questions, seeking to identify methods and impact of service user engagement in health service reconfiguration. Evidence on differential impact of methods, the sustainability of methods and outcomes, and methods of resolving differences between interested parties was less well documented. We noted some difficulty in distinguishing the stages of reconfiguration presently recommended by NHS England, but we uncovered some congruence between this guidance and what appears to be happening in practice, particularly in relation to the concentration of activity noted at stages 3 (discussion) and 5 (consultation). There is also some accordance between these specific NHS England stages and the theoretical basis provided by Arnstein’s ladder of participation. Across all evidence types, engagement activity appeared to move beyond basic information-giving towards the consultation and collaboration steps in the Arnstein’s ladder. It appears that future activity might usefully engage more readily with earlier and later stages in the process.

Given that service reconfiguration dominates the health policy agenda in almost all countries, it is essential to build upon the practical and research foundations that have already been laid.

Implications for health care

The NHS England stages of reconfiguration3 may provide a helpful framework on which to base plans for future service user engagement programmes. However, this framework should not necessarily be considered as a linear process or a set of distinct elements. Arnstein’s ‘ladder of engagement and participation’ (based on the work of Sherry Arnstein47) contained in NHS England guidance4 seems to offer less practical value. The four-dimensional framework developed by Gibson et al. 50 may be worth further evaluation.

Within the NHS England framework, some key factors contributing to successful engagement and/or service reconfiguration appear to be:

• ensuring a clear understanding of the local context

• early engagement; consulting widely

• demonstrating clinical-led case for change, with focus on service improvement rather than cost savings

• demonstrating openness and developing shared understanding of change through local partnership working

• promoting ownership of the change model and feedback results of engagement

• implementing strong managerial leadership

• using mixed approaches, particularly deliberative methods of engagement, targeted where necessary for different population groups

• considering access and transport issues as part of service change

• evaluation; follow-up

• expecting the unexpected.

Of these key factors, one aspect that seemed the most pressing was the striking need for robust evaluation and follow-up in user engagement programmes. Where evaluation has taken place – particularly in some potentially valuable case studies – it was largely poorly reported and difficult to appraise from a research viewpoint. Potential limiting factors are time constraints in health-care practice and naturally less concern for academic rigour. However, present reporting made it difficult to learn and move forward. Evaluation of user engagement in future health service reconfiguration is vital if we are to avoid reinventing the wheel each time public interaction is required. Ideally, evaluation should be conducted independently of those directing the engagement programme and should be embedded throughout the entire process from planning to implementation. The reporting system in Scotland may provide some helpful pointers. 42 Specific attention should be paid to explicitly and consistently describing the contextual characteristics of the situation; the methods of engagement, outcomes measured and overall impact (including positive and negative impact, and differential effects of engagement methods); the sustainability of efforts (through appropriate follow-up); and the lessons to be learned. The need to publish evidence on methods and impact of patient and public voice activity is touched upon in NHS England’s Transforming Participation in Health and Care. 4

Implications for research

One of the difficulties of the review was that much of the research was context-specific, which made generalisation difficult. Reconfiguration of services might be considered for the general population (such as primary care); for people with diagnosed conditions (such as mental health problems); for people seeking specific services (such as maternity services); or across geographical areas. Although the contexts are diverse, many challenges will be common.

The NHS England guidance on stages of reconfiguration may be of value in providing a generalisable approach and basis for user engagement in practice. The guidance may also provide a foundation for the design of future research on the evaluation of user engagement in service reconfiguration. These aspects are recommended areas of future research, together with an exploration of how the guidance might apply beyond the NHS setting.

In addition, further longer-term evaluations are needed to test the sustainability of methods of engagement and their impact over time. Prospective evaluations with contemporaneous data collection, including use of observational methods, may be the most suitable methods to achieve this. More research may also be warranted on the specific impact of interventions in negotiating and resolving differing opinions between patients, the public and clinical experts. Cost-effectiveness evaluation of engagement methods would be beneficial.

Contributions of authors

Jane Dalton, Duncan Chambers and Alison Eastwood carried out the study selection, data extraction and critical appraisal, and wrote up the report.

Melissa Harden conducted all searching.

Andrew Street and Gillian Parker provided expertise and advice, contributed to the development of the protocol and commented on drafts of the report.

External advice

The review authors are grateful to the following individuals for their valuable comments on the review protocol and/or the draft report:

Dr Helen Barratt, Hon. Clinical Lecturer (National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care North Thames). Dr Barratt was particularly helpful in providing information on further study sources and in relation to interpretation of the review findings.

Dr David Evans, Professor in Health Services Research (Public Involvement), University of the West of England.

Lucy Brown, Head of Communications, York Teaching Hospital NHS Foundation Trust.

Kay Gamble, Lead for Patient Experience, York Teaching Hospital NHS Foundation Trust.

Representatives of York Teaching Hospital NHS Foundation Trust were engaged with this review via Dr Jane Dalton (in her role as a publicly elected governor at the trust). This process was helpful in providing local context to the research and to enable discussion on potential exemplars for the review.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health.

Search strategies for reviews

Search strategies for primary studies

Search strategies to locate grey literature

The following websites were browsed manually and/or searched using the website search function where available, depending on the size of literature contained on the website. The searches were carried out during the period 11 April 2014 to 2 May 2014. The search was limited to documents published in English from 2000 onwards. A total of 187 relevant documents were identified.

A focused search of Google, using the advanced search, was carried out to locate reports on service reconfiguration. The search was limited to UK PDFs published in English from 2000 onwards with the term ‘reconfiguration’ in the title of the web page. The first 100 results were scanned for relevance. The search was carried out on 2 May 2014 and identified 18 reports.

Study: Abelson et al. (2010)6

Authors’ objectives: To evaluate the effectiveness of interactive strategies for engaging the public in the development of health-care policies and programmes at provincial/regional level. Particular attention is given to public engagement of rural populations (and to the New Brunswick context in Canada), in both official languages (English and French) and regarding the determinants of health.

Inclusion criteria: Original and review articles of empirical studies of public engagement methods, practices and evaluations in the development of health-care policy and programmes. Theoretical and conceptual work that helps inform the evaluation of public engagement. Key background papers of methodological and theoretical issues relating to public engagement. Languages restricted to English and French.

Exclusion criteria: Not stated.

Search dates: 2000–9.

Number of included studies: 34.

Included study designs: Cochrane review (1); systematic review (1); scoping review (1); narrative reviews (2); empirical studies evaluating a public engagement intervention in health care (12) (including three comparative evaluations; where reported, comparisons were different intensities, not different interventions); non-empirical papers of evaluation approaches and methods (8); Canadian grey literature documents containing empirical and non-empirical studies (9).

Countries of included studies: 12 published empirical studies include Canada (7); the UK (3); the USA (1); and Australia (1).

Types of reconfiguration: Published empirical studies: priority-setting, planning and policy development in a number of areas including local health goal-setting; health promotion and healthy public policy; cancer-specific and general health service planning and delivery; and policy development related to new health technologies.

Definitions of engagement/involvement: Review authors’ definition: a category of methods for involving citizens in health-care decision-making that incorporates at least three elements, (1) the provision of information to participants about the topic/issue being discussed; (2) the opportunity for interactive discussion among participants and potentially between participants which the public engagement sponsors; (3) an explicit process for collecting individual or collective input. Definitions are also provided for included reviews.

Who was engaged/involved: Urban/rural/Francophone/Aboriginal populations. Participants recruited from community-based organisations; local residents/citizens/stakeholders/hard-to-reach groups.

Methods of engagement/involvement: Ad hoc deliberative meetings (e.g. citizens’ panels/juries) held over 1–3 days; long-term collaborative partnerships over several months or years. Three studies looked at partnerships as models of, or inputs to, effective and sustainable public engagement.

Method of synthesis: Critical interpretive synthesis (Dixon-Woods et al. 200653): conceptual translation of quantitative/qualitative/non-empirical studies.

Conceptual/theoretical framework or logic model used: There is a contextual framework mentioned (figure 1, p. 6) but it is not explicitly stated that this was used. Many of the included studies employed a conceptual framework in their evaluation.

Research questions addressed: 1, 4.

Stages of involvement: 1, 2, 3, 4?, 5.

Levels of involvement: Informing; consulting; involving; collaborating.

Results: Results from the review of reviews indicated that greater conceptual clarity is needed about the meaning of effective public engagement and common evaluative criteria, as is more rigorous evaluation of the effects of public engagement on a range of outcomes of interest. Key messages from the 12 published empirical studies: interactive public engagement can be implemented successfully in various situations, and success is dependent on contextual variables. Organisational commitment and topic-related characteristics appear to be key contextual variables. Public engagement mechanisms need to be adapted to the wider context of policy development around the issue, including topic, the group(s) to be engaged, the history of the issue and the perceived power dynamics. Skills required to conduct interactive processes can be learned in a supportive organisational environment. Participant satisfaction and increased topic-specific learning is higher when the engagement process is well designed, although process satisfaction does not necessarily lead to perceived impact of participation on policy decision-making. Group debate features as an important contributor to satisfaction. Partnerships are central to the effectiveness of community-based public engagement strategies. Institutionalisation of partnerships beyond the active phase can enable sustainable change. Interactive public engagement can influence participant views, but is less likely to change more dominant views. Results from the Canadian grey literature highlighted the need for open processes of engagement, with the involvement of policy-makers and participants in setting indicators, benchmarks and goals in evaluating public engagement activity. Further results are reported in the paper from the published non-empirical literature (focusing on theoretical and conceptual work in public engagement evaluation). Further results are also reported on engaging the public about the determinants of health; engaging rural populations; public engagement in different languages; and combining theory and practice.

Authors’ conclusions: Current interest in public engagement among Canadian health system managers and policy-makers needs to be matched by clear thinking from all interested parties (researchers, managers and policy-makers) about the terminology, goals, theoretical properties and benefits of public engagement.

Comments: Not all focused on reconfiguration. Related to this review are those focusing on priority-setting and resource allocation. Not clear if any study overlap with Mitton et al. (2009),20 as Mitton did not present study details. Crawford et al. (2002)19 also included in the Abelson review.

Study: Conklin et al. (2012)16

Authors’ objectives: To evaluate the outcomes of public involvement in health-care decision-making, priority-setting, resource allocation and health service planning at the macro and meso levels.

Inclusion criteria: Focus on evaluation studies. For this review, involvement was conceptualised as taking a broader societal perspective on public involvement where the public were (1) lay citizens/community members, (2) representatives of organised social interest groups, such as coalitions, partnerships, (3) organisations of staff members and/or patients/customers/clients, and input was from a broad group (not individual) perspective. Any outcome or impact was eligible, as defined by the study authors.

Exclusion criteria: Surveys of beliefs/attitudes about participation. Studies that examined involvement in service development, shared decision-making of individual treatment, clinical decisions, participatory action research, therapeutic and service delivery decisions, and clinical guidelines. Studies of preferences for participation among health-care professionals or users. Reviews, editorials, commentaries. Studies in languages other than English, German or French.

Search dates: Published articles from November 2000 to April 2010 were sought.

Number of included studies: 19.

Included study designs: Nine descriptive case studies; 10 empirical studies (cross-sectional; comparative pre–post design; qualitative; mixed methods). Follow-up (where reported) ranged from 1 year to 7 years.

Countries of included studies: Majority in England, Canada, the USA and Northern Ireland. Others included France, the Netherlands, Italy and Israel.

Types of reconfiguration: Relevant studies (where reconfiguration was the goal). Examples: resource allocation relating to local health integration networks; shaping policies and decisions about primary care provision and delivery; health-care priority setting; health policy decisions about the delivery of women’s health services; decision-making about local health services.

Definitions of engagement/involvement: The review authors defined six levels of involvement (consultation, participation, engagement, partnership, community development, representation), but stated that these were often used interchangeably, even within included studies.

Who was engaged/involved: Special interest groups; the public; patients; staff; NHS lay board members. More specific definitions of ‘the public’ varied and were generally unclear across the studies. Terms used: representatives of patient organisations; ordinary citizens; individuals with no particular axe to grind; those whose voices might not otherwise be heard.

Methods of engagement/involvement: Surveys; conference and website; community health councils; public meetings; local patient groups; citizen panels/juries; group simulation using roulette wheel; collaboration between agencies/groups/individuals.

Method of synthesis: Narrative.

Conceptual/theoretical framework or logic model used: Not stated.

Research questions addressed: 1, 2, 3, 4.

Stages of involvement: 3, 4, 5, 6, 7.

Levels of involvement: Informing; consulting; involving; collaborating.

Results: Where reported, details of specific and measurable data on outcomes were generally lacking. Studies demonstrated how public involvement had influenced decision-making and led to acceptance and implementation (one study involving small group meetings/plenary sessions); practical changes/improvement were achieved (three studies involving surveys and group meetings); priorities (identified through the involvement activity) were integrated to a regional programme (one study); and resources had been found for new services (three studies involving citizen’s juries, community group representations/collaborations). One study demonstrated repeatability of regional meetings (a deliberative consultation activity) in two other organisations. Other studies suggested that the impact or influence of public involvement on shaping strategic decisions was found to be minimal (three studies), geographically variable (two studies) or mixed (one study). Common outcome measures were participant views or ‘change’ as a measure of influence of public engagement on decisions, policy and practice. Some studies reported improved learning or knowledge of processes involved in consultation and priority-setting, and benefits for deliberative procedures. Negative consequences of public involvement were reported as costs and unintended outcomes for participating stakeholders when challenged by members of the public.

Authors’ conclusions: Evidence of impact on public involvement in health-care policy remains scarce, despite the growing body of work. Firm conclusions about effective and appropriate involvement activities are difficult to draw. Focus on outcomes risks missing the normative argument that involving the public in the health-care policy process may be seen to be of intrinsic value.

Comments: This review was designed as a scoping review.

Study: Crawford et al. (2002)17

Authors’ objectives: To examine the effects of involving patients in the planning and development of health care.

Inclusion criteria: Studies of patient involvement (defined as the active participation in the planning, monitoring and development of health services of patients, patient representatives and the wider public as potential patients) in the planning and development of health care.

Exclusion criteria: Studies of patient involvement in research, community development, health promotion, self-management and self-help, and the role of doctors and patients in determining individual treatment choices. Studies measuring patient satisfaction or patient preferences without describing how information was used to help develop a service.

Search dates: 1966 to 2000.

Number of included studies: 42.

Included study designs: Case reports (31); surveys (5); review of minutes of meetings (3); action research (3). Four of the included studies used qualitative research methods.

Countries of included studies: The UK (28); Sweden; the USA; Australia; and Canada.

Types of reconfiguration: Various services, including primary care, mental health, learning and physical disability, general health care, community services, inpatient and outpatient, social care, maternity, neurology, human immunodeficiency virus. Most studies looked at smaller-scale change. Approximately one-quarter of studies focused on larger-scale change, including changes to organisation of care and/or services. Of these studies, two involved a plan for hospital closure.

Definitions of engagement/involvement: Review author’s definition: active patient participation in the planning, monitoring and development of health services of patients, patient representatives and the wider public as potential patients.

Who was engaged/involved: Most studies described participants as patients. Others reported involvement of carers, service users, staff, health and welfare council, community health council, citizens, lay board of directors or mixed populations.

Methods of engagement/involvement: Patient groups, consultation meetings, committees and forums; interviews; citizen’s juries; survey; focus groups; representation on planning boards and panels; mixed methods.

Method of synthesis: Narrative.

Conceptual/theoretical framework or logic model used: No.

Research questions addressed: 1, 2, 4.

Stages of involvement: 3?, 4?, 5, 6.

Levels of involvement: Informing; consulting; involving; collaborating.

Results: Most frequently cited effects of patient involvement were the production of new or improved sources of information for patients and making services more accessible (smaller-scale changes). Seven reports described new services being commissioned, and two reports described hospital closure proposals being modified or abandoned as a result of patient involvement. There was a mixed response from patients in terms of satisfaction with the process of involvement. Organisational attitudes to involving patients became more favourable (four reports), resulted in culture change to being more open to involving patients (two reports) and resulted in further involvement projects (two reports). Other reports (6) discussed less favourable consequences, such as involvement being used to legitimise decisions already made, or a slower decision-making process.

Authors’ conclusions: Involving patients has contributed to changes in the provision of services across a range of different settings. The evidence base for the effects on use of services, quality of care, satisfaction or health of patients does not exist.

Comments: This review appears to be a subset (based on health-care context only) of that in Crawford et al. (2003)18 (which contains a wider scope of sectors). The wider review has a different objective: to look at transferability of user involvement from other sectors to health care. Therefore, it appears appropriate to treat these as two separate reviews.

Study: Crawford et al. (2003)18

Authors’ objectives: To evaluate user involvement in change management across a range of sectors (health, other public services, voluntary and private sectors). To identify factors that promote successful user engagement, and identify relevance to the NHS.

Inclusion criteria: Any study design focusing on the aims, processes and outcomes of service user involvement in change management at local organisational level.

Exclusion criteria: Studies not directly related to change management (such as campaigns for change in policy at national level).

Search dates: Databases: published after 1980, to 2002. Other searches (where reported) to 2002.

Number of included studies: 344 (of those reporting outcomes, 55 related to public sector health, social care and community care; 19 related to public sector non-health; and 23 related to the private sector). See Appendix 3.

Included study designs: Most were qualitative process evaluations or discussion papers on case studies.

Countries of included studies: Largely the UK. Others: Australia, Sweden, Denmark, the USA, Canada, Austria, Japan. See Appendix 3.

Types of reconfiguration: Specific reconfiguration not described. Reconfiguration contexts described as various within health, social and community care; non-health public sector (including postal services, social security, education, housing); private sector (including consumer goods, travel, entertainment); and voluntary sector (disability/neurological services).

Definitions of engagement/involvement: Review authors’ definition: taking an active part in change management at the organisational level.

Who was engaged/involved: Current, ex- and potential service users and their representatives. Providers.

Methods of engagement/involvement: Various, covering time-limited methods (to elicit user perceptions/preferences) and long-term approaches (building relationships with service users). Some initiated by provider; others initiated by service users. Public sector tended to use more deliberative approaches. Examples: surveys, focus groups; deliberative approaches (such as citizen’s juries, public conferences); user/community groups; formal bodies (such as community health councils, patient groups, advocates and link workers).

Method of synthesis: Thematic (using qualitative methods).

Conceptual/theoretical framework or logic model used: Not stated.

Research questions addressed: 1, 2, 3, 4.

Stages of involvement: Not clear. Difficult to apply to such a general review.

Levels of involvement: Informing; consulting; involving; collaborating; devolving.

Results: User involvement in health care (represents the majority of evidence in this review): small numbers of reports attributed changes in services to user involvement. The impact of changes on service quality has not been evaluated. User involvement in other public sector services: similar issues to health care, but greater emphasis on involving local residents who are not service users. Emphasis on complaints procedure in social services. User involvement in the voluntary sector: very limited number of reports. Emphasis on user-led services. User involvement in the private sector: user engagement more frequently applied to shaping products and services (not structural or organisational policy change). Less emphasis on methods of involvement than other sectors. Quality of evidence reported to be poor (reported for private sector only). See Crawford et al. (2003)18 pp. 10–11.

Authors’ conclusions: There is little evidence of independent research in any of the sectors examined. No best method for involving users within the NHS. Methods need to be selected on the basis of aims of involvement, capacity of local users and providers, and financial/other constraints. Facilitators of successful user involvement appear to be increase clarity of aims of involvement to service users; increase willingness of users to engage; reduce organisational resistance to user involvement; and increase appropriately resourced evaluation on where user involvement has made a difference.

Comments: In the first 55 studies (on public sector health, social care and community care) this review overlaps with Crawford et al. (2002)17 (their review of 42 studies focusing on health care).

Study: Daykin et al. (2007)19

Authors’ objectives: To evaluate the impact of strategies, including the Patient Advice and Liaison Service, for PPI in health care.

Inclusion criteria: Initial inclusion criteria: quantitative studies (with measurable outcomes) or qualitative research focusing on defined PPI/user involvement interventions in an identified health-care population. English language and published between 2000 and 2005. Second stage inclusion criteria: randomised controlled trial/experimental studies. Quantitative studies measuring outcomes against an identified baseline. Qualitative and mixed methods studies focusing on successful or unsuccessful elements of the strategy under investigation.

Exclusion criteria: Second stage exclusion criteria: studies focusing on individual treatment and care. Anecdotal reporting only, no recognisable research process.

Search dates: 2000 to 2005.

Number of included studies: Eight evaluations.

Included study designs: Four case studies; two independent evaluations using qualitative methods; one project evaluation using mixed methods; one action research study.

Countries of included studies: The UK.

Types of reconfiguration: General/strategic development; cancer services; mental health services; older people’s services.

Definitions of engagement/involvement: Definition not stated.

Who was engaged/involved: Staff, members of the public, patients.

Methods of engagement/involvement: Employment of an individual to work with community groups; interprofessional cancer education programme; user groups; forums for service users and officials; community-based exercise facility for people with mental health problems; citizen’s juries; community initiative to elicit and respond to the views of older people; regional-level action research programme with staff.

Method of synthesis: Narrative.

Conceptual/theoretical framework or logic model used: Realist framework: identifying contexts and mechanisms that lead to favourable or unfavourable outcomes.

Research questions addressed: 1, 2.

Stages of involvement: Not clear. Possibly 3, 4?, 5.

Levels of involvement: Informing; consulting; collaborating.

Results: Difficulties were reported in documenting impacts from PPI. Problems were a result of time lag between intervention and outcomes, and difficulty isolating the direct impact of PPI from other potential contributory factors. Social benefits (two studies): engagement with community groups and activists can assist in shifting service provision towards a social model of care. New services provided as a result of PPI intervention. PPI can help to destigmatise mental illness and offer a means of reintegration to the community. Other outcomes were reported, but they did not correspond with the review inclusion criteria (e.g. focused on improvements to individual care and in relation to community health improvement). Prerequisites for effective PPI identified as follows. Structure and resources: organisational structures that support partnership working; community capacity for change; programme-level structures to support staff and safeguard users’ well-being; and mechanisms to identify impact of user involvement on decisions. Politics and discourse: commitment to deliberative process involving discursive democracy and community development. Attitudes and culture: adaptations and change by staff needed to enable successful participation of users in service development. Specific results in relation to Patient Advice and Liaison Service are reported in the paper.

Authors’ conclusions: Authors’ conclusion appeared to focus on limitations of the evidence, in terms of relevance to the evaluation of PPI and quality criteria.

Comments: The review authors excluded a number of reviews on the basis that they did not meet the review criteria.

Study: Mitton et al. (2009)20

Authors’ objectives: To examine methods and impact of public engagement in health-care priority-setting decision-making.

Inclusion criteria: Study design: English-language studies in peer-reviewed journals. Population: ordinary or lay citizens/community members; representatives of organised social interest groups (including coalitions, partnerships); organisation staff members/employees and/or patients/customers/clients. Intervention: public engagement or involvement in decision-making. Setting/context: decisions at macro (high-level funding) and meso (specific services and programmes) levels; decisions about treatment eligibility at group or typical patient level; monitoring and evaluation relating to priority-setting or resource allocation. Perspective: societal.

Exclusion criteria: Study design: news items, letters, editorials, book reviews, non-peer-reviewed journal articles. Population: exclusively professionals or experts. Context/setting: studies of wider policy frameworks and choices, (such as clinical rationing) unless they focused directly on how public engagement was used to develop such frameworks. Perspective: individual.

Search dates: 1981 to 2006.

Number of included studies: 175 articles (190 distinct cases of public engagement).

Included study designs: Observational.

Countries of included studies: The USA (40%), the UK (26%), the rest of Europe (8%), Canada (9%), Australia/New Zealand (7%) and elsewhere (10%).

Types of reconfiguration: Largely macro-level decision-making. General assessments of public priorities for government spending or development of broad vision statements in health care (97 cases). Non-health care (largely environmental science; urban planning) (77 cases). Most health care reconfiguration focused on public health and health promotion. Others on general policy and spending, regional health authorities and integrated systems, mental health, acute care, pharmaceuticals or health technology assessment, and long-term care. Sixteen cases included health and non-health care sectors.

Definitions of engagement/involvement: Three levels: communication; consultation; participation.

Who was engaged/involved: Where defined (167 of 190 cases): public as individuals (speaking on their own behalf) (42). Public as organised interest groups (speaking on behalf of a membership) (20). Public as patients or consumers of services (speaking on issues outside their personal experience) (14). Most of the 190 cases reported engagement with multiple public audiences. Thirty-eight per cent of cases reported particular attention to disadvantaged populations or groups with special needs.

Methods of engagement/involvement: Communication: traditional publicity; public meeting/hearing; drop-in centre; internet information; telephone hotline. Consultation: opinion poll/survey; referendum; consultation document with select population; electronic consultation/interactive website; focus group; study circle or open space; standing citizens’ advisory panel. Participation: citizens’ jury or consensus conference; negotiated rule-making or task force; deliberative poll or planning cell; town meeting with voting. Most cases (93 of 183 codable) used multiple methods. In total, 405 techniques were employed. Fifty-eight per cent consultation; 24% communication; 18% participation. From the codable data, 49% one-off events; 45% ongoing over a period of time (mean duration approximately 4 years). Forty per cent of cases involved face-to-face interaction, and this was more likely to occur within deliberative engagement processes.

Method of synthesis: Narrative.

Conceptual/theoretical framework or logic model used: Lomas (1997):54 three-level scale to assess the scope of public engagement (with added fourth category of monitoring and evaluation). Rowe and Frewer (2005):55 coding framework for intensity and methods of public engagement.

Research questions addressed: 1, 2, 3, 4.

Stages of involvement: 1 (largely macro-level priority-setting).

Levels of involvement: Informing; consulting; involving; collaborating.

Results: Evaluation of effectiveness carried out in only 32% of included cases, and most of this was process evaluation. No direct experimental comparison of different methods. Despite the lack of evaluation, two-thirds of included articles concluded that participation processes were successful (as defined by the study authors). Six per cent of cases were deemed to be unsuccessful and 14% inconclusive. Cases using deliberative methods were more successful than those employing lower levels of engagement (e.g. communication or consultation). Cases with face-to-face contact were perceived to be more successful than those without direct interaction. In studies where the intention of engagement was making a difference (or leading to a decision other than what would have been arrived at in the absence of engagement), 60% of cases stated an impact; 10% stated this had not been achieved; and in 30% of cases the impact was unclear or unspecified.

Authors’ conclusions: Some practices for public engagement in priority-setting are promising; for example, use of multiple methods and balancing broad consultations with in-depth engagement using new deliberative techniques. Lack of evaluation limits the ability to suggest supportive evidence for any particular approach to public engagement in priority-setting.

Study: Mockford et al. (2012)5

Authors’ objectives: To identify the impact of PPI on UK NHS health-care services and to identify the economic cost of PPI. Secondary objectives were to examine how PPI has been defined, theorised and conceptualised, and how the impact of PPI is captured or measured.

Inclusion criteria: Studies of user/patient activity which involved patients, carers or the public working (1) in a collaborative way with health professionals or management or (2) in a user-led way. All types of study designs were eligible except those specifically excluded.

Exclusion criteria: Studies of children’s services; health research; NHS research and development; voluntary sector; independent health-care services; and educational services. Studies of involvement as part of an individual’s health care. Discussion papers, ‘think pieces’ and editorials.

Search dates: 1997 to 2009.

Number of included studies: 28 (42 publications).

Included study designs: Case studies (20); ‘evaluations’ (5); secondary data analyses (2); survey (1).

Countries of included studies: The UK.

Types of reconfiguration: General.

Definitions of engagement/involvement: Not reported. Authors stated that included studies rarely provided an explicit definition of PPI, engagement or similar terms.

Who was engaged/involved: Patients/carers; the public.

Methods of engagement/involvement: Lay membership of boards, panels and working groups; user groups.

Method of synthesis: Narrative.

Conceptual/theoretical framework or logic model used: No.

Research questions addressed: 1, 2.

Stages of involvement: Unclear.

Levels of involvement: Consulting; involving; collaborating.

Results: Results indicated that PPI took many forms and had a range of impacts on health-care services, including design, location of and access to services; provision of additional services; reorganisation of existing services; and changes in organisation of acute trusts. Reporting quality was poor and there was little evidence on costs.

Authors’ conclusions: There is a need for significant strengthening of the evidence base around the impact of PPI on NHS health-care services. Development of clear concepts and robust forms of measurement is required.

Comments: Focused on small-scale (such as ward-level) change as well as larger-scale change.

Study: Rose et al. (2003)21

Authors’ objectives: To evaluate user and carer involvement in organisational change management in a mental health context.

Inclusion criteria: English-language articles published between 1987 and 2002. Studies of user and carer involvement in organisational change in a mental health context.

Exclusion criteria: Participants aged under 18 years and over 65 years. User/carer involvement in auditing and evaluation of services, where changes resulting from involvement are not described.

Search dates: 1987 to 2002.

Number of included studies: At least 112.

Included study designs: Most included studies were discussion papers (61 discussing practice; 43 discussing concept; some discussed both). There were 34 qualitative papers, 16 quantitative papers (including four trials), seven organisational reports and two unpublished dissertations.

Countries of included studies: The UK (71); the USA (42); Europe (1); Australia/New Zealand (3); and more than one country (9).

Types of reconfiguration: Promoting democracy and representation and/or cultural change (over 50% of included studies). Strategic planning, restructuring of services (very few studies) and policy initiatives. New service provision and the employment of service users in organisations.

Definitions of engagement/involvement: Review authors’ definition: consumerism (individual), including day-to-day personal care planning; consumerism (collective), group involvement in the planning, delivery, implementation and monitoring of services; citizenship.

Who was engaged/involved: Majority of studies focused on service users and professional staff. One-quarter of studies involved carers. Others involved: user groups, carer groups, the public, community health councils.

Methods of engagement/involvement: Majority of studies focused on collective consumerism, involving consultation, representation, partnership, evaluation, involvement in staff recruitment. Methods not explicitly stated.

Method of synthesis: Narrative. Use of coding frame to elicit most frequently occurring aspects of user involvement activity.

Conceptual/theoretical framework or logic model used: Review authors’ definition could also be classed as a conceptual framework for the review: consumerism (individual), including day-to-day personal care planning; consumerism (collective), group involvement in the planning, delivery, implementation and monitoring of services; citizenship. Adapted version of Arnstein’s ladder of participation was also used.

Research questions addressed: 1.

Stages of involvement: 5.

Levels of involvement: Consulting; involving; collaborating.

Results: Outcomes were rarely measurable (and based on subjective judgement of the study author). Fifty-eight papers reported unknown or unclear outcomes; 52 papers reported positive outcomes; 14 reported negative outcomes of user involvement. The included trials did not focus on reconfiguration. Among the other studies, various relationships were proposed between the type of user involvement and type of change. For example, collective consumerism was associated with restructuring and strategic planning but could equally be conceptualised as tokenism. Partnership working showed no systematic relationship with any form of organisational change (small number of reports). Citizenship and campaigning were associated with change of mission. User control was associated with new service provision. Drivers for successful user involvement included a facilitating organisational culture, information provision, funding and service user training.

Authors’ conclusions: The review shows a complex picture of user and carer involvement in change management. The evidence is diverse and of variable quality.