Using patient experience data to support improvements in inpatient mental health care: the EURIPIDES multimethod study

What do we mean by programme theory?

Programme theories explain how a programme or intervention works. They embody what the problem is, what the solution might be and how this is hypothesised to bring about change, and so encapsulate ideas about causation. Programme theory should not be so specific that it explains things only at the individual level and should not be so abstract that it is generally applicable.

Programme theory is a ‘middle-range theory’ that helps to explain in what ways and how a programme or intervention may or may not be operating successfully in particular circumstances for particular types of actors. Initial programme theory can be developed in a range of ways. In our study, the initial programme theories were based on previously published research, practice knowledge and lived experience. These explanations about how the programme or intervention works were drawn from published literature, practitioners, people with lived experience of mental illness and distress, and the NIHR funding call. We used these initial programme theories to develop the research proposal. These were then refined throughout the study, based on the outcomes of each WP.

What do we mean by generative mechanisms?

Programmes or interventions work or do not work as a result of different individuals responding (or not responding) to that programme or intervention. These responses result in different patterns of outcomes. The biological, psychological or social drivers underpinning each individual’s response are referred to as generative mechanisms. 37,38

Mechanisms are likely to operate at different levels of a system, over different time scales, and to involve interactions that may not be observable. If we define mechanisms in relation to the response of individuals to a programme or intervention, there are only a finite number of ways in which they can respond (e.g. in respect of the experience of inpatient mental health care, trusting staff or feeling disempowered). These responses are commonly recognised as (observable) aspects of human behaviour; however, what is not clear is how these responses are triggered by particular aspects of programmes or interventions in specific contexts. It is not just high-level mechanisms such as trust that we need to understand, but we also need to understand how the programme/intervention works in triggering individual responses that are characterised as the experience of trust.

Realist evaluation and context–mechanism–outcome configurations

Realist evaluation is operationalised through developing context–mechanism–outcome (CMO) configurations. In realist evaluation, outcomes are understood as the product of the context and mechanism. For example, trust may be an obvious mechanism in relation to patient experience; however, how that trust becomes activated (or not) in the inpatient setting is less clear. Understanding this is the work of realist evaluation. 39,40

The ‘context’ of realist evaluation is significant, as it can influence reasoning and behaviour, and it is the context that ‘activates’ (through the circumstances being right) mechanisms and that influences if, and if so which, mechanisms operate. 36,37 Outcomes are contingent, therefore, on both context (which may provide alternative explanations of different observed patterns of outcomes) and mechanism (based on the reasoning and resources38 of actors).

The development of CMO configurations helps to understand both proximal outcomes (which are more immediate and the result of generative mechanisms being activated in relation to the reasoning and resources of actors) and distal outcomes (which are more slowly developing patterns that are observed and built, often from the accumulation of proximal outcomes).

In this study, the outcome being examined was the collection, analysis and use of patient experience data in order to change services, and linked to quality improvement strategies in adult inpatient mental health settings in NHS England.

What do we mean by laminations in case study design?

In a classic case study design, a case would be clearly bounded as a unit of analysis and viewed as embedded within a wider system,42 and extraneous factors are viewed in relation to their impact on that case. If utilising a classic multiple case study research design, one would compare distinct and bounded cases. 43 Each case would be discrete and the comparison would take place between cases (Figure 3).

FIGURE 3.

Classic case study research design comparing between cases (adapted with permission from Fenton44).

adapted with permission from Fenton44

We adopted a different type of case study research design using the idea of a laminated system43,44 with defined, distinct levels of agency. We operationalised the idea of laminations in our research with the individual- or biographical-level experience understood as patients or carers, the micro- and small-group-level understood as staff working on the wards, the meso-level understood as corporate staff including PELs and senior managers in trusts, and the macro-level understood as policy-makers and organisations responsible for collecting nationally mandated patient experience feedback (Figure 4). Research interviews were conducted across these laminated levels (WP2 and WP3).

FIGURE 4.

Inverted case study research design comparing across and within the laminated structure (adapted with permission from Fenton44).

adapted with permission from Fenton44

To utilise the concept of laminations within a realist evaluation framework, we undertook research in these sites (WP3) to explore and understand the generative mechanisms operating within and between laminations in each of the case sites. 44 This approach enabled us to read down and through those laminations to understand the generative mechanisms being activated (or not) at different levels in case sites, and facilitated reading across the case sites at the level of each lamination to understand the reach and impact of extraneous factors.

The rationale for using this particular research design was twofold:

1. Rather than use a lamina (or level; interviews with inpatients, for example) as a window into those patients as the object of study, we used those interviews as windows out onto the wider system. While we used context to inform our understanding of generative mechanisms, we privileged narratives at the different levels identified. This makes it is easier to determine what contributes to middle-range theory and to determine those generative mechanisms that are particularly context driven and so relevant to specific case contexts.

2. Case studies traditionally have strong internal validity but suffer from weak external validity and generalisability from the case to the wider population. 42 In our approach, although cases were identified and bounded, we were able to read within levels across case sites to explore what features of a programme or intervention are common to everyone at that laminate level and what features are contextually driven and unique to the case. 44 This strengthened our external validity, without compromising or weakening our internal validity.

How was temporal sequencing introduced to realist evaluation?

Using inpatient experience feedback to improve the quality of care has an embedded temporal sequence, which forms part of the context for our realist evaluation. We brought this into our realist analysis through using a temporal coding framework: collecting, receiving, analysing, implementation and change (see Report Supplementary Material 10).

We adapted and developed these laminated and temporal additions to our realist analysis to ensure generative mechanisms were not identified at too superficial a level to be useful and to understand how combinations of different actors’ reasoning and different contexts interacted to produce different patterns of outcomes.

Patient and public involvement team

The PPIT comprised people who were unlikely to have prior experience of, or involvement with, research, but were experts by experience, having had historical or recent experience of adult mental health inpatient admission or of informal care for someone who had been an inpatient.

We aimed to recruit 8–10 PPIT members through the MHF’s extensive network, ensuring that the group included a degree of diversity with respect to age, gender, ethnicity and geographical location.

Our initial meeting was poorly attended, for a variety of reasons. Only two of the eight people recruited attended, with four sending apologies in advance and two failing to arrive owing to transport issues. After consultation with a number of the project team members and discussions with a service user and carer group, which met regularly close to the MHF’s London office (at a club known as the Dragon Café), we agreed to strengthen the PPIT with members from the Dragon Café. We also increased the payment offered to PPIT members and provided additional support to ensure that people were able to attend.

This led to a much stronger attendance at subsequent meetings and, because the new members knew each other, the PPIT immediately gelled as an entire group. There were a number of consequences of this approach. We sacrificed some geographical diversity (the vast majority of the PPIT were from London, with one regular attendee from the West Midlands and one from the south coast), but we increased ethnic and age diversity. We also increased the number of people involved who had current or very recent experience of inpatient care. Only a small proportion of the group had any experience of PPI in academic research.

We therefore increased support to ensure that the PPIT meetings were accessible for the participants. Initially, we proposed that there would be four PPIT meetings during the lifespan of the study. It became apparent after the first meeting that there were ways in which we might be able to do things differently and more meaningfully. PPIT members gave feedback after the first meeting and it was clear that, to make the most of their expertise, we would need to take more time to get know each other and we would need to plan our meetings more creatively (and with much less paper). As a consequence, we held seven PPIT meetings over 2 years. The format of these meetings was innovative and the benefits were significant. Adaptation enabled more equitable and honest engagement that allowed the PPIT to challenge the research team about the data and ways of working.

Some members of the PPIT found it difficult to maintain concentration, some were less confident with literacy and technical language, and some were voice hearing. There were periods over the 2 years of the study in which people were re-admitted to inpatient units or suffered significant life events. We worked to ensure that nobody felt left out or left behind. We adapted our approach to ensure that everyone was able to participate at the level they felt was appropriate. We used role play and small group discussions and we built in time for plenty of breaks and sandwiches. We visited the group in locations familiar to them outside the formal meetings to ensure that they had space for reflection and to ask for changes or updates ahead of the next meetings. This enabled us to engage people with no prior research experience.

A link worker at MHF (Jo Ackerman) supported the group outside the meeting times to ensure that there were opportunities for everyone to articulate their own opinions, both during and outside meetings. We ensured that travel arrangements were as simple as possible and that no one was left out of pocket on the days of the meeting. Together, we thought carefully about how to support the group in the context of their EURIPIDES study meetings. Working in this way has allowed us to be sensitive to disclosures and helped us to manage risk when it presented.

We began each meeting by demonstrating how the research had been shaped by the PPIT’s involvement (i.e. from changing the ways we interviewed and the tools we used, to implementing a coding structure that they helped us co-design). We also adapted our techniques of working with the PPIT to meaningfully include everybody and let them lead on the approach taken (i.e. through the workshop-style activities that included dramatisation and role play as opposed to reviewing textual materials). It was particularly important to be responsive to fluctuations in the group dynamics if people were unwell and to keep striving to engage people in a supportive way that was accepting of their individual situations. We co-designed (by asking how, in what ways and when they felt the next meetings should take place) the approach, not just the study, and we recognised their role as integral to steering the project.

The members of the PPIT played a critical role in the design of the research tools, in designing coding frameworks and in reflecting on the data throughout analysis. For example, they commented on the systematic review and they refined programme theories (the hypotheses we were testing), and their comments relating to the systematic review findings about what patients said about their experience were built into the interview tools for patients through a process of co-designing flash cards to support realist interviewing (WP3).

In addition, the PPIT chose the priorities for the ongoing research questions in their meeting forum and this shaped the entire data collection process. The PPIT also developed and refined our programme theories (the core work of realist analysis). This work refined our research questions and shaped the conduct of the fieldwork. The PPIT analysis workshop provided us with categories crucial to our data-coding. The impact of this involvement has been significant and it has led each planning and development phase of the primary data collection. Members of the PPIT attended the consensus conference and, to mark the end of the study, we hosted a celebration event to thank participants and recognise the enormous contribution they made to the EURIPIDES study.

Survivor researchers

The survivor researchers (who were people with lived experience of mental illness and its treatment who also held professional roles and had experience conducting research) were people who either had previously worked with the MHF or were recommended by the MHF. We adopted the term ‘survivor’ (rather than ‘patient’ or ‘service user’), as this was preferred by the individuals concerned.

The intention was for survivor researchers to be involved in the study by (1) advising on the questionnaire, topic guide and recruitment materials and on selection of case study sites, (2) advising on all aspects of the conduct of WP3, including planning and undertaking interviews with service users and (3) recruiting service user and carer participants and acting as facilitators at the consensus conference (WP4).

Four survivor researchers were initially recruited to the EURIPIDES study team through the National Survivor User Network and MHF networks and were actively involved in the early stages of the research: advising on recruitment materials, topic guides and approaches to realist interviewing. However, we had a difficult start to the fieldwork, as gaining access to case study sites was complex and time-consuming, as was the bureaucracy around research passports. This contributed to communication issues with the survivor researchers and impeded involvement. In addition, two of the survivor researchers made the decision to withdraw from the study prior to the fieldwork. Once research passport administration issues had been resolved, the two remaining survivor researchers were involved in data collection at only two of the six case study sites.

As the project progressed, we were able to integrate the survivor researchers into the project. Survivor researchers were able to exchange ideas and perspectives during the last two site visits with other members of the research team and were also involved in two meetings to design and refine coding frameworks at the analysis stage. One of the survivor researchers (Emma Ormerod) also participated in role plays of some sections of the interviews for discussion at the PPIT meetings. Both survivor researchers were involved in the final stages of the research process and co-authored this chapter.

Limitations

There were also limitations to the PPI work undertaken in the course of this research. During the early stages of the project, it may have helped with communication and mutual trust to have spent additional time getting to know each other by sharing different perspectives and experiences. It may also have helped for the whole team to have met together (PPIT, survivor researchers and other members of the research team), as this did not happen until the later stages of the research process. For example, when some members of the PPIT met one of the survivor researchers at a later stage in the project, they expressed surprise at realising that service users can also work as researchers (i.e. that some use their own experiences as part of their work as ‘survivor researchers’ and others work as researchers but choose not to integrate their experiences into their research work). Discussions of this nature may have been encouraging for some members of the PPIT, in addition to acknowledging that research skills can be obtained through various means – not just postgraduate and doctoral routes.

Time constraints and limited involvement in fieldwork prevented the survivor researchers from developing their approaches to realist interviewing informed by their own lived experiences of mental distress. The two WP3 case study sites where the survivor researchers were involved were ‘green-rated’ sites that had scored most highly in the WP2 survey of inpatient providers. As some interviews were conducted jointly (with other study researchers) as well as independently, we were unable to fully assess the extent to which specific interview dynamics between interviewee and researcher with lived experience might have shaped the study findings. This is something that could be reflected on as part of a collaborative research process in future projects.

Sampling

We invited members of the PPIT, members of the research team most involved in working with the PPIT, colleagues from the MHF and the survivor researchers to take part in these research interviews. Potential participants were given the option of taking part in a one-to-one interview or a group interview.

Data collection

At a pre-evaluation meeting with the PPIT, we co-designed a timeline for the project, which captured memorable milestones. We subsequently developed an interview guide around this timeline. The interview guide explored people’s experiences of being involved with the EURIPIDES study and also asked about key barriers to and facilitators of people’s involvement, drawing on a structure commonly used in critical incident technique interviewing.

Interviews with PPIT members were conducted by Michael Larkin and Elizabeth Newton. The remaining participant groups were interviewed by Lizz Kimber. Twelve interviews were conducted; two of these were group interviews. A total of 18 people took part, with representation from each of the four stakeholder groups. All interviews were audio-recorded, transcribed in full and then anonymised.

Data analysis

We used template analysis to code and organise the data. A template in this sense is a set of categories or headings for organising the content of the data. In template analysis, preliminary templates can be developed using an existing framework or they can be developed from exploratory coding of the data. We took the latter approach and we had the luxury of independent coders to help us with this. Nicole De Valliere began coding the interviews conducted with PPIT members (six transcripts) and Lizz Kimber began coding the remainder (six transcripts). When each coder had completed three transcripts, they shared their developing template structures with Michael Larkin, who produced a merged template structure. The remaining interviews were coded according to the structure of this preliminary template. The template was further refined and developed during this stage. The final template structure is detailed and complex, and can be used as the basis for identifying cross-cutting themes. In subsequent reporting, we will take this approach. For the purposes of this document, what follows is a more ‘concrete’ focus on barriers and facilitators.

Getting involved gradually

Participants from the PPIT reflected on the gradient of their involvement. To begin with, it was ‘a bit hard to take in, it was a bit complicated’. They described getting to grips with their role, getting to know the study team and coming to an understanding of what the study was about. They described how the study became ‘a constant’ for them. This was positive: a reliable break from other routines and an enjoyable experience. It soon entailed a commitment: ‘[i]t’s a commitment, and it means “yes” all the time. I’ve got my (other regular group activity). I should’ve gone. We told them we couldn’t go today. This has become more important than that.’

Different kinds of contributing

The participants also described ways in which they had contributed, emphasising particularly their role in helping the researchers to understand the problem they were studying: ‘I’ve made them feel informed’. When they did this, they grounded their claims in the value of their experiential insights: ‘I think we’ve all got that experience . . . We could share what we knew about it.’

In addition, PPIT members reflected on other contributions, such as group facilitation (‘I hope that, from time to time, I’ve put forward an idea that might get people moving together to get a consensus’), problem-solving (‘[i]t’s [trying to reconcile the range of views within the group] sort of quite infuriating sometimes, but it’s also very interesting’) and data analysis (‘[w]e went through the interviews . . . we picked out themes that were important’).

One issue that merits some reflection is that some PPIT members seemed to be unclear about the connection between research and practice. There was a tendency to overestimate the speed and directness with which their input might change practices. There is a difficult balance to strike between presenting PPI as a genuine opportunity to make a difference (which it can be) and the frustratingly slow process of translating research insights into practical or political change.

Challenges to contributing

Some of the things that made contributing more difficult were simple practical issues, which research teams can easily resolve if they are prepared to adapt: confusing or complicated concepts, being presented with too much information and a lack of clarity about the PPI role or the research process. Others issues were relational. There was a lot of material about the atmosphere and environment on the first (unsuccessful) meeting, for example. One participant described a ‘quite daunting’ atmosphere, with the group spaced out around a ‘massive table’, listening to strange people talk ‘in quite lengthy terms’, and then said: ‘[a]nd I was sort of sweating and shaking. I’m like “What have I let myself in for?”.’

Some of the relational issues were about getting to know the group. Several participants reflected on the challenges of speaking in large groups. It helped that many PPIT members knew each other, either from the Dragon Café or other activities, but not everybody did, and they also had to get to know the research team and MHF workers. In addition, they were mindful of treading on toes: ‘I think carers have to be very, very careful not to speak for service users’.

Personal factors could also be a barrier at times. A generic issue was the background phenomenon of stigma and personal history. One participant reflected on having to get used to the novelty of being asked for their opinion, for example: ‘[be]cause, by and large, having lived experience of mental health that has you down, and makes you cowed’. Others talked about more specific or intermittent difficulties (such as getting worked up by a particular issue or getting side-tracked) and life events (such as getting sectioned and missing a meeting).

Facilitators of contributing

Factors that helped with involvement were often driven by the motivation to make a difference. Participants described wanting ‘to break down the taboo’, ‘to put across the real’, ‘to help people’ and ‘to improve mental health’. Many people reported very upsetting inpatient experiences (‘[y]ou had to fight to protect yourself’), so the topic was very salient for them. This went hand-in-hand with the appraisal of the research as worthwhile or important: ‘[i]t can change things’.

Practical details such as travel arrangements, appropriate payment and appropriate refreshments were still important (‘[t]hat sort of thing has helped’ but ‘[w]e don’t come here just for the money’). The research team and the MHF participants reflected on the considerable amount of planning and communication that went on to build the right team to support these activities, to get all of the practical details right and to maintain a positive experience for the PPIT.

Some of the above-mentioned relational and personal barriers were overcome because the research team created an environment that felt comfortable (‘welcoming, friendly’, ‘easy going and relaxed’, ‘they’re all calm’, ‘we’ve been treated with respect’, ‘very caring’) but sustained a clear focus and purpose (‘it’s a general discussion’, ‘it’s a structured environment’). In part, this was achieved by stepping outside the routine day-to-day world and coming to the university: ‘[i]f we was in London, and we was talking about this, I don’t think we could be so relaxed. It’s a different environment and we can speak about what we speak about.’ In part, it was achieved by academics offering a different form of professional interest (‘[w]e know that what we said can be trusted. Some places you can’t be trusted – on the wards, cause they say they’re assessing you.’). The participants from the research team described how they had tried to promote equality and offer engaging ways of working. PPIT members were positive about the range of ways to contribute (‘[w]e talked all about it, everybody, and then we changed, swapped and went round over to other people and it was a different experience. It was brilliant.’).

The sense of the group as a group of friends, with a shared purpose, began to cut through worries about group dynamics: ‘[h]aving my friends around me, who also have been going through the same things I’ve been going through, it kind of makes you feel it’s a family’. Positive feedback from the research team about how the PPIT contributions were being taken forward was appreciated (‘[t]his is what you did last time – that’s been important and useful’). Efforts by the team to make sure that everyone had a chance to contribute were also noticed and valued. Honest feedback was appreciated in all quarters and all participant groups described times when it had been provided in a helpful way.

Summary

A more detailed analysis is required, but our initial evaluation highlights the importance of research teams being flexible and for this to be understood and supported by funders and other partners. The PPI component has made an invaluable contribution to the conduct of the EURIPIDES study and it would appear that this has largely been a positive experience for the people involved. These are both excellent outcomes, but they are a consequence of effective adaptation more than a consequence of our initial plans.

Issues with the employment of the survivor researchers (outlined in the previous section) were relatively difficult to resolve, because it proved difficult for partner organisations (and particularly NHS partners) to adapt quickly. By contrast, the problems at the beginning of our PPI programme were resolved quite quickly, because we were supported in responding flexibly to the feedback we received.

Commitment to principles

To ensure that involvement is genuine and meaningful rather than tokenistic, it needs to be taken seriously, both in principle and in practice. It is important to have a clear understanding of and commitment to shared principles of involvement from the outset. We aimed to do this by using resources such as the 4Pi national involvement standards,45 which is a framework for good practice developed by service users and carers themselves. However, we could have spent further time exploring the purpose of involving people with lived experience, particularly with the survivor researchers in relation to issues such as disclosure of service use and the extent to which interview dynamics between interviewee and researcher with lived experience might be present or potentially contributing to the kinds of knowledge produced.

Appropriate resources need to be allocated to PPI work for it to be effective. We realised that, to support the work of the PPIT, we needed to hold more meetings than were originally planned. We also needed to develop ways of supporting the group outside meetings and the PPI work itself (e.g. by arranging visits by team members to the Dragon Café in London and ongoing support from the link worker at the MHF). Although there are often many competing priorities, when making resource allocation decisions it is important to prioritise these principles of involvement when negotiating funding and spending.

Diversity of experience

Although our initial aim was to ensure that PPIT members included a degree of diversity with respect to age, gender, ethnicity and geographical location, issues with recruitment meant that, ultimately, the majority of PPIT were members from the Dragon Café in London. The benefits were substantial in that there was already a degree of mutual trust and understanding between members, which meant that it may have been easier for them to be open during discussions and to disclose experiences. The Dragon Café was also a place to meet the group members where they felt comfortable. However, it may also have meant that a narrower range of perspectives were represented in the PPIT (e.g. this was a very urban/London-centric group) and collective views may already have been consolidated around some issues.

Curiosity and flexibility

It is important to be curious and to listen to the feedback of everyone involved. What do people need or want to make involvement work for them? We discovered that these do not necessarily need to be ‘big things’, for example ensuring that information was presented in an accessible way without the need for paperwork, finding ways to ensure that people felt less anxious about travelling and accommodating preferences in relation to food and refreshments.

As the project progressed, we learned the importance of flexibility and the need to be responsive to other members of the team, even if this meant making changes to original plans. This meant that initial problems were swiftly resolved and it helped to build trust and relationships between the research team and the PPIT.

Communication

For people to be fully involved, communication needs to be open and transparent throughout the research process. This encompasses discussion of shared principles and values at the start of the project, establishing clear lines of communication and support, discussing boundaries and issues around disclosure, negotiating and clarifying roles for different parties, encouraging feedback from team members, etc.

We have acknowledged some of the initial difficulties with regard to communication with the survivor researchers and recognise that improved communication at the research passport and administration stage of the process would have helped to build trust and relationships. However, as the project progressed, all members of the team began to communicate more effectively with each other and integrate the skills and perspectives of everyone involved in the EURIPIDES study.

Sharing of power

The NIHR INVOLVE principles53 state that research needs to be jointly owned. People with a range of prior experiences need to be involved at all levels and stages of the research process. During the course of the study, we acknowledged the importance of sharing power and letting the PPIT lead and manage as much of the process as possible (e.g. with regard to how we worked on tasks and the pace of meetings). Involving and including everyone required a degree of flexibility and creativity and the methods we used were different at each stage in the research process (e.g. role playing sections of interviews for PPIT members to discuss emerging themes, which fed into coding frameworks).

Power-sharing also involves an explicit recognition that, although members of the team occupy specific roles on account of their research experience or their experience of mental/emotional distress, the reality is, of course, that people bring a variety of different overlapping experiences. Challenging hierarchies within research teams is vital for effective PPI and our learning outcomes for future projects would include the need for us to reflect together on our own identities and experiences to create knowledge in more collaborative ways.

Aim

The aim was to identify and synthesise evidence-based patient experience themes relevant to mental health inpatient care.

Objectives

The objectives were to systematically:

• review studies reporting patient experiences of mental health inpatient services

• identify patient experience themes and subthemes that are relevant to delivering high-quality inpatient mental health care.

Protocol and registration

The systematic review was registered as PROSPERO 2016 CRD42016033556.

Scoping review

A scoping review was first conducted to ascertain the extent, range and nature of studies, to map emerging key themes without describing the findings in full or performing a quality check64 and to inform the main review. Six key authors known to be experts in mental health patient experience were contacted for new or unpublished reports and studies.

Patient and public involvement team

Draft themes from the scoping review were presented to the PPIT to ensure face and content validity and to discuss gaps in the research (21 September 2016). The PPIT comprised 10 people with experience of inpatient care or caring for someone who had been an inpatient and who had been recruited by the MHF. Members of the PPIT were invited to two consultation meetings for the purpose of the review. The first meeting was to discuss the research team’s early familiarisation with the literature exploring the themes identified in the scoping review, to obtain their views on these and to add further concepts that PPIT members felt were important but that had not been identified from papers. The second meeting was to discuss the themes identified from the systematic review, to obtain PPIT members’ opinions on these and to identify perceived gaps in the literature. Both discussions were important for assuring content and face validity of the subthemes and themes. Public involvement in this WP is reported in Appendix 19 using GRIPP2.

Identification of studies for the systematic review

The themes that emerged from the scoping review guided the development of search terms and the search strategy for the systematic review. These were applied to MEDLINE, the Cumulative Index to Nursing and Allied Health Literature (CINAHL) and PsycINFO. An example of search terms and results are reported in Appendix 14. Reference lists of included papers were scanned. The search deviated from the protocol in that only three of five databases were searched owing to the large numbers of abstracts retrieved.

Inclusion and exclusion criteria

All study designs were considered if papers included experiences of current or former inpatients of mental health institutions. No restrictions were applied based on country. Articles were included if they reported primary research, were peer reviewed and were published in English between January 2000 and January 2016. Papers were excluded if they were not primary studies, were based on pre-2000 data, included children and adolescents (aged < 18 years) or were not in English. When study participants included both inpatients and outpatients, only data regarding inpatient experiences were extracted. Reviews (see Report Supplementary Material 1) were noted and reference lists were scanned, but these were excluded from the review to avoid bias.

Study selection

Titles and abstracts were screened (by Carole Mockford and Greg Chadburn), of which 20% were independently cross-checked for agreement prior to obtaining full-text articles (Sophie Staniszewska and Carole Mockford). Full texts were obtained when the abstract was unclear. Any disagreements could be resolved by consensus (Carole Mockford, Greg Chadburn and Sophie Staniszewska), but no disagreements occurred.

Data extraction

The data were extracted to Microsoft Excel^® (version 2013; Microsoft Corporation, Redmond, WA, USA), and included citation details, sample recruitment and research methods and findings related to key concepts; any other emerging concepts were added (Carole Mockford).

Quality and risk of bias in individual studies

The quality of included studies was evaluated using the Critical Appraisal Skills Programme (CASP) qualitative checklist;65 the evaluations were undertaken by Carole Mockford. Owing to the heterogeneity of the studies included, many of which were descriptive in their approach, this checklist provided an appropriate basis for comparison between studies. The only slight question change in the CASP checklist was as follows: ‘is the qualitative methodology appropriate for this study?’ was changed to ‘is the methodology appropriate for this study?’. This was to ensure that the quality check did not discriminate against the few relevant quantitative studies that had measured experiences.

Data analysis

The scoping review shaped the content of the thematic framework in the main review, when a detailed narrative synthesis of identified themes was undertaken. 23 The researcher read each study and undertook an initial preliminary synthesis to identify emerging subthemes and compared themes and subthemes within and across studies to develop the main themes. Themes were summarised in a descriptive form, allowing the findings of all studies included, regardless of design, to be aggregated and summarised. The concept of data saturation was used to form a judgement about the point at which no new themes were identified. 66 This approach was considered appropriate for a large review in which the addition of further papers was unlikely to change key findings.

Identification of key themes

Patient experience themes were categorised into four overarching themes or dimensions of experience: the importance of high-quality relationships; averting negative experiences of coercion; a healthy, safe and enabling physical environment and ward milieu; and authentic experiences of patient-centred care. These key themes, accompanied by subthemes, are described in detail below.

Setting and participants

This survey was conducted with PELs in NHS mental health trusts in England with > 50 adult mental health beds. At the time of undertaking WP2, there were 71 providers of NHS inpatient mental health services in England. Bed availability and occupancy statistics (the KH03) were used to determine eligibility. 132

Ethics approvals

Study materials were approved by the West Midlands (South Birmingham) NHS Research Ethics Committee. These were sent out to NHS trusts and included an invitation letter outlining the study (see Report Supplementary Material 2), a consent form and a participant information sheet; trusts were also issued with a full copy of the research protocol. 133

Data collection

The data collection comprised three elements:

1. telephone interviews with PELs

2. completion of a short questionnaire by PELs

3. documents showcasing patient experience work in the trust.

Telephone interview

Audio files were fully transcribed, reanonymised for any information that may be considered identifying, and analysed using MAXQDA12 (VERBI Software, Berlin, Germany). Transcripts were coded using a thematic analysis framework. Information gleaned from coded transcripts was then summarised into an overarching matrix (see Report Supplementary Material 3). In addition, any programme theories that trusts had identified (even if they were partially developed) were extracted from the interview data and recorded in a programme theory log created to collate these.

Questionnaire

Questionnaire data were summarised using a matrix to facilitate WP3 case site selection (see Report Supplementary Material 3). These data were used with the other summarised information to ensure a balanced selection of case sites in terms of size and scale.

Supporting documents

Documents were entered into MAXQDA12 and reviewed to assist with programme theory development by augmenting data in questionnaires and interview transcripts.

Analysis

We initially identified a descriptive framework developed by The Health Foundation134 for classifying patient experience data collection methods reported by participating trusts. This framework is based on the two dimensions of ‘descriptiveness’ and ‘generalisability’ and classifies tools such as surveys or online ratings as less descriptive than in-depth interviews, complaints and compliments. Using this framework, tools were additionally categorised by their generalisability. Surveys or in-depth interviews were seen as more generalisable, whereas patient stories with a specific individual narrative, anonymous complaints or online ratings were viewed as less generalisable. 134

Although this framework offered a simple means of classifying how feedback was sought from inpatients, it did not provide sufficient detail to enable selection of WP3 case study sites. Our methods therefore evolved to include a more nuanced understanding of the timing and sequence of approaches to patient experience data collection and use. This enabled us to ask more realist questions about how, when, for whom and why particular tools were adopted.

This broader approach was drawn from implementation research related to the heuristic ‘stages’ of the policy cycle. We based our final design on the rationale, objectives, appraisal, monitoring, evaluation and feedback (ROAMEF) cycle, as used to evaluate policy by Her Majesty’s Treasury and the Department of Health and Social Care. 135,136 The ROAMEF cycle phases were adapted to help in understanding the ‘journey’ of patient experience data as it moves through a trust, which we conceptualised in four stages: (1) data collection, (2) data analysis, (3) change informed by the results of analysis and (4) feedback within the patient experience process (i.e. patients and/or carers told about how their feedback had changed service delivery).

We used this four-step cycle to frame our programme theories to try and understand at what points patient experience data were collected, analysed and used. We used this cycle in the semistructured interview schedule to determine what happened in each of these phases, to get a more nuanced understanding of how the patient experience processes operated in participating trusts, and to compare these.

Once we had completed PEL interviews, an iterative process took place (through re-reviewing and triangulating the three sources of information) to ascertain which of our programme theories needed refining and what generative mechanisms could be observed underlying the patient experience work taking place in each trust that might inform the realist evaluation in WP3. The main purpose of the survey, however, was to populate a sampling frame to select case sites.

Collated survey data (transcribed interviews, field notes, best-practice documents and questionnaire data) were coded by study researchers using MAXQDA12. Coding was thematic and purposive in order to pull information into descriptive summary categories. For example, information was extracted on the size, composition and organisational location of patient experience teams. This extracted information was summarised and inserted into a matrix (see Report Supplementary Material 3).

The initial organising principle for these data was the four-step process described above and we initially sought to establish how ‘far around’ the cycle trusts were in relation to collecting and using patient experience data. We then expanded this to reflect information provided by PELs about wider patient experience work (and associated governance and patient engagement processes) in participating trusts. These categories were based on our initial and refined programme theories about what might lead to successful collection, analysis and use of patient experience data in relation to quality improvement and, thus, the sampling frame was strongly influenced by our programme theory development (see Appendix 17).

Developing the appraisal system

We created a system for appraising the data to support the selection of case study sites and minimise bias. Our approach was iterative. The initial matrix spreadsheet document and extracted data were reviewed and analysed in the light of the emergent programme theories (see Appendix 17). We developed a weighted appraisal system based on key themes identified as potentially important in this process. For example, one of the programme theories expressed in the original bid was133 (see Appendix 17):

. . . there will be discernible differences between organisations that demonstrate genuine commitment to, and capacity for, using patient experience data to improve services, and that these differences will be most clearly manifest through the existence of processes that support innovation and quality improvement, including commitment to service improvement among senior leaders, decentralised decision-making (through identifiable champions for change), role clarity within the organisation, and support for risk-taking.

Reproduced from Weich et al. 133 © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted. This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

To test this theory (including the level of commitment among senior leaders), we used organisational proximity of the PEL role to trust boards as a proxy. One of the scored items developed, therefore, concerned where the PEL post sat within the trust and how far removed this post was from the trust board in terms of reporting structures.

Based on our initial programme theories, the first iteration of the appraisal system included items about:

• the completeness of the cycle of data collection, analysis, implementation and feedback

• the variety of methods used to collect patient experience data

• the nature and extent of service user and carer or PPI work within the trust

• if the trust had a separate budget for patient experience work

• where the PEL post sat organisationally and how far removed this was from the trust board in terms of reporting structures

• if there were any quality initiatives or any quality data explicitly linked to or triangulated with patient experience data

• if Patient Advice and Liaison Services (PALSs) or complaints were linked to the patient experience work being undertaken in the trust

• whether the PEL was a temporary or a permanent post, and if they were supported by a wider team

• if patient safety data were linked to or triangulated with patient experience data.

The majority of trusts focused on the earlier part of the data collection cycle. Therefore, to differentiate trusts for the purposes of selecting WP3 case study sites, we refined our approach to include evidence to support or refute the following key programme theories133 (see Appendix 17):

• Organisations that innovate also value information, and ‘we anticipate that organisations that use patient experience data most effectively will also have robust data-collection strategies.’. 133

• There is a dearth of evidence about the processes required to analyse, interpret and translate these data into tangible actions, better outcomes for patients, and more efficient and cost-effective care. 133

• Organisations that are genuinely patient-centred will also demonstrate investment in and adoption of codesign approaches to service improvement, and will involve service users and carers meaningfully, as partners. Feedback from patients can be perceived as critical or threatening by professionals, and may be avoided or denigrated. Codesign is a partnership approach that neutralises perceived threats through collaborative working to find mutually agreeable solutions to problems or difficulties experienced by patients when receiving care.

The final matrix was structured to reflect these three key domains (the variety of methods for collecting patient experience data, service user and carer engagement, and the extent of completion of the patient experience cycle from data collection, to analysis, to implementation and feedback; Figure 6).

FIGURE 6.

Weighted appraisal system for WP2.

Data from each trust were scored independently by three researchers. Scores were then discussed to reach consensus if they differed.

Development of typologies of NHS mental health trusts

After rating each trust, we sought to red, amber, green (RAG) rate trusts into three bands of low, medium and high scores (Table 2). We used these three bands to form typologies of trust based on ‘levels of embeddedness’ of patient experience work. This term emerged from our programme theories and is used here to refer to the breadth of patient experience work in relation to the whole feedback cycle from data collection to its use in service improvement. ‘Embeddedness’ includes service user and carer involvement in collecting and using patient experience data, and triangulation of patient experience data with quality and safety data.

Thus, we were interested in what kinds and how much service user and carer involvement and engagement was taking place in each trust, and the quality of this involvement and engagement; how patient experience linked to quality improvement or other processes through organisational structures; and how seriously it was taken by trust boards. These ideas were explicitly linked to programme theory drawn from the original bid and the findings from WP1 (see Appendix 17).

At the end of the rating process, we had three categories of trust to select case sites from: those with low levels, those with medium levels and those with high levels of embeddedness of patient experience work.

Selection of trusts as potential case sites

The sampling process for WP3 involved a purposive realist approach. 137 Emmel137 suggests that the ‘trick in choosing [realist] cases is to assemble the optimal configuration of ingredients to refine a theory’. 137 Realist sampling strategies are theory driven, with theory being applied to investigate a particular context in the social world. The selection of comparative case sites was purposive in relation to our embeddedness typology. Purposive realist selection took place to refine our understanding of the programme theories and the generative mechanisms operating to create different outcomes between and within trusts. The final selection of case study sites was undertaken by the Project Oversight Group, supported by two service user researchers (Emma Ormerod and Stephen Jeffreys). All data were anonymised (and the initial numbers given to trusts were reanonymised during the analysis phase).

Case sites were chosen based on patterns of outcomes that emerged during the appraisal process. We chose sites with ratings that were similar in their overall totals but different in patterning, to allow for testing and refinement of our programme theories. Therefore, we decided that trusts would be recruited from sites rated in the medium- and high-scoring categories. Case sites with low levels of embeddedness of patient experience work were excluded, as we judged that programme theories would not be testable if little patient experience activity was being carried out. It was also felt that visiting trusts where limited patient experience work was taking place would not be fruitful and would risk upsetting staff.

A longlist of potential sites was established using anonymous data. At this stage, additional information was considered to ensure the sites accounted for the urban or rural setting, size of the trust (e.g. number of wards and beds) and location, to ensure geographical diversity in the sites selected (Table 3).

Turnover of patient experience leads

There was a high turnover rate among PELs in participating trusts. Between April 2016 and March 2017, 16 PELs left their posts or were replaced, and a further three leads were covering short-term or were in interim posts. This meant that a total of 19 PELs across the 42 trusts left their post in the first year of the study.

This staff churn reflected the state of the NHS at the time and offers an insight into the precarious nature of many PEL roles. Patient experience work seemed to be viewed in some trusts as a secondary function. This was relevant when considering some of our original programme theories, in particular those that related to trusts’ commitment to meaningful patient experience work linked to quality improvement, for example: ‘[w]e expect that organisations that set out to improve care quality will be able to evidence of methodologies for achieving this, including clear cycles of planning, implementation and reflection as opposed to small, piecemeal initiatives’ (see Appendix 17).

It became clear from our survey that the PEL role was vulnerable in many places and that, in at least one trust, it had been eliminated. This information was significant for refining programme theories and for site selection.

Completeness of the patient experience cycle

Trusts’ completion of the patient experience cycle from data collection to implementation and change was variable. Some trusts had seemingly long-established and well-developed systems for collecting feedback, whereas others were at an embryonic stage of patient experience work. Although this was unsurprising, what was of note was how few trusts reported using the data collected and explicitly linking this to evidence of service change.

The majority of trusts were collecting patient experience data but reported that these data were not used to drive change (51%). Some trusts were still struggling with collecting patient experience feedback at all (22%), with a similar number reporting using feedback data to support service improvement (27%). However, no trust achieved the potential maximum score (17 points) in relation to completing the patient experience feedback cycle (the highest score awarded was 15). In reality, few trusts were able to articulate how patient experience data were analysed and then used to drive service improvements or change. Some trusts are still in their infancy in relation to developing approaches to collecting patient experience feedback in adult inpatient settings, whereas others are more mature and able to use this feedback.

The extent of completion of the feedback cycle was influenced by multiple factors (resourcing, senior management buy-in, competing system pressures, etc.). It was clear that data analysis was the weakest point in the cycle. Although most trusts placed significant emphasis on data collection, little more than a basic description of the data took place. PELs could often not tell us when and how patient experience data were collected and by whom. This was partly because they did not know and because there was significant variation between wards within trusts. We developed a greater understanding of patient experience in inpatient settings as these interviews progressed. These two findings (relating to the completeness of the cycle and the weakness of analysis) led to programme theories being expanded ahead of WP3.

Comparing patient experience data collection in inpatient mental health services

The range of tools used in adult inpatient settings to capture patient experience data was not dissimilar to that identified by The Health Foundation134 in acute and general health-care settings. The most commonly reported tool used for collecting patient experience information was the FFT; however, trusts varied in how this was used. This varied from administering this test at discharge to monthly, or sometimes just ad hoc, administration. This variation occurred both between wards and between trusts.

Other common sources of patient experience feedback included complaints and compliments, patient stories (often presented to executive boards), community meetings on wards and electronic feedback through Patient Opinion (an independent feedback service) and the NHS Choices website (which is now known simply as the NHS website). Some trusts reported piloting different techniques, ranging from active use of peer support workers or volunteer-led listening clinics to recovery colleges with a patient experience function and more creative and innovative approaches, such as the example of a community-touring feedback camper van.

We identified three dimensions that characterised (and differentiated) approaches to data collection. First (as identified by The Health Foundation134), there was a distinction between approaches that were more descriptive and less generalisable (such as complaints or patient stories) and surveys that were more generalisable but not descriptive (such as the FFT).

The second dimension was the distinction between tools that were ‘formally’ implemented (in as much as they were reported on and used in any form of analysis) and those that were implemented ‘informally’. Exploring formal versus informal patient experience data was identified as important for WP3, to understand how both types of data linked to quality improvement (or not). This also had implications for WP5 (the economic evaluation; see Chapter 8).

The third dimension was the difference between trusts that collected patient experience data using just one formal method and those that employed a broader portfolio of approaches. Trusts that used a more layered (multiple) approach to collecting data appeared to be better at using the findings (i.e. some evidence of implementation following analysis). We were not able to identify outcomes from WP2 data alone, nor could we identify which combination of data collection methods was most effective in which particular settings and why. Again, this finding contributed to the refinement of programme theory and the development of the interview questions for WP3 (see Appendix 17).

The national imperative to report on patient experience was a key driver for routine collection of patient experience data. Although there were explicit criticisms made by PELs of the FFT, many acknowledged the usefulness of a national benchmarking measure and the systems change brought about by mandatory reporting. Nevertheless, FFT data were seen as insufficiently informative for the purposes of action-planning at the ward level. In addition (or perhaps consequently), FFT data were rarely fed back to staff. Analysis was often automated, outsourced or superficial, and there was limited analysis of the free-text data.

Concerns were raised in relation to inpatient mental health, and more so in forensic inpatient mental health settings, about the emotive and inappropriate nature of asking ‘would you recommend this service to your friends and family?’, particularly when admission was involuntary. This was cited as a reason for resistance among staff in administering FFT surveys to inpatients on adult wards.

Analysis and use of patient experience feedback to inform change

Patient experience work in NHS mental health trusts in England among adults of working age in inpatient settings was highly variable. Analysis of patient experience data was the weakest reported area of activity across all trusts. Variation in approaches to data collection was associated with differences in the ways in which these data were analysed and used. Some trusts had developed in-house data collection and reporting systems and ‘dashboards’ for reporting information; one trust had developed a bespoke patient experience dashboard. For many trusts, however, the management of patient experience data was not as robust and, in many places, FFT returns or other information was entered into local systems manually. Some trusts had procured external services and outsourced the analysis of patient experience data, which was received back in summary form and cascaded to staff. Even in trusts in which data collection and management processes were more sophisticated, many still reported that data were held in siloes; triangulation between patient experience data, PALS compliments and complaints, patient safety and other data sources was poor and typically relied on senior individuals taking the time to pull this together.

Often, data were collated and held centrally (even if some data entry was done locally). The initial descriptive analysis of collected feedback took place either through outsourcing or at a corporate level. Following this, information was cascaded through management lines back to wards.

When patient experience feedback led to changes, these were more often ‘environmental’ than ‘cultural’. ‘Environmental’ change was used to refer to changes that related to the physical environment or milieu, which resonated with WP1 findings (see Chapter 4). Examples cited often related to diet, seating areas in wards, temperature control and the physical environment of the ward.

‘Cultural’ change involved responding to patient feedback about relationships (i.e. feedback about staff attitudes). There were only two instances cited by PELs of patient experience leading to staff training. It was acknowledged that these kinds of changes were more difficult to make, particularly as they required additional resources. This resulted in a focus on short-term ‘quick wins’ rather than more complex and further-reaching quality improvement initiatives.

The issue of how difficult it was to know how to weight feedback was also raised by PELs. For example, they mentioned how one very powerful patient story may affect and change a service when it reached board level, whereas a single response to a survey rarely had the same effect. There was discussion in interviews about the ‘critical mass’ of feedback needed to drive change.

How feedback directly translated into quality improvement or the relationship of patient experience to ongoing quality improvement initiatives was often unclear. Triangulation of data across systems proved difficult in all settings. Many PELs highlighted the need for further progress:

After putting systems in place and changing culture around collection of patient experience, the next stage should be to identify what happens to data; the role may naturally evolve into quality improvement.

WP2Lead31

Patient experience may ‘promote’ FFT as they report on it for commissioning; they may also review Patient Opinion and NHS Choices, but these do not drive patient experience or any service improvement culture.

WP2Lead08

There was also awareness among PELs that change is difficult to facilitate and that there were many reasons for this:

Change doesn’t happen because of money, time, lack of support for change and the culture within the organisation; there are a lot of things stopping change.

WP2Lead06

Finite resources are a reason for lack of implementation of patient experience feedback; however, we need to think about what levels patient experience feedback can be used, e.g. in planning for change.

WP2Lead33

There was pervasive concern among WP2 interviewees about finite resources and resource pressure (particularly in a culture in which cost improvement decisions were increasingly difficult). There was also acknowledgement that the reason for not effecting cultural change in response to patient experience feedback was not solely because of a lack of resources. This provided useful programme theory refinement for the case study research (see Chapter 6 and Appendix 17).

Refinement of the EURIPIDES study programme theories

The WP2 survey provided an overview of the range of ways in which patient experience data were being collected in inpatient mental health settings in England. The first refinement of the programme theories related to developing a better understanding of the processes in place and methods that trusts were using to obtain patient experience feedback. One of the initial programme theories from the original bid was that organisations that use patient experience data most effectively will be those with the most robust data collection strategies (see Appendix 17). This theory necessarily evolved, as many trusts had similar data collection strategies, yet this did not, in and of itself, result in high levels of embeddedness of patient experience work being detected through the survey.

Although trusts used a broadly similar range of methods (and instruments) to collect data, these were being used differently within and between trusts. However, it was not clear that it was just the way in which the instruments were being used that influenced what happened next in relation to their success in obtaining genuine patient experience feedback or not. We could not tell from this survey which methods were most effective or why different patterns of outcomes (including uptake) were reported despite similar instruments (i.e. why patients engaged differently with the same instruments to give experience feedback). The implementation of similar tools that were generating different responses offered useful framing for the evolving programme theories that informed WP3. Realist evaluation tries to unearth the unseen mechanisms that generate different patterns of outcomes (see Chapter 2).

We also noted strongly contrasting, and contradictory, views about the approaches needed to obtain meaningful feedback from patients on inpatient units. An example occurred when PELs discussed the successful uptake of their survey or other measures for obtaining patient experience feedback, or tried to explain their FFT return rates. When talking about response rates from adult inpatient services, PELs offered logics including ‘getting their best data from inpatient services because they had a “captive audience”’, which was contradicted by another PEL who explained that they get their worst response rates from inpatient services because ‘patients are too unwell’. Both of these statements hold a potential truth. As the settings and clinical conditions of patients are broadly similar, something different must be operating; in realist terms, this suggests that different mechanisms are being ‘fired’ to warrant such different claims.

Strengths and limitations

This was the first survey of its kind in England. The high rate of participation (42 out of the 54 eligible trusts were represented) was likely to reflect the use of telephone interviews to collect data. Other strengths were the realist approach and in-depth data collection, informed by WP1 and the emergent programme theories that informed our data analysis. Broadening the focus to the embeddedness of patient experience work in trusts, rather than simply the ways in which patient experience data were collected, was a particular strength. This approach ensured that we were able to test and refine programme theories in a way that informed the conduct of WP3 interviews.

The development of the matrix for collating data, and an appraisal system to help summarise the latter, assisted greatly in WP3 case site selection. The main limitation was the absence of robust evidence of reliability or validity for item scoring or for the collation of results into three categories of ‘embeddedness’. The fact that the (blindly) chosen sites met the selection criteria (for diversity, against which to test and refine programme theories) provides a measure of validity for these approaches.

The realist evaluation within the overall project

This chapter reports on the comparative realist evaluation of the collection, analysis and use of patient experience data to improve the quality of care in six case study sites in NHS England. This theory-based evaluation builds on the findings of earlier WPs. WP1 used programme theories from the original research bid to interrogate the literature about inpatient experiences of mental health settings (see Chapter 4). The themes identified in WP1 informed the interview schedules for the national telephone survey (WP2) and were used to refine the programme theories (see Appendix 17). These refined programme theories were then used to shape WP3.

The sites selected for the evaluation were identified from the survey of PELs in mental health trusts in England, as described in WP2. The survey provided an overview of the types of data that were being collected and the range of ways in which they were being collected. In WP3, we examined how, why, in what circumstances and for whom the process of patient experience data collection and analysis was being used to improve the quality of care. Our aim was to understand the contradictory statements noted in WP2 interviews, such as ‘[w]e get our best’ versus ‘[w]e get our worst’ patient experience data in inpatient settings through examining the programme theories behind these statements, in an effort to explain the divergence of practices observed (see Appendix 17).

Aim and purpose

Our aim was to undertake an in-depth comparative realist evaluation of the collection, analysis and use of patient experience data to improve the quality of care in adult inpatient mental health settings using six case study sites. Our purpose was to refine programme theories operating at the middle range to ensure that, while respectful of individual trust contexts, generalisable findings could be extracted from across the case study sites (see Appendix 17). From this, we aimed to develop a series of recommendations to take to our consensus conference (WP4) for discussion with key stakeholders.

Research design

We evaluated the collection, analysis and use of patient experience data in adult inpatient settings. To understand how, why and in what context this process was (or was not) successful, a laminated qualitative research design44 using semistructured interviews was developed that involved interviewing patients, carers, clinical staff working on the wards and corporate staff (see Chapter 2).

Selection and recruitment of case study sites

We considered a case to be an NHS mental health trust in England that manages at least 50 adult mental health inpatient beds. The sampling process involved adopting a purposive realist approach. 137 Emmel137 suggests that the ‘trick in choosing [realist] cases is to assemble the optimal configuration of ingredients to refine a theory’. The selection and recruitment of the six case study sites is reported in Chapter 5.

Setting for the realist evaluation

The six case study sites recruited were geographically dispersed across England and included trusts serving predominantly rural, predominantly urban and mixed rural and urban populations (Box 4).

Sampling and recruitment of participants in the case sites

The EURIPIDES study principal investigator at each case site worked alongside the research team and the local research and development (R&D) teams to identify and recruit participants (staff, patients and carers) for interview. Researchers visited and spoke to key staff on all acute adult mental health wards at each site to understand the context of patient experience activities. We excluded wards that were outside the project scope: children and adolescent mental health service wards, forensic wards, rehabilitation or step-down services.

All adults of working age (18–64 years) who were admitted to inpatient services in the case study sites were eligible to take part, providing they had the capacity to consent. Clinical staff, supported by local R&D teams, identified patients who were well enough to participate and who had the capacity to consent, and made the initial approach. The EURIPIDES study research team arranged to meet with participants in a confidential room (often a relatives’ meeting space) on the wards at a time that was convenient to the participant. The study team would then complete any outstanding consent paperwork, ensure that the participant had a chance to ask any questions and ensure that the participant understood what they were participating in; they would then conduct an interview. Immediately prior to each patient interview, the research team checked with the ward staff that the patient was still well enough to be interviewed, as the initial approach may have been made a few days earlier. If the patient had become unwell, the interview was rescheduled.

Carers were recruited based on their having a friend or family member (for whom they had some caring responsibility) on an adult inpatient ward for adults of working age in the case study sites. Carers were excluded if their experience was historical (> 6 months since supporting a friend or family member who had been an inpatient in mental health services). Clinical staff and the local R&D team made the first approach to carers coming on to the wards or those identified through carer forums, where these existed. There was not necessarily any link between carer and patient participants.

In conjunction with the local principal investigator, key corporate staff were identified who had a role in relation to patient experience data (collection, analysis, management or use of these data from the corporate organisational perspective). Clinical staff were recruited from a range of professions and specialties and a range of bandings: nurses, occupational therapists, psychologists, psychiatrists and health-care assistants. The principal investigator sent an e-mail to relevant staff inviting them to participate and spoke to ward teams asking for volunteers.

Through purposive sampling, we sought to recruit 30 participants per case site, based on research practice relating to minimum participant numbers and saturation of themes. 139 Attention was paid to the balance between recruiting an optimum number of participants for the study and the disruption caused to service delivery. The research team reviewed and discussed their interview experience and reflective field notes daily. Once they perceived that thematic saturation had been reached at a case site, recruitment ceased.

Data on how many potential participants were approached and on how many of these agreed to participate were not recorded. Recruitment was often undertaken opportunistically, for example because a patient’s mental state improved or a carer arrived on the ward. Recording recruitment approaches was felt to be too great an additional burden on overstretched ward staff.

Study materials

Realist interviewing is described as a process whereby ‘I’ll show you my theory if you show me yours’. 36 In a realist interview, theories are placed before the interviewee for them to comment on, with a view to providing refinement. The realist approach assumes that participants’ accounts have a direct relationship with the real experiences of (in this case) the inpatient setting. The subject matter of the interview is the research team’s theory and the subject (participant) is there to confirm, falsify and refine that theory. 140 Separate semistructured interview guides for each participant group were created.

Patient and carer interview schedules were developed in conjunction with the PPIT. The PPIT reviewed WP1 results and, through discussion, added factors that might influence whether or not patients engage in feedback processes. Three key themes were then identified: ways of giving feedback, relationships between staff and patients, and communication and trust. Flash cards were created with the theme label on one side and illustrative quotations from the PPIT discussion on the other (Box 5; see Appendices 9, 11 and 13). Patient and carer quotations differed from each other, reflecting their different perspectives.

The clinical and corporate staff interview schedules drew on the findings of WP2. Flash cards for clinical and corporate staff had six themes: changes in service in response to patient feedback, ways of communicating patient experience, resources for patient experience feedback, understanding patient experience data, collecting and using patient experience feedback, and service user and carer involvement (see Chapter 5 and see Appendices 9, 11 and 13). Illustrative quotations were taken from WP2 interviews.

Participant information sheets, consent forms for each participant group and an easy-to-digest recruitment flyer for service users and carers were developed with advice from the MHF (see Report Supplementary Material 2).

Conduct of the interviews

Realist interviewing is often described as a process that ‘involves a highly specific and carefully planned route march which goes between the qualitative and quantitative traditions’. 36 This is because of the way in which the semistructured interviews are constructed and conducted, namely because they contain material about the programme theories, which is then presented to participants and discussed in a very purposeful way. In realist interviewing, the participants are likely to be sensitised to mechanisms and understand contextual constraints on outcomes; the interviewers are likely to bring theoretical knowledge to the understanding of patient experience feedback processes. 140

Realist interviews proceed iteratively. They employ a strategy of curiosity, whereby you see if the kernel of knowledge you uncovered in one interview holds true in a different context or if it is, in fact, being moderated by mechanisms that were perhaps not present in the other setting, resulting in different patterns of outcomes.

At the start of each interview, consent was obtained or, if it had been obtained by the R&D team, consent was checked with the participant. Interviews were audio-recorded. Reflective field notes were written after each interview about the interview process and content, how it was experienced and the ward environment.

Patients and carers were initially asked to describe their experience of giving feedback about their experience in their inpatient settings. This set the context and served to give the participant a chance to become more comfortable in talking about themselves. Once a participant had settled into the interview, the flash cards (see Appendices 9, 11 and 13) were introduced for discussion. Patients and carers were asked to select whichever card they felt was most important to discuss first and asked to explain their reasons for the selection. The theme labels were usually presented first to see if they agreed with them before the cards were turned over, uncovering the quotations, to aid further probing of their ideas. If participants had difficulty with literacy or reflection, the ideas on the cards were raised without necessarily presenting the cards. The researchers undertaking patient interviews also adapted their interview style based on how unwell the patient was at the time of the interview.

Interviews with staff followed a similar format using the staff flash cards (see Appendices 9, 11 and 13). A few staff interviews took place in pairs if this was the only way to engage the staff members (e.g. over a tea break).

Interviews with patients and staff all took place in a confidential room in clinical settings. Carers were mostly interviewed by telephone, with some being interviewed in person in a confidential room on the ward.

If participants became distressed during their interview, they were offered the opportunity to take a break or to terminate the interview. All participants were informed that the study would not affect their clinical care/work and they were given 2 weeks to withdraw their consent to the use of their data following the interview.

After each day of fieldwork, the three researchers collecting data discussed and reflected on the challenges they had faced. The senior researcher in the team provided in-the-field training, particularly on how to manage interviews in the busy, understaffed and often disrupted setting of the ward and how to manage interviews with the more unwell patients.

Data management

Interviews were fully transcribed, anonymised and checked against the audio-recording. Each interview and the accompanying field notes were given a unique identifier. All data were imported into data analysis software (MAXQDA12) for coding. Coded data were exported to Microsoft Excel for identifying CMO configurations. All data were stored securely. Audio-recordings were destroyed at the end of the project.

It was expected that, during these interviews, disclosures might be made that might require action to be taken outside the project. A detailed disclosure log was maintained and the chief investigator reviewed each disclosure as it was logged and ensured that appropriate action was taken.

Data analysis

Analysis took place in two phases. Initially, the research team explored the raw data with the PPIT and together they developed key themes for the thematic coding framework. Thematic coding was then undertaken to provide a platform for the second analysis phase. The latter phase involved refining the analysed data into CMO configurations. Retroduction was used in both analysis phases, as follows.

Analysis phase 1: development of the thematic coding framework for analysis

A draft coding framework was developed with the PPIT. At the outset, PPIT members said that they did not want to receive or review written material or presentations. They wanted material presented in conversations for discussion.

Research team members identified portions of transcripts that they thought contained CMO configurations and acted them out (anonymously) at PPIT meetings. They asked PPIT members to identify key themes and points of significance. PPIT members worked in small groups with a facilitator who made mind maps or lists of these ideas, dependent on group preference. From this, key superordinate themes were developed and imported into MAXQDA12.

Members of the full research team each read transcripts from different types of participants and discussed the superordinate themes identified by PPIT members. The superordinate themes then formed the basis of the coding framework for the staff data. The coding framework was further developed during further coding and research team discussion and emergent themes were added during analysis (see Report Supplementary Material 4). The thematic coding of all the data was done by the three researchers on the research team and included independent coding by two researchers of 100% of the data, followed by comparison and discussion to ensure consistency of coding.

During the process of coding, it became clear that we could not view time as a flat concept. We needed to break down the process of collection and the use of patient experience data to understand in what ways, when and where decisions were being taken and actors’ reasoning was shifting in relation to the resources available to them. 38

To capture this notion of a process, we superimposed the time dimension onto the developing themes by laminating the coding framework (in keeping with the patient experience data cycle) according to the following process sequence: collecting/giving feedback, receiving/listening to feedback, acting (analysis and response to feedback), and quality improvement and change, including feeding back to patients/carers [collecting and giving, receiving and listening, analysing, and quality improvement and change (CRAICh); Figure 7].

FIGURE 7.

The CRAICh coding architecture. QI, quality improvement.

A further lamination within the coding structure was the different participant perspectives (patients, carers and staff). Therefore, the coding framework consisted of the CRAICh framework providing the time dimension, themes identifying content relevant to the development of CMOs and the participant perspective (staff, patients and carers).

A note was made during the coding process of any potential CMO configurations. In addition, there were codes relating to the data needed for the economic analysis (see Chapter 8). During data coding, further themes not directly related to CRAICh but of relevance to the collection and use of patient feedback emerged. These were added to the coding framework.

Study participants and data

Care was taken with the theoretical sampling of all three recruitment groups. At least one participant was recruited from each ward providing care for adults with mental illness in each trust (see Appendix 15). Across the six sites, 182 interviews were completed, 62 with patients, 19 with carers and 101 with staff (Table 5). The average interview durations were as follows: staff, 43 minutes; carers, 32 minutes; and patients, 32 minutes. There were 40,000 words of field notes generated.

The staff who were interviewed were drawn from a range of professional backgrounds including psychiatry, nursing, occupational health, health-care assistance and health-care management. Staff were sampled at all levels, including those who worked in business support or on information technology (IT) operational functions to ensure that complexity in relation to the case was uncovered. We had representation from all relevant staff groups in the interview data for each of the six case sites and no professional groups were overlooked.

One patient who had agreed to be interviewed became unwell and did not recover sufficiently for interview before fieldwork was completed at that field site. Although some participants became distressed during interviews, no participant chose to terminate the interview. Sixteen disclosures including one whistleblowing incident were made during the course of fieldwork.

The recruitment of carers proved difficult. There were several reasons for this: carers were generally poorly engaged with wards, and many patients were often isolated and did not have carers or people they were close to. Such patients reported either becoming isolated as a result of their illness or isolation being a contributing factor to their illness. Some family members did not want to identify themselves as being in a role related to the care of the patient and some were resentful of being seen in this role. A number of carers also reported struggling financially themselves, so finding time (and fares) to visit relatives or friends was increasingly difficult.

Findings from analysis phase 1: key themes

The PPIT identified five superordinate themes that were retained throughout coding and analysis discussions. The five themes identified were as follows: (1) wellness, (2) the importance of feedback, (3) relationships and communication, (4) resources and (5) power. Each of these operates in different ways within different laminations of the framework, namely at different stages of CRAICh (the patient experience feedback cycle) and in relation to different types of participant.

The thematic analysis relevant to CRAICh was based on 10,812 of a total of 12,026 coded excerpts. The other coded extracts related to the economic analysis or to emergent themes not directly related to CRAICh: professional caring capacity, the experience of delusions and the experience of the Mental Health Act. 33 The analysis reported here will focus on the themes related to CRAICh and to the emergent themes.

Data from some participant perspectives were more dominant than data from other participant perspectives within different parts of CRAICh. For example, the majority of coded excerpts from patients and carers were in ‘collecting and giving’, with relatively few patient- and carer-coded excerpts in the remaining CRAICh lamina (Table 6). For staff, coded excerpts were more evenly distributed, although the majority were located in the first two CRAICh lamina. This, in part, reflects the attention that all trusts pay to collecting feedback, but also that more staff time is invested in obtaining feedback than in making sense of it, implementing changes or feeding back to staff, patients and carers.

The full coding framework can be found in Report Supplementary Material 4.

We report each of the superordinate themes using data from across the six field sites along with supporting quotations. We indicate links between themes and links between themes and the second phase of analysis.

In bold we summarise what our data indicate to be the key issues for health-care organisations regarding how to collect, analyse and use patient experience feedback to improve the quality of inpatient services. These summaries formed the basis for what we termed ‘rules’ developed for presentation to the consensus conference (see Chapter 7).

Findings from analysis phase 2: context–mechanism–outcome configurations

Although the thematic analysis was rich in detail, on its own, it did not provide sufficient evidence to help us understand how, for whom, in what circumstances and why underlying generative mechanisms were being activated. This was the purpose of the realist analysis in which CMO configurations were developed.

From the 12,026 coded excerpts related to CRAICh, 154 CMO configurations (see Appendix 16 and Report Supplementary Material 5–9) were developed. Not all coded excerpts from the thematic analysis were used in building CMO configurations, as not all excerpts helped to uncover generative mechanisms or to inform the wider context or outcomes (Table 7).

As expected, the relevance of each theme varied across CRAICh. For example, the themes of wellness and of relationships and communication were important in relation to the collection and receiving of patient experience feedback but not at other stages of CRAICh. The importance of feedback (which includes ‘seeing change’) is predominant at the implementation, quality improvement and change stages in CRAICh (see Table 7).

The majority of CMO configurations based on patient and carer data relate to collecting patient experience feedback, whereas the CMO configurations based on staff data mostly relate to receiving that feedback, although there are CMO configurations across the CRAICh cycle reflecting different staff roles within the process (Table 8).

All 154 CMO configurations are available in Appendix 16 (see also Report Supplementary Material 5). The results presented in the following sections draw on all 154 CMO configurations that we developed within CRAICh. We present here only examples of CMO configurations, which were chosen as they had the greatest number of underpinning data.

What it is important to get right in all patient settings

During our analysis, we identified issues talked about by subgroups of patients. We know from our analysis that it is difficult to get honest feedback from patients on issues other than the ward environment. As these deeper issues were often sensitive for patients, we think they are significant, even though they were raised by only a few patients.

Although there were specific examples of poor care in some sites, we studied each of the following themes, as each was important to get right in all inpatient settings. The four themes were:

1. tackling racism in inpatient settings

2. supporting patients with auditory and visual hallucinations or delusions to communicate about their experiences of care

3. supporting patients to understand their medication

4. supporting patients to understand their care pathways and experience of the Mental Health Act. 33

These four themes were central to a patient’s experiences of services and interacted with all of the other mechanisms we have discussed related to patients, carers and provision of care.

Strengths and limitations of work package 3

We successfully recruited staff and patients across all sites. For staff, we recruited a range of relevant staff. For patients, we are unable to determine if certain types of patients who were well enough to participate were not invited to participate by recruiting staff. However, we interviewed patients and carers who had both positive and negative things to say about their experiences of care and of giving feedback and some who were willing to talk about sensitive issues such as racism.

Our realist evaluation included lamination to explicitly differentiate between actors in CMO configurations, to differentiate between different points in time and to differentiate between stages in the cycle of patient feedback and change. This has allowed us to manage, in one analysis, a large data set about a very complex process.

We have presented our CMO tables in Appendix 16 (see also Report Supplementary Material 5), along with lists of the data sources for each CMO configuration. Our results are as auditable as a more traditional thematic analysis.

Our evaluation focused on UK NHS adult mental health wards. The purpose of the research was to inform how patient experience feedback is collected, analysed and used in this setting. Our findings are likely to be transferable to settings that are somewhat similar, for example adult mental health wards in other UK NHS provider organisations. However, from our data, we are not able to suggest what determines the transferability of our data to settings beyond this.

Development of the rules for presentation to the stakeholder conference

The rules were developed from the overarching programme theories and the thematic analysis and CMO configurations reported in WP3 that relate to the collection, analysis and use of patient experience data in adult inpatient settings. They were developed from our in-depth knowledge of the data and through research team discussion and discussion with the PPIT. Research team and PPIT experiences were not used as data; instead, the research team and PPIT worked to co-produce the rules from the primary data (see Appendix 17).

Stakeholder conference

The original rules

The rules presented to the stakeholder conference are presented in the final column of the EURIPIDES programme theory development table (see Appendix 17). Prior to the conference, the PPIT raised a concern about rule C2; the wording of this rule was as follows: ‘Make sure you collect positive patient experience feedback.’ They felt that it was important to highlight that all feedback (not just positive feedback) needed to be listened to. It was not the reasoning of the rule that they called into question, rather its phrasing. The rule was changed to the following: ‘Make sure you get both positive and negative patient experience feedback.’

The stakeholder conference

The conference was attended by 44 delegates from across England, including individuals with lived experience of acute inpatient care from our service user and carer reference group and survivor researchers, policy-makers, and both clinical (staff members from inpatient wards) and corporate (corporate patient experience roles) staff (see Appendix 18).

In the EURIPIDES programme theory development table (see Appendix 17), we present the details of the discussion group data and how these were used to refine and add to the original rules.

The view was expressed during several group discussions that more emphasis was needed on the importance of seeing change that resulted from giving feedback. In the original rules, this formed part of the evidence for several rules. We re-interrogated the WP3 data and developed rule C5 (see Rule C5: seeing changes based on both positive and negative feedback is important for patients, carers and staff). The discussion groups highlighted the absence of carers from the rules. Again we re-interrogated the WP3 data and developed a further rule, C6 (see Rule C6: trusts must engage carers proactively in the inpatient care of the person they care for and staff must communicate to obtain feedback from carers). The final rules are given in the final column of the EURIPIDES programme theory development table (see Appendix 17) and are presented in the following section.

The discussion groups highlighted the need for patient experience processes to be aligned with strategic priorities set by NHS England, to be formally recognised in the inspection criteria of the CQC and to be aligned with explicit priorities for commissioning bodies. The need for a more consistent approach to implementing thoughtful change in response to high-quality feedback and communicating those changes was emphasised, rather than a reactive underfunded superficial approach to quality improvement. There was recognition that change costs money. Discussion groups suggested piloting what was termed ‘engaged ward’ programmes, in which the CRAICh rules are implemented using existing resources from which further quality improvement can be built. Any such initiative needs to take into consideration staff retention, staff well-being and compassion fatigue.

The final rules: the programme theory – how to collect, analyse and use patient experience data to improve the quality of care in adult inpatient mental health settings

Strengths and limitations of work package 4

The process by which the EURIPIDES programme theories were developed and refined throughout WPs 1–4 is available in Appendix 17. We have made our process as transparent as possible. The programme theories – expressed as rules – are limited to CRAICh, following the original purpose of the study. Other issues identified in WP3 that were of relevance to quality improvement, although important, were not included in our final rules.

All stakeholders were well represented at the stakeholder conference. Different types of stakeholders engaged with each other in the facilitated discussion groups. Stakeholders were respectful of the rules presented to them, given the number of data and the detailed analysis that underpinned them. This allowed the stakeholders to focus on the use of these rules.

Step 1: identifying and specifying patient experience processes

To be able to cost patient experience activities, it was first necessary to identify what patient experience activities take place. Given that sites were selected for their differing approaches to collecting and using patient experience data, the activities representing patient experience data collection and analysis were expected to differ between sites.

The first stage of the microcosting exercise was, therefore, to map the patient experience activities at each site. The exact processes and activities involved in the patient experience data pathway for each site were identified via in-depth interviews with trust staff. Thus, the analysis of cost data relied on granular resource use data extracted from in-depth interviews at each site in relation to the key steps on the patient experience data pathway: data collection, data management, data analysis and presentation/dissemination. For each site, patient experience data processes were mapped to identify how data collection, analysis and use were broadly conducted at each site.

Step 2: identifying and quantifying all resource inputs

For each enumerated activity, resource inputs were identified and quantified using data from the in-depth interviews. Resources can be broadly separated into labour costs and capital costs. Labour refers to the time burden associated with any patient experience activities; for example, 1 hour spent collecting patient experience data. Capital refers to any equipment that is utilised; for example, this could be electronic gadgets for collecting data on the wards or the cost of the software used for analysing the data.

This bottom-up costing approach aimed to identify every item of resource use associated with the patient experience pathway. Whenever patient experience activities were discussed, the interviewer was tasked with eliciting ‘who’ was involved, ‘how long’ they spent conducting such activities, ‘how often’ they occurred and ‘what’ equipment was used. Owing to the number of staff operating at each site, it was not feasible to interview every member of staff; consequently, exemplars were used. Exemplars at each site were to be used to estimate and extrapolate across each ward. Labour resource inputs were quantified in terms of time (hours), whereas capital resources were recorded in their natural units.

Step 3: defining ‘low-intensity’ and ‘high-intensity’ patient experience packages

A goal of the previous WPs was to identify what components of the patient experience data cycle work well and where the shortcomings in patient experience processes lie.

Two simplified, hypothetical packages (or vignettes) of patient experience data collection and use were created, one representing ‘low-intensity’ and the other representing ‘high-intensity’ processes. Low intensity can be thought of as the minimum requirements for collecting and using inpatient mental health-care patient experience feedback within existing regulatory and policy structures. High-intensity processes, on the other hand, comprised activities that were found at sites with more extensive and embedded patient experience data collection and use (and which were rated as such in WP2).

Patient experience activities were enumerated and included in (i.e. allocated to) either the low-intensity or the high-intensity package (or, in some cases, both). These two packages were then used to examine the burden related to low-intensity and high-intensity patient experience data processes. This allowed us to assess the expected cost of adopting best practice in the collection and use of patient experience data in inpatient settings (as identified in WP3 and WP4) across all mental health trusts in England.

Step 4: identifying unit costs for all inputs

Unit costs refer to the cost of one unit of input. For labour, this could be the salary costs (including pensions, overheads and other on-costs) per hour, whereas for software it may be the ongoing cost of a licence. For inputs identified in step 2, a unit cost was attached. For labour, the costs of the Personal Social Services Research Unit145 were used as the primary source for costing. For capital costs, methods were dependent on what capital was being used. When appropriate, for example in the case of software for analysing patient experience data, the key PEL was asked how much was spent on the specified software licences.

Step 5: combining resource input and unit cost data

To calculate the cost for each individual activity in the two packages, we combined resource input data (identified in step 3) with the unit cost data (identified in step 4), that is we multiplied resource use data by unit cost data. This provided us with activity-specific costing for the packages. Thus, we calculated a cost associated with each activity identified in step 1. We then allocated these activities to the ‘low-intensity’ and/or ‘high-intensity’ packages to examine the resource use associated with different levels of implementation of patient experience processes.

Step 6: creating conceptual models of cost-effectiveness

One of our aims was to assess the benefits resulting from the implementation of best practice in patient experience processes across all mental health trusts in England. Acute mental health inpatient stays are expensive, typically costing £404 per bed-day. 145 Consequently, if patient experience data and analysis can lead to change that improves patient experience and earlier discharge, or a lower risk of relapse and re-admission, there is potential for cost savings. Likewise, if patient experience is improved, then there are benefits in terms of improved health-related quality of life and broader well-being.

However, as reported in Chapter 6, a theme across all sites was that data analysis was often limited, thus restricting the potential for benefits arising from data collection. Consequently, to examine the cost-effectiveness implications of collecting and using patient experience data and adopting best practice, three logic models were created. Two logic models representing low- and high-intensity patient experience data processes (as identified in WP3) were developed. A third model was created to illustrate how implementing patient experience feedback could improve cost-effectiveness in the delivery of inpatient care, using the example of alleviating boredom on the ward.

Contextualising the health economic results: opportunity cost

Within current frameworks, the formal costs associated with patient experience data are relatively low. For example, in the low-intensity package, the estimated formal costs amounted to just £466.97 per ward per month. In the high-intensity setting, these increase to £1227.92 per ward per month. How these costs are perceived is subjective and related to the size of budgets that departments are operating on and potential economies of scale. With a bed-day costing the NHS approximately £404 per day, the formal patient experience costs in the high-intensity scenario equate to three bed-days per ward per month. Whether or not that cost can be justified can be determined only through a formal analysis of the benefits that such a use of money would bring.

Contextualising the health economic results: formal versus informal patient experience data

Formal costs were far outweighed by the costs associated with collecting informal patient experience data. Earlier chapters (Chapters 5 and 6) highlight the importance of informal patient experience to improve care. Given this, there is tension between the costs of informal patient experience data processes and their benefits.

The results from the two vignettes demonstrate that, when the opportunity cost of staff time is considered, informal patient experience costs can be substantial. If, as in the low-intensity scenario, these data are not used, the implication is that these data are not cost-effective and should be discontinued. However, there is a degree of nuance that needs to be considered in relation to defining informal patient experience data. Some of the costs presented in the vignettes relating to informal data experience processes can be very clearly defined as a patient experience mechanism, for example community meetings. These discrete and well-defined activities can be clearly tied to patient experience processes and quantified. The biggest cost driver in both vignettes was related to informal one-to-one patient experience time. However, activities that we classified as informal patient experience activity, for example 15 minutes of one-to-one contact time, could just as easily be classified as normal therapeutic care. This should be borne in mind when comparing the costs and benefits of informal patient experience data with those of formal processes.

Cost-effectiveness

As reported in Chapter 6, there was limited evidence of analysis of patient experience data across study sites. Trusts were not recording or measuring the benefits of patient experience data analysis or the implementation of change. As a result, there was a dearth of evidence for use in a cost-effectiveness framework. Instead, conceptual models were created to show the mechanisms by which patient experience data could possibly be cost-effective. These models highlight the data and tools that would be required to evaluate cost-effectiveness in future trials of patient experience implementation. This includes the prospective measurement of resource use and patient-reported outcome measures alongside the implementation of patient experience processes. Likewise, large national databases could be harnessed to examine whether or not the implementation of new patient experience policies improve patient outcomes over time.

Quantifying benefits

We did not aim to undertake a formal cost-effectiveness analysis. Instead, data were used to conceptualise how patient experience data might theoretically be cost-effective.

Economic evaluation has two components: costs and benefits. This study was not designed to collect such data in a way that would be amenable to economic evaluation. On the benefits side, there were few quantifiable benefits. More explicit benefits, for example concerning how a patient experience policy change might lead to increased discharge rates, would have provided a quantifiable benefit. The majority of the benefits, however, were diffuse and environmental (e.g. improved food or locker facilities). It is challenging to model such intangible benefits and, thus, the benefits side of the cost-effectiveness equation was limited. Consequently, we chose to develop theoretical logic models, informed by interview data, to explore how best practice in the collection and use of patient experience data might lead to cost-effective service improvements. These models highlighted the data that would be required to evaluate cost-effectiveness in future.

It is important to note that, although few benefits across sites were reported, there was no evidence to indicate what would happen in the absence of (even low-intensity) formal patient experience processes. That is, although there were few perceived benefits, we do not know what would have happened had there been no patient experience work at all.

Lessons for future health economic studies on patient experience processes

The present study was designed for the primary purpose of identifying and understanding best practice in the collection and use of patient experience data, using a realist approach. Given this, the data collected were not particularly well suited to traditional health economic analyses. We therefore designed the economic WP to reflect this and to develop more conceptual models of potential cost-effectiveness, including modelling key parameters and potential pathways to cost-effectiveness.

Formal evaluation of cost-effectiveness in any future study (e.g. of interventions to disseminate the present study findings) will require a different design. Patient experience data activities will need to be operationalised and defined, and distinguished from (broader) activities designed to enhance experiences of care. Approaches to capturing the former activities (in both nature and duration) will need to be developed and tested, for instance through resource use diaries or time use surveys. 148 This would involve, for each site, giving an exemplar for each role a resource use diary to complete daily. This would seek to get participants to allocate the time they spent each day on various tasks including patient experience activities. This, however, could be very burdensome on the exemplar and alternative strategies could be employed, for example having a researcher shadow exemplars and complete these tasks on behalf of the patient experience staff.

To fully appraise the impact of new patient experience procedures, a prospective trial design could be implemented to rigorously assess not only the costs of patient experience data processes, but also the benefits that this type of systems change could bring.

Systematic review of inpatient experiences (work package 1)

The systematic review of the most salient aspects of the experiences in inpatient mental health settings (WP1) identified four themes: (1) the importance of high-quality relationships, (2) averting negative experiences of coercion, (3) healthy, safe and enabling physical environment and ward milieu and (4) authentic experiences of patient-centred care.

Survey of patient experience leads (work package 2)

In the survey of PELs (WP2), we found that patient experience work was universally regarded as a positive influence but was rarely embedded in the core business of mental health trusts. More worryingly, PEL posts were subject to high rates of turnover and many were insecurely funded and hence acutely vulnerable to cost improvement pressures. We found that most trusts collect patient experience data (albeit in varying ways and at different intervals across wards), but few analyse or use it; therefore, the feedback cycle is rarely completed. Consequently, we found few examples of service change that had occurred as a result of patient feedback and, when this did happen, it was more likely to take the form of environmental rather than cultural change. This was reflected in the overwhelming predominance of our data in WP3 that concerned the conditions necessary for patients to be able to give, and staff to elicit and receive, meaningful feedback. When we observed best practice, the staff saw the collection of patient experience feedback as integral to caring for the patient, prioritised spending time with patients (which is difficult when reporting demands are high so staff have to spend a lot of time in the office) and were able to discern from patients’ accounts of their experience what was due to illness and what related to the experience of the ward. When staff were over stretched, demoralised because they felt that they did not have the agency or resource to provide good-quality care, and frustrated at the lack of change that resulted from patient experience feedback, they stopped attempting to collect the feedback. This negative cycle was exacerbated by aggression on wards, which required investigation, and by patient complaints (both of which were more likely when quality of care was poor). Staff became fearful of engaging with patients about their experience and, in extreme cases, blamed patients for problems with care.

We found much less evidence of a nuance in approaches to analysing these data or indeed to using these data to bring about service improvements. This was in contrast to the importance that many senior staff placed on patient-centred care and improving the patient experience of care in their organisation.

Case studies (work package 3)

Our analysis of WP3 (case studies) data was undertaken in two stages. First, a thematic analysis was undertaken that spanned all four stages of the patient experience data cycle, for which we coined the acronym CRAICh (collecting and giving, receiving and listening, analysing, and quality improvement and change). In the second part of our analyses, we developed 154 CMO configurations to understand how, for whom, in what circumstances and why underlying generative mechanisms were activated. The generative mechanisms were reported using the CRAICh framework. The five themes identified in the first phase of analysis were wellness, the importance of feedback, relationships and communication, resources, and power.

The wellness theme revealed that patients who are acutely psychotic, or particularly unwell, can describe their experiences, and many such patients do wish to communicate these. However, many such patients feel that staff may discourage this, a view that has some basis in fact, particularly when staff feel that it may be countertherapeutic to encourage patients to discuss delusional beliefs or distressing, internally driven experiences. Consequently, patients may come to view certain topics as ‘off limits’ (i.e. implicitly and explicitly discouraged), which may, in turn, inhibit them from sharing their experiences of care as they approach discharge. We also found that power imbalances (highlighted in another of the five themes) shape patients’ willingness to share certain aspects of their experiences of care for fear that this might interfere with their care, leave or discharge plan. This means that there may be only a small window of opportunity to ask about certain types of care experience, at the point of discharge.

We found that, no matter how unwell a person was, they were able to say if they were having a good or bad experience of care. When they are feeling slightly better, they can offer more nuanced reflective feedback on the episode of care or the setting. Patients should be encouraged to speak freely and authentically (including about their delusions), by suitably skilled and trained staff, to support them in making sense of their experiences. If this is discouraged, it inhibits the development of trust and reduces patients’ ability and willingness to give meaningful feedback about their care and treatment.

Collecting meaningful feedback from patients takes time and emotional resources (another of the five themes), both of which were in short supply in a system that was evidently under pressure. There were identifiable impacts (i.e. the availability of staff to talk to patients about their inpatient experiences) of resource pressure in inpatient settings; however, the realist evaluation uncovered that, although resource pressure and the wider climate of austerity may exacerbate the problems of collecting authentic patient experience feedback, this alone did not determine the culture of the inpatient setting. Instead, it was a collection of activated mechanisms that generated an environment in which patients felt safe and supported, which, in turn, enabled them to trust staff in order to engage in giving honest feedback. Conversely, what was most detrimental to quality of care was when poor culture developed that hampered authentic patient experience work.

We found that patients will give feedback only to staff whom they trust, a finding that resonates with the results of WP1, which highlights the importance of trusting, available, consistent therapeutic relationships (the fourth of the themes) with ward staff as the most important determinant of patient experience. What realist evaluation has offered, to both support and enhance this finding, is the mechanisms by which the loss of trust takes place and how, why and for whom that happens. This is particularly relevant when thinking about institutional racism (i.e. staff not tackling racist abuse between patients) as damaging particular populations’ abilities to trust within inpatient settings. The culture of the ward or organisation is a powerful contextual moderator of the outcomes of obtaining meaningful inpatient experience data.

Linked to this point, the realist evaluation demonstrated that, although the conditions in which trust may be activated vary, establishing trust was a critical ingredient for patient experience work. Our results support the view that meaningful patient experience data are available only in places where the quality of care is good and where patients feel able to trust staff. Paradoxically, therefore, it may be impossible to obtain meaningful (and vitally important) feedback when the care quality is poor. This is even truer for carers, who often feel excluded and that their views are unwelcome. Moreover, they fear for the safety of their friend or family member and worry about giving honest feedback because they do not want to negatively influence their care.

One consequence of this was that ward staff (who rarely see the results of feedback, let alone actions that might arise from its use) often perceived patient experience data as serving corporate purposes (e.g. the obligation to report these data outside the organisation) rather than as leading to service improvement. Because service users do not see changes occurring as a result of their feedback (the final theme), they may be loath to give it. By contrast, ward staff, who were closest to the patient experience data and, therefore, best placed to effect meaningful change as a result, often did not get sight of the formal patient experience data and were therefore denied the opportunity to act on it. At the same time, they were not empowered to act on the informal patient experience feedback that was given to them or their colleagues directly.

Another key finding was that positive feedback is important for patients to give (from their perspective), but is often dismissed. Negative feedback (in the form of complaints) was often considered to carry far more weight and is subject to formal processes. Positive feedback, which was often dismissed (and, therefore, lost) is a rich source of data for planning and service improvement. However, some staff perceive feedback as threatening and avoid asking for it. This was particularly evident in trusts in which the corporate function (i.e. service management) was visible at ward level only in response to criticism, complaints or serious incidents, rather than in response to feedback in the round.

Health economic modelling (work package 5)

To examine the cost-effectiveness implications of collecting and using patient experience data and of adopting best practice, three logic models were created. Two logic models, representing low- and high-intensity patient experience data processes (as identified in WP3), were developed. A third model was created to illustrate how implementing patient experience feedback could improve cost-effectiveness in the delivery of inpatient care, using the example of alleviating boredom on the ward. In the absence of prospective data, we sought to compare the costs (obtained using standard unit costs for activities characterised from our WP3 case study data) and potential benefits of these two archetypes.

A key conclusion of our economic modelling (WP5) was that the costs of collecting informal patient feedback (i.e. staff time) far outweighed all other aspects of the patient experience data cycle. Although it might be argued that some of this time is part of routine patient care, it highlights a dilemma: investment in collecting patient experience data this way carries substantial risk given the dearth of evidence showing that these data lead to substantive improvements in patient care.

Further findings

There were a number of findings that, although less common in our data set, require comment. The first of these is experiences of racism. This took the form of patients complaining that they had been subject to racist remarks by other patients and that staff had not responded to or addressed these adequately. We are unable to comment on whether or not this represents instances of institutional racism, but these experiences had a profound effect on the patients concerned. In terms of the aims of our research, our conclusion is that this created a divide between patients and staff, prevented the formation of trusting therapeutic relationships and deterred patients from BAME communities from giving honest feedback. One reason why this is so important is because patients of black ethnicity, in particular, continue to be over-represented in inpatient mental health settings. Unless experiences of racism are addressed effectively, it will continue to prove difficult for patients from BAME groups to give honest patient experience feedback. As long as this continues, services will be unable to improve in response to poor episodes of care, further marginalising and disadvantaging particular populations.

We also found that many (if not the majority of) patients that we interviewed reported that they did not understand their medication, had questions they wished to ask about their medication or were concerned about the side effects of medication. Likewise, many patients told us that they did not understand their care pathway, the implications of being detained under the Mental Health Act33 or their rights in relation to their health and care. Explaining medication and other aspects of treatment, and offering patients more opportunity to consult about their medication, was linked to the likelihood of obtaining patient experience feedback. Giving people time to talk needs to be a priority in inpatient settings, because, without having their questions answered and their concerns (e.g. about side effects) addressed, patients remain mistrustful of professionals. This resonates with the ‘patient-centred’ theme from the WP1 systematic review and reminds us that the processes that underlie good-quality care also increase the chances that patients will be motivated (and feel sufficiently secure) to provide meaningful feedback.