Notes
Article history
The research reported in this issue of the journal was funded by the HS&DR programme or one of its preceding programmes as project number 13/07/39. The contractual start date was in September 2014. The final report began editorial review in March 2017 and was accepted for publication in August 2017. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.
Declared competing interests of authors
none
Permissions
Copyright statement
© Queen’s Printer and Controller of HMSO 2018. This work was produced by Graham et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.
2018 Queen’s Printer and Controller of HMSO
Chapter 1 Introduction
Context
The importance of measuring patient experience for understanding the quality of care in an organisation is indisputable. Although a wide range of approaches are used by organisations to gather feedback from patients, there is a lack of evidence on the effectiveness of the various initiatives employed and how data are used to improve the quality of care. 1 Despite a growing emphasis on using real-time methods for gathering feedback while patients are still in hospital or shortly following discharge, the effectiveness of these approaches for driving service improvement remains largely under-researched. 2,3
This study set out to provide evidence to assess how effective near real-time data collection is for driving service improvement. The focus was on measuring the relational aspects of patient care, such as compassion, dignity and emotional support. Previous measures, such as the national Adult Inpatient Survey and the national Emergency Department Survey, have focused mainly on the functional or transactional aspects of care. This means that the majority of questions on these surveys aim to assess care aspects, such as access to services, waiting times and cleanliness of hospital areas.
Specifically, the Francis Report4 revealed deficiencies in these ‘softer’ aspects of care, and evidence suggests that these aspects are important for determining overall patient experience. 1,5 The NHS National Quality Board, in 2011, agreed on the NHS Patient Experience Framework as the guide to measurement of patient experience across the NHS. Respect for patient-centred values and emotional support, and key aspects of relational care, feature in this framework.
Moreover, a real-time approach to collecting data is considered to increase the chance of feedback being put to effective use as staff have a greater sense of ownership of the results; the data are more recent and have the potential to be more granular. 1,3 However, there is some evidence that surveys administered at the point of care produce more positive results than traditional-based surveys. 6 For instance, a survey of hospital trusts on hospital-acquired infections showed that the responses from the real-time data collection were significantly more positive than those from the paper-based survey collected months after discharge. 7
Another commonly cited limitation of the quality of real-time data is potential sampling bias: respondents choose to take part rather than being selected through a formal sampling strategy. 2,3 Similarly, there is a potential for introducing sampling bias from staff, who select which patients are most suitable to provide feedback. 8 However, it has been argued by Davies and colleagues2 that the aims of real-time data collection are different and, rather than trying to accurately measure the views of all patients, the purpose is to feed back data quickly to staff so that the necessary changes can be identified and acted on. There is evidence to suggest that using real-time patient feedback approaches can drive improvement in the quality of care provided to patients. Research carried out at the Royal Devon and Exeter NHS Foundation Trust, during which feedback was collected from patients on a daily basis, revealed a number of advantages of implementing the real-time data collection, including better teamwork, fewer complaints, better communication between staff and patients and improved service delivery. 9 Other studies that have employed a real-time survey using hand-held electronic devices have shown improvement in patient experience following changes implemented as a result of the real-time data collection. 10,11
It is widely accepted that the success of any survey approach for generating improvements in patient experience requires staff engagement and their involvement in interpreting and using the results for quality improvement. 1,2,9,10,12 Davies and colleagues2 argued that organisations must have an understanding of how to use the findings from near real-time feedback data collections in order to make organisational improvements. The potential barriers to using patient feedback data for bringing about service improvement may be caused by a lack of knowledge and understanding among staff of the changes planned, defensiveness, professional autonomy and limited time and resources. 2,9 Explaining the benefits of the study to staff and dealing openly with issues of scepticism and resistance to change will increase a project’s likelihood of success. 9,12
This research used a participatory action research approach to engage staff in the process of implementing the near real-time feedback (NRTF) survey and to identify the factors that promote and limit effective use of data, such as the information and support needs of frontline staff and service leads. The research conducted by the Royal Devon and Exeter NHS Foundation Trust, mentioned above, illustrates the advantages of properly engaging staff in the implementation of a real-time patient feedback approach. The study used a cyclical change management approach (the ‘plan, do, study, act’ model), which led to raised staff morale and improved care. 9
There is growing evidence that the experiences of NHS staff and patients are closely linked; improving staff experience will improve the patients’ experiences of care. 13–16 For example, one study found that patients were more satisfied when they received treatment and support from teams that had good team processes, such as teams that communicated effectively and whose members gave support for ideas that would lead to improved patient care. 15
Our approach, which was focused on working with staff and patients to identify what needs to change, reflects the fundamental principles of organisational change theory. The ‘emergent approach’ to organisational change fits well with the context and aims of this study. This approach sees change as a process of learning as the organisation responds to internal and external factors. 17 In their literature review of organisational change management, Barnard and Stoll17 state that proponents of the emergent school, such as Kotter,18 Kanter and colleagues19 and Luecke,20 have suggested a sequence of actions that organisations should take to increase the chances of change being successful, including that a set of suggested actions are shared, establishing a sense of urgency, creating strong leadership and empowering employees.
Although it was not the aim of this research to use a tool to measure and assess ‘organisational cultures’, the study utilised findings from previous research to better understand the nature of organisational culture, how it can be assessed and measured and how such assessments can be integrated into beneficial programmes of change. 21,22 The work carried out by Mannion and colleagues21,22 informed the planning and delivery of the participatory workshops and evaluation interviews with NHS staff. Of particular relevance were the findings of the research undertaken by Mannion and colleagues22 on the needs and interests of key NHS stakeholders with regard to understanding, assessing and shaping organisational cultures.
The work by Mannion and colleagues22 has also shown what users and patients consider to be the most important cultural attributes for high-quality care. These were ‘patient-centredness’, senior management commitment, a quality focus, clear governance/accountability and safety awareness. Patient-centredness was considered to be a key component for the following reasons: it was believed to lead to better process and clinical outcomes; it challenges cultures that are not aligned with the interest of patients; and, by putting the patient at the centre of decision-making, leads to a health service that is more accountable to them. 22
Our study built on these research findings and had a special focus on ensuring that patient representatives, such as the patient collaborators at the case study sites, were involved in the participatory workshops, in reviewing the findings from the NRTF data collection and in developing the resources to be used by organisations for driving quality improvement measures that improve the relational aspects of care.
As the project specifically sought to address the recommendations of the Francis Inquiry, it was referred to as the ‘After Francis’ project.
Literature review
Person-centred care is considered to be an essential factor of high-quality health care, and looking at patient experiences is one way for trusts to measure and improve the quality of patient-centred care. 4,23,24 Patient experiences can also influence other aspects of care quality, such as better safety and effectiveness,24–26 better treatment outcomes, fewer complications and overall lower service use,27 as well as better staff experiences. 28
The Francis Report4 recently identified substantial deficiencies in emotional or relational aspects of care, which are key determinants of overall patient experience. 29,30 These aspects of care focus on the relationships staff form with patients during their time in hospital and can include communication, providing the space for patients to discuss concerns or fears and treating patients with respect and dignity. 5,26 Owing to these deficiencies, the Francis Report recommended that hospitals should place particular focus on improving relational aspects of hospital care, especially for older patients and those visiting accident and emergency (A&E) departments. 27,28 An increase in the collection, and use, of NRTF was also recommended. Current patient experience data collection tends to focus data capture on transactional aspects of care, such as cleanliness, waiting times and pain management, with only a fraction of data collection dedicated to measuring relational aspects of care.
Two instruments that predominantly measure relational aspects of care were identified;31,32 however, these instruments have not been designed or tested for use with a near real-time approach or hard-to-reach patients. Although the instruments could be used with a NRTF approach or hard-to-reach populations, they would require further testing and refinement in order to do so. For example, the Patient Evaluation of Emotional Care during Hospitalisation (PEECH) measure32 was developed and tested in Western Australia. Further validation of the measure was carried out in four English acute hospitals. 30 Another instrument that primarily measures relational aspects of care is the Consultation and Relational Empathy (CARE) measure. 31 This measure was developed and tested for clinical use in primary care settings, with special focus placed on the instrument’s applicability to patients with various social economic backgrounds. An additional four instruments that measure some relational aspects of care were found; however, the main purpose of these instruments is to measure transactional aspects of care, such as pain management, waiting times and cleanliness. 33–36
As only two existing instruments focus primarily on relational aspects of care and, of these, only one has been designed for use in hospitals, we developed and tested an instrument specifically for use with a NRTF approach in A&E departments and inpatient wards that provide care primarily to elderly patients.
Need for the research
All NHS organisations were expected to respond to the Francis Report4 and to take action to strengthen patient voice, improve frontline care and change organisational culture. The health and social care quality regulator in England (the Care Quality Commission) has subsequently published strategic and business plans that place strong emphasis on the importance of patient voice. For example, the 2013–16 Care Quality Commission strategy, titled Raising Standards, Putting People First, highlights continuous engagement and relationship-building with patients and public representatives. 37 For NHS organisations, assuring and demonstrating the quality of care that they provide is a priority as never before; there is a window of opportunity for developing and disseminating an evidence base that will support NHS quality assurance and improvement initiatives pertaining to the Francis Report. 4
There is no one ‘best’ method for capturing data about patients’ experiences of care – each method has its strengths and limitations. Real-time feedback (RTF) is mandated, as of April 2013, for adult inpatients, A&E patients and maternity service users via a national policy directive, the Friends and Family Test (FFT). As of 2014, RTF is also mandated for other areas of care provision, such as mental health services, community health care and general practice. The Francis Report4 recommends that:
Results and analysis of patient feedback including qualitative information needs to be made available to all stakeholders in as near real-time as possible, even if later adjustments have to be made.
Similarly, in the review into the quality of care and treatment provided by 14 hospital trusts in England, published in July 2013, Sir Bruce Keogh also called for real-time patient feedback, stating that:
Real-time patient feedback and comment must become a normal part of provider organisations’ customer service and reach well beyond the Friends and Family Test.
There is therefore an urgent need for research that evaluates the introduction and impact of RTF approaches in the NHS in order to establish best practice and to ensure that data collection, presentation and communication supports frontline staff, service leads and managers in effectively improving services. 2
The Francis Report4 and the published literature alike38,39 highlight ongoing concerns about the quality and consistency of care that older people receive in hospital, especially around relational aspects of care, including communication and information provision. Similarly, the Francis Report4 raises concerns about neglect and poor care in A&E, and the government has launched a review of urgent and emergency services. The first phase of the review has been completed and provides the evidence base for changes to these services. 38 By focusing on older people’s wards and emergency departments, this evaluation of the introduction of NRTF is directly relevant to current and near-future priorities for NHS staff, service leads and managers.
Although the existing national surveys capture relational aspects to a small extent, a new methodology for understanding relational aspects of care more comprehensively and using a NRTF approach is warranted. Specifically, a new way of improving hospital quality that does not require clinical staff time for its implementation is needed. If NRTF could successfully be used for improving the relational aspects of care, the approach may potentially replace other internal data collection mechanisms in hospitals.
Research objectives
The overall aim was to develop and disseminate evidence-based recommendations to support the implementation of NRTF data collection about patient experiences in the NHS. The main objective was to evaluate the impact of a NRTF survey for driving improvements in patients’ experiences of the relational aspects of care.
The research used a mix of quantitative, qualitative and participatory research approaches in evaluating the collection, communication and use of real-time data about patients’ experiences of the ‘softer’ aspects of care. The aim was to explore and understand the inputs, processes and impacts of NRTF data collection, as reflected in patient experience data and as understood by NHS frontline staff, service leads and managers. Our key research questions (RQs) were:
-
Can NRTF be used to measure relational aspects of care?
-
Can NRTF be used to improve relational aspects of care?
-
What factors influence whether or not NHS staff can use NRTF to improve relational aspects of care? Specifically, what are the barriers and enablers?
-
What should be considered best practice in the use of NRTF in the NHS?
Structure of the report
Following this introduction, Chapter 2 describes the methodology for the set-up, data collection, implementation, and evaluation and reporting phases of the project. Chapter 3 presents findings to address the question, ‘Can NRTF be used to measure relational aspects of care?’ In Chapter 4, we showcase evidence of using RTF to improve relational aspects of care and the factors that influence whether or not NHS staff can use NRTF to improve relational aspects of care, such as the barriers and enablers. Chapter 5 highlights what should be considered best practice for using NRTF for making improvements in the NHS. Chapter 6 presents a synthesis and discussion of the findings, including the robustness of the collected data. Finally, Chapter 7 showcases conclusions and implications for policy and practice, as well as recommendations for future research.
Chapter 2 Methodology and data sources
This chapter presents the quantitative and qualitative methods used to evaluate the development and implementation of a near real-time survey for improving relational aspects of care. The chapter provides an overview of the data sources and methods used to answer the four RQs guiding this work. Additional details are presented for each data collection activity, including sampling and recruitment approaches and participant characteristics.
Figure 1 displays a flow chart that outlines the research activities, with arrows indicating how previously collected data feed into the subsequent research activities.
FIGURE 1.
Flow chart of research activities.
Research question 1
Table 1 presents the data collection methods and research activities that were employed to answer the first RQ, which explores whether or not NRTF can be used to measure relational aspects of care.
| RQ 1: can NRTF be used to measure relational aspects of care? | |
|---|---|
| Data collection methods | Data sources |
| Literature review to identify relational aspects of care | Published literature |
| Focus group and interviews | Recent A&E patients and hospital inpatients aged ≥ 75 years |
| Factor analyses of existing data | Existing data from the national 2012 NHS Emergency Department Survey35 and 2013 NHS Adult Inpatient Survey36 |
| Composite scores of existing data | Existing data from the national 2012 NHS Emergency Department Survey35 and 2013 NHS Adult Inpatient Survey36 |
| Other research activities | Methods basis |
| Recruit six case study sites | All trusts included in the national patient experience surveys for whom we had details |
| Develop draft survey instrument | Existing survey questions, findings from literature, focus group and interviews, and advisory group members’ suggestions |
| Apply for ethics approval | Draft survey instrument |
Literature review
An extensive search and review of the literature on existing scales used to measure relational aspects of care was conducted. Literature was accessed using the Web of Science online database, as well as the online search platforms PubMed and Google Scholar (Google Inc., Mountain View, CA, USA). The following search terms were used to identify relevant literature:
-
relational aspects of care
-
emotional care
-
patient–doctor relationship
-
patient–doctor communication
-
patient emotional care
-
relational communication.
In addition to databases and search engines, the research team used a snowball strategy to identify seminal literature based on the citations provided in the previously accessed literature.
Focus group and interviews
To understand what patients thought constituted relational aspects of care, one focus group with individuals who had visited an A&E department in the last 3 months was conducted on 24 September 2014. Participants were recruited through adverts in local newspapers. Although seven participants confirmed their participation on the telephone, only four attended the meeting. Of the four participants, three (75%) were females and all (100%) described themselves as white British. Focus group participants ranged in age from 28 to 65 years. The focus group was conducted over the course of 2 hours and included a discussion on what constitutes relational aspects of care and a ranking of their importance using a sort card exercise. Questions asked in the focus group were designed to explore what different aspects of care mean to patients in terms of their recent experiences. As little detail on what matters most to patients is provided in existing literature it was also important to explore this. Questions asked were:
-
What comes to your mind when you think about emotional care or relational care?
-
Sometimes it seems that one or two hospital or A&E department staff members are especially good with patients all around. Did you get this sense during your recent A&E department visit? If so, what made the care these staff provided stand out?
-
During your recent A&E department visit, were these relational or emotional aspects of care present or addressed by staff?
-
Why are these relational or emotional aspects important when you receive treatment?
-
Which components are most important to patients’ overall experiences of care?
-
What relational or emotional aspects cannot be missing from a good care experience?
Eight face-to-face interviews with older adults who had recently stayed overnight in hospital were conducted in lieu of a second focus group during the week commencing 13 October 2015. Three interview participants were male (37.5%) and aged 75 or 76 years. Five interviewees were female (62.5%) and between the ages of 75 and 85 years. All participants identified themselves as white British. Interviewers utilised the same discussion questions and prompts as in the focus group. Each interviewee also completed the sort card exercise. Interviews lasted between 40 and 50 minutes each.
For the sort card exercise, 19 cards were prepared prior to the focus group and six statements were prepared during the focus group, based on aspects of relational care that arose during discussions. The previously prepared statements were based on relational aspects of care identified in the following existing survey instruments:
-
the PEECH measure32
-
the 2013 NHS Adult Inpatient Survey36
-
the 2012 NHS Emergency Department Survey35
-
the CARE measure31
-
the 27-item Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) questionnaire
-
the General Practice Patient Survey (GPPS). 33
The exercise was completed by the eight interviewees and by two groups of four focus group participants. Therefore, data were available from a total of 10 card sort activities. During the exercise participants grouped the 25 statements into three categories, namely ‘most important’, ‘quite important’ and ‘least important’. The 25 statements and their importance rankings were entered into a Microsoft Excel® 2013 file (Microsoft Corporation, Redmond, WA, USA). The following values were assigned to each category: 2 = most important; 1 = quite important; and 0 = least important.
The focus group discussions and interviews were transcribed verbatim. Three researchers reviewed the transcriptions independently and created an initial coding framework. Comparison of frameworks highlighted extensive similarities among themes and categories. The framework with the least specific categories and which utilised participants’ own words for themes to the greatest extent was chosen for analysis purposes. It was anticipated that the framework would evolve during the coding process; however, only three additional categories were added for two themes during the coding process.
Data from the nine transcripts were then coded by assigning or coding each section or newly shared thought to the best-fitting categories or themes. Where sections of the transcripts appeared to fit under multiple themes or categories, they were coded for each. The coding process was conducted independently by two researchers, each using the software package Nvivo, version 10 (QSR International, Warrington, UK). Inter-rater reliability was computed and found to be 97%.
Factor analyses
To build on the understanding of relational aspects of care gained through the literature review, focus group and interviews with recent patients, factor analyses were conducted on patient experience data collected for the 2012 NHS Emergency Department Survey35 and the 2013 NHS Adult Inpatient Survey. 36 The purpose of these secondary data analyses was to identify a cohesive set of existing questions that measure relational aspects of care in the national surveys. At present, national inpatient and A&E department surveys do not specifically set out to measure relational aspects of care; however, they do include items which may be considered to tap into many of the aspects of relational care identified in our preliminary literature review and qualitative analyses. Survey items included in the analyses were selected based on themes of relational aspects of care identified in the literature review. Themes included patient-perceived level of security, knowledge and personal value during their hospital stay or visit. 32
Prior to conducting factor analyses, multiple imputation40 was conducted to replace missing and unscored responses to all items. For all survey questions, missing data ranged from 2% to 54%. However, the majority of survey questions had missing responses of ≤ 5%. Five imputations were completed for each survey data set. Next, factor analyses were conducted utilising data from 15 questions on the NHS Emergency Department Survey and from 18 questions on the NHS Adult Inpatient Survey. Data for both surveys were collected from separate patients and could not be linked across the two surveys. A polychoric correlation matrix for each survey was generated from the imputed data sets.
Exploratory factor analysis was then conducted on each data set using Factor v9.3 (Rovira i Virgili University, Tarragona, Spain). Each matrix was subject to two tests to determine the likely number of factors appropriate in each case. 41 The first test was parallel analysis, in which data eigenvalues are compared with those that would be obtained from random data. The second was Velicer’s minimum average partial (MAP) test, which seeks to minimise the amount of residual variance after different numbers of factors are extracted. The indicated numbers of factors were extracted using unweighted least-squares factoring and rotated (where appropriate) using Promin oblique rotation. The resulting loadings, communalities, residuals and fit statistics were examined to determine the suitability of the solution. Items were then allocated to scales on the basis of their loading patterns.
Scale analysis was conducted on the item sets for both surveys by fitting a single-factor model to each proposed scale. The resulting loadings were then used to analyse the overall reliability of the scales and the contribution of individual items. Reliability was evaluated using Cronbach’s alpha and McDonald’s omega (a more general coefficient of which alpha is a special case). 42 Item contribution was evaluated using McDonald’s item information index (the ratio of communality to uniqueness). 42
Composite scores
To select six case study sites, we first sought to identify high- and low-performing trusts on the 2012 NHS Emergency Department Survey35 and the 2013 NHS Adult Inpatient Survey. 36 Both were the most recent relevant national survey administrations for which complete data sets were available. Composite scores were computed based on data collected using existing survey questions that capture relational aspects of care.
Those survey questions that were identified to most closely measure relational aspect of care in previous factor analyses were utilised to compute composite scores. Composites were computed for all respondents who had answered at least 60% of the selected questions. For the 2012 NHS Emergency Department Survey,35 only one composite score was computed for each trust as the previous factor analysis indicated one underlying factor. For the 2013 NHS Adult Inpatient Survey,36 factor analyses indicated two underlying factors of relational aspects of care. Therefore, two composite scores were computed for each trust on the inpatient survey data, based on question loadings for each factor. To allow for comparison across trusts, an average score was calculated for each trust based on the three composite scores. Trusts were then sorted by the average score to identify the top and bottom one-third of the distribution.
Recruit six case study sites
All trusts falling within the top or bottom one-third of the composite score distribution described in Composite scores were initially contacted and informed about the opportunity for participation as case study sites; however, sufficient and timely interest was not generated using this approach. Therefore, trusts falling within the middle of the ranked distribution were also contacted. At the same time, reminder messages were sent to the trusts that were initially approached.
All trusts were sent an information sheet describing the project and the case study participation opportunity. Individual telephone conference calls were held with 26 trusts to further inform them about the purpose and benefits of the study and time and resource requirements for their participation.
Following the telephone discussions, interested trusts were asked to confirm in writing their interest in participating in the study as a case study site. The first three trusts falling below and above the median of the composite score distribution from which written confirmation was obtained were selected to participate in this study.
The six case study sites participating in the research were Hinchingbrooke Healthcare NHS Trust, Milton Keynes University Hospital NHS Foundation Trust, North Cumbria University Hospitals NHS Trust, Northern Lincolnshire and Goole NHS Foundation Trust, Poole Hospital NHS Foundation Trust and Salisbury NHS Foundation Trust. More details are presented about the case study sites in Chapter 3.
Develop draft survey instrument
The survey items identified as most closely related in the factor analyses were mapped to the themes of relational aspects of care identified in the literature, focus group and interviews. Additional survey questions were subsequently developed to ensure adequate coverage of each theme and that a pool of at least five survey items was available for each theme. This resulted in a pool of 62 potential survey items.
An expert advisory group, comprising nine members, was consulted at various stages of the project. Of these nine members, seven were female and two were male. The group included two patient and public representatives, two academic experts and two voluntary organisation representatives with a special focus on ageing, as well as three members of NHS organisations (specifically, one deputy chief nurse and two heads of patient experience).
During the meeting, members reviewed the themes and associated draft survey items. Under each theme, the two most representative and easy to understand questions were selected for face validity based on the following criteria:
-
Do the survey questions make sense?
-
How well do the survey questions represent their theme?
-
Which survey questions were easiest to understand?
-
Do any survey questions appear to be duplicates for this theme?
This selection process ensured that each theme was addressed with at least two survey items, while also reducing the burden placed on survey respondents by a lengthy survey instrument. Suggestions were offered for making the survey easier to understand and operate on a tablet. The researchers incorporated all suggestions, focusing on the tablet-based survey process and question wording, into the draft survey instrument.
Apply for ethics approval
The East of Scotland Research Ethics Service (EoSRES) was assigned through the NHS Integrated Research Application System to review this research, including the instrument development process. After the draft survey instrument was developed and prior to cognitive testing, the survey instrument was submitted to the same Research Ethics Service for review as a substantial amendment.
Research questions 2 and 3
Table 2 presents the data collection methods and research activities that were employed to answer the second and third RQs guiding this study. The second RQ investigates whether or not NRTF can be used to improve relational aspects of care. The third RQ aims to gather additional details on the context of the research activities to understand what factors influence whether or not NHS staff can use NRTF to improve relational aspects of care. Specifically, the third RQ aims to shed light on the different barriers to, and enablers of, using a NRTF approach for improving relational aspects of care.
|
RQ 2: can NRTF be used to improve relational aspects of care? RQ 3: what factors influence whether or not NHS staff can use NRTF to improve relational aspects of care? Specifically, what are the barriers and enablers? |
|
|---|---|
| Data collection methods | Data sources |
| Cognitively test survey instrumenta | Patients aged ≥ 75 years and A&E department visitors |
| Train volunteers and implement diary of their experiences | Hospital volunteers collecting patient feedback |
| Patient experience data collectiona | Patients aged ≥ 75 years and A&E department visitors |
| Participatory workshops | Hospital staff and patient collaborators |
| Online staff survey | Frontline staff |
| Networking event for all case study sites | Staff administering research at case study sites and frontline staff |
| Meetings with key staff at case study sites | Staff administering research at case study sites and frontline staff |
| Support calls for volunteers | Volunteers actively engaged in data collection |
| Telephone and in-person interviews with staff | Hospital and staff patient collaborators |
| Specific survey for interviewees | Staff who participated in first round of telephone interviews |
Cognitively test survey instrument
The draft patient survey was cognitively tested in three rounds with a total of 30 patients. This cognitive testing provided the basis for finalising the survey instrument. The testing took place at the following three case study sites at the beginning of May 2015:
-
Poole Hospital NHS Foundation Trust on 6 May 2015
-
Salisbury NHS Foundation Trust on 7 May 2015
-
Milton Keynes University Hospital NHS Foundation Trust on 11 May 2015.
A specific probing approach, including preprepared and spontaneous probes, was used to test each survey item with patients. 43,44 The cognitive process of responding in terms of the model described by Tourangeau45 was considered, seeking to establish consistency in:
-
comprehension – people understand what the question is asking in a consistent way that matches the intended RQ
-
retrieval – people are able to retrieve from memory the information necessary to evaluate their response to the question
-
evaluation – people are able to use retrieved information to evaluate the question meaningfully, and do this in an unbiased manner (e.g. not simply acquiescing or providing socially desirable responses)
-
response – people are able to match their evaluation to one of the available responses in a meaningful and appropriate way; the response selected adequately reflects the person’s experience.
Patients were approached on two wards and within the A&E department at each of the three trusts. Senior staff members directed the researchers to all patients who had the capacity to consent and were well enough to participate in a survey. Consenting patients then completed all survey questions, if necessary with help from a researcher. Assistance provided to respondents included reading questions and response options, entering responses or physically holding the tablet up for the patient to read. After each survey question was answered, a researcher asked two or three prepared follow-up prompts for each question. This question-by-question approach was selected as participants, who were primarily elderly patients, did not want to complete the survey and then return to discuss individual survey questions once they had already provided a response.
Specific probes were used to gauge respondents’ understanding of more complex words, such as condition, treatment and conversation. 43,44 In addition to these specific probes, if needed, spontaneous probes were also used to further assess patients’ understanding of survey questions and answer options.
Two examples of survey questions and corresponding specific probes are listed below:
-
If you have needed attention, have you been able to get a member of medical or nursing staff to help you?
-
Yes, always.
-
Yes, sometimes.
-
No, never.
-
I have not needed attention.
What do you think is meant by ‘attention’?
Who do you consider to be medical staff? Do you think they are different from nursing staff?
-
-
Has a member of staff told you about what danger signals regarding your condition or treatment to watch for?
-
Yes, completely.
-
Yes, to some extent.
-
No.
-
I have not needed this type of information.
What does ‘watch for’ mean to you?
What do you think is meant by ‘danger signals’?
-
A total of 38 patients were approached, of whom 30 completed the cognitive testing of the survey. Four elderly patients began the process but decided that they did not want to continue after a few questions and another four patients did not have the capacity to consent. Of the 30 participants, 13 were male and 17 were female. Twenty-eight patients identified themselves as white British. English was not a native language for two patients. One participant had dementia and a family member provided answers from the patient’s perspective for most survey questions. Twenty-one patients were currently inpatients in wards that provide care primarily to the elderly. The remaining nine patients were currently visiting A&E departments.
Volunteer training
Volunteer training sessions were conducted at each of the six case study sites to prepare volunteers thoroughly to collect patient feedback using the near real-time survey. The sessions lasted for 2.5–3.5 hours. As it was not possible for all volunteers involved in the data collection to attend the training, NHS trust staff, such as the project co-ordinators, patient experience leads and volunteer co-ordinators, also participated in the volunteer training. This way, additional volunteers could be trained by trust staff throughout the project.
Two members of the research team co-led the volunteer training at each site. The volunteer training was designed to address the following learning objectives:
-
understand the overarching goals of the After Francis research project
-
understand how the current RTF approach differs from existing approaches, such as the FFT
-
assist patients in reading the patient information sheet and understanding the purpose of the survey, including risks and benefits
-
obtain informed consent from patients
-
understand what constitutes capacity to consent and how to judge it based on patient’s communications
-
follow up on patient concerns, if necessary
-
provide assistance to patients by reading the survey questions and response options
-
follow standard infection control procedures for hands, tablet and patient information sheet
-
use computer tablets to administer the survey
-
understand the purpose of a weekly volunteer diary and discuss frequency and mode of administration
-
understand the patient collaborator role, descriptively termed peer researcher, and have an opportunity to ask questions or register interest.
In total, 61 volunteers were involved in the training sessions led by the research team, with further volunteers subjected to cascade training by the trained volunteers. We are unable to determine the number of volunteers who were subject to cascade training; however, we know that at least 35 individuals were trained by staff members or other volunteers at the trusts.
During the training, volunteers also practised administering the tablet-based survey to a colleague, followed by administering the survey to patients on study wards. The research team members were available to provide directions and answer any questions.
Following the initial training, the patient data collection process for volunteers was as follows:
-
check in with volunteer co-ordinator to receive ward/department assignment and collect tablet
-
report to ward manager or senior sister on the ward/department and get list of patients who should not be approached
-
approach remaining patients and enjoy interacting with them.
Volunteer diary and interviews
Volunteer diary
An online diary was made available to all volunteers who collected patient experience data on the wards and the A&E departments. The diary was in the form of an online survey, accessed via the tablets used for collecting patient experience feedback. The diary’s purpose was to provide a format for volunteers to share their experiences and challenges. All volunteers were asked to complete the diary on a weekly basis, provided they had approached patients for this project during the week.
The diary consisted of a demographic question, six closed-ended questions and one open-ended question. The six closed-ended questions prompted volunteers to rate the level of ease/difficulty in approaching and surveying patients, checking in with staff to receive information on which patients could be surveyed and the patients’ level of interest to participate in the survey. Volunteers could also indicate the reasons patients gave for not wanting to participate in the survey during the previous week. The open-ended question asked volunteers to further describe their experiences using their own language. The demographic question asked the volunteer to indicate which trust they were volunteering in.
The diary took ≤ 5 minutes to complete each week. The research team reviewed the diary entries on a weekly basis and informed the case study sites of any reported challenges so that they could be addressed immediately.
Volunteer interviews
The patient collaborators and one other volunteer were interviewed at each case study site. They shared their experiences with the tablet-based data collection, including challenges, barriers, what worked well and the perceived benefits to themselves resulting from participation in this research. Lessons learnt by the volunteers and patient collaborators were also explored.
During the month of September 2016, semistructured interviews were conducted with the volunteers involved in the project. If possible, volunteers were interviewed in pairs to encourage discussion; however, some volunteers wanted to be interviewed alone. Interviews were conducted either face to face or via telephone.
The purpose of the interviews was to understand the experience volunteers had collecting data for the After Francis project. For instance, we wanted to learn what worked well and what did not work well. Examples of some of the questions asked included:
-
What was it like being a volunteer and collecting data from patients?
-
In your opinion, what were the benefits of being a volunteer on this project?
-
What challenges did you encounter during your shifts for the After Francis project?
-
Please describe the types of support you received from staff, a volunteer co-ordinator or other volunteers?
Patient experience data collection
To understand patients’ experiences of the relational aspects of care, a 10-month-long survey data collection phase was implemented using trained volunteers to administer the survey. Data collection began during June 2015; one case study site started to collect near real-time data on 1 June 2015 and two other trusts began their data collection at a later date during the month. Two of the remaining case study sites began their data collection during the month of July and one case study site began its patient data collection in early August. Discrepancies in the dates on which case study sites began their data collection were a result of the varying times of volunteer availabilities. Moreover, most volunteer training was conducted at the beginning of June, prior to which trusts could not begin their NRTF survey data collection.
Patient experience data were collected on one to five wards/departments selected by staff at each trust. These included wards providing care primarily to patients aged ≥ 75 years and A&E departments. The recruitment targets were a minimum of 20 completed surveys per month for each ward and 50 completed surveys per month for each A&E department.
Initially, the research team planned to provide a stationary kiosk to each trust to facilitate data collection in the A&E department and two protected tablets for data collection on study wards. Four of the six sites requested a third tablet for their data collection in lieu of a stationary kiosk placed in the A&E department. These requests were made for a number of reasons, including lack of space, previous unsuccessful experiences with a kiosk in A&E and a strong desire to avoid placing a burden on staff who would be directing patients towards the kiosks. During the data collection, the remaining two trusts switched to using a tablet in the A&E department as a result of the low response rate obtained with the kiosk methodology.
Over 43 weeks of data collection, the recruitment targets were met 50 times out of the 643 potential times across the 15 participating wards. In A&E departments, recruitment targets were met 40 times out of a potential 258 times across the six sites.
An interactive reporting system using Microsoft Excel was designed by the research team to showcase weekly results from the patient experience survey. Feedback from case study sites, shared through staff interviews, the online staff survey and e-mail communications and at the participatory workshops, was used to guide the continued refinement of the reporting system to best meet the informational needs of the trusts. The reporting system continued to evolve throughout the data collection phase of the research. Specifically, weekly and monthly summaries were added to the reporting system, along with printable dashboard reports. Case study sites also requested the option to remove non-specific responses from results displayed for each survey question.
Following the 10-month data collection period, the quantitative patient experience data from all six sites were analysed using factor analyses. These are described in detail in Chapter 4. Free-text comments from all sites were combined and underwent a thematic analysis. Themes and example comments are presented in the results section of this report.
Participatory workshop
Two participatory workshops were conducted at each case study site for staff involved with the project implementation. Frontline staff, such as nurses, matrons and ward managers, were especially encouraged to attend the events. Workshops were held 3 months into patient data collection and again on completion of the data collection phase.
Prior to the workshops, project advisory group members were asked to share their thoughts and suggestions for increasing the usefulness of participatory workshops to staff involved with the project at the case study sites.
First, advisory group members discussed the usefulness of giving staff time off from their regular duties to consider and plan for improvements, as they considered it a task unable to be undertaken by administrators.
Another advisory group member stressed the importance of having a team of nurses or staff from each ward or department attending the workshops. This way, a ‘shared group perspective’ forms the basis of discussions and one person does not ‘feel overwhelmed’ by the perception of bearing the sole responsibility for improvements to patient care.
One advisory group member explained that some language used by the research team, such as ‘improvements’ or ‘changes to patient care’, may be perceived negatively by staff. The wording ‘working together for the future’ was suggested for use instead of or in addition to the current wording. Additionally, it was suggested to make the workshop content and patient data meaningful to staff by continuously relating discussions back to the patient. One way to make the data meaningful is to relate them to activities associated with the professional revalidation process for nurses. As part of this process, staff must engage in, and demonstrate, reflection and self-learning. Feedback from patients collected using the tablet-based survey can be used for this purpose, especially as responses provided in free-text comments may provide ‘personal feedback’ needed for the professional revalidation.
Advisory group members also proposed asking staff to focus on small and manageable changes for the action-planning activity. All changes must be easy to implement. In addition, a short- and long-term focus of the improvements should be apparent to staff. It was also suggested that ‘staff not shy away from understanding underlying reasons for improvements, as it may be necessary so staff can truly understand and improve patient care’.
Based on staff suggestions, participatory workshop activities were designed to provide a collaborative space for staff to interpret patient experience feedback collected as part of the project, and to maximise engagement. Staff identified and prioritised areas of patient experience that needed improvement and developed concrete action plans for each ward/department. In addition to engaging staff in a guided action-planning process, the research team sought to understand the levels of staff engagement with the near real-time data and discuss factors that encourage patients to provide feedback and staff to use the NRTF for improvements and decision-making. Finally, as part of the workshops, any barriers identified as hindering the use of NRTF could be explored as a group and mitigating strategies shared with the site leadership.
Staff survey
An online staff survey was designed to understand the types and methods of patient experience data collected and methods of data collection on the wards/departments involved in the research. In addition, the survey instrument explored the following contextual factors associated with patient experience data collection at each case study site:
-
how results from patient experience data collection are communicated to staff, including by whom and how frequently
-
preferences for communication modes and their desired frequency
-
preferred reporting formats
-
usefulness of patient experience feedback
-
perceived barriers to, and enablers of, using patient feedback
-
experiences with RTF data collection and using volunteers to collect patient feedback.
To capture changes in perceptions of and experiences with near real-time data collection, the staff survey was administered before (pre) and after (post) the patient data collection. This survey was made available to all staff working on wards and in A&E departments involved in the patient experience survey. At each case study site, an additional two wards that were not involved with the project were selected as control groups. All staff from these wards were eligible to take the online survey.
In the first administration, the survey was made available to staff in an online format. To notify staff of the upcoming survey opportunity, project leads at each trust prenotified all staff or senior staff members via e-mail, as well as in person if possible. If senior staff were notified, they were asked to pass the information on to their team members.
The survey was administered using the Snap survey platform (version 11; Snap Surveys, Tidestone Technologies, Overland Park, KS, USA) and hosted on Snap WebHost. Invitations were sent to trust contacts on 21 May 2015. Five project leads at the trusts forwarded the invitations on to their teams shortly thereafter. One trust was still awaiting the site-wide NHS permission letter, which delayed the survey invitation mailings by 2 weeks. Reminder messages were sent to project leads at the case study sites 1 week following the initial survey invitation mailings.
Owing to the low number of survey responses (see Table 3), project leads at the case study sites were asked to share any suggestions or ideas they had for increasing the number of responses for the second survey administration. The main suggestions received focused on the substitution of the online survey for of a paper-based version, which could be distributed to all staff working on the wards.
Advisory group members were also asked for suggestions to improve the response rates. Similar to staff, they suggested the use of an alternative survey format, as staff on wards are generally not provided with access to e-mail during their work hours. A paper- or a tablet-based survey version that volunteers could take to staff was suggested for use instead.
Advisory group members recommended shortening the survey, as frontline staff have limited time. It was also considered beneficial to make it clear to staff why it was important to complete the survey.
For the second administration, the research team reduced the number of questions on the survey and made it available in a paper-based format or on the tablets previously used for patient data collection. In addition, the survey prenotification, invitation and reminder messages were replaced by in-person communication from the project team with the ward/department leaders and senior staff.
The paper-based survey was printed as an A5 booklet and handed out to all staff members working on wards/departments by the senior staff members. Where possible, time was made available in ward or department meetings for staff to complete the survey. Staff placed their completed surveys into envelopes, which were collected by a trust research and development (R&D) team member and returned to the research team. The survey was available for completion over a 2-week period.
Three trusts chose to administer the survey as a tablet-based version, which was also available over a 2-week period. Using this method, R&D team members took the tablets to the wards during handover times and encouraged staff to complete the survey at that time. Table 3 presents the number of responses and selected survey administration modes for each of the two survey time points at each case study site, along with the change in number of completed staff surveys between the first and second time point.
| Case study site | Responses, % (n) | Difference in recruitment, % (n) | Mode of administration post data collection | |
|---|---|---|---|---|
| Pre-patient experience data collection | Post-patient experience data collection | |||
| Hinchingbrooke | 28.2 (20) | 28.3 (41) | 105.0 (21) | Paper |
| Milton Keynes | 4.2 (3) | Not available | –100.0 (3) | Paper |
| North Cumbria | 14.1 (10) | 1.7 (3) | –70.0 (7) | Paper |
| Northern Lincolnshire and Goole | 2.8 (2) | 30.2 (52) | 2500.0 (50) | Paper and tablet |
| Poole | 16.9 (12) | 29.1 (50) | 316.7 (38) | Tablet |
| Salisbury | 33.8 (24) | 15.1 (26) | 8.3 (2) | Tablet |
Networking event for case study staff
A networking event was held on Tuesday 24 November 2016, at Harris Manchester College, Oxford, to enable key staff from each of the six case study sites to meet each other and share their experiences regarding the project. Specifically, staff provided feedback on experiences of data collection; barriers and enablers that staff had come across; impacts the data have had within the trust; and lessons learnt.
Meetings with key staff at case study sites
Five meetings with key staff from each study site were held on 24 and 25 November 2016 to coincide with the networking event. As one of the sites was unable to send a representative to the networking event, a conference telephone call was held on 2 December 2016 with four staff members involved in the project.
Support calls
To support volunteers with their data collection, the research team invited all volunteers to join them in monthly conference telephone calls. Support telephone calls were scheduled separately for each trust. During these telephone calls, volunteers could discuss any questions, comments or concerns with the research team. Conference call opportunities were available on the following dates:
-
23 October 2015
-
13 November 2015
-
4 December 2015.
Invitations to participate in calls were sent to the key contacts at the case study sites, who confirmed that they forwarded the messages and reminders on to all their volunteers currently involved in data collection for the project.
Telephone interviews with staff
Prior to implementing patient data collection, the project leads at each case study site were asked to assist the research team in scheduling 10 telephone interviews with staff who would be involved with the research project. A total of 52 interviews were conducted via telephone during the months of May and June 2015. This constituted the first round of staff interviews. Interview duration ranged between 14 and 35 minutes.
Through the interviews, we aimed to understand, in detail, the types of patient experience data currently collected at the trusts, how results are communicated, and the factors or structures that facilitate or hinder the use of patient experience data for improvement purposes. In addition, staff shared examples of improvements or changes to services that have been made based on patient experience feedback. Finally, staff shared their expectations for the upcoming patient data collection, the benefits and challenges associated with real-time data collection and the use of volunteers to collect patient experience feedback.
Some example questions and follow-up probes are as follows:
-
What types of patient experience data are your trust currently collecting? How frequently? Are any volunteers involved?
-
What procedures or structures are in place to make improvements or changes based on patient feedback?
-
What are your experiences with NRTF of patient experiences? What benefits do you anticipate? What drawbacks do you anticipate?
-
How could your trust benefit from volunteer involvement to collect NRTF? What should be considered when working with volunteers?
The following number of interviews were conducted at each case study site:
-
Hinchingbrooke Health Care Trust (n = 10)
-
Milton Keynes University Hospital NHS Foundation Trust (n = 10)
-
North Cumbria University Hospitals NHS Trust (n = 7)
-
Northern Lincolnshire and Goole NHS Foundation Trust (n = 5)
-
Poole Hospital NHS Foundation Trust (n = 10)
-
Salisbury NHS Foundation Trust (n = 10).
The roles of the interviewed staff working on wards and departments included the director of nursing, head of nursing, senior sister, senior staff nurse, deputy sister, matron, ward sister, staff nurse, lead consultant, unit lead, Foundation (year 2) doctor, health-care assistant, ward clerk and receptionist.
In addition, a lead governor and staff working in R&D departments, voluntary services and customer care were interviewed. Staff whose roles do not typically involve contact with patients were selected to participate in interviews as they were involved in collecting patient experience feedback from patients. For example, staff administered the FFT or other short surveys designed specifically for the trust to collect data from patients while they are still in hospital. Interviewees’ length of employment at the sites ranged from 6 months to 41 years.
Specific survey for interviewees
On completion of the 10-month patient data collection period, the 52 staff members across the six case study sites who participated in the initial telephone interviews were asked to complete an online survey consisting of closed and open questions. The purpose of the survey was to understand their involvement and experiences with the project and to determine how their previously shared expectations of near real-time data collection and experiences working with volunteers to collect patient experience feedback have changed. Staff were also asked about improvements to care made as a result of the NRTF collection. Similar questions were asked to those during the initial telephone interview, with answer options derived from the responses to the telephone interviews. Additional space for other answers was available.
This survey was available for completion online. It was set up in Snap and hosted on Snap WebHost. Staff received a prenotification message from their colleagues who were responsible for the implementation of the research at each case study site. Following this initial communication, the research team invited staff by e-mail to participate in the survey. The survey was available for a 2-week period in April 2016. Non-respondents received two reminder e-mails. Of the 52 staff members who participated in the initial telephone interviews, seven had left the trusts or were currently on maternity leave. Therefore, a total of 45 staff members were invited to take the online survey and 31 completed the survey, yielding a response rate of 68.9%.
In-person interviews with staff
A second round of staff interviews was conducted following the end of the patient experience data collection. Five staff members from each trust were interviewed in person or via telephone by the research team. Questions asked were similar to those asked during the first round of interviews. Instead of expectations for the data collection, we asked about their experiences with the data collection as part of this research. Interviews allowed us to explore any changes in opinions about NRTF and working with volunteers to collect patient experience feedback that resulted from their involvement in the current research. Interviews lasted between 14 and 35 minutes and staff members with the following roles participated:
-
matrons
-
ward sisters
-
deputy sister
-
director of nursing.
Research question 4
Table 4 presents the research activities that were employed to answer the fourth RQ, which explores what should be considered best practice in the use of NRTF in the NHS.
| RQ 4: what should be considered best practice in the use of NRTF in the NHS? | |
|---|---|
| Research activities | Methods basis |
| Develop a survey toolkit and evidence-based practice recommendations | Synthesis of evidence from all data sources presented under RQs 1–3 |
| Disseminate toolkit and practice recommendations to NHS trusts | Policy commentary, publication, conference presentations and toolkit launch events |
Develop a survey toolkit and evidence-based recommendations
Based on the patient experience data collection and the evaluation data collected from staff and volunteers through surveys and interviews, a toolkit was developed to share the survey instrument, including guidelines and recommendations, with other NHS hospitals. These data sources evidenced the success of the NRTF approach for improving relational aspects of care. With the help of the advisory group, the toolkit contents were developed and refined to make them more relevant and meaningful to the target audience at NHS hospitals. In addition, the advisory group selected the branding, including colours, font, title and pictures for the toolkit. Four guides and three case studies were also included alongside the survey instrument. These documents showcase the challenges, lessons learnt and impacts of the work at the case study sites. Volunteer training materials were made available to assist trusts in implementing the approach.
Disseminate toolkit and practice recommendations to NHS trusts
To share the toolkit, three regional toolkit launch and networking events were held at the following locations in February and early March 2017:
-
Novotel Paddington, London, 20 February 2017
-
Novotel Leeds Centre, Leeds, 27 February 2017
-
Harris Manchester College, Oxford, 2 March 2017.
The launch events commenced with a presentation providing an overview of the research, followed by a detailed description of the purpose and components of the toolkit. This was followed by time for questions about the content and functionality of the toolkit. Time for networking and further small group discussions were provided.
In February 2017, the toolkit was made freely available on the Picker website (URL: www.picker.org/compassionatecare, accessed 22 November 2017) and publicised through a published policy commentary. 46 In addition, the toolkit was disseminated through Twitter (Twitter, Inc., San Francisco, CA, USA) and e-mails sent to our contacts at NHS trusts nationally.
Chapter 3 Using near real-time feedback to measure relational aspects of care
Results for research question 1: ‘can near real-time feedback be used to measure relational aspects of care?’
Literature review
Relational aspects of care
Person-centred care is considered to be a key component of high-quality health care and patient experiences provide one important avenue for measuring and improving the quality of patient-centred care. 4,23,24 Although patient experiences are important in and of themselves, they have also been shown to be correlated with other aspects of care quality: they are related to safety and clinical effectiveness24–26 and are associated with better treatment outcomes, fewer complications and overall lower service use,27 as well as better staff experiences. 28 Relational aspects of care have been established as more challenging to measure and improve than transactional aspects of care. 5,32
The need to strengthen relational aspects of care is consistently identified by policy-makers. 5,47 Recently, the Francis Report4 identified important deficiencies in emotional or relational aspects of care, which, the evidence suggests, are the key determinants of overall patient experience. 29,30 Relational aspects of care, also referred to as compassionate care, focus on the relationships staff form with patients during their time in hospital and can include communication, providing the space for patients to discuss concerns or fears and treating patients with respect and dignity. 5,26 Owing to the documented deficiencies, which were especially salient in experiences of older patients and those visiting A&E departments, the report of the Francis Report recommended that a special focus should be placed by hospitals on improving relational aspects of hospital care, especially for older patients and those visiting the A&E departments, as they are an important component of the patient experience, which may in turn affect treatment outcomes. 27,28 In addition, an increase in the collection and use of NRTF was recommended.
Current patient experience data collection primarily captures transactional aspects of care, such as cleanliness, waiting times and pain management. Although some relational aspects of care, such as being treated with dignity and respect and being fully informed about treatment, can be included, these have not been the focus of existing patient experience data collection.
Two instruments were found that aim to measure mainly relational aspects of care. 31,32 These instruments have not been designed or tested specifically for use with a near real-time approach or with patients who are generally considered to be harder to reach. Although the instruments might be used with this approach or hard-to-reach populations, further testing and refinement may be necessary.
The PEECH measure was developed and tested in Western Australia. 31 Emotional care was defined as the ‘interpersonal interactions that facilitate or enhance the state of emotional comfort’. 31 As part of the study, a new intervention was designed to improve emotional care on an oncology ward and the PEECH survey was created to measure the success of the intervention. The PEECH measure was developed based on a framework identified in previous research, which suggested that three components are key contributors to emotional care. These components are titled level of security, level of knowing and level of personal value. Face and content validity were assessed, as well as internal consistency of the measure. Adequate sensitivity of the PEECH measure was established for use in a ward setting. The PEECH survey contained 25 questions, constructed as Likert scales, which were grouped into three subscales matching the components described above.
Further research on the PEECH measure examined the psychometric properties of the instrument. 32 Internal consistency was assessed using Cronbach’s alpha, which ranged from 0.60 to 0.79 for the three subscales. Exploratory factor analyses were conducted to verify the internal structure of the measure and whether or not the three components or subscales could be sustained. The results showed that an additional subscale of emotional care, named level of connection, was assessed by the PEECH measure. These results further highlighted the complexity of emotional care.
The PEECH measure, which was originally developed and validated in Australia, was also validated in four English acute hospital settings. 30 Factor analyses revealed a different underlying structure of the instrument and two new factors, named personal interactions and feeling valued, emerged. These differences may in part be attributable to the different timings of the data collection. In the Australian samples, data were collected while patients were in hospital, whereas in the English samples, data were collected following discharge. The researchers concluded that further testing of the measure is warranted to inform the measurement and reporting of emotional or relational aspects of care.
Another instrument measuring primarily relational aspects of care is the CARE measure. 31 This measure was developed by researchers with the help of patients and experts in emotional care. It was developed and tested specifically for clinical use in primary care settings. A special focus was placed on the measure’s applicability to patients with various social economic backgrounds. The final instrument comprises 10 questions, which can be answered using a five-point Likert-type rating scale. A ‘does not apply’ answer option is available. The measure has not been designed or tested for use in hospitals or with a NRTF approach.
An additional four instruments were found to measure some relational aspects of care; however, their main purpose is to measure transactional aspects of care such as pain management, waiting times and cleanliness. 33–36 The instruments include the NHS Adult Inpatient Survey, the NHS Emergency Department Survey, the GPPS and the HCAHPS questionnaire. The last two instruments are described in more detail as examples below.
The GPPS, administered in primary care since 2007, focuses on waiting times, scheduling, access to a preferred doctor or practice nurse, care planning and overall satisfaction. 33 However, this survey also includes some questions which capture relational aspects of care, including having sufficient time to ask questions, staff listening to patients, obtaining clear and sufficient explanations, being treated with care and concern, and having confidence and trust in their doctor. This survey instrument has been developed and tested for use in a variety of geographic areas, as well as socioeconomically deprived practices. Question wording has been revised to ensure consistent understanding for patients with a range of disabilities. Owing to the primary focus on transactional aspects of care and the instrument’s intended use in primary care, the measure is not suitable in its current form for the measurement of relational aspects of care in hospital settings.
The 27-item HCAHPS questionnaire was developed to provide a standardised tool to capture experiences of patients on seizure monitoring units following discharge. 34 The survey was developed specifically to provide a means for comparing the experiences of patients on seizure monitoring units with the experiences of patients on other similar units. Twenty-seven questions are included in the instrument. These are answered using 4- and 10-point rating scales, as well as through dichotomous answer options.
Although the instrument focuses primarily on capturing patient experiences of the transactional aspects of care, including the provision of discharge information and the hospital environment, some relational aspects of care are also assessed. These focus on both physician and nursing care with regard to treating patients with dignity and respect, listening carefully, providing clear explanations and providing assistance to patients when requested. The survey can be used in hospital settings as a tool to capture transactional aspects of care.
In conclusion, as only two existing instruments focus primarily on relational aspects of care and, of these, only one has been designed for use in hospitals, we developed and tested an instrument specifically for use with a NRTF approach in A&E departments and inpatient wards that provide care primarily to elderly patients.
Focus group and interviews
Focus group and interview responses were analysed thematically, first separately then again in combination, using a coding framework developed specifically for these activities. This framework applied to both the individual and combined analyses. The same themes appeared across all transcripts and were mentioned at a similar rate of frequency in interviews and the focus group, suggesting that there was a general agreement on themes’ importance between the focus group and interviews.
Table 5 presents the most frequent themes and the number of occurrences found in the focus group and interview analysis. It presents a simple count of the number of times statements related to each theme were mentioned; therefore, the counts for each theme can exceed the total number of participants.
| Theme | References in (n) | Total (n) | |
|---|---|---|---|
| A&E focus group | Eight inpatient interviews | ||
| Feeling informed | 12 | 26 | 38 |
| Being listened to and feeling involved in care | 3 | 19 | 22 |
| Confidence – trust in staff knowledge | 5 | 16 | 21 |
| Kindness | 2 | 16 | 18 |
| Surpassing expectations or making special effort | 3 | 14 | 17 |
| Showing empathy | 0 | 16 | 16 |
| Competent staff | 2 | 14 | 16 |
| Feeling alone, vulnerable or unnoticed | 5 | 8 | 13 |
| Efficiency | 2 | 10 | 12 |
| Staff ability to cope when understaffed | 0 | 11 | 11 |
| Feeling like a priority or adequately monitored | 0 | 11 | 11 |
| Opportunity to ask questions and knowing what to expect | 2 | 9 | 11 |
| Stress, anxiety and/or fear | 4 | 7 | 11 |
| Feeling staff were thorough | 0 | 9 | 9 |
| Feeling secure in knowledge others are in control – safe hands | 3 | 5 | 8 |
| Feeling comfortable – at ease with staff | 2 | 5 | 7 |
| Use of humour – friendly | 3 | 4 | 7 |
| Good rapport | 2 | 4 | 6 |
| Feeling respected | 0 | 5 | 5 |
| Facilitating communications with friends and family | 0 | 2 | 2 |
| Emotional state | 0 | 1 | 1 |
Frequent themes
Additional details on the seven most frequent themes are presented below.
Feeling informed
The theme of feeling informed focused on how well the care staff explained the condition or treatment. This included whether or not patients were told of any danger signals to watch out for once they left hospital and whom to contact if they were concerned:
If you can’t communicate with someone adequately for the situation, the person’s not sufficiently stress free to be able to cope with what they’ve got to cope with.
Patient
It is considered an especially important aspect of relational care because it suggests that good communication between the patient and care staff is fundamental to allowing the patient to be aware of what is happening and what steps to take. It gives the patient the ability to understand and, subsequently, utilise the information he or she is given to aid them in coping with the situation.
Being listened to and feeling involved in care
This theme included if staff listened to what patients had to say and whether or not patients felt they were involved as much as they wanted to be in decisions about care and treatment:
I think she [the doctor] was genuinely struggling with whether I should be admitted or not so she did want input from me, this is after I was diagnosed as stabilised and I’d had pain relief and she, I wasn’t expecting this but it was almost like she left it to me and I so wanted to go home that that was a decision, she seemed really open.
Patient
This theme was important to participants as being listened to and involved in care was seen to empower the patient. This allows individuals to feel like they are in control of their treatment, not just being told what will be happening to them:
I think it’s essential if you’re going to get better, if you’re both [self and hospital staff] working to the same aim you’re bound to get better sooner than if I’m sitting back saying ‘What are you going to do now?’
Patient
Confidence: trust in staff knowledge
Trust in staff knowledge included whether or not patients were confident in the staff who treated them. In addition, this theme encompassed whether or not patients trusted staff’s judgement when it came to examining and treating them. Patients considered the theme an important aspect of relational care as they are leaving their treatment in the hands of professionals and they want to trust them to do the best that they can do:
. . . suddenly he [my son] was having all these X-rays from different angles of his skull and neck and then doing all these special measurements and stuff and I kind of, my thoughts were that he’d slept awkwardly on his pillow, but I was glad that they were really, really thorough and it’s like they weren’t taking any risks really.
Participant
Confidence in care staff also helps to reduce any anxiety experienced by patients:
I was never afraid of the operations you know, even the first one I wasn’t scared because I trusted them.
Patient
Kindness
The theme of kindness addressed the reassurance patients received from staff if they felt distressed, as well as having someone to talk to about their worries and fears. It was important to recent patients because being received in a kind and friendly manner helps to reassure them that they are being cared for as a person and treated compassionately:
Each person is an individual, each person is treated with the same compassion, professionalism, dignity.
Patient
When patients were treated with kindness, they did not feel isolated during their stay:
You know, you felt like you were with a friend.
Patient
Showing empathy
Aspects of this theme included staff helping to reassure patients when they felt distressed, treating patients with respect and dignity, and affording patients an opportunity to discuss any worries or fears about their treatment or condition. It was highlighted as an important aspect of relational care because being in hospital can be a frightening and disorientating experience. For example, patients may not know what is wrong with them when they are first admitted and have to cope with a lot of pain or distress. Therefore, having an empathic and reassuring approach to patients is important and serves to avoid additional distress:
. . . a senior doctor brings a great bevy of people around the bed . . . and then he shouts across to me, ‘oh you’ve got lymphoma, do you know what that is?’ he shouts, so I say ‘Yes it’s cancer of the lymph glands’, he said ‘Yes it’s probably quite treatable these days, we’re referring you on to the lymph people’ and off he went . . . now that was completely out of the blue, I was not expecting that diagnosis at all and in fact it was incorrect . . . and the only person who spoke to me after that was the woman in the next bed who said ‘oh lymphoma, my neighbour died of lymphoma’.
Patient
Surpassing expectations or making a special effort
Patients felt that it was important that staff appeared to make a special effort or surpassed their expectations occasionally. This made patients feel valued or provided them with reassurance that they were being looked after and that staff were doing all they could to help them. One person described the special effort the A&E department nurse made:
. . . when I saw the second nurse, it was 8 o’clock on a Sunday evening, she seemed a bit tired and fed up and maybe had a long shift. I had already had to go and see a nurse and get an X-ray and then get a cast and then get another X-ray. I had been there for 5 or 6 hours. By the end the nurse, instead of like making me wait to see her again, she made the effort to come into the waiting room and talk to me and my girlfriend in there which was good. It was a nice gesture, just to get us home quicker.
Patient
Competent staff
The theme of competent staff included staff responding quickly and effectively to requests for assistance and staff appearing confident and able to perform specific tasks when caring for patients. It is considered an important aspect of relational care because patients are required to leave their care and treatment in the hands of professionals. Patients need reassurance and trust that staff are doing everything and the best they can:
. . . you need reassurance and you need that from a person who is confident and competent and relaxed, sufficiently relaxed.
Patient
Feeling alone, vulnerable or unnoticed
Unlike the previously described themes, which focused on positive staff behaviours or attitudes, this theme focused on the negative consequences resulting from a lack consideration of relational aspects of care. The aspects that fall under this theme include nurses not responding to calls for assistance, patients not being checked on or monitored frequently and patients’ general feelings of isolation. It is important that patients are not left feeling alone or vulnerable or do not go unnoticed because it makes them feel that there is a lack of care and attention, which leads them to question the treatment they receive:
Yeah well, the nurses weren’t brilliant when you wanted some attention and I didn’t want much attention, they weren’t brilliant, they were talking at their stations.
Patient
. . . rang my bell, nobody came, they just sort of looked down, I was still alive, they didn’t come, rang my bell again, I needed to go to the loo and I couldn’t get out of bed, I wasn’t allowed to get out of bed, this was after surgery, called again, nobody came . . .
Patient
Sort card exercise results
A total of 25 statements were used for the sort card exercise. Answers were categorised into ‘very important’, ‘quite important’ and ‘least important’. However, to determine agreement, the categories of very important and quite important were collapsed to create a dichotomous variable of importance. The two categories were then reassigned 1 for important and 0 for not important. Percentages were calculated across the 10 rankings (eight interviews and two focus groups).
The percentages of agreement were calculated for sort card responses for the focus group with recent A&E patients and the eight interviews with recent inpatients. Once averages were calculated, the most frequent aspects of the sort cards were aligned to themes identified in the focus group and interviews (Table 6). The average percentage of agreement ranged from 44% to 100%.
| Sort card statement | Corresponding theme from the focus group/interviews |
|---|---|
| Staff are confident and able to care for patients | Competent staff |
| Being treated as a person not a case | Showing empathy |
| Openness and honesty about what was happening and what to expect | Opportunity to ask questions and knowing what to expect |
| Good communication skills | Feeling informed |
| Staff listening to what you have to say | Being listened to and feeling involved in care |
| Feeling informed | Feeling informed |
| Answering questions and being given information in a way you can easily understand | Opportunity to ask questions and being given information in a way you can easily understand |
| Having enough time to discuss problems | Efficiency |
| Being able to talk about worries and fears about your condition or treatment | Feeling comfortable/at ease with staff |
| Positive staff | Competent staff |
| Treatment is explained thoroughly before it is given | Feeling informed |
| Having confidence in your care staff | Competent staff |
| Helping you take control, not giving lectures | Feeling secure in knowledge others are in control/safe hands |
| Manner in which staff communicate | Showing empathy |
| Friendly staff | Use of humour/friendly |
Aligned themes with sort card statements
The results from the sort card exercise further validate that the frequent themes identified from the interviews and focus group discussions are consistent with areas of importance within the current literature. It also provides evidence on the consistency of rankings in terms of order of importance within relational care.
Cross-validation of themes
To provide cross-validation of themes identified in the focus group and interviews, NHS staff from various trusts were consulted. The project team participated in regularly scheduled A&E department survey results workshops held by Picker. These workshops were held for NHS staff members from trusts across the UK. As part of the workshops, staff were asked to list what they thought patients considered important in terms of relational care. Themes discussed by NHS staff were consistent with themes highlighted as important in the focus group and interviews carried out with recent patients. Important aspects of relational care raised by staff members are as follows: staff introductions and goodbyes (building relationships); communication on waiting times; a dialogue which acknowledges the patient’s pain; empathy (care/kindness/compassion/appropriate touch); and having time to chat with the patient. This convergence of themes for relational aspects of care, as identified by both patients and NHS staff, further highlights the themes’ validity for this research.
Factor analyses
Results for the trust-level discrimination analyses conducted in preparation for the factor analyses are reported in Tables 7 and 8 by survey question for the intraclass correlation, illustrative trust-level reliability and the number of respondents needed to achieve threshold reliability levels of 0.7 and 0.8.
| Survey item | Interclass correlation | Trust-level reliability (Φ) with 250 responses | Threshold reliability level | |
|---|---|---|---|---|
| Number required for Φ = 0.70 | Number required for Φ = 0.80 | |||
| Q4 | 0.008 | 0.66 | 307 | 526 |
| Q5 | 0.029 | 0.88 | 78 | 134 |
| Q6 | 0.015 | 0.79 | 157 | 269 |
| Q7 | 0.033 | 0.89 | 70 | 119 |
| Q8 | 0.027 | 0.87 | 85 | 146 |
| Q9 | 0.014 | 0.79 | 160 | 274 |
| Q10 | 0.030 | 0.88 | 77 | 131 |
| Q11 | 0.006 | 0.59 | 401 | 687 |
| Q12 | 0.007 | 0.64 | 326 | 558 |
| Q13 | 0.006 | 0.61 | 378 | 647 |
| Q14 | 0.006 | 0.59 | 402 | 688 |
| Q15 | 0.008 | 0.66 | 299 | 513 |
| Q16 | 0.011 | 0.74 | 207 | 354 |
| Q17 | 0.006 | 0.60 | 397 | 680 |
| Q18 | 0.004 | 0.47 | 656 | 1124 |
| Q19 | 0.008 | 0.68 | 276 | 473 |
| Q20 | 0.012 | 0.75 | 191 | 328 |
| Q21 | 0.007 | 0.63 | 336 | 576 |
| Q22 | 0.006 | 0.58 | 416 | 713 |
| Q24 | 0.006 | 0.60 | 391 | 671 |
| Q25 | 0.007 | 0.63 | 341 | 583 |
| Q26 | 0.003 | 0.46 | 689 | 1181 |
| Q28 | 0.013 | 0.77 | 171 | 294 |
| Q29 | 0.041 | 0.91 | 56 | 95 |
| Q30 | 0.057 | 0.94 | 39 | 66 |
| Q31 | 0.016 | 0.80 | 148 | 254 |
| Q32 | 0.013 | 0.77 | 171 | 293 |
| Q35 | 0.007 | 0.64 | 328 | 561 |
| Q36 | 0.010 | 0.71 | 235 | 402 |
| Q37 | 0.010 | 0.71 | 244 | 417 |
| Q38 | 0.012 | 0.76 | 188 | 322 |
| Q39 | 0.010 | 0.71 | 244 | 417 |
| Q40 | 0.007 | 0.64 | 332 | 568 |
| Q41 | 0.017 | 0.81 | 139 | 237 |
| Q42 | 0.010 | 0.71 | 236 | 404 |
| Q43 | 0.015 | 0.79 | 153 | 262 |
| Q44 | 0.015 | 0.79 | 158 | 270 |
| Survey item | Interclass correlation | Trust-level reliability (Φ) with 250 responses | Threshold reliability level | |
|---|---|---|---|---|
| Number required for Φ = 0.70 | Number required for Φ = 0.80 | |||
| Q3 | 0.010 | 0.73 | 221 | 378 |
| Q4 | 0.011 | 0.74 | 209 | 358 |
| Q6 | 0.018 | 0.82 | 130 | 222 |
| Q7 | 0.012 | 0.74 | 201 | 344 |
| Q8 | 0.008 | 0.67 | 284 | 487 |
| Q9 | 0.028 | 0.88 | 81 | 138 |
| Q11 | 0.017 | 0.81 | 134 | 230 |
| Q14 | 0.033 | 0.89 | 69 | 119 |
| Q15 | 0.014 | 0.78 | 165 | 282 |
| Q16 | 0.005 | 0.57 | 440 | 754 |
| Q17 | 0.031 | 0.89 | 74 | 126 |
| Q18 | 0.035 | 0.90 | 65 | 111 |
| Q19 | 0.002 | 0.32 | 1221 | 2092 |
| Q20 | 0.004 | 0.50 | 585 | 1003 |
| Q21 | 0.039 | 0.91 | 57 | 98 |
| Q22 | 0.018 | 0.82 | 131 | 223 |
| Q23 | 0.019 | 0.83 | 120 | 205 |
| Q24 | 0.010 | 0.71 | 237 | 406 |
| Q25 | 0.008 | 0.66 | 295 | 505 |
| Q26 | 0.010 | 0.71 | 236 | 404 |
| Q27 | 0.014 | 0.78 | 161 | 276 |
| Q28 | 0.013 | 0.77 | 176 | 301 |
| Q29 | 0.012 | 0.75 | 190 | 325 |
| Q30 | 0.025 | 0.86 | 92 | 157 |
| Q31 | 0.007 | 0.64 | 336 | 575 |
| Q32 | 0.012 | 0.75 | 190 | 326 |
| Q33 | 0.007 | 0.65 | 313 | 537 |
| Q34 | 0.020 | 0.84 | 113 | 193 |
| Q35 | 0.018 | 0.82 | 127 | 218 |
| Q36 | 0.008 | 0.66 | 295 | 505 |
| Q37 | 0.005 | 0.53 | 512 | 877 |
| Q39 | 0.011 | 0.73 | 218 | 374 |
| Q40 | 0.030 | 0.88 | 77 | 131 |
| Q42 | 0.006 | 0.62 | 358 | 614 |
| Q43 | 0.005 | 0.57 | 432 | 741 |
| Q44 | 0.005 | 0.56 | 463 | 793 |
| Q45 | 0.007 | 0.65 | 321 | 551 |
| Q47 | 0.004 | 0.51 | 563 | 964 |
| Q48 | 0.010 | 0.71 | 236 | 405 |
| Q49 | 0.011 | 0.73 | 215 | 368 |
| Q50 | 0.011 | 0.74 | 209 | 357 |
| Q52 | 0.014 | 0.78 | 161 | 275 |
| Q53 | 0.018 | 0.82 | 125 | 214 |
| Q54 | 0.015 | 0.79 | 158 | 270 |
| Q55 | 0.009 | 0.68 | 271 | 464 |
| Q56 | 0.015 | 0.79 | 155 | 265 |
| Q57 | 0.008 | 0.66 | 297 | 509 |
| Q58 | 0.012 | 0.75 | 196 | 335 |
| Q59 | 0.017 | 0.81 | 133 | 228 |
| Q60 | 0.012 | 0.75 | 198 | 340 |
| Q61 | 0.011 | 0.74 | 208 | 356 |
| Q62 | 0.015 | 0.79 | 152 | 260 |
| Q63 | 0.015 | 0.80 | 150 | 257 |
| Q64 | 0.005 | 0.58 | 431 | 738 |
| Q65 | 0.093 | 0.96 | 23 | 40 |
| Q66 | 0.009 | 0.69 | 262 | 449 |
| Q67 | 0.013 | 0.77 | 175 | 300 |
| Q68 | 0.023 | 0.85 | 100 | 171 |
| Q69 | 0.020 | 0.84 | 113 | 194 |
| Q70 | 0.018 | 0.82 | 128 | 220 |
Determining the factor structure
A correlation matrix was selected corresponding to each subset of items chosen for further analysis. Exploratory factor analysis was then conducted using Factor v9.3. Each matrix was subject to two tests to determine the likely number of factors appropriate in each case. 41 The first test was parallel analysis, in which data eigenvalues are compared with those that would be obtained from random data (essentially a signal-to-noise analysis). The second was Velicer’s MAP test, which seeks to minimise the amount of residual variance after different numbers of factors are extracted. Both approaches were in agreement, indicating that the A&E data reflected one latent variable, whereas the inpatient data reflected two latent variables.
Factor extraction
The indicated number of factors were extracted using unweighted least-squares factoring and rotated (where appropriate) using Promin oblique rotation. The resulting loadings, communalities, residuals and fit statistics were examined to determine the suitability of the solution.
Inpatients
The model fit was good by some indices [goodness of fit index (GFI)/adjusted goodness of fit index (AGFI) > 0.95; root-mean-square of residuals (RMSR) < 0.05], but not acceptable according to others (Table 9):
-
non-normed fit index = 0.81
-
comparative fit index = 0.86
-
GFI = 0.99
-
AGFI = 0.99
-
RMSR = 0.049.
| Variable | Factor | |
|---|---|---|
| First | Second | |
| Q3 | 0.366 | 0.264 |
| Q24 | 0.629 | 0.209 |
| Q25 | 0.731 | 0.106 |
| Q26 | 0.547 | 0.045 |
| Q27 | 0.902 | –0.115 |
| Q28 | 0.982 | –0.234 |
| Q29 | 0.729 | –0.134 |
| Q32 | 0.627 | 0.188 |
| Q33 | 0.586 | 0.292 |
| Q34 | 0.665 | 0.097 |
| Q35 | 0.896 | –0.067 |
| Q42 | –0.006 | 0.896 |
| Q43 | –0.155 | 1.013 |
| Q44 | –0.031 | 0.958 |
| Q45 | –0.030 | 0.798 |
| Q47 | –0.055 | 0.760 |
| Q48 | 0.133 | 0.650 |
| Q67 | 0.933 | –0.099 |
One item (survey question 3) loaded only weakly with the first factor and almost as much with the second. All other items loaded clearly with one factor or the other. On the basis of these results, survey question 3 was dropped from further consideration.
Accident and emergency
The model fit was acceptable by some indices (GFI/AGFI > 0.95; RMSR < 0.08), but not by others (Table 10):
-
non-normed fit index = 0.78
-
comparative fit index = 0.81
-
GFI = 0.99
-
AGFI = 0.98
-
RMSR = 0.068.
| Variable | First factor |
|---|---|
| Q10 | 0.279 |
| Q11 | 0.838 |
| Q12 | 0.854 |
| Q13 | 0.882 |
| Q14 | 0.818 |
| Q17 | 0.767 |
| Q19 | 0.676 |
| Q21 | 0.552 |
| Q22 | 0.827 |
| Q24 | 0.713 |
| Q25 | 0.516 |
| Q26 | 0.758 |
| Q40 | 0.549 |
| Q41 | 0.635 |
| Q42 | 0.794 |
The loadings indicated that one variable (survey question 10) did not load strongly enough with the factor, whereas others loaded strongly. On this basis, survey question 10 was dropped from further consideration.
Scale analysis
Scale analysis was conducted on the revised item sets for both surveys (Tables 11–13). In the case of the NHS Adult Inpatient Survey, items were mapped onto one of two scales (see Tables 11 and 12) according to which of the two factors they loaded with. The resulting loadings were then used to analyse the overall reliability of the scales and the contribution of individual items. Reliability was evaluated using Cronbach’s alpha and McDonald’s omega (a more general coefficient of which alpha is a special case when certain restrictive conditions apply). 42 Item contribution was evaluated using McDonald’s item information index (the ratio of communality to uniqueness). 42
| Variable | Factor loadings on factor 1 | Communalities | Uniquenessesa | Item information | Scale reliability |
|---|---|---|---|---|---|
| Q24 | 0.787 | 0.619 | 0.381 | 1.626 | |
| Q25 | 0.816 | 0.666 | 0.334 | 1.994 | |
| Q26 | 0.582 | 0.339 | 0.661 | 0.512 | |
| Q27 | 0.813 | 0.662 | 0.338 | 1.956 | α = 0.940 |
| Q28 | 0.801 | 0.642 | 0.358 | 1.792 | |
| Q29 | 0.626 | 0.392 | 0.608 | 0.645 | |
| Q32 | 0.769 | 0.591 | 0.409 | 1.446 | |
| Q33 | 0.790 | 0.624 | 0.376 | 1.660 | ω = 0.941 |
| Q34 | 0.736 | 0.541 | 0.459 | 1.180 | |
| Q35 | 0.847 | 0.718 | 0.282 | 2.544 | |
| Q67 | 0.857 | 0.735 | 0.265 | 2.772 |
| Variable | Factor loadings on factor 1 | Communalities | Uniquenessesa | Item information | Scale reliability |
|---|---|---|---|---|---|
| Q42 | 0.892 | 0.796 | 0.204 | 3.900 | |
| Q43 | 0.915 | 0.838 | 0.162 | 5.168 | α = 0.930 |
| Q44 | 0.941 | 0.885 | 0.115 | 7.700 | |
| Q45 | 0.784 | 0.615 | 0.385 | 1.597 | ω = 0.933 |
| Q47 | 0.731 | 0.535 | 0.465 | 1.149 | |
| Q48 | 0.728 | 0.529 | 0.471 | 1.125 |
| Variable | Factor loadings on factor 1 | Communalities | Uniquenessesa | Item information | Scale reliability |
|---|---|---|---|---|---|
| Q11 | 0.844 | 0.712 | 0.288 | 2.473 | |
| Q12 | 0.851 | 0.725 | 0.275 | 2.633 | |
| Q13 | 0.892 | 0.796 | 0.204 | 3.900 | |
| Q14 | 0.822 | 0.676 | 0.324 | 2.085 | α = 0.941 |
| Q17 | 0.774 | 0.599 | 0.401 | 1.494 | |
| Q19 | 0.684 | 0.468 | 0.532 | 0.879 | |
| Q21 | 0.550 | 0.302 | 0.698 | 0.433 | |
| Q22 | 0.829 | 0.687 | 0.313 | 2.196 | ω = 0.943 |
| Q24 | 0.698 | 0.488 | 0.512 | 0.952 | |
| Q26 | 0.719 | 0.517 | 0.483 | 1.070 | |
| Q40 | 0.550 | 0.303 | 0.697 | 0.434 | |
| Q41 | 0.642 | 0.412 | 0.588 | 0.701 | |
| Q42 | 0.805 | 0.648 | 0.352 | 1.841 |
Composite scores
All survey questions identified as most closely measuring relational aspect of care in previous factor analyses were utilised to compute composite scores. Composites were computed for all respondents who had answered at least 60% of the selected questions. For the NHS Emergency Department Survey, only one composite score was computed for each trust as the previous factor analysis indicated one underlying factor. For the NHS Adult Inpatient Survey, factor analyses indicated two underlying factors of relational aspects of care. Therefore, two composite scores were computed for each trust on the NHS Adult Inpatient Survey data, based on question loadings for each factor. To allow for comparison across trusts, an average score was calculated for each trust based on the three composite scores. Composite scores ranged from 73.95 to 87.23. Trusts were then sorted by the average score to identify the top and bottom one-thirds of the distribution. A total of 47 trusts fell into each one-third of the distribution. The two scores separating the distribution into one-thirds were 80.92 and 82.85.
Recruit six case study trusts
Following the composite score identification and the recruitment process detailed in Chapter 2 (see Recruit six case study sites), the six trusts listed below served as case study sites for the remainder of the project:
-
Hinchingbrooke Health Care NHS Trust (composite score of 82.18)
-
Milton Keynes University Hospital NHS Foundation Trust (composite score of 79.60)
-
North Cumbria University Hospitals NHS Trust (composite score of 82.41)
-
Northern Lincolnshire and Goole NHS Foundation Trust (composite score of 79.46)
-
Poole Hospital NHS Foundation Trust (composite score of 83.97)
-
Salisbury NHS Foundation Trust (composite score of 86.05).
Figure 2 presents a map of trust locations across England.
FIGURE 2.
Case study trust locations. Map data: Google, DigitalGlobe.
Develop draft survey instrument
The final survey instrument, including survey information sheet and informed consent statement, can be found in Appendix 1. The instrument was developed based on the review of the literature, focus group and interviews with recent patients, factor analyses of data collected using existing survey items and consultation with the advisory group.
Apply for ethics approval
An application for full ethics approval was submitted using the Integrated Research Application System. The EoSRES Research Ethics Committee 1 reviewed our application, including instrumentation, and provided a favourable opinion on 29 August 2014. The draft questionnaire, measuring relational aspects of care, which was developed during the project set-up phase, was submitted to the Research Ethics Committee for review as a substantial amendment on 11 December 2014. Subsequent approval of the substantial amendment was granted by the EoSRES Research Ethics Committee 1 on 22 December 2014.
Chapter 4 Using near real-time feedback to improve relational aspects of care
Results for research questions 2 and 3: evidence of using near real-time feedback to improve relational aspects of care
Cognitively test survey instrument
During the first round of cognitive testing with 10 patients, six patients said that the survey contained too much repetition or too many redundancies, specifically, that questions 17–19 were near duplicates of each other. The three survey questions are listed as follows:
-
Question 17 – if you had any anxiety or fears about your condition or treatment, has a member of staff discussed them with you?
-
Question 18 – has a member of staff told you about what danger signals regarding your condition or treatment to watch for?
-
Question 19 – if you have felt distressed while in hospital, has a member of staff helped to reassure you?
In addition, four patients felt strongly that questions 7 and 8 were nearly identical:
-
Question 7 – have you been told how to contact the nurses if you needed any help?
-
Question 8 – if you have needed attention, have you been able to get a member of medical or nursing staff to help you?
Taking into account patients’ feedback, the research team removed questions 7 and 19 from the survey instrument. Question 7 was removed as all patients explained that they had been shown how to ring the bell or buzzer to call the nurses. Question 19 was removed as patients found the word ‘distressed’ much more difficult to understand than the wording of similar questions.
The revised survey was then cognitively tested in two subsequent rounds with a total of 20 patients. These 20 patients did not report any duplication or redundancies in the remaining survey questions. No other changes to the survey instrument were suggested by patients or evidenced in their answers in the cognitive testing process.
Answers to question prompts highlighted good, consistent understanding among participants. Detailed and coherent explanations of terms were given from all respondents and consistent responses were observed across the three rounds of cognitive testing. For example, patients provided the following statements when explaining what feeling safe meant to them:
I know where everyone is always and I am looked after.
Patient
You trust staff.
Patient
Well cared for.
Patient
Cognitive testing provided confirmation to the research team that the survey took most patients around 10 minutes to complete.
Volunteer training
Two members of the research team co-led the volunteer training at each case study site. A total of 55 individuals were trained to collect data by the research team. This included a few staff members at each case study site, who then used the training materials to train additional volunteers to collect data as needed.
A practice opportunity was included as part of the training. Volunteers practised administering the tablet-based survey to peers, then administered the survey to patients on study wards. The research team members were available to provide directions and answer any questions. After practice sessions, the volunteers rejoined as a group to debrief and share their experiences of the practice surveys. Based on feedback shared during the debriefing, volunteers thought that the tablets were easy to use and allowed for a user-friendly navigation throughout the survey. Volunteers felt welcomed to the clinical areas by the ward staff and found patients were generally happy to participate and welcomed the opportunity to speak with a volunteer.
During the question and answer session of the meeting, volunteers and trust staff obtained additional clarification on the need to always make the patient information sheet available to all approached patients and the type of help volunteers can give to assist patients in understanding the information presented. Volunteers at one trust also felt that a plain English version of the information sheet was needed for older patients. One volunteer assisted the researchers in designing a plain English version, which was submitted to the Research Ethics Committee as part of a substantial amendment and was subsequently made available to all case study sites. Volunteers also asked for clarification regarding judging patients’ capacity to consent, as this was considered especially important when working with older patients with long-term conditions.
Volunteer diary and interviews
Volunteer diary
A total of 258 respondents completed the diary. The highest percentage of responses was obtained from Poole Hospital NHS Foundation Trust (24.4%), followed by Northern Lincolnshire and Goole NHS Foundation Trust (17.8%). The weeks during which volunteers completed the diary most often were weeks 17 and 32 (both resulting in 12 responses, 4.7%) and the weeks when volunteers completed the diary least often were weeks 1 and 5 (both resulting in one response, 0.4%).
In-person interviews with volunteers
A total of nine volunteers were interviewed across the six case study sites. When asked about their involvement in the project, some of the volunteers stated that they were involved in the initial research study set-up only. Others were involved substantially in data collection for the duration of the project. Some were volunteers with no paid roles in the trust, whereas others were research nurses, who assisted with the study set-up and supplemented data collection when volunteers were not available.
The interviews were audio-recorded and lasted approximately 15–40 minutes. Below is a breakdown of interviews conducted per trust:
-
Hinchingbrooke Healthcare NHS Trust (n = 1)
-
Milton Keynes University Hospital NHS Foundation Trust (n = 3)
-
North Cumbria University Hospitals NHS Trust (n = 0)
-
Northern Lincolnshire and Goole NHS Foundation Trust (n = 1)
-
Poole Hospital NHS Foundation Trust (n = 2)
-
Salisbury NHS Foundation Trust (n = 2).
Typically, when collecting data, volunteers first checked in with staff to make them aware of their presence in an area and check which patients should not be approached to be interviewed because of their health status. Next, volunteers informed the patients of the purpose and nature of the survey, collected informed consent and, if necessary, aided patients in completing the survey on the tablets.
Results
Analysis of the volunteer interviews and diary entries showed frequent overlap between the results of the data sources. Therefore, the results for the volunteer diary are accompanied by supporting statements from the volunteer interviews.
Question 1: speaking with ward managers for patient selection
Of the 258 volunteers who completed the diary, 34.9% found it very easy to speak with the ward managers to determine which patients were suitable for participation in the patient survey. The use of volunteer diaries was considered easy by 30.2% and neither difficult nor easy by 22.5%. Only 7.8% of respondents found it difficult to speak with ward managers and only 0.8% found it very difficult to do so.
Question 2: were patients interested in the survey?
Volunteers were asked to feed back on how interested the patients appeared to be in taking part in the survey. The majority (60.7%) of volunteers reported that patients seemed somewhat interested in taking part in the survey, followed by patients seeming very interested in taking part in the survey (25.6%). Only 4.7% of volunteers reported that patients seemed not interested in taking part in the survey.
Question 3: reasons why patients declined to participate
The most frequent reason for declining to take part in the survey was the patient feeling unwell (40.7%), followed by the volunteer approaching the patient at an inconvenient time (18.6%), lack of interest (17.8%), no reason given (8.5%) and language barriers (3.1%). Other reasons why patients declined to participate were also given (16.0%):
Felt that it would be a bad report as she is very dissatisfied.
Volunteer
Banana more important!
Volunteer
Patient extremely anxious and awaiting crisis team.
Volunteer
Never take part in surveys!
Volunteer
Question 4: patients answering the survey questions
Volunteers also reported that, for the most part, patients found it easy to answer the survey questions (43.9%), with approximately one-third (33.7%) of patients finding it neither difficult nor easy to do so. Thirty-one responses suggested that patients found it very easy to answer the survey questions (12.0%). Only 15 responses suggested that it was difficult for the patients to answer the questions (5.8%) and only two volunteer responses stated that patients found the survey very difficult to answer (0.8%).
Patients’ privacy was also a concern when completing the survey. Often the wards lacked privacy for patients to answer survey questions, causing some patients to feel uneasy when giving negative feedback:
Yeah, you would often see them following the member of staff with their eyes. And they didn’t want to answer the question . . .
Volunteer
Question 5: engaging hard-to-reach patients
Volunteers were asked how easy or difficult they found it to engage hard-to-reach patients to take part in the survey. The most common response was that it was neither difficult nor easy (37.2%) to engage hard-to-reach patients. The proportions of hard-to-reach patients that were reported to be easy or difficult to engage were similar (20.9% and 20.5%, respectively). Fourteen responses reported that it was very difficult (5.4%) to engage hard-to-reach patients, whereas only 11 responses (4.6%) indicated that the volunteers found this aspect very easy.
Question 6: forming relationships with patients
When asked if they were able to form relationships with patients during data collection, the majority of volunteers said that they could form relationships with most patients (46.9%), all patients (24.0%) or some patients (23.6%). In only six cases were volunteers unable to form a relationship with the patient (2.3%).
Question 7: free-text comment
Respondents were asked ‘Please use the box below to tell us about your experiences with data collection this week’. Of the 258 responses to the volunteer diary, 160 responses to this question were given. These experiences provided additional detail to illustrate the answer options selected in response to the previous closed questions in the survey, as well as restating some of the complaints already shared by patients.
Themes
Seven themes were identified in responses. These themes are described below and have been further split into whether or not the experiences were positive, negative or both.
Patient experience and feedback
Comments from volunteers suggest that some patients found their experiences in hospital to be a positive experience and that they reported being happy with their care:
It was interesting to hear from one of the patients who had been in a few years ago say how much better the care they have received this time has been.
Volunteer
[Patients] have shared positive and beneficial [feedback] about the hospital, as well as their health, statements about their care and the outcomes of it.
Volunteer
However, some patients explained to volunteers that their experiences and care while in hospital required much improvement. Some discussions focused on the lack of staff availability, low staff levels and the fact that patients have been unhappy regarding the lack of explanations surrounding their diagnoses, which have impacted negatively on their emotional well-being:
One lady had very negative experiences on the ward. She has no relatives and is anxious about her care. She has asked to speak to the Sister three times but still has not seen her. She has praised all of the foreign nurses, and said that they are very kind, but that some of the English nurses are very hard and not kind.
Volunteer
Concern on [department/ward name redacted] with staffing levels, referred patient to PALS [Patient Advice and Liaison Service].
Volunteer
One patient stated that the professionals had not taken the time to explain the anatomy associated with the diagnosis made as well as treatment. The lack of this knowledge impacted the emotional aspect of the health status of the patient.
Volunteer
Some comments recorded in volunteers’ diaries were positive and some were negative. Negative comments related to concerns about staffing levels and, in one case, the fact that the patient, although happy with the care received, was unhappy with the ward environment:
Feedback was mostly positive, however a recurring comment was how understaffed patients felt the ward is.
Volunteer
Most people are very happy with their treatment and care. But we keep hearing about the noise in the wards at night with patients crying out and shouting while the rest of the patients are trying to sleep!!
Volunteer
Patient recruitment
Volunteers also described their experiences with the recruitment of patients to participate in the survey. Some volunteers reported having no problems in engaging the patients to participate in the survey, often commenting that they seemed ‘keen’ to take part. During an interview, one volunteer explained that, generally, 99.9% of the people were willing to give their opinion:
A very willing sample of patients today!
Volunteer
No problems – all the patients were keen to take part and generally very positive about the ward. Patients were easy to talk to – interviews took longer because of very interesting conversations with patients.
Volunteer
However, some responses suggested that the volunteers also often found it difficult to get patients to participate in the survey. Volunteers learnt that they only had short windows of opportunity to survey the patients, because of protected mealtimes and patient care, causing challenges with recruiting patients. However, in the A&E department there was a different challenge – the faster turnover of patients:
Difficulty with a deaf lady. I had to sit really near to her and enunciate the words clearly. Three patients did not want to complete the survey, looked ill, hot and bothered in the heat of the ward.
Volunteer
Still a problem getting on to some wards before 10.00 a.m. due to patient care time and have to get out for mealtimes which causes a bit of a panic.
Volunteer
Some mixed feedback was also provided in the diary, suggesting that, although volunteers experienced some problems with data collection, in general, it was easy to recruit and engage with patients:
The patients were approachable and unless they felt unwell were happy to participate, although one patient fell asleep part-way through the questionnaire so I had to end that session.
Volunteer
As I am assigned to a ward for hip, knee replacements, etc. There is a quick turnover of patients so there is usually no problem in finding participants. Sometimes there have been very few patients and when some are asleep it can be a wasted visit!
Volunteer
Staff interest and buy-in
Volunteer comments within the diary suggested that levels of staff buy-in of the project can differ (e.g. depending on the ward/department the volunteer is working in). The levels of staff interest or buy-in directly affect the experiences of volunteers.
Substantial staff buy-in is important, as volunteers see staff as fundamental to ensuring that patient recruitment goes smoothly, by directing volunteers to approach the correct patients:
The [department/ward] staff was really friendly and was able to be approached. They guided me to people I can talk to and others I couldn’t which was helpful for me.
Volunteer
Staff very helpful in assisting me with advising me of which patients I could approach with the survey on the [department/ward]. A member of staff on the [department/ward] was extremely helpful and showed me respect.
Volunteer
Despite it being the day of the strike the staff were very accommodating and helpful.
Volunteer
On the other hand, volunteers also explained that some staff acted as barriers towards data collection. Some volunteers reported that staff reacted negatively towards them when entering the ward or department. Partly, this was as a result of staff being suspicious of the data collection or because staff remained unaware of the project and its purpose despite frequent explanations from the volunteers:
One consultant questioned me about the survey and wanted to know why we chose his department, which is one of the top three departments in the country. He felt that relationships with patients are very good in [department/ward], and although it is not perfect, it would be more use to use another ward.
Volunteer
Nursing staff did not seem to think that patients were appropriate for approach. I am not sure that the nursing staff understood my brief.
Volunteer
These barriers were experienced during the first month of data collection only, and were brought to the attention of the site lead, who explained the purpose of the research in more detail to the staff.
Comments on the survey
Feedback in the diary suggested that the majority of volunteers, and some patients, found some survey questions more challenging to answer than other questions. These issues were mainly associated with question repetition, or questions confusing patients because of a lack of understanding or language barriers. Volunteers also reported that patients found the survey to be lengthy:
Question 17 seems to always confuse patients. But other than that it’s very good and easy to communicate and get answers.
Volunteer
Question, ‘have family or carers been informed’ patients found difficult to answer as some didn’t know whether their family had been or not – no option for this answer.
Volunteer
A lady whose first language was not English found some of the vocabulary hard to understand so had to simplify some questions. One person thought the questions needed reviewing as some were repetitive.
Volunteer
However, despite an opinion that the survey requires further review, feedback also suggested that the survey was well constructed. Yet some patients gave vague, non-direct answers that required follow-up from the volunteer:
On the whole well-constructed survey although one or two patients perceived some of the questions to be repetitive.
Volunteer
The questions are straightforward for the patients but, sometimes the patients are too vague with their answers.
Volunteer
Tablet and technology
The majority of volunteers experienced some technological issues with their tablets, which were resolved during the initial 2 months of the data collection. Some volunteers also found their tablets heavy to work with, especially when standing next to patients to hold the tablet on their behalf. However, these issues applied to only two case sites:
I find the tablet heavy to work with. No one seems interested in using it themselves on elderly care – it isn’t familiar or easy for patients who are not lying very comfortably in bed.
Volunteer
Technical difficulties: tablet froze on several occasions or went to friends and family or patient experience demo and unable to get it back to Picker had to wait a while so missed a couple of patients.
Volunteer
Having volunteers to aid patients with the survey meant that patients could still complete the survey if they were hesitant to use the tablets. Volunteers often reported that few patients used tablets. When given a choice between completing the survey themselves and getting aid from the volunteer, patients would often request that the volunteer fill in the survey on their behalf.
Although the tablets provided initial challenges, they were not problematic for the duration of data collection. Generally, tablets were found to be easy to use. Sometimes, volunteers were unsure whether or not survey responses were received by Picker:
Most people found the tablet easy to use with my assistance.
Volunteer
I am a little concerned as to whether all the surveys are sent off properly when we give the iPad [Apple Inc., Cupertino, CA, USA] to the patients to fill in by themselves. And they hand it back to us completed? Otherwise all good.
Volunteer
It was also suggested that the tablets were beneficial to the approach. The electronic devices were generally held in higher regard than paper surveys.
Volunteer benefits
Comments suggested that volunteers generally had a positive experience with data collection and found that they received many benefits from spending time with the patients. These benefits include feeling useful in carrying out an important role, forming relationships with patients, enjoying the opportunity to talk to patients and finding it rewarding and interesting:
I was talking so some older patients and was told some amazing stories while filling out the questions. I helped fill it in as they were not great with the tablets but we built a relationship and had a great time.
Volunteer
Had end-of-life patient in [department/ward]. Had a great conversation with them about their past and they said I made a difference to their day. I felt good and happy.
Volunteer
Plain that many patients on [ward/department name redacted] relish opportunity to talk to someone. The process of the survey often opens memory of long and interesting life experiences. This provides a gratifying sense of usefulness, likely to encourage volunteers to undertake this kind of work.
Volunteer
Other
Other comments that did not align with the previously discussed themes have been grouped into general positive and negative comments. Positive comments mainly described pleasant and generally positive experiences with data collection on the wards and how volunteers can help improve the patient experience by feeding back to staff after data collection. For example, they informed staff of immediate patient needs, such as needing food or water:
Very pleasant experience.
Volunteer
In my opinion and experience I have seen patients have been treated with respect, dignity and compassion.
Volunteer
Relatives helped with two patients especially with one more confused.
Volunteer
Other comments were considered less positive. They detailed that some staff were unaware of how the weekly reports related back to their wards, issues related to data collection, such as ward closures, and that there were generally very few volunteers available to collect patient feedback. For example, one trust relied only on one volunteer to collect data. This was considered challenging, as volunteers felt more pressure to survey patients:
The senior Sister told me that in the weekly report from Picker, she does not know which results are attributed to [department/ward name redacted].
Volunteer
[Department/ward name redacted] became [department/ward name redacted] which was then closed and is now temporarily a discharge lounge, long-term future is unknown.
Volunteer
When asked if they would like to be involved in future NRTF within the trusts, volunteers stated that they would like to see a continued use of the approach. However, there needs to be comprehensive volunteer training and a support system in place for volunteers to use. In addition, volunteers stated that they would like to have the results of the feedback frequently shared with them on a regular basis.
Patient experience data collection
Quantitative analyses
The survey of patient experiences was completed by 3928 patients over a 10-month data collection period. Of these respondents, 22% of surveys were completed at Poole Hospital NHS Foundation Trust, 20% at Salisbury NHS Foundation Trust, 19% at North Cumbria University Hospitals NHS Trust, 15% at Northern Lincolnshire and Goole NHS Foundation Trust, 13% at Hinchingbrooke NHS Trust and 12% at Milton Keynes University Hospital NHS Foundation Trust.
Fifty-five per cent of respondents were female and 45% were male. Twenty-two per cent of respondents were between the ages of 81 and 90 years. The next most common age groups were 71–80 years (20% of respondents) and 61–70 years (14% of respondents). Around 9% of patients were aged between 18 and 30 years and the same proportion were aged 41–50 years. Seven per cent of patients were aged between 31 and 40 years and 7% between 91 and 120 years. In total, 12% of respondents did not report their age.
Nearly all surveys (92.4%) were completed on tablets, with only around 8% of surveys completed at kiosks (7.6%).
After Francis preliminary analyses
Prior to the post-data collection analyses, the following preliminary analyses were conducted:
-
item frequencies
-
analysis of interview length
-
mode of administration
-
measurement properties of the questionnaire
-
assessment of data dimensionality
-
scale analysis
-
reducing the item pool
-
handling missing item responses in the scale
-
unit-level reliability
-
score variation by demographic groups
-
differential item functioning (DIF).
The complete analyses and corresponding results are detailed as follows.
Item frequencies provide basic information about questionnaire functioning, including completion or ‘skip’ rates, and potential differentiation issues through ‘floor’ or ‘ceiling’ effects.
A ‘point reached’ analysis (Table 14) indicated that just under 95% of respondents (n = 3713) proceeded to the end of the scored survey questions, also known as closed survey questions. Thirty-two respondents did not complete question 20 (Overall, do you feel you have been treated with respect and dignity while in hospital?). Other than that, there appeared to be no particular point at which responses ceased. Refer to Appendix 1 for the full list of survey questions.
| Last scored item completed | Frequency | Cumulative (%) |
|---|---|---|
| No scored items answered | 2 | 0.1 |
| Q1 | 3 | 0.1 |
| Q2 | 11 | 0.4 |
| Q3 | 19 | 0.9 |
| Q4 | 11 | 1.2 |
| Q5 | 12 | 1.5 |
| Q6 | 10 | 1.7 |
| Q7 | 10 | 2.0 |
| Q8 | 10 | 2.2 |
| Q9 | 12 | 2.5 |
| Q10 | 12 | 2.9 |
| Q11 | 14 | 3.2 |
| Q12 | 10 | 3.5 |
| Q13 | 9 | 3.7 |
| Q14 | 8 | 3.9 |
| Q15 | 7 | 4.1 |
| Q16 | 7 | 4.3 |
| Q17 | 8 | 4.5 |
| Q18 | 8 | 4.7 |
| Q19 | 32 | 5.5 |
| Q20 | 3713 | 100.0 |
| Total | 3928 |
All further analyses were restricted to the 3889 responses deemed ‘useable’ (i.e. having at least four valid responses to scored items).
Basic item frequencies for all scored items are presented in Appendix 2. The highest proportion of ‘skipped’ or missing responses was for question 17 (Has a member of staff told you about what danger signals regarding your condition or treatment to watch for?): 6.2% of cases. For all other questions, at least 95% of respondents recorded an answer.
For two questions, > 90% of respondents gave the same answer:
-
question 18 (During your time in hospital, have staff made you feel safe?) – 90.9% of respondents selected ‘yes, completely’
-
question 20 (Overall, do you feel you have been treated with respect and dignity while in hospital?) – 92.8% of respondents selected ‘yes, all of the time’.
These questions provided limited differentiation in the current sample.
The proportion giving a response of ‘do not know’ was generally low (< 2% of responses). However, for a number of questions, the proportion of respondents indicating that the question was not applicable to them (e.g. ‘I have not asked for help’) was high: these responses are not scored and, therefore, the contribution of these questions to evaluating provider performance is reduced.
Descriptive statistics for the scored items are presented in Table 15. Refer to Appendix 1 for the full list of question and answer options. Five items had very high mean scores (> 90): questions 3, 4, 10, 18 and 20. For all items, the mean score was < 1 standard deviation (SD) from the scale maximum (range 0.26–0.88 SDs).
| Survey item | Respondents (n) | Mean score | SD |
|---|---|---|---|
| Q1 | 3865 | 87.3 | 25.4 |
| Q2 | 3718 | 69.3 | 34.8 |
| Q3 | 3879 | 91.6 | 20.8 |
| Q4 | 3786 | 91.4 | 20.4 |
| Q5 | 3837 | 89.8 | 22.4 |
| Q6 | 3841 | 85.0 | 27.8 |
| Q7 | 2974 | 88.7 | 23.2 |
| Q8 | 2887 | 81.6 | 28.4 |
| Q9 | 1665 | 81.2 | 31.4 |
| Q10 | 3771 | 94.2 | 17.6 |
| Q11 | 3673 | 84.4 | 28.9 |
| Q12 | 2738 | 80.8 | 34.8 |
| Q13 | 3597 | 82.9 | 30.2 |
| Q14 | 3483 | 85.4 | 28.8 |
| Q15 | 3324 | 89.9 | 23.7 |
| Q16 | 2017 | 75.9 | 36.1 |
| Q17 | 2288 | 62.8 | 43.6 |
| Q18 | 3692 | 94.5 | 18.4 |
| Q19 | 3663 | 89.9 | 23.5 |
| Q20 | 3713 | 95.8 | 16.0 |
Interview length is an aspect of usability. Statistics for the time taken to complete the questionnaire and the mean time per question are shown in Table 16, both overall and split by mode of administration. These times are broken down in Table 17 by who completed the questionnaire. For these analyses, data were restricted to questionnaire sessions that started and finished on the same day, to avoid problems caused by data not being submitted directly on completion.
| Mode of administration | Time (minutes) | |
|---|---|---|
| Duration of interview | Mean time per question | |
| Tablet | ||
| Mean | 8.69 | 0.35 |
| n | 3575 | 3575 |
| SD | 9.85 | 0.40 |
| Median | 6.63 | 0.26 |
| Minimum | 0.35 | 0.04 |
| Maximum | 267.53 | 10.29 |
| Kiosk | ||
| Mean | 6.44 | 0.25 |
| n | 295 | 295 |
| SD | 9.56 | 0.37 |
| Median | 4.79 | 0.19 |
| Minimum | 1.08 | 0.04 |
| Maximum | 142.88 | 5.50 |
| All | ||
| Mean | 8.52 | 0.34 |
| n | 3870 | 3870 |
| SD | 9.84 | 0.40 |
| Median | 6.46 | 0.26 |
| Minimum | 0.35 | 0.04 |
| Maximum | 267.53 | 10.29 |
| Respondent type | Time (minutes) | |
|---|---|---|
| Duration of interview | Mean time per question | |
| The patient | ||
| Mean | 7.77 | 0.30 |
| n | 1552 | 1552 |
| SD | 10.69 | 0.42 |
| Median | 5.58 | 0.22 |
| Minimum | 1.08 | 0.04 |
| Maximum | 267.53 | 10.29 |
| A friend or relative of the patient | ||
| Mean | 5.76 | 0.23 |
| n | 304 | 304 |
| SD | 4.33 | 0.17 |
| Median | 4.70 | 0.18 |
| Minimum | 1.33 | 0.05 |
| Maximum | 55.80 | 2.15 |
| Both patient and friend/relative together | ||
| Mean | 8.28 | 0.33 |
| n | 133 | 133 |
| SD | 8.74 | 0.36 |
| Median | 6.90 | 0.27 |
| Minimum | 1.47 | 0.06 |
| Maximum | 92.55 | 3.70 |
| The patient and the volunteer together | ||
| Mean | 9.93 | 0.38 |
| n | 1716 | 1716 |
| SD | 9.97 | 0.39 |
| Median | 7.92 | 0.31 |
| Minimum | 0.93 | 0.04 |
| Maximum | 240.28 | 9.24 |
The survey was completed by 295 patients using kiosks and by 3575 patients using the tablets (see Table 16). Two case study sites started with kiosks and tablets and then switched to using tablets only as kiosks did not yield sufficient responses despite staff directing patients to them. Owing to the low number of responses obtained from the kiosks, further analyses comparing differences in responses by mode of administration were not conducted.
It was the intention of the study to use composite patient experience scores based on multiple questions to measure provider performance. In evaluating a composite score computed from data across a number of items, there are two questions that should be addressed:
-
Does the composite score measure (essentially) one aspect of experience?
-
How well does it measure that aspect of experience?
If a score does not relate to a single aspect of experience, the score cannot be interpreted unambiguously. This is the question of dimensionality, which is a necessary but not sufficient condition for valid score interpretation. The second point concerns how precisely the score measures that aspect of experience or, equivalently, the extent to which it is free of measurement error, and this is the question of reliability.
Both dimensionality and reliability are properties of the responses, rather than of the questionnaire itself, and are sample dependent.
The number of major constructs (aspects of experience) being measured using the questionnaire was investigated by examining the underlying dimensionality of the responses. This was carried out using exploratory factor analysis. 48 The analysis was intended to identify a parsimonious set of latent variables that account for as much of the variance among the individual question responses as possible.
Accurate factor analysis requires complete data, otherwise biases can be introduced. Because of the number of missing item scores (only 609 respondents completed every scored item), multiple imputation of the item scores was used to substitute plausible values for the missing scores. 40 Five data imputations were carried out using AmeliaView v1.7.4 (Honaker J, King G, Blackwell M, Harvard University, Cambridge, MA, USA). An inter-item polychoric correlation matrix was computed for each of the five complete data sets and the mean of the five matrices was used for factor analysis.
The correlation matrix was subject to two respected tests of dimensionality41 using Factor v10.3. The first test was parallel analysis, in which data eigenvalues are compared with those that would be obtained from random data (essentially a signal-to-noise analysis). The second was Velicer’s MAP test, which seeks to minimise the amount of residual variance after different numbers of factors are extracted. Both approaches were in agreement, indicating that the data were essentially unidimensional. Additionally, the first factor was found to account for > 50% of total item variance, almost 10 times that of the second factor, a further indicator of essential unidimensionality.
One factor was therefore extracted using unweighted least-squares factoring. Key analysis output is reported in Appendix 3. Model fit was good by some indices (GFI/AGFI > 0.95; RMSR < 0.05), but not according to others. Factor loadings were generally high, that is, all were > 0.4 and all but two were > 0.5.
Reliability coefficients indicate how well a score differentiates between cases that differ in whatever construct the score is measuring. Reliability can be measured at the respondent level (how well a score differentiates levels of experience of individual patients) or at the group level (how well a score differentiates between wards or other units in terms of the experience of their patients).
The loadings from the factor analysis were used to analyse the overall respondent-level reliability of a single scale derived from all the items and the contribution of the individual items to reliability. Reliability was evaluated using Cronbach’s alpha and McDonald’s omega (a more general coefficient, of which alpha is a special case when certain restrictive conditions apply). 42 Item contribution to reliability was evaluated using McDonald’s 42 item information index (the ratio of communality to uniqueness) (Table 18). The ratio increases as the item’s contribution increases. Refer to Appendix 1 for a full list of survey questions.
| Survey item | Factor loading on factor 1 | Communality | Uniquenessa | Item information42 |
|---|---|---|---|---|
| Q1 | 0.601 | 0.362 | 0.638 | 0.567 |
| Q2 | 0.626 | 0.392 | 0.608 | 0.645 |
| Q3 | 0.825 | 0.680 | 0.320 | 2.125 |
| Q4 | 0.813 | 0.661 | 0.339 | 1.950 |
| Q5 | 0.781 | 0.610 | 0.390 | 1.564 |
| Q6 | 0.770 | 0.593 | 0.407 | 1.457 |
| Q7 | 0.660 | 0.436 | 0.564 | 0.773 |
| Q8 | 0.642 | 0.412 | 0.588 | 0.701 |
| Q9 | 0.548 | 0.300 | 0.700 | 0.429 |
| Q10 | 0.771 | 0.594 | 0.406 | 1.463 |
| Q11 | 0.718 | 0.516 | 0.484 | 1.066 |
| Q12 | 0.458 | 0.210 | 0.790 | 0.266 |
| Q13 | 0.732 | 0.536 | 0.464 | 1.155 |
| Q14 | 0.727 | 0.528 | 0.472 | 1.119 |
| Q15 | 0.732 | 0.535 | 0.465 | 1.151 |
| Q16 | 0.547 | 0.300 | 0.700 | 0.429 |
| Q17 | 0.478 | 0.229 | 0.771 | 0.297 |
| Q18 | 0.811 | 0.658 | 0.342 | 1.924 |
| Q19 | 0.870 | 0.757 | 0.243 | 3.115 |
| Q20 | 0.856 | 0.733 | 0.267 | 2.745 |
The overall reliability was very high, indicating that different levels of individual experience could be distinguished with high precision. Two items with relatively low loadings (questions 12 and 17) did not contribute to reliability, and overall reliability could be improved by a very small increment if these were excluded from the scale. Question 17 was subsequently removed from the scale; however, question 12 was kept.
Taking into account the extent of missing or non-evaluative item data, and qualitative information from cognitive interviews and other feedback on use of the questions, it was decided to remove question 9 (Have staff responded quickly when you ask for help?), question 16 (Has a member of staff answered your questions in a way that you could easily understand?) and question 17 (If you have had any anxiety or fears about your condition or treatment, has a member of staff discussed them with you?) from contributing to the overall score.
The exploratory factor analysis was rerun using the reduced item set (see Appendix 4). The key output is given in Appendix 5 and factor loadings in Table 19. The results from the analysis were virtually identical to those from the earlier analysis incorporating all items. Refer to Appendix 1 for the full list of survey questions.
| Survey item | Factor loading on factor 1 | Communality | Uniquenessa | Item information42 |
|---|---|---|---|---|
| Q1 | 0.607 | 0.368 | 0.632 | 0.582 |
| Q2 | 0.626 | 0.392 | 0.608 | 0.645 |
| Q3 | 0.834 | 0.695 | 0.305 | 2.279 |
| Q4 | 0.821 | 0.673 | 0.327 | 2.058 |
| Q5 | 0.781 | 0.610 | 0.390 | 1.564 |
| Q6 | 0.773 | 0.597 | 0.403 | 1.481 |
| Q7 | 0.663 | 0.440 | 0.560 | 0.786 |
| Q8 | 0.642 | 0.412 | 0.588 | 0.701 |
| Q10 | 0.777 | 0.604 | 0.396 | 1.525 |
| Q11 | 0.713 | 0.508 | 0.492 | 1.033 |
| Q12 | 0.458 | 0.210 | 0.790 | 0.266 |
| Q13 | 0.726 | 0.528 | 0.472 | 1.119 |
| Q14 | 0.716 | 0.512 | 0.488 | 1.049 |
| Q15 | 0.725 | 0.526 | 0.474 | 1.110 |
| Q18 | 0.807 | 0.652 | 0.348 | 1.874 |
| Q19 | 0.869 | 0.756 | 0.244 | 3.098 |
| Q20 | 0.860 | 0.739 | 0.261 | 2.831 |
The retained items included some with a relatively large number of missing data, which presented potential problems in the computation of an overall score; only 1561 cases had complete data on the reduced item set. It was therefore decided to test a short form of the scale using only items with at least a 95% response and to compare this with the full scale.
Scale statistics are reported in Table 20. Figure 3 shows the relationship between the two forms for cases with complete data on both the long and short forms.
| Scale descriptive | Mean score | Number of cases with complete data | SD | SE of the mean |
|---|---|---|---|---|
| All items | 88.02 | 1561 | 15.91 | 0.40 |
| Short form | 88.73 | 1561 | 15.93 | 0.40 |
| Scale difference | Minimum | Maximum | Mean | SD |
| All: core items | –16.91 | 12.25 | –0.71 | 3.51 |
FIGURE 3.
Scatterplot of long- and short-form scores.
The differences between the two scales were assessed using a paired-samples t-test. The difference was statistically significant in raw score form [t = –8.017, degrees of freedom (df) = 1560; p < 0.0005], but not when both scales were standardised (t = 1.304, df = 1560; p = 0.192). Agreement between the scales was measured using the intraclass correlation coefficients for consistency (0.976) and exact agreement (0.975) and for ordinal agreement using Spearman’s ρ (0.945). 49
The level of consistency between the two forms was high and considered sufficient for overall scores to be based on the short form of the questionnaire. Respondent-level reliability for the short form was estimated from the factor loadings (see Table 19) as Cronbach’s α was 0.944 and McDonald’s ω was 0.945. Incorporating the scale data from Table 20, this gives a standard error (SE) of measurement of 3.74 and a 95% confidence band between 81.4 and 96.1 for a score at the sample mean. Refer to Appendix 1 for the full list of survey questions.
The item–total correlations for this scale are reported in Table 21.
| Survey item | Polyserial correlations |
|---|---|
| Q1 | 0.687 |
| Q2 | 0.706 |
| Q3 | 0.871 |
| Q4 | 0.866 |
| Q5 | 0.829 |
| Q6 | 0.794 |
| Q10 | 0.822 |
| Q11 | 0.775 |
| Q13 | 0.801 |
| Q18 | 0.876 |
| Q19 | 0.890 |
| Q20 | 0.907 |
Although the items selected for the overall score had high completion rates, it was necessary to determine a rule for how missing item responses are handled in score computation, in order to optimise data use. The options considered were to take the mean of the completed items (regardless of the extent of missing data); impose a minimum of six items completed; impose a minimum of eight items completed; replace missing responses with the survey mean; or replace missing responses with the ward mean. Each of these approaches involves some kind of imputation (either at the case level or the group level) and has attendant advantages and disadvantages. It was therefore necessary to balance minimisation of data loss and distortion of the score and its variance.
The relationships between these alternatives were explored using scatterplot matrices and intraclass correlations for exact agreement49 (see Appendix 6). The agreement measures were high for all comparisons, but the scatterplots showed that discrepancy at the individual level was large for some cases when comparing the imputation approaches with the most liberal approach to averaging the item scores. This discrepancy decreased as the tolerance of missing items was reduced. The eight-item minimum rule produced the highest correspondence with the imputation methods, making the process of averaging the survey items effectively redundant.
As the extent of data loss was small with this approach (about 3% of cases), the decision was made to adopt the rule of requiring scores to be present for at least 8 out of 12 items for these to be averaged in computing the overall score.
Reliability of scores at the unit (ward) level was estimated using the generalisability theory approach. 50 This relates to the capability of distinguishing differing levels of performance at the unit level, using scores on the short form calculated at the patient level and then aggregated to the ward level. Data for the 12 short-form items were entered in a three-level variance components model, with item responses nested within respondents within wards. 51 Variance estimates at each level were obtained using MLwiN v2.36 (MLwiN, Centre for Multilevel Modelling, Bristol, UK).
Generalisability theory follows the same classical ‘true score’ measurement model as for respondent-level reliability and seeks to estimate the proportion of overall variance that is ‘true’ variance between (in this case) wards. This affords both an estimate of unit-level reliability for the study (based on the harmonic mean numbers of respondents per ward and items per respondent) and also a minimum number of respondents required to achieve a threshold reliability of 0.80. The results of this analysis are reported in Table 22.
| Variance component | Estimatea | Number of units (harmonic mean) |
|---|---|---|
| Object of measurement (ward) | ||
| σ2(ward) | 26.960 | |
| Respondents within wards | ||
| σ2(respondent : ward) | 206.160 | 78.46 |
| Items within respondents | ||
| σ2(item : respondent : ward) | 368.656 | 11.67 |
| Intraclass correlation | 0.045 | |
| Unit-level reliability | 0.899 | |
| Minimum n for reliability = 0.80 | 36 | |
Variation in the overall score was investigated using a multilevel regression model with ward as a random effect and with indicators for the various demographic factors entered as main effects. This approach enabled all demographic variables to be taken into account when assessing the effects of each.
The variable groups entered were who completed the questionnaire; patient sex; patient age group; presence of long-term health condition(s); and ethnic group. Within each factor, there were groups whose score differed significantly from the reference group. In some groups – particularly those for ethnicity – numbers were very small and the coefficient estimates must therefore be regarded as tentative. Estimates for the fixed effects were obtained using MLwiN and are reported in full in Appendix 7. A chart of the score differences associated with each demographic factor is shown in Figure 4.
FIGURE 4.
Score differences associated with demographic factors.
Differential item functioning is concerned with score differences at the question level: whether or not certain groups respond differently to a particular question. This is examined by conditioning score differences between a ‘focal’ group and a reference group on their overall scale score. 52 Two types of DIF are possible: uniform DIF shows a consistent difference between focal and reference group responses; and non-uniform DIF describes a difference in one direction for high scores and a difference in the opposite direction for low scores.
The analysis was applied to two grouping variables that were of particular interest in use of the overall score. These were (1) the use of a proxy respondent and (2) white British compared with other ethnicities. All else being equal, and taking overall experience into account, we would hope to see no difference in the score allocated to a given item between the patient themselves and a friend or relative responding on their behalf, or between a patient of white British ethnicity and one of another ethnic group. For the purposes of this analysis, joint responses completed by a friend/relative or the volunteer in conjunction with the patient were excluded.
The method used followed closely that recommended by Zumbo. 52 This involves a sequence of three ordinal logistic regression models in which the response to a selected question is conditioned (1) on the overall scale score, (2) on the scale score and a group membership indicator and (3) on the scale score, group indicator and the interaction between these. An item is considered to show DIF if (a) there is a significant improvement in model fit between models 1 and 3 (p < 0.01) and (b) the magnitude of the effect is greater than a recommended threshold (> 0.13 increase in the pseudo-R2 index). The analyses were conducted using the PLUM procedure in IBM SPSS Statistics v23 (IBM Corporation, Armonk, NY, USA). Two pseudo-R2 indices were examined: the Nagelkerke and McFadden indices.
None of the questions showed evidence of substantive DIF according to these criteria, for either grouping variable. One question showed a statistically significant, but small, effect for both comparisons: question 2 (Have staff taken the opportunity to learn about you as a patient?). Estimated adjusted mean scores for this question, controlling for overall score, are given in Tables 23 and 24. Similarly, question 6 (During your time in hospital, have you had enough contact with staff?) showed a small but statistically significant effect for respondent type. The estimated mean scores are shown in Table 25. Additional results are presented in Appendices 8 and 9.
| Ethnicity | Score | ||
|---|---|---|---|
| Mean | SE | 95% CI | |
| White British | 69.347 | 0.462 | 68.442 to 70.252 |
| Other ethnicity | 74.125 | 2.348 | 69.522 to 78.727 |
| Respondent | Score | ||
|---|---|---|---|
| Mean | SE | 95% CI | |
| The patient | 68.105 | 0.666 | 66.799 to 69.412 |
| A friend or relative of the patient | 75.813 | 1.526 | 72.819 to 78.807 |
| Respondent | Score | ||
|---|---|---|---|
| Mean | SE | 95% CI | |
| The patient | 85.609 | 0.480 | 84.668 to 86.550 |
| A friend or relative of the patient | 81.670 | 1.093 | 79.527 to 83.814 |
After Francis results analysis, following the validation study
Following the preliminary analyses, an analysis of the impact of the intervention was conducted first at the whole-study level, then at the individual ward level. The main focus of the analysis was the overall score and the difference between scores at the baseline ‘pre-intervention’ period (weeks 1–18) and the ‘post-intervention’ period (weeks 21–43).
Figure 5 shows the trajectory of scores for the survey as a whole, calculated as the mean of the ward-level means. The source data are in Appendix 10. The chart also shows a 95% confidence interval (CI) for the null hypothesis of no overall change in scores for the 11 months. The general trend for the score is upwards, although mostly it remains within the CI.
FIGURE 5.
Trend in overall score for whole survey. LCL, lower confidence limit; UCL, upper confidence limit.
Results for the whole study were analysed using a multilevel regression model of the overall score, with ward as level 3, respondent as level 2 and item score as level 1 in the model. Modelling was done using MLwiN v2.36. Fixed effects were entered for items and for the results period (pre or post intervention). The relevant model terms are presented in Table 26 and show a small but statistically significant effect for the intervention term.
| Model term | Coefficient | SE | 2.5% CI | 97.5% CI | p-value |
|---|---|---|---|---|---|
| Intercept (pre intervention) | 85.141 | 1.187 | 82.815 | 87.468 | 0.000 |
| Post intervention | 1.186 | 0.588 | 0.034 | 2.337 | 0.044 |
| Ward-level variance | 25.613 | 8.598 | |||
| Respondent-level variance | 207.856 | 5.685 | |||
| Residual variance | 367.208 | 2.654 |
Ward-level results were analysed using t-tests of the score difference between the pre- and post-intervention periods (see Appendix 11).
The large majority of differences were positive: the post-intervention score was higher than the baseline. However, very few of these differences were statistically significant at a p-value of < 0.05. It should be noted that a number of the samples did not reach the recommended minimum of 36 responses required for unit-level reliability of 0.80.
Qualitative analyses
Overall, 1429 free-text comments were analysed from the patient experience survey. The greatest number of comments were provided by North Cumbria University Hospitals NHS Trust (n = 324), followed by Salisbury NHS Foundation Trust (n = 299), Poole Hospital NHS Foundation Trust (n = 245), Hinchingbrooke Healthcare NHS Foundation Trust (n = 238), Milton Keynes University Hospital NHS Foundation Trust (n = 171) and Northern Lincolnshire and Goole NHS Foundation Trust (n = 152).
Thematic analysis was used to code all 1429 comments. The following key themes were highlighted within patient comments.
Quality of care
Generally, patients who provided a comment reported that they experienced good-quality care. Specifically, feedback on care was related to the care provided by staff, rather than their overall experiences of care. Staff were mostly found to be helpful, compassionate, kind and caring. Many patients left a thank-you comment at the end of the survey, which reflected their positive experience of care:
I am very happy, we had a laugh last night.
Patient
The nursing staff have been amazing, kind, caring and thoughtful.
Patient
However, some patients fed back that the quality of care received was lower than expected. In particular, some patients felt that staff appeared unfriendly. Agency, night and reception staff were often perceived to provide a lower standard of care:
Reception staff were unfriendly, rude and dismissive.
Patient
There is a nurse who never smiles.
Patient
X-ray staff member very rude. 7-year-old scared to go in on her own as mum had a newborn and she huffed and puffed and told her to hurry as others waiting!
Patient
Poor care. Left in your urine and left with no dignity . . . Nurses not caring enough and don’t check on you enough.
Patient
One member of staff did not like me and I was scared of her.
Patient
Some patients also highlighted that their care staff seemed to lack knowledge, confidence and ability to perform their duties:
One nurse did not know what coeliac meant.
Patient
Staff did not really understand mental health. [I was] made to feel uncomfortable.
Patient
Not enough contact with doctors or nurses
At times, patients felt that they had too little contact with staff, especially with the doctor in charge of their care. However, patients also recognised that the nurses were often overworked and so could not always attend to their needs immediately:
I have not met the consultant named on the board by my bed and I have been here 2 months.
Patient
I haven’t seen a doctor for 3 days to discuss my health.
Patient
Not seen much of the doctors. Staff are very busy but come over as quick as they can.
Patient
The nurse who dealt with me was clearly not very interested in me as a person, seem to give me minimum contact.
Patient
Involved in decisions about care
Some patients reported that they did not feel as involved as they would like to be, if at all, in decisions regarding their care. In one case, a patient stated that family and doctors had seemingly made decisions without their involvement:
Don’t always feel involved as much as I would like.
Patient
Cross that family have apparently taken control of decisions about where she will live on discharge.
Patient, as reported by volunteer assisting with data collections
On the other hand, one patient did report that doctors had considered them to be an expert in their condition and complex health needs, so always consulted the patient in care decisions:
The doctors have been really good at treating me as an expert patient due to my complex health needs and have consulted me in regards to my treatment and what I would like to do.
Patient
Communication
Patients referencing communication varied in their feedback. Some frequently shared that the communication they experienced was not acceptable. Patients often felt that the information they were given regarding their health was not clear, or was seriously lacking in content, leaving them confused and without the opportunity to ask questions to gain a better understanding of their situation.
Some patients also reported witnessing discussions between staff members, which came across as unprofessional and made them uneasy:
Staff complain about being short-handed which is alarming for a patient.
Patient
Last night I was very upset by a female member of staff after my other [half] asked her at reception for a bottle for urine. She came to my bedside and said that I should be using the call button . . . She then told the nurse in charge of our care to look after her patients better in front of us. Very unprofessional.
Patient
Communication between departments was also erratic at times, causing problems and uncertainty for patients about their treatment, medication, dietary needs or care after discharge:
Mealtimes are quite erratic which is not good for a diabetic. There seemed to be a mix up and they did not know I was diabetic.
Patient
Contact with social services to enable timely discharge has been impossible.
Patient
The medicine I take at home wasn’t given to me correctly at the hospital for the duration of my stay.
Patient
On the other hand, some patients responded that levels of communication between members of staff, and between the staff and patient, were excellent. They reported being well informed about their condition and care, that family members were kept updated and that staff answered any questions they had.
Isolation
In their feedback, some patients spoke of feeling isolated during their stay in hospital. Patients described being left alone, frequently commenting that the nurses were too busy to spend time with them:
They don’t listen. No one talks to you. You sit all day. No one comes around.
Patient
However, this was not the case for all patients, as some fed back that their care staff were always happy to spend time with them and get to know them and took an interest in them as people, not patients:
We chat, we laugh. In a roundabout way you get to know them very well. I feel very relaxed about talking about ‘personal’ things without any blushing.
Patient
They were there for me morning, noon and night . . . never felt alone always someone to talk to.
Patient
Hygiene
Hygiene was also briefly mentioned in some feedback provided by patients. In particular, a few patients noted that student nurses did not always adhere to hygiene and infection control practices, and one patient stated that they had not received the personal hygiene care that they would have liked:
Had feet washed [today] for first time since admission.
Patient
Staffing levels
In their comments, several patients reported on the visible lack of staff during their hospital stay, stating that staff often look ‘busy’ or ‘rushed off their feet’. In particular, patients perceived that there were often not enough staff at night. Patients mostly recognised that staff were unable to spend time with them because they were overworked and understaffed:
Sometimes short staffed this is no problem to me but it does sometimes effect how much time they can spend with you.
Patient
Short staffed on here but they do their best.
Patient
Language barriers
Difficulties in understanding care staff as a result of language barriers were also cited by patients. Specifically, patients reported having difficulty understanding what was being said to them during conversations with foreign staff with unfamiliar accents:
Difficulty at times understanding some of the overseas staff.
Patient
Discharge
Discharge was also identified as an area of care with which some patients were dissatisfied. Reasons included being discharged with little to no notice or too much advance notice:
I was told I could go home 2 days ago.
Patient
Discharged with little notice which made the patient very anxious.
Patient
Call bells and waiting times
Comments received in the patient experience survey also described longer than expected waiting times in A&E, as well as on wards, once the call bell had been used. The wait on wards, at times, left patients in an undignified situation because they were unable to attend the bathroom. The wait in A&E – while understandably long – was found to cause some patients distress:
Having broken foot to the extent where my foot was black, in pain and crying, I was left for 7 hours before having my cast on, my treatment was unpleasant and cost me almost losing my foot.
Patient
Patient next to me wet the bed because staff took too long to help her at night.
Patient
Similarly, one patient highlighted that the check-in process at reception can be lengthy:
[On] one occasion, after pushing the buzzer for 15 minutes and getting no assistance I resorted to picking my mobile phone and calling the main hospital, asking to be connected to [ward].
Patient
However, although waiting times were frequently commented on, most patients understood that the long waiting time to be seen on wards was related to staff being busy and acknowledged that they would usually be seen when staff were available.
Food
Generally, feedback regarding the food provided during the hospital stay was positive, with statements suggesting that food was always served warm and that choices which catered to different dietary needs were offered.
On the other hand, some patients reported that the food should be of a better quality. For instance, it was sometimes described as tasteless, cold and often ‘stuck to the plate’. Similarly, patients described that they often did not receive regular meals or were completely missed on food rounds because they were in a side room:
Only meal I got offered was breakfast this morning. I arrived via ambulance at 14.00 yesterday.
Patient
Environment
Feedback describing the hospital environment was predominantly negative, with many patients stating that it was unclean and noisy. Similarly, some patients requested that equipment be installed (e.g. handrails in bathrooms) or stated that their beds were uncomfortable. In addition, some patients were not comfortable with being on a mixed-sex ward; some who wanted to be moved had their request denied because of a lack of beds on single-sex wards. Other concerns raised related to the lack of entertainment, such as radios, or the fact that the TVs were broken or too expensive to use. Other requests were simple, such as requests for air-fresheners, more wheelchairs and access to more pillows.
Comments regarding the survey
Feedback on the questionnaire was also given by some patients. Comments referred to some repetition within survey questions and that many questions should be split between nursing staff and doctors to make it easier to give an accurate response. One comment also suggested that the tablet was a little awkward to use. Positive feedback was also given on the survey, with patients praising it as an excellent tool to improve health care.
Participatory workshops
Findings from the first workshops
The first round of participatory workshops was attended by 8–15 participants at each site, with the exception of one case study site, where no participants attended the event because of lack of availability. The following sections detail the challenges experienced by the case study sites during the first 3 months of the patient experience data collection, as well as the areas for improvement identified in the action-planning activities.
Data collection challenges encountered
Staff at some sites explained that weekly results from patient experience data collection were not shared with nurses on the participating wards or departments. Some matrons found the weekly reports ‘too time-consuming’ to read or ‘too difficult to understand’ and did not cascade them down to their staff. In addition, ward staff were not given sufficient time to review the weekly reports as part of their regular duties. Hence, the case study sites requested a printable dashboard report page that could provide them with an overview of progress and allow them to quickly identify any areas that received lower scores.
At this time, staff were unaware if any changes or improvements had already been made based on the collected patient experience feedback. With the exception of staff from one case study site, all workshop participants found that the research project at their trust needed more management and leadership support in order to drive improvements. Patient experience boards, if available, should also be informed of the research and involved where possible.
The two case study sites using stationary kiosks in the A&E departments found that these did not generate a comparable number of data to those generated by the tablet-based approach, despite staff directing patients to use the equipment.
Some sites found that they did not have sufficient volunteers to continue with the data collection, as some of the initially trained volunteers stopped collecting data for the project. Volunteers stopped data collection because of personal preferences or being asked to move on to other tasks by the trust. Some volunteers also found it difficult to access the wards as a result of ‘protected times’, such as mealtimes.
Areas for improvement
Communication between staff and patients was a common theme for improvement identified by case study sites. Specifically, not all patients understood their conditions, treatment plans and medications, although these had been communicated to them by their doctors. Strategies or action points developed to improve communication and understanding included spot checks of consultants to gauge patients’ understanding, nurses restating or explaining again what consultants had previously described to a patient and consultants providing information to patients in writing. Staff introductions were also reintroduced as an area of focus to help patients feel more informed. Some sites planned a fortnightly coffee hour with the matron when families and patients could discuss any questions in a safe and supportive environment.
Management of patient expectations was also seen as an area for improvement for the case study sites. Specifically, staff wanted to make patients more aware of what they could expect from their hospital stay and how long it generally takes before call bells are answered.
In addition to improving staff communication with patients, staff across the sites identified a need for greater engagement of the participating ward/department staff with the project. Staff suggested that involvement of the trust leadership and the inclusion of key stakeholders, such as patient experience board members, could encourage staff to actively engage with the project and utilise the collected patient experience data. By including the project on a trust’s quality agenda, continued engagement and sustainable changes could be fostered.
Staff also planned to share the patient feedback with each other as part of regularly scheduled monthly ward meetings. Wards recognised the need to integrate the results from this research with areas for improvement identified through other feedback collected at the trust, such as the FFT, informal feedback and call bell response time audits.
Findings from the second workshops
A second round of participatory workshops was conducted after the data collection for the patient experience survey had been completed. Four of the six case sites were able to release a sufficient number of staff from their regular duties to attend a workshop. The discussions and findings from the workshops build on the experiences of staff and action plans for improvement derived during the first round of participatory workshops for staff.
The workshops were held on the following dates:
-
Hinchingbrooke Healthcare NHS Trust, 19 September 2016
-
North Cumbria University Hospitals NHS Trust, 29 September 2016
-
Northern Lincolnshire and Goole NHS Foundation Trust, 21 September 2016
-
Poole Hospital NHS Foundation Trust, 30 September 2016.
Discussions during the events aimed to gather feedback on the implementation of the patient experience survey. The sections below outline the thoughts shared at the events, which are organised into four sections to describe challenges faced, their resolutions, factors that made the NRTF approach a success and improvement efforts based on the weekly patient feedback.
Challenges faced
Staff at the case study sites reported that their teams experienced a variety of challenges related to data collection. First, it was difficult for the trusts to maintain volunteer engagement throughout the 10-month patient data collection period. Wards and A&E departments were busy areas and volunteers sometimes felt they were ‘in the way’. At times, staff were found to be ‘quite aggressive’ when approached to determine which patients could be surveyed. Volunteers could be seen as presenting an additional task during an ‘already busy time’.
Privacy, survey fatigue and resurveying patients were also found to be issues when collecting data, especially on the wards. Staff on wards often did not receive notice of when the volunteers were expected to come. Volunteers learnt that certain times of the day, such as mornings and mealtimes, were generally not suitable times to collect data.
The use of stationary kiosks in A&E departments was difficult, as patients rarely used the kiosks despite being directed to them by staff. Staff also found it challenging to maintain the functionality of the kiosks and to ensure that patients completed surveys at the end rather than at the beginning of their visit.
Staff also reported challenges affecting their use of the weekly patient experience feedback. In general, staff found the weekly reports to be difficult to navigate and understand as they provided a great level of detail. Staff preferred to see only data for their ward or department, otherwise ‘there was too much data to look at’. ‘Flat files’ were preferred over ‘interactive reporting documents’.
Despite their organisations’ receipt of weekly reports, staff reported a delay in getting feedback to wards. Lack of communication within the trusts themselves inhibited the ‘regular trickledown’ of results to the wards/departments involved. Therefore, results were no longer available to some areas in near real time.
Engagement of staff with the project on participating wards and departments developed over the course of the initial months. In retrospect, some staff reported an initial ‘suspicion that the project was there to criticise them’.
Addressing challenges
To address the challenges described above, staff derived strategies to ensure the success of the NRTF approach.
To address volunteer retention and engagement, staff recognised the need for continuous volunteer recruitment. As the data collection role is not suitable for all volunteers, the use of a job description or person specification was suggested to recruit specifically for this role. To maintain continuous engagement, ongoing volunteer support was key. However, the ‘right type of volunteer’ was needed for the data collection task.
Staff recognised that a substantial amount of patient experience feedback was being collected in each organisation, but the capacity to reflect on and use the data was lacking. Some staff suggested that, in retrospect, it might have been better to hold shorter weekly rapid staff briefings on each ward or department. NRTF results could be presented every week as part of these meetings so they can be used immediately. It was suggested that volunteers should be included in these presentations as they already understand the results through their interactions with patients during data collection. Staff also suggested sharing results by displaying posters showing an infographic on wards. One case study site decided to create a new post, ‘head of patient experience and clinical operations’, to provide additional capacity to review and use ongoing patient experience feedback.
Factors that make the near real-time feedback approach a success
Staff noted several factors that contribute towards the success of a NRTF approach to improving compassionate care. To reduce the burden of data collection for volunteers and patients, staff suggested the use of a shorter survey instrument. In addition, some suggested that each survey question should only have two instead of three answer options to facilitate recall.
To maintain volunteer engagement, recruitment efforts should target volunteers who wish to use technology and approach patients to collect patient feedback. This purposive recruitment is anticipated to contribute towards volunteer retention and provide greater benefits to volunteers.
The weekly reports should have an ‘at a glance’ section with graphs. Staff would like to see a single page with graphs and pictures that clearly highlight where improvements are needed.
Although the project was predominantly administered by R&D teams, retrospectively staff recognised that it should have been in the domain of the quality matrons and operational matrons to drive trust engagement. There should be involvement from everyone, at all levels of the trust. One staff member said, ‘We bought into the research as a trust but there should have been more done on a trust level, more board interaction, as opposed to having it purely in R&D’s domain’. Senior management also need to provide support, as the majority of staff did not have supporting professional activity time to review data, action plan and implement changes.
Improvement efforts based on weekly reports
In most trusts, staff introductions were revisited with the help of the ‘Hello, my name is . . .’ campaign. Similarly, staff were now more aware of the need to inform patients about their conditions on multiple occasions and took the time to give patients the information they needed.
To communicate staff availability, one trust began displaying staff numbers on electronic screens so that patients could see how many staff are currently available. Although this was not a relational aspect of care that we set out to capture in the survey, it was frequently mentioned in response to the open-ended question and staff addressed it nevertheless.
Staff often felt that patient expectations were not realistic; for example, patients often felt that the discharge process took too long. Therefore, patients were informed about the purpose of the discharge process to explain why it can be lengthy.
Based on responses to the patient experience survey, some trusts carried out an audit of call bell times and began placing a nurse in the bay at all times. However, they did not have the resources to fund this permanently, so they placed a sign visible to patients, stating the average time of a call bell response.
Staff survey
Tables 27–57 present the results of the staff surveys carried out pre and post patient experience data collection. As the survey administered post data collection had a reduced number of questions (to increase staff participation), for a number of questions only pre-patient data collection results are displayed. For tables showing a breakdown of results by study and control wards/departments for both pre and post data collection of staff surveys, see Appendix 12.
It is important to note that, for any question with multiple-choice answer options, the percentages are calculated separately for each answer option. Therefore, the percentage and number of responses for the question as a whole can exceed the total number of responses received.
A total of 71 respondents completed the staff survey pre patient data collection. The greatest proportion of responses were obtained from Salisbury NHS Foundation Trust (33.8%), followed by Hinchingbrooke Healthcare NHS Trust (28.2%). Only two respondents (2.8%) who participated in the survey were from Northern Lincolnshire and Goole NHS Foundation Trust (Table 27).
| Trust | Time point | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| Missing | 3.4 (6) | |
| Trust | ||
| Hinchingbrooke | 28.2 (20) | 23.0 (41) |
| Milton Keynes | 4.2 (3) | |
| North Cumbria | 14.1 (10) | 1.7 (3) |
| Northern Lincolnshire and Goole | 2.8 (2) | 29.2 (52) |
| Poole | 16.9 (12) | 28.1 (50) |
| Salisbury | 33.8 (24) | 14.6 (26) |
Of the 71 staff members who completed the survey pre data collection, 42 worked on the study wards and departments involved in the project and 29 worked on the control wards. The control wards were not involved with any other aspects of the After Francis research project.
A total of 178 respondents completed the survey post patient data collection. The largest proportion of responses were obtained from Northern Lincolnshire and Goole NHS Foundation Trust (29.2%), followed by Poole Hospital NHS Foundation Trust (28.1%). Only three respondents (1.7%) who participated in the survey were from North Cumbria University Hospitals NHS Trust and no responses were obtained from Milton Keynes University Hospital NHS Foundation Trust (see Table 27).
Post data collection, of the 178 staff who completed the survey, six staff members did not want to identify which ward they worked on, leaving a total of 172 responses. Of these, 113 staff were on the study wards and 59 were on the control wards.
Data collection at trust level
Types of patient experience data collected
The first questions in the staff survey focused on patient experience data collected at the trust level (Table 28). Staff were asked to indicate the types of patient experience data that had been collected, at their trust, during the current year.
| What types of patient experience data have you collected in your ward or department during the year 2015? Select all that apply | Response, % (n) |
|---|---|
| Total | 71 |
| No reply | 1.4 (1) |
| Response option | |
| National surveys (e.g. the national survey, the FFT) | 84.5 (60) |
| Surveys developed specifically for your hospital, ward or department | 56.3 (40) |
| Informal patient feedback (e.g. through conversations with staff or volunteers) | 60.6 (43) |
| Not sure | 4.2 (3) |
| Other, please specify | 1.4 (1) |
Pre data collection, staff most frequently reported the collection of patient experience data through national surveys (84.5%), followed by informal patient feedback (60.6%) and bespoke surveys developed exclusively for their ward, department or hospital (56.3%). Other patient experience feedback collected by the trust was highlighted:
Formal anonymised 360-degree feedback from a random sample of my patients for my revalidation.
Staff member
Methods used to collect patient experience data
Staff were asked about the methods used in their trusts to collect patient experience feedback (Table 29). The most common method was comment cards, reported by 88.7% of respondents, followed by conversations, reported by 50.7%. Only 4.2% of staff said that tablet-based surveys were used to collect patient experience feedback. Other methods used to collect patient feedback reported by staff were the use of tokens in boxes and a wipe board on which to record comments.
| What methods are used to collect patient experience data in your trust? Select all that apply | Response, % (n) |
|---|---|
| Total | 71 |
| No reply | 1.4 (1) |
| Response option | |
| Comment cards | 88.7 (63) |
| Conversations | 50.7 (36) |
| Focus groups | 18.3 (13) |
| Interviews | 12.7 (9) |
| Online surveys | 19.7 (14) |
| Paper surveys | 29.6 (21) |
| Tablet-based surveys | 4.2 (3) |
| Telephone surveys | 7.0 (5) |
| Not sure | 5.6 (4) |
| Other, please specify | 2.8 (2) |
Focus of patient experience data
Staff also specified that most patient experience data collected focused on recommendations to family and friends (87.3%), relationships with staff, such as being treated with kindness, dignity and respect (76.1%), and interactions with staff (69.0%) (Table 30).
| What parts of the patient experience have data collection at your trust mainly focused on? Select all that apply | Response, % (n) |
|---|---|
| Total | 71 |
| No reply | 1.4 (1) |
| Response option | |
| Cleanliness | 54.9 (39) |
| Interactions with staff | 69.0 (49) |
| Medications | 22.5 (16) |
| Pain relief | 23.9 (17) |
| Recommendations to family and friends | 87.3 (62) |
| Relationships with staff (e.g. being treated with kindness, dignity and respect) | 76.1 (54) |
| Waiting times | 22.5 (16) |
| Other, please specify | 8.5 (6) |
Additional aspects of the patient experience, captured by data collection, were specified:
All aspects of stroke – patients are invited to comment on everything and anything.
Staff member
Meals and nutrition.
Staff member
General comments, such as ‘what we can improve and what we did well’.
Staff member
Timing of data collection
Staff were asked at what points within the patient journey feedback is collected (Table 31). For the pre-patient data collection survey, one-third of staff reported that data are collected during the patient’s stay (33.8%) and 25.4% said that data are collected at all points of the patients’ journey. Only 2.8% of staff said that data are collected on a patient’s admission to hospital.
| At what point in the patient journey is most of your data collection carried out? | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 1.4 (1) | 3.9 (7) |
| Response option | ||
| On admission to the hospital | 2.8 (2) | 7.9 (14) |
| During the patient’s stay | 33.8 (24) | 35.4 (63) |
| After the patient has left hospital | 14.1 (10) | 17.4 (31) |
| At all points of the patient’s journey | 25.4 (18) | 23.6 (42) |
| Other, please specify | 22.5 (16) | 11.8 (21) |
The post-data collection survey showed similar results, with again around one-third of respondents saying that most patient experience data collection is carried out during the patient’s stay (35.4%). This was followed by 23.6% saying that collection are carried out at all points of the patient’s journey. Again, admission to the hospital was the least reported time point for data collection (7.9%).
Other data collection points reported by staff were at the point of discharge or shortly before discharge.
Data collection at the ward or department level
Types of patient experience data collected
Staff were asked to indicate the types of patient experience data collected on their ward or department during the past year (Table 32). Similar results were reported in both the pre- and post-data collection surveys.
| What types of patient experience data have been collected in your ward or department during the years 2015–16? Select all that apply | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 179 |
| No reply | 1.4 (1) | 5.1 (9) |
| Response option | ||
| National surveys (e.g. the national survey, the FFT) | 81.7 (58) | 62.4 (111) |
| Surveys developed specifically for your hospital, ward or department | 38.0 (27) | 32.6 (58) |
| Informal patient feedback (e.g. through conversations with staff or volunteers) | 52.1 (37) | 60.1 (107) |
| Other, please specify | 5.6 (4) | 6.2 (11) |
The majority of staff specified that national surveys were used (81.7%, pre; 62.4%, post), followed by informal patient feedback (52.1%, pre; 60.1%, post). Other types of patient data collection at the ward/department level were stated as:
Personal letters/cards.
Staff member
Suggestions box.
Staff member
Methods used to collect patient experience data
Staff were asked to describe the methods used to collect patient feedback during the current year. Similar responses were given pre and post data collection (Table 33). Comment cards were mentioned by the majority of staff (83.1%, pre; 88.2%, post), followed by conversations (49.3%, pre; 58.4%, post) and paper surveys (26.8%, pre; 32.6%, post). Pre data collection, the method reported by staff to be used the least was telephone surveys (5.6%). Post data collection, the method used the least was focus groups (3.4%).
| What methods were used to collect patient experience data in your ward or department during the year 2015? Select all that apply | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 1.4 (1) | |
| Response option | ||
| Comment cards | 83.1 (59) | 88.2 (157) |
| Conversations | 49.3 (35) | 58.4 (104) |
| Focus groups | 8.5 (6) | 3.4 (6) |
| Interviews | 12.7 (9) | 3.9 (7) |
| Online surveys | 14.1 (10) | 10.7 (19) |
| Paper surveys | 26.8 (19) | 32.6 (58) |
| Tablet-based surveys | 7.0 (5) | 13.5 (24) |
| Telephone surveys | 5.6 (4) | 3.9 (7) |
| Not sure | 3.4 (6) | |
| Other, please specify | 4.2 (3) | 4.5 (8) |
Visits by a dedicated patient experience team and the use of tokens were other methods identified by staff.
Focus of patient experience data
Pre data collection, staff reported that data collection most often focused on recommendations to family and friends (81.7%), followed by relationships with staff (76.1%) and interactions with staff (70.4%). Waiting times (19.7%) were mentioned the least. Post data collection, the majority of staff reported that data collection focused on interactions with staff (71.9%), followed by recommendations to family and friends (69.7%). Just over one-quarter of respondents said that medications were the main focus of surveys (29.2%) (Table 34).
| What parts of the patient experience have data collection in your ward or department mainly focused on? Select all that apply | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 1.4 (1) | 2.8 (5) |
| Response option | ||
| Cleanliness | 50.7 (36) | 46.1 (82) |
| Interactions with staff | 70.4 (50) | 71.9 (128) |
| Medications | 23.9 (17) | 29.2 (52) |
| Pain relief | 28.2 (20) | 33.1 (59) |
| Recommendations to family and friends | 81.7 (58) | 69.7 (124) |
| Relationships with staff (e.g. being treated with kindness, dignity and respect) | 76.1 (54) | 66.9 (119) |
| Waiting times | 19.7 (14) | 37.6 (67) |
| Other, please specify | 8.5 (6) | 11.8 (21) |
Other aspects of patient experiences captured by data collection at the ward or department level were specified as:
All aspects important to patients or generic experience.
Staff member
Meals and nutrition.
Staff member
Pre data collection, some differences were observed in the percentage of staff reporting a focus on waiting times between those working on the study (23.8%) and those working on the control (13.8%) wards/departments. Differences were also observed post data collection between the study and control wards/departments for waiting times (41.5% and 30.5%, respectively), recommendations to family and friends (72.6% and 61.0%, respectively) and relationships with staff (63.7% and 71.2%, respectively).
Communication of results
Staff were asked how results from the patient experience data collection were communicated in their ward or department (Table 35). Here, communication on staff noticeboards was mentioned most frequently (62.0%, pre; 59.0%, post), followed by communication during staff meetings (59.2%, pre; 53.4%, post). A complete lack of communication of results within the ward or department was reported least frequently (1.4%, pre; 4.5%, post).
| How are results from patient experience data collection communicated to staff in your ward or department? Select all that apply | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 2.8 (2) | 1.7 (3) |
| Response option | ||
| Communicated during staff meetings | 59.2 (42) | 53.4 (95) |
| On staff noticeboards | 62.0 (44) | 59.0 (105) |
| Online portal/trust intranet | 32.4 (23) | 20.2 (36) |
| Written report (electronic version) | 29.6 (21) | 11.8 (21) |
| Written report (hard copy) | 15.5 (11) | 9.0 (16) |
| They are not communicated | 1.4 (1) | 4.5 (8) |
| Not sure | 1.4 (1) | 12.4 (22) |
| Other, please specify | 9.9 (7) | 10.7 (19) |
The following other methods of communication were specified by staff:
Results are sent via e-mails to every staff on the ward.
Staff member
Communication folder.
Staff member
Newsletter.
Staff member
Staff were asked who communicates results from the patient experience data collection to them (Table 36). Staff most frequently reported communications from ward managers (77.5%, pre; 87.1%, post), followed by patient experience leads (21.1%, pre; 9.0%, post). Board members were reported as the staff group least likely to communicate patient experience data to staff (2.8% pre/post data collection). Other individuals communicating patient experience results were specified as:
Consultant.
Staff member
Department administrator or communications team.
Staff member
Matron or ward sister.
Staff member
Clinical director or operational lead.
Staff member
| Who communicates patient experiences results with you? Select all that apply | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 2.8 (2) | 3.9 (7) |
| Response option | ||
| Board members | 2.8 (2) | 2.8 (5) |
| CEO | 7.0 (5) | 3.4 (6) |
| Patient experience leads | 21.1 (15) | 9.0 (16) |
| Speciality leads | 12.7 (9) | 5.1 (9) |
| Ward managers | 77.5 (55) | 87.1 (155) |
| Other, please specify | 14.1 (10) | 9.0 (16) |
Similar responses were provided by staff working on study and control wards.
Staff were then asked to indicate how they would like to have the results of patient experience data collection communicated to them (Table 37). Most staff selected communications during staff meetings (57.7%, pre; 57.9%, post) and on staff notice boards (57.7%, pre; 51.7%, post). Only 1.4% (pre) and 2.2% (post) of staff reported that patient experience data were not communicated to them.
| How would you prefer to have the results of patient data collection communicated to you? Select all that apply | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 5.6 (4) | 2.2 (4) |
| Response option | ||
| Communicated during staff meetings | 57.7 (41) | 57.9 (103) |
| On staff noticeboards | 57.7 (41) | 51.7 (92) |
| Online portal/trust intranet | 29.6 (21) | 21.9 (39) |
| Written report (electronic copy) | 33.8 (24) | 21.9 (39) |
| Written report (hard copy) | 7.0 (5) | 11.2 (20) |
| They are not communicated | 1.4 (1) | 2.2 (4) |
| Not sure | 3.4 (6) | |
| Other, please specify | 4.2 (3) | 3.4 (6) |
Other preferred reporting formats included:
Via e-mail.
Staff member
[I] think this needs to be a more formal process where themes are pulled out, prioritised and actions derived.
Staff member
Pre data collection, some differences were noted for preferences for online portal or trust intranet communications (38.1% and 17.2%, respectively) by staff working on study and control wards. No differences were identified post data collection.
When asked about the frequency of communication, most staff indicated that results of patient experience data collection are communicated to them monthly (60.6%, pre; 23.6%, post) or weekly (22.5%, pre; 9.0%, post). Daily communication of results was not reported by any staff members in the pre data collection survey and by only 5.1% in the post data collection survey. Less than 2% of staff reported results being communicated annually (1.4%, pre; 1.7%, post) (Table 38).
| How frequently are results communicated with you? | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 4.2 (3) | 50.0 (89) |
| Response option | ||
| Annually | 1.4 (1) | 1.7 (3) |
| Quarterly | 7.0 (5) | 5.1 (9) |
| Monthly | 60.6 (43) | 23.6 (42) |
| Weekly | 22.5 (16) | 9.0 (16) |
| Daily | 5.1 (9) | |
| Other, please specify | 4.2 (3) | 10.1 (18) |
The following other frequencies of communication were specified:
As they come in.
Staff member
It varies when they are communicated.
Staff member
Pre data collection, some differences in frequencies of communication can be observed between the study and control ward staff for weekly communications (26.2% and 17.2%, respectively). Post data collection, differences were identified between selections for monthly (21.2% and 30.5%, respectively) and weekly (11.5% and 5.1%, respectively) communication.
Staff were then asked to indicate their preferred frequency for the communication of patient experience results (Table 39). Similar to current frequencies of communication, as reported in Table 38, staff preferred monthly communication (63.4%, pre; 53.9%, post), followed by weekly communication (25.4%, pre; 25.3%, post). Annual communication of results was preferred by the fewest respondents (1.4%, pre; 1.1%, post). Some differences between respondents from study and control wards can be observed for weekly data collection, which was preferred by 31.0% of staff from study wards pre data collection and by 17.2% of staff from the control wards pre data collection. Similarly, differences were observed in preferences for monthly data collection, which was desired by 52.2% of staff from study wards post data collection and 61.0 % of staff on the comparison wards post data collection. These differences may be attributable to the priming of staff on study wards. Specifically, staff on these wards may expect weekly reporting of results, as this frequency is provided by the current project.
| How frequently would you like to have results communicated with you? | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 1.4 (1) | 3.4 (6) |
| Response option | ||
| Annually | 1.4 (1) | 1.1 (2) |
| Quarterly | 5.6 (4) | 15.2 (27) |
| Monthly | 63.4 (45) | 53.9 (96) |
| Weekly | 25.4 (18) | 25.3 (45) |
| Daily | 2.8 (2) | 5.6 (10) |
| Other, please specify | 5.6 (10) | |
In addition, respondents indicated their preferred reporting format to facilitate understanding and use of the results presented (Table 40). Staff indicated that they would like to see reports that include example quotes detailing patients’ feedback in their own words (64.8%, pre; 47.8%, post). This was followed by preferences for written narrative (40.8%, pre; 42.1%, post) and charts and graphs (40.8%, pre; 29.2%, post).
| What report format would make it easier for you to understand and use patient experience findings? Select all that apply | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 4.2 (3) | 3.9 (7) |
| Response option | ||
| Tables | 16.9 (12) | 17.4 (31) |
| Charts and graphs | 40.8 (29) | 29.2 (52) |
| Written narrative | 40.8 (29) | 42.1 (75) |
| Example quotes of patients’ feedback provided in response to open-ended questions | 64.8 (46) | 47.8 (85) |
| Other | 2.2 (4) | |
Usefulness of patient experience data
Many staff indicated that they found patient experience data to be extremely useful (50.7%, pre; 40.4%, post) or very useful (25.4%, pre; 30.9%, post) for their work with patients. Only 1.4% (pre) and 1.1% (post) indicated that they found patient experience data to be not at all useful for their work (Table 41).
| How useful is patient experience data for your work with patients? | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 1.4 (1) | 2.8 (5) |
| Response option | ||
| Extremely useful | 50.7 (36) | 40.4 (72) |
| Very useful | 25.4 (18) | 30.9 (55) |
| Somewhat useful | 16.9 (12) | 20.8 (37) |
| Not very useful | 4.2 (3) | 3.9 (7) |
| Not at all useful | 1.4 (1) | 1.1 (2) |
To follow up on the perceived usefulness of patient experience data, staff were asked to explain their response selection provided for the previous question. Staff who considered the patient experience data to be extremely useful or very useful to their work provided explanations that can be grouped into three themes. Specifically, staff explained that patient experience data help them understand how patients and their families view their care:
Because it shows areas that need improvement to ensure that patients are getting a high level of care. Also it shows what patients feel is important to them.
Staff member
The patient perceives their care very differently and the impact we have as carers. It is good to get that perspective to change the way we think and nurse.
Staff member
Staff also explained that patient experience data help them identify areas for improvement and make changes to their services. Under this theme, improving care to meet higher standards was mentioned by several staff:
Informs us of areas for improvement and patient wishes – helps us identify areas of good practice.
Staff member
The only way we will improve the care given is to listen to our patients. This data will enable us to make changes where needed to keep our hospital at the top for great patient care.
Staff member
Staff who considered patient experience data to be extremely or very useful for their work also highlighted that patient feedback provided a source of motivation:
It can help us to improve standards as well as giving encouragement to staff who have been mentioned for doing a good job.
Staff member
Few staff members who considered patient experience data to be somewhat useful to their work explained their opinion. Some staff explained that patient feedback can help to make the patient’s stay more pleasant. However, one staff member felt that patient feedback was not necessary for their role, as they do not have a clinical-based patient interaction.
Other staff members highlighted challenges in collecting or interpreting patient feedback, such as awkwardness in handing out surveys:
I find it awkward . . . giving out feedback cards to patients. I feel that they will think that I only delivered good care because I want good feedback. I deliver good care regardless.
Staff member
Another member of staff questioned the content validity of the patient feedback data collection instrumentation, such as the FFT:
Patients tend to come to the nearest hospital in an emergency or perceived emergency. So asking if they would recommend the trust to family, etc., doesn’t really make sense because no one would travel miles to an emergency department in an emergency to attend one recommended by a family member or friend.
Staff member
Only one respondent, who considered patient experience data to be not very useful to their work, explained that data collection highlighted logistical issues beyond their power to change:
It mainly says that patients think there should be more staff which is not something we can change on the ward.
Staff member
The staff member who thought patient experience data to be not at all useful to their work explained that feedback may not include a sufficient level of detail to provide a meaningful basis for improvements:
[The] token system doesn’t give [an] opportunity to say what is good or bad about the dept [department]. Numbers are also very small.
Staff member
Using patient experience data to drive service improvements
In the first administration, staff were asked about their awareness of changes that have been implemented at their trust as a result of patient experience data collection (Table 42). In response to this question, staff were most likely to say that they were aware of changes implemented by themselves and colleagues (31.0%), with almost as many being aware of others who had implemented changes or not aware of any implemented changes (21.1% and 21.1%, respectively).
| Are you aware of any changes that have been implemented in your trust as a result of patient experience data? | Response, % (n) |
|---|---|
| Total | 71 |
| No reply | 1.4 (1) |
| Response option | |
| Yes, I have implemented changes | 11.3 (8) |
| Yes, others have implemented changes | 21.1 (15) |
| Yes, myself and others have implemented changes | 31.0 (22) |
| No | 21.1 (15) |
| Not sure | 14.1 (10) |
To follow up on the previous question, staff were asked to identify the types of changes made at their trust as a result of patient experience data collection (Table 43). Again, this was only asked pre patient experience survey. Here, staff most frequently listed changes to the way care is provided to patients (73.9%) and changes to the way staff interact with patients (69.6%). Both categories can encompass relational aspects of care. Staff least frequently mentioned changes to the patient survey instrument (4.3%) as a result of patient experience feedback. Other changes mentioned by staff were:
Changes to how the ward runs.
Staff member
Signage to remind staff of care essentials, such as care bells.
Staff member
Waiting times for medications.
Staff member
| What changes have been made based on the results from patient experience data collection? Select all that apply | Response, % (n) |
|---|---|
| Total | 46 |
| No reply | 2.2 (1) |
| Response option | |
| Changes to the way staff interact with patients | 69.6 (32) |
| Changes to the way staff interact with colleagues | 34.8 (16) |
| Changes to the way care is provided to patients | 73.9 (34) |
| Changes to the layout of the hospital, ward or department | 32.6 (15) |
| Changes to patient survey | 4.3 (2) |
| Other, please specify | 8.7 (4) |
Some differences can be noted between responses from staff on study and responses from those on control wards for changes to the way staff interact with colleagues (39.3% and 27.8%, respectively) and physical changes to the layout of the hospital, ward or department (28.6% and 38.9%, respectively).
Staff were also prompted to specify whether or not any changes have been made on their ward or department as a result of patient experience feedback (Table 44). Staff most frequently selected that they themselves and their colleagues have implemented changes at the ward level (32.4%, pre; 25.8%, post), followed by responses stating that the changes were made by other staff (23.9%, pre; 19.1%, post). Only 14.1% of staff were not aware of any changes made pre patient data collection, compared with 41.0% of staff post data collection.
| Are you aware of any changes that have been implemented in your ward as a result of patient experience data? | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 1.4 (1) | 2.8 (5) |
| Response option | ||
| Yes, I have implemented changes | 9.9 (7) | 11.2 (20) |
| Yes, others have implemented changes | 23.9 (17) | 19.1 (34) |
| Yes, myself and colleagues have implemented changes | 32.4 (23) | 25.8 (46) |
| No | 14.1 (10) | 41.0 (73) |
| Not sure | 18.3 (13) | |
Next, staff described the types of changes made on wards/departments based on results of patient experience data collection (Table 45). The majority of staff listed changes to the way staff interact with patients (75.0%, pre; 61.0%, post), followed by responses listing changes to the way care is provided to patients (72.9%, pre; 58.1%, post). Changes to the patient survey instrument were mentioned least frequently (4.2%, pre; 3.7%, post). Other changes listed by staff included enforcing staff introductions, reducing noise at night, discussing medication changes with the patient and allowing for added, visible information to increase positive communication and reduce anxiety:
| What changes have been made based on the results from patient experience data collection? Select all that apply | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 48 | 105 |
| No reply | 4.2 (2) | 7.6 (8) |
| Response option | ||
| Changes to the way staff interact with patients | 75.0 (36) | 61.0 (64) |
| Changes to the way staff interact with colleagues | 27.1 (13) | 30.5 (32) |
| Changes to the way care is provided to patients | 72.9 (35) | 58.1 (61) |
| Changes to the layout of the hospital, ward or department | 22.9 (11) | 25.7 (27) |
| Changes to patient survey | 4.2 (2) | 3.8 (4) |
| Other, please specify | 6.3 (3) | 4.8 (5) |
Folders at the end of beds to provide extra stroke info, contact numbers, etc.
Staff member
We now routinely leave a sign by the bed when a patient leaves the ward to reduce anxiety if a relative visits.
Staff member
Differences in the responses provided by staff on study and control wards (pre survey) were observed for changes to the way staff interact with colleagues (35.7% and 15.0%, respectively). Post data collection differences for study and control wards were also observed in staff interactions with patients (64.2% and 51.5%, respectively) and changes to the layout of the ward/department (25.4% and 18.2%, respectively).
Barriers that affect the collection and use of patient experience data
When asked to give examples of the barriers encountered when collecting and using patient experience feedback, staff reported that the most frequent barrier was the lack of staff time to administer surveys (43.7%, pre; 49.4%, post) (Table 46).
| What are the barriers, if any, that affect your collection or use of patient experience data? Select all that apply | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 1.4 (1) | 7.3 (13) |
| Response option | ||
| Cost | 9.9 (7) | 13.5 (24) |
| Lack of staff interest | 19.7 (14) | 18.5 (33) |
| Lack of staff time to administer survey | 43.7 (31) | 49.4 (88) |
| Lack of volunteer workforce | 18.3 (13) | 18.0 (32) |
| Language barriers | 9.9 (7) | 10.7 (19) |
| Not enough importance placed on patient experience | 11.3 (8) | 12.9 (23) |
| Patients are receiving multiple surveys already | 23.9 (17) | 7.9 (14) |
| No barriers affect the collection or use of patient experience data | 25.4 (18) | 15.7 (28) |
| Other, please specify | 16.9 (12) | 11.2 (20) |
Differences between study and control wards/departments were identified pre data collection. These differences were highlighted in cost (14.3% and 3.4%, respectively) and lack of staff interest (23.8% and 13.8%, respectively). Post data collection differences were also seen in lack of staff interest (20.4% and 11.9%, respectively), language barriers (13.3% and 6.8%, respectively) and patients receiving multiple surveys (8.0% and 1.7%, respectively).
Other barriers encountered include patients not wanting to complete forms and staff sometimes being so busy they forget to hand patients a survey card.
Resolution of barriers that affect the collection and use of patient experience data
To understand how trusts could overcome the barriers listed in Barriers that affect the collection and use of patient experience data, staff were asked their opinion on how they think barriers could be resolved. At the beginning of the research, suggestions focused primarily on the use of volunteers and technology to reduce the burden of data collection for staff. Towards the end of the research, suggestions provided more detail around maintaining staff and volunteer engagement throughout the research. The suggested solutions focused on using a dedicated volunteer workforce, increasing staff buy-in through encouragement or reward and increasing staff knowledge of patient experience through training. One strategy to maintain staff engagement would be to make more use of open-ended questions, because of the insights they provide to staff. The meaningful insights gained from patient comments are described in the quote that follows:
. . . more meaningful and allow for conversation rather than question and answer.
Staff member
Promoters or enablers that support the collection and use of patient experience data on a ward or department level
Similarly, staff were asked to report any factors that they have found to be conducive to the collection and use of patient experience data. Similar promoters were shared during the first and second administrations of the staff survey. These related to having a designated team of volunteers to collect patient experience feedback:
Volunteers on the ward to ensure the cards are completed.
Staff member
Likewise, having a dedicated team of staff to engage in action-planning and use the patient feedback was considered to be an enabling factor for participating wards. Staff also highlighted that greater visibility of how patient feedback is used in turn enables continued use of patient feedback.
Improving the collection and use of patient experience data at a ward or department level
Staff were asked to describe in their own words how the collection and use of patient experience data could be improved at a ward or department level. Increased resources, such as ‘more time or people’ dedicated to data collection, were thought to be key to the use of patient experience data. This allows for continuity of the efforts if ‘the main responsible person is away’. In addition, communication of results, as well as senior staff buy-in and morale, were equally important. These suggestions were consistent across the two administrations of the staff survey.
Using near real-time feedback
Staff were asked about their awareness of NRTF data collection on their ward or department. In response to this question, 49.3% pre and 32.6% post data collection indicated that they or their colleagues have used NRTF methods to collect patient experience data in their ward or department (Table 47).
| In the past, have you used NRTF to collect patient experience data in your ward or department? | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 2.8 (2) | 18.0 (32) |
| Response option | ||
| Yes | 49.3 (35) | 32.6 (58) |
| No | 26.8 (19) | 9.6 (17) |
| Not sure | 21.1 (15) | 39.9 (71) |
To follow up on the previous question, staff were asked about their experiences with, and perceptions of, NRTF. Feedback shared in response to these questions was similar for the first and second administrations of the staff survey and highlighted the benefits as well as challenges associated with the approach.
Staff were asked about their reasons for using or not using NRTF. Those who already used NRTF explained that it allowed them to act or make changes based on the data more quickly. In addition, staff explained that it provided them with a greater level of specificity, particularly as patient experiences are still fresh in patients’ minds at the point of data collection:
It pinpoints the exact thoughts, feelings and emotions of the patient/visitor at the time of their visit and helps us understand how they are experiencing their time in our department. Time for reflection is good, but I feel surveying in [near] real-time gives us more of a raw opinion.
Staff member
Staff who did not use NRTF explained that this was as a result of the lack of awareness of this method or opportunities and resources to collect data in this way. A lack of time and a lack of a volunteer workforce were also reported as reasons for not using NRTF in the past.
Staff who had used NRTF in the past were asked to describe their experiences with this approach. Positive experiences provided in response to this question echoed the usefulness of this type of feedback as a way of implementing quick actions and fast improvements. Staff also reported that ‘Patients were honest and are pleased to be given the opportunity to have their say’. At the same time, the approach was found to ‘boost staff morale’ by keeping patient experiences at the forefront of their work:
Feedback from your service users can help drive change and improve our service as quickly as possible.
Staff member
Positive responses, any concerns are dealt with immediately.
Staff member
However, staff who have used NRTF previously highlighted some challenges associated with its use, specifically the time required to collect and analyse NRTF. Some staff described an occasional ‘lack of patient interest or capacity to complete surveys’.
Staff who worked on the wards involved with the After Francis project were then asked to describe any concerns or thoughts they have about the introduction of NRTF on their wards or departments. Most staff who provided an answer to this question stated that they had ‘no concerns’ about NRTF. However, a few staff explained that the time required by staff to implement NRTF collection and carry out analyses presented an area for concern:
. . . not enough staff for patient care so this might take staff away from their duties.
Staff member
. . . lack of time, especially when we have acute patients.
Staff member
Other concerns described by staff included language barriers among staff and patients, the effects of mental states of patients on survey results and vandalism of the stationary kiosks.
Suggestions of lessons learnt or any new ideas to consider when using NRTF were also given by staff. It was indicated that it was ‘helpful to have someone assist patients when completing their surveys’. In addition, volunteers ‘encouraged patients to voice their concerns while in hospital’. Other suggestions by staff were to remove the patient data collection from the responsibility of clinical staff.
Using volunteers to collect data
Pre data collection, most staff indicated that they have used volunteers to collect patient experience feedback on their ward or department (39.4%). However, post data collection, most staff were not sure if they had used volunteers to collect data previously (34.3%).
Pre data collection, differences in responses for staff from study and control wards can be noted for those using volunteers (45.2% and 31.0%, respectively) and those unsure about the use of volunteers (21.4% and 41.4%, respectively). Post data collection differences can also be found for those using volunteers (38.9% and 20.3%, respectively) (Table 48).
| In the past, have you used volunteers to collect patient experience data in your ward or department? | Time point, response % (n) | |
|---|---|---|
| Pre data collection | Post data collection | |
| Total | 71 | 178 |
| No reply | 1.4 (1) | 21.3 (38) |
| Response option | ||
| Yes | 39.4 (28) | 33.1 (59) |
| No | 29.6 (21) | 11.2 (20) |
| Not sure | 29.6 (21) | 34.3 (61) |
Next, staff were asked about their reasons for using or not using volunteers. Those who had used volunteers explained that they were useful in that they can provide time to patients, which is something that nursing staff are unable to do on many occasions as a result of time constraints. The volunteers also alleviate staff pressures by being able to spend time with the patients, which in turn allows staff to carry out more tasks. Staff also highlighted that volunteers are, generally, more available to and flexible with the patients. Other staff explained that patients and their families feel they can be more honest with volunteers about their experiences, as they do not provide care to patients. Volunteers were seen as ‘neutral’.
Another member of staff explained that volunteers can relate to patients if they have experienced similar health issues and so can build rapport with the patients:
[We] used stroke survivors as I thought they would understand how patients with stroke feel.
Staff member
Finally, some staff explained that they did not have a specific reason for using volunteers to collect data; instead, the volunteers ‘just appear on the wards and do work’.
Some staff who had not used volunteers for data collection explained that they were a ‘limited or non-existent resource’ in the hospital.
Staff who had used volunteers were asked to suggest any lessons learnt or new ideas that should be considered when working with volunteers. Suggestions focused on different ways of using volunteers in order to collect patient experience data:
Volunteers can be used to collect and give out data in the future. This saves staff a job.
Staff member
Organised hours that staff will be expecting them.
Staff member
Finally, staff who worked on the wards/departments involved with the After Francis project were asked to describe any concerns or thoughts about the introduction of volunteers as part of this project. The majority of comments reflected that staff had no concerns over using volunteers to collect patient experience data and that it was a good idea to use them. Volunteer safety on wards and the A&E departments, confidentiality of responses and volunteers’ knowledge or skills to interact with patients were raised by a few staff members.
Networking event for case study staff
Two members of staff each from five of the study sites attended the networking event. The team from North Cumbria University Hospitals NHS Trust was unable to attend the meeting.
Key points from discussions
All trusts experienced some volunteer recruitment difficulties and their retention was seen as a barrier towards data collection.
The study benefited greatly from senior leadership and staff involvement. Getting the right people involved was considered important. Senior leaders were seen as being able to ‘unblock issues’ that affect data collection or the use of results. Matrons can be gatekeepers and affect the success of the study.
A flat and concise one-page summary was requested as an addition to the weekly reports.
Ward teams and trusts were engaged in multiple action-planning activities throughout the year. The challenge would be to consolidate the plans arising from the results of this project with findings from other data collection.
Some impacts could already be observed from the project. For example, English-language skills training was put in place and calling cards were created that provide treatment and care details in writing for patients.
Experiences of data collection
Case study sites need ‘the right type of volunteer’ for data collection. This means that ‘purposeful volunteer recruitment’ should be in place. Not all volunteers are comfortable collecting data from patients, and some ‘see it as spying’. Volunteers need to be ‘robust and not take things personally’ when patients do not want to participate or staff question them. To facilitate volunteer recruitment, some trusts have compiled a set of desirable volunteer characteristics and provided these to the volunteer co-ordinator. This was similar to a person specification assembled for recruitment purposes. The role must also be ‘sold to volunteers’ so that they develop an interest in the project.
Volunteers were seen as ‘a fragile resource’, who can be unavailable during certain times, such as the winter cold and flu seasons. New volunteer recruitment is not quick enough to replace volunteers when they become unavailable. A back-up option needs to be available to maintain ongoing data collection.
Teams observed a mix of staff engagement in the wards/departments. Some volunteers found it easier to collect data on wards because ward managers know that they are there and give them a list of patients to talk to. Others found it easier to collect data in A&E because volunteers already collect FFT data there. Ward managers were, at times, seen as more concerned with patient capacity and allowed only certain patients to be approached. Some volunteers thought that staff were too restrictive and may inadvertently ‘cherry pick’ ward patients for participation. That is, staff may be unnecessarily cautious in excluding patients who they believe are too ill or lack capacity; some volunteers felt that a significant proportion of excluded patients would be able to complete a survey if asked.
Experiences of near real-time feedback reporting
Free-text comments were seen by staff as ‘very valuable’ and ‘really powerful’. In the weekly reports, staff said that they wanted to see successes and challenges highlighted in a way that was ‘quick and punchy’. Furthermore, when circulating results, it would be helpful to put the weekly results into the body of the e-mail, rather than in an attachment.
Sharing results with patients was found to be challenging by staff. As the collected feedback is anonymous, staff cannot go back to patients and tell them what they have done to address their feedback.
Many trusts have multiple action plans that have various degrees of overlap. The challenge is to find the gaps in the plans and to consolidate them to make them more actionable.
Feedback on analysis and interpretation of weekly results
In some trusts, the head of nursing sets the standards for improvement (e.g. ‘anything below “yes, always” is not good enough’). Free-text comments were seen as useful and provide details on staffing levels. Ward leaders liked the free-text comments as they could keep an eye out and scan for similar comments over time.
Some case study sites felt that it would be useful to see results from other trusts for benchmarking between trusts. This would facilitate a collaborative exchange to learn strategies from each other (i.e. how to address call bell response times).
Observed impacts of changes in practice
Communication and language barriers were addressed by providing additional training in communicating in English to incoming foreign staff. Following training, responses to free-text comments were monitored for further references to communication challenges and language barriers. Communication was also improved by introducing ‘calling cards’, which recorded in writing any communications with staff. Patients were also encouraged to review them with their families or carers and ask staff questions. At some sites, efforts focused on improving communication through consistent introductions by all staff. The ‘Hello, my name is . . .’ campaign was brought to the forefront again and staff committed to providing job roles to patients along with names.
Although the survey focused on relational aspects of care, some transactional aspects of care could be identified in the free-text comments.
Barriers towards changes in practice
Staffing levels were seen to be a barrier as higher demands leave less time for staff to reflect and implement improvements. In addition, a high level of reliance on agency staff was associated with a lack of continuity with regard to improvement action implementation.
Lessons learnt about near real-time feedback
Staff found that it could take up to 10 days before the departments can access results if they are not shared within the trust immediately. In addition, the kiosks in the A&E departments have been challenging for staff because of technical issues. The survey software needs to be streamlined to be effective and make sure settings in the tablet do not change. Also, very few patients actually used the kiosks despite being directed to them by staff. Several factors influenced patients’ decisions not to use the kiosks to complete a survey; these were based on their recent experiences of shock, pain or trauma, as well as a strong desire to return home following their visit.
Lessons learnt when working with volunteers
Staff discussed the following lessons learnt about working with volunteers, in relation to their experiences as part of the After Francis project:
-
Volunteers need lots of support (especially younger ones); support can focus on being assertive and being ready to explain the purpose of the project.
-
It can take a long time (approximately 3–4 months) to get volunteers into position.
Meetings with key staff at case study sites
Progress and impact
The sites found that their teams made progress with the patient experience data collection and encountered fewer challenges. However, recruiting patients was still seen to be challenging at times for volunteers and it was observed that ‘elderly patients could be reluctant to criticise care’. Data collection in A&E was generally seen as ‘more challenging for volunteers’ than that on the wards.
Some site teams felt that clinical staff should become more engaged with the project and begin to use the data collected. To facilitate use of the patient feedback, some trusts reported great benefits by collaborating with a matron, senior sister or director of nursing to drive staff engagement.
Some changes to relational aspects of care have already been implemented. These include increased call bell response time monitoring, a greater focus on staff introductions and leaving patients with written information about their treatment and care. At some case study sites, foreign nurses now complete a 6-week English course as part of the recruitment policy, to strengthen their communication skills with patients.
Lessons learnt
Patients appreciated the opportunity to share their experiences, which has improved some relationships between staff and patients, as patients are generally happier with their care. Staff found the free-text comments particularly ‘very useful and powerful’.
To maintain a consistent effort for the patient experience data collection, specific volunteer personalities and skills are needed for engaging with patients. At times, volunteers have to ‘cope with some form of abuse in the A&E departments’, as patients are in pain and worried and staff have very limited time and physical space. A person/job specification was a useful tool to recruit the desired types of volunteers for the study. A team of volunteers solely dedicated to the project would be beneficial because of the time requirement associated with data collection, changing volunteer schedules and unanticipated absences.
Support calls
During the first call in October, volunteers from the case study sites did not utilise this opportunity for support. One volunteer from Poole Hospital and one volunteer from Milton Keynes University Hospital participated in the support calls with the researcher during the second opportunity in November. For the third opportunity in December, three research nurses from Scunthorpe General Hospital participated in the support call. The following information was shared by the volunteers, which showcases their involvement with the project and the types of challenges that were encountered as part of their work.
In the second call opportunity, both volunteers described some challenges in retaining other volunteers to collect data for the project. They reported that some volunteers realised shortly after their training that it was not a good fit for them to collect the data using a tablet. However, other volunteers truly enjoyed being able to interact with patients and collect their feedback. Both volunteers stated that their trusts would continue to recruit volunteers.
One volunteer explained that they wanted to take on a more active role in providing support to new volunteers, in an effort to retain them with the project. They were to discuss this possibility with the project team at the trust.
Both volunteers stated that their team did not consistently meet the minimum targets of 20 completed surveys per month per ward and 50 completed surveys per month in the A&E department. They felt personally responsible for ensuring that the minimum targets were met, but expressed that they ‘could only collect so many responses in a day’ on their own.
One volunteer further said that the time available for data collection was severely limited because of ward rounds, mealtimes and visiting times. These events left the volunteer with a 2-hour window during which data could be collected on study wards. During this time, the volunteer would typically receive six or seven completed questionnaires from a ward.
The research nurses who rang up on the third opportunity wanted to discuss how they could better retain the volunteers who were assisting with data collection. The possibility of providing the volunteers with certification of research involvement was raised, as the volunteers at the trust were students looking to establish a career within health care. It was agreed that this would help ensure that the volunteers remain engaged with the research, as it evidences their participation in research and would aid them in their career development.
Telephone interviews with staff
Patient experience data collection
Staff were asked about the types of patient experience data collection methods used within the trust. Although most staff were able to comment on the different methods used to collect patient experience feedback and the frequency with which it is collected, other staff members were not able to provide a detailed account of the collection methods.
A total of 20 different types of patient experience feedback were reported to be used by the staff at the six case study trusts. Table 49 represents the types and frequencies of patient experience data collection.
| Type of patient experience data | Frequency of data collection |
|---|---|
| Audits and observations | Monthly |
| Catering feedback cards | Not specified |
| Clinical trials | Ad hoc |
| Comment cards | Continuous |
| Community and PPI representatives | Not specified |
| Feedback during conversation | Continuous |
| Focus groups | Monthly |
| FFT | On discharge |
| Governor meetings | Not specified |
| In-trust research | Monthly |
| Mystery shoppers | Not specified |
| National surveys | Annually |
Feedback reported and communicated within the trust
When asked about the methods of reporting and communicating patient experience feedback within the trust, the staff identified a number of communications or reports and the frequencies with which they are used (Table 50).
| Methods of feedback reporting and communication | Frequency of reporting |
|---|---|
| Annual report | Annually |
| Bulletin boards (you said we did) | Not specified/monthly |
| Chief executive e-mail | Weekly/fortnightly |
| E-mail bulletin | Daily/weekly/monthly |
| Feedback database | Ad hoc |
| Newsletter | Weekly/monthly |
| Online Huddleboard | Monthly |
| Performance reviews | Monthly/quarterly |
| Posters | Every 30 responses/quarterly |
| Quality accounts (online FFT, PALS, etc., results) | Monthly |
| Quote sheet on ward round | At every ward round |
| Reports/feedback at ward level | Monthly |
| Roadshow (trust-wide discussion) | Monthly |
| Senior management meetings | Monthly/quarterly |
| Team meetings | Monthly |
| Trust intranet | Fortnightly/monthly/quarterly |
| Trust-wide formal reports | 6-monthly/annually |
| Ward/staff meetings | Every few weeks/monthly |
Service improvements
Factors that promote improvements based on patient experience feedback
Factors that help improvement as discussed by staff are listed below.
Staff identified that staff and patients having the belief that they can change things goes a long way to helping improvement:
I think making them [staff] see the sense of it, rather than it just being a sort of tick-box exercise, that it’s actually valuable, the patients find it valuable and if staff can see the value in it, then it helps to get buy-in.
Staff member
Changing the staff perspective to limit negativity was also found to be useful when making change:
I don’t call it as ‘this is what we did right, this is what we do wrong’, I say ‘this is our successes, these are our challenges’.
Staff member
Communication was also seen as a helpful factor to implement change. By having clear communication either verbally or on written displays, patients and staff are more aware about what has been done and what is being done to make the trust services better.
Incentives are also seen to be useful by staff, to help boost staff morale:
We have something called the golden rules within the trust, which each letter of golden rules stands for something and if staff are named more than three times in a month, they get a golden rules award which is given to them personally. If patients are giving their feedback then we need to utilise it to encourage staff as well because there’s nothing better than somebody saying that you did a really good job.
Staff member
Another helpful factor that staff identified was making sure that the trust engaged with the patients even after they have provided their feedback or left hospital:
Engaging with patients directly is what helps because it creates that understanding. We have a standard phrase, a standard sentence that if any patient complains or raises concern about any of my services they’re automatically and systematically, their choice, they are always invited to join one of the relevant user groups, so that’s a standard. So you know, if somebody writes and says, ‘well I felt the corridor was a bit dusty’, they will get a letter that will say what we’ve done about it, what we’re doing about it, but they’ll also get a paragraph that invites them to join us on a walk around or an audit or to come in and meet with the relevant service manager to talk to us, and that’s absolutely universal across all of our services within the directorate.
Staff member
Finally, staff also considered triangulation of data to be a useful factor when making change:
I can’t use the National Patient Survey to write an action plan, it just tells me that they’re not happy, doesn’t tell me why. So I have to use, other data as well.
Staff member
Factors that hinder trusts in making improvements
Factors that hindered improvement within the trust as discussed by staff are as follows.
Staff recognised the difficulty in implementing changes without the resources to support them, be it time, finances or staffing levels. Examples of quotes given by staff are:
Some feedback may be about response times to call bells or people getting care when they need it. If its staffing issues, that could be a problem for the staff on the wards, management that need to make the changes and maybe the money’s not there or they can’t recruit staff. That’s a big problem, understaffing on the wards at the moment.
Staff member
Negative feedback was also seen as a barrier to improvement, as some staff can be unwilling to believe that their behaviours may have contributed to a patient’s negative experience:
People don’t like being told negative stuff, so just trying to get through to them [staff] that people have had this experience, people don’t like to hear that sort of thing, that people have been complaining about behaviour and everybody always things ‘oh it’s definitely not me’.
Staff member
Another barrier to improvement was seen to be delayed feedback and not being able to get feedback from a patient because they have been moved to another ward:
It’s an assessment unit, it’s a very busy environment so gaining feedback is somethings quite difficult because a lot of the patients now aren’t going home directly from our environment. They’re either too acutely unwell to actually give you proper feedback or they’re being moved onto other wards.
Staff member
High expectations of patients placed on staff and the inability to make changes that suit all patients were seen as other barriers to improvement:
Patients’ expectations are a lot higher than they’ve ever been because we’ve encouraged them to have those expectations, but it has to be within the limits and financial constraints of what the health service can actually provide.
Staff member
Staff found it challenging to use the data collected from patients. Although their teams had the ability to use the data, at times there were delays in interpreting the data and feeding back to the trust. In addition, staff’s busy workload and staff shortages affected their ability to use the weekly feedback from patients:
I suppose the biggest hindrance is, it depends on the data, how it’s been interpreted and how it’s been fed back to us, how it’s cascaded back to staff. It doesn’t always get back to us in a timely manner.
Staff member
Similarly, staff also identified the issue that, even though they had the data, they could not always use them because it was not always feasible to make the changes identified in the data.
Experiences of near real-time feedback
Benefits of near real-time feedback
The benefits perceived by staff regrading NRTF are outlined as follows.
One of the benefits that staff reported was that they could act more quickly in response to feedback as a result of the fast reporting that is associated with NRTF. An example of how staff consider fast reporting to be a benefit is:
We can respond in a timely fashion to issues and incidents and also, learn as things are happening so we can prevent them happening proactively. I think the benefit of [near] real-time feedback is that you can be proactive rather than reactive in nature.
Staff member
Another benefit of NRTF was that staff can involve the trust governors and volunteers in data collection, to help alleviate the demands on staff time:
For me as lead governor, it’s a fantastic way for the governors to get in and you know, be the eyes and ears of the trust and really be on the wards, get a very strong sense of what it’s like here.
Staff member
Staff also highlighted the benefit of having much more detailed data through NRTF in comparison with other data collection methods. Staff or volunteers are able to clarify details with patients. In addition, patients would provide more honest feedback.
The quick turnaround of positive feedback helps to boost staff morale and it can become more personal to staff on the wards:
If you get feedback, you know, positive feedback about the system that we have in place and what we’re doing, then that boosts staff morale, so that people, when you read the newspapers, quite often it’s the bad stories you’re really reading about, but if you get feedback that’s very recent about the positive aspects about our systems, what’s working and what people are happy about, then it boosts staff morale, they feel better, even though everything isn’t going 100% right, at least there’s aspects of it that its’s good for staff to hear about.
Staff member
Another benefit of NRTF that was discussed was the likelihood that there would be an increase in patient feedback as a result of patients completing the survey while in the trust, in comparison with feedback being requested once they are home:
If you’re collecting the information in the here and now, it’s easier and you won’t get as much participant attrition as you would do if it’s being collected at a later time.
Staff member
Another point for discussion was the fact that NRTF can help reduce patient isolation, as the time spent with patients when collecting their feedback can be another point of contact for patients:
The staff don’t always have a lot of time to just sit and listen to that kind of thing, so I think we fulfil more than just one role, it’s not just the information that we’re collecting, it’s sometimes just a friendly face.
Staff member
Staff felt that NRTF would help make it clear to patients that their views count, by being able to implement change quickly, based on the patient feedback.
Drawbacks of near real-time feedback
Drawbacks of NRTF which staff identified are as follows.
One of the drawbacks that staff discussed was whether or not the trust was actually able to make the improvements based on the patient feedback. Therefore, the expectations of staff and patients regarding changes based on feedback collected must be managed:
You can’t please everybody all of the time, which means that some patients will still remain unhappy because something hasn’t been done quickly or cannot be done.
Staff member
Cost implication was another drawback that staff recognised with NRTF, as money would need to be spent to purchase the equipment and software needed to establish a NRTF approach.
Staff recognised the possibility that feedback may not accurately represent patient experiences at the time of collection. This can be because of patients’ conditions, current informational needs or levels of experienced pain. These factors can interact and alter the patient experiences:
Don’t get me wrong, if there’s things that we’re doing wrong I’m first in line to want to know about them but equally, these people are in situations where their emotions are so heightened and dependent on their personality. Some people you put them in a situation where they’re dependent on others and they could not be more grateful for the help that they get, you get other people and you put them in the situation where they’re dependent on others and it ignites something in them that is just fear and discomfort and very often that manifests itself as anger.
Staff member
Some difficulties arose around maintaining staff and volunteer engagement in the NRTF data collection:
We’ve actually committed to quite a big data collection, so 90 [completed surveys] a month is quite a lot to collect. That’s a big commitment from the volunteers and it’s maintaining that commitment and maintaining the enthusiasm of the volunteers over a 10-month period and over the summer as well when lots of people go on holiday.
Staff member
Another drawback was the problem of having to continuously and quickly cope with negative feedback. It can sometimes be impossible to come up with a solution if the patient is expecting an immediate response to the situation:
You’ve got somebody facing you, telling you what’s wrong there and then so you’ve got to then deal with it, so it doesn’t give you the opportunity to think at any length or resolve something before anything.
Staff member
Staff also identified that ward and department teams can sometimes not see the benefit of collecting the data and have the perception that data collectors aim only to find the negatives:
Initially, staff tend to be a bit wary of you coming in because they think you’re there to capture the negative, but I think sometimes, once they realise that there’s a lot of positive coming out of the reports, and that there’s a chance to celebrate successes, I think they tend to welcome you a bit more. But initially, you’re seen as the enemy.
Staff member
One more problem with NRTF that staff reported was the need for the survey to be accessible for all types of patients:
My patient cohort tend to have communication and or cognition problems. We spent literally 18 months devising our current survey sheets because of the need to make sure that they were visually appealing and they were able to be managed to a certain extend by people with communication and cognition issues. If you’re talking about something like an iPad or electronic device then you’re talking about manual dexterity.
Staff member
Finally, staff also identified some issues with how the data are being used within the trusts, specifically data being used as a management tool rather than being given to ward staff:
If it goes back to senior managers it becomes a management tool, and it’s used to performance manage, if it comes back to the practitioners, the practitioners are in a sense auditing their own practice.
Staff member
Volunteer involvement in data collection
Benefits of volunteer use
Staff identified many benefits to using volunteers within trusts. These benefits are outlined as follows.
Staff felt that one of the benefits of using volunteers is that they are able to spend the time with patients and help them use the survey technology if needed:
Some of the older people wouldn’t be very au fait with using a tablet so they’d have to have someone to do it for them. It would work if there was someone to help them obviously as long as the person was impartial and didn’t try to coerce them into saying something.
Staff member
Staff discussed the benefit of having volunteers as another perspective for patient experience:
They [volunteers] will get that insight into what happens to the patients, how their care progresses, what their experience is like, what their journey’s like, how joined up it is, how they relate to the staff, so I think using volunteers is a really powerful thing to do.
Staff member
One of the most commonly discussed benefits was that volunteers are able to free up staff time, so that staff can carry on with other tasks:
Because volunteers are supernumerary, so it wouldn’t be like you’re taking on staff off their normal duties to be doing something like that. The supernumerary volunteers have the time to sit down and you know, engage in a conversation so it’s not just, it’s not just a very quick thing.
Staff member
Staff also felt that volunteers benefit from additional skills and experience that come with involvement in patient experience data collection.
The neutral and objective position of the volunteer was seen as especially beneficial to the staff, as they felt this would encourage patients to give a truer account of their experience:
[A volunteer] is completely neutral and there might be an instance where patients feel if they have a problem that they could be totally honest about it because it’s not a nurse or somebody involved with the ward, so I think maybe it would give some patients a chance to be very open.
Staff member
Considerations around volunteer use
The considerations around volunteer use within trusts, as identified by staff, are as follows.
Collecting accurate data with the help of volunteers was a concern for some of the staff members. They explained that sometimes volunteers can influence the feedback given by patients through their own biases.
Other staff acknowledged that volunteers should be extensively trained to recognise their biases and adhere to study protocols for data collection.
One of the concerns raised regarding the use of volunteers is making sure that the patients remain safe within the trust. Staff felt that this could be ensured by maintaining patient privacy and confidentiality, having volunteers learn escalation protocols and making sure that the patients provide consent before completing the survey:
You’ve got to make sure that it’s someone that you can trust not to give the information out . . . So perhaps they would need to be checked out before actually letting them volunteer for these jobs.
Staff member
Staff felt that it was important to give volunteers the support they need while they are collecting patient experience data. They explained that volunteers should be able to get support if they have witnessed something uncomfortable or traumatic and to be shown that their time and work within the trust is valued:
Training, keeping them motivated, keeping them supported. If they’ve got any problems that they’ve discovered on the ward, it’s giving them the support to be able to either go back or take it to the ward staff.
Staff member
It was recognised that volunteers need to be able to understand the patient’s condition and how that can affect collecting feedback. They need to communicate with ward staff to become aware of any important information; for example, if the patient has received recent bad news, or is in pain, the volunteer needs to be sensitive to that.
Having the volunteer draw on family members and carers to help provide answers to the survey was also brought up by staff:
. . . sometimes it’s good to get visitors’ feedback too because they sometimes can talk for patients who are a bit poorly and yet they appreciate the care. If we don’t speak to them too then we’re not getting the full capture of patients’ experiences.
Staff member
One of the biggest considerations highlighted by staff was the difficulty they experience when trying to recruit volunteers, as not only are available resources low, but they have to make sure the right kind of volunteer is used:
They are unpaid, you can’t expect them always to be there, on a certain day and a certain time. You hope that they are and you ask them to have a commitment, but that’s something that you can never guarantee.
Staff member
Specific survey for interviewees
A total of 31 respondents completed the staff survey (Table 51). Of the six trusts participating in this study, Poole Hospital NHS Foundation Trust had the highest number of responses, accounting for 32.3% of the overall sample. Hinchingbrooke Healthcare NHS Trust provided 19.4% of the responses received, whereas Salisbury NHS Foundation Trust and Northern Lincolnshire and Goole NHS Foundation Trust each represented 16.1% of the overall sample. Milton Keynes University Hospital NHS Foundation Trust and North Cumbria University Hospitals NHS Trust had the lowest representation in the overall sample, accounting for 9.7% and 6.5%, respectively.
| Trust | Percentage (frequency, n) | Number of initial interviews | Number of people who left during interim |
|---|---|---|---|
| Poole Hospital NHS Foundation Trust | 32.3 (10) | 10 | 1 |
| Hinchingbrooke Healthcare NHS Trust | 19.4 (6) | 10 | 2 |
| Salisbury NHS Foundation Trust | 16.1 (5) | 10 | 1 |
| Northern Lincolnshire and Goole NHS Foundation Trust | 16.1 (5) | 5 | 2 |
| Milton Keynes University Hospital NHS Foundation Trust | 9.7 (3) | 10 | 3 |
| North Cumbria University Hospitals NHS Trust | 6.5 (2) | 7 | 0 |
| Total | 100 (31) | 52 | 9 |
The respondents represented a range of backgrounds and interests, including patient experience officers, line managers who analysed and evaluated the After Francis project data and oversaw improvement actions, a matron for emergency care, project sponsors, volunteer supporters and those responsible for the recruitment and selection of volunteers.
Experiences with the After Francis project
Respondents were asked to provide free-text comments regarding experiences of their involvement in the After Francis project. Overall, having completed the 10-month data collection period, staff viewed the After Francis project favourably. A number of respondents commented that the After Francis NRTF was compatible with, and complimentery to, existing data collection ongoing in their trust, including the FFT and their own patient feedback surveys.
Receiving NRTF based on patient experiences was found to be worthwhile in enabling staff to understand experiences closer to when they occur from the patient’s perspective. Staff valued both positive and negative feedback, which boosted morale, yet also allowed them to address issues and concerns of their patient groups:
It has been a very positive experience with identification of good areas of practice and the ability to review and address areas where improvements can be made.
Staff member
The use of technology to collect data and feed back results was largely welcomed. With regard to data collection, one staff member particularly described being interested in the use of a new method of data collection compared with existing pen-and-paper methods. Although staff observed some patients needing help with technology, on the whole, they observed general enjoyment with their use of the tablet. Staff described that the near real-time feedback boosted staff morale by allowing them to see the impacts of their work quickly. Staff used the near real-time feedback as a way to celebrate their success when an issue could be resolved right away, which made a difference in the subsequently collected feedback:
This has been really helpful to the team and real-time feedback has allowed us to celebrate as a team successes and to deal with issue quickly after the event.
Staff member
Three respondents who were involved in the early stages of the project through recruiting volunteers or leading/sponsoring the research within their trust acknowledged experiencing initial set-up challenges prior to seeing study benefits:
From the trust’s perspective this has been an excellent and invaluable research study. It took a while to get started and iron out a few technical and recruitment issues. Once resolved it gathered momentum. Staff, particularly in one speciality, have become fully engaged and actions as a result of the survey are gradually making a difference.
Staff member
It was a hard process to get off the ground and recruit enough volunteers. I expected the young volunteers to excel with the project as they normally are better with technology. However, as the project went on it was the older volunteers with more life skills that were able to get the higher number of surveys.
Staff member
Further comments regarding challenges to implementation of the After Francis survey included the difficulty of finding an appropriate time to approach patients, avoiding treatments and mealtimes. One respondent working in the emergency department noted that data collection may have been better targeted in the major injuries area so that data could be collected from patients who had already been seen and treated. One respondent described the results output as complicated, which made it difficult for them to understand what problem areas need to be looked at.
Service improvements
Staff reported improvements made in their trusts as a result of the After Francis feedback. Actions were put in place to make staff more aware of how they approached and introduced themselves to patients:
I think the named nursing has improved safety and quality and facilitates a more in depth ward handover.
Staff member
Patient calling cards were introduced in order to learn more about the patient as an individual and about their lives from a non-clinical perspective. Carers and relatives were encouraged to contribute to the calling card, given patient consent. In one trust, the introduced calling card system extended to other wards across the trust that were not involved in the After Francis project. Feedback also highlighted some changes to transactional aspects of care, such as access to call bells and reducing response time.
Steps were taken to improve communication and patient information to ensure that patients were given explanations of treatment. Actions were put in place to provide better information on discharge in emergency departments and explanations of care delivery. Information flows to carers and families were also improved (e.g. through the setting up of a family clinic where patients’ families can arrange dedicated time with the ward lead).
In addition to the improvements within participating wards, some respondents reported that their learnings had been disseminated to other teams in the wider trust, for example across the older people directorate and information-sharing at the care group level. Action-planning issues were also reported to have been discussed between senior sisters and lessons learnt shared with teams. A barrier to the sharing of improvements across other teams within a trust was the provision of time in order to communicate more widely.
Respondents were asked to select the most important factors in making use of patient feedback for improvements to service (Table 52). The most important factor cited for encouraging patient feedback was to ‘Gain a better understanding of patients’ needs’, with 74.2% and at least one person from each trust selecting this option. ‘Feeling responsible for improving the patient experience’ (67.7%) and ‘Improved communication between staff and patients’ (54.8%) were the next two most popular choices. Interestingly, ‘Incentives or awards’ was the least endorsed category (9.7%). One respondent gave a free-text response indicating that they would have liked to have the feedback on the same day it is collected or when the minimum number of completions had been gathered.
| Response | Percentage (frequency, n) |
|---|---|
| Better understanding of patients’ needs | 74.2 (23) |
| Feeling responsible for improving the patient experience | 67.7 (21) |
| Improved communication between staff and patients | 54.8 (17) |
| Improvements are part of the quality agenda | 51.6 (16) |
| Improved communication among staff | 48.4 (15) |
| Faster reports (RTF) | 41.9 (13) |
| Belief that one can change things | 35.5 (11) |
| Management or leadership support | 35.5 (11) |
| Ability to triangulate data | 32.3 (10) |
| Incentives or awards | 9.7 (3) |
With regard to challenges that staff encountered to using patient feedback to make improvements (Table 53), the top three hindrances were listed as ‘Managing patients’ expectations for improvements’ (54.8%), ‘Making changes to suit all patients is difficult’ (48.4%) and a ‘Lack of resources’ (41.9%). One respondent who replied to the free-text response option suggested that it would also be useful to collect data outside office hours to investigate if any differences in responses were evident.
| Response | Percentage (frequency, n) |
|---|---|
| Managing patients’ expectations for improvements | 54.8 (17) |
| Making changes to suit all patients is difficult | 48.4 (15) |
| Lack of resources | 41.9 (13) |
| Lack of detail in patient feedback | 29.0 (9) |
| Feeling solely responsible for improving patient experiences rather than it being a team effort | 19.4 (6) |
| No incentives for making improvements | 16.1 (5) |
| Patient feedback was collected a long time ago and is no longer applicable | 9.7 (3) |
| Lack of management or leadership support | 6.5 (2) |
| Negative feedback is not used or accepted | 0 (0) |
| No time to review or consolidate patient feedback | 29.0 (9) |
Impact of the near real-time feedback data on staff to make improvements
Respondents were asked to rate the effectiveness of the After Francis survey in helping staff make changes to care. Six respondents rated the survey as very effective, 18 rated the survey as somewhat effective and four rated the survey as not effective at all. Those rating it as not effective were from two trusts.
In free-text comments, respondents largely felt that staff were motivated to make changes in response to the NRTF, as data were gathered directly from patients. The feedback was a mechanism through which valuable information could be collected on patient views:
[Staff] have a more comprehensive account of how their care affects our patients and, therefore, are motivated to improve.
Staff member
The feedback has been shared with all the older people service nurse staff and associated support services staff – there has been multidisciplinary involvement in taking forward changes in practice supported by the information gathered.
Staff member
One suggestion of how staff on wards could receive feedback on the After Francis project data was made:
It would have been nice to run a feedback quick secession within the recruiting wards at maybe handover time, so that staff rather than managers could see the type of things patients were reporting back. Sisters who attended the feedback meetings seemed to very much take on board the feedback.
Staff member
Using near real-time feedback
As part of the After Francis project evaluation, it was important to ask staff for their view of collecting and responding to NRTF. When asked how valuable the NRTF approach was for the work of their ward or department, 71.0% (n = 22) of staff viewed it as very valuable, 22.6% (n = 7) viewed it as somewhat valuable and 6.5% (n = 2) viewed it as not at all valuable. This demonstrated that a large majority viewed NRTF as valuable in their work. When asked if they would like to continue to collect NRTF in their ward or department beyond the After Francis project, the majority 66.7% (n = 20) indicated that they would, whereas 26.7% (n = 8) were unsure. Two respondents felt that they would not wish to collect NRTF after the trajectory of the project.
Respondents were asked to select the main benefits of NRTF (Table 54). The highest number of responses were received for the answer options ‘Shows patients their views count’ (74.2%); the ‘Ability to act fast’ (71.0%); and ‘Can use volunteers instead of staff to collect data’ (67.7%).
| Response | Percentage (frequency, n) |
|---|---|
| Shows patients their views count | 74.2 (23) |
| Ability to act fast | 71.0 (22) |
| Can use volunteers instead of staff to collect data | 67.7 (21) |
| Can follow up or clarify with patients | 48.4 (15) |
| Boosts staff morale | 38.7 (12) |
| Reduces patient isolation | 38.7 (12) |
| More detailed data | 32.3 (10) |
One respondent, answering in free text, described how NRTF gives ownership to staff as their actions are responsible for feedback. This resulted in negative responses driving the team to improve their care and raise standards.
When considering the drawbacks of NRTF, the most frequently selected reason (64.5%), which was selected by staff from all trusts, was that the ‘Patients’ condition may influence the result’ (Table 55). ‘Maintaining volunteer engagement’ (51.6%) was also endorsed by representatives in all trusts and ‘Managing patient expectations of change’ was selected by 51.6%.
| Response | Percentage (frequency, n) |
|---|---|
| Patient conditions may influence feedback | 64.5 (20) |
| Maintaining volunteer engagement | 51.6 (16) |
| Managing patient expectations of change | 51.6 (16) |
| Problems with electronic equipment | 29.0 (9) |
| Lack of staff buy-in | 19.4 (6) |
| Using the data for performance management vs. improvement | 19.4 (6) |
| Coping with negative feedback | 16.1 (5) |
| Cost and time implications | 16.1 (5) |
The use of volunteers to collect patient experience data
Overall, staff considered the use of volunteers for collecting patient feedback valuable, with 82.1% (n = 23) reporting that they were very valuable and 10.7% (n = 3) reporting that they were somewhat valuable. Two respondents felt that volunteers were not valuable. These two responders were from two different trusts. There was a generally positive response to the use of volunteers to collect patient experience feedback data (Table 56). In particular, responders from all trusts felt that the use of volunteers freed up staff time (87.1%), reduced bias in responses as a result of volunteers being viewed as neutral (83.9%) and reduced patient isolation (51.6%). Volunteers were also thought to add to the care experience in a manner staff could not:
The impartiality of using volunteers has been a bonus and on two occasions a patient has chosen to ‘disclose’ to a volunteer. This has been investigated and followed up. Concerns may not have come to light if a volunteer had not been involved.
Staff member
| Response | Percentage (frequency, n) |
|---|---|
| Ability to help with technology | 29.0 (9) |
| Frees up staff time | 87.1 (27) |
| Reduces patient isolation | 51.6 (16) |
| Volunteers are seen as neutral so patients are more likely to answer honestly | 83.9 (26) |
Despite the positive endorsement of the use of volunteers, staff were mindful of the many aspects which should be considered when working with volunteers (Table 57). Aspects included having adequate support for volunteers (83.9%), being mindful of the skills required by volunteers to interact with patients effectively (80.6%), the challenges of maintaining volunteer engagement or retaining them (74.2%) and safeguarding patients (74.2%). Recruiting volunteers was also considered to be a key factor when working with volunteers (67.7%). Some staff specifically highlighted the challenges of retaining volunteers and the impact that may have on the number of data collected:
We had a lot of interest initially however in the end we only maintained two volunteers throughout the process one who did come for every session this could have been [due to] the age of our volunteers and their other work commitments.
Staff member
| Response | Percentage (frequency, n) |
|---|---|
| Recruiting volunteers | 67.7 (21) |
| Safeguarding patients | 74.2 (23) |
| Supporting volunteers | 83.9 (26) |
| Maintaining volunteer engagement or retaining them | 74.2 (23) |
| Volunteer skills to interact with patients effectively | 80.6 (25) |
Reflections on the survey
Staff completing the survey were asked for further comments. These comments were largely positive; however, some challenges were highlighted, including the length of the survey and technical issues with tablet completion. Comments reflected the need for fewer questions, particularly for older people. Having one space for free-text comments at the end of the survey was also troublesome if patients wished to comment throughout, yet risked forgetting their comments by the time they reached the final page.
In-person interviews with staff
Project involvement
The staff were mostly involved with the project throughout the 10-month data collection period, with only a few becoming involved part-way through as a result of starting new roles.
Activities
Activities that staff were involved with throughout the project are as follows:
-
initial set-up of the project at the trust
-
attending the two participatory workshops and a networking event
-
directing volunteers to appropriate patients
-
participating in the project staff interviews and surveys
-
disseminating results to staff.
Generally, there were no changes in staff involvement throughout the data collection period. Staff reported only changes if they had stepped into a new role and were adapting to their new duties as well as having involvement within the project.
Using a near real-time feedback approach again
When staff were asked if they would like to have a NRTF approach implemented in their trusts again, the consensus was that staff would like to see the approach used again to collect patient experience feedback, especially if tablets were to be used. For trusts which already used NRTF before the onset of the project, thoughts were focused on wanting to move away from paper surveys to using tablets, as the technology helped streamline data collection and speed up the reporting process. One staff member explained, ‘Patients are less resistant to this computerised thing, and you can see that it’s anonymous and there are no bits of paper that can go astray, so it’s much better than old fashioned sheets’.
Project challenges
Although staff would like to use a NRTF approach again to collect patient experience feedback, they experienced challenges throughout the project related to communication, engagement, recruitment and reporting. The challenges are outlined in more detail below.
Communication
Communication was the main challenge that staff experienced throughout the project, which caused delays with the weekly reports getting back to the ward staff on time. In addition, staff at one trust reported that, initially, volunteers were not approaching staff to check whom they could and could not interview.
Staff engagement
It was reported that some of the ward staff could be resistant to change, particularly when the focus was on implementing changes to target areas of improvements. Staff found that, with senior staff buy-in, overall engagement of other staff with the research improved. However, there was still resistance to improvements or changes, especially from the junior members.
Recruiting patients
Recruiting patients to participate in the survey was often a challenge because of the hospital settings. Staff found that it was harder to recruit on the wards because of the low patient turnover, resulting in lower patient recruitment numbers.
All trusts continued with other feedback methods, such as the FFT, after which some patients refused to complete the tablet-based survey, stating ‘no, we’ve already done that’. After combining the After Francis survey results with other patient experience data within the trust (e.g. the FFT scores), staff saw recruitment numbers improve.
Recruiting volunteers
Staff perceived that a large number of patients did not complete the survey. They suspected that not all volunteers were confident or comfortable enough to approach all patients to gather their feedback. It was highlighted that it was important to recruit volunteers specifically suited to data collection in the future.
Reporting format
It was difficult for the staff to interpret the weekly reports in the format that the research team provided. Some trusts tackled this challenge by creating their own shorter reports to distribute to wards if there were the resources to do so.
Lessons learnt
Below are the main lessons learnt by the teams involved in the project at each case study site.
Reporting format
Staff reported that they found it difficult to understand the weekly feedback using the reporting format that the research team sent to them. After some discussion, staff explained that, although they found the free-text comments to be extremely valuable, the quantitative data needed to be displayed visually, using colours and graphs, rather than plain line graphs or bar charts. This would allow them to quickly view the data and identify areas of improvement.
Value of volunteers
Volunteers were seen to be a valuable resource throughout the project, as they have more time to spend talking to patients and can encourage patient participation in the survey. Volunteer interactions with patients also provided an opportunity for patient concerns to be resolved quickly, at times while the patients were still on the wards.
Implementing little changes often
It was found that, when addressing challenges in the feedback, ward staff were often more receptive to making small changes often, as opposed to making numerous large changes in one go. This ensured that staff were not overwhelmed by any sudden changes and gave them the time to incorporate changes within their daily routine before the next improvement was made.
What needs to be in place for successful implementation of a near real-time feedback approach
Staff identified three key factors that need to be in place in order to successfully implement a NRTF approach for collecting patient experiences: communication, senior staff buy-in and a dedicated volunteer workforce.
Communication related to not only how staff communicate with each other, but also how the results were fed back to the ward staff and used to make improvements. There needs to be coherent and frequent communication to make the approach successful.
Senior staff support is also essential, to ensure that the feedback is used in action-planning and to aid staff buy-in and engagement. Although staff still need to maintain ownership of any improvements implemented, they need the support of senior staff to move forward with any changes (e.g. those that may require financial support).
A dedicated volunteer workforce was considered key to the NRTF approach. This relieved staff of the burden of data collection so that they could prioritise tasks involved with patient care. However, in order to gain a dedicated volunteer workforce, purposeful recruitment is required, to recruit volunteers who are best suited to data collection and will be dedicated to the project. Similarly, ongoing support must be in place for the volunteers should they need it. For example, volunteers should be able to ask questions and discuss their experiences with someone. In addition, comprehensive training must be available to make sure that health and safety protocols are followed and to provide volunteers with knowledge regarding safeguarding, patient capacity and informed consent.
Improvements made based on patient feedback
Based on the patient experience feedback collected as part of the project, staff reported that several improvements had been made. Staff from some of the trusts reported a decrease in complaints on their wards after implementing some of the following improvements.
Call bells
Call bell response times have repeatedly been a challenge for staff to address. One case study trust conducted an audit of response times and found that patients were often not waiting as long as they thought. As a result, the site’s wards began to display posters with the average wait time to help manage patients’ expectations of how long it may take to be seen by a member of staff once called.
Communication
Several challenges relating to communication were identified throughout the project: the way staff communicated with patients about their treatments and conditions, as well as the way staff communicated with their colleagues about staffing levels. In order to improve the delivery of information, trusts used boards to highlight patient feedback results and the number of staff available (and their roles) and to provide up-to-date information about waiting times.
Similarly, staff encouraged each other to reintroduce the ‘Hello, my name is . . .’ campaign and provide more information to patients regarding discharges. One staff member said:
We introduced the Ticket Home, which is a patient’s discharge summary . . . it was completely de-medicalised, everything was simple . . . people that were more forgetful or where the family needed something in writing, you were able to give them something to read and know what was going on.
Privacy
Privacy was a concern of patients, specifically in A&E when checking in. As a result, screens were put up around reception to give patients more privacy when speaking to the reception about their conditions.
Other changes
Some changes, although they did not specifically target relational aspects of care, were implemented to help improve the overall patient experience. At one site, these included employing a housekeeper whose role was to ensure that the patients on the ward had enough drinks, thus relieving the nurses of some pressure. Matrons and senior sisters worked with ward staff to make them more aware of how their verbal and behavioural communications could be perceived by patients. At another site, improvements were made to the overall environment, such as repainting to brighten up areas for the patients.
Chapter 5 Best practices in the use of near real-time feedback
Results for research question 4: best practices in the use of near real-time feedback in the NHS
Develop a survey toolkit and evidence-based recommendations
Based on the patient experience and evaluation data, and with the help of feedback from the advisory group, a survey toolkit was developed which included evidence-based recommendations for the use of the survey instrument to improve relational aspects of care. The toolkit consists of an introductory section outlining the purpose of the toolkit, practice recommendations and four guides addressing survey implementation, volunteer training, reporting and evaluation. Practice recommendations focused on planning for the survey implementation, working with volunteers, reporting findings to staff regularly and evaluating the success of the NRTF approach.
Practice recommendations were based on barriers to, and facilitators of, the NRTF approach, which were obtained from interviews, surveys and workshops, with staff and volunteers. A report outlining the barriers and facilitators has been published in the Health Policy and Technology journal. 53
Three case studies are included to provide an overview of challenges experienced as part of the NRTF approach, lessons learnt and impacts generated from the methodology. The short form of the survey instrument to measure relational aspects of care is also available as part of the toolkit. The full volunteer training materials are provided alongside contact information for three of our six case study sites that would be happy to answer questions from other NHS trusts interested in improving relational aspects of care using a NRTF approach. The toolkit is freely available to NHS trusts at www.picker.org/compassionatecare (accessed 4 December 2017).
Disseminate toolkit and practice recommendations to NHS trusts
To share the toolkit, three regional toolkit launch and networking events were held. Launch events were held at the following locations in February and early March 2017:
-
Novotel Paddington, London, 20 February 2017
-
Novotel Leeds Centre, Leeds, 27 February 2017
-
Harris Manchester College, Oxford, 2 March 2017.
Each toolkit launch event was open to 40 attendees. Between 27 and 36 participants attended each event. Attendees represented NHS trusts, voluntary organisations (such as Macmillan Cancer Support and Point of Care Foundation), NHS England, Collaborations for Leadership in Applied Health Research and Care, universities and clinical commissioning groups.
The launch events commenced with a presentation providing an overview of the research, which was followed by a detailed description of the purpose and components of the toolkit. This was followed by time for questions about the content and functionality of the toolkit. Time for networking and further small group discussions was provided.
Discussions during the toolkit launch events focused on volunteer recruitment with the help of a person specification, increasing the use of patient experience data by frontline staff, the sustainability of the approach beyond the funded project, the need for ethics board review for implementation of the survey, and survey software providers.
In February 2017, the toolkit was made freely available on the Picker website (www.picker.org/compassionatecare) and publicised through a published policy commentary. 46 The policy commentary outlined the importance of improving compassionate care and more widespread use of NRTF for decision-making. The available toolkit offers one avenue for improving relational aspects of care using this more timely approach to data collection, analysis and reporting.
In addition to this, the toolkit was disseminated through Twitter and e-mails to our contacts at NHS trusts. E-mails were sent to our existing database of patient experience leads at NHS trusts across England. This list included 157 contacts at NHS trusts and 60 contacts at clinical commissioning groups.
Chapter 6 Discussion
Introduction
In this report, we describe the development, implementation and evaluation of a survey tool which measures patient experiences of relational care using a NRTF approach.
This report provides a useful context for trusts across the UK as they continue to respond to the recommendations of the Francis Report4 and take measures to strengthen patient voice, improve frontline care and challenge current organisational culture. This report also provides a comprehensive evaluation of the effectiveness of a NRTF approach to gather patient feedback and improve relational aspects of care. Despite the increasing support for the NRTF approach, it remains an area which is under-researched. 2,3
In this section, we summarise the quantitative and qualitative methodologies used in our study and then summarise the key findings which pertain to each of our RQs. Finally, we accumulate these findings and discuss them in the context of the robustness and limitations of the research.
Methods for the evaluation
This evaluation utilised a participatory mixed-methods approach, consisting of both qualitative and quantitative data collection. We obtained feedback not only from patients but also from hospital staff, volunteers and patient and public representatives. Through the comprehensive evaluation methodology, we aimed to understand the effectiveness of a NRTF approach for improving relational aspects of care at six case study sites, with a particular focus on improving the experiences of older patients and those visiting A&E departments.
Our data collection was tailored to address the following four overarching RQs:
-
Can NRTF be used to measure relational aspects of care?
-
Can NRTF be used to improve relational aspects of care?
-
What factors influence whether or not NHS staff can use NRTF to improve relational aspects of care? Specifically, what are the barriers and enablers?
-
What should be considered best practice in the use of NRTF in the NHS?
Based on the data collected to answer these RQs, a series of outputs was produced for NHS staff and managers, policy-makers, commissioners, researchers and academics. Outputs include a toolkit, which included the validated patient survey instrument, publications in peer-reviewed and professional journals and three conference presentations. 46,53
Synthesis of findings for research question 1: can near real-time feedback be used to measure relational aspects of care?
To answer this RQ, a variety of data were collected and analysed to develop and test a survey measuring patient experiences of relational aspects of care, specifically for use with a NRTF approach.
First, a literature review was conducted to understand the factors encompassing compassionate or relational aspects of care and identify existing tools used to measure these. To supplement the learning from the literature review and gain a more in-depth understanding of what recent patients consider the most important relational aspects of care, a focus group and eight interviews with recent A&E patients, and with hospital patients aged ≥ 75 years, were carried out. Based on the relational aspects of care identified through the literature and data collected from recent patients, relevant existing survey questions were identified in the national Adult Inpatient Survey and the Emergency Department Survey. Factor analyses identified a comprehensive set of existing items which measured relational aspects of care within the national surveys.
Next, six case study sites were recruited to participate in the 10-month patient experience data collection period, as well as other collections of data to capture staff and volunteer feedback. Case study sites were selected to ensure an adequate geographic spread and composite scores from the items measuring relational aspects of care on the national surveys were utilised to ensure that higher- and lower-performing trusts were equally represented in the final case study sites.
To address all relational aspects of care identified in the focus group and interviews, additional survey questions were developed to supplement existing survey questions. Advisory group members then selected the final survey questions, which were tested cognitively with older inpatients and A&E visitors across three case study sites.
Survey development
The literature review identified four existing scales which measure relational aspects of care. Specifically, two of these instruments were found to measure predominantly relational aspects of care. 31,32 An additional two instruments addressed some relational aspects of care; however, their main purpose is to measure functional or transactional aspects of care. 33,34 None of the existing measures was developed and tested for use specifically with older patients, or using a NRTF approach. Hence, a new survey instrument measuring relational aspects of care, using a NRTF approach for use with older patients and in A&E departments, needed to be developed and tested.
Through our qualitative research, a total of 22 themes of relational aspects of care were identified by recent patients. The eight most frequently occurring themes were prioritised as being essential in ensuring a positive patient experience. These aspects included patients ‘feeling informed’ and having staff treat them with ‘kindness’ and ‘empathy’.
To supplement existing survey items measuring relational aspects of care, additional survey questions were developed to ensure adequate coverage of each theme, with a pool of at least five survey items available per theme. This increased the available survey item pool to 62.
The advisory group eliminated 40 potential survey questions and provided suggestions to tailor the wording of the question text and answer options to the target populations.
The resulting 22 questions were tested using cognitive interviewing. Following this process, two survey questions were removed as these were perceived to be duplicates by the 30 respondents. The final survey instrument included 20 closed survey questions, one open question and seven demographic questions. The survey development process was followed by volunteer training, after which the 10-month-long patient experience data collection began at the six case study sites.
Throughout the patient experience data collection period, the questionnaire instrument was used to gather feedback from A&E attendees and older people in the case study sites. There were challenges around recruiting and maintaining an appropriate set of volunteers to support survey administration and, partly as a result of this, sites were generally not able to meet the targeted numbers of participants. Nevertheless, older people and A&E attendees alike were generally able to complete the questionnaire as intended, particularly when supported by interviewers. Analysis of data quality from the questionnaire, particularly in terms of item completion and termination rates, suggests that it generally performed well. There were some items that performed less well individually, in terms of high non-response or ceiling effects, andwe did receive feedback that a shorter set of questions would be preferred. Subsequent analysis showed that a 12-item short form retained good measurement characteristics and provided a valid and concise means of measuring people’s experiences of relational aspects of care during hospitalisation or A&E attendance.
The successful development of the questionnaire instrument and short form, along with data from the 10-month collection period, support the conclusion that a NRTF approach can be used to effectively measure people’s experiences of relational aspects of care.
Synthesis of findings for research questions 2 and 3: can near real-time feedback be used to improve relational aspects of care and what factors influence whether or not NHS staff can use near real-time feedback to improve relational aspects of care? Specifically, what are the barriers and enablers?
To address these RQs, we collected data from patients at six case study sites over 10 months. Interviews and surveys with staff were conducted before patient data collection began and again afterwards. To fully understand the factors that influence the success of the NRTF approach to measure and improve relational aspects of care, two participatory workshops were conducted at each case study site. Here, staff discussed barriers to, and facilitators of, improvements and made plans for how to work around them to improve patient experiences. Volunteers were also encouraged to provide weekly feedback on their data collection experiences using an online diary.
Using near real-time feedback to improve relational aspects of care
Based on the patient experience data, a small but statistically significant improvement in overall experiences of relational aspects of care was observed over the course of the 10-month data collection period. Although the number of completed surveys for each participating ward per week was relatively small, overall, sufficient responses were collected to detect a significant change in patient-reported experiences.
Volunteers and staff reported that patients benefited from having their voice heard during their hospital stay or visit. Interactions with volunteers gave patients the chance to share and make sense of their experiences. Therefore, speaking with volunteers and processing their experiences in this manner may have been an intervention in itself, which helped to improve patient experiences. Volunteers reported that they could help improve patient experiences by, in cases where patients gave permission, immediately reporting patients’ needs to staff, who could address them right away.
Staff received weekly reports which included graphics and printable dashboards to monitor changes over time. Through these weekly reports alone, patient experiences of relational aspects of care were brought to their attention regularly. Over the course of the project, staff collaborated with colleagues from their wards/departments to identify areas for improvements to relational aspects of care, and developed action plans to work towards improving these right away. Following the data collection, the case study sites reported that most action points had been successfully addressed by incorporating them into current clinical practices. At some case study sites, clinical leaders, such as ward managers, quality matrons or directors of nursing, considered it within their remit to monitor progress towards improvements and to develop and drive further action-planning with their teams based on the weekly reports of patient experiences.
Overall, the findings support the view that a NRTF measure of people’s experiences of relational care in hospital can be a valuable tool in supporting service improvement. However, we found that the value of the data and the success of improvement plans depended on a number of factors.
Factors influencing NHS staff’s use of near real-time feedback to improve relational aspects of care
A variety of factors were found to influence staff use of NRTF to make improvements to relational aspects of care. Specifically, the reporting format was found to be a key enabling factor. Clinical staff generally lacked time and did not have access to e-mail as part of their regular duties. Therefore, short summaries of findings (i.e. printable dashboards and graphs) were essential tools that could help staff understand areas for improvements quickly. When these were absent, it led to underuse of the data.
Free-text comments were considered an important enabling factor by staff to identify and prioritise improvements. During action-planning workshops held at the six case study sites, staff preferred to work with patients’ free-text comments rather than quantitative survey data to identify areas for improvement. Staff also reported monitoring free-text comments over time to get a sense of progress towards improvements. Free-text comments from patients also included feedback on transactional aspects of care. Staff found these equally helpful and addressed these at the same time as relational aspects of care in their action plans.
Another enabling factor was senior colleagues who felt personally responsible for ensuring that improvements were made on their ward or department. For example, directors of nursing or ward leaders would revisit action plans during regularly scheduled meetings and drive improvements through progress monitoring.
At the same time, senior colleagues could also hinder the use of patient experience feedback to improve relational aspects of care. For example, at some sites, frontline staff reported not receiving weekly reports on a regular basis, or at all, from their colleagues. Others reported receiving only partial reports that were thought to be applicable only to their ward or department. Staff responsible for disseminating reports to their teams reported not sharing them for fear that they were ‘too complicated’ for their teams to understand. Others reported a preference to share a verbal summary of the findings with their teams in meetings rather than sharing the full weekly reports.
Technical barriers were experienced. Initial start-up challenges related to the tablet computers and the survey software were encountered. Stationary kiosks in A&E departments presented a challenge to teams as sufficient patient responses could not be collected despite staff attempting to direct patients to the kiosks.
In addition, volunteer engagement and retention were reported to be barriers for all six case study sites because of the specific nature of the data collection task, high expectations from volunteers and the need to balance data collection with other volunteer activities.
Synthesis of findings for research question 4: what should be considered best practice in the use of near real-time feedback in the NHS?
Various best practices were identified for the use of a NRTF approach in a NHS hospital setting. These were identified through qualitative and quantitative data collected from staff, volunteers and patients. Specifically, they focused on planning the survey implementation, working with volunteers to collect patient feedback, reporting findings to staff on a regular basis and evaluating the success of the NRTF approach to improving the relational aspects of care. These findings were consolidated as part of the final toolkit. Three case studies were shared describing the contextual factors of NHS hospital settings and the lessons learnt which led to the development of the best practice recommendations.
Practice recommendations were disseminated to NHS trusts and other interested stakeholders through three regional toolkit launch and networking events. Following the events, the toolkit was also made available free of charge online. The resource and events were publicised via e-mail, Twitter, an online event management system and the Picker and the University of Oxford’s websites.
Robustness of results and limitations
To address our four RQs, we examined the effectiveness of a NRTF approach to measure and improve relational aspects of care in NHS hospitals. We collected evidence from patients, hospital staff and volunteers on multiple occasions at six case study sites. Over the course of the research, a small but statistically significant improvement in relational aspects of care was observed across the case study sites. Hospital ward or department teams reported making improvements to communication with patients and among staff.
A toolkit has been developed to make the survey instrument and NRTF approach to improving relational aspects of care accessible to other NHS hospitals. However, it should be noted that the six case study sites experienced challenges with retaining their trained and available volunteer workforce. Therefore, the number of completed surveys per ward or department each week was relatively small. Nevertheless, over the course of the 10-month data collection period, the sample sizes were adequate to analyse progress over time. Other methodological limitations that affected the analysis of patient experience data were high levels of missing data on a survey item level and ceiling effects on responses to some survey items. In addition, the survey of relational aspects of care was developed and tested for use in A&E departments and elderly care wards. Therefore, the generalisability of the survey instrument to other areas within a hospital should be evaluated prior to wider implementation.
Chapter 7 Conclusions
To briefly summarise the key outcomes of the research, the practice implications and recommendations for future research are noted below.
Practice implications
Collecting patient experience data from older patients presents unique challenges, as many patients present with long-term conditions and multimorbidities that often affect hearing, speech, vision and cognitive processing. 30 Although these challenges may not apply to patients visiting A&E departments, the transient environment, combined with recent or ongoing acute pain, shock and trauma experienced by patients, makes data collection equally challenging in A&E departments. 54
Therefore, no single mode of administration can be considered the best method of collecting data from these patient groups. In fact, as the needs and experiences differ greatly across these groups, a flexible or responsive data collection mode is needed, which can offer assistance to patients during the data collection process. 55 Well-trained volunteers can provide a responsive approach to data collection from lesser-heard groups. By rapidly determining any needs for assistance and providing help, data collection can be facilitated. 56 An additional benefit of the use of trained volunteers is that the data collection burden, which may otherwise fall on clinical staff, is reduced. 55,57
The NRTF approach offers a new and valuable approach to NHS hospitals wishing to improve relational aspects of care. A small but statistically significant improvement was found in patient-reported experiences of relational care using a near real-time approach to data collection. Staff, volunteers and patients described the value of the NRTF approach to their work and care.
It is important that organisations and staff are aware of the approximate number of completed surveys that are achievable per month with a NRTF approach using trained volunteers. The size of the volunteer workforce and the transient nature of A&E departments as well as lower patient turnover on elderly care wards all affect the number of surveys that can be completed in a given month. Therefore, trusts looking to implement this approach should not expect to collect large numbers of data that allow for comparisons across trusts or generalisation of findings to other areas. However, fewer completed surveys do not pose a great limitation for the use of the resulting data for ward- or department-based improvement purposes.
Based on these findings, a toolkit was developed to assist other interested NHS trusts in improving compassionate care using this NRTF approach. Practical implications and considerations related to data collection and improvement are outlined below.
The use of volunteers was found to have unexpected benefits to the trusts. For instance, staff reported that patients valued the opportunity to talk with the volunteers. Where appropriate, volunteers also helped staff by immediately alerting them to concerns on the patients’ behalf.
However, volunteer recruitment and retention was a challenge at all case study sites. The sites noted that volunteers are a fragile resource and can become unavailable at a moment’s notice. During the 10-month data collection phase, trusts found that they needed to constantly recruit volunteers, as they could easily become unavailable as a result of other volunteering duties at the hospital, sicknesses, travel or personal commitments. In some cases, staff perceived that, initially, volunteers did not understand the nature of the work and the amount of patient contact required as part of the data collection. Therefore, some volunteers dropped out shortly after beginning their data collection.
Staff engagement and buy-in was found to help facilitate improvements to care and action-planning, especially if senior staff, such as quality matrons, directors of nursing or ward matrons, were engaged in the research. They were seen to be key enablers for driving forward action plans for improvement. However, engagement is also needed from the whole ward/department team. This buy-in ensures that volunteers are not restricted by barriers created by ward staff who may not be fully aware of the research and that data are reported back to staff frequently for discussion.
The weekly reporting format was found to be a determinant of staff use of the patient feedback. Initial versions of the reporting format used in this research were not easily utilised by the hospital staff and so underwent a process of co-design with staff involved in the research. The co-design process created a reporting format that was accessible for frontline staff, including printable versions, dashboard overviews and graphs indicating changes or trends over time. This enabled staff to take a quick look at the data during their limited free time, without needing to focus on the interpretation of the quantitative patient feedback.
Tablet computers offer a practical and easy way to collect feedback from patients. In practice, they eliminate data entry and collation, which often requires substantial staff time. Though tablets provide a relatively cost-effective alternative to conventional paper surveys, there is a need for initial investments into equipment and software if such tools are not yet available for use in the trust. Similarly, time must be spent to set up the questionnaire for use on the tablets, and staff and volunteer training must take place to ensure that the tablets and software are used correctly. In some case study sites we noted initial teething problems in introducing this approach to data collection; although relatively easy to resolve with support, early issues could be a barrier to use in sites where there is not adequate commitment to embedding the approach. Therefore, this method is considered to be best suited if there is an intention for continued use of the equipment and volunteer-led data collection.
In practice, stationary kiosks were ineffective as data collection tools, even given staff and volunteer efforts to direct patients to use the kiosks. Staff reported more technical difficulties with kiosks and perceived that patients wanted to leave the A&E department immediately on completion of their treatment, thus forgoing the use of the kiosks to provide feedback on their experiences. Owing to the very low numbers of weekly responses collected through kiosks, their use was discontinued within the initial 3 months in favour of volunteers with portable tablets.
The patient experience data collection was tested and implemented in hospitals serving a primarily white British population. Therefore, if data on patient experiences of the relational aspects of care were to be collected at hospitals serving populations with different backgrounds, the instrumentation should be reviewed and tested again. Persons from other cultural backgrounds may see other elements of relational aspects of care as important and the value they place overall on relational aspects of care may be different. Therefore, interviews, focus groups and cognitive testing would be helpful tools for exploring these culture-specific differences.
In conclusion, the NRTF methodology offers a valuable approach for hospitals to measure relational aspects of care and provides them with ongoing feedback to form the basis of improvements. However, it should be recognised that the approach required resources and commitment from health-care providers to overcome its challenges. Costs associated with the approach are primarily start-up costs, with few ongoing costs. Time to train volunteers and support them should also be considered, as well as time to develop a reporting system that matches the hospital’s or trust’s preferences.
Other considerations include quality assurance, information security and data storage and retention. The sustainability of the NRTF approach, especially around reporting and the interaction of trained volunteers with patients, should be considered. Trusts might involve their information technology team in the design of a sustainable reporting template that requires little ongoing maintenance. In addition, trusts might discuss with their patient experience team leads the implications of the introduction and potential removal of trained volunteers who interact with patients and their impact on patient experiences. To address these considerations, the establishment of an initial working group may be beneficial prior to the implementation of the NRTF approach.
Despite these considerations, the NRTF approach offers a valuable tool for improvement purposes. However, for continued understanding and use of a NRTF approach to strengthening compassionate care and the patient’s voice, further research should build on the findings of this study.
Recommendations for future research
To assist staff in using patient feedback on a regular basis, further research is necessary to explore the impact of different styles of reporting formats. There is a need to find an optimal reporting format that takes into account staff ability to analyse and interpret data, the amount of time available for review of results, the setting in which the data are reviewed and other reporting formats with which staff are already familiar. The use of graphics to explore and monitor changes and trends over time should be examined. Finally, future research is needed to understand the best channels for dissemination of results within trusts. It is crucial for staff and management to understand what works best with regard to internal dissemination of results.
Staff and volunteers believed that patient experiences were improved by their interactions with volunteers. Volunteer visits provided patients with a chance to speak about their experiences, which left them feeling grateful for the opportunity to share their views and having someone who listened to them. Although volunteers did not encourage patients to share details about their experiences until the very last survey question, patients did so regardless. This highlights the desire for patients to share and process their experiences of hospital care. The impact of providing patients with a one-on-one opportunity to discuss their experiences should be explored, especially as communication with patients is an import component of relational aspects of care.
By alerting staff to patient concerns, where appropriate, volunteers can play a central role in helping staff improve patients’ experiences. However, they are a fragile resource, so further research is necessary to understand the roles volunteers play in various activities at NHS trusts. This understanding may help shed light on why volunteers chose to stay on or leave the data collection and may assist hospitals in developing more bespoke volunteer recruitment, training methods and support mechanisms that encourage their retention in this type of role.
As other NHS trusts begin to implement NRTF approaches to improving relational aspects of care, local-level evaluations will be necessary to continue to assess effectiveness, and to capture and address challenges as they arise. A large-scale or national-level evaluation will be beneficial to understand the full costs and benefits of the approach to hospitals and the NHS. As part of a larger-scale evaluation, implementation differences among trusts should be explored, as these can provide valuable insights into what works for whom and under what conditions. The effect of implementation differences on patient experiences of relational aspects of care should also be considered as part of a large-scale evaluation.
Further research should also explore the acceptability of the survey instrument designed to measure relational aspects of care in different hospital settings. In addition, as a result of the limited diversity in the patient population who completed the survey as part of this study, future research is needed to validate the survey instrument with other patient populations, including those of different ethnicities and in other clinical areas.
Additional research may also consider whether or not existing instruments already used within the NHS could be adapted to include a greater focus on relational aspects of care and for use with a NRTF approach. These considerations are especially important given the financial and time constraints of the NHS and its workforce.
Acknowledgements
We would like to thank the volunteers who collected feedback from patients over the 10-month data collection period. We would also like to thank the patients who supported us with our survey development by participating in focus groups, interviews and testing the draft survey. Your support has been invaluable to this work.
We acknowledge our advisory group members, who have provided us with expert advice and guidance on numerous occasions throughout the project. Thank you for making space in your busy schedules to support us. Thank you also to the staff at our six case study sites for their great collaboration and communication with us over the past 30 months. It has been a pleasure to work with you.
Finally, we would like to thank our programme manager, Ms Margarida Marques-Resende, for her expert guidance and support. We also thank the Patient Feedback Team at Picker for their technical and implementation support during the patient data collection.
Contributions of authors
Chris Graham (chief investigator, corresponding author and Director of Research and Policy) oversaw the implementation of the project and contributed to all aspects of the research. He actively participated in activities held at the case study sites and collaborated continuously with the project advisory group.
Susanne Käsbauer (project manager, Senior Research Associate) managed the day-to-day implementation of the project and contributed to all aspects of the project. She wrote Chapters 1–7 and produced the full report.
Robyn Cooper (Research Associate) contributed to survey development, volunteer training, trust workshops, running weekly data reports, support for patient, staff and volunteer surveys, staff and volunteer data collection and analysis, toolkit development and dissemination of findings. She contributed to the writing of Chapters 2–7 and assisted with the production of the full report.
Jenny King (Associate Director of Research and Policy) assisted with the survey development, data collection and reporting of the project. She was involved throughout the project and provided advice and guidance to keep the project on track with the timetable.
Steve Sizmur (Chief Statistician) conducted secondary analysis of the national survey data and analysis of the near real-time patient experience data. He wrote sections in Chapters 2–4 pertaining to the survey development (factor analysis and composite scores) and patient experience data analysis.
Crispin Jenkinson (co-applicant, Professor of Health Services Research) contributed to all phases of the research. He provided expertise and guidance to keep the project on track and ensure the case study sites benefited from the research. He also worked with the advisory group and the case study sites throughout the project.
Laura Kelly (Research Officer) contributed to the survey development, co-led volunteer training and contributed to participatory workshops, analysis of the near real-time patient experience data, toolkit development and dissemination of findings.
Publications
Käsbauer S, Cooper R, Kelly L, King J. Barriers and facilitators of a near real-time feedback approach for measuring patient experiences of hospital care. Health Policy Technol 2016;6:51–8.
Cooper R, Käsbauer S, Ainley E, Tallett A. Thematic analysis of free-text comments provided in a compassionate care questionnaire. Qual Prim Health Care 2017;1:1–6.
Data sharing statement
Requests for access to data should be addressed to the corresponding author, Mr Chris Graham.
Patient data
This work uses data provided by patients and collected by the NHS as part of their care and support. Using patient data is vital to improve health and care for everyone. There is huge potential to make better use of information from people’s patient records, to understand more about disease, develop new treatments, monitor safety, and plan NHS services. Patient data should be kept safe and secure, to protect everyone’s privacy, and it’s important that there are safeguards to make sure that it is stored and used responsibly. Everyone should be able to find out about how patient data are used. #datasaveslives You can find out more about the background to this citation here: https://understandingpatientdata.org.uk/data-citation.
Disclaimers
This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health and Social Care.
References
- Robert G, Cornwell J, Brearley S. What Matters to Patients? Developing the Evidence Base for Measuring and Improving Patient Experience 2011. http://webarchive.nationalarchives.gov.uk/*/http://www.institute.nhs.uk/patient_experience/guide/the_patient_experience_research.html (accessed 11 November 2017).
- Davies E, Shaller D, Edgman-Levitan S, Safran DG, Oftedahl G, Sakowski J, et al. Evaluating the use of a modified CAHPS® survey to support improvements in patient-centred care: lessons from a quality improvement collaborative. Health Expect 2008;11:160-76. https://doi.org/10.1111/j.1369-7625.2007.00483.x.
- Brown H, Davidson D, Ellins J. NHS West Midlands Investing for Health Real-time Patient Feedback Project. Birmingham: University of Birmingham; 2009.
- The Mid Staffordshire NHS Foundation Trust . Mid Staffordshire NHS Foundation Trust Public Inquiry. Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry: Executive Summary 2013. www.gov.uk/government/uploads/system/uploads/attachment_data/file/279124/0947.pdf (accessed 11 October 2016).
- Bridges J, Flatley M, Meyer J. Older people’s and relatives’ experiences in acute care settings: systematic review and synthesis of qualitative studies. Int J Nurs Stud 2010;47:89-107. https://doi.org/10.1016/j.ijnurstu.2009.09.009.
- Gribble RK, Haupt C. Quantitative and qualitative differences between handout and mailed patient satisfaction surveys. Med Care 2005;43:276-81. https://doi.org/10.1097/00005650-200503000-00010.
- The Healthcare Commission . A Snapshot of Hospital Cleanliness in England 2005. http://news.bbc.co.uk/1/shared/bsp/hi/pdfs/15_12_05_hospital_cleanliness_snapshot.pdf (accessed 11 October 2016).
- Crocker JC, Beecham E, Kelly P, Dinsdale AP, Hemsley J, Jones L, et al. Inviting parents to take part in paediatric palliative care research: a mixed-methods examination of selection bias. Palliat Med 2015;29:231-40. https://doi.org/10.1177/0269216314560803.
- Larsen D, Peters H, Keast J. Using real time patient feedback to introduce safety changes: Debra Larsen and colleagues describe how the use of a process based on the ‘plan, do, study, act’ model has raised staff morale and improved care. Nurs Manag n.d.;18:27-31. https://doi.org/10.7748/nm2011.10.18.6.27.c8718.
- Lomas C. Instant patient feedback improves care. Nursing Times 2009. www.nursingtimes.net/whats-new-in-nursing/instant-patient-feedback-improvescare/1986679.article (accessed 11 October 2016).
- Simpson C. Realtime views of patient feedback. Nurs Stand 2013;27.
- Reeves R, West E, Barron D. Facilitated patient experience feedback can improve nursing care: a pilot study for a phase III cluster randomised controlled trial. BMC Health Serv Res 2013;13. https://doi.org/10.1186/1472-6963-13-259.
- Tzeng HM, Ketefian S, Redman RW. Relationship of nurses’ assessment of organizational culture, job satisfaction, and patient satisfaction with nursing care. Int J Nurs Stud 2002;39:79-84. https://doi.org/10.1016/S0020-7489(00)00121-8.
- Peltier J, Dahl A, Mulhern F. The relationship between employee satisfaction and hospital patient experiences. Forum People Perform Manag Meas 2009;1:1-29.
- Borrill C, West M, Carter M, Dawson J. The Relationship Between Staff Satisfaction and Patient Satisfaction: Results from Wolverhampton Hospitals NHS Trust. Birmingham: Aston University; 2003.
- Department of Health . Does the Experience of Staff Working in the NHS Link to the Patient Experience of Care? n.d. http://webarchive.nationalarchives.gov.uk/+/www.dh.gov.uk/en/Managingyourorganisation/Workforce/NHSStaffExperience/index.htm (accessed 11 October 2016).
- Barnard M, Stoll N. Organisational Change Management: A Rapid Literature Review. Short Policy Report No. 10/01. Bristol: Centre for Understanding Behaviour Change, University of Bristol; 2010.
- Kotter JP. Leading Change. Boston, MA: Harvard Business School Press; 1996.
- Kanter RM, Stein BA, Jick TD. The Challenge of Organizational Change. New York, NY: The Free Press; 1992.
- Luecke R. Managing Change and Transition. Boston, MA: Harvard Business School Press; 2003.
- Mannion R, Davies HT, Marshall MN. Cultural characteristics of ‘high’ and ‘low’ performing hospitals. J Health Organ Manag 2005;19:431-9. https://doi.org/10.1108/14777260510629689.
- Mannion R, Davies H, Konteh F, Jung T, Scott T, Bower P, et al. Measuring and Assessing Organisational Culture in the NHS (OC1). Report to the National Institute for Health Research Service Delivery and Organisation Programme 2008. www.netscc.ac.uk/hsdr/files/project/SDO_FR_08-1501-091_V01.pdf (accessed 11 October 2016).
- De Silva D. Helping Measure Person-Centred Care 2014. www.health.org.uk/publication/helping-measure-person-centred-care (accessed 11 October 2016).
- Darzi A. Quality and the NHS next stage review. Lancet 2008;371:1563-4. https://doi.org/10.1016/S0140-6736(08)60672-8.
- Vincent C, Amalberti R. Safer Healthcare: Strategies for the Real World. London: Springer Open; 2016.
- Black N, Varaganum M, Hutchings A. Relationship between patient reported experience (PREMs) and patient reported outcomes (PROMs) in elective surgery. BMJ Qual Saf 2014;23:534-42. https://doi.org/10.1136/bmjqs-2013-002707.
- Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open 2013;3. https://doi.org/10.1136/bmjopen-2012-001570.
- Anhang Price R, Elliott MN, Zaslavsky AM, Hays RD, Lehrman WG, Rybowski L, et al. Med Care Res Rev 2014;71:522-54. https://doi.org/10.1177/1077558714541480.
- Truths H. The Journey to Putting Patients First: Volume One of the Government Response to the Mid Staffordshire NHS Foundation Trust Public Inquiry 2. London: Department of Health; 2014.
- Murrells T, Robert G, Adams M, Morrow E, Maben J. Measuring relational aspects of hospital care in England with the ‘Patient Evaluation of Emotional Care during Hospitalisation’ (PEECH) survey questionnaire. BMJ 2013;3. https://doi.org/10.1136/bmjopen-2012-002211.
- Mercer SW, Maxwell M, Heaney D, Watt GC. The consultation and relational empathy (CARE) measure: development and preliminary validation and reliability of an empathy-based consultation process measure. Fam Pract 2004;21:699-705. https://doi.org/10.1093/fampra/cmh621.
- Williams AM, Kristjanson LJ. Emotional care experienced by hospitalised patients: development and testing of a measurement instrument. J Clin Nurs 2009;18:1069-77. https://doi.org/10.1111/j.1365-2702.2008.02586.x.
- Campbell J, Smith P, Nissen S, Bower P, Elliott M, Roland M. The GP Patient Survey for use in primary care in the National Health Service in the UK – development and psychometric characteristics. BMC Fam Pract 2009;10. https://doi.org/10.1186/1471-2296-10-57.
- Giordano LA, Elliott MN, Goldstein E, Lehrman WG, Spencer PA. Development, implementation, and public reporting of the HCAHPS survey. Med Care Res Rev 2010;67:27-3. https://doi.org/10.1177/1077558709341065.
- 2012 NHS Emergency Department Survey n.d. https://discover.ukdataservice.ac.uk/catalogue/?sn=7406&type=Data%20catalogue (accessed 11 October 2016).
- 2013 NHS Adult Inpatient Survey n.d. https://discover.ukdataservice.ac.uk/catalogue/?sn=7509&type=Data%20catalogue (accessed 11 October 2016).
- Care Quality Commission . Raising Standards, Putting People First: Our Strategy for 2013–2016 2013. www.cqc.org.uk/sites/default/files/documents/20130503_cqc_strategy_2013_final_cm_tagged.pdf (accessed 11 October 2016).
- NHS England . The Keogh Urgent and Emergency Care Review 2015. www.nhs.uk/NHSEngland/keogh-review/Documents/UECR.Ph1Report.FV.pdf (accessed 11 October 2016).
- Tadd W, Hillman A, Calnan S, Calnan M, Bayer T, Read S. Dignity in Practice: An Exploration of the Care of Older Adults in Acute NHS Trusts. NIHR SDO Report 2011. www.bgs.org.uk/pdf_cms/reference/Tadd_Dignity_in_Practice.pdf (accessed 11 October 2016).
- Zhang P. Multiple imputation: theory and method. Int Stat Rev 2003;71:581-92. http://dx.doi.org/10.1111/j.1751-5823.2003.tb00213.x.
- Gorsuch RL, Schinka JA, Velicer WF. Handbook of Psychology: Volume 2 Research Methods in Psychology. Hoboken, NJ: John Wiley; 2003.
- McDonald RP. Test Theory: A Unified Treatment. Mahwah, NJ: L. Erlbaum Associates; 1999.
- Beatty P, Presser S, Rothgeb M, Couper MP, Lessler JT, Martin E, et al. Methods for Testing and Evaluating Survey Questionnaires. Hoboken, NJ: John Wiley; 2004.
- Willis G. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage; 2005.
- Tourangeau R, Jabine T, Straf M, Tanur J, Tourangeau R. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: National Academies Press; 1984.
- Picker . After Francis Policy Commentary n.d. www.picker.org/news/francis-policy-commentary/ (accessed 22 November 2017).
- Raleigh VS, Hussey D, Secombe I, Qi R. Do associations between staff and inpatient feedback have the potential for improving patient experience? An analysis of surveys in NHS acute trusts in England. Qual Saf Health Care 2009;18:347-54. https://doi.org/10.1136/qshc.2008.028910.
- Fabrigar LR, Wegener DT. Understanding Statistics: Exploratory Factor Analysis. New York, NY: Oxford University Press; 2012.
- McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psych Methods 1996;1. https://doi.org/10.1037/1082-989X.1.1.30.
- Cronbach LJ, Gleser GC, Nanda H, Rajaratnam N. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. New York, NY: John Wiley & Sons; 1972.
- Jeon MJ, Lee G, Hwang JW, Kang SJ. Estimating reliability of school-level scores using multilevel and generalizability theory models. Asia Pacific Ed Rev 2009;10:149-58. https://doi.org/10.1007/s12564-009-9014-3.
- Zumbo BD. A Handbook on the Theory and Methods of Differential Item Functioning (DIF). Ottawa, ON: National Defense Headquarters; 1999.
- Käsbauer S, Cooper R, Kelly L, King J. Barriers and facilitators of a near real-time feedback approach for measuring patient experiences of hospital care. Health Policy Technol 2016;6:51-8. https://doi.org/10.1016/j.hlpt.2016.09.003.
- Brostow DP, Hirsch AT, Kurzer MS. Recruiting older patients with peripheral arterial disease: evaluating challenges and strategies. Patient Prefer Adherence 2015;9:1121-8. https://doi.org/10.2147/PPA.S83306.
- Johnson R, Kuczawski M, Mason S. Why is it so difficult to recruit patients to research in emergency care? Lessons from the AHEAD study. Emerg Med J 2016;33:52-6. https://doi.org/10.1136/emermed-2014-204401.
- Naylor C, Mundle C, Weaks L, Buck C. Volunteering in Health and Care: Securing a Sustainable Future. London: The King’s Fund; 2013.
- Wright C, Davey A, Elmore N, Carter M, Mounce L, Wilson E, et al. Patients’ use and views of real-time feedback technology in general practice. Health Expect 2017;20:419-33. https://doi.org/10.1111/hex.12469.
Appendix 1 Survey of patient experiences
Appendix 2 Scored item frequencies
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, all staff have introduced themselves | 3017 | 77.6 | 78.1 | 78.1 |
| Yes, some staff have introduced themselves | 716 | 18.4 | 18.5 | 96.6 |
| No, staff have not introduced themselves | 132 | 3.4 | 3.4 | 100.0 |
| Total | 3865 | 99.4 | 100.0 | |
| Missing | ||||
| No answer | 24 | 0.6 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, definitely | 1894 | 48.7 | 50.9 | 50.9 |
| Yes, to some extent | 1362 | 35.0 | 36.6 | 87.6 |
| No | 462 | 11.9 | 12.4 | 100.0 |
| Total | 3718 | 95.6 | 100.0 | |
| Missing | ||||
| Do not know | 147 | 3.8 | ||
| No answer | 24 | 0.6 | ||
| Total | 171 | 4.4 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, always | 3289 | 84.6 | 84.8 | 84.8 |
| Yes, to some extent | 526 | 13.5 | 13.6 | 98.4 |
| No | 64 | 1.6 | 1.6 | 100.0 |
| Total | 3879 | 99.7 | 100.0 | |
| Missing | ||||
| No answer | 10 | 0.3 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, all of the time | 3178 | 81.7 | 83.9 | 83.9 |
| Yes, some of the time | 563 | 14.5 | 14.9 | 98.8 |
| No | 45 | 1.2 | 1.2 | 100.0 |
| Total | 3786 | 97.4 | 100.0 | |
| Missing | ||||
| Do not know | 35 | 0.9 | ||
| No answer | 68 | 1.7 | ||
| Total | 103 | 2.6 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, definitely | 3132 | 80.5 | 81.6 | 81.6 |
| Yes, to some extent | 631 | 16.2 | 16.4 | 98.1 |
| No | 74 | 1.9 | 1.9 | 100.0 |
| Total | 3837 | 98.7 | 100.0 | |
| Missing | ||||
| No answer | 52 | 1.3 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, definitely | 2882 | 74.1 | 75.0 | 75.0 |
| Yes, to some extent | 767 | 19.7 | 20.0 | 95.0 |
| No | 192 | 4.9 | 5.0 | 100.0 |
| Total | 3841 | 98.8 | 100.0 | |
| Missing | ||||
| No answer | 48 | 1.2 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, always | 2359 | 60.7 | 79.3 | 79.3 |
| Yes, sometimes | 555 | 14.3 | 18.7 | 98.0 |
| No, never | 60 | 1.5 | 2.0 | 100.0 |
| Total | 2974 | 76.5 | 100.0 | |
| Missing | ||||
| I have not needed attention | 862 | 22.2 | ||
| No answer | 53 | 1.4 | ||
| Total | 915 | 23.5 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, always | 1955 | 50.3 | 67.7 | 67.7 |
| Yes, to some extent | 803 | 20.6 | 27.8 | 95.5 |
| No | 129 | 3.3 | 4.5 | 100.0 |
| Total | 2887 | 74.2 | 100.0 | |
| Missing | ||||
| I have not asked for help | 912 | 23.5 | ||
| Do not know | 25 | 0.6 | ||
| No answer | 65 | 1.7 | ||
| Total | 1002 | 25.8 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, definitely | 1173 | 30.2 | 70.5 | 70.5 |
| Yes, to some extent | 359 | 9.2 | 21.6 | 92.0 |
| No | 133 | 3.4 | 8.0 | 100.0 |
| Total | 1665 | 42.8 | 100.0 | |
| Missing | ||||
| I have not raised any concerns | 2088 | 53.7 | ||
| Do not know | 57 | 1.5 | ||
| No answer | 79 | 2.0 | ||
| Total | 2224 | 57.2 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, always | 3370 | 86.7 | 89.4 | 89.4 |
| Yes, to some extent | 361 | 9.3 | 9.6 | 98.9 |
| No | 40 | 1.0 | 1.1 | 100.0 |
| Total | 3771 | 97.0 | 100.0 | |
| Missing | ||||
| Do not know | 39 | 1.0 | ||
| No answer | 79 | 2.0 | ||
| Total | 118 | 3.0 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, definitely | 2744 | 70.6 | 74.7 | 74.7 |
| Yes, to some extent | 709 | 18.2 | 19.3 | 94.0 |
| No | 220 | 5.7 | 6.0 | 100.0 |
| Total | 3673 | 94.4 | 100.0 | |
| Missing | ||||
| I have not seen a doctor or nurse | 104 | 2.7 | ||
| No answer | 112 | 2.9 | ||
| Total | 216 | 5.6 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, definitely | 2024 | 52.0 | 73.9 | 73.9 |
| Yes, to some extent | 374 | 9.6 | 13.7 | 87.6 |
| No | 340 | 8.7 | 12.4 | 100.0 |
| Total | 2738 | 70.4 | 100.0 | |
| Missing | ||||
| There has been no need to inform my family or carers | 899 | 23.1 | ||
| There are no family or carers to be informed | 124 | 3.2 | ||
| No answer | 128 | 3.3 | ||
| Total | 1151 | 29.6 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, definitely | 2620 | 67.4 | 72.8 | 72.8 |
| Yes, to some extent | 727 | 18.7 | 20.2 | 93.0 |
| No | 250 | 6.4 | 7.0 | 100.0 |
| Total | 3597 | 92.5 | 100.0 | |
| Missing | ||||
| I have not been well enough to be involved in decisions about my care | 129 | 3.3 | ||
| No answer | 163 | 4.2 | ||
| Total | 292 | 7.5 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, completely | 2686 | 69.1 | 77.1 | 77.1 |
| Yes, to some extent | 578 | 14.9 | 16.6 | 93.7 |
| No | 219 | 5.6 | 6.3 | 100.0 |
| Total | 3483 | 89.6 | 100.0 | |
| Missing | ||||
| I have not needed an explanation | 257 | 6.6 | ||
| No answer | 149 | 3.8 | ||
| Total | 406 | 10.4 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, completely | 2760 | 71.0 | 83.0 | 83.0 |
| Yes, to some extent | 459 | 11.8 | 13.8 | 96.8 |
| No | 105 | 2.7 | 3.2 | 100.0 |
| Total | 3324 | 85.5 | 100.0 | |
| Missing | ||||
| I have not had any questions | 407 | 10.5 | ||
| No answer | 158 | 4.1 | ||
| Total | 565 | 14.5 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, completely | 1318 | 33.9 | 65.3 | 65.3 |
| Yes, to some extent | 424 | 10.9 | 21.0 | 86.4 |
| No | 275 | 7.1 | 13.6 | 100.0 |
| Total | 2017 | 51.9 | 100.0 | |
| Missing | ||||
| I have not had any anxieties or fears | 1698 | 43.7 | ||
| No answer | 174 | 4.5 | ||
| Total | 1872 | 48.1 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, completely | 1240 | 31.9 | 54.2 | 54.2 |
| Yes, to some extent | 395 | 10.2 | 17.3 | 71.5 |
| No | 653 | 16.8 | 28.5 | 100.0 |
| Total | 2288 | 58.8 | 100.0 | |
| Missing | ||||
| I have not needed this type of information | 1361 | 35.0 | ||
| No answer | 240 | 6.2 | ||
| Total | 1601 | 41.2 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, completely | 3356 | 86.3 | 90.9 | 90.9 |
| Yes, to some extent | 266 | 6.8 | 7.2 | 98.1 |
| No | 70 | 1.8 | 1.9 | 100.0 |
| Total | 3692 | 94.9 | 100.0 | |
| Missing | ||||
| Do not know | 39 | 1.0 | ||
| No answer | 158 | 4.1 | ||
| Total | 197 | 5.1 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, definitely | 3033 | 78.0 | 82.8 | 82.8 |
| Yes, to some extent | 521 | 13.4 | 14.2 | 97.0 |
| No | 109 | 2.8 | 3.0 | 100.0 |
| Total | 3663 | 94.2 | 100.0 | |
| Missing | ||||
| Do not know | 54 | 1.4 | ||
| No answer | 172 | 4.4 | ||
| Total | 226 | 5.8 | ||
| Total | 3889 | 100.0 | ||
| Response | Frequency (n) | Respondents (%) | Valid (%) | Cumulative (%) |
|---|---|---|---|---|
| Valid | ||||
| Yes, all of the time | 3445 | 88.6 | 92.8 | 92.8 |
| Yes, some of the time | 222 | 5.7 | 6.0 | 98.8 |
| No | 46 | 1.2 | 1.2 | 100.0 |
| Total | 3713 | 95.5 | 100.0 | |
| Missing | ||||
| No answer | 176 | 4.5 | ||
| Total | 3889 | 100.0 | ||
Appendix 3 Factor analysis output
Appendix 4 Short-form survey
Appendix 5 Factor analysis output (reduced item set)
Appendix 6 Comparison of missing data options
FIGURE 6.
Scatterplot matrix and intraclass correlation for missing data options. Intraclass correlation = 0.984 (two-way mixed model, absolute agreement). CoreItems.1, score computed using at least one completed item; CoreItems.All, replace missing item scores with whole-survey mean; and CoreItems.Wd, replace missing item scores with ward mean.
FIGURE 7.
Scatterplot matrix and intraclass correlation for missing data options. Intraclass correlation = 0.994 (two-way mixed model, absolute agreement). CoreItems.6, score computed using at least six completed items; CoreItems.All, replace missing item scores with whole-survey mean; and CoreItems.Wd, replace missing item scores with ward mean.
FIGURE 8.
Scatterplot matrix and intraclass correlation for missing data options. Intraclass correlation = 0.997 (two-way mixed model, absolute agreement). CoreItems.8, score computed using at least eight completed items; CoreItems.All, replace missing item scores with whole-survey mean; and CoreItems.Wd, replace missing item scores with ward mean.
Appendix 7 Score variation mode coefficients
| Score variation source | n | Coefficient | SE | CI | p-value | |
|---|---|---|---|---|---|---|
| 2.5% | 97.5% | |||||
| Intercept | 3333 | 83.507 | 1.011 | 81.526 | 85.487 | 0.000 |
| Who responded (reference: the patient)? | 1365 | |||||
| A friend or relative of the patient | 259 | –3.614 | 1.088 | –5.746 | –1.482 | 0.001 |
| Both patient and friend/relative together | 112 | –3.157 | 1.507 | –6.110 | –0.204 | 0.036 |
| The patient and the volunteer together | 1597 | –0.230 | 0.661 | –1.524 | 1.065 | 0.728 |
| Sex (reference: male) | 1482 | |||||
| Female | 1851 | –1.636 | 0.535 | –2.684 | –0.587 | 0.002 |
| Age group, years (reference: 18–30 years) | 316 | |||||
| 31–40 | 222 | 4.409 | 1.320 | 1.821 | 6.996 | 0.001 |
| 41–50 | 313 | 2.636 | 1.200 | 0.284 | 4.988 | 0.028 |
| 51–60 | 375 | 5.559 | 1.166 | 3.275 | 7.844 | 0.000 |
| 61–70 | 467 | 7.850 | 1.141 | 5.614 | 10.086 | 0.000 |
| 71–80 | 663 | 6.411 | 1.099 | 4.257 | 8.565 | 0.000 |
| 81–90 | 754 | 6.795 | 1.124 | 4.593 | 8.998 | 0.000 |
| 91–120 | 223 | 5.807 | 1.431 | 3.003 | 8.611 | 0.000 |
| Chronic conditions (reference: absence of each condition) | ||||||
| Deafness or severe hearing impairment | 440 | 0.052 | 0.827 | –1.570 | 1.673 | 0.950 |
| Blindness or partial sight | 184 | –2.259 | 1.182 | –4.576 | 0.058 | 0.056 |
| A long-standing physical condition | 676 | –0.983 | 0.672 | –2.300 | 0.334 | 0.144 |
| A learning disability | 48 | –7.309 | 2.220 | –11.659 | –2.959 | 0.001 |
| A mental health condition | 139 | –7.864 | 1.337 | –10.484 | –5.243 | 0.000 |
| Dementia | 92 | –1.200 | 1.674 | –4.480 | 2.080 | 0.473 |
| A long-standing illness | 935 | 0.428 | 0.615 | –0.777 | 1.633 | 0.486 |
| Ethnic group [reference: white (British)] | 3209 | |||||
| Irish | 14 | 4.234 | 4.073 | –3.749 | 12.218 | 0.299 |
| Gypsy or Irish traveller | 8 | –7.949 | 5.309 | –18.355 | 2.456 | 0.134 |
| White and black Caribbean | 4 | –4.019 | 7.561 | –18.838 | 10.800 | 0.595 |
| White and black African | 4 | –1.216 | 7.576 | –16.066 | 13.633 | 0.872 |
| White and Asian | 4 | 15.299 | 7.577 | 0.448 | 30.150 | 0.043 |
| Indian | 8 | –1.484 | 5.295 | –11.862 | 8.893 | 0.779 |
| Pakistani | 6 | –20.076 | 6.227 | –32.281 | –7.871 | 0.001 |
| Bangladeshi | 8 | 5.807 | 5.324 | –4.627 | 16.242 | 0.275 |
| Chinese | 7 | –1.955 | 5.705 | –13.137 | 9.226 | 0.732 |
| African | 8 | –3.684 | 5.332 | –14.135 | 6.766 | 0.490 |
| Caribbean | 6 | 2.945 | 6.104 | –9.019 | 14.909 | 0.629 |
| Arab | 2 | –19.196 | 10.586 | –39.944 | 1.551 | 0.070 |
| Any other ethnic group | 45 | 1.683 | 2.264 | –2.754 | 6.120 | 0.457 |
Appendix 8 Differential item functioning analysis (patient versus friend/relative)
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 392.151 | 1 | 0.329 | 0.221 | |
| + uniform DIF | Chi-squared | 392.157 | 2 | 0.329 | 0.221 | |
| + non-uniform DIF | Chi-squared | 393.204 | 3 | 0.33 | 0.221 | |
| Difference chi-squared | 1.052 | 2 | 0.295 | |||
| Effect size | 0.001 | 0.001 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 744.622 | 1 | 0.454 | 0.259 | |
| + uniform DIF | Chi-squared | 756.818 | 2 | 0.46 | 0.263 | |
| + non-uniform DIF | Chi-squared | 762.848 | 3 | 0.463 | 0.265 | |
| Difference chi-squared | 18.226 | 2 | < 0.001 | |||
| Effect size | 0.009 | 0.006 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 668.761 | 1 | 0.566 | 0.445 | |
| + uniform DIF | Chi-squared | 670.539 | 2 | 0.567 | 0.446 | |
| + non-uniform DIF | Chi-squared | 670.666 | 3 | 0.567 | 0.446 | |
| Difference chi-squared | 1.905 | 2 | 0.193 | |||
| Effect size | 0.001 | 0.001 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 656.149 | 1 | 0.573 | 0.457 | |
| + uniform DIF | Chi-squared | 661.147 | 2 | 0.576 | 0.461 | |
| + non-uniform DIF | Chi-squared | 661.517 | 3 | 0.577 | 0.461 | |
| Difference chi-squared | 5.368 | 2 | 0.034 | |||
| Effect size | 0.004 | 0.004 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 553.689 | 1 | 0.515 | 0.407 | |
| + uniform DIF | Chi-squared | 554.426 | 2 | 0.516 | 0.408 | |
| + non-uniform DIF | Chi-squared | 554.469 | 3 | 0.516 | 0.408 | |
| Difference chi-squared | 0.78 | 2 | 0.339 | |||
| Effect size | 0.001 | 0.001 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 697.051 | 1 | 0.503 | 0.351 | |
| + uniform DIF | Chi-squared | 708.98 | 2 | 0.51 | 0.357 | |
| + non-uniform DIF | Chi-squared | 712.465 | 3 | 0.512 | 0.359 | |
| Difference chi-squared | 15.415 | 2 | < 0.001 | |||
| Effect size | 0.009 | 0.008 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 468.164 | 1 | 0.515 | 0.426 | |
| + uniform DIF | Chi-squared | 470.623 | 2 | 0.517 | 0.429 | |
| + non-uniform DIF | Chi-squared | 470.903 | 3 | 0.517 | 0.429 | |
| Difference chi-squared | 2.738 | 2 | 0.127 | |||
| Effect size | 0.003 | 0.002 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 632.097 | 1 | 0.468 | 0.322 | |
| + uniform DIF | Chi-squared | 632.218 | 2 | 0.468 | 0.322 | |
| + non-uniform DIF | Chi-squared | 632.482 | 3 | 0.468 | 0.322 | |
| Difference chi-squared | 0.386 | 2 | 0.412 | |||
| Effect size | 0 | 0 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 693.372 | 1 | 0.489 | 0.332 | |
| + uniform DIF | Chi-squared | 694.759 | 2 | 0.49 | 0.333 | |
| + non-uniform DIF | Chi-squared | 697.601 | 3 | 0.492 | 0.334 | |
| Difference chi-squared | 4.229 | 2 | 0.06 | |||
| Effect size | 0.002 | 0.002 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 564.53 | 1 | 0.558 | 0.458 | |
| + uniform DIF | Chi-squared | 564.533 | 2 | 0.558 | 0.458 | |
| + non-uniform DIF | Chi-squared | 564.548 | 3 | 0.558 | 0.458 | |
| Difference chi-squared | 0.018 | 2 | 0.496 | |||
| Effect size | 0 | 0 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 925.616 | 1 | 0.673 | 0.539 | |
| + uniform DIF | Chi-squared | 930.266 | 2 | 0.675 | 0.542 | |
| + non-uniform DIF | Chi-squared | 931.755 | 3 | 0.676 | 0.543 | |
| Difference chi-squared | 6.139 | 2 | 0.023 | |||
| Effect size | 0.003 | 0.004 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 505.614 | 1 | 0.626 | 0.554 | |
| + uniform DIF | Chi-squared | 505.988 | 2 | 0.627 | 0.554 | |
| + non-uniform DIF | Chi-squared | 507.941 | 3 | 0.629 | 0.557 | |
| Difference chi-squared | 2.327 | 2 | 0.156 | |||
| Effect size | 0.002 | 0.003 |
Appendix 9 Differential item functioning analyses (white British versus other ethnic group)
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 774.216 | 1 | 0.329 | 0.221 | |
| + uniform DIF | Chi-squared | 775.383 | 2 | 0.329 | 0.222 | |
| + non-uniform DIF | Chi-squared | 775.514 | 3 | 0.329 | 0.222 | |
| Difference chi-squared | 1.298 | 2 | 0.261 | |||
| Effect size | 0.000 | 0.000 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 1436.611 | 1 | 0.443 | 0.249 | |
| + uniform DIF | Chi-squared | 1439.881 | 2 | 0.444 | 0.25 | |
| + non-uniform DIF | Chi-squared | 1445.235 | 3 | 0.445 | 0.251 | |
| Difference chi-squared | 8.624 | 2 | 0.007 | |||
| Effect size | 0.002 | 0.001 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 1113.054 | 1 | 0.538 | 0.434 | |
| + uniform DIF | Chi-squared | 1113.397 | 2 | 0.539 | 0.434 | |
| + non-uniform DIF | Chi-squared | 1113.398 | 3 | 0.539 | 0.434 | |
| Difference chi-squared | 0.344 | 2 | 0.421 | |||
| Effect size | 0.000 | 0.000 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 1151.983 | 1 | 0.537 | 0.427 | |
| + uniform DIF | Chi-squared | 1152.927 | 2 | 0.537 | 0.428 | |
| + non-uniform DIF | Chi-squared | 1154.451 | 3 | 0.538 | 0.428 | |
| Difference chi-squared | 2.469 | 2 | 0.145 | |||
| Effect size | 0.001 | 0.001 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 1065.286 | 1 | 0.479 | 0.364 | |
| + uniform DIF | Chi-squared | 1066.345 | 2 | 0.479 | 0.365 | |
| + non-uniform DIF | Chi-squared | 1067.983 | 3 | 0.48 | 0.365 | |
| Difference chi-squared | 2.697 | 2 | 0.13 | |||
| Effect size | 0.001 | 0.001 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 1216.174 | 1 | 0.463 | 0.321 | |
| + uniform DIF | Chi-squared | 1216.781 | 2 | 0.463 | 0.321 | |
| + non-uniform DIF | Chi-squared | 1221.43 | 3 | 0.464 | 0.322 | |
| Difference chi-squared | 5.256 | 2 | 0.036 | |||
| Effect size | 0.002 | 0.001 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 697.74 | 1 | 0.424 | 0.347 | |
| + uniform DIF | Chi-squared | 700.447 | 2 | 0.426 | 0.349 | |
| + non-uniform DIF | Chi-squared | 700.584 | 3 | 0.426 | 0.349 | |
| Difference chi-squared | 2.844 | 2 | 0.121 | |||
| Effect size | 0.002 | 0.001 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 1136.923 | 1 | 0.426 | 0.283 | |
| + uniform DIF | Chi-squared | 1137.028 | 2 | 0.426 | 0.283 | |
| + non-uniform DIF | Chi-squared | 1138.706 | 3 | 0.426 | 0.283 | |
| Difference chi-squared | 1.782 | 2 | 0.205 | |||
| Effect size | 0.001 | 0.000 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 1273.025 | 1 | 0.455 | 0.299 | |
| + uniform DIF | Chi-squared | 1273.144 | 2 | 0.455 | 0.299 | |
| + non-uniform DIF | Chi-squared | 1274.043 | 3 | 0.455 | 0.3 | |
| Difference chi-squared | 1.018 | 2 | 0.301 | |||
| Effect size | 0.000 | 0.000 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 831.171 | 1 | 0.5 | 0.42 | |
| + uniform DIF | Chi-squared | 831.852 | 2 | 0.501 | 0.421 | |
| + non-uniform DIF | Chi-squared | 832.21 | 3 | 0.501 | 0.421 | |
| Difference chi-squared | 1.039 | 2 | 0.297 | |||
| Effect size | 0.001 | 0.001 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 1456.285 | 1 | 0.606 | 0.484 | |
| + uniform DIF | Chi-squared | 1456.331 | 2 | 0.606 | 0.484 | |
| + non-uniform DIF | Chi-squared | 1457.2 | 3 | 0.606 | 0.484 | |
| Difference chi-squared | 0.915 | 2 | 0.316 | |||
| Effect size | 0.000 | 0.000 |
| DIF type | Statistic | Value | df | p-value | Nagelkerke R2 | McFadden R2 |
|---|---|---|---|---|---|---|
| Scale only | Chi-squared | 821.817 | 1 | 0.572 | 0.506 | |
| + uniform DIF | Chi-squared | 822.347 | 2 | 0.572 | 0.506 | |
| + non-uniform DIF | Chi-squared | 822.35 | 3 | 0.572 | 0.506 | |
| Difference chi-squared | 0.533 | 2 | 0.383 | |||
| Effect size | 0.000 | 0.000 |
Appendix 10 Survey-level results by month
| Month | Mean score | SE | 95% CI |
|---|---|---|---|
| 1 June | 89.443 | 1.787 | 85.940 to 92.946 |
| 2 July | 84.413 | 0.905 | 82.640 to 86.186 |
| 3 August | 87.345 | 0.837 | 85.704 to 88.986 |
| 4 September | 86.695 | 0.804 | 85.119 to 88.272 |
| 5 October | 86.970 | 0.743 | 85.513 to 88.428 |
| 6 November | 89.523 | 0.787 | 87.979 to 91.066 |
| 7 December | 89.218 | 0.878 | 87.495 to 90.940 |
| 8 January | 88.223 | 0.768 | 86.717 to 89.730 |
| 9 February | 88.342 | 0.729 | 86.912 to 89.772 |
| 10 March | 89.575 | 0.746 | 88.111 to 91.038 |
Appendix 11 Ward-level scores and t-tests
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 22 | 75.42 | 20.73 | 4.42 | ||||
| Post intervention | 1 | 100.00 | 24.58 | 1.16 | 21 | 0.259 | |||
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 43 | 91.52 | 10.83 | 1.65 | ||||
| Post intervention | 104 | 91.90 | 8.82 | 0.87 | 0.38 | 104 | 91.90 | 8.82 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 23 | 74.88 | 22.29 | 4.65 | ||||
| Post intervention | 6 | 76.97 | 27.58 | 11.26 | 2.08 | 6 | 76.97 | 27.58 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 28 | 82.65 | 13.11 | 2.48 | ||||
| Post intervention | 3 | 93.06 | 4.81 | 2.78 | 10.41 | 3 | 93.06 | 4.81 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 84 | 87.84 | 13.29 | 1.46 | ||||
| Post intervention | 133 | 87.45 | 15.08 | 1.31 | –0.39 | 133 | 87.45 | 15.08 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 0 | |||||||
| Post intervention | 40 | 84.98 | 16.14 | 2.55 | 40 | 93.06 | 4.81 | ||
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 29 | 78.51 | 20.34 | 3.78 | ||||
| Post intervention | 71 | 86.74 | 16.90 | 2.01 | 8.23 | 2.08 | 99.00 | 0.040 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 15 | 80.66 | 14.89 | 3.84 | ||||
| Post intervention | 36 | 85.66 | 17.54 | 2.92 | 5.00 | 0.97 | 49.00 | 0.338 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre-intervention | 43 | 83.15 | 19.07 | 2.91 | ||||
| Post-intervention | 97 | 83.80 | 18.57 | 1.89 | 0.65 | 0.19 | 138.00 | 0.850 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 43 | 81.95 | 19.47 | 2.97 | ||||
| Post intervention | 73 | 84.40 | 16.91 | 1.98 | 2.45 | 0.71 | 114.00 | 0.477 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 24 | 76.19 | 16.00 | 3.27 | ||||
| Post intervention | 15 | 83.76 | 11.19 | 2.89 | 7.58 | 1.60 | 37.00 | 0.118 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 59 | 97.34 | 4.29 | 0.56 | ||||
| Post intervention | 62 | 95.71 | 7.23 | 0.92 | –1.63 | –1.50 | 119.00 | 0.137 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 37 | 85.33 | 22.08 | 3.63 | ||||
| Post intervention | 475 | 89.87 | 16.18 | 0.74 | 4.55 | 1.60 | 510.00 | 0.110 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 147 | 90.04 | 12.88 | 1.06 | ||||
| Post intervention | 192 | 88.51 | 14.53 | 1.05 | –1.53 | –1.01 | 337.00 | 0.315 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 54 | 76.72 | 21.25 | 2.89 | ||||
| Post intervention | 86 | 83.36 | 15.36 | 1.66 | 6.64 | 2.14 | 138.00 | 0.034 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 49 | 81.48 | 17.04 | 2.43 | ||||
| Post intervention | 78 | 88.81 | 12.95 | 1.47 | 7.32 | 2.74 | 125.00 | 0.007 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 76 | 88.44 | 15.34 | 1.76 | ||||
| Post intervention | 107 | 90.07 | 10.99 | 1.06 | 1.63 | 0.84 | 181.00 | 0.404 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 142 | 78.06 | 25.86 | 2.17 | ||||
| Post intervention | 46 | 69.80 | 34.14 | 5.03 | –8.26 | –1.73 | 186.00 | 0.085 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 31 | 92.32 | 9.34 | 1.68 | ||||
| Post intervention | 270 | 89.39 | 13.23 | 0.81 | –2.93 | –1.20 | 299.00 | 0.231 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 41 | 92.76 | 8.93 | 1.40 | ||||
| Post intervention | 36 | 88.35 | 14.29 | 2.38 | –4.38 | –1.65 | 75.00 | 0.104 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 36 | 82.75 | 18.33 | 3.06 | ||||
| Post intervention | 29 | 90.08 | 11.67 | 2.17 | 7.33 | 1.87 | 63.00 | 0.066 | |
| Scale | Period | n | Mean | SD | SE of the mean | Difference | t-test | df | Significance |
|---|---|---|---|---|---|---|---|---|---|
| Overall score: short form | Pre intervention | 245 | 92.64 | 11.02 | 0.70 | ||||
| Post intervention | 340 | 93.33 | 11.38 | 0.62 | 0.69 | 0.73 | 583.00 | 0.464 | |
Appendix 12 Breakdown of pre- and post-staff survey results by study/control ward
| Breakdown | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113a | 59a |
| Missing | – | – | – | 3.4 (6) | – | – |
| Trust | ||||||
| Hinchingbrooke | 28.2 (20) | 35.7 (15) | 17.2 (5) | 23.0 (41) | 32.7 (37) | 6.8 (4) |
| Milton Keynes | 4.2 (3) | 2.4 (1) | 6.9 (2) | – | – | – |
| North Cumbria | 14.1 (10) | 11.9 (5) | 17.2 (5) | 1.7 (3) | 0.9 (1) | 3.4 (2) |
| Northern Lincolnshire and Goole | 2.8 (2) | 2.4 (1) | 3.4 (1) | 29.2 (52) | 20.4 (23) | 49.2 (29) |
| Poole | 16.9 (12) | 9.5 (4) | 27.6 (8) | 28.1 (50) | 30.1 (34) | 27.1 (16) |
| Salisbury | 33.8 (24) | 38.1 (16) | 27.6 (8) | 14.6 (26) | 15.9 (18) | 13.6 (8) |
| What types of patient experience data have you collected in your trust during the year 2015? Select all that apply | Pre patient data collection, % (n) | ||
|---|---|---|---|
| Total | Study | Control | |
| Total | 71 | 42 | 29 |
| No reply | 1.4 (1) | – | 3.4 (1) |
| Response option | |||
| National surveys (e.g. the national survey, the FFT) | 84.5 (60) | 85.7 (36) | 82.8 (24) |
| Surveys developed specifically for your hospital, ward or department | 56.3 (40) | 52.4 (22) | 62.1 (18) |
| Informal patient feedback (e.g. through conversations with staff or volunteers) | 60.6 (43) | 59.5 (25) | 62.1 (18) |
| Not sure | 4.2 (3) | 4.8 (2) | 3.4 (1) |
| Other, please specify | 1.4 (1) | 2.4 (1) | – |
| What methods are used to collect patient experience data in your trust? Select all that apply | Pre patient data collection, % (n) | ||
|---|---|---|---|
| Total | Study | Control | |
| Total | 71 | 42 | 29 |
| No reply | 1.4 (1) | – | 3.4 (1) |
| Response option | |||
| Comment cards | 88.7 (63) | 88.1 (37) | 89.7 (26) |
| Conversations | 50.7 (36) | 57.1 (24) | 41.4 (12) |
| Focus groups | 18.3 (13) | 14.3 (6) | 24.1 (7) |
| Interviews | 12.7 (9) | 7.1 (3) | 20.7 (6) |
| Online surveys | 19.7 (14) | 23.8 (10) | 13.8 (4) |
| Paper surveys | 29.6 (21) | 26.2 (11) | 34.5 (10) |
| Tablet-based surveys | 4.2 (3) | 4.8 (2) | 3.4 (1) |
| Telephone surveys | 7.0 (5) | 7.1 (3) | 6.9 (2) |
| Not sure | 5.6 (4) | 4.8 (2) | 6.9 (2) |
| Other, please specify | 2.8 (2) | 2.4 (1) | 3.4 (1) |
| What parts of the patient experience have the data collection at your trust mainly focused on? Select all that apply | Pre patient data collection, % (n) | ||
|---|---|---|---|
| Total | Study | Control | |
| Total | 71 | 42 | 29 |
| No reply | 1.4 (1) | – | 3.45 (1) |
| Response option | |||
| Cleanliness | 54.9 (39) | 45.2 (19) | 69.0 (20) |
| Interactions with staff | 69.0 (49) | 66.7 (28) | 72.4 (21) |
| Medications | 22.5 (16) | 14.3 (6) | 34.5 (10) |
| Pain relief | 23.9 (17) | 21.4 (9) | 27.6 (8) |
| Recommendations to family and friends | 87.3 (62) | 90.5 (38) | 82.8 (24) |
| Relationships with staff (e.g. being treated with kindness, dignity and respect) | 76.1 (54) | 73.8 (31) | 79.3 (23) |
| Waiting times | 22.5 (16) | 26.2 (11) | 17.2 (5) |
| Other, please specify | 8.5 (6) | 4.8 (2) | 13.8 (4) |
| At what point in the patient journey is most of your data collection carried out? | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 1.4 (1) | – | 3.4 (1) | 3.9 (7) | 2.7 (3) | 6.8 (4) |
| Response option | ||||||
| On admission to the hospital | 2.8 (2) | 4.8 (2) | – | 7.9 (14) | 8.8 (10) | 6.8 (4) |
| During the patient’s stay | 33.8 (24) | 28.6 (12) | 41.4 (12) | 35.4 (63) | 38.9 (44) | 32.2 (19) |
| After the patient has left hospital | 14.1 (10) | 11.9 (5) | 17.2 (5) | 17.4 (31) | 13.3 (15) | 20.3 (12) |
| At all points of the patient journey | 25.4 (18) | 23.8 (10) | 27.6 (8) | 23.6 (42) | 23.9 (27) | 23.7 (14) |
| Other, please specify | 22.5 (16) | 31.0 (13) | 10.3 (3) | 11.8 (21) | 12.4 (14) | 10.2 (6) |
| What types of patient experience data have been collected in your ward or department during the years 2015–16? Select all that apply | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 179 | 113 | 59 |
| No reply | 1.4 (1) | – | 3.4 (1) | 5.1 (9) | 3.5 (4) | 8.5 (5) |
| Response option | ||||||
| National surveys (e.g. the national survey, the FFT) | 81.7 (58) | 83.3 (35) | 79.3 (23) | 62.4 (111) | 61.1 (69) | 66.1 (39) |
| Surveys developed specifically for your hospital, ward or department | 38.0 (27) | 35.7 (15) | 41.4 (12) | 32.6 (58) | 31.0 (35) | 30.5 (18) |
| Informal patient feedback (e.g. through conversations with staff or volunteers) | 52.1 (37) | 50.0 (21) | 55.2 (16) | 60.1 (107) | 61.1 (69) | 57.6 (34) |
| Other, please specify | 5.6 (4) | 4.8 (2) | 6.9 (2) | 6.2 (11) | 7.1 (8) | 5.1 (3) |
| What methods were used to collect patient experience data in your ward or department during the year 2015? Select all that apply | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 1.4 (1) | – | 3.4 (1) | – | – | – |
| Response option | ||||||
| Comment cards | 83.1 (59) | 83.3 (35) | 82.8 (24) | 88.2 (157) | 86.7 (98) | 89.8 (53) |
| Conversations | 49.3 (35) | 54.8 (23) | 41.4 (12) | 58.4 (104) | 57.5 (65) | 61.0 (36) |
| Focus groups | 8.5 (6) | 4.8 (2) | 13.8 (4) | 3.4 (6) | 2.7 (3) | 3.4 (2) |
| Interviews | 12.7 (9) | 4.8 (2) | 24.1 (7) | 3.9 (7) | 4.4 (5) | 3.4 (2) |
| Online surveys | 14.1 (10) | 19.0 (8) | 6.9 (2) | 10.7 (19) | 9.7 (11) | 11.9 (7) |
| Paper surveys | 26.8 (19) | 21.4 (9) | 34.5 (10) | 32.6 (58) | 35.4 (40) | 25.4 (15) |
| Tablet-based surveys | 7.0 (5) | 9.5 (4) | 3.4 (1) | 13.5 (24) | 18.6 (21) | 1.7 (1) |
| Telephone surveys | 5.6 (4) | 9.5 (4) | – | 3.9 (7) | 4.4 (5) | 3.4 (2) |
| Not sure | – | – | – | 3.4 (6) | 3.5 (4) | 3.4 (2) |
| Other, please specify | 4.2 (3) | 2.4 (1) | 69 (2) | 4.5 (8) | 6.2 (7) | 1.7 (1) |
| What parts of the patient experience have the data collection in your ward or department mainly focused on? Select all that apply | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 1.4 (1) | – | 3.4 (1) | 2.8 (5) | 4.4 (5) | – |
| Response option | ||||||
| Cleanliness | 50.7 (36) | 42.9 (18) | 62.1 (18) | 46.1 (82) | 40.7 (46) | 54.3 (32) |
| Interactions with staff | 70.4 (50) | 66.7 (28) | 75.9 (22) | 71.9 (128) | 69.0 (78) | 76.3 (45) |
| Medications | 23.9 (17) | 16.7 (7) | 34.5 (10) | 29.2 (52) | 23.9 (27) | 35.6 (21) |
| Pain relief | 28.2 (20) | 21.4 (9) | 37.9 (11) | 33.1 (59) | 27.4 (31) | 39.0 (23) |
| Recommendations to family and friends | 81.7 (58) | 85.7 (36) | 75.9 (22) | 69.7 (124) | 72.6 (82) | 61.0 (36) |
| Relationships with staff (e.g. being treated with kindness, dignity and respect) | 76.1 (54) | 73.8 (31) | 79.3 (23) | 66.9 (119) | 63.7 (72) | 71.2 (42) |
| Waiting times | 19.7 (14) | 23.8 (10) | 13.8 (4) | 37.6 (67) | 41.6 (47) | 30.5 (18) |
| Other, please specify | 8.5 (6) | 7.1 (3) | 10.3 (3) | 11.8 (21) | 14.2 (16) | 6.8 (4) |
| How are results from patient experience data collection communicated to staff in your ward or department? Select all that apply | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 2.8 (2) | 2.4 (1) | 3.4 (1) | 1.7 (3) | 1.8 (2) | 1.7 (1) |
| Response option | ||||||
| During staff meetings | 59.2 (42) | 54.8 (23) | 65.5 (19) | 53.4 (95) | 49.6 (56) | 57.6 (34) |
| On staff noticeboards | 62.0 (44) | 59.5 (25) | 65.5 (19) | 59.0 (105) | 58.4 (66) | 55.9 (33) |
| Online portal/trust intranet | 32.4 (23) | 33.3 (14) | 31.0 (9) | 20.2 (36) | 20.4 (23) | 20.3 (12) |
| Written report (electronic version) | 29.6 (21) | 33.3 (14) | 24.1 (7) | 11.8 (21) | 15.0 (17) | 1.7 (1) |
| Written report (hard copy) | 15.5 (11) | 16.7 (7) | 13.8 (4) | 9.0 (16) | 10.6 (12) | 6.8 (4) |
| They are not communicated | 1.4 (1) | 2.4 (1) | – | 4.5 (8) | 5.3 (6) | 3.4 (2) |
| Not sure | 1/4 (1) | 2.4 (1) | – | 12.4 (22) | 14.2 (16) | 10.2 (6) |
| Other, please specify | 9.9 (7) | 4.8 (2) | 17.2 (5) | 10.7 (19) | 13.3 (15) | 6.8 (4) |
| Who communicates patient experience results with you? Select all that apply | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 2.8 (2) | 2.4 (1) | 3.4 (1) | 3.9 (7) | 2.7 (3) | 6.8 (4) |
| Response option | ||||||
| Board members | 2.8 (2) | 2.4 (1) | 3.4 (1) | 2.8 (5) | 2.7 (3) | 3.4 (2) |
| CEO | 7.0 (5) | 7.1 (3) | 6.9 (2) | 3.4 (6) | 4.4 (5) | 1.7 (1) |
| Patient experience leads | 21.1 (15) | 19.0 (8) | 24.1 (7) | 9.0 (16) | 6.2 (7) | 15.3 (9) |
| Specialty leads | 12.7 (9) | 9.5 (4) | 17.2 (5) | 5.1 (9) | 6.2 (7) | 1.7 (1) |
| Ward managers | 77.5 (55) | 78.6 (33) | 75.9 (22) | 87.1 (155) | 86.7 (98) | 86.4 (51) |
| Other, please specify | 14.1 (10) | 11.9 (5) | 17.2 (5) | 9.0 (16) | 12.4 (14) | 3.4 (2) |
| How would you prefer to have the results of patient data collection communicated to you? Select all that apply | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 5.6 (4) | 7.1 (3) | 3.4 (1) | 2.2 (4) | 1.8 (2) | 3.4 (2) |
| Response option | ||||||
| During staff meetings | 57.7 (41) | 54.8 (23) | 62.1 (18) | 57.9 (103) | 56.6 (64) | 61.0 (36) |
| On staff noticeboards | 57.7 (41) | 57.1 (24) | 58.6 (17) | 51.7 (92) | 54.0 (61) | 44.1 (26) |
| Online portal/trust intranet | 29.6 (21) | 38.1 (16) | 17.2 (5) | 21.9 (39) | 19.5 (22) | 27.1 (16) |
| Written report (electronic copy) | 33.8 (24) | 31.0 (13) | 37.9 (11) | 21.9 (39) | 20.4 (23) | 20.3 (12) |
| Written report (hard copy) | 7.0 (5) | 4.8 (2) | 10.3 (3) | 11.2 (20) | 8.8 (10) | 15.3 (9) |
| They are not communicated | 1.4 (1) | 2.4 (1) | – | 2.2 (4) | 0.9 (1) | 5.1 (3) |
| Not sure | – | – | – | 3.4 (6) | 2.7 (3) | 3.4 (2) |
| Other, please specify | 4.2 (3) | – | 10.3 (3) | 3.4 (6) | 3.5 (4) | 3.4 (2) |
| How frequently are results communicated with you? | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 4.2 (3) | 2.4 (1) | 6.9 (2) | 50.0 (89) | 51.3 (58) | 42.4 (25) |
| Response option | ||||||
| Annually | 1.4 (1) | 2.4 (1) | – | 1.7 (3) | – | 5.1 (3) |
| Quarterly | 7.0 (5) | 9.5 (4) | 3.4 (1) | 5.1 (9) | 1.8 (2) | 11.9 (7) |
| Monthly | 60.6 (43) | 54.8 (23) | 69.0 (20) | 23.6 (42) | 21.2 (24) | 30.5 (18) |
| Weekly | 22.5 (16) | 26.2 (11) | 17.2 (5) | 9.0 (16) | 11.5 (13) | 5.1 (3) |
| Daily | – | – | – | 5.1 (9) | 6.2 (7) | 3.4 (2) |
| Other, please specify | 4.2 (3) | 4.8 (2) | 3.4 (1) | 10.1 (18) | 11.5 (13) | 8.5 (5) |
| How frequently would you like to have results communicated with you? | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 1.4 (1) | – | 3.4 (1) | 3.4 (6) | 3.5 (4) | 3.4 (2) |
| Response option | ||||||
| Annually | 1.4 (1) | 2.4 (1) | – | 1.1 (2) | 0.9 (1) | – |
| Quarterly | 5.6 (4) | 7.1 (3) | 3.4 (1) | 15.2 (27) | 14.2 (16) | 18.6 (11) |
| Monthly | 63.4 (45) | 59. (25) | 69.0 (20) | 53.9 (96) | 52.2 (59) | 61.0 (36) |
| Weekly | 25.4 (18) | 31.0 (13) | 17.2 (5) | 25.3 (45) | 25.7 (29) | 20.3 (12) |
| Daily | 2.8 (2) | – | 6.9 (2) | 5.6 (10) | 6.2 (7) | 5.1 (3) |
| Other, please specify | – | – | – | 5.6 (10) | 7.1 (8) | 3.4 (2) |
| What report format would make it easier for you to understand and use patient experience findings? Select all that apply | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 4.2 (3) | – | 10.3 (3) | 3.9 (7) | 2.7 (3) | 6.8 (4) |
| Response option | ||||||
| Tables | 16.9 (12) | 16.7 (7) | 17.2 (5) | 17.4 (31) | 18.6 (21) | 16.9 (10) |
| Charts and graphs | 40.8 (29) | 42.9 (18) | 37.9 (11) | 29.2 (52) | 28.3 (32) | 30.5 (18) |
| Written narrative | 40.8 (29) | 35.7 (15) | 48.3 (14) | 42.1 (75) | 41.6 (47) | 40.7 (24) |
| Example quotes of patients’ feedback provided in response to open questions | 64.8 (46) | 61.9 (26) | 69.0 (20) | 47.8 (85) | 42.5 (48) | 54.2 (32) |
| Other | – | – | – | 2.2 (4) | 2.7 (3) | 1.7 (1) |
| How useful is patient experience data for your work with patients? | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 1.4 (1) | – | 3.4 (1) | 2.8 (5) | 2.7 (3) | 3.4 (2) |
| Response option | ||||||
| Extremely useful | 50.7 (36) | 50.0 (21) | 51.7 (15) | 40.4 (72) | 41.6 (47) | 40.7 (24) |
| Very useful | 25.4 (18) | 26.2 (11) | 24.1 (7) | 30.9 (55) | 28.3 (32) | 32.2 (19) |
| Somewhat useful | 16.9 (12) | 14.3 (6) | 20.7 (6) | 20.8 (37) | 22.1 (25) | 18.6 (11) |
| Not very useful | 4.2 (3) | 7.1 (3) | – | 3.9 (7) | 4.4 (5) | 3.4 (2) |
| Not at all useful | 1.4 (1) | 2.4 (1) | – | 1.1 (2) | 0.9 (1) | 1.7 (1) |
| Are you aware of any changes that have been implemented in your trust as a result of patient experience data? | Pre patient data collection, % (n) | ||
|---|---|---|---|
| Total | Study | Control | |
| Total | 71 | 42 | 29 |
| No reply | 1.4 (1) | – | 3.4 (1) |
| Response option | |||
| Yes, I have implemented changes | 11.3 (8) | 9.5 (4) | 13.8 (4) |
| Yes, others have implemented changes | 21.1 (15) | 26.2 (11) | 13.8 (4) |
| Yes, myself and others have implemented changes | 31.0 (22) | 31.0 (13) | 31.0 (9) |
| No | 21.1 (15) | 23.8 (10) | 17.2 (5) |
| Not sure | 14.1 (10) | 9.5 (4) | 20.7 (6) |
| What changes have been made in your trust based on the results of patient experience data collection? Select all that apply | Pre patient data collection, % (n) | ||
|---|---|---|---|
| Total | Study | Control | |
| Total | 46 | 28 | 18 |
| No reply | 2.2 (1) | – | 5.6 (1) |
| Response option | |||
| Changes to the way staff interact with patients | 69.6 (32) | 67.9 (19) | 72.2 (13) |
| Changes to the way staff interact with colleagues | 34.8 (16) | 39.3 (11) | 27.8 (5) |
| Changes to the way care is provided to patients | 73.9 (34) | 75.0 (21) | 72.2 (13) |
| Changes to the layout of the hospital, ward or department | 32.6 (15) | 28.6 (8) | 38.9 (7) |
| Changes to patient survey | 4.3 (2) | 3.6 (1) | 5.6 (1) |
| Other, please specify | 8.7 (4) | 7.1 (2) | 11.1 (2) |
| Are you aware of any changes that have been implemented in your ward as a result of patient experience data? | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 1.4 (1) | – | 3.4 (1) | 2.8 (5) | 3.5 (4) | 1.7 (1) |
| Response option | ||||||
| Yes, I have implemented changes | 9.9 (7) | 7.1 (3) | 13.8 (4) | 11.2 (20) | 11.5 (13) | 11.9 (7) |
| Yes, others have implemented changes | 23.9 (17) | 26.2 (11) | 20.7 (6) | 19.1 (34) | 15.0 (17) | 22.0 (13) |
| Yes, myself and colleagues have implemented changes | 32.4 (23) | 33.3 (14) | 31.0 (9) | 25.8 (46) | 29.2 (33) | 20.3 (12) |
| No | 14.1 (10) | 16.7 (7) | 10.3 (3) | 41.0 (73) | 40.7 (46) | 44.1 (26) |
| Not sure | 18.3 (13) | 16.7 (7) | 20.7 (6) | – | – | – |
| What changes have been made based on the results of patient experience data collection? Select all that apply | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 48 | 28 | 20 | 105 | 67 | 33 |
| No reply | 4.2 (2) | 3.6 (1) | 5.0 (1) | 7.6 (8) | 7.5 (5) | 9.1 (3) |
| Response option | ||||||
| Changes to the way staff interact with patients | 75.0 (36) | 71.4 (20) | 80.0 (16) | 61.0 (64) | 64.2 (43) | 51.5 (17) |
| Changes to the way staff interact with colleagues | 27.1 (13) | 35.7 (10) | 15.0 (3) | 30.5 (32) | 26.9 (18) | 33.3 (11) |
| Changes to the way care provided to patients | 72.9 (35) | 71.4 (20) | 75.0 (15) | 58.1 (61) | 56.7 (38) | 57.6 (19) |
| Changes to the layout of the hospital, ward or department | 22.9 (11) | 17.9 (5) | 30.0 (6) | 25.7 (27) | 25.4 (17) | 18.2 (6) |
| Changes to patient survey | 4.2 (2) | – | 10.0 (2) | 3.8 (4) | 6.0 (4) | – |
| Other, please specify | 6.3 (3) | 7.1 (2) | 5.0 (1) | 4.8 (5) | 4.5 (3) | 6.1 (2) |
| What are the barriers, if any, that affect your collection or use of patient experience data? Select all that apply | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 1.4 (1) | – | 3.4 (1) | 7.3 (13) | 6.2 (7) | 10.2 (6) |
| Response option | ||||||
| Cost | 9.9 (7) | 14.3 (6) | 3.4 (1) | 13.5 (24) | 12.4 (14) | 15.3 (9) |
| Lack of staff interest | 19.7 (14) | 23.8 (10) | 13.8 (4) | 18.5 (33) | 20.4 (23) | 11.9 (7) |
| Lack of staff time to administer survey | 43.7 (31) | 42.9 (18) | 44.8 (13) | 49.4 (88) | 50.4 (57) | 47.5 (28) |
| Lack of volunteer workforce | 18.3 (13) | 19.0 (8) | 17.2 (5) | 18.0 (32) | 15.0 (17) | 23.7 (14) |
| Language barriers | 9.9 (7) | 9.5 (4) | 10.3 (3) | 10.7 (19) | 13.3 (15) | 6.8 (4) |
| Not enough importance placed on patient experience | 11.3 (8) | 14.3 (6) | 6.9 (2) | 12.9 (23) | 11.5 (13) | 15.3 (9) |
| Patients are receiving multiple surveys already | 23.9 (17) | 16.7 (7) | 34.5 (10) | 7.9 (14) | 8.0 (9) | 1.7 (1) |
| No barriers affect the collection or use of patient experience data | 25.4 (18) | 26.2 (11) | 24.1 (7) | 15.7 (28) | 13.3 (15) | 18.6 (11) |
| Other, please specify | 16.9 (12) | 14.3 (6) | 20.7 (6) | 11.2 (20) | 10.6 (12) | 13.6 (8) |
| In the past, have you used NRTF to collect patient experience data in your ward or department? | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 2.8 (2) | – | 6.9 (2) | 18.0 (32) | 14.2 (16) | 27.1 (16) |
| Response option | ||||||
| Yes | 49.3 (35) | 54.8 (23) | 41.4 (12) | 32.6 (58) | 33.6 (38) | 28.8 (17) |
| No | 26.8 (19) | 23.8 (10) | 31.0 (9) | 9.6 (17) | 8.0 (9) | 13.6 (8) |
| Not sure | 21.1 (15) | 21.4 (9) | 20.7 (6) | 39.9 (71) | 44.2 (50) | 30.5 (18) |
| In the past, have you used volunteers to collect patient experience data in your ward or department? | Time point, response % (n) | |||||
|---|---|---|---|---|---|---|
| Pre patient data collection | Post patient data collection | |||||
| Total | Study | Control | Total | Study | Control | |
| Total | 71 | 42 | 29 | 178 | 113 | 59 |
| No reply | 1.4 (1) | – | 3.4 (1) | 21.3 (38) | 19.5 (22) | 27.1 (16) |
| Response option | ||||||
| Yes | 39.4 (28) | 45.2 (19) | 31.0 (9) | 33.1 (59) | 38.9 (44) | 20.3 (12) |
| No | 29.6 (21) | 33.3 (14) | 24.1 (7) | 11.2 (20) | 7.1 (8) | 18.6 (11) |
| Not sure | 29.6 (21) | 21.4 (9) | 41.4 (12) | 34.3 (61) | 34.5 (39) | 33.9 (20) |
List of abbreviations
- A&E
- accident and emergency
- AGFI
- adjusted goodness of fit index
- CARE
- Consultation and Relational Empathy
- CI
- confidence interval
- df
- degree of freedom
- DIF
- differential item functioning
- EoSRES
- East of Scotland Research Ethics Service
- FFT
- Friends and Family Test
- GFI
- goodness of fit index
- GPPS
- GP Patient Survey
- HCAHPS
- Hospital Consumer Assessment of Healthcare Providers and Systems
- MAP
- minimum average partial
- NRTF
- near real-time feedback
- PEECH
- Patient Evaluation of Emotional Care during Hospitalisation
- R&D
- research and development
- RMSR
- root-mean-square of residuals
- RQ
- research question
- RTF
- real-time feedback
- SD
- standard deviation
- SE
- standard error