Improving patient safety through the involvement of patients: development and evaluation of novel interventions to engage patients in preventing patient safety incidents and protecting them against unintended harm

Data sources and searches

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed in conducting this systematic review [see www.prisma-statement.org/ (accessed 8 August 2016)]. A variety of strategies was used to search the literature to 20 November 2010. Clear identification of studies that identified the contributory factors in active failures was hampered by the lack of consistent terminology used across studies. First, search terms were developed and electronic database searching was performed across the following databases: MEDLINE, PsycINFO, ISI Web of knowledge, Cumulative Index to Nursing and Allied Health Literature (CINAHL) and EMBASE. Second, the reference lists of all downloaded articles were manually searched to identify potentially relevant papers. Third, a number of patient safety organisation websites were searched to identify other potentially relevant published or unpublished reports. Fourth, registered study databases were searched using the term ‘patient safety’ to identify any ongoing or finished projects relevant to the current review that may have provided relevant material. A summary of these search strategies can be found in Appendix 1. Finally, key patient safety authors were contacted and asked to provide details of any relevant published or unpublished reports. This search strategy identified a total of 1502 potential articles. All article titles and abstracts were reviewed for inclusion (by RM). A random sample of 10% of the titles and abstracts was double coded with respect to inclusion or exclusion (by RS). The kappa value of 0.73 indicated an acceptable level of agreement. If there was disagreement about inclusion or exclusion, the full-text article was obtained and reviewed (by RM and RS) and agreement reached. In total, 95 papers (representing 83 studies) met the inclusion criteria and were included in the review (Figure 2).

FIGURE 2.

Flow chart of the search strategy and included studies.

Study selection

Studies were included in this review if they reported data from:

1. secondary care or hospital environments

2. primary research that either specifically aimed to identify the contributory factors (often referred to as ‘causes’ within studies) of active failures or threats to patient safety or reported a clear framework for the categorisation of contributory factors of errors or threats to patient safety in the results section.

Studies were excluded if they reported data from:

1. active failures as causes of errors or threats to patient safety rather than underlying latent domains (e.g. only specific human ‘errors’ causing failure of a barcode checking system71)

2. contributory factors of behaviours or processes that were not active failures (e.g. factors affecting the likelihood of staff to report serious medication errors in hospitals72)

3. case studies reporting contributory factors of a specific adverse event (e.g. Chassin and Becher73)

4. studies that applied proactive risk assessment methods to identify potential failures (e.g. failure mode and effects analysis) as these papers focused on exploring potential problems of specific elements of a health-care system or process.

Data extraction and quality assessment

The study characteristics of 83 data sets were coded. All included articles were blind double coded (by RM and RS) and data were extracted and uploaded onto a Microsoft Access^® database (2010; Microsoft Corporation, Redmond, WA, USA). Kappa values are reported only for dichotomous variables. Articles were coded according to the following characteristics: country of origin; description of setting; study method; study sample; theoretical frameworks informing the research (following the quality coding framework of Sirriyeh et al. ,74 studies were coded as explicit use of theory, i.e. explicit statement of theoretical framework applied to the research; specific use of theory, i.e. reference to specific theoretical basis; broad use of theory, i.e. reference to broad theoretical basis; or none at all, i.e. no theory mentioned); whether identification of contributory factors was a primary or secondary aim of the study (κ = 0.66); whether contributory factors were identified by a human factors expert (κ = 0.79); and, finally, whether patients or staff reported the raw data used to identify contributory factors (κ = 1, perfect agreement). Studies varied in the extent to which they used primary data collection methods to elicit contributory factors or whether they used a predefined set of contributory factors; therefore, the following additional information was gathered to glean more details about the elicitation of contributory factors: whether the contributory factor list was fully developed before empirical data were collected (yes or no, κ = 0.74); the method for eliciting contributory factors (if different from the overall study method); and any further details about the sample used to elicit contributory factors if different or a subset of the overall study sample. Disagreements were discussed and resolved. As we were interested in how contributory factors were identified, regardless of whether this was the primary aim of the study, we did not engage in any further ‘quality assessment’ coding, as often very little detail about how contributory factors were identified was reported. All included papers and extracted data can be found online [see http://qualitysafety.bmj.com/content/suppl/2012/03/14/bmjqs-2011-000443.DC1/Final_Appendix_v2_2_11_11.pdf (accessed 8 August 2016)].

All contributory factors reported within the papers were transcribed verbatim into a Microsoft Excel^® spreadsheet.

Data synthesis analysis

To develop the contributory factor framework, two of the authors (RL, a human factors expert, and RM, a behavioural scientist) first independently grouped all of the transcribed verbatim contributory factor items into categories according to their underlying semantic meaning (e.g. equipment not working, equipment failure and equipment malfunction were all be grouped as equipment failure). Items could be categorised into more than one category. Second, each author further grouped these categories into their higher-order domains (e.g. equipment failure was grouped with equipment unavailability and insufficient supplies to become ‘equipment and supplies’). At this stage the authors did not explicitly distinguish between latent conditions and local working conditions. Next, the two authors met to discuss and agree the number of higher-order domains and label and define them (e.g. ‘equipment and supplies’ was defined as ‘the availability and functioning of equipment and supplies’). A decision was made to include all factors contributing to PSIs in this framework, both the proximal factors (e.g. active failures) and those more distal or external to the organisation (e.g. design of equipment and supplies and external policy context). This process resulted in a framework of 19 domains with a definition for each (Figure 3 and Table 1). Finally, the same two authors applied the framework, again independently, to the raw data to classify each of the contributory factors based on the framework and to assess agreement. At first, 10% of the factors were coded and at this stage agreement was 55%. Following clarification and modification of the definitions (e.g. ‘human factors design of equipment and supplies’ became ‘design of equipment and supplies’), the remaining 90% of the contributory factors were coded. Agreement at this second stage was 90%. Disagreements were discussed and resolved through consensus.

FIGURE 3.

The Yorkshire Contributory Factors Framework. 172 This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode. Reproduced with permission from Bradford Teaching Hospitals NHS Foundation Trust.

To ensure that the framework of domains had relevance and meaning beyond the two authors who developed it, 10% of the data sets (n = 9) and their respective contributory factors were extracted from the database and sent to two academic health professionals (IW, a general practitioner, and JW, a hospital physician). Both were provided with instructions and definitions of the domains and were asked to code each of the contributory factors using the framework. They were asked to include ‘can’t code’ when they were uncertain of the correct response. Initial agreement between the first two authors and each academic health professional was 62.5% (RL and RM with IW) and 85% (RL and RM with JW). After discussion with the first independent reviewer (IW) and further minor modifications of the definitions of domains, agreement rose to 80.1%. Given that agreement with the second reviewer was initially high (85%), further discussion with this reviewer was not deemed necessary.

As noted in the introduction, contributory factors can vary according to their level of proximity to the ‘active failure’ being accorded to the individual, local working conditions (e.g. management of staff and staffing levels) or more latent conditions (e.g. design of equipment and supplies). The contributory factors elicited in this review also reflected these distinctions. In a final step, an expert panel of clinicians (n = 5), researchers (n = 8), managers (n = 2) and lay people (n = 2) were provided with a list of all contributory factors and definitions and asked to identify the extent to which each factor was removed in time and space from PSIs on a 5-point scale from 1 (very close in time and space) to 5 (very distant in time and space). Contributory factors scoring 4 or 5 were deemed to be more ‘latent’ organisational factors, whereas those scoring 2 or 3 were deemed to be more related to local working conditions or situational factors. This allowed us to ground the taxonomy in a hierarchical framework, which we have described in Figure 3.

Country of origin

The majority of the studies identified by this review were conducted in the USA (n = 34), the UK (n = 13), Australia (n = 7) and Canada (n = 5). One study reported multinational data from 27 countries,162 one reported data from three countries147 and one reported data from the USA and Canada. 167

Setting

Thirty studies reported data collected from general hospital settings. Other studies focused particularly on intensive care on its own (n = 17), in combination with coronary care (n = 1) or in combination with medicine and surgery (n = 1); surgery settings (n = 16), including in combination with intensive care;163 and anaesthesia (n = 7), maternity (n = 2), pharmacy (n = 2) or transfusion settings (n = 2). Other settings included geriatric and cardiovascular wards158 and the emergency department. 154 Two studies reported incidents from US general reporting systems (the US Vaccine Adverse Event Reporting System92 and the National Medication Error Database166). Finally, one study reported data from a cohort of student nurses. 120

Aim of study (primary/secondary) and theoretical basis

The majority of studies explicitly aimed to identify contributory factors (or more commonly referred to as ‘causes’) of errors or active failures (n = 55). Over half of the included studies made no reference to a theoretical basis driving the identification of contributory factors (n = 48). When theory was explicitly mentioned and related to methodology (n = 8), all studies referred to Reason’s20 model of accident causation. Only six studies included explicit human factors expertise in the elicitation of contributory factors.

Description of empirical data collection methods

One-third of studies (n = 30) reported data collected as part of an incident reporting scheme based within the hospital; see http://qualitysafety.bmj.com/content/suppl/2012/03/14/bmjqs-2011-000443.DC1/Final_Appendix_v2_2_11_11.pdf for details. Typically, these studies reported the frequency with which staff identified contributory factors of a reported incident from a predefined list (e.g. Beckmann et al. 80) but they also included studies in which free-text input from incident reports was analysed qualitatively (e.g. Nast et al. 135). Other papers reported results from observational studies (n = 14), interviews (n = 9) and focus groups (n = 1), surveys (n = 8) or case note reviews (n = 4). Seventeen studies reported using multiple methods; see http://qualitysafety.bmj.com/content/suppl/2012/03/14/bmjqs-2011-000443.DC1/Final_Appendix_v2_2_11_11.pdf.

Use of a contributory factors framework

The coders assessed the extent to which studies had generated a deductive predefined list of contributory factors (e.g. the London Protocol63) that then informed data collection or whether studies used inductive methods to elicit contributory factors from participants. For example, within incident reporting studies, the deductive use of lists would take the form of a tick box list given to participants, whereas within interview studies a list of closed questions might be used to elicit responses about particular contributory factors. The use of a deductive list in these contexts means that no new contributory factors can be elicited from participants; rather, only the prevalence with which they are endorsed can be assessed.

In total, 49 studies used a predefined contributory factor list as a basis for data collection. Twenty-six of these were based solely on previous frameworks (seven studies used a variation of the Australian Incident Monitoring Study framework,57 three studies used the Eindhoven classification,61 two studies used the London Protocol63 and 14 reported frameworks from miscellaneous previous publications). Six studies78,109–111,131,148 used a combination of literature reviewing and author or other expert opinion to identify the list of contributory factors, one of which used previous literature (in addition to pilot work not reported in the paper140), and one study used only expert opinion. 163 Fifteen studies that used a predefined contributory factors list did not specify how that list was obtained. The use of a contributory factors framework was unclear in two studies. Of the 35 remaining studies that elicited contributory factors from analysis of primary data, 25 used qualitative methods such as interviews, focus groups or free-text coding of incident reports, eight used observational methods and two used both.

Identification of contributory factors

As described in the methods section, through the coding of the 1676 contributory factors, a list of 20 contributory factor domains was independently identified by two reviewers (RM and RL) and this list was verified by two further coders (IW and JW, both clinicians). Based on this list we also sought to identify contributory factors that were identified most frequently within the literature. The number of times each of the 20 contributory factors was identified across all of the study settings is shown in Table 2 (total column). Across study settings, the five contributory factors identified most frequently were active failures (slips, lapses, mistakes, deviations from policy) (18.2%), individual factors (11.0%), communication systems (7.9%), equipment and supplies (6.6%) and management of staff and staffing levels (5.8%). This pattern varied little according to the hospital setting in which the data were collected, with active failures and individual factors consistently being the most frequently identified contributory factors. However, there was some variation. For example, team factors (8.5%) was among the top five contributory factors for surgery but for no other setting. For anaesthesia, equipment and supplies was the second most cited contributory factor, accounting for 15.2% of the codes. Physical environment was also among the top five factors for anaesthesia. For the general hospital setting, patient factors (7.4%) was among the highest-ranked contributory factors but equipment and supplies was not.

Table 3 shows the contributory factors identified by each of the different study methodologies. Studies using incident reporting methodology more commonly identified active failures than interview or observational studies. This is intuitive as generally incident report forms are limiting in terms of the details of the event that can be recounted and the options for contributory factors available to the reporter. Interview studies appeared to more commonly identify individual factors and staff workload as contributory factors. Observational studies tended to identify equipment and supplies marginally more frequently than other methods.

We also investigated variation in the identification of contributory factors as a function of whether or not a human factors expert was involved in the identification. Caution must be exercised because of the low number of studies explicitly utilising a human factors expert in the elicitation of contributory factors. However, there was some evidence that, compared with others, human factors experts tended to identify active failures less frequently (11% vs. 19%) and more latent contributory factors such as team factors (10% vs. 3%) and the physical environment (7% vs. 3%) more frequently. However, despite some evidence that human factors experts were more likely to identify distal than proximal causes, they were more likely to identify individual factors (e.g. fatigue, inexperience; 16%) than others (11%). A similar pattern of findings was apparent when comparing studies that employed a theoretical framework in developing their contributory factors coding scheme with those that did not.

Figure 3 is a diagrammatic summary of the findings of the review, which represents the speculated hierarchical nature of the identified domains. This figure depicts the domains as a series of concentric circles, with active failures at the centre and external policy context as the outer circle. This diagram helps to illustrate the extent to which a domain is proximal to the active failure.

Conclusions and policy implications

The poor quality of the current evidence base and the lack of a consistently adopted framework limits the accurate reporting of factors that contribute to error and hence the opportunity to learn from error. We conducted a systematic review of contributory factors identified from a wide range of settings using multiple data collection methods. We then developed an empirically based framework of contributory factors. This framework has the potential to be applied across hospital settings to improve the identification and prevention of factors that cause harm to patients.

Chapter summary

This chapter has outlined a review of the extant evidence of contributory factors to PSI and the development of a framework for their conceptualisation within a systems approach to safety. Alongside the obvious implications for improving the identification of errors for health-care professionals, it provides a framework for the further development of a tool to allow patients to systematically feed back about the safety of their care, effectively engaging in the assessment of risk within hospital settings. The next chapter describes the development and validation of such a tool: the Patient Measure of Safety.

Publication statement

This chapter is based on a previously published paper. 172 We reproduce it here with permission. This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

Development

The development study can be divided into two distinct stages. Stage 1 involved a series of qualitative interviews with patients to explore which contributory factors they were able to identify, using the YCFF as a basis. The data from these interviews were used to inform the development of items for the PMOS tool. Stage 2 involved testing the PMOS tool with health professionals and patients using a ‘think-aloud’189 approach. A multidisciplinary panel of experts including policy makers, health professionals, academic researchers and patients informed the design of the study and the patient panel aided the development of the questionnaire.

Patients in stage 1 (interviews) and patients and health professionals in stage 2 (think aloud) were recruited from six units (maternity, renal, physiotherapy outpatients, vascular surgery, ear, nose and throat and cancer services) in a NHS trust in the north of England. These units were selected using purposive sampling to ensure that the views of a broad range of patients were elicited encompassing those with regular interaction with the health service (renal patients) and those with relatively short one-off stays (maternity), young and old patients, men and women and patients of different ethnic backgrounds. The data were collected by two researchers, one of whom was able to speak Mirpuri (a dialect spoken by three-quarters of the Pakistani population in Bradford) [see www.30-days.net/muslims/muslims-in/europe/mirpuris-britain/ (accessed September 2012)]. Ethical approval for this study was gained from a local research ethics committee (reference number 09-H1302–115).

Validation

The validation study used two separate cross-sectional surveys (one with patients and one with staff) within a large acute teaching hospital in the north of England. Data were collected between 1 September 2011 and 30 November 2011 (10 wards) and between 3 and 5 April 2012 (one ward).

In line with recommendations, a minimum sample size of 250 was considered acceptable for the patient survey (in relation to planned factor analysis). 193 There was no a priori required sample size for the staff survey (against which the patient survey was validated) although we aimed to sample a minimum of 50% of staff on participating wards.

Development study

All participants approached for interview at both stages agreed to take part. This high uptake was because patients who were too unwell or who did not have the capacity to complete the interview were filtered out by staff. In the stage 1 interviews, participants (14 male patients and 19 female patients) ranged in age from 18 to 83 years.

Validation study

Limitations: development study

The development of the PMOS tool was based on the YCFF, which in turn was based on a review of studies conducted almost exclusively with health-care professionals. This means that there is a danger that the PMOS tool may not truly reflect the views of patients if patients tend to identify different contributory factors. However, taking an unstructured approach to half of the qualitative interviews meant that we were not constrained by this a priori structure. It is worth noting that, aside from the dignity and respect domain, all contributory factors that patients identified mapped onto the YCFF model.

In addition, the PMOS tool contains only those domains from the YCFF that patients were able to comment on, which means that five of the domains within the YCFF are not represented. Although a major strength of the PMOS tool is that it allows identification of contributory factors from a patient’s perspective, it must be recognised that it does not capture all possible contributory factors. It must therefore be used in conjunction with other tools, such as staff safety culture surveys and failure modes and effects analysis,210 to explore upstream organisational failures.

Limitations: validation study

Some of the dimensions within the PMOS tool failed to achieve the recommended Cronbach’s alpha guideline of 0.70. Despite this, the average inter-item correlations suggest that the dimensions are located within the optimum range of good reliability (0.20–0.40). The PMOS positive index was also found to relate to only two of the AHRQ patient safety culture survey outcomes; no significant correlations were found with the two incident reporting outcomes. However, given the unclear relationship between levels of incident reporting and actual PSIs, one might not expect to see a clear relationship between the number of incident reports and a measure of safety. The AHRQ hospital survey of patient safety was chosen as a measure of convergent validity as it has been used and validated a number of times; however, recent research has questioned its reliability and validity in a UK setting. 211 The response rates for this staff survey also varied widely, with an overall response rate of 48%, much lower than responses to the patient survey, which has implications for assessing the convergent validity of the measure. The two retained single indicator items exhibited poor test–retest reliability and, as such, caution should be exercised in their interpretation.

General limitations

Finally, the PMOS tool could be completed only by patients (or their relatives/carers) who understood English. Despite the above limitations, patients were found to be very willing to engage with the measure, demonstrated by the high consent rate into the study (86%). The PMOS tool was developed specifically for patients in hospital settings, which may limit its generalisability to other settings. Further work should explore whether this tool can be used for different patient groups in a range of settings (e.g. community-based outpatient services or general practice). The PMOS tool may not represent the views of the general population internationally because, despite being developed with a diverse ethnic population, the sample was drawn from a single hospital in the north of England. However, future work will explore the utility of the PMOS tool with a larger number of patients across a number of NHS trusts within England.

Recommendations for future research

• The PMOS tool should be tested between wards in different hospital settings or between different hospitals to determine whether it has acceptable reliability and validity outside of the validation study described in this chapter. The research team plan to undertake a large multicentre evaluation of the tool in hospitals in the north of England.

• Linked to the above, further work should explore whether this tool can be used for different patient groups in a range of settings such as community-based outpatient services or general practice. To achieve this, further validation work will be necessary to tailor the PMOS tool to these specific patient groups and clinical environments.

Conclusion

Patients are in a very good position to observe the safety of their care and that of others on the same ward/unit and are able to respond to items in the PMOS tool that are indicative of the safe organisation of their care. In this study we have developed a tool that patients can complete during their hospital stay.

Overall, the tool exhibited acceptable reliability and validity, subject to the caveats mentioned above. Based on a clear theoretical framework, the PMOS tool uses the patient’s perspective with the intention of identifying ‘latent’ weaknesses that could contribute to future PSIs. The development of this tool has implications for practice and research. Clinicians can use this tool to gain an important patient perspective on safety, capturing areas of weakness that might otherwise go unreported or unidentified. There is no other such tool available at present to collect this type of information from patients. The tool also offers an opportunity for clinicians and hospitals to track changes in safety over time by repeated assessments within wards.

Chapter summary

This chapter has outlined the development and validation of the first known tool to allow patients to feed back about factors associated with their care that are known to contribute to PSIs. This evidence and theory-based tool effectively allows health-care organisations to systematically capture a perspective on safety – the patient’s – that has been missing in the assessment of risk to date. However, alongside reporting prospectively on factors that may impact on patient safety in the future, it is important to understand how patients might also report retrospectively on patient safety problems that they have already experienced. The next chapter outlines a review of the evidence of patient reporting in hospital settings, addressing the question of whether patients can help us learn about errors in hospitals.

Publication statement

This chapter represents a summary of two previously published papers. 181,212 We reproduce them here with permission under the terms of the Creative Commons Attribution Non-commercial License.

Rationale

The patient ‘voice’ is emerging as a key part of the research, development and management of patient safety both internationally and within the UK. The main driver for this shift in focus was the political move towards ‘patient choice’ as part of creating a more dynamic and responsive health service. 213 This change of policy was aimed at empowering patients to act as partners in their health care and in terms of patient safety has been translated in practice within the UK through the establishment of national initiatives such as the Patient Safety Champion Network214 and, at a local level, the Patient Advice and Liaison Service (PALS). 215 This was brought sharply into focus by the UK coalition government’s White Paper outlining the legal duty of those with health service commissioning responsibility; the principal aim is to facilitate active participation from patients and public. 186 The UK government health minister at the time encapsulated his vision of the patient perspective in the words, ‘No decision about me, without me’. 216

In addition to establishing patient choice in shaping health-care services, patients have also more recently been viewed as key stakeholders in the management of patient safety. The former UK NPSA recognised this by incorporating patient reports into its National Reporting and Learning System, alongside clinician reports. However, the current position within the NHS and health-care services internationally is still very much dominated by clinician-led reporting of PSIs, a position that has also been apparent in the most recent data published by the UK Patient Safety Observatory. 217

It is intuitive that patients would be a useful source of information on patient safety. Patients are often the only common link between the different treatments and consultations that make up their health-care experience and, as such, are uniquely motivated and positioned to contribute to the quality and safety of their own care. 42,218 However, despite this, it has often been commented that one of the main issues for the patient safety movement has been the lack of patient perspective. 7,219–222 Indeed, a central question for patient safety research must be how relevant and effective research and management of patient safety is if one of the protagonists in the patient safety experience is effectively excluded.

To address such questions, and in line with policy development as already discussed, researchers have more recently started to focus efforts on understanding more about how best to engage patients in patient safety initiatives. One emerging area is how best to involve patients in the reporting of PSIs or issues. A recent systematic review of patient reporting across a variety of settings177 concluded that, despite a relative paucity of studies in this area, patient reporting has been shown to be reliable following corroboration of reports. 36,38,223 However, this timely review also revealed that the little evidence that is available comes from a disparate body of work and that further research is required to identify the optimal method for capturing patient reports, cognisant of different clinical settings and the duration of stay.

In line with such a recommendation, we undertook this systematic review as part of a wider research project aimed at developing and evaluating a range of patient-led PSI reporting tools. Crucially, this research addressed the need for a system that can be used across a hospital with its diversity of clinical settings and which can allow patients to ‘hot report’, that is, to report PSIs while receiving treatment in hospital, thus reducing retrospective recall bias known to be an issue in incident reporting systems across high-risk domains. 224,225 Given the developmental nature of this work, it was clear that a detailed examination of the current evidence was required, informed by a human factors perspective, to ensure that any reporting tool developed would both build on the existing knowledge base and contribute to effective clinical governance. This review builds on previous reviews177,226 by widening the search strategy (through increased numbers of databases searched), examining the quality of the included papers and, most importantly, focusing only on studies exploring patient reporting of PSIs experienced within a hospital setting. This focus has allowed us to consider in greater depth how patient reporting has been examined within the wider context of the systems that exist in hospitals dedicated to the measurement of patient safety, the management of this information and the clinical governance or quality improvement agenda.

Objectives

The nature of the literature examining patient-reported safety concerns precluded a full meta-analysis. Therefore, for this review, our specific objectives were to explore:

• the types of PSIs identified by patients, how this differs from that found for other reporting methods and where patient reporting fits in relation to other methods of measuring patient safety

• the settings in which patients have been asked to report on PSIs

• the timing of patient reports of PSIs in relation to the experience of the patient safety ‘event’

• how patients have been asked to report on PSIs and what has been done with this information.

Eligibility criteria

The inclusion and exclusion criteria were decided on with respect to the aims of the review and defined in terms of the population, intervention/comparators, outcome measures and study design, as advised by the Centre for Reviews and Dissemination. 227 Studies were included if they satisfied one or more criterion under each of the following headings.

Data sources and search strategy

Five databases were searched for this review: MEDLINE, EMBASE, CINAHL, (The King’s Fund) HMIC and PsycINFO. These databases were selected to cover both medical and psychological literatures. The search strategy was developed iteratively, with reference to the study aims and an assessment and ongoing revision of the keywords of target articles. In addition, in conjunction with the specialist librarian at Bradford Teaching Hospitals NHS Foundation Trust, subject headings that mapped onto the keywords were identified, to ensure that papers not using the keywords but within the subject area were also picked up within the search. For each of the databases, the subject headings were identified separately to ensure optimum coverage. The final list of search terms is detailed in Appendix 3. Final searches across all five databases were run on 9 August 2010.

Given that patient reporting is a relatively new phenomenon in both research and practice, that the related terminology lacks standardisation177 and that through pilot searches we identified some inaccuracy of indexing in electronic databases, we opted for a search based on a high-sensitivity, low-specificity strategy. This also necessitated hand searching the literature.

Data extraction

Figure 4 outlines the process for study selection. First, results from the searches were merged using reference management software and duplicates were removed. The titles and abstracts of retrieved citations were reviewed by a researcher (JW) using the inclusion and exclusion criteria, following which the full texts of 51 studies were retrieved for further assessment. Reasons for excluding studies at this stage were primarily because they did not relate to the subject area and because they were non-empirical studies such as letters, editorials or position papers. The retrieved full-text articles were then scrutinised against the inclusion criteria and the reference sections examined, which identified a further 17 articles. At this stage, concordance in the decision for inclusion or exclusion was achieved by two reviewers (JW and GA). Following this, 12 articles were selected for inclusion in the review. Typically, studies excluded at this stage discussed patient involvement in general patient safety initiatives (rather than incident reporting) or concerned patient reporting in a non-hospital setting (e.g. primary care or outpatient/ambulatory care, community-based surveys, etc.). A further paper was included within the review during the manuscript preparation process, resulting in a final total of 13 papers34–36,38,228–236 selected for inclusion in the review.

FIGURE 4.

Flow chart of the study selection process. a, One paper was included during the manuscript preparation process.

Quality assessment and data synthesis

The heterogeneous nature of the study designs, aims and key findings precluded a full meta-analysis of the data from this review. For this reason, we do not report here on analytical strategy or summarise measures. Furthermore, given the range of methods used within the identified studies, a validated quality assessment tool was utilised that allows comparative analysis of quantitative, qualitative and mixed-methods studies. 74 This tool facilitates assessment of the robustness of the study design and methods; reference to theory; sample size and representativeness; any validation of measures; extent of user involvement; and evidence of critical discussion and limitations. Studies were rated against 16 criteria (14 applying to qualitative papers only, 14 to quantitative papers only and 16 to mixed-methods papers), with scores ranging from 0 to 3 for each criterion (0 = no evidence, 3 = ‘complete’). Total scores were translated to percentages allowing comparison across studies using different methods. Included studies were judged against the scoring criteria by a researcher (JW), with a random sample of three studies further scrutinised by another researcher (GA). Differences in scores were resolved by discussion.

Data from included studies were extracted under the following headings:

• authors

• sample size

• target population

• location

• design

• aim

• method of data collection

• nature of questions eliciting patient-reported PSIs

• timing of reporting and length of patient recall period

• comparison with other methods of error/incident detection

• concordance across error detection methods.

To allow for further comparison, we also extracted data pertaining to the following outcome measures:

• total number of patient-reported PSIs

• percentage of patient sample reporting a PSI

• number of PSIs per patient

• method of classifying patient-reported PSIs

• total number of patient incident reports classified as a PSI

• percentage of patient sample reporting a classified PSI

• number of classified PSIs per patient.

Terminology

All papers concluded that patients were able to report on PSIs in a hospital setting. However, the terminology used to describe such incidents did vary considerably across papers (Table 9). Four papers were concerned only with issues related to medication or treatment. 228–231 When a broader perspective was taken, papers were split between those concerned only with adverse events as categorised by physician review34,232–234 and those that widened this categorisation to also include near misses/close calls and medical errors with minimal or no risk of harm. 35,36,235 Two of these latter papers took a more analytical approach to differentiate between PSIs (adverse events, near misses/close calls and medical errors with a minimal risk of harm) and service quality incidents and process of care problems, respectively. 35,36 The last category was those papers in which patients were asked to what extent they had experienced any ‘undesirable events’ from a prespecified list. 38,236

Nature of patient reports

Can patients report patient safety incidents in a hospital setting?

It is clear when one considers the results in their totality that patients are in a position to report on safety-related issues experienced in a hospital setting. Furthermore, these studies do suggest that patients are able to identify PSIs from across a range of incident ‘types’, referencing a variety of different parties, and across the full range of preventability and severity. On this last point, although patients generally reported PSIs that were not life-threatening, they did report a large number of PSIs rated as ‘significant’ by physicians, suggesting that the patient role in error detection is unlikely to be limited to information deemed to be clinically insignificant. Furthermore, in those studies undertaking physician classification, on average, nearly half of all PSIs reported by patients were judged to be ‘definitely’ or ‘probably’ preventable. This clearly demonstrates that, if asked the right questions about the incident context, patient reporting may offer health-care providers a valuable source of information about how to proactively manage safety.

Implications for patient reporting: research

None of the reviewed papers used any theoretical underpinning to inform either their design or analysis of patient reports. A number of models may be of value in investigating patient reporting, for example social cognition models such as the theory of planned behaviour. 237 However, we believe that taking a human factors perspective is perhaps the most appropriate foundation for research in this area, because of its focus on the multilevel, multifactorial nature of PSI causation, as well as its increasing adoption by service providers in safety improvements. Additionally, this perspective also attributes a high value to near-miss events as well as harm events, thereby widening the opportunity for learning from PSIs. Developing a method for capturing patient reports without recognising human factors may lead to a superficial interpretation of PSIs and one that may inappropriately focus on the role of individuals in causation. This could be a particular issue for nurses, who as a professional group are frequently mentioned in patient reports, which may be largely because of their ongoing visibility through the ‘patient lens’ and numerous encounters as the last point of direct professional contact during a process of care. It has been suggested that patients do not have knowledge of the reasons for, or consequences of, adverse events. 233 We would contend that has yet to be fully established empirically and would likely vary across different patients and their level of contact with health services. Furthermore, research from staff incident reporting suggests that such schemes fail to routinely capture the context and causes of PSIs. 170,238 As for the value of patient reporting, we can infer from such research that, even when those reporting may understand the clinical reasons for preventable events, reporting schemes may not facilitate the capture of such information, leading to the erroneous conclusion that they are unaware of any causal antecedents.

The length of recall period between experiencing and reporting a PSI remains unexplored within the literature. Some authors have commented that lengthy recall periods may introduce ‘recall bias’ into patient reporting. 232,233 The authors of one study did report that PSI rates did not decrease as a function of time,233 but this was related only to rates of PSI reports, which is different from any impact on the accuracy of information. This literature seems to currently lack a sound understanding of, first, the key biasing influences on patient reporting of PSIs and, second, the optimal period of recall. Further research is needed to clarify the optimum period for recall based on the experience of real patients with associated issues of acuity, length of stay, severity of illness, the emotional impact of a PSI and the potentially disorienting hospital environment.

A related issue is the method used by patients to report PSIs within these studies. All of the included studies actively ‘solicited’ reports from patients, either through an interview or through a written survey. None of the study designs allowed for patients to spontaneously report a PSI. This is significant as it may be that research risks inflating the extent to which patients may be willing to report PSIs, simply through the methods used to collect such reports. Some authors have reflected on this point, highlighting the related issue of how the role of the researcher and the nature of the questions may preferentially elicit certain responses. 35 Perhaps the key research question should no longer be ‘Can patients report?’ but rather ‘Can patients report in a system designed to collect this information routinely in a clinical setting?’ Consequently, there is currently no evidence whether patient reporting is feasible outside of a research study or whether it could be an integral and complementary element of a service provider’s safety intelligence network. To assess the latter and examine the validity of patient reporting, we argue that future studies should routinely compare not only the type but also the quality of patient reports with conventional methods of incident detection such as case note review.

Irrespective of the specific study design or the nature of capturing the patient safety reports, there are ongoing issues that researchers, practitioners and managers need to be cognisant of when designing studies or indeed systems to capture PSI reports from patients. A significant issue is the somewhat paradoxical situation that those who are least able to report PSIs may also be at most risk of experiencing them. A number of authors have commented previously on this paradox with reference to the inherent bias arising from asking only those who are discharged from hospital about PSIs, when those unfortunate enough not to have survived and been discharged may have been at a higher risk of experiencing a PSI. 34,233 It has been demonstrated empirically that older people do experience more PSIs,239 but there is also emerging evidence to suggest that other factors influence the likelihood of experiencing a PSI (e.g. non-native language speaking),240 factors that may also lead to under-representation in the studies conducted so far. Overall, current estimates of patient reporting may be particularly inaccurate on the basis that some of the most vulnerable groups are under-represented in patient safety research. Further research should focus on the best ways to engage with these patient groups, to gain a fuller understanding of patient-reported PSIs.

Implications for patient reporting: practice

If patient reporting is to become a valid tool for measuring ‘performance’ in patient safety terms, consideration must be given as to how it fits with other existing error detection methods. Some authors have discussed the problem of a higher false-positive rate for patient reporting of medication errors than for errors detected through physician and nurse reporting. 228 Perhaps this finding highlights a weakness in the proposition that patient reporting can be a valid error detection tool. However, others have presented the counter argument that, as false-positive reports can be ‘validated’ by clinical review, the bigger issue is that patients might suffer from higher rates of false negatives than clinicians, meaning that many potential PSIs may go undetected. 236 Thus, the evidence seems to suggest that patient reporting may risk both over- and underestimating the PSI rate because of misunderstanding of what is normal within the clinical context. There is evidence from the wider incident reporting literature that, when triangulated, different error detection methods may lack a high degree of overlap in the PSIs identified. 228,241–244 Taking this into consideration, patient reporting may suffer from some of the perennial problems inherent in staff reporting,245 but as a part of an error detection jigsaw it may also prove a valuable, and as yet untapped, resource. Given the continuing policy emphasis on patient involvement in care, it would seem entirely appropriate to integrate patient reporting of safety concerns as a formal component of clinical governance.

Limitations

Because of the focus of the overarching project (on the basis of which this review was conducted), the search was limited to studies within a hospital setting. This clearly does exclude other health-care settings, for example primary/ambulatory care. However, very little information is published about incident reporting (staff or patient) in primary care and so the inclusion of such studies here may skew the findings of hospital-based studies.

Recommendations

As discussed above, future research is clearly needed to demonstrate that patient reporting can move beyond the research domain and become an established part of clinical governance. Some authors make suggestions for implementing patient reporting in practice and one has discussed the possibility of distributing notepads to patients to write down concerns, events or questions to share with health-care staff. 228 Others discuss the possibility of ‘hot reporting’, with systems designed to allow patients to call a dedicated telephone line to the hospital pharmacy, to report medication errors. 231 Both suggestions take patient reporting into the realm of workable systems, but with the caveat that it is combined with other error detection methods to form part of an overall safety strategy. Furthermore, to be successful, there should be a ‘collective responsibility’ for development of any patient reporting system, with ‘coordinated improvement efforts involving all members of the healthcare team (including patients)’ (p. 172). 246 Indeed, for a given system to be workable for patients, methods of reporting should be designed, tested and evaluated in consultation with patients. The work of Bate and Robert247 would be useful here, which suggests that, whatever patients design, it should be part of a carefully managed emancipatory process that incorporates staff as stakeholders, increasing a sense of co-ownership but also ultimately demonstrating a fit with the pragmatics of clinical governance.

As with all patient involvement, before establishing patient reporting tools within clinical settings, consideration does need to be given to the issue of patient burden. It is an issue that has been raised previously,13 with the concern expressed that blanket patient involvement interventions may risk shifting the responsibility of safety onto patients, at a time when they are arguably at their most vulnerable. Furthermore, we know that not all patients will be willing or able to be engaged with such interventions and therefore it is important that such patients are not negatively affected as a result of their lack of engagement. Going forward, both research and practice must be mindful that any approach must be flexible enough to deal with such differing levels of engagement.

Conclusions

Patient involvement is a policy imperative. It would appear that hospitalised patients have the potential to report safety concerns. However, the evidence base is currently equivocal and dominated by studies that have focused on active solicitation to the neglect of ‘hot reporting’. Future study designs should be underpinned by a human factors approach, developed in collaboration with patients, taking account of memory recall and other cognitive biases, and use terminology that is understandable to patients but also reflects the predominant language of patient safety. Samples should be representative of the entire hospital population and the tool or tools developed must complement existing organisational governance and improvement strategies.

Chapter summary

This chapter presents a detailed examination of the evidence for the role of patients in reporting PSIs within hospital settings. The findings suggest that patients may have a role in helping organisations learn from error but that there are many unanswered questions about how best to capture this information from patients. The next chapter presents a study that provides further information about one of these key questions: ‘What is the most effective mechanism for collecting patient feedback about patient safety?’

Publication statement

This chapter is based on a previously published paper. 15 We reproduce it here with permission under the terms of the Creative Commons Attribution Non-commercial License.

Study design

The study presented here was an exploratory pilot study using the principles of cluster randomisation.

Sample

In total, 432 patients were approached and provided with an information leaflet about the study. Of these, 178 agreed to participate, equating to a response rate of 41%. The sample demographics are presented in Table 13 (two participants did not provide their demographic details).

Number of reports per patient

There was a significant effect of condition (mechanism) on the number of reports per patient after controlling for the effects of the covariates (F_2,154 = 8.292, p < 0.01). Post hoc tests showed that patients in the face-to-face interviewing condition provided significantly higher numbers of reports than those in the paper-based form condition [t(154) = –3.064, p < 0.01] and those in the patient safety hotline condition [t(154) = –3.781, p < 0.01]. The mean scores for the number of reports per patient are provided in Table 14.

Likelihood of reporting

There was a significant association between the condition (type of reporting mechanism) and whether a patient reported one or more safety concerns [χ²(2) = 19.626, p < 0.01]. The contingency table is displayed in Table 15. In total, 64% of patients in the face-to-face interviewing condition reported one or more safety concerns, compared with 41% in the paper-based form condition and 19% in the patient safety hotline condition. Overall, nearly half (49%) of all patients reporting a safety concern were in the face-to-face condition. The standardised residuals indicate that fewer than expected patients reported a safety concern in the patient safety hotline condition and more than expected reported a safety concern in the face-to-face interviewing condition.

Number of patient reports classified as a patient safety incident

No association was found between condition and the number of reports classified as a PSI. Table 16 displays the numbers of PSIs by type of reporting mechanism.

Preventability and severity of patient reports classified as a patient safety incident

Because of the numbers of patient reports classified as PSIs it was not possible to undertake the planned chi-squared analyses. Summaries of the preventability and severity scores by condition are detailed in Table 17.

Nature of patient-reported safety concerns

The categorisation of the content of the patient-reported safety concerns and experiences is summarised in Table 18. To further illustrate this, Table 19 provides typical examples of the patient reports from each of the five overarching categories.

Conclusion

Previously published work in this area has been based on patients’ retrospective reports of safety-related information, the information in these reports having typically been generated using structured surveys. Our work heralds a significant departure from this approach. We have developed and tested a novel intervention, codesigned by patients, health-care professionals and researchers. Mindful of the tendency to locate PSI definitions in the clinical risk paradigm, we gave patients licence to define safety on their terms. We now know that hospitalised patients can report their concerns using a variety of mechanisms, but facilitated discussion at the bedside appears to be the preferred and most fruitful option. The diverse range of safety intelligence that patients can provide substantiates their position at the centre of care and their capacity to actively contribute to a more holistic view of risk.

Chapter summary

This chapter provides evidence about a key but previously unanswered issue: how best to collect feedback about safety. In doing so, it informs not just the further development of tools to collect the patient perspective on safety, but, more generally, it suggests that health services might be best exploring ways to speak to patients face-to-face rather than simply relying on passive methods of feedback collection. In addition, this chapter has explored how using an unstructured approach to soliciting feedback from patients can result in a wide range of safety-relevant data being sourced from an as yet untapped, but potentially crucial, perspective. The next chapter explores the potential role of patient feedback about safety as part of a range of error detection methods.

Publication statement

This chapter is based on a previously published paper. 257 We reproduce it here with permission from How might health services capture patient-reported safety concerns in a hospital setting? An exploratory pilot study of three mechanisms, O’Hara JK, Armitage G, Reynolds C, Coulson C, Thorp L, Din I, et al. ,257 2016, with permission from BMJ Publishing Group Ltd.

Setting and sample

This study was carried out between 5 September and 29 November 2011 in a university teaching hospital in northern England. The total number of patients to be recruited was driven by the PMOS, an associated intervention necessitating a specific sample size to meet predefined requirements for factor analysis. There were nine study wards, each of which were recruited through liaison with senior managers. The wards included three medical wards, three surgical wards, two maternity wards and a paediatric ward. In the last ward parents were able to report concerns specific to their child’s care.

Recruitment, measures and procedure

Data were collected by a team of trained recruiters, which consisted of research nurses, an operating department practitioner and research fellows, all specifically employed to work on this research programme. They were each allocated to collect data on specific wards; however, there was some minor variation in allocation because of recruiter sickness or absence. PIRT data were collected from all those patients who had disclosed and who were willing to disclose a concern about their safety and who had also completed the PMOS questionnaire described earlier. The team used the PIRT pro forma via face-to-face patient interview in the study wards (see Box 1). The research team visited each ward once a day from Monday to Friday over the study period. They first asked the ward manager’s permission to visit all resident patients except those who were very poorly, who were contagious or who lacked capacity to consent. Following permission, each patient considered appropriate to be approached was invited to participate. Those patients wishing to participate were provided with an information pack before being consented. We did not include patients who were unable to speak English. We excluded all non-reports from the comparative process, that is, those reports that did not include an identifiable safety-related concern.

We reviewed case notes for those patients on the study wards who had submitted patient safety concerns, which for the purpose of this study were defined by the former NPSA definition, now used by NHS England,253 as ‘any unintended or unexpected incident which could have or did lead to harm for one or more patients receiving NHS care’. We did not review case notes for those patients who did not consent to review or whose notes could not be accessed by the research team because of ongoing hospitalisation. To reduce any expectation bias, the review team were informed that the case notes had been randomly selected from all patients in the study wards in the study period, not only those who had submitted reports. Additionally, case notes for patients who had submitted reports were not allocated for review to the research team member who had collected the PIRT submission; this avoided the reviewer associating any memory of the patient report with the patient’s clinical profile.

The case notes were reviewed for the admission that coincided with the PIRT submission and covered all aspects of care up until the point of discharge from hospital. The review process was informed by a holistic, two-stage model of case note review, originally used in the Harvard Medical Practice Study239,261 and now routinely employed in clinical audit in the UK. This incorporated two stages: the stage 1 review was carried out by four members of the research team (CC, CR, LT and SM); if a PSI(s) was identified, a stage 2 review was carried out by a medical practitioner. The stage 2 team included five medical staff, four of whom were seconded to the quality and safety research team at the Bradford Institute for Health Research, and one anaesthetist from the study hospital who was not associated with any of the study wards. The stage 1 reviewers did not inform the stage 2 reviewers of the PSIs that they had identified.

Staff incident reports submitted from the study wards over the period of recruitment and data collection were collated and analysed retrospectively via an electronic reporting system. Data from PALS were supplied by the department concerned; there were no patient complaints received from the study wards over the recruitment period.

For the patient reports, two reviewers (CR and SM) individually identified PSIs before forwarding those patient reports with PSIs to two medical reviewers, who then confirmed or did not confirm agreement about the existence of PSIs. Two reviewers (CR and SM) counted PSIs from all sources and cross-referenced between the sources for agreed overlap.

Analysis

To compare the safety incident data from the various data sources with patients’ reported concerns, we matched dates (when possible) and patient identity through use of a specially created patient identity number (PIN) that corresponded with the patients’ unique Patient Administration System number. CR and SM compared all data sources and if there was an agreed overlap between sources this was recorded.

All patients in the study wards were consented before participating in the study. A NHS research ethics committee approved the study.

Patient reports

Analysis of each patient narrative revealed a wide range of patient concerns. Box 2 provides a range of illustrative examples of patient concerns that were reported only via the PIRT and, as can be seen, these emanate from observations of the immediate environment, including other patients’ care, the activities of the entire health-care team, from consultants to cleaners, and of course the impact of patients’ particular experiences. It can also be seen that patients appear especially interested in standards of hygiene and lack of attention to detail. Box 3 shows patient concerns that were also evident in other sources of patient safety data.

Evidence of the reporting of PSIs from multiple sources was very limited. Table 21 shows the overlaps. Only 11 patient-reported concerns were also evident in other sources; of these, four were evident in two or more sources. Case note review yielded a total of 1055 incidents. Staff incident reports and information about safety concerns reported by patients to PALS generated just 24 incidents, with no overlap.

Box 4 shows examples of PSIs considered documentation errors. However, it was difficult to determine whether some PSIs were documentation errors or not and examples of these are shown in Box 5.

Profile of patient reports

There was a variation in the number of patient reports by ward and specialty. Length of stay and complexity of care may be influential, as has been seen in adverse events studies in critical care settings,262 with patients being more likely to experience threats to their safety as a direct consequence of experiencing more interventions. We were unable to examine any association between either length of stay or care complexity for those patients who submitted more than one report. On the basis of our observations when visiting the study wards to recruit patients, we can add that those patients who were receiving the most complex care were also likely to be the most ill. Consequently, we did not collect data from these patients as this would, understandably, have violated the conditions of our ethical approval.

The content of patient reports is inevitably a reflection of care through the lens of patients’ experience and self-knowledge and many aspects of their experiences may be relatively invisible to clinical staff. 207 It would appear that patients have intently watched not only how care is given by the entire staff team but also the different standards of attention given to particular patient needs and their consequences. Whether patients were reporting concerns about their own care or concerns about the care of others, we argue that the content often reflected the possible precursors or antecedents of future PSIs, which possess an intrinsic learning value. A good example of this is the porter (see Box 2) who apparently did not ‘gel his hands’ prior to patient contact. However peripheral porters may be to clinical procedures, they routinely handle large numbers of patients, within and across wards, and possess considerable potential as a vector for infection. Conversely, many reports might also be judged by some staff as ‘soft issues’, especially those who conceptualise patient safety through a (clinical) outcome-driven lens. 207 Indeed some of the examples illustrate that patients’ reports could be described as perception driven and, consequently, they are affected by individual patients’ values, expectations and of course knowledge of their care and treatment. Furthermore, they can be contentious, as seen in the first report in Box 2. Although it is therefore accepted that there are limitations in the content validity of patient reports, it is also clear that some patients use this novel medium to identify phenomena that paint a vivid picture of safety in their care environment and which would either be causing distress or be likely to contribute to an eventual harm, albeit not often imminently. In summary, patient-reported concerns are centred on what has been called the ‘whole sequence of care’. 244 As indicated earlier, it is apparent that such phenomena may not be recorded in other sources of patient safety data.

Overlap of definitive patient safety incidents between different patient safety data sources

We found a distinct lack of overlap across different methods of collecting safety data, with different safety data sources identifying different PSIs. Our findings reflect those from several other studies that have compared the overlap between staff-based sources of patient data, such as incident reporting and case note review,241,243,260,263 but also those that have added patient sources to the sampling frame. 228,244

In this study we did not find any event that was evident in all data sources but there was some overlap in PSIs that had been identified through case note review. Although our findings confirm those from earlier studies that case note review is likely to provide the greatest number of PSIs compared with other established methods such as incident reporting,241,260,263 it may also be an indication of the multidisciplinary detail available in case notes and the systematic process that we applied to the process of analysis, one that was uniformly carried out by research staff. The total number of PSIs (n = 1055) identified from the second-stage case note review included a large number of documentation errors, which in many cases may not have led to any form of patient harm (e.g. incorrect terminology as defined in Box 4), although this cannot be certain because we found that the majority of documentation errors did not have any consequences recorded in the case notes. More importantly, however, the overlap between the various forms of PSI data was extremely small. This study has, however, further confirmed that the root causes of harm events are unlikely to be evident in either case notes,264 which record clinical information (measurable signs, symptoms and biometric data), or incident report data. 170

The paucity of the PALS data is significant. This may be a reflection of patient awareness about the service and its purpose but also that contacting PALS is something that must be instigated by patients or their relatives, unlike the PIRT process, which was brought to the patient and facilitated by staff (often with a clinical background) but independent of the study hospital.

Implications for patient safety and organisational learning

This study suggests that organisational learning can be enhanced by gathering patient-reported concerns about their safety and that the content of these reports varies greatly from PSIs identified from case note review, incident reports or other forms of institutionally sourced data. These concerns might also act as a very early warning of forthcoming harm events and, we argue, if part of an integrated analysis based on the full range of safety data sources available, may shed further light on the root causes of incidents that have already occurred. We believe that the first step in realising any potential is through asking patients the right questions at the right time for patients and having trained independent personnel to ask these questions. In this study both the questions and the approach to data collection were developed with patients and ratified by staff to be complementary to their existing means of gathering patient safety intelligence. Coproduction is an ideal method for shared decision making265 and systematically collecting and applying the patient perspective; coproduction equally recognises that a parallel staff perspective is crucial for successful implementation. The health-care staff involved in the PIRT development were able to hear the patient view outside of the frantic environment of practice and question that view without creating tension. We argue that this reduced the likelihood of the intervention being perceived by the greater body of staff as a top-down initiative borne out of an inappropriately idealistic, institutional drive to unconditionally prioritise the patient voice. However, the second step in realising the potential of patient reports is involving patients and staff in their analysis and the subsequent application of this knowledge in the risk management or clinical governance process. This remains a considerable challenge. Furthermore, the study described here was generously funded by a national research institution. In the current financial climate, it is hard to foresee health-care providers being able to implement such an initiative without additional finance unless those collecting these data are volunteers. We are currently investigating the feasibility of implementing this intervention (in tandem with PMOS) across three other acute health-care providers in the UK.

Limitations

This was a single-centre study but one that included a wide range of patients from different specialties. We were unable to recruit all of the patients from the wards during the study period, which would have provided a more reliable estimate of the number of PSIs perceptible to all patient admissions in a given ward in a specific time frame. Although we have confirmed that patients can report their concerns, based on our experience we contend that the volume of patients recruited is also likely to be an indicator of how comfortable UK patients currently feel about formally commenting on their care during its provision. Patients may well have experienced a PSI after having had the opportunity to provide data through the PIRT as they were asked on only one occasion if they had experienced a PSI. We did not review the case notes for all patients on the wards in the study period, which would have allowed identification of those PSIs not identified as concerns by patients (or at least by those who had consented). There were also some difficulties in accessing all staff incident reports through the existing electronic reporting system.

Conclusion

There is a growing evidence base which demonstrates that patients can, when asked, report on their safety and that of other patients. It is also apparent that these data and other sources of routinely collected patient safety data have remarkably little overlap. This study provides an important addition to this evidence base but also shows that patients can work effectively with staff to develop PIRTs that can then be used to gather patient feedback while they are receiving care. The collection of this type of patient-reported data requires a sensitive approach and one that is likely to have resource implications for health-care providers. The importance given to the patient voice, however, demands that health-care providers have a systematic means of collecting these data and ideally one that has been rigorously tested. The content of these reports can collectively provide a unique perspective on the overall process of care from the only participants who are constantly present and who can, through their ongoing experience, identify the antecedents of PSIs and the incidents themselves. Their purpose is to add another, as yet, largely invisible source of intelligence about patient safety as part of what must be a fully integrated, multimethod approach to organisational learning.

Chapter summary

The findings from this chapter suggest that speaking to patients about their experiences of safety may well provide health services with an, as yet, untapped perspective on error detection. However, as with all measurement within health care, this information will be of use only if it is somehow interpreted and acted on as part of ongoing service improvement. The next chapter outlines the bringing together of the PMOS survey and the PIRT to form the basis of a patient-centred patient safety intervention. With patient feedback about safety at the heart of this intervention, we move these two tools from simply measuring safety to being the foundation for patient-centric service improvement.

Development of the intervention

The PRASE intervention has two measurement tools at its core: the PMOS tool and the PIRT. The development of these two measurement tools has been described in previous chapters. However, to form an intervention, these tools needed to be part of a practical process that included the receipt of patient feedback about safety by ward teams and a means by which they might take action. To achieve this, we formed an intervention development group that included a number of managers responsible for patient safety and experience from the region, medical and nursing professionals, patient representatives and the research team. This group met four times over 4 months between January and May 2012. The remit of the group was to identify and explicate how the measurement tools might be used as a basis for an intervention focused on using patient feedback about safety to improve ward-level patient safety and how this would be implemented within acute NHS trusts.

The Patient Reporting and Action for a Safe Environment intervention

Under the guidance of the intervention development group, the PRASE intervention was developed as a cyclical improvement process, which used as its base the patient perspective on the safety of care. Figure 6 presents the outline cycle for the intervention.

FIGURE 6.

The PRASE intervention cycle.

Design

The study used a wait-list RCT design (Figure 7). A randomised design was used to test the feasibility of randomising wards (including impact on the recruitment of participants) and the acceptability of randomisation to a wait-list control group. To allow focus on intervention fidelity this randomisation was in a 2 : 1 ratio of intervention to control wards. Outcomes were measured at three points (baseline, 3 months and 6 months). Intervention wards received the PRASE intervention, with two 3-month cycles of patient feedback and action planning. Patients on control wards were also measured at three time points but these data were not fed back to staff until the end of the study. Data were collected between July 2012 and March 2013. Wards were randomly assigned to condition using a random number generator.

FIGURE 7.

Outline study design.

Setting and participants

This study was undertaken in a large acute teaching hospital in the north of England. Six wards were purposively sampled from the trust to represent a wide range of patient demographics, clinical specialty and type of admission (acute/elective). Specialties included in the study were (1) paediatric surgery, (2) ear, nose and throat surgery, (3) medical admissions, (4) orthopaedics, (5) renal, gastroenterology and rheumatology and (6) urology. There were two participant groups in this study: patients on participating wards and staff on participating wards.

Procedure

Research aim 1: exploring the feasibility of collecting feedback from patients about the safety of care

Research aim 2: exploring the feasibility and acceptability of the Patient Reporting and Action for a Safe Environment intervention process for staff

Research aim 3: exploring the feasibility of collecting feedback from staff about the culture of safety

Recommended changes to the Patient Reporting and Action for a Safe Environment intervention and cluster randomised controlled trial design

In light of our findings from the feasibility testing, the authors recommend the following changes to the PRASE intervention process.

• An increase in the time between cycles of measurement, feedback and action planning from 3 months to 6 months.

• Facilitated action planning meetings to encourage teams to integrate the different types of patient feedback and support longer-term systems-based change over inappropriate ‘quick fixes’.

• An augmented action planning pro forma to encourage staff to consider how they will measure success in addition to what action they will undertake.

• Better engagement with and support from senior management to support action planning and the implementation process.

• An initial training session to introduce the intervention to staff and describe more fully the process of using patient feedback for service improvement.

• Update sessions to provide an opportunity for shared learning between groups and with senior managers. Issues that transcend ward boundaries can be discussed and a wider improvement approach taken.

• Explicitly recognise (within the APG facilitation guidance) the need of the PRASE intervention to fit within other service improvement activity to avoid duplication of effort.

• Better support the process of collecting staff feedback about safety culture through education at training sessions as well as informal information sharing when meeting with ward representatives.

Developing a logic model

Having an explicit theory of change for the implementation and evaluation of service improvement interventions is becoming increasingly recognised as crucial for both researchers and improvers. 16 Figure 8 outlines the logic model based on the findings and recommendations of this study. This model helps to explicate the hypothesised relationships between the programme activities and the proximal and distal outcomes, providing a structure for further testing and evaluation. It was developed iteratively through discussion within the research team and is based on the findings from the qualitative research described in this chapter. The logic model forms the basis of the testing of the PRASE intervention in the cluster RCT described in the following chapter.

FIGURE 8.

Outline logic model for the PRASE intervention.

Limitations

This study had a number of limitations. First, we were limited to six wards within which to undertake this study. Although this was sufficient to address the feasibility study aims, it did not allow for estimates of effectiveness, which may have informed the subsequent study further. Second, with respect to the recruitment of patients into the study, it is possible that, in asking nursing staff about the capacity of patients to participate, a bias was introduced into the sampling regimen. However, it is strongly felt by the research team that this risk must be balanced against the greater risk of causing distress to patients unable to participate because of acute illness or capacity to consent. Furthermore, given that engagement with the intervention by ward staff is an important factor, it was felt by the team that we must at all times work with staff in ways that were acceptable to them.

Conclusion

The PRASE intervention represents an innovative approach for health services wishing to systematically collect feedback from patients about the safety of their care and use this as a basis for making improvements in the way that they deliver care. It is possible that the intervention will be as much about changing culture as about the specifics of service improvement. To this end it will be important for health services seeking to use the intervention to provide adequate resources to support it, in terms of senior management support, as well as the time and space for staff to learn about, interpret and act on patient feedback about safety. It will be crucial not to introduce this as just a top-down safety intervention (the next ‘audit’ to achieve), focused as it is on encouraging ‘bottom-up’ change from the patient perspective upwards. If wards engage wholeheartedly with the intervention, it is possible that this will aid the journey towards achieving greater transparency in how we deliver care, as well as a public commitment to codesigning shared solutions to patient-centred problems.

Chapter summary

This chapter described the feasibility testing of an innovative patient-centred patient safety intervention. The study demonstrated not only that it feasible to systematically collect information from patients about the safety of their care but also that staff – with support – were willing and able to engage with the feedback to make improvements to service design and care delivery. The following chapter will present a more rigorous examination of the intervention within a cluster RCT across three NHS trusts in the Yorkshire and Humber region.

Trial design

This study was a multicentre, cluster, wait-list design RCT to assess the efficacy of the PRASE intervention in achieving patient safety improvements over a 12-month period. A cluster design was utilised as the intervention was at the ward level, not at the individual patient level.

Participants

The study was conducted in 33 hospital wards spread across three NHS trusts over five different hospital sites (eight wards at a small district general hospital, 10 wards at a medium-sized teaching hospital and 15 wards at a large teaching hospital). We invited five trusts to participate and received responses from four, with one declining and three agreeing to participate. At the small district hospital, all adult non-intensive wards were recruited. At the medium-sized teaching hospital, the chief nurse recruited 10 adult wards, both surgical and medical, to the study. Within the final trust, wards were asked to volunteer to take part. Study participants were patients within participating wards. An average of 25 patients within each ward meeting the following eligibility criteria were recruited at three different time points.

Interventions

The PRASE intervention (described in earlier chapters) uses patient feedback about the safety of care as a means of achieving patient-centred service improvement and, ultimately, improved patient safety outcomes. In Chapter 7, the ‘logic’ of this intervention was explicated within a programme theory, which outlines the hypothesised ways in which the programme activities will lead to the desired ‘proximal’ and ‘distal’ outcomes. Following the feasibility study, and mindful of the recommended changes to the process of delivering the intervention, the PRASE intervention was modified slightly prior to the cluster RCT described here. In line with the programme theory, the PRASE intervention was designed to collect feedback from patients about the safety of their care, using robust tools. This patient feedback was then collated and presented to each ward in a formal ‘feedback report’. During a facilitated action planning meeting, ward staff used an action planning pro forma to agree on a set of ward-specific improvements/actions to address areas of patient concern. The philosophy of this intervention is that it is an iterative process with a cycle of measurement, feedback and change lasting for a period of 6 months. Ward staff engaged in two cycles during the 12-month intervention period. In response to some of the key moderating factors identified within the programme theory, within this cluster RCT the intervention now included (1) a ‘start-up’ meeting at which all intervention wards within each trust met together to better understand the PRASE intervention, systems approaches to service redesign and the importance of patient feedback for achieving patient safety improvement; (2) a ‘mid-point’ (6-month) meeting for intervention wards at which staff could discuss any collective learning and actions that might require senior-level support; (3) visible support from senior managers, with the director of nursing and medical director invited to attend these two learning sessions; (4) action planning sessions facilitated by members of the research team, to support action planning and discussion of patient feedback; and (5) explicit guidance about the importance of action planning within multidisciplinary teams. As this represented an efficacy rather than effectiveness trial, the intervention required extra resources in the form of (1) data collectors (in this case research nurses), (2) a facilitator to support action planning and (3) the time of health-care staff to develop and implement actions.

Control wards received no intervention during the study period. At the end of the study, control wards received a feedback report for the three time points, amalgamated into one. They were also provided with a 1-hour training session on how to interpret and use their data.

Intervention fidelity

Our assessment of fidelity was informed by the programme theory for the intervention. The fidelity assessment was designed to aid our understanding of the degree to which intervention wards adhered to key components of the programme activities, the effect of a number of the key moderating factors and our achievement of a number of the desired proximal outcomes. Adherence was assessed in relation to eight components of the intervention listed below; each was rated independently by three members of the research team. We followed a framework proposed by Carroll et al. 272

1. Attendance of at least one ward representative at the orientation meeting (0 = no; 3 = yes).

2. Multidisciplinary team present at the phase 1 action planning meeting (0 = no action planning meeting; 1 = one staff group represented; 2 = at least two staff groups represented; 3 = more than two staff groups represented).

3. Creation of action plans in phase 1 (0 = no action plans; 1 = limited action plans including mainly quick fixes; 2 = considered action plans reflecting issues identified, with the potential for short-term impact; and 3 = as 2, but with the potential as longer-term solutions).

4. The extent to which action plans were implemented in phase 1 (0 = no implementation; 1 = at least one action plan partially implemented; 2 = most action plans implemented; and 3 = all action plans implemented).

5. Multidisciplinary team present at the phase 2 action planning meeting (as above).

6. Creation of action plans in phase 2 (as above).

7. Implementation of action plans in phase 2 (as above).

8. Attendance of at least one ward representative at the mid-point meeting (as orientation meeting).

It was decided a priori that scores of 0 and 1 would be classed as non-adherence, whereas scores of 2 or 3 would imply adherence.

Outcomes

Each trust had a start-up meeting following randomisation. There were three outcome measurement periods – at baseline (pre randomisation) and at 6 and 12 months post start-up.

Sample size

The study was powered to detect a small to medium difference (effect size = 0.3) between the allocated groups with respect to the PMOS score (0–100). A small to medium effect size seemed reasonable as each ward would be focusing on developing and implementing tailored action plans, meaning that the intervention might not impact on all areas measured by the PST or PMOS tool. To achieve 80% power (with alpha = 0.05) with an average cluster size of 25 patients and an assumed intracluster correlation coefficient (ICC) of 0.05, 32 wards were required (16 per arm). This estimate of the ICC was considered reasonable given the secondary care setting and patient-reported outcome. 39

Randomisation

Wards were randomly assigned to the intervention or control group on an equal basis by the trial statistician at York Trials Unit using a secure computer system. Minimisation was used to balance the groups with respect to ward type (medical or surgical), age (low, middle or high based on tertiles of the distribution from all wards), male/female/mixed sex wards and ward size (low, middle or high based on tertiles of the distribution from all wards).

Blinding

Baseline data were collected at least 1 month before randomisation. The nature of the intervention meant that it was not possible to blind wards. All meetings with wards, creation of feedback reports and facilitation of action planning meetings were conducted by researchers not involved in data collection. Blinding of research nurses collecting data was planned so as not to bias responses but in reality some would be inadvertently unblinded through conversations with ward managers.

Statistical methods

Analysis was conducted in Stata version 13 (StataCorp LP, College Station, TX, USA) using the principles of ‘intention to treat’, meaning that wards (and associated patients) were analysed according to the trial arm that they were randomised to, regardless of intervention implementation or fidelity. Minimisation factors and clustering were accounted for when required. Statistical significance was assessed at the two-sided 5% level. For regression analyses, the difference in adjusted (least-square) means are summarised with 95% CIs. Mixed regression took a restricted maximum likelihood (REML) approach with random effects to account for any potential clustering by ward. Effect size estimates presented are Cohen’s d effect sizes. Estimates of ICC are presented for both primary outcomes.

Participant flow

Consolidated Standards of Reporting Trials (CONSORT) diagrams were used to show the flow of wards through the trial and cluster size at each time point in relation to primary outcomes. 273

Primary analyses

Secondary analyses

Participant flow and recruitment

Thirty-four wards were recruited, with one ward excluded prior to randomisation because of an insufficient number of patients available. Participant data collection periods ran between May and July 2013 for baseline, between January and April 2014 for 6 months post start-up and between June and September 2014 for 12 months post start-up.

Figure 9 shows a CONSORT diagram for the PMOS outcome. The percentage of patients who were approached who provided feedback was 89%, 88% and 82% for baseline, 6 months and 12 months, respectively, giving an overall completion rate of 86%.The average cluster size was 25, as planned in the sample size calculation. Figure 10 shows an analogous diagram for the PST outcome; cluster sizes are large as 3 months’ worth of data were used at each time point. Thirty-three wards were randomised (16 allocated to control, 17 to intervention), with all wards retained throughout the trial.

FIGURE 9.

Consolidated Standards of Reporting Trials (CONSORT) diagram relating to PMOS data.

FIGURE 10.

Consolidated Standards of Reporting Trials (CONSORT) diagram relating to PST data.

Implementation of the intervention

Table 29 shows summary statistics for fidelity scores for aspects of the intervention. Multidisciplinary team scores were slightly higher at the first meeting than at the second [1.88 (SD 0.86) vs. 1.18 (SD 0.64)]. Fidelity related to the action plan and implementation was similar in both phases; however, a score was not available for three wards for phase 1. Figure 11 shows the number of components that intervention wards complied with; 11 of the 17 (64.7%) intervention wards complied with at least 50% of the intervention at 12 months and four intervention wards (23.5%) complied with at least 75% of the intervention at 12 months.

FIGURE 11.

Compliance with elements of the intervention for each of the 17 intervention wards.

Baseline data

Summary statistics for the minimisation factors are reported in Table 30; all proportions were fairly similar between allocated groups.

One ward was incorrectly listed as being a female-only ward when it was a mixed ward; summaries reported are corrected. Analyses were conducted treating this ward as a female-only ward (as randomised).

Ward characteristics

Table 31 shows summary statistics for ward characteristics over the study period (May 2013 to September 2014); the allocated groups are generally well balanced. Overall, the mean size of wards was between 25 and 26 beds (range 10–32 beds). The number of discharges ranged between 32 and 223 patients per month with a mean of 121 patients per month and the rate of staff sickness ranged between 1.9% and 9.4% with a mean of 4.9%.

Summary statistics for patients completing the PMOS questionnaire at each time point are presented by trial arm in Table 32. At baseline, mean age was similar between the randomised groups as was the proportion of women. Most patients were British (92% in both trial arms) and just fewer than 40% of patients in each group were receiving ongoing treatment elsewhere at the time of data collection. The mean number of times that participants had been an inpatient in the hospital trust in the past 5 years was just over two in both groups and the mean length of time spent in hospital until enrolment in the trial was around 7 days. Balance was also observed on all variables at 6 months and 12 months when comparing the intervention and control groups.

Patient Safety Thermometer

Table 33 shows summary statistics for PST score. The sample size was > 1000 and the mean percentage HFC was high (> 90%) at each time point. Percentages were similar in both allocated groups and were higher at 6 months than at baseline or 12 months.

The mean cluster size per ward was similar in both groups. There were no missing data in relation to the PST at 6 or 12 months; however, one ward had no baseline percentage score because of missing data for a required month and hence was excluded from the primary analysis.

Patient Safety Thermometer secondary analyses

An analogous model to the primary analysis to assess differences in the percentage of HFC at 6 months showed no evidence of a difference between the allocation groups (p = 0.58), with a non-significant increase of 1.09% for intervention wards compared with control wards (95% CI –2.94% to 5.12%, effect size 0.20).

Figure 12 shows the mean PST HFC by allocated group at baseline alongside the adjusted HFC at 6 and 12 months. There was a larger, although still non-significant, difference between the two allocated groups at 6 months than at 12 months. The baseline average was lower in the intervention group than in the control group. This was not the case at 6 months; however, very little difference was seen between the two groups at the primary end point.

FIGURE 12.

Mean PST scores by allocated group at baseline, 6 months and 12 months.

Patient Measure of Safety questionnaire

Mean cluster size was very similar between the allocated groups at each time point and was 25 overall as planned, meaning that the study was adequately powered in relation to the PMOS tool.

Table 35 shows summary statistics for the PMOS completion method. The proportion of self-completion was relatively low at each time point, with a maximum of just under 25%; the majority of completions were facilitated.

Regression analyses

Linear mixed regression models were used to compare differences in PMOS scores between the two allocated groups. Neither the overall PMOS score nor any domain scores (Tables 37 and 38, respectively) showed evidence of a difference between the intervention wards and the control wards at 12 months. The ICC ranged between 0.01 and 0.03; these values are < 0.05, which was the ICC assumed for the sample size correlation. This does not imply any loss of power based on correlation.

TABLE 37 - Summary of the overall PMOS score results

Overall PMOS score	Difference (95% CI)	p-value
Primary analysis	0.08 (–0.01 to 0.17)	0.09
With adjustment for ward characteristics	0.05 (–0.04 to 0.14)	0.27
With adjustment for method of completion	0.08 (–0.01 to 0.16)	0.08
CACE analysis (50% compliance cut-off)	0.15 (–0.05 to 0.36)	0.13
CACE analysis (75% compliance cut-off)	0.43 (–0.28 to 1.14)	0.23

TABLE 38 - Summary of the PMOS domain regression analysis results

Intervention – control, so positive values indicate a more positive response in the intervention group.

PMOS domain	n	Differencea (95% CI)	p-value	Effect size	ICC
Access	807	0.00 (–0.09 to 0.09)	0.93	–0.01	0.01
Communication and teamwork	815	0.01 (–0.07 to 0.10)	0.81	0.02	0.01
Delays	766	0.09 (–0.08 to 0.25)	0.29	0.10	0.02
Equipment	734	–0.01 (–0.13 to 0.12)	0.93	–0.01	0.03
Information flow	799	0.02 (–0.09 to 0.14)	0.71	0.04	0.02
Organisation and care planning	810	0.02 (–0.08 to 0.11)	0.69	0.04	0.01
Staff roles and responsibilities	817	0.00 (–0.16 to 0.17)	0.96	0.01	0.03
Staff training	726	–0.01 (–0.12 to 0.11)	0.92	–0.01	0.01
Ward type and layout	817	0.02 (–0.06 to 0.10)	0.65	0.04	0.01

Table 37 shows a summary of the results from the analyses conducted on the overall PMOS scores. The primary analysis, conducted on 656 individuals, showed a non-significant difference of 0.08 (95% CI –0.01 to 0.17, ES = 0.20; p = 0.09) between the two allocated groups, with no evidence that this difference was significant. The marginal mean was lower in the control group (3.96, 95% CI 3.90 to 4.02) than in the intervention group (4.04, 95% CI 3.98 to 4.10). The ICC associated with the overall score was estimated at 0.03. Sensitivity analyses similarly showed no evidence of a difference between the allocated groups.

Figure 13 shows the mean overall PMOS scores by allocated group at baseline alongside adjusted scores at 6 and 12 months. Intervention group scores were slightly lower at baseline than control group scores; this was reversed at 12 months but the differences were non-significant.

FIGURE 13.

Mean overall PMOS scores by allocated group at baseline, 6 months and 12 months.

Secondary outcomes analysis

Health economic analysis

Resource use for all health- and non-health-related resources is presented in Table 42.

The mean time spent with patients collecting data was similar between the allocated groups, with a non-significant difference of 0.36 minutes in favour of the intervention (95% CI –0.38 to 1.11; p = 0.34).

Action planning meetings that took place during phase 1 lasted on average 46.23 (SD 14.00) minutes and were attended by 3.82 (SD 1.81) members of staff. During the second phase those meetings were shorter [40.11 (SD 19.02) minutes] and were attended by fewer members of staff [2.23 (SD 1.25)].

A total of 89 action plans were produced. Of the 89 actions, 53 (60%) were planned during phase 1 and 36 (40%) during phase 2. There was only one ward that designed at least one action per category. Table 43 shows the focus of the action plans. One-third aimed to improve roles and responsibilities and over one-quarter aimed to improve communication and information exchange. The mean number of actions associated with roles and responsibilities was 0.85 (SD 0.65) per ward, with a maximum of two actions within this category. The median number of actions was zero for all classes of actions except for the roles and responsibilities and improving communication categories.

The results show that most wards considered that patient safety and quality can be developed by means of actions focused on improving roles and communication. It was common to find that staff roles were unclear or that patients were unsure which nurse or consultant was responsible for them. Actions relating to roles and communication frequently aimed to (i) ensure guidance and supervision in the absence of senior staff; (ii) ensure clear allocation of staff and planning of the team each day or (iii) define team roles and responsibilities for the prevention of falls and infections. Often actions were straightforward such as buying whiteboards to show different staff roles or bedside handovers with explicit definitions of ‘who is in charge’. Noise was another major concern for patients. Actions to reduce noise ranged from low-cost activities such as earplugs or soft-close bins to more complex activities that involved education of staff about the implications of noise for the patient experience.

The results of the cost–consequence analysis are shown in Table 44.

Summary of the key findings

The cluster RCT reported here ran smoothly. All recruited wards were retained throughout the study and data were collected for an average of 25 patients per ward, as planned. This, together with a lower ICC than expected (0.03 rather than 0.05), means that the trial was adequately powered to detect a medium effect size with respect to the primary outcomes.

Despite the ability to retain wards, adherence with the intervention was poor, with 11 wards adhering with 50% of scored intervention components and only four wards complying with 75%. Ward teams struggled to pull a multidisciplinary team together (particularly in the second cycle) and to develop and implement high-quality action plans.

The primary outcome of mean percentage HFC was high (> 90%), meaning that achieving a substantial increase was challenging. The percentage HFC also demonstrated some seasonal variation but not in the anticipated direction, such that scores were higher (better) at 6 months than they were at baseline or 12 months, despite the 6-month follow-up being in winter when one might anticipate that winter pressures would impact patient safety negatively. These unexpected findings require further investigation.

For the primary PST outcome there was no evidence of a difference between the allocated groups at 12 months (p = 0.99) or 6 months (p = 0.58), despite the intervention wards showing some improvement over time, which was not evident for the control wards. Results from the CACE analysis were consistent, suggesting that intervention wards that adhered to the intervention did not achieve a significantly greater improvement in safety than comparable control wards

There was also no evidence of a difference in HFC between the allocated groups at 12 months when considering only new harms arising during the care of patients on the study wards (p = 0.15), However, the difference between the two groups was larger (1.60%) for this outcome, with an effect size of 0.51. It is perhaps unsurprising that HFC occurring on the study wards themselves rather than elsewhere in the hospital might be a more sensitive measure of a ward-based safety intervention. This is a more positive finding but still non-significant at conventional levels. However, the CACE analyses relating to this outcome identified a difference as large as 5.38% for wards adhering with 75% of the intervention. Thus, those intervention wards that engaged fully with the intervention showed a much larger increase in HFC (for new harms) than comparable control wards.

The second primary outcome was the PMOS questionnaire. This measure was developed during the course of the programme grant (see previous chapters). For this reason, the completion rate, reliability and validity of the PMOS measure were assessed again in this chapter. A high percentage of patients completed a sufficient number of items to receive an overall score (85.6%). Most missing data resulted from patients responding that items were not applicable. The majority of patients were facilitated by the researchers to answer the questions; those self-completing were generally younger and more likely to be male. The overall score showed reliability with high Cronbach’s alpha values at each time point (> 0.9) and predictive validity in known group comparisons such that percentage HFC was higher in wards where PMOS scores were higher.

For this second primary outcome there was no evidence of a difference between allocated groups in relation to the PMOS overall score at 6 or 12 months, although, again, the control group remained the same throughout, whereas the intervention group did show some improvement. CACE analyses also demonstrated a non-significant effect when considering adhering wards although the results do suggest a trend in favour of the intervention wards, with improvements in PMOS scores being largest for those wards that adhered to the intervention.

No significant differences between allocated groups were identified for the secondary outcomes of recommendation by friends and family and staff safety culture. However, the staff grading of the safety of the ward did show some interesting trends. At baseline 70.5% of control ward respondents rated their ward either excellent or very good, whereas at 12 months this proportion was slightly lower at 68.5%. In the intervention wards, 74.7% graded positively at baseline, whereas this dropped to 62.2% at 12 months. The greater decrease in perceptions of safety of the ward in the intervention group may be a direct result of obtaining feedback from patients that highlights possible areas for improvement or from attempts to make improvements on the basis of feedback that have been difficult to implement. Being non-significant, these changes over time may of course just be down to chance.

The main costs associated with the intervention arise from staff time (i) from involvement in the design and implementation of actions and (ii) associated with patient data collection. It is worth highlighting that future models for the PRASE intervention might use volunteers to collect patient data, which would reduce the overall cost of the intervention.

Cost–consequences analyses are not commonly conducted to recommend whether an intervention is worth funding as that judgement is usually related to the gains that would likely be achieved by the patients. However, the intervention was not found to result in any significant differences for outcomes. Combined with the cost of delivery of approximately £2040 per ward and the cost of implementation of £1018.80 per ward it is highly unlikely that the intervention is worth funding. Consequently, given that the opportunity costs are likely to exceed the benefits, policy makers may wish to consider other options for investment to improve health gains for NHS patients.

Interpretation of findings

The main finding of this trial is that an intervention based on feedback from patients to wards about the safety of their care was not effective in improving patient safety during the 12-month study period. There are many possible explanations for this but the two most probable – measurement and fidelity – are discussed in the following sections.

Limitations

This trial was adequately powered and conducted across different wards from a range of hospitals. Data collection proceeded as anticipated. One of the key limitations of this trial was that we did not collect information about the impact of completing a questionnaire about safety on the patients themselves. It is possible that completing such a measure might lead to improvements in patients’ knowledge and understanding of safety, which may change behaviour during the current or subsequent hospital visits. Our model for change focused on the feedback given to staff and the changes in their behaviour rather than patient-mediated change. Although we attempted to blind research nurses collecting the data, several wards identified themselves as being intervention wards through comments about engaging in action planning. Although unlikely, this may have had an impact on the way that research nurses asked questions when collecting feedback from patients.

Generalisability of the findings

We attempted to recruit three very different NHS trusts to this trial (large, medium and small; teaching and non-teaching; one and two or more sites) and to involve a range of different types of surgical and medical wards to make the findings generalisable across the acute care sector. However, we did not engage any paediatric, emergency, critical care or maternity wards/units and therefore it is not possible to generalise our findings to these populations. Critical care patients are unlikely to be able to provide feedback because of their status as seriously ill patients. Patients in accident and emergency care departments stay for only a short period and may also be seriously ill, meaning that the intervention may not be appropriate in these settings. Although possible to deploy the intervention in maternity and paediatric wards, none was recruited into the current study. This may be because patient engagement is already strongly promoted in these settings, but it is possible that this cultural maturity may serve to enhance the impact of an intervention such as the PRASE intervention and therefore evaluation of this intervention in these settings is important.

Conclusion

This intervention, based on collecting information from patients on the safety of their care environment and feeding this back to staff, was not effective based on our primary outcome measures. A secondary analysis revealed that for those wards that were complying with the intervention and in which only new harms were measured, the impact of the intervention was greater. This is worthy of further exploration.

Together with the cost of delivering the PRASE intervention, these two factors would suggest that it is highly unlikely that the PRASE intervention is worth funding.

The collection of data from patients was feasible, with 86% of patients approached willing to respond. The main challenge in implementation was the development and implementation of action plans by ward staff to improve safety.

Chapter summary

This chapter has presented the findings from an ambitious cluster RCT testing the efficacy of a novel patient-centred patient safety intervention. Although the authors found no direct effects on the patient safety outcomes measured, what is clear is that patients are willing to provide information that supports the proactive assessment of risk, as well as the retrospective identification of safety concerns, and that this combination of data can be used as a basis for service improvement. However, there are other potential ways that patients can be involved in improving patient safety within hospital settings and the next chapter focuses on one of these – directly engaging with health-care professionals to improve safety and reduce harm.

Background

A number of current patient safety improvement initiatives prompt patients to be vigilant about their health care and to intervene directly if they have a concern or think that there has been a mistake (e.g. the Speak Up initiative in the USA39 and the Please Ask campaign in the UK40). A common goal of these patient-mediated initiatives is to ‘push’ improvement by encouraging patients to ‘check’ the care that they receive and to speak directly to staff if they have any concerns or questions. Evidence suggests that patients would be willing to take on such a role in improving their safety284,285 but that this is contingent on a number of factors, including illness severity and type,286,287 the proposed action12,179 or question12,13,179,288,289 and their self-confidence. 222,290 Patient involvement also depends on the receptiveness of staff (real or perceived)291–297 and related concerns about damage to the patient–provider relationship. 297 This questions the acceptability and effectiveness of patient-push approaches and highlights a need to better understand how best to involve patients in improving their own safety.

A key criticism of initiatives using a patient-push approach has been the apparent lack of involvement of patients, their families or frontline staff in their development, a lack of theoretical rationale for the choice of intervention approach or materials and a dearth of robust evaluation. This project aimed to address this major gap through the systematic development of a fully piloted intervention grounded in patient and professional experience, underpinned by relevant theory and informed by current evidence and best practice.

Ethical statement

The research was approved by the Newcastle and North Tyneside 2 Research Ethics Committee (reference 10/H0907/24 for phases 1 and 2) in May 2010 and by the National Research Ethics Service (NRES) Committee (proportionate review) East of England – Norfolk (reference 12/EE/0277) in June 2012. NHS research and development research governance approval was obtained from both participating trusts for work undertaken from May 2010 and from June 2012.

Clinical setting

The clinical setting was UK secondary care, involving two NHS foundation trusts in the north-east of England. Patient and staff participants were recruited from acute and elective general medical and surgical wards across a range of clinical specialties.

Aim of the overall project

The aim of this study was to develop an intervention(s) that supports patients to work collaboratively with health-care staff to improve safety and reduce the risk of harm.

Design of the overall project

Guided by the MRC framework for developing and evaluating complex interventions267 the project progressed across four phases, using methods appropriate to a series of objectives relevant to each phase (Table 45).

International scoping exercise of ongoing work to identify best practice

A scoping exercise was undertaken of local, national and international ongoing initiatives, including identifying any evidence of piloting or evaluation, theoretical underpinning and/or evidence of patient and family involvement in the development of the identified intervention approaches and materials (Table 46). Initiatives were identified through personal contacts with organisational leads and experts and by visiting organisations’ websites and related external links.

Examples of intervention materials identified as a result of the scoping exercise were used to elicit patient and staff perceptions about the use of these approaches by patients in the UK NHS acute care setting during subsequent qualitative interviews (see Qualitative study). The learning and resources were also used during the intervention development phase (phase 2) to guide the form of the prototype intervention(s) and the chosen mode of delivery.

Systematic review

A previous systematic review of the effectiveness of interventions to involve patients in improving patient safety identified 15 eligible studies reporting on a range of interventions to improve medication safety. 180 This original review found evidence in favour only of patient self-management of medication (mainly in relation to anticoagulant therapy), but little evidence for other approaches to improving safety. The review was updated and expanded to:

1. identify evaluations of interventions published subsequent to the original review

2. extract data across the full review relating to the process of intervention development and the extent, if any, of patient involvement in this process

3. extract data relating to behaviour change techniques (BCTs) used in the interventions described in all included studies, with a focus on BCTs associated with successful interventions [defined for the purposes of this work as present more frequently (> 50%) in positive studies than in negative studies].

Qualitative study

A qualitative study297 was undertaken to explore patients’, relatives’ (service users’) and health-care staff members’:

• understanding and experience of patient safety and the potential patient/family role in harm prevention

• attitudes towards, and beliefs about, patient and family involvement in improving safety

• perspectives on how best to support the involvement of patients and families, including potential barriers and facilitators.

Novel ideas workshops

Participants in the qualitative study struggled to come up with novel ideas for involving service users in improving patient safety.

To address this, two brainstorming workshops, one with experts in the field of safety in health care, ergonomics and human factors and one with expert patients including regional Local Involvement Networks (LINks) leads and patient safety champions, were held. Using a specific creative thinking technique called ‘bootlegging’,308 these workshops generated ‘out of the box’ thinking and a number of novel approaches to using technology and social marketing to promote and support patient involvement (Table 48). A broad theme across ideas generated within both workshops was again of facilitating the exchange of information between patients and relatives and their health-care providers.

Identification of relevant theory

Previous work has found that few studies reported a theoretical underpinning for the choice of intervention approach. 180 A theoretical rationale provides understanding of how interventions are intended to work (i.e. the proposed mechanisms of change) and where and when they might have their effect (i.e. at what point in the causal chain). To guide the systematic development of a theoretical basis for the project intervention a three-step process was used. 309

1. Key themes and related beliefs identified by the qualitative study were mapped onto behavioural dimensions within the theoretical domains framework (TDF)310 (Figure 14). The TDF is a tool derived from psychological theories of behaviour and suggests behavioural determinants that can then be directly targeted within the study intervention. Six TDF domains were relevant to understanding the asking and telling behaviours to be promoted in patients and five were relevant to understanding the ‘fostering engagement’ behaviour to be promoted in staff:

   • domains relevant to both:

     • motivation and goals

     • beliefs about capabilities

     • beliefs about consequences

     • social/professional role and identity

     • nature of the behaviour

   • additional domain relevant to patients:

     • social influences.

2. To further support targeted behaviour change, behavioural domains identified in step 1 were then mapped on to relevant BCTs303 using a related matrix311 (see Figure 14).

3. Identifying relevant theory involved mapping the domains identified as key to explaining patient involvement behaviour back to constituent theories (Figure 15). Systematic methods for doing this are less well developed312 but a number of established theoretical models of behaviour (listed in the appendix of Michie et al. 310) were identified as potential frameworks for understanding patient and staff behaviour in relation to patient involvement in improving patient safety. Constructs within TDF domains identified by step 1 were present in two theories of motivation and three theories of action (see Figure 15). Theories identified included the theory of planned behaviour,313 social cognitive theory,314 self-regulation theory315 and learning theory. 316 The rationale for selecting these theories was supported by evidence on their previous use in studies examining patient involvement in improving patient safety222 and as models for understanding and changing patient and health-care professional behaviours309,317–319 and by their direct linkage to BCTs (social cognitive theory and self-regulation theory). 311

FIGURE 14.

Systematic mapping of qualitative themes to the TDF and to candidate BCTs (steps 1 and 2).

FIGURE 15.

Mapping domain constructs to theory. Bold text = theoretical domain. a, Mapped to BCTs only; b, addressed by Talk Time component.

A further theory – role theory320 – which is not included in the TDF list (as it is not a theory of behaviour change), was also included as part of the analytical framework, as most of the themes emerging from the qualitative study related to traditional patient and health-care role perceptions. Support for the utility of this theory for understanding the patient and relative perspective in relation to a patient role in improving safety is provided by previous qualitative work in this area. 291

These theories formed the basis for the initial analytical framework for the pilot evaluation trial undertaken in phase 3 of the study. The framework facilitates modelling of the promoted patient and health-care professional patient safety behaviours and testing of the hypothesised relationships between behavioural antecedents and behaviour, and behaviour and patient safety indicators.

Intervention components

Pilot and preliminary evaluation of ThinkSAFE

Methods

The evaluation design was a non-randomised before-and-after study with a parallel control group (Figure 17). Twelve wards (six within each trust) took part in the pilot study; eight received the intervention and four acted as controls. Mixed-methods evaluation318–322 was used to investigate a range of process and outcome measures relevant to the intervention approach (Table 49). Process measures evaluated intervention fidelity, acceptability and feasibility and behavioural antecedents in both service user and health-care staff participants. Outcome measures evaluated the potential impact of the ThinkSAFE intervention on self- and proxy-reported service user and staff behaviours and on medication safety. Medication safety was chosen because of the higher rate of opportunity for patient involvement at different time points across the inpatient stay, the persistently high frequency of particular error types322–326 and the existence of robust, routine methods of measurement of these errors. 259 A power calculation based on published error rates300–302,321–324 for the outcomes targeted by the tracer topic suggested that 230 patient records per group would be needed (460 in total) to detect a difference in proportions of unintended medication omissions of between 50% and 36.5% (13.5% difference) with 80% power, allowing for a 10% loss to follow-up. Sample sizes to allow the reliable capture of safety events relating to hospital-acquired infection and patient deterioration were beyond the scope of a pilot evaluation.

FIGURE 17.

Exploratory pilot trial design. DVD, digital versatile disc.

TABLE 49 - Outcome and process measures

CAHPS, Consumer Assessment of Healthcare Providers and Systems; NICE, National Institute for Health and Care Excellence.

NICE 2007 – see www.npsa.nhs.uk/corporate/news/guidance-to-improve-medicines-reconciliation/ (accessed 8 August 2016); ARQH 2011 – see www.ahrq.gov/sites/default/files/publications/files/match.pdf (accessed 8 August 2016); International SOP – see www.who.int/patientsafety/implementation/solutions/high5s/h5s-sop.pdf (accessed 8 August 2016).

Instrument	Informed by	Outcome measures	Process measures
ThinkSAFE patient safety questionnaire	Theoretical framework derived from phase 1 project work (theory of planned behaviour, social cognitive theory, learning theory, role theory) UK NHS and US CAHPS national surveys	Motivation (willingness, intention) Behaviour (self-report and proxy report): self-efficacy satisfaction level of perceived involvement staff retention; morale; sickness	Cognitions and beliefs regarding patient involvement in improving safety: confidence outcome expectancies, consequences normative, control and role beliefs attitudes perceptions of trust
Medications reconciliation audit tool	Review of current audit tools and guidance (UK and international, e.g. NICE 2007; Bradford Institute 2011; ARQH 2008; International SOP 2011a)	Unintended omissions Unintended dose/frequency errors in: initial admission prescription chart (home medications) discharge prescription (all medications)
Interview topic guide	Normalisation process theory, TDF	Experience: knowledge/understanding use of intervention	Intervention fidelity Acceptability Feasibility
Evaluation framework	Implementation literature272,327		Organisational issues and buy-in Intervention fidelity

Measures (see Table 49)

• Theory-based, self-completion questionnaires for both patients and staff captured pre-intervention (baseline) and post-intervention data for behavioural process and outcome measures (see Appendix 11). Items were measured on a 5-point Likert scale. To enable comparison of the evaluation data with existing survey data, relevant items from validated UK305 and US325 national patient and staff survey instruments were included in the post-evaluation questionnaire. The post-intervention questionnaire asked patient intervention respondents if they had received a logbook and if they had viewed the video/digital versatile disc (DVD).

• A standardised medications reconciliation audit tool was devised in collaboration with pharmacy teams at both participating trusts, which captured frequency data on target medication errors (see Appendix 12).

• Post-intervention, face-to-face interviews with patients and/or relatives and health-care staff exposed to the intervention explored user experience and the feasibility and acceptability of ThinkSAFE.

• Data on the fidelity of the intervention approach was also captured both quantitatively and qualitatively (numbers of logbooks distributed, exposure of ward staff to educational session, staff and patient report of process and use of the intervention).

Setting

The pilot took place on six surgical and six medical wards across three hospitals within two NHS foundation trusts in the north-east of England. Purposive sampling was used to identify wards receiving both acute and elective admissions across a range of specialties and to match intervention and control wards on these features as far as possible. The evaluation began on all control wards on 1 September 2012 and ran without interruption to 14 December 2012. Each control ward aimed to sample around 60 patients at admission and at discharge. Start-up on intervention wards was staggered across this evaluation period, with each intervention ward aiming to sample 30–35 patients during a 2- to 4-week active intervention period.

Participants

Participants were adult elective or acute patients (≥ 18 years) admitted to the participating wards during a 2- to 4-week evaluation period for a stay of ≥ 2 nights (actual or anticipated). An opt-out, blanket approach was used in the delivery and evaluation of the ThinkSAFE approach. When patients lacked capacity or were too ill to participate in the evaluation study, relatives or carers were invited to participate. All staff (nurses, doctors, health-care assistants, ward pharmacists) working on participating wards during the evaluation period were eligible to take part.

Pre-intervention survey

Patients: elective admissions only

Two weeks prior to their admission, patients on elective lists were sent a study pack containing a letter of invitation from the research team, information about the survey, the theory-based questionnaire and a pre-paid envelope for return of the questionnaire directly to the research team within 1 week, prior to receipt of the intervention materials. Patient questionnaires were returned anonymously.

Staff

All staff on participating wards (intervention and control) were provided with a study pack 3 weeks before the ward’s evaluation period. Staff questionnaires were marked with a participant code to enable matching of pre- and post-intervention questionnaire data.

Pre-intervention set-up and delivery of the staff intervention

Intervention wards were set up 2 weeks in advance of their evaluation period to enable the organisational-dependent aspects of the intervention process to be arranged (e.g. time within current routines to provide the Talk Time sessions) and advance rehearsal of the intervention process. Following the pre-intervention survey, staff educational sessions were held on the ward, delivered by the study senior research associate (SH). Reference copies of the patient-focused intervention materials (the video and health-care logbook) were provided to each ward.

Delivery of the patient-focused intervention components

The patient safety video (copied to a DVD) and the health-care logbook were mailed to elective patients 1 week before their planned admission (i.e. 1 week following receipt of the study questionnaire) on intervention wards. Information explaining their purpose and use was included and patients were informed of the availability of Talk Time. Patients were encouraged to read and complete the different sections of the logbook and to watch the DVD prior to admission, sharing this with relatives when possible. Patients arriving on participating wards (or their relatives) following acute or emergency admission were given the logbook by staff, along with the information about its purpose and use. Acute wards were provided with portable DVD players to allow patient viewing of the DVD.

Post-intervention survey

Patients: elective and acute admissions

Questionnaires were either given to patients at discharge or, when administrative support was available to wards, posted to patients following their discharge. A pre-paid envelope was included for (anonymous) return of questionnaires to the research team. No reminders were sent to patients.

Staff

Staff on intervention wards were invited to complete the post-intervention questionnaire at the end of the evaluation period for their ward; staff on control wards received the questionnaire at the end of the overall evaluation period. A pre-paid envelope was provided for return direct to the study team. Reminders were sent at weekly intervals for 2 weeks following the close of the evaluation period.

Post-intervention interviews

Purposive sampling was used to recruit subsamples of patient and staff participants with a broad range of characteristics (intervention wards only). Informed consent and interviews were undertaken by the study senior research associate (SH) and took around 20 minutes for staff and up to 1 hour for patients. All interviews were audio recorded and transcribed verbatim.

A semistructured interview schedule (see Appendices 9 and 13) was used that was informed by two frameworks for exploring implementation of health-care innovation: the TDF310 and normalisation process theory. 326 Within the domains proposed by these frameworks, interviews explored service user and staff perspectives, attitudes, beliefs and experiences in relation to:

• use of the intervention materials and participation in the overall intervention approach (service user behaviour; staff behaviour)

• understanding and knowledge about the approach and patient safety issues

• acceptability and feasibility of the approach

• perceived levels of engagement and involvement in identifying patient deterioration and alerting staff to concerns

• perceived impact on patient safety

• barriers to and facilitators of implementation

• unintended consequences (e.g. service user loss of trust in services; service user dissatisfaction with care; damage to staff morale, plans to leave; work burden).

Medication audit

Using a medications reconciliation process, the medication history of patients admitted to participating wards throughout the intervention period was reviewed at admission and discharge. Ward pharmacists coded for discrepancies related to two outcome measures (see Table 49) using a standardised data collection pro forma. No patient-identifiable information was extracted, although data were coded for ward, age group (e.g. 18–30 years, 31–50 years), mode of admission (elective/acute) and sex.

Results

The evaluation period was from 1 September 2012 to January 2013. The intervention ran on most wards for 4 weeks but was extended to 5 weeks on two wards to achieve the target sample of 460 patients (230 intervention and 230 control patients). Some post-intervention interviews with staff and patients were completed during January 2013 for these wards.

Participant flow

A CONSORT diagram (Figure 18) shows the flow of wards through the evaluation trial at each time point in relation to the primary outcomes. 273

FIGURE 18.

Consolidated Standards of Reporting Trials (CONSORT) diagram showing the flow of participants through the trial. A, acute; E, elective.