Notes
Article history
The research reported in this issue of the journal was funded by PGfAR as project number RP-PG-0608-10050. The contractual start date was in October 2010. The final report began editorial review in December 2015 and was accepted for publication in June 2016. As the funder, the PGfAR programme agreed the research questions and study designs in advance with the investigators. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The PGfAR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.
Declared competing interests of authors
Martin Roland and John Campbell have acted as advisors to Ipsos MORI, the Department of Health and, subsequently, NHS England on the development of the English GP Patient Survey. Jenni Burt has acted as an advisor to Ipsos MORI and NHS England on the GP Patient Survey. Pete Bower has received funding from the National Institute for Health Research in addition to the programme grant.
Permissions
Copyright statement
© Queen’s Printer and Controller of HMSO 2017. This work was produced by Burt et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.
Chapter 1 Introduction to the IMPROVE (improving patient experience in primary care) programme
Context
Improving the health status of individuals and populations is a central ambition of health-care systems in high-income countries, and the US Institute of Medicine has suggested that high-quality health-care delivery should be safe, effective, patient-centred, timely, efficient and equitable. 1 Berwick et al. 2 have noted the importance of patient experience of care as one of the suggested ‘triple aims’ of an advanced health-care system. A recent US report highlighted the important contribution that listening to, and acting on, patient feedback can potentially make to health-care improvement efforts. 3
New developments within the English NHS highlight the embedding of public performance assessment within the regulation of the health-care system, including NHS England’s consultation on the production of general practitioner (GP) league tables4 and the Care Quality Commission’s (CQC) parallel development of a rating system for primary care. 5 A transparent health-care system is regarded by policy-makers as essential to enable patients to make informed choices about the care that they receive6 and patient feedback on health-care services is now commonly gathered in the USA, Canada, Europe, Australia and China.
Efforts to improve quality of care in the NHS over the last 15 years have focused on providing prompt access to care (e.g. the time taken to see a GP or hospital waiting times) and on providing evidence-based clinical care [e.g. through the development of National Service Frameworks and the UK Quality and Outcomes Framework (QOF)7]. A direct link between patient feedback and quality improvement efforts was previously operationalised by including results arising from patient surveys as a component of the QOF. 7 This performance management system provides financial incentives for GPs within the NHS to achieve agreed quality indicators covering areas including chronic disease management, practice organisation and additional services offered. With the introduction of the QOF it was possible, for the first time, to rank all practices according to their patient feedback and the results of surveys, aggregated at practice level, formed the basis of a pay-for-performance scheme between 2009 and 2011, when the UK government withdrew the pay-for-performance arrangements for patient experience.
Some of these policies have been highly effective. For example, associated with a wide range of quality improvement initiatives over a decade, there have been greater improvements in the UK for the clinical care of conditions such as heart disease and diabetes than in any other major developed country. 8 Although relatively neglected in the early years of the millennium, patient experience of health care is now a high policy priority, and in 2008 the Next Stage Review suggested that:
. . . quality of care includes quality of caring. This means how personal care is – the compassion, dignity and respect with which patients are treated. It can only be improved by analysing and understanding patient’s satisfaction with their own experiences.
Department of Health, p. 47, emphasis in original. 9 © Crown copyright 2008, contains public sector information licensed under the Open Government Licence v3.0
The review9 noted, however, that ‘[up until 2008] progress has been patchy, particularly on patient experience’ (p. 48) and announced the development of quality accounts for all NHS organisations in which ‘healthcare providers will be required to publish data . . . looking at safety, patient experience, and outcome’ (p. 51) (© Crown copyright 2008, contains public sector information licensed under the Open Government Licence v3.0).
Since 2008, therefore, there has been a major policy initiative to improve patient experience in the NHS. Most recently, the focus on patient experience has been enshrined in the NHS Outcomes Framework, which, in Domain 4, focuses on ensuring that ‘patients have a positive experience of care’10 (contains public sector information licensed under the Open Government Licence v3.0). In primary care, these policy initiatives and statements have been implemented primarily through the development and conduct of the GP Patient Survey,11 first sent to 5.6 million patients in January 2009. The large sample size was intended to provide sufficient responses to characterise patient experience of primary care in all 8300 general practices in England. Detailed responses for individual practices were published on the NHS Choices website12 and made available online and included information on access to GP services and interpersonal aspects of care, out-of-hours care and care planning. The questionnaire specifically included validated questions about interpersonal aspects of care based on questionnaires that the authors of the present report previously designed and on which we have previously reported. 13 This large-scale survey is, of course, an expensive undertaking and its utility and impact need to be commensurable with this investment.
In seeking to achieve improvement in the quality of NHS services, gathering data is important both to inform the process of service development and innovation and to assess the impact of such changes in practice. It has been suggested that data to support such improvement initiatives need to be of sufficient quality to assess whether or not an innovation can be made to work, rather than being the more rigorous level of research data needed to assess whether or not an innovation works. 14
Communication in the consultation has always been an important part of primary care and is closely linked to continuity of care. At the outset of this research, there had been many anecdotal accounts that GPs were more focused on meeting clinical targets identified on their computer screens than on the needs of the patient sitting in front of them. It seemed therefore an appropriate time to balance the focus on improving clinical care with a renewed focus on interpersonal care and communication in the consultation. The ability of patients to choose their own doctor is also important. Our research prior to commencing this programme showed that continuity of care had deteriorated since the introduction of the new GP contract in 200415 and previous research had also highlighted that patients were less likely to report overall positive experiences if they were not able to choose a doctor whom they know. 16,17
Experience and satisfaction
Previous research has identified considerable confusion and overlap relating to the concepts of patient experience and satisfaction. The two concepts are closely linked, although at a simple level reports of experience relate to recounting or commenting on what actually happened during the course of a clinical encounter whereas reports of satisfaction focus on the patient’s or carer’s subjective evaluation of the encounter (i.e. asking for ‘ratings’ of care rather than simple ‘reports’ of care). Individual items in a survey may thus examine patients’ reports of their experience of care, whereas other items may explore patients’ evaluation of that care, with the linkage between report and evaluation/rating item pairs offering potential for the development of cut points in scales of performance. 18 In practice, however, the terms are often used interchangeably and survey items designed as report items often have an evaluative component; for example, the question ‘Were you involved as much as you wanted to be in decisions about your care and treatment?’ from the NHS Inpatient Survey19 contains elements of both. Within the GP Patient Survey,11 the instrument behind much of this programme of work, items often relate to ratings of care. For example, the communication questions ask patients to consider ‘how good’ the doctor was at providing various elements during a consultation, including giving enough time, involving in decisions about care and treating with care and concern.
Patient satisfaction may be seen as a multidimensional construct, focusing on the subjective experiences of patients, and related to their expectations of care and the perceived technical quality of the care provided. 20 Russell21 has recently summarised some of the problems associated with surveys of patient satisfaction with care, including problems with the validity and reliability of satisfaction survey instruments, the lack of a universal definition of the term ‘satisfaction’, the disinclination for patients to be critical of care received because of not wanting to jeopardise their treatment, satisfaction being determined by factors other than the actual health care received and the frequently non-specific nature of the findings arising from such surveys. In contrast, reports and surveys of patient experience may offer the potential to discriminate more effectively between practices than do reports of patient satisfaction,22 thus potentially offering greater external accountability of health-care providers, enhanced patient choice and the ability to improve the quality of care and measure the performance of the health-care system as a whole. 23
Patient experience matters
Patient experience is an important end point for NHS care in its own right. Patients consistently report that personal care is central to effective care and, in that context, the development and refinement of GPs’ interpersonal skills is a key priority. 23,24 It is noteworthy that many complaints regarding care centre not on technical and ‘clinical’ aspects of care, but on issues relating to interpersonal aspects of care and communication. 25,26 Good communication with patients is not just an end in its own right; it brings three important additional benefits.
First, our research27 has confirmed earlier work which showed that patients balance a range of beliefs and concerns when making decisions about taking medicines. Adherence is related both to the quality and duration of the consultation and to the doctor’s ability to elicit and respect the patient’s concerns. 28–30 Better communication may lead to improved patient outcomes31 through, for example, improved blood pressure control in hypertensive patients. 32
Second, there is a close relationship between poor communication and serious medical error. 33 This is partly because not listening to the patient’s perspective may lead doctors to miss important clinical information and partly because patients react more negatively when things go wrong if communication has been poor during the clinical episode in question. A significant proportion of cases referred to medical defence societies have at their heart poor communication in the consultation34 and improving communication with patients and engaging them more closely in their care is seen as key to improving patient safety. 35,36
Third, the increasing emphasis in the NHS on self-care and prevention demands good information and shared decision-making in the consultation. Our research shows that GPs and practice nurses are currently poorly prepared for roles in which they encourage patients to take greater responsibility for their own care37 or their lifestyle choices.
Although intuitively of importance, enhanced patient experience of care also matters on account of an important range of other associations reported in the research literature, including improved safety-related outcomes,38 improved self-reported health and well-being,39 enhanced recovery,31 increased uptake of preventative health interventions40,41 and reduced utilisation of health-care services including hospitalisation and emergency department visits. 42
Capturing patient experience of care
Although several approaches have been adopted to obtain information on patient experience of care – for example through the use of focus groups, patient participation groups, in-depth patient interviews, feedback booths placed in health-care settings and direct observation of patient experience43 and the use of compliment and complaint cards to capture qualitative feedback – the only practical approach to capturing large-scale feedback with the intent of providing actionable information remains the use of surveys of patients. In primary care in England, this culture of feedback has been embedded into routine practice in several ways. Central among these is the use of structured patient feedback obtained through surveys of patients’ experience of care, at both national and practice levels. 44
Qualitative approaches may be judged to offer greater depth of feedback than quantitative approaches,45 but such approaches are intensive in respect of data collection, although Locock et al. 46 have drawn on secondary analysis of a large national qualitative data archive to inform service improvements.
Newer forms of capturing feedback, such as the use of tablets and kiosks to capture real-time feedback (RTF), is an area of great current interest, but, as yet, these newer forms lack a strong evidence base from primary care. During the course of this research a report from a preliminary observational study47 suggested that RTF offers potential in primary care settings and similar findings48 have emerged from reports provided by patients with cancer attending oncology outpatient departments. Although there may be potential for the widespread use of real-time data capture of patient experience in primary care, the acceptability and feasibility of the approach in routine primary care is not known and nor is the nature of the feedback provided. Such an investigation needs detailed feasibility and pilot work using an experimental design of RTF of patient experience of primary care.
Large-scale surveys of NHS patients and staff have been in use since the mid-1990s, building on the experience of smaller-scale surveys conducted at local level or on the experience of surveys conducted for research purposes. Large-scale surveys of patient experiences of primary care were first introduced in 199849 with the express purpose of addressing issues relating to the quality of care and reducing inequalities in care by taking account of patients’ views in informing local service developments. Surveys of patients have been used extensively since the introduction of the UK QOF in 2004, when two questionnaires [General Practice Assessment Questionnaire (GPAQ)13 and CFEP50] were ‘approved’ for use by the NHS and adopted as the basis of linking the pay of GPs to their participation in the patient survey programme. 51
Such surveys may be administered in a variety of ways. In health-care contexts, paper-based surveys are most commonly used, although digital e-platforms are now commonly and widely used as a means of capturing information, most frequently using online processes. Computer-administered personal interviews52 and computer-administered telephone interviews may also be used, most commonly in research settings.
The NHS has established a major programme of surveys53 developed for a wide range of settings. Several of these surveys focus on patient experience of care, emulating the suite of Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys introduced in the USA in 1995. 54
The content of primary care surveys of patient experience
Historically, the content of UK primary care surveys has evolved from the 1998 survey,55 which covered a wide range of issues including primary care access and waiting times, GP–patient communication, patients’ views of GPs and practice nurses in terms of knowledge, courtesy and other personal aspects of care and the quality and range of services provided such as out-of-hours care and hospital referrals. The GP Patient Survey in 2008 developed and presented an expanded suite of items from the surveys of 2006 and 2007, which were focused almost exclusively on the accessibility of GP services; the 2008 survey focused on domains of care identified as being of importance to patients56,57 including the accessibility of care, technical care, interpersonal care, patient centredness, continuity, outcomes and hotel aspects of care. More recently, the English NHS has outlined eight domains believed to be of critical importance in respect of patient experience. Overlapping with earlier thinking, these include respect for patient-centred values, information, communication and education, emotional support, physical comfort, continuity and access to care,58 all being reflected, at least to some extent, in the ongoing GP Patient Survey programme. 11
Most recently, the Friends and Family Test has been introduced widely across the NHS, acting as a single-question proxy for patient experience based on the willingness of respondents to recommend their health-care provider to close acquaintances. The widespread use of the test has been accompanied by specific guidance on its implementation in practice59,60 and research reports have recently started to emerge following the use of the test in hospital settings, in which concerns have been raised about the reliability of the test. 61,62 The test was rolled out to general practice settings in December 2014.
Out-of-hours services
Beyond the domains mentioned in the previous section, additional areas of enquiry incorporated in the 2008 version of the GP Patient Survey included out-of-hours care and care planning. Variation in patients’ experiences of out-of-hours care has been identified as an area of concern since 2000, with numerous influential reports considering the structures suitable for delivering out-of-hours care, as well as highlighting the variable experience of patients across the UK in respect of service delivery. In 2000, Dr David Carson reported on the structural aspects of out-of-hours care pertaining at the time and recommended an expanded role for NHS Direct as a facilitator of access to these GP-led services, proposing that patients should use a single telephone access point to enter the system. 63 Much less emphasis was placed on patients’ experience of out-of-hours care, although recommendations were made regarding the need to monitor patients’ experience of the developing service. The transfer of responsibility for out-of-hours care from GPs to primary care trusts (PCTs) was foreshadowed in a report by the House of Commons Health Committee,64 which once again focused on structural and organisational issues relating to care provision. It was not until 2006,65 following the publication of national quality requirements in respect of out-of-hours care in October 2004,66 that patient experience of such services began to attract serious attention, with a recognition that, by 2006, although patient experience of out-of-hours services was generally ‘good’, one in five patients was dissatisfied with the service at that time. In addition, 40% of respondents in an independent survey of out-of-hours service users reported that the overall quality of the service was less than ‘good’. 65 The incorporation of six items in the 2008 GP Patient Survey with the intent of capturing information on aspects of out-of-hours GP services thus represented an extension of earlier versions of the questionnaire, recognising the growing importance of patient experience of care, and offered the potential to examine the experience of patients from various subgroups of the population and the potential to compare out-of-hours service providers in respect of their patients’ experience of care.
Measuring patient experience of care
The potential utility of questionnaires capturing patient feedback is, like other questionnaire-based feedback, dependent on the psychometric performance of the questionnaire in practice. Issues centring on the validity of the resulting data – whether or not the questionnaire items are measuring what is intended to be measured rather than some extraneous domain – underpin the reliability of inferences and conclusions that might be drawn following data collection. Validity and reliability themselves each consist of several elements and demonstrating validity of an assessment is generally regarded as having priority over demonstrating reliability.
Our previous research identified concerns expressed by doctors regarding the use of patient survey data for the purposes of providing individual feedback regarding a doctor’s performance. 67 Some of those concerns focused on the reliability and validity of the resulting data and on the conclusions being drawn regarding a doctor’s professional practice.
The validity of items within a questionnaire may be assessed in a number of ways, for example in exploring the pattern of item response using quantitative approaches such as factor analysis to investigate the latent variables identifiable within the theoretically related item responses. Qualitative approaches may also be used in the questionnaire design phase; for example, cognitive interviews were undertaken with patients from a range of sociodemographic backgrounds in the early stages of developing the GP Patient Survey. 68 Such interviews are designed to assess the interpretability and accessibility of the putative questionnaire items. Qualitative studies using cognitive interviewing or similar approaches may, however, be undertaken following respondents’ completion of questionnaires, seeking to explore the basis on which respondents are providing their evaluation. Such studies are unusual, but they potentially offer great value in exploring respondents’ insights and whether or not items presented are interpreted as originally intended.
Patients’ varying experiences of care
Our earlier research, and the research of others, has previously identified substantial variation between practices in patients’ reports of their experience of, and satisfaction with, care received and recent studies have also identified a range of patient experience being reported among doctors providing care in similar clinical specialties and settings. 69–73 This acted as the basis for the inclusion of patient feedback as an element required by UK regulatory authorities for the routine appraisal and revalidation of doctors. Despite these observations, few studies have examined the relationship between feedback on patient experience aggregated at practice level and the performance of individual doctors within practices, with one observational study74 identifying a substantial range of performance among doctors from a sample of eight Scottish general practices; the authors noted a number of possible contributing factors that might have accounted for differences observed at doctor level, including the experience of the doctors themselves, as well as the doctors’ mental health and professional disillusionment.
Systematic differences in patients’ reports of their experience of care have also been reported to be related to the characteristics of patients themselves. Older patients, patients from white ethnic backgrounds, the better educated, the less deprived and those reporting a better health status have generally reported more favourable experiences of care than younger patients, those from minority ethnic groups, the less well educated and more deprived and those with a poorer health status. Similar differences have been reported across many health-care systems and have given rise to calls to take account of the characteristics of participating patients when considering the results of patient feedback on care. To date, however, such calls have generally not been heeded in practice, as the relative contribution of practice, doctor and patient to overall variation in feedback remains to be defined. Uncertainty regarding the need for, and effect of, such ‘case-mix adjustment’ remains a concern for doctors in their consideration of patient feedback.
Specifically in respect of variation in experience among patients from different ethnic backgrounds, previous analyses have identified variations in patient experience in relation to ethnic group, age and gender and have found an interaction between ethnicity and age for cancer referrals. 75,76 However, no studies to date have yet investigated such an interaction in respect of patients’ experience of communicating with their GP, for example investigating differences between older and younger patients, by gender and among patients representing a range of minority ethnic groups.
In addition, although communication between doctors and patients is a core component of patient experience,77 and minority ethnic groups have reported lower patient experience scores for communication than the majority population,75,78,79 such differences are not consistent for all minority ethnic groups. Previous analysis of patient experience data conducted by the authors highlighted that South Asian patients reported particularly negative experiences, including for waiting times for GP appointments, time spent waiting in surgeries for consultations to start and continuity of care. 75 However, such analyses have not been repeated using GP Patient Survey data.
A number of potential explanations have been suggested for the lower ratings provided by South Asian and other minority ethnic groups in respect of their experience of care. Broadly, these relate to whether South Asian patients (1) receive lower-quality care or (2) receive the same care but rate this more negatively. 75 For example, differences in the use of questionnaire response scales might lead to South Asian groups being less likely to endorse the most positive options when asked to evaluate a doctor’s communication skills. 80 Alternatively, there could be systematic variations in evaluations of consultations because South Asian respondents vary in their expectations of, or preferences for, care. However, recent evidence from the USA points to lower quality of care as the main driver of variations. 81 Gaining understanding of why minority ethnic groups give relatively poor evaluations of their care is key to forming an effective response, as determining appropriate action is difficult until it is ascertained whether differences in evaluations relate to true differences in care or to variations in expectations, scale use and preferences. Exploring these observed differences between patients from various ethnic backgrounds is challenging using only observational, real-world data. More robust approaches are required, drawing on experimental designs in which some key elements of the consultation–interaction can be accounted for in the analysis, for example through the use of standardised consultations and video vignettes. 81,82
Using patient survey data to improve care
Although there is a belief, articulated in the Darzi Review,9 that patient surveys can be used to improve care, a systematic review from 200883 suggested that there is considerable uncertainty about how and whether or not this can actually be achieved. Several causal pathways for achieving improvements in provider performance through the release of publicly reported performance data have been proposed. 84–86 Some invoke market-like selection, claiming that patients will modify their choice of provider using publicly available data, such as that provided by patient experience websites. 84,87–89 Evidence to support this pathway is, however, weak. 85 A more likely mechanism driving performance improvement in response to the publication of performance data is health professionals’ concern for reputation, in which peer comparison motivates individuals and organisations to improve their care. 85,86
Furthermore, at the outset of this research, PCTs were poorly prepared to support and work with general practices to improve patient experience. In addition, the Darzi review9 had noted that progress in improving patient experience in the NHS had been slow, and our research had identified that some aspects of care, especially out-of-hours care90,91 and continuity of care,15 may actually have worsened in recent years. In addition, as observed earlier, it had been noted that patients from minority ethnic communities consistently reported lower evaluations of the quality of primary care. 75 Although these problems had been clearly identified in published research, the research had provided less clarity about the meaning and interpretation of these findings and the best way to intervene to improve patient experience.
Irrespective of its potential to stimulate change, the publication of performance data is central to the openness and transparency that are seen as essential for a safe, equitable, patient-centred health-care system. 92 Thus, regardless of any effect on quality improvement, such initiatives are likely to be here to stay. 85 In refining the information made public, it is important that performance data are accurate and relevant to all potential users. The US-based Robert Wood Johnson Foundation93 has noted that, whilst there is a patient preference for information to be provided at the level of individual clinicians (and not at practice level), such information is only rarely available. Currently, there is some move towards publication of performance data from an organisational level to that of individual doctors. In the UK, for example, patients referred to the cardiology service at the University Hospital of South Manchester NHS Foundation Trust may go online to view both mortality and patient experience data for each cardiologist or cardiac surgeon. 94 However, within English primary care, the practice-level aggregation of data from the GP Patient Survey used to derive practice performance indicators potentially masks variation in performance among individual GPs, thereby inappropriately advantaging or disadvantaging particular doctors. Current indicators may consequently fail to provide users, providers or commissioners with an accurate assessment of performance within a practice.
Although intuitively simple, patient satisfaction is a complex concept95 and patient questionnaire scores must be interpreted carefully. For example, practices need to understand if low ratings for communication reflect particular consultation behaviours or whether they are in fact the result of broader issues such as practice culture or the structure and availability of consultations and appointments.
Once the causes of low ratings have been better understood, interventions to improve care can then be designed. However, the current literature on the effects of feedback of patient assessments is insufficient in scope, quality and consistency to design effective interventions. 83 There are many reasons why simple feedback on patients’ experience of care is likely to have limited effects. Our research is designed to address these gaps in knowledge, enable managers, patients and professionals to have confidence in the meaning of patient assessments and provide effective interventions to improve care when problems are identified.
Summary
In summary, therefore, capturing patients’ experience of primary care is a current ambition of major importance in UK government health policy. Patient surveys, incorporating opportunities for people to comment on various aspects of their care, are a convenient means of capturing relevant information at scale. It is not clear, however, how health-care staff operating in practices respond to the resulting information. Previous experience suggests that staff may rationalise scores on the basis of concerns regarding the scientific properties of the survey, or uncertainty regarding the implications arising from providing care in their particular circumstances, for example taking account of the sociodemographic mix of respondents. On a similar vein, it remains unclear the extent to which overall practice performance, based on aggregated patient feedback, might relate to the performance of individual doctors within the practice. It is also unclear whether or not patients provide reliable evaluations of care – and the extent to which such evaluations might vary according to the sociodemographic characteristics of respondents. New modes of capturing patients’ experiences of their care have become available in recent years, but to date it is not clear whether or not novel, technology-based approaches can be successfully implemented in routine primary care settings, nor the extent to which any resulting data might reflect the results of the wider population.
In recent years, care provided by out-of-hours GP services has been a particular area of interest for the NHS and has been the subject of national audit and standard setting. However, it is not clear whether or not patients’ reports of their experience of out-of-hours care are valid and reliable. Neither is it clear the extent to which factors relating to the structure and organisation of such care might be associated with systematic differences in patients’ reports of their care. Furthermore, as for in-hours care, it is not clear how staff providing out-of-hours care might respond to patient feedback and how service managers might utilise such information in the planning and design of services aimed at being responsive to the needs of NHS patients.
Aims of the programme
This programme had seven aims:
-
to understand how general practices respond to low patient survey scores, testing a range of approaches that could be used to improve patients’ experience of care
-
to estimate the extent to which aggregation of scores to practice level in the national study masks differences between individual doctors
-
to investigate how patients’ ratings on questions in the GP Patient Survey relate to actual behaviour by GPs in consultations
-
to understand better patients’ responses to questions on communication and seeing a doctor of their choice
-
to understand the reasons why minority ethnic groups, especially South Asian respondents, give lower scores on patient surveys than the white British population
-
to carry out an exploratory randomised controlled trial (RCT) of an intervention to improve patient experience, using tools developed in earlier parts of the programme
-
to investigate how the results of the GP Patient Survey can be used to improve patients’ experience of out-of-hours care.
In presenting our work, we report our research and findings under three major themes: (1) understanding patient experience data, (2) understanding patient experience in minority ethnic groups and (3) using data on patient experience for quality improvement. These are outlined in brief in the following sections. The relationships between individual studies and the three themes are shown in Figure 1, which also outlines methods and participants.
FIGURE 1.
The IMPROVE (improving patient experience in primary care) programme themes and studies contained within workstreams: (a) understanding patient experience data; (b) understanding patient experience in minority ethnic groups; and (c) using data on patient experience for quality improvement. OOH, out of hours.
During the course of the programme we conducted empirical studies across a number of general practices and out-of-hours providers. General practices were initially recruited to take part in a suite of studies (presented in Chapters 7–9) in which we conducted a patient experience survey at the level of individual GPs, gave feedback from this survey to both the practice and the individual doctors (see Chapter 9) and, for some practices, conducted focus groups with practice staff and interviews with GPs. Sampling was initially designed around the survey study: practices were sampled on the basis of location (two study areas, the South West and North London/East of England, covering both urban and rural settings), performance on the GP Patient Survey, practice size and area-level deprivation. Once the survey was completed, a number of practices were purposively sampled to take part in focus groups with staff (see Chapter 7) and interviews with GPs (see Chapter 8) and additional filming of consultations (see Chapter 3). Out-of-hours providers were recruited from across England. We worked with up to 11 providers in varying workstreams (see Chapter 11). We additionally completed multiple analyses of GP Patient Survey data11 (see Chapters 4, 5 and 11) and, for an experimental vignette study, collected data from members of the general public (see Chapter 6).
1. Understanding patient experience data
In this theme we explored the meaning of data gathered through patient experience surveys by video recording (with consent) a large number of GP–patient consultations. Patients and GPs completed a questionnaire evaluating the quality of communication during the consultation and trained external raters (all GPs) also scored a small number of filmed consultations for quality. We additionally interviewed a sample of patients who consented to have their consultations filmed, reviewing their recorded consultation with them while talking through the options that they chose on the questionnaire about their experiences. This theme relates to aims 3 and 4. In addition, we conducted analyses of GP Patient Survey data to explore variations in patient experience in patients whose contact was with a nurse rather than a GP. This was additional to the original aims of the programme.
2. Understanding patient experience in minority ethnic groups
Here, we conducted a number of studies to explore why South Asian groups often have lower patient experience scores than white British patients in national surveys and provide more robust evidence of the drivers of this variation. These included a series of analyses of GP Patient Survey data and an experimental vignette study in which we showed simulated GP–patient consultations to white British and Pakistani respondents. This theme relates to aim 5.
3. Using data on patient experience for quality improvement
In trying to understand how patient experience data are currently used, and how they could be used, we carried out a wide range of studies. We completed a large-scale survey of patients at 25 general practices and carried out focus groups with practice staff and interviews with GPs. We conducted similar research in out-of-hours services. Finally, we looked at a different way of collecting patient experience data, using RTF kiosks in general practices. This theme relates to aims 1, 2, 6 and 7.
Patient and public involvement
Our programme of work was supported by two study advisory groups: one was based in Cambridge and provided support across all streams of work except for the out-of-hours research and the other was based in Exeter and was convened specifically to provide input to the out-of-hours workstreams. In this section we briefly outline the formation and working of these groups over the course of the programme.
Formation and composition of the main study advisory group
In the original application for the IMPROVE (improving patient experience in primary care) programme we set out our plans to establish an advisory group composed of 50% lay members and 50% professional members, to provide continuing advice and input throughout the course of the programme. We envisaged that this group would provide advice on the design of all strands of the work, assist with the production of study materials and work with us on the interpretation of data. At the start of the study we therefore set out to invite a mix of lay people registered with a general practice, GPs and practice managers to join the group.
We worked with the patient and public involvement (PPI) co-ordinator of the West Anglia Comprehensive Local Research Network to identify potential lay members with an interest in patient experience and primary care research. Potential patient representatives were provided with guidance outlining what was involved in advisory group membership and were informed that any costs incurred in preparing for or attending advisory group meetings would be reimbursed. Four lay members were recruited through this route. Additionally, we recruited one local GP to the advisory group from a practice with a large minority ethnic patient list. Despite a number of attempts to recruit an additional GP and two practice managers to the group, to ensure that we had input from all key stakeholders in the research, we were unable to do so. In spite of offering reimbursement to practices (e.g. we paid for a locum to enable our one GP member to attend advisory group meetings), GPs and practice managers were reluctant to commit to provide input into a research study over a number of years. We therefore proceeded with input from one GP only.
As a large focus of our work was on patient experience in minority ethnic communities, and particularly South Asian communities, we had additionally planned at the start of the study to recruit two additional lay members from a minority ethnic background to join the advisory group and provide specific advice on the development of our work in this area. In the event, this proved very difficult to achieve and we were unable to locate suitable representatives willing to sit on a formal group. We therefore considered alternative approaches to ensuring that we had input from these communities as we developed our study ideas and materials. As a result, we recruited a part-time researcher, Hena Wali Haque, and a senior advisor, Professor Cathy Lloyd, with specific expertise in and knowledge of research with minority ethnic groups. Hena liaised early on in our work with community groups representing Pakistani and Bengali communities and provided input on study materials and design. Although we would have preferred to have had such representation directly on our study steering group, through this route we were able to benefit from guidance on the most appropriate and effective approach to our research in this area.
We drew on guidance from INVOLVE to develop policies and documentation relating to the involvement of our advisory group members. 96 These included details of payment for particular activities, reimbursement, confidentiality, and data security. Group members completed a checklist to indicate what they were willing to assist with during the course of their involvement (e.g. reviewing different types of documents or attending meetings).
Formation and composition of the out-of-hours study advisory group
A stakeholder advisory group was convened specifically to provide guidance throughout the out-of-hours research. This consisted of three members from out-of-hours service providers, two academics with a particular expertise in this area and one lay representative. We had originally aimed to recruit two out-of-hours service users through local service providers, with assistance and guidance from local PPI groups; however, despite significant efforts, it proved difficult to recruit service users with relevant, lived experience. Our experiences were echoed by out-of-hours service providers, who noted that the relatively infrequent contacts that people made with out-of-hours services may in part drive the difficulties in recruiting service users to sit on advisory groups such as ours.
Activities of the main study advisory group
We set out to convene a face-to-face meeting of the main programme advisory group once a year throughout the course of the research. All meetings took place in Cambridge, with the first meeting taking place in October 2011 and the fifth and final meeting taking place in March 2015. At these meetings, group members reviewed and suggested changes to the study design, reviewed progress, discussed challenges and reflected on findings and interpretation. Particularly crucial input came, for example, in designing our approach to the recruitment of patients to our workstream involving the video recording of GP–patient consultations and in reflecting on the findings of our video elicitation interviews with patients. To keep group members up to date with progress and the research team, we sent out study newsletters on a roughly quarterly basis, with 13 being sent over the course of the programme.
Informal contact with group members by e-mail and letter continued throughout the rest of the year outside of the more structured meetings. One advisory group member, for example, was instrumental in organising a pilot focus group to reflect on study questionnaires. Additionally, all study materials aimed at patients or GPs (information sheets, consent forms and questionnaires) were reviewed and commented on by advisory group members and members were sent a summary of all findings and our conclusions to reflect on.
Activities of the out-of-hours study advisory group
Our out-of-hours advisory group, based in Exeter, had a more specific remit in guiding our research in this area. The group met initially to review study methods and procedures in light of the findings of the preliminary piloting and testing of methods (see Chapter 11, Workstream 2) and to comment on topic guides supporting interviewing in workstream 3. However, because of the logistical challenges of organising face-to-face meetings around staff availability, after an initial face-to-face meeting we communicated with the advisory group by e-mail and telephone.
Section A Understanding patient experience data
Chapter 2 How do patients respond to communication items on patient experience questionnaires? Video elicitation interviews with patients
Abstract
Background
Patient feedback instruments used in national survey programmes are robustly tested and evaluated, yet there remains a paucity of evidence on the drivers of a patient’s choice of response option. The objective of this study was to understand how patients’ responses to a questionnaire relate to their actual experience of a consultation with a GP, focusing on both implicit and explicit processes that respondents use to answer survey items.
Methods
We video recorded GP–patient consultations at 13 practices. Immediately following the consultation, patients were asked to complete a questionnaire about the GPs’ communication skills. We purposively approached a sample of these patients to take part in a video elicitation interview (n = 52), in which they were shown the video of their consultation and asked to reflect on their completion of the questionnaire.
Results
Although participants were able to raise concerns about doctors’ behaviours during the interview, they were reluctant to do so in their questionnaire responses. We identified three important drivers of this mismatch: (1) the patients’ relationship with the GP, (2) the patients’ expectations of the consultation and (3) perceived power asymmetries between patients and doctors.
Conclusions
Patients were inhibited in providing feedback to GPs through the use of questionnaires, with patients struggling to transform their experiences into a representative quantitative evaluation of GP performance. Our results suggest that patient surveys, as currently used, may be limited tools for enabling patients to feed back their views about consultations.
Introduction and rationale for the study
The overall purpose of patient surveys in primary care, such as the national GP Patient Survey,11 is to improve patient experience by feeding back patients’ evaluations to GPs and to the public. This process makes an important assumption, which is that the behaviours that doctors need to change are accurately assessed by responses given in patient experience questionnaires. For questionnaire items that relate to doctor–patient communication, the evidence that the items reflect doctors’ behaviour rests on their face validity and the cognitive testing that has already been carried out. Face validity is often taken as sufficient. However, further understanding of questionnaire completion is needed before helpful advice can be given to GPs. For example, if more is understood about the nature of poor consultations identified by patients, better support and advice can be provided to GPs to improve the quality of their consultations.
Previous studies have examined the process of patient questionnaire completion in specialist clinics. 97,98 These highlighted that patients may struggle to accurately represent their experiences of a consultation on standard survey instruments. Further, concerns have been raised about the perceived requirement for patients to assess health care from a ‘consumerist’ perspective. 99,100
To date, little is known about the ways in which questionnaire responses relate to patient experience within primary care and, specifically, patients’ perceptions of communication during GP consultations. The aim of this study was to understand, through the use of video elicitation interviews, how patients’ responses to a questionnaire relate to their experience of a consultation with a GP.
Changes to study methods from the original protocol
The aim of this workstream, as stated in the original protocol, was to understand better patients’ responses to questions on communication and seeing a doctor of their choice (aim 4).
In our application we set out plans to address this by conducting interviews with 40 patients, with 20 from a white British background and 20 from an Asian background. Interviews with minority ethnic participants were designed to contribute to our understanding of variations in patients’ experience of care in these groups, complementing our analyses of GP Patient Survey data and our experimental vignette study (see Chapters 5 and 6, respectively). We envisaged all interviews drawing on psychological approaches to cognitive interviewing, focusing on (1) comprehension of the question, (2) recall and assessment, (3) decision processes and (4) response processes.
We have expanded on our original design in several important ways. First, following our application, literature on the use of video elicitation interviews to stimulate recall and reflection on a medical encounter was published and to us appeared to be of direct utility for the aims of this study. 101 Video elicitation approaches, outlined in the methods section, use a series of detailed and specific prompts to enable participants to ‘relive, recall and reflect’ on their recent medical consultation. 101 We therefore adopted this approach in preference to that of cognitive interviewing.
Second, following discussions with practices, we were concerned that a ‘one size fits all’ approach to recruiting patients to the study from both white British and South Asian backgrounds was unlikely to be sufficiently sensitive and robust. We therefore made the decision to conduct the South Asian interviews as a stand-alone study, recruiting three additional practices with a particularly large proportion of South Asian patients on their lists and using dedicated researchers fluent in South Asian languages, together with appropriate study materials. This resulted in 23 interviews specifically with patients from a Pakistani background, conducted in the language of their choice. Our analyses of these interviews identified broadly similar concerns between our South Asian sample and the sample in the main study and we report these briefly in this chapter.
Finally, we expanded our original sample size of 20 interviews with white British patients to > 50, from a variety of backgrounds (but all were fluent in English). Video elicitation interviews are challenging to conduct well and we felt that it was important to enable the research team to build up sufficient confidence and expertise to generate rich data, as well as to reach a more diverse patient population. This chapter focuses in the main on interviews with the English-speaking population (n = 52).
Methods
This strand of work was conducted alongside the quantitative study outlined in Chapter 3. The recruitment of practices, GPs and patients was, thus, the same for both. The work outlined in this chapter focuses on subsequent interviews with patients who gave consent for their consultation to be video recorded. The IMPROVE study advisory group made important contributions to study design, particularly our approach to recruiting patients and the use of both a ‘brief’ and a ‘full’ study information sheet, and reflected on our analysis and findings.
Recruitment of general practices
The study was conducted in general practices in two broad geographical areas (Devon, Cornwall, Bristol and Somerset and Cambridgeshire, Bedford, Luton and North London). Practices were eligible if they (1) had more than one GP working a minimum of four sessions a week in direct clinical contact with patients and (2) had low scores on GP–patient communication items used in the national GP Patient Survey [defined as practices below the lower quartile for mean communication score in the 2009/10 survey, adjusted for patient case mix (age, gender, ethnicity, self-rated health and deprivation)102]. Low-scoring practices were chosen to maximise the chance of consultations within the practice being given low patient ratings for communication (nationally, 94% of patients score all questions addressing doctor communication during consultations as ‘good’ or ‘very good’ in the GP Patient Survey). Some, but not all, of these practices had previously participated in our individual GP-level patient experience survey (see Chapter 9 for details).
Recruitment of patients and recording of consultations
Video recording of GP–patient consultations took place for one or two GPs at a time within each participating practice. A member of the research team approached adult patients on their arrival in the practice to introduce the study. The patients were given a summary of the study as part of a brief information sheet, as well as a detailed full information sheet and a consent form. A member of the research team discussed these documents with each patient and sought consent to video record their consultation. Video cameras, set up in participating GPs’ consulting rooms, were controlled by the GP; physical examinations took place behind a screen and were thus not captured on camera. Data collection ceased when we reached our required number of video-recorded consultations that patients judged to be less than good for communication, as required for the quantitative analysis described in Chapter 3. All videos were stored on an encrypted secure server accessible only to members of the core research team. The recordings were made available to GPs for the purposes of continuing professional development. Immediately after the consultation, the patients were asked to complete a short questionnaire. This contained items relating to GP communication that were adapted from the GP Patient Survey (Table 1), alongside participant information including age, ethnicity and health status.
| Thinking about the consultation that took place today, how good was the doctor at each of the following? Please put a ✗ in one box for each row | Very good | Good | Neither good nor poor | Poor | Very poor | Doesn’t applya |
|---|---|---|---|---|---|---|
| Giving you enough time | □ | □ | □ | □ | □ | □ |
| Asking about your symptoms | □ | □ | □ | □ | □ | □ |
| Listening to you | □ | □ | □ | □ | □ | □ |
| Explaining tests and treatments | □ | □ | □ | □ | □ | □ |
| Involving you in decisions about your care | □ | □ | □ | □ | □ | □ |
| Treating you with care and concern | □ | □ | □ | □ | □ | □ |
| Taking your problems seriously | □ | □ | □ | □ | □ | □ |
Video elicitation interviews and analysis
The patient questionnaire contained a tick-box question asking patients if they were willing to participate in a face-to-face interview about their experience of the consultation. We subsequently contacted (by telephone or e-mail) those patients who expressed an interest in taking part. We aimed to interview at least one patient per participating GP. When more than one patient expressed an interest, we used a maximum variation sampling approach to reflect a mix of patient characteristics and questionnaire responses. Prior to the commencement of the study, we were particularly interested in interviewing patients who had given at least one response of ‘poor’ or ‘very poor’ in relation to a doctor’s communication skills.
We conducted video elicitation interviews with all participants (n = 52). In these interviews, participants were shown a recording of their consultation with the GP and asked specific questions relating to the consultation and their questionnaire responses (Box 1). The video elicitation technique is an established interview method that allows in-depth probing of experience during the interview by enabling participants to ‘relive, recall and reflect’ on their recent consultation. 101
Data generation focused particularly on participants’ recall of and reflection on the consultation and how this was expressed in their choice of responses on the questionnaire immediately post consultation. In each interview, the video of the consultation was used to encourage more accurate recall of specific events during the interaction. Our approach did not aim to establish the facts of what occurred, but rather explored the meaning to patients of actions that were performed in the consultation. The interview guide used was semistructured; however, we maintained a tight focus on specific moments and events captured in the recording.
Participants were asked some brief introductory questions about whether or not they had previously consulted with this doctor and whether the problem that they were consulting about was new or ongoing. Participants were then shown their consultation on the researcher’s laptop. They were encouraged to reflect as they watched the recording. Participants were also given their questionnaire responses and invited to talk through them. The recorded consultation was used as a prompt, enabling further in-depth discussion of their experiences in the consultation and their responses to the survey questions. Participants were also asked to identify behaviours in the consultation that they considered as contributing to their question responses and which could be changed to improve consulting performance.
The interviewers watched each consultation usually on at least two occasions before the interview and identified particular points at which they wanted to stop the recording or when they wanted to use prompts specific to the consultation content or to the respondent’s answers on the questionnaire. During the interview, the video recording of the consultation was shown to the participant, usually on two occasions. The participant was encouraged to stop the recording at any point to discuss a particular element of the consultation with the interviewer. The interviewer also stopped the recording as appropriate in response to a request from the participant, or to something said by the participant, or to her own prepared notes.
The analysis followed the principles outlined by Lofland et al. 103 These form a series of reflexive steps through which data are generated, coded and recoded, making particular use of memos to aid analytical thinking. Data analysis took place in two stages. The first stage occurred during data collection. A coding frame was devised from the topic guide, previous literature and early interviews. Each interviewer (JN, NL and AD) coded her own interviews in NVivo 10 software (QSR International, Warrington, UK). A number of analysis meetings were convened in which the interviewers and other members of the project team (JB, NE and JBe) discussed the data and themes. To ensure familiarisation with all of the data, the lead author (JN) listened to all interviews and read all of the transcripts. The coding frame was refined in response to discussions and as analysis progressed.
Ethics considerations
Approval for the study was obtained from the National Research Ethics Service (NRES) Committee East of England – Hertfordshire on 11 October 2011 (reference number 11/EE/0353).
Results
Participant recruitment
Consultations were videoed with 45 participating GPs from 13 general practices. During the period of data collection, a total of 908 patients had face-to-face consultations with participating doctors. Of these, 167 (18.4%) were ineligible (mostly children) and 529 completed a questionnaire (71.4% response rate) (Figure 2).
FIGURE 2.
Flow of patients through the video elicitation interview recruitment process.
Video elicitation interviews
A sample of patients whose consultation was video recorded participated in a video-elicitation interview. In total, interviews were conducted with 52 patients (35 women and 17 men) who had consulted with 34 different doctors across 12 GP surgeries in rural, urban and inner-city areas in the South West and East of England.
The interviews took place between August 2012 and July 2014, and were conducted in a location chosen by the participant within a maximum of 4 weeks of the recorded consultation (Table 2). Researchers preferred that interviews were not conducted at the GP surgery in case it inhibited patients in their narrative. However, a few participants specifically requested that their interviews be held at the GP surgery.
| Location | Number of interviews |
|---|---|
| Participant’s home | 44 |
| GP surgery | 6 |
| Other location (chosen by participant) | 2 |
All of the interviews were conducted in English and lasted between 26 and 97 minutes (average 58 minutes). The participants were aged between 19 and 96 years, with 22 (42%) aged > 64 years and 30 (58%) aged between 19 and 64 years. The participants consulted for a range of conditions, some chronic and some minor. The names used in the following sections are not respondents’ real names.
Questionnaire completion
In interviews participants were well disposed towards the process of questionnaire completion and generally keen to contribute their views. Most participants described completing the questionnaire with relative ease and as a simple task. Despite this willingness to contribute, there was little variety in questionnaire responses: the majority of participants reported care to be ‘good’ or ‘very good’ across all seven communication items on the questionnaire. Indeed, no respondents in our interview sample chose to score ‘poor’ or ‘very poor’, despite our original aspiration to focus in particular on patients who expressed dissatisfaction with their care. Twelve respondents did, however, use the ‘neither good nor poor’ option in at least one domain, although five of these also scored ‘very good’ on at least one other domain. As a result, in our small sample we had a lower proportion of scores in which every domain of GP communication was judged to be ‘good’ or ‘very good’ than in the national GP Patient Survey sample (77% in our sample vs. 94% nationally). Thus, despite the lack of ‘poor’ responses, we were able to explore patients’ responses in those who had expressed more dissatisfaction than average.
Disconnect between the ‘tick and the talk’
Although scores on the questionnaire were largely positive, some narratives in the interviews were more critical of aspects of GP communication. We outline three types of narrative relating to the relationship between questionnaire responses and further reflection on the consultation experience expressed during the interview.
Rewatching the consultation endorsed positive questionnaire scores
For some participants, their reflection on the consultation during the video-elicitation interview led to a repeated endorsement of the questionnaire responses they had given, and thus their narrative account was consistent with their previous evaluation of care. In all cases these responses were positive. Participants had been pleased with the quality of the consultation at the time of completion of the questionnaire. On rewatching the consultation this view was endorsed and in some cases further strengthened. Some respondents pointed to elements in the videoed consultation that had impressed them:
. . . his [GP’s] movements, his mannerisms . . . I’m asking the question, he didn’t exactly ignore me, he says no, that’s for gout. He actually explained it . . . And he’s still doing some work . . . So he’s not stopped and put all his attention on me, because if you stop doing that you probably forget what you’re doing here, so he’s done both. He’s answered my question and he’s also continued working, and that’s a good thing for me.
Colin (53151034)
High quantitative scores were followed by some criticism in interviews
Some participants scored the consultation highly on the questionnaire, yet the subsequent interview was peppered with tones of criticism about aspects of the consultation.
Criticism in the interview was often subtle, with participants often seemingly unaware of the discrepancy between their narrative and their questionnaire responses. Although they spoke of their consultation in a tone that was not particularly positive, participants remained loyal to the positive scoring they had applied on the questionnaire immediately following the consultation:
I gave it ‘good’ because . . . well she was listening to me, but I guess most of the time she was the one talking rather than listened to what I was saying . . . Not in a negative way, like completely, but I feel she didn’t really give me proper time to properly explain myself a little more . . . giving me a little bit more time, to explain my symptoms.
Steven (60121017)
Participant reappraises the consultation during the interview
A small group of participants who had scored their GP highly on the questionnaire underwent a process of reappraisal of the consultation during the video-elicitation interview. They voiced criticism of the doctor’s behaviour and proceeded to review their original score. Through the process of rewatching, participants spontaneously identified more negative aspects of the consultation that they had not been aware of previously:
I suppose you’re proving to me that I marked that wrong [taps questionnaire] [laugh] . . . Yeah, but he [GP] did, he did, he was concentrating on my leg and not worrying about the fact that the tablets were upsetting me.
Mm. And how did you feel?
Well, I felt the same thing. He, sort of, ignored the fact that he’d got all these side effects and all that.
Emma had scored elements of her consultation as ‘very good’ on the questionnaire:
. . . now I’m thinking, well no, he didn’t really sort of ask about symptoms or think, y’know, so perhaps not so good. Listening – yeah he listened but didn’t pick up on things, like you say, like the cough, he didn’t sort of pick up on erm, little things.
Emma (27131004)
On occasion there was a dramatic shift in point of view when the consultation was reshown. During the rewatching of the interview, Martha began to critique more aspects of the consultation, such as the doctor’s lack of explanations and unexpected examination:
I remember him just like, because he, because it’s quite rushed . . . you, er, can’t, you don’t, I don’t know, you’re just, it’s just like, er, er, and then, fine, I don’t know, I suppose I remember thinking why is he taking my temperature, and then just seeing how it must be OK, erm, I, I definitely remember him when he was just doing that with my, feeling my neck [slight pause] wondering what he was doing. [laughs] I just remember thinking, this is a bit weird, like why is this connected to my ear.
Martha (62111010)
In a number of the interviews, therefore, there was a mismatch between the subsequent account and previous responses to questions. At times participants were happy to critique an experience during the interview, sometimes at great length, yet they had been reluctant to do so on the questionnaire. Participants were able to explain in great detail elements of the consultation that they experienced to be negative, yet when asked to complete the questionnaire on that basis they still scored the doctor as ‘good’. The use of the video-elicitation method identified the possibility that other factors fed into the choice of response options on the questionnaire, aside from the doctor’s behaviour in the consultation.
There was, therefore, on a number of occasions, a disconnect between the ‘tick’ and the ‘talk’: differences between the narrative given in the interview and the responses recorded previously in the questionnaires. Although participants were able to raise concerns about doctors’ behaviours during the interview, at times they appeared reluctant to do so in their questionnaire responses.
Factors that influence patients’ reluctance to criticise on the questionnaire
This reluctance to record negative responses on the questionnaire leads to the question of why patients were reluctant to do so, given the negative views often apparent in their narratives. We therefore sought to further understand this phenomenon. We identified three key factors that appeared to influence patients’ reluctance to criticise doctors’ communication skills within the questionnaire:
-
the patients’ relationship with their GP
-
the patients’ expectations of the consultation
-
perceived power asymmetries between patients and doctors.
The following sections will examine each of these explanations in turn.
The patients’ relationship with their general practitioner
Participants often spoke about the significance of the GP or the surgery in their lives. This affiliation was sometimes with the practice, even if the doctors had changed over the years. Some elderly patients interviewed had been with their surgery most of their lives and a number of participants expressed loyalty to a practice even if they did not often visit. Gratitude towards the wider health service and NHS provision was also commonly expressed.
In commenting on relationships with individual GPs, care given previously to participants or to their family and friends was often praised:
. . . but I mean I’ve known him for – I mean he actually phoned when my mum died, you know. So that was nice of him, you know.
Janice (67131043)
Some participants spoke particularly of their relationship with the doctor. In some cases this was the notion of ‘getting on with’ the doctor and liking him or her as a person. For some there were specific interests that were shared, such as an interest in sport or knowledge of the GP’s family:
Oh yes, yes [laughter] they go to my church as well you see, so in that sense the relationship I had with Fiona and Paul is very much the old-fashioned family doctor, where you know them.
Alan (19144016)
Some participants used the term ‘friend’ to describe their relationship with their GP, often going on to explain that the relationship was different from a friendship:
And as I said, y’know, he isn’t a friend but you feel as if you are seeing a friend.
Bob (25111005)
I can see now the relationship. I have to be careful, you know, when I said to somebody one day well, you and I have a very good relationship, and I thought oh no, that’s not the right word.
Janet (53181024)
This was the case for participants who had not previously seen the doctor they consulted with, as well as for those who had a long relationship with their doctor. However, some found it difficult to create a relationship with the GP, which could lead to challenges in building a rapport.
Most respondents articulated that they were responding to the questionnaire based on the recorded consultation at hand. However, in explaining their scores, participants would often reflect on previous consultations with the GP. Consequently, it appeared challenging for them, when evaluating communication skills, to differentiate between this particular consultation and their wider relationship with the GP. This loyalty to and closeness with the GP at times inhibited patients from giving negative survey responses.
The patients’ expectations of the consultation
A variety of patient expectations influenced the scoring process. First, we identified expectations that related to the communication skills of the GP, based either on previous experience of consulting with that same GP or on experiences of consultations with other GPs. Expectations were important, and participants used this relational knowledge of other GPs to compare care received in the consultation with experience of previous consultations as they reflected during the interview. Some participants compared the care they had received with high-quality care they had received from other doctors. More commonly, however, participants compared a GP’s behaviour with poorer care that they had received. In particular, patients appeared to benchmark GPs’ skills based on their experiences with other GPs:
And he’s [GP], he’s not as clipped as Dr Williams, but he can be sometimes a bit clipped in the way he speaks to you.
Dave (67131012)
Narratives covered both positive and negative expectations of care from a particular GP, so, for example, if the participant had high expectations and these were not met, he or she was disappointed. Conversely, if a participant had poor expectations of a GP but the consultation was better than he or she had expected, he or she might score the GP more positively, even if overall the experience of the consultation was poor.
Second, expectations relating to the outcome of consultations were often referred to in justifying questionnaire scores. Participants tended to rate consultations more positively on the questionnaire if the outcome was what they desired, for example if they wanted a particular type of medication, a referral or reassurance about their medical problem:
I worry that, like, yeah, that he’s just going to be really dismissive. So the fact that he gave me medicine meant that it was higher than I expect . . . like it was better than I expected it to be, em, but perhaps by the more standards it wasn’t amazing.
Martha (62111010)
Perceived power asymmetries between patients and doctors
Descriptions of power asymmetries were prevalent in many of the accounts. Participants were often reluctant to criticise their GP, fearing repercussions for him or her. One respondent, who shared a story of poor experience as a patient in her general practice, had scored the GP as ‘neither good nor poor’ on some elements. When asked why she had not used the ‘poor’ or ‘very poor’ options on the questionnaire, she replied:
. . . you don’t want to get anybody into trouble, you know, but you do wish they did behave a little better, you know, treated you a little better, you know, in their response to you.
Esther (53131010)
At times participants expressed an associated dependency on the GP, with a corresponding view that they could not be critical for fear of compromising the relationship.
Participants often spoke of the trust they placed in the doctor. For some, the doctor had a status that they were in awe of. Although Amanda felt that this relationship had changed over time, the doctor was still revered:
And I think the gap is not as wide as it used to be between doctors and patients is it? I mean when I was a young girl the doctor was even more of a god, whereas now it’s less, it’s definitely getting lesser, yeah definitely.
And in terms of the GPs at the surgery, I know you mentioned about GPs being on a pedestal, how do you feel the GPs are there?
Oh I think they all are, yeah definitely yeah. But years ago perhaps it was a six-foot pedestal whereas now it’s probably a couple of foot [laughs].
In some ways participants had an inability to critique the doctor, or at least had a reluctance to do so. For example, some participants seemed to feel that they lacked the authority to judge the doctor’s communication skills.
Allowances were often made by participants for elements of the GP’s behaviour that they did not like, such as the GP looking at the screen a lot during the consultation:
She [GP] was reading so and I mean there’s an awful lot on there [laugh] there’s loads on that screen, bless her, so she’s probably thinking oh my God, how many [laugh] but no I don’t take a lot of notice to be honest.
Sue (24155004)
Respondents often commented on how busy the doctor was that day or how much he or she had to do. On a number of occasions participants were dismissive of other patients and the unreasonable requests they made of GPs.
During interviews, participants could be critical of their own behaviour, taking responsibility for their own poor communication during the consultation:
What makes you say that you weren’t a very good patient?
Because I was spending too much time . . . I wasn’t giving out information as clearly as I should do, and, you know, I had gone in with an agenda.
Additional interviews with South Asian respondents
Alongside the interviews conducted as part of the main study, we set out to recruit participants from a South Asian background to explore in particular their experiences of GP care and the factors that influenced their choice of response options on the patient experience questionnaire. We followed exactly the same procedures as for our main study sample but worked in three practices (in Bedford, Peterborough and Luton) that had a high proportion of Pakistani patients on their practice list. Our researchers for this workstream were fluent in Urdu and Punjabi, and all study documents were available in Urdu. We followed standard procedures for forward and backward translation of these documents.
We conducted 23 video elicitation interviews with respondents who self-identified as Pakistani (18 male and five female aged between 18 and 74 years). Study transcripts were analysed separately to consider the determinants of patient experience and questionnaire response tendencies in this patient group, following the same approach as outlined earlier for the main study sample. Only three participants chose ‘neither good nor poor’ as at least one response option for the communication items, with none choosing ‘poor’ or ‘very poor’.
In line with our main study sample, respondents were broadly positive about their GP’s communication skills when asked directly about these. Respondents were able to identify a number of approaches used by GPs that they rated positively, for example in explaining tests and treatments:
Yes, so I liked that [referring to the recording], the way she showed me on the . . . err . . . she had like a diagram of a body, she even pointed to . . . like, like . . . the nerve and where I’ve got a spasm. So I like a bit of that. So that explains to me more of the situation . . . So yeah, that part was quite good.
Tahir (66 18 5090)
And in relation to being given enough time:
Obviously they’re very short of time, and he was obviously still getting prepared to see me, when I came in. So when I started the conversation he was still looking at the screen, but he immediately, once he got through that piece of work, he immediately established eye contact, which again helps, certainly, to put me at ease, and know that somebody’s listening, responding and understanding. So that was good.
Sajid (65 13 5113)
However, within interview narratives it became evident that they drew on a number of factors external to the immediate encounter with the GP in evaluating communication competence. In spite of differences in cultural background, we identified the same issues driving evaluations of communication. First, respondents often drew on their relationship with their GP and others within the practice in making evaluations of care, rather than the specific events in a consultation. Many participants expressed high regard for their current GP and often compared them with GPs from their past to explain why they considered them to be so good:
I’ve been to other surgeries as well . . . and they’re not really interested, they just want to get you in and out, but this practice itself, the doctor listens to you, gives you a lot of time, yeah.
Imran (65 13 5110)
Second, expectations of the consultation could influence assessments of care. For some Pakistani respondents, experiences of health care abroad and a sense that the NHS provided high-quality care meant that, regardless of a consultation experience, they were grateful for and positive about encounters:
In Pakistan, if you have money, it is OK otherwise you are on the road. There are very competent doctors in Pakistan, but you need a lot of money for them. Everyone can’t afford that – some can and some can’t . . . It means a lot to me. Big deal for me! Fine. Allah forbid, if I have to go to private, I can’t do it, can I? I can’t afford it. So we are on NHS account.
Mohammed (65 13 5085)
Finally, perceived power asymmetries between patients and doctors were often prominent: as one respondent clearly articulated, a core theme in Pakistani patients’ narratives of experience was that doctors were perceived as having a sacred position in society:
The profession of a doctor is holy. Life and death is commanded by Allah. He saves, but the doctor is the best means.
Mohammed (65 13 5085)
Limited English proficiency did, however, have the potential to compound these issues; for example, participants could additionally blame their own difficulties in communicating as contributing to challenges during the consultation:
Our issue is the language, all doctors are good. Why would we question them? They have studied to be good, so for me the language creates hurdles.
Anam (65 13 5061)
Discussion
Although participants commonly showed reluctance to criticise GPs in their survey responses, our video-elicitation interviews opened up more nuanced discussions, in which patients voiced a number of criticisms about the GP’s communication skills. Previous studies have identified a reluctance to critique doctors in users of mental health services98 and patients undergoing elective orthopaedic surgery. 97 Our findings confirm that such reluctance persists for some patients in general practice, a setting in which an ongoing relationship between doctor and patient is more commonplace than in the secondary health-care setting.
Medical encounters have long been characterised by an asymmetrical power balance, despite attempts over the last 15 years of government policy104 to embrace a more ‘patient-centred’ approach to health-care provision. Goodyear-Smith and Buetow105 have urged that, although seeking to empower patients, we must be sure not to disempower doctors; they note that power can be beneficial in the consultation. Others have also argued that asymmetry is essential to the success of the medical encounter. Pilnick and Dingwell106 distinguish between functional and dysfunctional asymmetry in this role, arguing that the former may prove useful in shaping the medical encounter. Within our findings the notion of power asymmetry was evident in many narratives, and this was not necessarily viewed negatively by participants. In fact, patients’ accounts often displayed a respect and at times even reverence towards GPs and the work that they do. However, this relationship may make it difficult for patients to be critical when giving feedback to doctors and they therefore may need more encouragement or permission to report negative aspects of the consultation in questionnaire responses.
The role of expectation was important in our study. Previous work has suggested that non-fulfilment of expectations of care, such as examinations, tests and referrals, can be associated with lower patient satisfaction. 107,108 Our work suggests that reported experience may be influenced by the meeting of overall expectations, even if the overall standard of the communication was not seen favourably by participants.
Throughout our interviews, we identified a number of ways in which patients may be inhibited in choosing negative responses on experience questionnaires. Lupton,100 in her examination of the concept of consumerism in health care, asserts that participants can hold both a ‘consumerist’ and a ‘passive patient’ position simultaneously or variously in interactions with doctors. In this study, patients appeared to struggle to inhibit the purely consumerist approach to health care, as Lupton et al. 99 found in the earlier study with patients in Australian general practices. Health may hold vested emotional significance in patients’ lives, making it more challenging for them to provide ‘objective’ assessments of care so common in other areas of consumer experience. Coyle’s109 work has also highlighted the personal nature of health-care experiences and the threat to personal identity experienced when problems with health-care provision occurred.
Our study identified a number of contextual factors that impact on patients’ choice of response to a questionnaire concerning their experience, including power dynamics, expectations of care, ongoing relationships and previous experience. For some patients, this translated into an inhibition to provide negative evaluations of care on a questionnaire, despite being able during interviews to identify a number of concerns about the quality of communication that they experienced. Questionnaires, although an important tool for gathering patient feedback, may be limited in the information provided in their absolute scores. Our quantitative evaluation of patient assessments of care, reported in Chapter 3, provides more details on how questionnaires may best be used for quality assurance and improvement initiatives. We note that GPs’ professional development may benefit from other methods of feedback in addition to patient questionnaires, such as recording and reflecting on their own consultations or having peers watch and discuss their consultations.
Strengths and limitations
Our use of video-elicitation methods enabled us to probe in detail the link between a patient’s responses on a questionnaire and his or her experience of the consultation. A number of patients attending participating practices declined to have their consultation recorded. It may be that these patients had particular conditions that made them more conscious of their privacy, such as gynaecological or mental health issues. We acknowledge that, as GPs and patients knew that the consultations were being video recorded, this may have altered behaviours. However, as the camera was in the room for most of the day a number of GPs commented on how its presence became normalised during the session. GPs were able to opt in or out of the study and it may be that doctors who were less confident in their communication skills declined to participate.
The use of the video-elicitation method and the ability of patients to experience the consultation through rewatching it after the event created a unique environment. Inevitably, the method prompts patients to reflect on a consultation in a novel way. The temporal element of the experience in the consulting room was emphasised and the re-experiencing sometimes led to an altered view of the consultation. For example, the critical self-reflection seen in the data may in part be an artefact of the method in which participants viewed themselves in the consultation during the interview. We also note that the time delay between the consultation and the interview may mean that any number of events (e.g. a worsening of their condition) may lead to a re-evaluation of the nature of the consultation and a more negative critique, particularly as the patients were (in most cases) further removed from the general practice and the consultation.
Although researchers gently prompted participants regarding their responses to the questionnaire, it may be that some felt the need to give an account of their responses in a socially acceptable way rather than their actual thoughts at the time of interview. For example, they may have preferred to present a rationalised explanation for their responses rather than admit that they rushed the questionnaire and did not give consideration when completing it. For interviews with Urdu-speaking patients, we used materials translated into Urdu; we did not, in this qualitative work, consider the cultural equivalency of the translated instrument using consensus meetings and there may be unidentified issues in understanding as a result. However, our bilingual interviewer had the opportunity to draw on shared understanding of concepts during interviews, albeit in an ad hoc manner.
Conclusions
Our findings suggest that patients may, on occasion, be inhibited in providing feedback to GPs through a questionnaire. The factors that we identified may account for some of the tendency of patients to score consultations highly on questionnaires, with issues including previous experiences, ongoing relationships and perceived power asymmetries contributing to evaluations of communication skills. Our results suggest that patient surveys, as currently used, may be limited tools for enabling patients to feed back their views about consultations. Doctors whose communication skills are rated ‘very good’ on a patient questionnaire are likely to conclude that no change in their consultation style is required; however, this work suggests that even a rating of ‘very good’ may in fact mask patient reservations about the quality of the encounter.
Chapter 3 The association between patients’, raters’ and general practitioners’ assessments of communication in a consultation
Abstract
Background
Although patient feedback is widely used with the aspiration of quality improvement, the association between patients’, external observers’ and GPs’ own evaluations of communication performance during a consultation remains little explored.
Methods
We video recorded 529 consultations with 45 GPs in 13 practices. Following each consultation, the patient rated the GP’s communication skills and the GP did likewise. Subsequently, 56 consultations were sampled to include a range of patient scores for communication. Each video was rated by four trained clinical raters using the Global Consultation Rating Scale (GCRS). The ratings of patients, raters and GPs were compared.
Results
There was a modest positive correlation between patient ratings and those made by trained raters (ρ = 0.29, increasing to 0.33 after accounting for measurement error/reliability; p = 0.054). Consultations scored highly for communication by trained raters were also scored highly by patients. However, when trained raters judged communication to be of lower quality, mean patient scores ranged from ‘poor’ to ‘very good’. There was no evidence that GP scores were associated with the scores of trained raters (p = 0.721) or with the scores of patients (p = 0.854).
Conclusions
Compared with patients, trained raters tended to give more negative scores for communication during consultations. This is consistent with the finding from the patient interviews that patients find it difficult to criticise GPs when completing questionnaires. Patient surveys are a useful tool for measuring relative performance of doctors’ communication skills, but absolute scores should be interpreted with caution. Our results also cast doubt on how useful doctors’ assessment of their own performance is when used as part of reflective practice.
Introduction and rationale for the study
A clear aspiration of the national GP Patient Survey programme is to facilitate changes in overall experience of care by feeding back patients’ evaluations both to GPs and to the wider public. Confidence in the instruments used to assess – and potentially rank – performance is therefore essential if they are to make a meaningful contribution to quality assurance and improvement. 110 There has been extensive work on the reliability and validity of patient experience questionnaires. 111–116 However, although the face validity of communication items in questionnaires such as the GP Patient Survey has been well studied, evidence is sparser on whether or not the scores have construct validity, that is, whether or not behaviours that doctors may need to change are accurately represented by responses given in the questionnaires. For example, do patients reflect specifically on their experience of communication with the GP in their choice of response options or are they drawing on wider influences, which may be internal or external to the consultation? And how do patients’ concepts of ‘good’ communication relate to professionally agreed norms of ‘good’ communication?
One approach to investigating the construct validity of items is to compare patient evaluations of consultation behaviours with those of external observers. Previous research has explored the relationship between patient and examiner ratings of trainee GP communication skills and has found either no evidence of an association (in an underpowered study, with a sample size of 19)117 or a weak to moderate association. 118 This workstream aimed to provide more robust evidence of the association between patient assessments of communication skills using items from national survey programmes and observer assessment of the performance of practising GPs.
An additional area of concern for quality improvement efforts is that, despite the extensive psychometric testing of patient experience instruments, research shows that doctors often struggle to make sense of, and act on, feedback from patient surveys119 (see Chapter 8 for our work on this). A possible contributory factor in this may be incongruence between self (doctor) and patient assessments of performance. Evidence suggests that doctors tend, in fact, to rate themselves more negatively than patients or peers. 120,121 Indeed, there is a substantial body of evidence showing that doctors’ perceptions of their own competence are frequently out of kilter with external assessments of the same. 122–124 Of particular concern, however, is that the highest levels of incongruence are found in doctors who are, by external evaluation, the least skilled but the most confident in their abilities. 122,125
Previous research has tended to focus on the associations between doctor and other assessments at the level of overall performance, rather than performance at the level of a particular consultation. To understand in more detail where discrepancies arise between doctor and patient assessments of care, this workstream also considered how GPs’ and patients’ assessments of communication compared at the level of the individual consultation.
Changes to study methods from the original protocol
The aim of this workstream, as stated in the original protocol, was to investigate how patients’ ratings on questions in the GP Patient Survey relate to actual behaviour by GPs in consultations (aim 3).
Our application envisaged this workstream taking place as part of our wider patient experience survey, with participants being drawn from patients attending the lowest-ranking 15 (out of 25) GP practices (programme aim 2, reported in Chapter 9). This would have entailed asking patients for consent to film their consultation as well as consent to participate in the exit survey, planned to take place face-to-face. However, with the change in survey mode from face-to-face to postal (see Chapter 9 for details), we made the decision to separate this study entirely from the larger-scale survey. We thus recruited a sample of low-scoring practices specifically to participate in the filming of consultations, making this a completely stand-alone piece of work.
As this became a fully separate study, we were able to additionally ask GPs to rate, after each video-recorded consultation, their communication performance. This enabled us to undertake additional, originally unplanned analyses on how GPs’ perceptions of their own performance relate to those of the patient or external raters.
Furthermore, in our original application we planned to ask external raters to use the GP Patient Survey communication items to evaluate the quality of communication during a consultation. However, we decided to take a more robust approach to assessment and thus developed our own instrument to assess communication quality (the GCRS126), based on a widely used international approach to communicating within a consultation, the Calgary–Cambridge guide to the medical interview. 127,128
Finally, our original application had a focus on identifying specific behaviours that may have been associated with patient-reported communication scores. In particular, we were interested in identifying which dimensions were of most importance to patients and thus those that GPs might want to change to improve patient experience. However, the study design was powered to detect an overall association between patient and other ratings. The consequence of this is that we were underpowered to differentiate between different doctor behaviours and this was confirmed in the initial analysis. Given this realisation, we have chosen not to present that analysis and to concentrate instead on the overall associations.
Methods
This study took part alongside the video elicitation interviews described in Chapter 2. Briefly, we obtained consent from patients and GPs to video record face-to-face consultations in participating practices. Full details of our approach to sampling, recruitment and recording of consultations are provided in Chapter 2. As already stated, immediately following the consultation, patients were asked to complete a short questionnaire including a set of seven items taken from the national GP Patient Survey11 to assess GP–patient communication (Table 3) and basic sociodemographic questions. At the same time, GPs answered the same questions about their own performance. We calculated two GP–patient communication scores, one from the patient responses and one from the GP responses. In line with previous work, we calculated communication scores by linearly rescaling responses between 0 and 100 and taking the mean of all responses when four or more informative answers were given. 129–131
| Thinking about the consultation that took place today, how good was the doctor at each of the following? Please put a ✗ in one box for each row | Very good | Good | Neither good nor poor | Poor | Very poor | Doesn’t applya |
|---|---|---|---|---|---|---|
| Giving you enough time | □ | □ | □ | □ | □ | □ |
| Asking about your symptoms | □ | □ | □ | □ | □ | □ |
| Listening to you | □ | □ | □ | □ | □ | □ |
| Explaining tests and treatments | □ | □ | □ | □ | □ | □ |
| Involving you in decisions about your care | □ | □ | □ | □ | □ | □ |
| Treating you with care and concern | □ | □ | □ | □ | □ | □ |
| Taking your problems seriously | □ | □ | □ | □ | □ | □ |
Ratings by trained external raters
Of the video-recorded consultations for which the patient had completed the communication items on the questionnaire, we sampled 56 for rating by experienced trained clinical raters. Raters scored each of the selected consultations using the GCRS. 126 We designed the GCRS to assess the effectiveness of communication across an entire GP–patient consultation; it is based on the widely used Calgary–Cambridge guide to the medical interview. 127,128 The instrument provides a basis for raters to score each consultation in 12 domains (including gathering information, building the relationship, providing structure and achieving a shared understanding) and results in a final score between 0 and 10 (see Appendix 1). Raters were GPs experienced in the teaching of communication skills; all attended a 2-hour training session on the GCRS delivered by one of the original authors of the Calgary–Cambridge guide (Jonathan Silverman). We used four raters for each consultation to increase reliability. Raters accessed videos via a secure online portal. Each rater scored the consultations in a different random order to minimise the consequences of any order effects, and the same raters were used for all consultations. A simple mean of the four raters was calculated for each consultation and used in subsequent analyses.
From the rating of 56 consultations we expected 80% power (0.05 significance level) to detect a correlation coefficient of 0.37. To obtain the strongest correlation we designed our sampling strategy to include consultations with a wide range of scores: 28 (half) from those for which all patient responses to the seven communication items were either good or very good and 28 (half) from those for which at least one rating was less than good. For the 28 ‘less than good’ consultations, we selected those with the lowest patient communication scores. The 28 ‘good’ consultations were selected at random. We placed a restriction on the selection of consultations that barred the inclusion of more than two consultations involving the same GP.
Statistical analyses
Reliability of Global Consultation Rating Scale scores
We assessed the reliability of the GCRS scores by fitting a mixed-effect linear regression model to the 224 individual ratings (four ratings of 56 consultations). We anticipated that some raters would give systematically higher scores than others, resulting in an inflation of the within-consultation variance. As the same four raters were used to rate all 56 consultations this source of variation did not contribute to the reliability, as it manifests itself as a fixed offset in the mean consultation rating used in the analysis outlined below. Thus, a categorical fixed effect was included for rater in the models to account for this source of variance. The model additionally had a random intercept for consultation. In this model the variance of the random intercept represents the between-consultation variance (σb2) and the residual variance represents the within-consultation, between-rater variance (σw2) in ratings (after accounting for systematic differences between raters). The reliability (λGCRS) of the mean GCRS rating is then given by:
Consultation scores
The 56 consultations selected for rating were used to explore the association between GPs’ ratings of their own GP–patient communication, patient ratings of communication and the scores given by trained raters. The much larger sample of all videoed consultations was used to explore the association between GP and patient scores.
The association between patient scores and trained clinical raters’ scores
We explored the association between patient ratings and the ratings obtained by trained raters using a simple correlation coefficient and scatterplot. This coefficient can be corrected to account for the attenuation produced by the less than perfect reliability of the GCRS rating by multiplying by λGCRS. Consideration was given to adjusting for patient sociodemographic characteristics only if this resulted in reduced standard errors; however, this was not the case and so unadjusted results are shown. Because of potential concerns over normality assumptions, bootstrapping was used with 1000 bootstrap samples. To account for the non-independence of observations because of some GPs being represented twice, we performed the bootstrap sampling clustered by GP. Finally, we illustrated the relationship between single consultation ratings and GP ratings made up of many individual patient ratings by simulating scores for 100 hypothetical GPs with a range of communication skills as measured by the GCRS. The patient ratings for a given GCRS score were drawn from an appropriate distribution, informed by the findings of the observational work, and then, for each GP, mean patient scores were calculated for 1, 10, 30 and 100 patients.
The association between general practitioner scores and trained clinical raters’ scores
We explored the association between GP ratings of their performance and the ratings obtained by trained raters by calculating correlation coefficients. Consideration was given to adjusting for patient sociodemographic characteristics only if this resulted in reduced standard errors; however, this was not the case and so unadjusted results are shown. Because of potential concerns over normality assumptions, bootstrapping was used, in this case with 500 bootstrap samples. Again, to account for the non-independence of observations because of some doctors being represented twice, we performed the bootstrap sampling clustered by doctor.
The association between general practitioner scores and patient scores
To compare GP and patient scores we used all available consultations. First, we carried out a correlation analysis, as above. Subsequently, we conducted a regression analysis with doctor rating as the outcome, adjusting for patient age, gender, ethnicity and self-rated health. Finally, to evaluate the within-doctor association between patient and doctor scores, we augmented the previous model with a random effect for doctor. This final model accounted for the fact that some doctors may, in general, be more generous or more critical than other doctors. Standardised regression coefficients (betas) are reported, being directly comparable to (and in the case of models with a single exposure equal to) correlation coefficients. As above, clustered bootstrapping was used for all analysis.
All analysis was carried out using Stata® 13.1 (StataCorp LP, College Station, TX, USA).
Ethics approval
Ethics approval for the study was obtained from the NRES Committee East of England – Hertfordshire on 11 October 2011 (reference number 11/EE/0353).
Results
Consultations were videoed with 45 participating GPs from 13 general practices. During the period of study, a total of 908 patients had face-to-face consultations with participating doctors. Of these, 167 (18.4%) were ineligible (mostly children) and 529 completed a questionnaire (71.4% response rate) (Figure 3). A further 26 (5.1%) consultations were excluded from our analyses because of missing data. The videos selected for rating using the GCRS came from all 13 general practices and represented 37 GPs. One further consultation was excluded from our analysis of how GP and rater scores compared because of a rated consultation missing the communication score from the GP.
FIGURE 3.
Flow chart illustrating the recruitment and participation of patients.
Table 4 shows the self-reported demographics of those patients who completed a questionnaire and those whose consultation was selected for rating by trained raters. Male patients, patients aged 18–24 years and Asian patients were somewhat more likely to have been selected to have their consultation rated.
| Characteristic | Completed questionnaire, n (%) | Rated consultations, n (%) |
|---|---|---|
| Gender | ||
| Male | 212 (40.15) | 26 (46.43) |
| Female | 316 (59.85) | 30 (53.57) |
| Age (years) | ||
| 18–24 | 39 (7.41) | 10 (18.18) |
| 25–34 | 78 (14.83) | 7 (12.73) |
| 35–44 | 64 (12.17) | 7 (12.73) |
| 45–54 | 82 (15.59) | 4 (7.27) |
| 55–64 | 85 (16.16) | 8 (14.55) |
| 65–74 | 103 (19.58) | 7 (12.73) |
| 75–84 | 60 (11.41) | 8 (14.55) |
| ≥ 85 | 15 (2.85) | 4 (7.27) |
| Self-rated health | ||
| Excellent | 50 (9.51) | 3 (5.36) |
| Very good | 173 (32.89) | 14 (25) |
| Good | 182 (34.60) | 23 (41.07) |
| Fair | 83 (15.78) | 13 (23.21) |
| Poor | 38 (7.22) | 3 (5.36) |
| Ethnicity | ||
| White | 474 (90.98) | 44 (81.48) |
| Mixed | 5 (0.96) | 1 (1.85) |
| Asian or Asian British | 15 (2.88) | 6 (11.11) |
| Black or black British | 22 (4.22) | 1 (1.85) |
| Chinese | 4 (0.77) | 1 (1.85) |
| Other | 1 (0.19) | 1 (1.85) |
Reliability of Global Consultation Rating Scale scores
The distribution of patient scores and GCRS ratings is shown in Figure 4. Patient scores were highly skewed: the most common score was 100 out of a possible 100 (i.e. very good for all reported communication items; found for 21/56 consultations). The median score was 91 [interquartile range (IQR) 71–100] and the lowest score reported was 31 out of 100. In contrast, the GCRS ratings are reasonably symmetrical: the median score was 4.3 (IQR 3.6–5.5) and scores ranged from 2.2 to 6.8 out of a possible 10. From the mixed modelling of GCRS ratings (adjusted for rater) the estimated variances were 1.01 between consultations and 1.18 within consultations. Reliability for the mean of four ratings was 0.77.
FIGURE 4.
Distribution of (a) patient scores based on GP–patient communication survey items; and (b) ratings by trained raters on the GCRS.
The association between patient scores and trained clinical raters’ scores
Figure 5 shows patient scores plotted against average GCRS ratings for each consultation. There is weak evidence (p = 0.054) of an association between patient scores and GCRS ratings, with a correlation coefficient of 0.29. This increases to 0.33 when corrected for attenuation because of the imperfect reliability of the mean GCRS rating. When trained raters assessed communication during a consultation to be of a high standard, patients tended to do the same (with the exception of a single outlying low patient score). However, when trained raters judged communication during a consultation to be of a poor standard, patients reported communication as anything from poor to very good.
FIGURE 5.
Scatterplot comparing patient scores based on GP–patient communication survey items and ratings by trained raters on the GCRS.
Figure 6 shows the results of our simulation study, which is based on a hypothetical set of consultations with a range of trained rater scores (GCRS). For each GCRS score we defined a range of possible simulated patient scores, shown by the shaded green areas in Figure 6. The lower limit of these simulated patient scores increased as GCRS score increased. However, the upper limit of simulated patient scores was set at 100 for all possible GCRS scores in the simulation. For any given GCRS score we allowed patient scores to take any value in this range, with equal probability. The simulation is designed for illustrative purposes only and is not intended to accurately reflect the findings presented here. Figure 6a, designed to be reminiscent of Figure 5, shows what would be observed with just a single patient score per GP, that is, a weak correlation between patient rating and communication skill. The remaining parts of Figure 6 illustrate the effect of combining scores (taking the mean) from multiple consultations, rather than using a single rating. As the number of patient ratings taken increases, the correlation between trained rater scores and patient scores gets stronger. When the number of consultations is 30 this correlation becomes very strong (ρ = 0.97), becoming stronger still when n = 100.
FIGURE 6.
Simulated GP–patient communication scores based on different numbers of patient ratings: (a) n = 1; (b) n = 10; (c) n = 30; and (d) n = 100. The green areas show the possible individual patient scores that could be given for any particular level of communication competence, as assessed by the GCRS.
The association between general practitioner scores and trained clinical raters’ scores
Histograms showing the distribution of scores given to the 55 consultations are shown in Figures 7a and b for GP scores and trained rater scores, respectively. Both distributions are reasonably symmetrical. The GCRS scores cover a wide range of the possible values; in contrast, the GP scores for their own performance were all > 50 out of 100, indicating that no GP scored themselves as poor or very poor consistently across the domains for any one consultation. A scatterplot comparing GP scores with the GCRS scores is shown in Figure 8a. The wide scatter is reflected in the low correlation coefficients shown in Table 5, with no evidence that GP scores are associated with the scores of trained raters using the GCRS (p = 0.721). Because only a small number of consultations were rated, confidence intervals (CIs) are wide. However, it is of particular note that the upper CI is < 0.25, indicating that moderate or strong correlations between GP scores and rater scores are highly unlikely to be consistent with these data.
FIGURE 7.
Distribution of scores given to consultations by GPs scoring themselves (a) and (c), raters using the GCRS scale (b) and patients (d). Panels (a) and (b) apply only to those consultations rated by trained raters (n = 55), whereas panels (c) and (d) relate to all consultations for which physician and patient scores were recorded along with patient demographics (n = 503).
FIGURE 8.
Scatterplots illustrating the association between GP scores and (a) trained rater scores using the GCRS scale (n = 55); and (b) patient scores (n = 503). Note: that in each case the blue line is a line of best fit. Panel (a) applies only to those consultations rated by trained raters, whereas panel (b) relates to all consultations for which physician and patient scores were recorded along with patient demographics.
| Relationship examined | Trained raters (GCRS) (n = 55) | Patients (n = 503) | ||
|---|---|---|---|---|
| Correlation coefficient (95% CI) | p-value | Correlation coefficient/standardise regression coefficient (95% CI) | p-value | |
| Global association | –0.052 (–0.336 to 0.232) | 0.721 | 0.009 (–0.086 to 0.104) | 0.854 |
| Within GP association | NA | 0.025 (–0.060 to 0.110) | 0.565 | |
| Within GP association adjusted for patient sociodemographics | NA | 0.023 (–0.064 to 0.110) | 0.608 | |
The association between general practitioner scores and patient scores
Figures 7c and d show the distribution of GP scores and patient scores, respectively, for all consultations for which both are present (along with patient sociodemographic information). The distribution of GP scores is similar to that seen in the selection used for rating. In contrast, the distribution of patient scores is highly skewed, with 63.4% of patients giving the maximum score of 100. A scatterplot comparing GP scores with patient scores for the same consultations in shown in Figure 8b. The skewed nature of the patient scores is evident in this figure, which also shows that, although GPs do not often give themselves a score of < 50, on average they give themselves lower scores than patients. The lack of any clear relationship in this figure is reflected in the very low correlation coefficient shown in Table 5, again with no evidence of an association (p = 0.854). The lack of association persists when considering within-GP associations and when further adjusting for patient sociodemographics. Because of the increased sample size, CIs are tighter than those found when comparing GP scores with rater scores, such that only very weak correlations between GP and patient scores would be consistent with these data.
Discussion
We found a modest correlation between patients’ and trained raters’ assessments of the quality of communication in GP–patient consultations. This suggests that there is an association between patient ratings of communication and professionally defined standards of care. Importantly, when trained raters identified communication as good, patients tended to agree with this. However, when trained raters identified communication as poor, patients ranged in their assessments of communication from poor to very good. By contrast, we found no evidence of an association between GPs’ and trained raters’ assessments of communication performance and no evidence of an association between GPs’ and patients’ assessments of communication quality.
The first aim of this workstream was to explore how far patient ratings reflect accepted professional standards of communication. Our findings suggest that, although trained raters and patients tend to agree what good communication looks like in a consultation, clinical raters are more likely than patients to judge communication as poor. We outline two possible mechanisms driving this divergence; both assume that raters’ assessments of communication quality are the ‘gold standard’ (an issue that we discuss further in Strengths and limitations). The first mechanism arises from the well-known phenomenon of skewed patient ratings, with a large proportion of patients rating communication as ‘very good’. 130,132,133 By contrast, GCRS ratings tend to cluster around the middle scores available to raters. It is therefore possible that the presence of ceiling effects inherent in the patient question items may artificially constrain the responses that patients would like to give, preventing them from being able to distinguish the very best consultations from those that they judge to be simply good. If our observed pattern is attributable to ceiling effects, this implies a weak correlation between underlying ‘true’ patient opinion (not the reported opinion expressed using the available survey instruments) and trained rater scores. Thus, this mechanism requires that patients differ from raters in their views of what good or poor communication in a consultation looks like. As a result, the more positive patient opinion is ‘held back’ by only being able to choose questionnaire options ranging from ‘very poor’ to ‘very good’ (and not, for example, ‘excellent’), despite extensive instrument development. 134
However, the second – and we argue more plausible – mechanism is that there are wider factors at play that inhibit some patients from assigning poor scores to consultations that they do perceive as involving poor communication. It is important to note that any such inhibition would have to apply unevenly between patients to explain the range of patient scores seen for consultations rated as poor by the trained raters – whereas some patients are able to choose ‘poor’ as an option, others feel less able to do so. For this mechanism to be driving our observed pattern, the ‘true’ opinion of patients would be more strongly correlated with the opinion of trained raters, with general agreement between patients and raters about what good or poor communication looks like. Such a phenomenon may lead to an overestimate of doctors’ ‘true’ patient experience scores, whereas the former mechanism (in which patients are constrained in their choice of responses) would lead to an underestimate of doctor performance.
Although we are unable to determine the relative contribution of either mechanism from the methodology of this current workstream, there is existing evidence that patients may be inhibited in their judgements of care. In particular, qualitative research has identified a number of psychological and social factors which suggest that patients struggle to criticise GPs’ performance in surveys. For example, an investigation into how patients evaluated community mental health services found that they frequently avoided giving negative scores on experience questionnaires; instead, allowances for poor care were constructed by referencing their perceptions of the duties and culpabilities of health-care providers. 98 Similarly, patients undergoing elective orthopaedic surgery reinterpreted their experiences in a positive light as a result of feelings of dependency on their health-care provider and a perceived need to maintain constructive relationships with their GP. 97 A tendency to excuse rather than report poor care has also been identified in breast cancer patients. 135 These findings were confirmed in the qualitative research that we undertook with our sample, as previously discussed in Chapter 2.
The lack of association between GPs’ own ratings of their performance and both trained raters’ and patients’ assessments of the same echoes previous research that has identified gaps in doctors’ and others’ evaluations. 122–124 The absence of agreement between GPs and trained raters and between GPs and patients suggests that patients’ assessments of what constitutes a ‘good’ consultation may vary. This has potentially important implications for the utility of GP self-reflection in developing their clinical practice. Reflective practice has become a core part of continuing professional development over the years and the identification of learning needs forms an important aspect of this. 136 Indeed, the collection and consideration of patient feedback is a central component of the supporting information required for current medical appraisal and revalidation and appraisers are required to explore ‘what [doctors] think the supporting information says about [their] practice and how [they] intend to develop or modify your practice as a result of that reflection’ (p. 2). 137 In this study, trained clinical raters (all GPs) were consistently more negative about communication performance than participating GPs were. In seeking to improve communication skills and patient experience, reliance on a doctor’s own assessment may not be a robust approach and there may be an important role for external assessment of communication performance.
Strengths and limitations
It was not necessary to seek a representative sample of practices or GPs for the purposes of this study. Instead, we intentionally approached a sample of practices previously found to be receiving lower patient experience scores for communication. Our sampling strategy was informed by the need to locate consultations that patients identified as less than good; the proportion of such consultations is small and so to increase study efficiency we deliberately approached practices that had received lower scores for communication in the national GP Patient Survey. Not all GPs in every practice took part and it is possible that the GPs who did so were more confident in their ability to communicate with patients.
Our patient consent rate was 71.4% of eligible patients. The research team missed only a small number of patients [2.0% (15/741) of those eligible] and so exclusions predominantly reflect those who did not consent to participate. Recorded consultations concerning some medical conditions may be under-represented as participants may have been more likely to decline being video recorded. However, participants’ age, gender, self-rated health and ethnicity were broadly representative of the population attending general practices.
We assessed communication using two well-validated instruments: the GP Patient Survey items for patients and GPs and the GCRS for trained raters. 126,134 The GCRS was derived from the Calgary–Cambridge guide, which is used widely for communication skills training and represents agreed professional norms of high-quality communication. 127,128,138 Recently, the question has arisen as to how and whether or not trained raters take account of contextual factors in assessing the communication skills of GPs, for example by allowing variations from ‘accepted practice’ when scoring performance in particular situations. 139,140 However, the GCRS has been explicitly designed to focus only on the consultation process and contains no task-based items that may be context specific (such as requiring a rating for specific physical examinations). Additionally, it enables raters to choose ‘not applicable’ when necessary, although, in fact, this was rarely endorsed by raters in this study.
As mentioned earlier, in drawing conclusions about the meaning of patients’ and GPs’ ratings of communication quality, we have positioned the trained raters as being ‘the gold standard’. This is not to suggest raters are more valued or competent assessors of communication than patients or GPs, but simply to use them as representative of professionally agreed norms of behaviour against which to judge patient and participating doctor evaluations of communication. In doing so, we are able to provide evidence that patient assessments tap in to the same underlying construct of communication drawn on by trained raters, but also that patients are less likely to judge consultations as poor. We are also able to provide evidence that GP assessments of their own performance do not appear to be associated with the same ideas of what ‘good’ communication looks like for trained raters.
Conclusions
Our findings support observations that patients may be inhibited in criticising doctors’ performances. If indeed patients are reluctant to give lower ratings that truly reflect their experience, mean survey scores may be overestimates of performance. We therefore suggest that the practice of taking mean survey scores at face value and assuming that they provide a realistic reflection of absolute performance level of either GPs or their practices is inadvisable, as such scores are likely to be biased. However, the use of relative rankings to identify GPs who are better or poorer at communicating with patients may be an acceptable approach to benchmarking performance, as long as statistically reliable figures are obtained. Previous research has demonstrated that the GP Patient Survey communication questions can differentiate between the performance of GPs and practices, as long as an adequate sample size is used to achieve acceptable statistical reliability. 129,141 This was confirmed by our simulation: with sufficient patient scores a strong correlation between patient ratings and competency will be observed. In the use of patient experience scores as quality indicators, our findings suggest that it is, therefore, possible to (1) trust aggregated patient scores that meet traditional standards of reliability as valid measures of comparative performance with respect to communication and (2) trust relatively low mean patient ratings. However, crucially, we cannot necessarily assume that a high mean patient rating means that all is well.
General practitioner assessments of their own standards of communication were poorly associated both with professionally defined norms of communication and with patients’ own assessments of what happened in a consultation. Taken together with our findings on the importance of rater feedback for identifying consultations in which communication is less than ideal, our findings suggest that there may be a current gap in the use of external assessments of communication competence.
Chapter 4 Ability of patients to see the clinician of their choice
Abstract
Background
This chapter describes analyses of data from the GP Patient Survey investigating which patients have a preference for seeing a particular GP and how successful they are in seeing that doctor. We report these trends over a 4-year period. In addition, we undertook analyses to examine whether or not patients’ expectations of who they wished to see and who they did see (a doctor or a nurse) influenced their assessment of the consultation.
Methods
This study involved analyses of data from the GP Patient Survey.
Results
The majority of patients have a particular GP who they prefer to see. This increased from just over 50% in those aged 18–25 years to 80% in those aged > 75 years. Of those patients who have a preference to see a particular GP, 30% were not able to see that doctor easily in 2010/11. That percentage has been rising year on year, to 39% in 2013/14, indicating substantial problems in terms of patients being able to see the doctor of their choice. Patients who saw a nurse when they wanted to see a GP gave scores for communication with the nurse that were substantially lower (adjusted difference 5.99%, 95% CI 5.71% to 6.28%) than those given by patients who wanted to see a nurse in the first place.
Conclusions
Patients’ ability to see a doctor of their choice is seriously compromised, with a high proportion of patients who have a preference for a particular doctor unable to see that doctor on a regular basis. This is a significant quality problem for the NHS.
Introduction and rationale for the study
This chapter relates specifically to the fourth aim of the programme: to understand better patients’ responses to questions on communication and seeing a doctor of their choice. In the rest of the programme we have focused on communication between doctors and patients. In this chapter we present our analyses of questions in the GP Patient Survey11 that relate to patients’ ability to see the doctor of their choice.
Continuity of care, specifically relational continuity, is valued by patients and is a core value of general practice. Nevertheless, changes to practice organisation and staffing (including targets relating to improved access) have all combined to make it more difficult for patients to see a regular doctor. There are no routinely collected measures of continuity of care. However, there are two questions that have remained largely unchanged in the GP Patient Survey for several years that are relevant to continuity of care. These are shown in Table 6.
| Q15. Is there a particular doctor you prefer to see at your GP surgery or health centre? | |
|---|---|
| □ | Yes . . . Please go to Q16 |
| □ | No . . . Please go to Section F |
| □ | There is usually only one doctor in my GP surgery or health centre . . . Please go to section F |
| Q16. How often do you see the doctor you prefer to see? | |
| □ | Always or almost always |
| □ | A lot of the time |
| □ | Some of the time |
| □ | Never or almost never |
| □ | Not tried at this GP surgery or health centre |
The wording of these questions recognise that not all patients want to see a particular doctor and enable the ability to see a particular doctor to be assessed among patients who have that preference. These questions are not a direct measure of continuity of care, but combine elements of continuity with an element of patient choice.
The aim of the analysis of these questions was, first, to identify which patients most valued having a particular doctor and, second, to examine the extent to which patients were able to see the doctor of their choice. Because these questions have remained stable for some years, we were also able to examine trends over time.
We also include in this chapter some analyses of patient experience with practice nurses. Although this was not part of our original programme of work, it is of interest in its own right and also gave us the opportunity to examine patients’ responses when they wished to see a doctor but were given an appointment with a nurse and when they wished to see a nurse but were given an appointment with a doctor.
Methods
Three sets of analyses are presented in this chapter:
-
determination of which patients express a preference for and manage to see a doctor of their choice
-
examination of trends in the proportion of patients able to see a doctor of their choice
-
the association between patient ratings of communication and the mismatch between the type of appointment wanted and the type of appointment received.
Analysis 1
Data from the 2009/10 GP Patient Survey11 were used for the analysis of which patients had a preference for, and which succeeded in seeing, a particular doctor. The results presented are a summary of those published in the British Journal of General Practice. 142
Responses to how often patients were able to see their preferred doctor (when they expressed a preference for doing so) were dichotomised into a ‘yes’ (‘always or almost always’ or ‘a lot of the time’)/’no’ (‘some of the time’ or ‘never or almost never’) measure. Survey weights were developed by Ipsos MORI (the survey provider) and were used in our analysis to account for the complex survey design and non-response in prevalence estimates of preference for, and success in, seeing a preferred doctor. These weights employed rim weighting with two rims: (1) age-by-gender (8 × 2 levels) and (2) practice (8362 levels).
Separate crude and multivariate logistic regression models were used to examine the association between various patient and practice characteristics and preference for, and success in, seeing a preferred doctor. We adjusted for gender, age group, ethnicity, deprivation quintile, self-reported chronic medical or psychological/emotional condition, number of practice doctors and type of appointments requested by the patient in the previous 6 months. Crude models made use of the weights and adjusted standard errors to account for the survey design. Multivariate models did not make use of the weights but did include random intercepts for practice to account for clustering of patients within practices and to better distinguish the experiences and preferences of patient subgroups from general variation in continuity at practice level.
Although patient registration with a given practice is largely determined by geographical proximity, some patients might choose to register with a smaller practice specifically to receive better continuity of care, in which case it would not have been appropriate to adjust sociodemographic associations for practice size. For this reason we performed a sensitivity analysis excluding the number of practice doctors. The results were very similar, for which reason data are not shown. Stata 11 was used for the descriptive analyses and SAS 9.2 (SAS Institute Inc., Cary, NC, USA) for the regression analyses.
Analysis 2
In this analysis we used data from 4 years of the GP Patient Survey (2010/11 to 2013/14),11 in which the questions addressing the ability to see a preferred GP have remained unchanged. We present annual national figures after applying the survey design and non-response weights such that percentages are representative of the national population rather than respondents to the survey.
Analysis 3
For this analysis we used data from the 2013/14 GP Patient Survey. 11 This included more detailed questions on appointments than previous surveys and, in particular, we were able to analyse the responses of patients who saw a nurse on their last visit, comparing those who contacted the practice wanting to see a nurse with those whose original request had been to see a doctor. Similarly, we were able to analyse the responses of patients who saw a doctor on their last visit, comparing those who wanted to see a doctor with those whose original request had been to see a nurse.
We first present a descriptive analysis to examine the extent to which the type of appointment that patients obtained was the same as or different from the type of appointment that they wanted. This was done by comparing responses to the question ‘Last time you wanted to see or speak to a GP or nurse from your GP surgery: what did you want to do?’ with responses to the question ‘What type of appointment did you get? I got an appointment . . .’. For both questions response options allowed patients to indicate that they wanted to/got to see a GP at the surgery, see a nurse at the surgery, speak to a GP on the telephone, speak to a nurse on the telephone or have a home visit. Additionally, when asking about what they wanted, there was an option to state ‘I didn’t mind/wasn’t sure what I wanted’. Because patients often endorsed more than one response we reduced responses to the first question into five categories:
-
those who wanted to either see or speak to a GP (or both)
-
those who wanted to either see or speak to a nurse (or both)
-
those who wanted a home visit
-
those who weren’t sure or didn’t mind
-
those who wanted more than one of the previous four categories.
For the second question the same categories were used, excluding the ‘didn’t mind/wasn’t sure’ category as this was not an option for this question. We then cross-tabulated what people wanted with what they got, again using the design and non-response weights.
Finally, we considered whether or not reported nurse–patient communication and GP–patient communication varied according to any mismatch between what people wanted and what they got in terms of who the appointment was with and what type of appointment it was. Two separate analyses were performed, one for nurse communication and one for GP communication. Each analysis was restricted to those reporting that they had had an appointment with the appropriate clinician on their last visit to their GP surgery. For relative simplicity this analysis was further restricted to those who endorsed only one box for both the question on what type of appointment they wanted and the question on what they got. Eight categories were created (for each analysis) covering the various combinations of seeing or speaking to someone and whether the person they wanted to see or speak to was a GP. Composite nurse–patient communication and GP–patient communication scores (between 0 and 100) were created in an identical way to that described earlier in this report, except using responses to the equivalent question about nurses when appropriate. Crude differences between the categories were estimated using linear regression (restricted to those who had complete information for age, gender, ethnicity, confidence in managing their own health, the presence of a long-standing heath condition and deprivation). Mixed-effects linear regression was then used for an adjusted analysis including age, gender, ethnicity, confidence in managing their own health, the presence of a long-standing heath condition and deprivation. Practice was included as a random effect (intercept).
Results
The overall response rate to the 2009/10 GP Patient Survey was 39%, with 2,169,718 completed responses from patients in 8362 practices.
Analysis 1a: preference for seeing a particular doctor
In total, 2% of patients reported that there was only one GP in their practice. After excluding those patients from further analysis, 62% of patients reported having a preference for seeing a particular doctor (Table 7). Such a preference varied across patient groups and was higher for women (68% vs. 56% in men), older patients (52% for age group 18–24 years increasing to 81% for age group 75–84 years), those with chronic medical or psychological/emotional conditions (75% and 78%, respectively, vs. 52% and 61%, respectively, for those without) and those living in more affluent areas (from 60% to 64% for most to least deprived patients). Preference for seeing a particular doctor ranged from 47% to 65% of respondents across the 16 ethnic groups and increased with the number of practice GPs (58% for practices with two GPs increasing to 63% for practices with six to nine GPs). Preference for seeing a particular doctor was higher in patients who had recently requested only non-urgent appointments in the previous 6 months (68%) than in patients who had requested only urgent appointments (58%). The crude odds ratios (ORs) (see Table 7) reflect the associations described above. All associations are stronger than would be expected by chance (p < 0.001).
| Patient/practice variables | Weighted prevalencea (95% CI) | Crude weighted ORa (95% CI) | Adjusted ORb (95% CI) |
|---|---|---|---|
| All survey respondents | 62.2 (61.9 to 62.4) | NA | NA |
| Gender | |||
| Male | 56.3 (56.1 to 56.6) | Reference | Reference |
| Female | 67.5 (67.2 to 67.7) | 1.60 (1.59 to 1.61) | 1.50 (1.49 to 1.52) |
| Age group (years) | |||
| 18–24 | 51.7 (51.1 to 52.2) | 0.49 (0.48 to 0.50) | 0.65 (0.64 to 0.66) |
| 25–34 | 51.0 (50.6 to 51.3) | 0.48 (0.47 to 0.48) | 0.55 (0.54 to 0.56) |
| 35–44 | 56.0 (55.6 to 56.3) | 0.58 (0.58 to 0.59) | 0.66 (0.65 to 0.67) |
| 45–54 | 61.4 (61.1 to 61.7) | 0.73 (0.72 to 0.74) | 0.79 (0.78 to 0.80) |
| 55–64 | 68.6 (68.3 to 68.9) | Reference | Reference |
| 65–74 | 76.0 (75.8 to 76.3) | 1.45 (1.44 to 1.47) | 1.36 (1.35 to 1.38) |
| 75–84 | 81.1 (80.8 to 81.4) | 1.97 (1.94 to 2.00) | 1.71 (1.69 to 1.74) |
| ≥ 85 | 80.0 (79.5 to 80.4) | 1.83 (1.78 to 1.88) | 1.54 (1.50 to 1.58) |
| Ethnic group | |||
| Whitec | |||
| White Britishd | 62.9 (62.6 to 63.2) | Reference | Reference |
| Irishd | 65.1 (64.2 to 65.9) | 1.10 (1.06 to 1.14) | 0.97 (0.94 to 1.00) |
| Any other whited | 57.5 (56.9 to 58.1) | 0.80 (0.78 to 0.82) | 1.03 (1.01 to 1.05) |
| Mixedc | |||
| White and black Caribbeand | 56.8 (54.8 to 58.9) | 0.78 (0.72 to 0.84) | 1.05 (0.97 to 1.14) |
| White and black Africand | 52.2 (49.6 to 54.7) | 0.64 (0.58 to 0.71) | 0.92 (0.84 to 1.02) |
| White and Asiand | 56.7 (53.8 to 59.6) | 0.77 (0.69 to 0.87) | 1.07 (0.99 to 1.16) |
| Any other mixedd | 59.7 (57.7 to 61.7) | 0.88 (0.81 to 0.95) | 1.09 (1.02 to 1.18) |
| South Asianc | |||
| Indiand | 63.0 (62.2 to 63.9) | 1.01 (0.97 to 1.04) | 1.49 (1.45 to 1.53) |
| Pakistanid | 61.4 (60.4 to 62.4) | 0.94 (0.90 to 0.98) | 1.49 (1.43 to 1.54) |
| Bangladeshid | 61.7 (60.2 to 63.2) | 0.95 (0.89 to 1.01) | 1.74 (1.64 to 1.84) |
| Any other Asiand | 59.0 (57.9 to 60.1) | 0.85 (0.81 to 0.89) | 1.28 (1.23 to 1.33) |
| Blackc | |||
| Black Caribbeand | 61.9 (60.9 to 62.8) | 0.96 (0.92 to 1.00) | 1.14 (1.10 to 1.18) |
| Black Africand | 47.3 (46.4 to 48.2) | 0.53 (0.51 to 0.55) | 0.81 (0.78 to 0.83) |
| Any other blackd | 59.1 (57.2 to 61.1) | 0.86 (0.79 to 0.93) | 1.08 (0.99 to 1.17) |
| Chinesec | |||
| Chinesed | 48.5 (47.0 to 50.0) | 0.56 (0.52 to 0.59) | 0.86 (0.81 to 0.90) |
| Otherc | |||
| Other ethnic groupd | 58.5 (57.9 to 59.1) | 0.83 (0.81 to 0.85) | 1.14 (1.12 to 1.17) |
| Deprivation quintile | |||
| 1 (affluent) | 64.1 (63.7 to 64.5) | Reference | Reference |
| 2 | 63.3 (62.9 to 63.6) | 0.96 (0.95 to 0.98) | 0.96 (0.95 to 0.97) |
| 3 | 62.2 (61.8 to 62.6) | 0.92 (0.90 to 0.94) | 0.92 (0.91 to 0.93) |
| 4 | 61.0 (60.6 to 61.3) | 0.87 (0.86 to 0.89) | 0.89 (0.88 to 0.90) |
| 5 (deprived) | 59.5 (59.1 to 60.0) | 0.82 (0.81 to 0.84) | 0.84 (0.83 to 0.85) |
| Presence of self-reported chronic medical condition | |||
| No | 52.1 (51.7 to 52.4) | Reference | Reference |
| Yes | 74.8 (74.5 to 75.0) | 2.73 (2.70 to 2.76) | 1.87 (1.86 to 1.89) |
| Presence of self-reported long-standing psychological or emotional condition | |||
| No | 61.3 (61.1 to 61.6) | Reference | Reference |
| Yes | 78.3 (77.9 to 78.7) | 2.28 (2.23 to 2.33) | 1.59 (1.57 to 1.62) |
| Number of practice GPs | |||
| 1 | 56.4 (55.6 to 57.2) | 0.94 (0.90 to 0.97) | 1.01 (0.97 to 1.05) |
| 2 | 58.0 (57.4 to 58.6) | Reference | Reference |
| 3 | 61.9 (61.3 to 62.5) | 1.18 (1.13 to 1.22) | 1.20 (1.15 to 1.25) |
| 4 | 63.5 (62.9 to 64.1) | 1.26 (1.22 to 1.30) | 1.29 (1.24 to 1.35) |
| 5 | 63.6 (63.0 to 64.2) | 1.26 (1.22 to 1.31) | 1.31 (1.26 to 1.37) |
| 6–9 | 63.0 (62.6 to 63.4) | 1.23 (1.19 to 1.27) | 1.30 (1.25 to 1.34) |
| 10+ | 62.2 (60.7 to 63.6) | 1.19 (1.11 to 1.27) | 1.28 (1.19 to 1.37) |
| Type of appointments sought in previous 6 monthse | |||
| No appointment requested | 47.5 (47.2 to 47.8) | 0.65 (0.64 to 0.66) | 0.67 (0.66 to 0.68) |
| Urgent only | 58.3 (58.0 to 58.6) | Reference | Reference |
| Non-urgent only | 67.5 (67.1 to 67.9) | 1.49 (1.46 to 1.51) | 1.40 (1.39 to 1.42) |
| Both urgent and non-urgent | 73.5 (73.2 to 73.7) | 1.98 (1.96 to 2.00) | 1.85 (1.83 to 1.87) |
In multivariate analysis there was strong evidence that differences exist in the preference for seeing a particular doctor across all sociodemographic groups after adjusting for other factors (p < 0.001 for all variables) (see Table 7). This preference was more common among women (OR 1.50), older people (OR 1.71 for age group 74–85 years vs. age group 55–64 years), respondents suffering from a chronic medical (OR 1.87) or psychological/emotional (OR 1.59) condition and those from more affluent areas (OR 0.84 for most deprived vs. most affluent areas). Patients from South Asian ethnic groups (Bangladeshi, Indian, Pakistani and ‘any other Asian’ groups) had a substantially higher preference for seeing a particular doctor (OR 1.74, 1.49, 1.49 and 1.28, respectively, vs. white British patients). Patients were more likely to express such a preference if they were registered with a practice with a greater number of GPs (OR 1.3 for patients registered with practices with six to nine GPs vs. patients registered with practices with two GPs) and if they had sought non-urgent appointments (OR 1.4 for patients seeking non-urgent appointments only vs. patients seeking urgent appointments only).
Analysis 1b: ability to see the doctor of the patient’s choice
The next analyses are restricted to patients with a preference for seeing a particular doctor. Of these patients, 72% were successful in seeing the doctor who they preferred ‘always or almost always’ or ‘a lot of the time’; we refer to those two response categories using the term ‘most of the time’ hereafter (Table 8). The proportion of patients who were successful in seeing their preferred GP most of the time was higher in men (74% vs. 70% in women), older patients (60% for age group 18–24 years increasing to 87% for age group 75–84 years) and those with chronic medical or psychological/emotional conditions (77% and 75%, respectively, vs. 66% and 72%, respectively, in those without). White patients were more likely to be able to see the doctor of their choice than patients in most other ethnic groups. More deprived patients were less successful in seeing the doctor who they preferred most of the time (67% for the most deprived patients rising to 74% for the least deprived patients). Success in seeing a particular doctor decreased as the number of practice GPs increased (79% for practices with one GP vs. 69% for practices with ≥ 10 GPs). Success in seeing a particular doctor was lowest among patients requesting urgent appointments only (69%), with it being greatest for patients requesting only non-urgent appointments (79%). The crude ORs (see Table 8) reflect the associations described above. All associations are stronger than would be expected by chance (p < 0.001).
| Patient/practice variables | Weighted prevalencea (95% CI) | Crude weighted ORa (95% CI) | Adjusted ORb (95% CI) |
|---|---|---|---|
| All survey respondents | 71.8 (71.4 to 72.1) | ||
| Gender | |||
| Male | 73.6 (73.2 to 74.0) | Reference | Reference |
| Female | 70.2 (69.9 to 70.6) | 0.85 (0.84 to 0.86) | 0.87 (0.86 to 0.88) |
| Age group (years) | |||
| 18–24 | 59.8 (59.1 to 60.5) | 0.43 (0.42 to 0.44) | 0.43 (0.42 to 0.44) |
| 25–34 | 60.2 (59.7 to 60.8) | 0.44 (0.43 to 0.44) | 0.48 (0.47 to 0.49) |
| 35–44 | 63.6 (63.1 to 64.0) | 0.50 (0.49 to 0.51) | 0.54 (0.53 to 0.55) |
| 45–54 | 69.8 (69.4 to 70.3) | 0.67 (0.66 to 0.68) | 0.68 (0.67 to 0.69) |
| 55–64 | 77.7 (77.3 to 78.1) | Reference | Reference |
| 65–74 | 84.3 (84.0 to 84.7) | 1.55 (1.52 to 1.58) | 1.53 (1.50 to 1.56) |
| 75–84 | 86.5 (86.1 to 86.8) | 1.84 (1.80 to 1.87) | 1.82 (1.79 to 1.86) |
| ≥ 85 | 85.3 (84.8 to 85.7) | 1.66 (1.61 to 1.72) | 1.56 (1.51 to 1.61) |
| Ethnic group | |||
| Whitec | |||
| White Britishd | 73.7 (73.4 to 74.1) | Reference | Reference |
| Irishd | 74.1 (73.2 to 75.1) | 1.02 (0.97 to 1.07) | 0.90 (0.86 to 0.94) |
| Any other whited | 66.9 (66.2 to 67.5) | 0.72 (0.70 to 0.74) | 0.85 (0.83 to 0.88) |
| Mixedc | |||
| White and black Caribbeand | 61.8 (59.1 to 64.4) | 0.58 (0.52 to 0.64) | 0.90 (0.81 to 1.00) |
| White and black Africand | 56.6 (53.4 to 59.8) | 0.46 (0.41 to 0.53) | 0.68 (0.60 to 0.78) |
| White and Asiand | 63.4 (60.7 to 66.1) | 0.62 (0.55 to 0.69) | 0.81 (0.72 to 0.90) |
| Any other mixedd | 62.4 (60.1 to 64.7) | 0.59 (0.54 to 0.65) | 0.74 (0.67 to 0.81) |
| South Asianc | |||
| Indiand | 60.7 (59.3 to 62.1) | 0.55 (0.52 to 0.58) | 0.73 (0.71 to 0.76) |
| Pakistanid | 54.4 (52.9 to 55.9) | 0.43 (0.40 to 0.45) | 0.66 (0.63 to 0.69) |
| Bangladeshid | 50.2 (48.1 to 52.3) | 0.36 (0.33 to 0.39) | 0.57 (0.53 to 0.61) |
| Any other Asiand | 56.8 (55.4 to 58.1) | 0.47 (0.44 to 0.49) | 0.59 (0.56 to 0.62) |
| Blackc | |||
| Black Caribbeand | 65.6 (64.2 to 67.0) | 0.68 (0.64 to 0.72) | 0.83 (0.79 to 0.87) |
| Black Africand | 52.3 (50.9 to 53.8) | 0.39 (0.37 to 0.42) | 0.55 (0.53 to 0.58) |
| Any other blackd | 58.0 (55.3 to 60.6) | 0.49 (0.44 to 0.55) | 0.70 (0.62 to 0.78) |
| Chinesec | |||
| Chinesed | 56.2 (54.1 to 58.4) | 0.46 (0.42 to 0.50) | 0.55 (0.51 to 0.59) |
| Otherc | |||
| Other ethnic groupd | 60.6 (59.9 to 61.4) | 0.55 (0.53 to 0.57) | 0.66 (0.64 to 0.68) |
| Deprivation quintile | |||
| 1 (affluent) | 74.3 (73.7 to 74.8) | Reference | Reference |
| 2 | 74.1 (73.6 to 74.6) | 0.99 (0.97 to 1.02) | 0.99 (0.97 to 1.01) |
| 3 | 72.3 (71.8 to 72.8) | 0.90 (0.88 to 0.93) | 0.95 (0.94 to 0.97) |
| 4 | 69.6 (69.1 to 70.1) | 0.79 (0.77 to 0.82) | 0.91 (0.89 to 0.93) |
| 5 (deprived) | 67.0 (66.4 to 67.6) | 0.70 (0.68 to 0.73) | 0.86 (0.84 to 0.88) |
| Presence of self-reported chronic medical condition | |||
| No | 66.3 (65.8 to 66.7) | Reference | Reference |
| Yes | 76.8 (76.4 to 77.1) | 1.68 (1.66 to 1.70) | 1.29 (1.27 to 1.30) |
| Presence of self-reported long-standing psychological or emotional condition | |||
| No | 71.9 (71.5 to 72.2) | Reference | Reference |
| Yes | 75.3 (74.8 to 75.8) | 1.19 (1.17 to 1.22) | 1.25 (1.22 to 1.27) |
| Number of practice GPs | |||
| 1 | 78.5 (77.3 to 79.6) | 1.10 (1.01 to 1.19) | 1.42 (1.33 to 1.52) |
| 2 | 76.9 (76.0 to 77.7) | Reference | Reference |
| 3 | 74.5 (73.6 to 75.4) | 0.88 (0.82 to 0.94) | 0.76 (0.71 to 0.81) |
| 4 | 73.4 (72.6 to 74.3) | 0.83 (0.78 to 0.89) | 0.66 (0.61 to 0.70) |
| 5 | 72.0 (71.1 to 72.9) | 0.77 (0.73 to 0.83) | 0.57 (0.53 to 0.61) |
| 6–9 | 69.7 (69.2 to 70.3) | 0.69 (0.66 to 0.73) | 0.48 (0.45 to 0.51) |
| 10+ | 68.8 (67.0 to 70.7) | 0.67 (0.60 to 0.74) | 0.44 (0.40 to 0.49) |
| Type of appointments sought in previous 6 monthse | |||
| No appointments requested | 73.6 (73.2 to 74.0) | 1.23 (1.21 to 1.25) | 1.17 (1.15 to 1.19) |
| Urgent only | 69.4 (68.9 to 69.8) | Reference | Reference |
| Non-urgent only | 78.8 (78.4 to 79.2) | 1.64 (1.60 to 1.68) | 1.59 (1.57 to 1.62) |
| Both urgent and non-urgent | 70.3 (69.9 to 70.7) | 1.05 (1.03 to 1.06) | 1.12 (1.10 to 1.13) |
In multivariate analysis there was strong evidence that differences relating to the success of seeing a preferred doctor persist after adjusting for other factors (p < 0.001 for all variables) (see Table 8). Women were less likely to be successful than men in seeing the doctor of their choice (OR 0.87). This contrasts with older patients (OR 1.82 for age group 74–85 years vs. age group 55–64 years), those with a chronic medical (OR 1.29) or psychological/emotional (OR 1.25) condition and white patients, all of whom were more likely to be successful than their respective reference groups. Success in seeing a preferred doctor was also less common in deprived areas (OR 0.86 for most deprived vs. most affluent). Patients registered with larger practices were less likely to report that they could see a doctor of their choice most of the time (OR 0.48 for patients registered with practices with six to nine doctors vs. patients registered with practices with two doctors). Patients who requested only non-urgent appointments were more likely to be successful in seeing the doctor who they preferred (OR 1.59 vs. patients requesting only urgent appointments).
Analysis 2: seeing the clinician of the patient’s choice – changes over time
Figure 9 shows the percentage of patients who have a preference for seeing a particular doctor who were actually able to do so last time they had a consultation. The percentage of patients able to see the GP of their choice has declined year on year for the 4 years from 2010/11 to 2013/14, from 70% to 61%. Note that percentages presented have been weighted for survey design and non-response such that they represent the national population rather than respondents.
FIGURE 9.
Percentage of patients able to see or speak to the GP they prefer ‘a lot of the time’, ‘almost always’ or ‘always’ (of those who say that they have a particular GP who they prefer to see – data from the GP Patient Survey).
Analysis 3: seeing the clinician of the patient’s choice – association with subsequent rating
Of the 903,357 people who responded to the 2013/14 GP Patient Survey, 870,085 answered the question regarding what they wanted to do last time they contacted the GP surgery. Table 9 shows how that question was answered. Accounting for non-response and design weighting suggests that over three-quarters of patients wanted only to see or speak to a GP, whereas just under 15% wanted only to see or speak to a nurse.
| Last time you wanted to see or speak to a GP or nurse from your GP surgery, what did you want to do? | n | Weighted % |
|---|---|---|
| Wanted to see and/or speak to a GP | 653,526 | 77.7 |
| Wanted to see and/or speak to a nurse | 139,300 | 14.5 |
| Wanted a home visit | 12,873 | 1.2 |
| Didn’t mind/wasn’t sure | 15,404 | 2.4 |
| One or more of the above | 48,982 | 4.3 |
| Total | 870,085 | 100.0 |
We then compared the type of appointment that patients wanted and the type of appointment that they actually received (Table 10). By and large the vast majority of patients got what they wanted, with 96% of people who wanted to see or speak to a GP achieving this compared with 92% of people who wanted to see or speak to a nurse. The percentage wanting a home visit who received one was lower at 80%.
| Last time you wanted to see or speak to a GP or nurse from your GP surgery, what did you want to do? | What type of appointment did you get? | ||||
|---|---|---|---|---|---|
| To see and/or speak to a GP | To see and/or speak to a nurse | A home visit | One or more of the above | Total | |
| Wanted to see and/or speak to a GP | 95.9 | 2.9 | 0.1 | 1.1 | 100.0 |
| Wanted to see and/or speak to a nurse | 6.3 | 92.2 | 0.1 | 1.4 | 100.0 |
| Wanted a home visit | 15.7 | 2.2 | 79.6 | 2.5 | 100.0 |
| Didn’t mind/wasn’t sure | 67.1 | 29.1 | 1.1 | 2.7 | 100.0 |
| One or more of the above | 29.7 | 8.6 | 0.7 | 61.1 | 100.0 |
| Overall | 77.6 | 17.5 | 1.1 | 3.8 | 100.0 |
The results of the analysis investigating the association between reported nurse–patient communication scores and the mismatch between what people wanted and what they received in terms of appointment is shown in Table 11. This analysis was restricted to the 121,086 patients who reported seeing or speaking to a nurse on their last visit to the GP surgery who also had complete information on the covariates used in the adjusted model. The mean communication score for those who wanted to speak to a nurse and did the same was 90.0 out of 100.0. For all other combinations nurse–patient communication scores were, on average, lower (p < 0.001). This difference was largest for those wanting to see or speak to a GP who then saw or spoke to a nurse, with the lowest scores for those who wanted to see a GP but spoke to a nurse (adjusted difference vs. those who wanted to see a nurse and did see a nurse –10.5, 95% CI –11.7 to –9.2).
| Wanted to happen | Actually happened | n (%) | Mean communication Score | Crude differencea (95% CI) | Adjusted differenceb (95% CI) |
|---|---|---|---|---|---|
| See a nurse | Saw a nurse | 105,140 (86.8) | 90.0 | Reference | Reference |
| See a nurse | Spoke to a nurse | 517 (0.4) | 88.3 | –1.70 (–2.97 to –0.44) | –1.40 (–2.63 to –0.18) |
| Speak to a nurse | Saw a nurse | 1170 (1.0) | 87.6 | –2.37 (–3.21 to –1.53) | –2.05 (–2.87 to –1.23) |
| Speak to a nurse | Spoke to a nurse | 1697 (1.4) | 88.9 | –1.09 (–1.79 to –0.38) | –0.93 (–1.61 to –0.24) |
| See a GP | Saw a nurse | 10,916 (9.0) | 82.7 | –7.31 (–7.60 to –7.02) | –5.99 (–6.28 to –5.71) |
| See a GP | Spoke to a nurse | 538 (0.4) | 77.8 | –12.16 (–13.40 to –10.92) | –10.47 (–11.68 to –9.27) |
| Speak to a GP | Saw a nurse | 819 (0.7) | 85.7 | –4.25 (–5.26 to –3.25) | –3.35 (–4.33 to –2.38) |
| Speak to a GP | Spoke to a nurse | 289 (0.2) | 80.2 | –9.80 (–11.49 to –8.11) | –8.66 (–10.30 to –7.03) |
The parallel analysis for those patients who saw or spoke to a GP is shown in Table 12. This analysis is restricted to the 497,302 patients who reported seeing or speaking to a doctor on their last visit to the GP surgery and also have complete information to the covariates used in the adjusted model. The mean communication score for those who wanted to speak to a doctor and did so was 85.4 out of 100. For the majority of the remaining categories, doctor–patient communication scores were, on average, lower (p < 0.001). The differences were small in most cases (1 to 2 points), although they were greatest when the patient wanted to see a GP and ended up speaking to either a GP or a nurse on the telephone.
| Wanted to happen | Actually happened | n (%) | Mean communication score | Crude differencea (95% CI) | Adjusted differenceb (95% CI) |
|---|---|---|---|---|---|
| See a GP | Saw a GP | 450,555 (90.6) | 85.4 | Reference | Reference |
| See a GP | Spoke to a GP | 9127 (1.8) | 80.0 | –5.41 (–5.77 to –5.05) | –4.49 (–4.84 to –4.14) |
| Speak to a GP | Saw a GP | 8281 (1.7) | 85.2 | –0.21 (–0.59 to 0.17) | –0.23 (–0.59 to 0.13) |
| Speak to a GP | Spoke to a GP | 22,039 (4.4) | 86.9 | 1.54 (1.30 to 1.77) | 0.96 (0.73 to 1.19) |
| See a nurse | Saw a GP | 5831 (1.2) | 84.5 | –0.85 (–1.30 to –0.40) | –1.22 (–1.65 to –0.80) |
| See a nurse | Spoke to a GP | 378 (0.1) | 82.0 | –3.41 (–5.17 to –1.64) | –3.76 (–5.43 to –2.10) |
| Speak to a nurse | Saw a GP | 913 (0.2) | 84.3 | –1.07 (–2.21 to 0.07) | –0.94 (–2.01 to 0.13) |
| Speak to a nurse | Spoke to a GP | 178 (0.0) | 83.7 | –1.64 (–4.21 to 0.93) | –2.16 (–4.59 to 0.26) |
Conclusions
Our analyses show that most patients have a particular GP who they prefer to see. It is sometimes suggested that this is important for only some population groups (e.g. not for young people) but we found that this is not the case. Even among those aged 18–24 years, > 50% of respondents to the GP Patient Survey have a particular doctor who they prefer to see, which rises to > 80% in those aged > 75 years. Disturbingly, a large percentage of people who have such a preference are unable to see the doctor of their choice. This percentage has risen from 30% to 39% between 2010 and 2015. We can only speculate on the reasons for this, with the rise likely to be the result of a range of factors including the pressure on GPs to increase access by offering same-day appointments and by opening for longer hours or on more days. In addition, the increasing proportion of GPs working part-time may make it more difficult for patients to see the GP of their choice.
One of the criticisms of patient surveys is the very positive scores that patients give, scores that may not represent the totality of their experience (as we have shown in Chapters 2 and 3). However, we do see less positive scores for ratings of being able to see a doctor of your choice, with 40% of patients responding to the GP Patient Survey now saying that they are regularly unable to see the doctor of their choice. This is clearly an important quality issue for the NHS that has received scant attention from governments, which remain focused on access. Providing good continuity is difficult in the context of contemporary general practice, but there are ways of organising practice to increase patients’ ability to choose the doctor who they see. The Royal College of General Practitioners has published a toolkit on the subject144 and we have also published guidelines on how practices can improve the continuity of care that they provide. 145 This is certainly an area that deserves more priority in the NHS.
We are able to get some insight into the impact of this on the patient experience from our analysis of data from patients who have seen a nurse when they had originally wanted to see a doctor. These patients report a substantially worse experience with their subsequent consultation with a nurse and we have no reason to think that the nurses were behaving differently towards these patients compared with any other patients (and nurses generally get very high scores for their communication with patients).
Section B Understanding patient experience in minority ethnic groups
Chapter 5 Analyses of GP Patient Survey data to explore variations in patient experience by ethnic group and practice
Abstract
Background
In the UK there is particular concern over South Asian patients’ experience of care, with consistently more negative ratings across a wide range of measures. The nature and potential drivers of the reported variations in care in South Asian groups have yet to be fully explored. In this workstream we aimed to investigate a number of potentially contributory factors to variations in communication with primary care professionals related to ethnicity and practice.
Methods
This study involved analyses of data from the GP Patient Survey.
Results
South Asian respondents report more negative experiences of GP–patient communication than their white British counterparts. Around half of this variation may be attributed to the concentration of these patients in low-performing practices. However, the effect of ethnicity on reported GP–patient communication varies by age and gender, with poorer experience scores being particularly marked in older, female Asian patients. There was no evidence of differential item functioning (DIF) of the communication items for white British and South Asian patients. These findings increase the likelihood that there are true differences in the quality of care received by South Asian groups and the white British majority. A substantial proportion of the variability in practice scores for GP–patient communication can be explained by practice factors.
Conclusions
Reports of communication with primary care professionals are more negative for South Asian respondents. Although practice factors are an important driver of this, even within the same practice, South Asian patients (particularly those who are older and female) are likely to experience a lower quality of communication.
Introduction and rationale for the study
Systematic variations in experience of health care in relation to ethnicity, age, gender, health and socioeconomic status have long been documented in the UK. 146–148 In 2014, NHS England149 reiterated concerns about variations in the quality of primary care for disadvantaged groups, stating that ‘People have a right to high quality services, irrespective of who they are, their social status, where they live, or what needs they have’ (p. 9) (contains public sector information licensed under the Open Government Licence v3.0). A particular focus has been the experience of some minority ethnic groups, who have reported consistently lower patient experience scores than the majority population in both the UK and the USA. 75,150–153 Previous analyses of patient experience data conducted by the authors highlighted that South Asian patients reported particularly negative experiences, including for waiting times for GP appointments, time spent waiting in surgeries for consultations to start and continuity of care. 75
Several potential explanations have been proposed for the lower patient experience ratings given by South Asian patients in response to surveys. Broadly, these relate to whether or not South Asian patients receive lower-quality care or whether or not they receive similar care but rate this more negatively. 75,80,154 For example, differences in the use of questionnaire response scales80 may lead to South Asian groups being less likely to endorse the most positive options when asked to evaluate a doctor’s communication skills. Alternatively, there may be systematic variations in evaluations of consultations because South Asian respondents vary in their expectations of, or preferences for, care. Finally, of course, it is possible that reported poorer experiences of care do reflect actual differences in the care received by these patient groups. In this workstream, we set out to explore in more detail the nature and potential drivers of the reported variations in care in South Asian groups, using existing GP Patient Survey data. Experimental work to explore how South Asian and white British participants rate simulated consultations is detailed in Chapter 6.
Structure of the work package
We undertook a series of analyses of GP Patient Survey data to investigate variations in patient experience for South Asian groups. This work was undertaken across four workstreams:
-
an exploration of whether or not the low scores of minority ethnic and other sociodemographic groups reflect their concentration in poorly performing primary care practices
-
building on the above, further analyses to determine how reported GP–patient communication varies between patients from different ethnic groups, stratified by age and gender
-
an analysis, using item response theory, to test for evidence of whether or not the GP Patient Survey communication items perform differently for South Asian and white British respondents
-
finally, in addition to the above patient-level analyses, we explored how differences between practices influence GP–patient communication scores.
Changes to study methods from the original protocol
The aim of this workstream, as stated in the original protocol, was to understand the reasons why minority ethnic groups, especially South Asians, give lower scores on patient surveys than the white British population (aim 5).
We conducted all analyses outlined in the original protocol. However, workstream 2, an exploration of how reported GP–patient communication varies by ethnicity stratified by age and gender, was an additional analysis undertaken to gain better insight into the particular combinations of patient characteristics associated with the most negative reported experiences of care.
Background to the GP Patient Survey
The GP Patient Survey was started in 2007 as a national postal survey of primary care patients. Each year it takes a random sample of patients registered at all NHS primary care practices and sends out a questionnaire covering key aspects of patient experience, including access, waiting times and communication with doctors and nurses. Findings from the survey are disseminated widely and are available to practitioners and patients through the dedicated GP Patient Survey website. 11 In 2014/15, a questionnaire was sent to 2.6 million patients, of whom 858,381 responded (a 32.5% response rate). Respondents may complete the survey by post or online, including in British Sign Language, and in 13 languages other than English, either online or by telephone.
The original GP Patient Survey questionnaire was developed iteratively, with guidance from stakeholders and experts, cognitive testing of items and extensive piloting. 130 It has been further developed over the years, with changes to the content and technical aspects including survey weighting.
Workstream 1: do poor patient experience scores of minority ethnic groups reflect their concentration in poorly performing primary care practices?
Aims and objectives
The aim of this workstream was to investigate the causes of sociodemographic variations in patient experience. There were two specific objectives:
-
Do minority ethnic group differences in reported GP–patient and nurse–patient communication arise from the concentration of minority ethnic patients in practices with lower than average performance?
-
Do minority ethnic group differences in reported GP–patient and nurse–patient communication vary substantially across practices?
Methods
We analysed data from the 2009/10 GP Patient Survey. 11 Drawing on our previous principal components analysis of survey data, we constructed a measure of reported GP–patient and nurse–patient communication from seven communication items (Table 13). 130 From these, we created a composite score for all responders who provided three or more informative responses; this was derived by linear rescaling of the responses between 0 and 100 and taking the mean of all subitems answered.
| Last time you saw or spoke to a GP/nurse from your GP surgery, how good was that GP/nurse at each of the following? Please put a ✗ in one box for each row | Very good | Good | Neither good nor poor | Poor | Very poor | Doesn’t applya |
|---|---|---|---|---|---|---|
| Giving you enough time | □ | □ | □ | □ | □ | □ |
| Asking about your symptoms | □ | □ | □ | □ | □ | □ |
| Listening to you | □ | □ | □ | □ | □ | □ |
| Explaining tests and treatments | □ | □ | □ | □ | □ | □ |
| Involving you in decisions about your care | □ | □ | □ | □ | □ | □ |
| Treating you with care and concern | □ | □ | □ | □ | □ | □ |
| Taking your problems seriously | □ | □ | □ | □ | □ | □ |
Patient-reported age group, gender, ethnicity, self-rated health and presence of a long-standing psychological or emotional condition were taken directly from survey responses. Socioeconomic status was measured using an area-based approach, the Index of Multiple Deprivation (IMD),155 based on the patients’ residential postcode. For analysis, we split the IMD into five groups, based on national quintiles.
To examine our first objective (to distinguish the effects of the concentration of some minority ethnic groups in low-scoring practices from the variation in scores of different population groups within practices), we combined two analytical strategies:
-
Fixed-effects multivariable linear regression models to predict patient experience measures only from patient sociodemographic characteristics. These models estimate overall sociodemographic differences in patient experience that arise both because some patient groups are concentrated in low-performing practices and because the scores of patients of different groups vary within the same practices.
-
Mixed-effects models that included patient sociodemographic variables as fixed effects plus a random effect (intercept) for practice. These models estimate only the sociodemographic differences that arise because the scores of patients from different groups vary within the same practices.
We used the difference between the coefficients of the first and the second models to indicate the amount of overall difference arising from the concentration of any given population group in practices with low scores.
To examine our second objective (to assess whether or not sociodemographic differences are consistent among practices) we added random effects (slopes) to the above models corresponding to the interaction of each patient characteristic variable with the ‘practice’ random effect (random slope random intercept models). From those models, using a normal approximation, we derived the ‘95% mid range of practice-level coefficients’ for each sociodemographic group, which indicates the range of practice-level sociodemographic differences within which 95% of all practices lie.
SAS 9.2 was used for random slope random intercept models and Stata 11 was used for all other analyses.
Results
There were 2,163,456 responses to the GP Patient Survey in 2009, representing an overall response rate of 38%. Table 14 shows the response by ethnic group.
| Ethnic group | Survey respondents, n | Percentage of survey respondents |
|---|---|---|
| White | ||
| White British | 1,718,133 | 82.0 |
| Irish | 29,930 | 1.4 |
| Any other white | 61,087 | 2.9 |
| Mixed | ||
| White and black Caribbean | 4549 | 0.2 |
| White and black African | 2825 | 0.1 |
| White and Asian | 4142 | 0.2 |
| Any other mixed | 3564 | 0.2 |
| South Asian | ||
| Indian | 53,484 | 2.6 |
| Pakistani | 33,517 | 1.6 |
| Bangladeshi | 10,974 | 0.5 |
| Any other Asian | 14,930 | 0.7 |
| Black | ||
| Black Caribbean | 25,231 | 1.2 |
| Black African | 28,349 | 1.4 |
| Any other black | 4174 | 0.2 |
| Chinese | 9759 | 0.5 |
| Other ethnic group | 90,644 | 4.3 |
The reported experiences of GP–patient communication by Bangladeshi, Pakistani, Indian and Chinese respondents were 9, 7, 6 and 8 percentile points more negative, respectively, than those by white British patients (Table 15).
| Ethnic group | Overall difference (SE)a | Difference (SE) attributable to different evaluations of care within the same practicea | Difference attributable to concentration of different patient groups in practices with different mean scores | Percentage of overall difference attributable to patient group concentration in practices with different mean scores |
|---|---|---|---|---|
| White | ||||
| White British | Reference | |||
| Irish | –0.2 (0.141) | 0.6 (0.138) | –0.8 | 353b |
| Any other white | –4.1 (0.096) | –3.2 (0.094) | –0.9 | 22 |
| Mixed | ||||
| White and black Caribbean | –1.9 (0.355) | –0.8* (0.346) | –1.1 | 56 |
| White and black African | –3.5 (0.447) | –1.9 (0.435) | –1.6 | 46 |
| White and Asian | –3.4 (0.358) | –2.2 (0.348) | –1.1 | 33 |
| Any other mixed | –4.7 (0.405) | –3.3 (0.394) | –1.4 | 31 |
| South Asian | ||||
| Indian | –6.1 (0.101) | –3.2 (0.109) | –3.0 | 48 |
| Pakistani | –7.2 (0.132) | –3.8 (0.145) | –3.4 | 48 |
| Bangladeshi | –8.6 (0.233) | –5.3 (0.242) | –3.4 | 39 |
| Any other Asian | –4.3 (0.194) | –2.1 (0.192) | –2.2 | 51 |
| Black | ||||
| Black Caribbean | –2.7 (0.155) | –0.5 (0.156) | –2.2 | 82 |
| Black African | –2.6 (0.143) | –0.2** (0.144) | –2.4 | 94 |
| Any other black | –2.0 (0.405) | –0.2*** (0.394) | –1.8 | 89 |
| Chinese | –8.3 (0.230) | –7.2 (0.225) | –1.1 | 14 |
| Other ethnic group | –4.7 (0.081) | –3.2 (0.081) | –1.5 | 32 |
There were similar findings for nurse–patient communication (Table 16).
| Ethnic group | Overall difference (SE) | Difference (SE) attributable to different evaluations of care within the same practice | Difference attributable to concentration of different patient groups in practices with different mean scores | Percentage of overall difference attributable to patient group concentration in practices with different mean scores |
|---|---|---|---|---|
| White | ||||
| White British | Reference | |||
| Irish | –0.5 (0.168) | 0.4 (0.166) | –0.9 | > ±100a |
| Any other white | –3.2 (0.118) | –2.4 (0.117) | –0.8 | 25 |
| Mixed | ||||
| White and black Caribbean | –1.7 (0.446) | –0.8 (0.439) | –0.9 | 51 |
| White and black African | –4.0 (0.570) | –2.5 (0.561) | –1.5 | 38 |
| White and Asian | –4.2 (0.459) | –3.3 (0.452) | –0.9 | 22 |
| Any other mixed | –4.1 (0.507) | –2.6 (0.499) | –1.5 | 37 |
| South Asian | ||||
| Indian | –7.2 (0.123) | –5.1 (0.134) | –2.1 | 29 |
| Pakistani | –7.8 (0.165) | –5.9 (0.179) | –1.9 | 24 |
| Bangladeshi | –9.6 (0.309) | –7.3 (0.319) | –2.3 | 24 |
| Any other Asian | –6.2 (0.244) | –4.3 (0.244) | –1.9 | 30 |
| Black | ||||
| Black Caribbean | –3.4 (0.192) | –1.4 (0.195) | –2.0 | 60 |
| Black African | –4.0 (0.1481) | –1.4 (0.184) | –2.2 | 55 |
| Any other black | –3.6 (0.492) | –1.9 (0.485) | –1.7 | 47 |
| Chinese | –9.4 (0.314) | –8.3 (0.311) | –1.2 | 12 |
| Other ethnic group | –4.7 (0.101) | –3.3 (0.102) | –1.4 | 29 |
Our first objective in this strand of work was to examine whether or not such overall minority ethnic differences in experiences of care arise from the concentration of these patients in practices with lower than average performance. By comparing the coefficients obtained from the fixed- and mixed-effects model for GP–patient communication, we identified that the concentration of minority ethnic groups in low-scoring practices was responsible for about 50% of the difference between South Asian and white British patients. However, even after accounting for the effect of the concentration of these groups in practices with lower scores, South Asian patients reported more negative experiences of care than their white British counterparts within the same practice.
Our second objective was to examine whether or not minority ethnic differences varied between as well as within practices. Table 17 shows that within-practice ethnic group differences in reported GP–patient communication varied substantially across practices, alongside other key measures of patient experience. On average, South Asian patients evaluated doctor communication more negatively than white British patients (–4 percentile points); however, in some practices South Asian patients reported more positive experiences of GP–patient communication than their white British counterparts (95% practice mid range for differences in doctor communication: –13 to +4 percentile points; positive values indicate a better patient experience than in the majority white British group). Again, we found a similar picture for nurse–patient communication (Table 18).
| Ethnic group | Mean difference in GP–patient communication scoresa | 95% mid range of practice differences |
|---|---|---|
| South Asianb | –4.3 | –12.6 to 4.0 |
| Blackb | –1.4 | –7.9 to 5.0 |
| Chineseb | –8.5 | –18.3 to 1.3 |
| Mixedb | –3.9 | –16.1 to 8.2 |
| Otherb | –4.3 | –11.7 to 3.1 |
| Ethnic group | Mean difference in nurse–patient communication scoresa | 95% mid range of practice differences |
|---|---|---|
| South Asianb | –5.9 | –14.0 to 2.1 |
| Blackb | –2.2 | –8.3 to 3.8 |
| Chineseb | –9.2 | –23.6 to 5.2 |
| Mixedb | –3.4 | –18.2 to 11.5 |
| Otherb | –3.9 | –12.0 to 4.3 |
Summary
This analysis of GP Patient Survey data confirmed that South Asian respondents report substantially more negative experiences of patient communication than their white British counterparts. Around half of this was due to the concentration of these patients in low-performing practices. However, differences in reported experiences also varied substantially between practices: as well as more negative reports of care, in some practices South Asian patients evaluated their experience similarly or more positively than their white British counterparts.
These findings suggest that there may be a number of drivers behind the more negative reports of GP–patient communication seen in national patient experience surveys. However, the experimental vignette work that we conducted (see Chapter 6) enabled us to determine more clearly where the most plausible explanations lie.
Workstream 2: how does reported general practitioner–patient communication vary between patients from different ethnic groups, stratified by age and gender?
This material is reproduced from Burt J, Lloyd C, Campbell J, Roland M, Abel G. Variations in GP–patient communication by ethnicity, age, and gender: evidence from a national primary care patient survey. Br J Gen Pract 2016;66:e47–52156 under the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.
Aims
Although our earlier analyses confirmed the variation in reported experience between minority ethnic groups within the GP Patient Survey, the question remained whether or not more negative experiences of care are consistent across respondents within a particular ethnic group. Recently, interactions between age and ethnicity have been identified for patient reports of the number of GP consultations that take place before hospital referral for cancer. 157 To explore whether or not such interactions exist for our focus area of patient experience, GP–patient communication, we undertook further analysis of GP Patient Survey data to determine how reported GP–patient communication varies between patients from different ethnic groups by age and gender.
Methods
We analysed data from the 2012/13 and 2013/14 GP Patient Survey. 11 By combining data from 2 years of the survey, we were able to increase the number of responses available for analysis from small ethnic groups. No patient receives the survey in 2 consecutive years so there was no risk of double counting respondents.
Following the same approach outlined earlier in this chapter, we constructed a measure of reported GP–patient communication from five of the communication items used in the most recent GP Patient Surveys; these were reduced from the seven used in earlier questionnaires (Table 19). 130
| Last time you saw or spoke to a GP/nurse from your GP surgery, how good was that GP/nurse at each of the following? Please put a ✗ in one box for each row | Very good | Good | Neither good nor poor | Poor | Very poor | Doesn’t applya |
|---|---|---|---|---|---|---|
| Giving you enough time | □ | □ | □ | □ | □ | □ |
| Listening to you | □ | □ | □ | □ | □ | □ |
| Explaining tests and treatments | □ | □ | □ | □ | □ | □ |
| Involving you in decisions about your care | □ | □ | □ | □ | □ | □ |
| Treating you with care and concern | □ | □ | □ | □ | □ | □ |
We created a composite score for all responders who provided three or more informative responses. This was derived by linear rescaling of the responses between 0 and 100 and taking the mean of all sub-items answered. Patient-reported age group, gender and ethnicity were taken directly from survey responses. Health-related quality of life was measured using responses to five questions that make up the EuroQol-5 Dimensions three-level version (EQ-5D-3L) descriptive system. 158 Socioeconomic status was measured using an area-based approach, the IMD, based on patients’ residential postcode. 155 For analysis, we split the IMD into five groups, based on national quintiles.
We used a mixed-effect linear regression model with GP–patient communication score as the outcome. The model included age, gender, ethnicity, EQ-5D score and deprivation as fixed effects, as well as a random effect (intercept) for practice to account for the fact that certain patient groups cluster in practices that may perform better or worse overall. We included in the model all possible two-way interactions between age, gender and ethnicity, as well as the three-way interaction between them, to allow the effect of ethnicity to vary between different age and gender groups. We used Wald tests of the interaction terms to assess evidence supporting this variation. We then used the models to estimate age- and gender-specific differences between white British patients and patients of the same age and gender from each of the other ethnic groups. All analyses were carried out using Stata 13.1.
Results
There were 1,874,589 responses to the GP Patient Survey across 2012/13 and 2013/14, representing an overall response rate of 35%. Of these responses, 1,599,801 (85%) had complete data for all items included in our analysis. Table 20 shows the numbers of respondents in each ethnicity group. The largest group of responders were white British (n = 1,323,621, 82%), although there were at least 1800 responders in all but one group (that of gypsy or Irish traveller). Figure 10 shows the age composition of each ethnic group. White British and white Irish responders tended to be older than those from other ethnic groups and are dominated by those aged ≥ 55 years. In contrast, for nearly all other ethnicities the majority of responders were aged < 45 years. We therefore had very few responses in the oldest age groups (particularly those aged ≥ 85 years) for a number of ethnicities (see Table 20 for details).
| Ethnicity | All ages, n (%) | Aged ≥ 85 years, n (%) |
|---|---|---|
| White | ||
| British | 1,323,621 (82.7) | 49,891 (93.1) |
| Irish | 16,330 (1.0) | 662 (1.2) |
| Gypsy or Irish traveller | 401 (0.0) | 6 (0.0) |
| Any other white | 71,105 (4.4) | 1386 (2.6) |
| Mixed/multiple ethnic groups | ||
| White and black Caribbean | 3413 (0.2) | 26 (0.0) |
| White and black African | 1865 (0.1) | 4 (0.0) |
| White and Asian | 3171 (0.2) | 18 (0.0) |
| Any other mixed | 3340 (0.2) | 15 (0.0) |
| Asian/Asian British | ||
| Indian | 38,705 (2.4) | 425 (0.8) |
| Pakistani | 20,729 (1.3) | 143 (0.3) |
| Bangladeshi | 6699 (0.4) | 23 (0.0) |
| Chinese | 7986 (0.5) | 66 (0.1) |
| Any other Asian | 19,812 (1.2) | 105 (0.2) |
| Black/African/Caribbean/black British | ||
| African | 21,131 (1.3) | 24 (0.0) |
| Caribbean | 13,715 (0.9) | 275 (0.5) |
| Any other black | 6061 (0.4) | 52 (0.1) |
| Other ethnic group | ||
| Arab | 2786 (0.2) | 16 (0.0) |
| Other | 38,931 (2.4) | 458 (0.9) |
| Total | 1,599,801 (100.0) | 53,595 (100.0) |
FIGURE 10.
Age composition of responders according to self-reported ethnicity. Reproduced from Burt J, Lloyd C, Campbell J, Roland M, Abel G. Variations in GP–patient communication by ethnicity, age, and gender: evidence from a national primary care patient survey. Br J Gen Pract 2016;66:e47–52156 under the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.
From the regression model (adjusting for deprivation, EQ-5D score and practice) there was strong evidence (p < 0.001 for age*gender*ethnicity three-way interaction term) that the effect of ethnicity on reported GP–patient communication varied by both age and gender. 156 Figure 11 shows the age- and gender-specific adjusted differences between white British responders and responders of the same age and gender from all Asian subgroups and white (non-British) ethnic groups. Negative differences indicate that responders reported a worse experience than their white British counterparts (i.e. of the same age and gender). Again, as with our previous analyses, the largest differences were seen in Asian ethnic groups, followed by white (non-British) ethnic groups.
FIGURE 11.
Age- and gender-specific differences with 95% CIs in reported GP–patient communication scores (0–100 scale) between white British patients and responders in Asian and white (non-British) ethnic groups. Burt J, Lloyd C, Campbell J, Roland M, Abel G. Variations in GP–patient communication by ethnicity, age, and gender: evidence from a national primary care patient survey. Br J Gen Pract 2016;66:e47–52156 under the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.
Differences in reported experiences of GP–patient communication between Asian groups and the white British group were particularly large for older responders (those aged ≥ 55 years). This differential effect of ethnicity was particularly marked in Bangladeshi responders and in women (see Figure 11 for details). For example, the difference in reported experience scores between a white British women aged from 75 to 84 years and a Bangladeshi woman of the same age range was –8.23 points on the 0–100 scale (95% CI –12.76 to –3.69 points). However, the differences between Indian, Pakistani and Bangladeshi groups and white British responders among younger age groups were fairly small. For example, the difference in reported experience scores between a white British women aged from 35 to 44 years and a Pakistani woman of the same age range was –2.72 points (95% CI –3.42 to –2.02 points). For Chinese responders, substantial negative differences compared with white British counterparts were seen across all age groups.
In contrast to Asian responders, for those responders identified as ‘any other white’, ethnic variations in reported communication were largest for younger responders (those aged < 55 years). We found few differences in reported experience over all ages for African, Caribbean and other black responders. Because of the smaller sample sizes, our ability to detect differences for mixed ethnic groups was limited. However, we note that there were more substantial (and statistically significant) negative differences for other Asian women (at all ages) and for white and Asian women (particularly at older ages).
Summary
This analysis of GP Patient Survey data has shown that the effect of ethnicity on reported GP–patient communication varies by age and gender. In comparison to white British responders of the same age and gender, poorer experience scores for GP–patient communication are particularly marked in older, female Asian patients and in younger ‘any other white’ patients. This highlights the need to focus not just on ethnic background but on how this interacts with other patient characteristics such as age and gender in its association with more negative reported experiences of care.
Workstream 3: is there evidence that the GP Patient Survey communication items perform differently for South Asian and white British respondents?
This material is reproduced with permission from Setodji CM, Elliott MN, Abel G, Burt J, Roland M, Campbell J. Evaluating differential item functioning in the English general practice patient survey. Comparison of South Asian and white British subgroups. Med Care 2015;53:809–17. 159
Aims and objectives
As already outlined, observations of poorer reported experience for certain minority ethnic groups may be attributed either to variations in the way in which they rate their care or to variations in the care actually received. Item response theory is one approach to exploring whether observed differences in survey responses may be attributable to true differences in health care or to differences in responses. 160,161 The aim of this strand of work was to use item response theory modelling to test for evidence that the GP Patient Survey communication items perform differently for South Asian and white British respondents, after controlling for other sociodemographic characteristics.
Methods
We analysed data from the 2011/12 GP Patient Survey. 11 We restricted the analysis to patients who responded to items about experience with GP and nurse care and who self-reported white British (n = 818,219) or South Asian ethnicity (n = 54,832). As before, we used the five GP–nurse–patient communication items (giving enough time, listening, explaining tests and treatments, involving in decisions about care and treating with care and concern).
Item response theory approaches were used to test for DIF for white British and South Asian responses (i.e. whether or not white British and South Asian patients have different understandings and scaling of the survey items). We conducted separate analyses for the GP and nurse communication items. In item response theory models, items vary in ‘difficulty’ (the extent to which they are easy for providers to ‘pass’) and patients differ in ‘ability’ (true health-care experiences). Item response theory models also allow subgroups, such as ethnic subgroups, to differ in true experiences or scale use in a way that is uniform across all items that attempt to measure a single construct (such as patient experience). Differences between groups, also known as DIF,161–163 provide evidence that items are not equivalent in meaning across subgroups and an unmeasured dimension other than the intended construct may be influencing item responses. Ideally, item response theory models can rely on an unimpeachable anchor item164 that measures the same construct as the other items but which is known to be completely unaffected by factors other than true care. This is quite rare in practice and so the all-items method, also known as the Wald-2 equating method,165,166 is more commonly used, in which designated anchors are not required. This approach links the metric of the construct of interest (patient experience) across South Asian and white British patients simultaneously and then all item parameters (item difficulty and ability) are estimated using the linked construct but they are free to vary between groups, effectively allowing the assessment of whether or not the differences between the groups that are being compared are consistent across items. In this approach, inconsistent differences across items are taken as evidence of DIF.
The absence of evidence of DIF is not conclusive evidence of equivalence, as it may reflect lack of power or it may reflect differences in scale use or expectations that are uniform across items in a scale. Because it is often considered unlikely that scale use and expectations would have the same effects on different items, lack of evidence of DIF in a well-powered study such as the present study is often seen as suggestive that true differences play a non-negligible role in observed differences in mean scores.
Unidimensionality and differential item functioning analysis
Because of the large sample sizes (white British, n = 818,219; South Asian, n = 54,832), power to detect statistical significance for even very small differences with a classic chi-squared or Wald test was very high, even after a Benjamini–Hochberg adjustment for multiple comparisons. 166 Consequently, the root-mean-square error of approximation (RMSEA), a transformed Wald chi-squared statistic that measures the degree of misfit independently of sample size, was used for DIF inference. 167 The item response theory DIF analyses were conducted in flexMIRT version 2.0. 166 To assess the possibility that our inferences of white British/South Asian differences were biased by potential confounders such as age, gender, chronic conditions and quality of life we conducted an additional sensitivity analysis with a matched sample of 54,484 South Asian and 54,484 white British patients with exactly the same characteristics on these potential confounders. The few South Asian patients with no match (0.63%) were dropped from the analysis.
Results
Using the DIF all-other anchor selection method with the Wald-2 equating algorithm, we found no items with DIF. These results are shown in Table 21 for both the full and the matched sample of patients. The RMSEA fit statistic was < 0.0085 for all GP items and < 0.0140 for all nurse items, suggesting the absence of DIF. In general, discrimination parameters typically range from 0.5 to 2, with higher values indicating items that better discriminate between levels of the latent construct,169 in this case patient experience. In this study, all item discrimination parameters (‘a’ in Table 21) exceeded 4.4, showing that all items are highly related to the overall score within the GP or nurse item set. The item difficulty parameters, which indicate the level of patient experience, θ, at which an item has a 50% chance of endorsement, would typically fall between –2 (2.5th percentile) and +2 (97.5th percentile). 170 In this study, they ranged from –2.62 to 0.03 for GP items and from –2.80 to –0.01 for nurse items (columns b2–b5 in Table 21), indicating that the scales best measure average and below-average experiences. They also indicate that a merely average patient experience (θ) results in a 50% chance of endorsing the highest response of ‘very good,’ consistent with the high numbers of patients endorsing the ‘very good’ response options.
| Items | Ethnicity | GP items | Nurse items | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Difficulty parameters | a | RMSEA | Difficulty parameters | a | RMSEA | ||||||||
| b2 | b3 | b4 | b5 | b2 | b3 | b4 | b5 | ||||||
| Full sample | |||||||||||||
| 1. Provider giving you enough time | South Asian | –2.44 | –2.03 | –1.39 | –0.18 | 4.82 | 0.0077 | –2.65 | –2.24 | –1.57 | –0.28 | 5.90 | 0.0084 |
| White British | –2.62 | –2.11 | –1.40 | –0.21 | 4.50 | –2.80 | –2.33 | –1.62 | –0.36 | 5.47 | |||
| 2. Provider listening to you | South Asian | –2.43 | –1.99 | –1.43 | –0.25 | 6.25 | 0.0068 | –2.62 | –2.20 | –1.56 | –0.30 | 7.86 | 0.0082 |
| White British | –2.44 | –1.96 | –1.38 | –0.26 | 6.82 | –2.66 | –2.19 | –1.48 | –0.30 | 8.11 | |||
| 3. Provider explaining tests and treatments | South Asian | –2.44 | –2.02 | –1.33 | –0.15 | 5.39 | 0.0083 | –2.63 | –2.26 | –1.50 | –0.26 | 6.36 | 0.0098 |
| White British | –2.57 | –2.08 | –1.29 | –0.15 | 5.00 | –2.76 | –2.30 | –1.44 | –0.26 | 5.71 | |||
| 4. Provider involving you in decisions about your care | South Asian | –2.34 | –1.88 | –1.16 | 0.02 | 5.15 | 0.0094 | –2.58 | –2.15 | –1.33 | –0.11 | 5.84 | 0.0147 |
| White British | –2.46 | –1.94 | –1.11 | 0.02 | 4.85 | –2.69 | –2.20 | –1.22 | –0.07 | 5.12 | |||
| 5. Provider treating you with care and concern | South Asian | –2.32 | –1.91 | –1.27 | –0.11 | 5.93 | 0.0068 | –2.56 | –2.17 | –1.47 | –0.22 | 6.52 | 0.0069 |
| White British | –2.39 | –1.93 | –1.24 | –0.15 | 6.07 | –2.59 | –2.18 | –1.49 | –0.28 | 6.15 | |||
| Matched sample for sensitivity analysis | |||||||||||||
| 1. Provider giving you enough time | South Asian | –2.28 | –1.87 | –1.24 | –0.04 | 4.86 | 0.0153 | –2.54 | –2.13 | –1.47 | –0.20 | 6.02 | 0.0184 |
| White British | –2.44 | –1.94 | –1.25 | –0.07 | 4.44 | –2.68 | –2.20 | –1.53 | –0.25 | 5.28 | |||
| 2. Provider listening to you | South Asian | –2.27 | –1.83 | –1.28 | –0.12 | 6.36 | 0.0083 | –2.50 | –2.09 | –1.45 | –0.21 | 8.04 | 0.0121 |
| White British | –2.28 | –1.79 | –1.24 | –0.12 | 6.70 | –2.54 | –2.07 | –1.39 | –0.22 | 7.99 | |||
| 3. Provider explaining tests and treatments | South Asian | –2.27 | –1.86 | –1.18 | –0.02 | 5.46 | 0.0187 | –2.52 | –2.15 | –1.40 | –0.17 | 6.42 | 0.0207 |
| White British | –2.41 | –1.91 | –1.15 | –0.01 | 4.89 | –2.66 | –2.18 | –1.37 | –0.18 | 5.63 | |||
| 4. Provider involving you in decisions about your care | South Asian | –2.18 | –1.73 | –1.01 | 0.15 | 5.15 | 0.0203 | –2.47 | –2.04 | –1.22 | –0.02 | 5.90 | 0.0318 |
| White British | –2.31 | –1.78 | –0.97 | 0.13 | 4.75 | –2.60 | –2.10 | –1.14 | –0.01 | 5.05 | |||
| 5. Provider treating you with care and concern | South Asian | –2.16 | –1.75 | –1.11 | 0.03 | 5.98 | 0.0108 | –2.44 | –2.06 | –1.37 | –0.14 | 6.63 | 0.0147 |
| White British | –2.21 | –1.76 | –1.10 | –0.01 | 6.16 | –2.44 | –2.05 | –1.40 | –0.20 | 6.32 | |||
Figure 12 illustrates the response curves for the parameters in Table 21. For each item, five response curves, each representing the probability of endorsing a specific category over the range of the underlying patient experience, showed no visual difference between South Asian and white patients, which is consistent with there being no meaningful DIF. The test characteristic curves depicting the expected scale scores of the GP and nurse items as a function of patient experience on the item response theory scale for the two groups also show no difference between the two groups (Figure 13).
FIGURE 12.
Full sample response curves for patient experience: (a) GP item 1: giving you enough time; (b) GP item 2: listening to you; (c) nurse item 2: listening to you; and (d) nurse item 4: involving you in care decisions. Plots for only two items each are reported, but all of the other GP/nurse items had similar response curves. Note: the x-axis is the item response theory parameter estimate (θ) for the construct (patient experience) on a z-score metric. The y-axis is the probability of endorsing each response option at given estimates of patient experience–quality level for South Asian and white British patients. Samples of 866,460 and 783,904 were used for the GP and the nurse survey items, respectively. IRT, item response theory. Reproduced with permission from Setodji CM, Elliott MN, Abel G, Burt J, Roland M, Campbell J. Evaluating differential item functioning in the English general practice patient survey. Comparison of South Asian and white British subgroups. Med Care 2015;53:809–17. 159
FIGURE 13.
Full sample test characteristic curves (TCCs): (a) TCC for GP questions; and (b) TCC for nurse questions. IRT, item response theory. Reproduced with permission from Setodji CM, Elliott MN, Abel G, Burt J, Roland M, Campbell J. Evaluating differential item functioning in the English general practice patient survey. Comparison of South Asian and white British subgroups. Med Care 2015;53:809–17. 159
Summary
These analyses found no evidence of meaningful DIF for white British and South Asian patients on GP–patient and nurse–patient communication items. These findings remained even after matching patients on a variety of sociodemographic characteristics. We suggest that the lack of evidence of DIF may be consistent with either (1) there being no differences in expectations or scale use between white British and South Asian respondents or (2) there being differences in expectations and/or scale use between groups that were the same across all items. It is possible that similar differences in scale use may occur across all items, as the response scale and labels remain the same. It is somewhat less likely that there are differences in expectations which remain constant across items that vary in content. Although we cannot exclude other possibilities, these findings do increase the likelihood that there is a role for true differences in the quality of care received by South Asian groups in comparison with the white British majority.
Workstream 4: how do practice factors influence general practitioner–patient communication scores?
Aims and objectives
The previous work described in this chapter focused on patient-level factors that may influence reported patient experience to try and gain insight into what is driving differences between different patient groups. In workstream 1 we explored to what extent clustering of certain patient groups in practices with better or worse patient experience scores overall explained differences between groups. Here, we take this one step further and investigate the factors describing a practice that are associated with GP–patient communication scores. We considered three different categories of practice factors: (1) practice geography, (2) practice professional team and (3) practice population. In each case we looked at the differences in communication scores that are associated with these factors and how much of the between-practice variance is explained by them.
Methods
We analysed data from the 2009/10 GP Patient Survey. 11 We used the same composite outcome measure for GP–patient communication, using the seven communication items (see Table 13) and taking values between 0 and 100. Identical patient-level exposure variables were used. In addition, we made use of practice-level variables from a number of sources.
Practice geography factors
As a proxy for geographical region, we used the former strategic health authority (SHA) to which a practice belonged, of which there are 10. We defined a rurality classification [based on the Office for National Statistics (ONS) definitions171] according to the postcode of the practice. Both SHA and rurality were included with the GP Patient Survey data set.
Practice professional team factors
Here, we used data from the GP census 2009 to calculate for each practice (data provided directly by the Department of Health):
-
the number of GPs excluding trainees
-
the number of patients per full-time equivalent GP
-
the mean number of years since qualification of the GPs
-
the proportion of male GPs
-
the proportion of GPs who trained in the UK for their primary medical qualification.
Practice population factors
We calculated a score for socioeconomic deprivation for each practice by applying the 2007 lower super output area IMD proportionately to the practice population. 170 We used GP Patient Survey results to estimate the proportion of black, Asian, Chinese, mixed race and other non-white patients in each practice. Registered patient numbers, broken down by gender and age group, were provided by the NHS Information Centre and used to calculate the proportion of patients in each practice who were children (aged < 15 years) and the proportion of adult patients in the following age groups: 15–44, 45–64, 65–74, 75–84 and ≥ 85 years.
Starting from the random intercept model used in workstream 1 (including fixed effects for patient age, gender, deprivation, ethnicity and self-rated health and a random intercept for practice), we added practice-level variables for the factors described above. To facilitate comparison between different variables with different distributions and units we scaled all continuous variables (including proportions) such that a difference of 1 corresponded to the difference between the 95th and 5th percentile of the distribution for that variable. The corresponding coefficients from the regression model can be interpreted as the adjusted differences between practices at either end of the distribution, ignoring outliers. We estimated the amount of variance in practice scores attributable to single or multiple variables by comparing the variance of the random effect in a model containing no practice-level variables with the variance of the random effect in a second model containing a single or multiple practice-level variables. We subsequently estimated the amount of variance in practice scores uniquely attributable to single or multiple variables by comparing the variance of the random effect in a model containing all practice-level variables with the variance of the random effect in a second model with one or more practice-level variables omitted.
Results
Table 22 shows the results of the regression model for practice-level variables (patient-level variables are not shown but are consistent with those shown in workstream 1). In relation to practice team factors, practices with a large number of GPs, a high number of patients per full-time equivalent GP and doctors who, on average, completed training a longer time ago tended to have worse GP–patient communication scores. In comparison, those with a high proportion of GPs trained in the UK had better GP–patient communication scores. For geographical factors, practices in London and urban areas received the worst scores. Finally, in relation to practice population factors, those practices that served populations with relatively more men and Asian and black patients and patients aged < 85 years tended to have worse scores for GP–patient communication. It is worth noting that these population coefficients have been controlled for individual characteristics and so do not represent the fact that these patient groups score worse, but rather that practices who have more of these patient groups tend to have worse scores for all patients.
| Practice-level variables | Regression coefficient (95% CI) |
|---|---|
| Practice professional team | |
| GP team (95th vs. 5th percentile) | |
| Mean years since qualification | –1.5 (–1.9 to –1.1) |
| Proportion male GPs | 0.3 (0.0 to 0.6) |
| Proportion UK qualified | 4.0 (3.6 to 4.3) |
| Patients per FTE | –1.9 (–2.2 to –1.7) |
| Number of GPs | |
| 1 | Reference |
| 2 | –0.6 (–1.0 to –0.3) |
| 3 | –0.8 (–1.2 to –0.4) |
| 4 | –1.0 (–1.5 to –0.6) |
| 5 | –1.1 (–1.5 to –0.6) |
| 6–9 | –1.1 (–1.6 to –0.7) |
| ≥ 10 | –1.4 (–2.0 to –0.7) |
| Practice geography | |
| SHA (surrogate for region) | |
| North East | 3.1 (2.5 to 3.7) |
| North West | 2.9 (2.4 to 3.3) |
| Yorkshire and the Humber | 2.3 (1.8 to 2.7) |
| East Midlands | 1.5 (1.0 to 2.0) |
| West Midlands | 1.6 (1.1 to 2.0) |
| East of England | 1.2 (0.8 to 1.7) |
| London | Reference |
| South East Coast | 1.2 (0.7 to 1.6) |
| South Central | 2.0 (1.5 to 2.5) |
| South West | 2.4 (1.9 to 2.8) |
| Rurality of practice | |
| Urban > 10,000 – less sparse | Reference |
| Urban > 10,000 – sparse | –0.2 (–2.3 to 1.8) |
| Town and fringe | 0.1 (–0.3 to 0.4) |
| Village | 2.1 (1.5 to 2.6) |
| Hamlet and isolated dwellings | 1.0 (–0.2 to 2.3) |
| Practice population | |
| Proportion of patients who are (95th vs. 5th percentile) | |
| Female | Reference |
| Male | –1.5 (–1.9 to –1.2) |
| White | Reference |
| Mixed | –0.1 (–0.4 to 0.3) |
| Asian | –1.3 (–1.7 to –0.9) |
| Black | –0.7 (–1.0 to –0.3) |
| Chinese | 0.0 (–0.1 to 0.1) |
| Other | 0.8 (0.1 to 1.4) |
| Children aged < 15 years | –0.7 (–1.1 to –0.3) |
| Proportion of adult (age ≥ 15 years) patients who are aged (years) (95th vs. 5th percentile) | |
| 15–44 | –2.8 (–3.8 to –1.7) |
| 45–64 | Reference |
| 65–74 | –0.7 (–1.6 to 0.3) |
| 75–84 | –2.4 (–3.3 to –1.6) |
| ≥ 85 | 0.3 (–0.3 to 0.9) |
| Mean deprivation quintile | |
| 1 (least deprived) | 0.2 (–0.2 to 0.6) |
| 2 | 0.2 (–0.2 to 0.6) |
| 3 | –0.1 (–0.5 to 0.2) |
| 4 | –0.1 (–0.4 to 0.3) |
| 5 (most deprived) | Reference |
In total, this model explained 35.4% of the between-practice variance in GP–patient communication scores. Practice team factors explained 25.9% of the total practice-level variance and 11.5% was explained uniquely by practice team factors (i.e. could not be explained by other factors). The corresponding percentages for practice geography were 13.8% and 2.7%, respectively, and those for practice population were 18.3% and 3.3%, respectively. Practice team factors are therefore the most important in explaining GP–patient communication scores. Of the practice team factors, the proportion of GPs trained in the UK was the most important factor, with 5.4% of the total variance uniquely attributable to that variable.
Summary
This analysis demonstrates that a substantial proportion of the variability in practice scores for GP–patient communication can be explained by practice factors. Factors related to the practice professional team were most important, particularly how many of the GPs were trained in the UK. When a large proportion of GPs were trained outside of the UK, GP–patient communication scores were substantially lower. Although this association might represent the quality of training received in the UK, it is more likely to be a marker of a GP being of non-British ethnicity. Thus, factors that may drive the observed association could plausibly include language and cultural barriers or discrimination on behalf of the patients. In workstream 1 we demonstrated that around half of the difference between white British and Asian patients resulted from the clustering of Asian patients in practices with worse scores overall. This is consistent with our finding that practices in London and other urban areas (where minority ethnic patients would be expected to cluster) and those practices in which doctors trained overseas are focused have lower scores. Even so, after controlling for these factors, we still found that practices with high proportions of Asian patients had lower GP–patient communication scores. Interestingly, in other work we have found that, when South Asian patients attend a practice where consultations in a concordant South Asian language are offered, the difference between white British and South Asian patients in terms of GP–patient communication scores decreases. 172
Overall conclusions
Our analyses of recent GP Patient Survey data have confirmed the presence of substantially more negative experiences of communication in minority ethnic groups, including South Asian groups. A consideration of the interactions between ethnicity, age and gender highlighted that older, female Asian patients are particularly likely to report negative experiences of communication. Although a substantial proportion of these differences may reflect the concentration of such patients in low-performing practices, even within the same practices patients report substantial variations in communication. Our analyses further found no evidence that South Asian and white British groups exhibit differential response tendencies to communication items. Although experimental work is required to understand whether or not variations are indeed attributable to poorer quality of care, these findings point to this as the most plausible explanation of the identified differences.
South Asian patients may face a number of barriers to high-quality care, including poor language proficiency, lack of acculturation and provider-side discrimination. Our analysis of the association of practice factors, particularly the proportion of GPs trained outside of the UK, with reported experience of communication confirms, from a different perspective, the importance of language and cultural factors in determining the quality of communication.
Language is only one part of communication, but an important one. ‘Language discordance’ occurs when a doctor and patient do not share the same language. The inability to speak English well or at all varies widely between and within ethnic groups: 16.2% of Bangladeshi, 15.2% of Chinese, 12.2% of ‘any other white’ and 11.1% of Pakistani 2011 census respondents fell into this category. 173 Older Bangladeshi and Pakistani women may be prevented from acquiring English proficiency through family obligations or cultural and community expectations. 174 A number of studies have suggested that language discordance in clinical encounters may negatively impact on quality of care. 172,175–177 Challenges in communicating in language-discordant consultations can lead to particularly strong tensions between ‘ideal’ standards of communication and what is ‘good enough’. 178
Acculturation is concerned with the modification of attitudes or behaviours as people come into contact with a culture other than their own. Although its definition and scope are contested, it is frequently used to explain inequalities in health care. 179 Levels of acculturation may lead to variations in perceptions and expectations of providers and care, and ability to navigate the health-care system, impacting on reported experience. 180 Previous analysis of patient experience in a US primary care setting with Hispanic patients found no relationship between acculturation levels and patient reports of provider communication, although there was an association with other aspects of patient experience. 180 However, the measurement of acculturation through commonly used language proficiency scales has been criticised for failing to capture its multidimensional nature. 181 Further, a focus on lack of acculturation as a driver of disparities may mask other causal factors, including poverty, the social construction of ethnic identities and inequities in treatment. 182 Nevertheless, the broad concept of acculturation may be a useful reminder that age, gender and ethnicity groupings could vary in their understanding and navigation of primary care for reasons that are additional to those involving language barriers.
Concerns about institutionally ingrained variations in attitudes to patients on the basis of ethnicity have led to a rise in cultural competency training. 183,184 These approaches have been criticised for placing emphasis on patient characteristics as the drivers of variations in care, rather than on provider- and system-level factors including the potential for stereotyping of or bias towards particular groups. 185 However, our analysis shows that provider- or system-side factors do not occur in reaction to ethnicity alone, but in response to the inter-relationship between ethnicity, gender and age. It is the combination of these factors that may identify groups with particular needs, such as those patients with the lowest levels of English proficiency. We therefore need to focus not only on differences between groups but also on differences within them, considering how ethnicity, gender, age and other categories of social identity interact with each other to create different experiences and outcomes. The study of such interactions has been termed ‘intersectionality’. 186
Strengths and limitations
The GP Patient Survey data are derived from a large, randomly selected sample designed to be representative of patients registered with a practice in England or Wales. 113 However, response rates to the GP Patient Survey are low: for the years that we analysed, the response rate ranged from 34% to 38%, although recent reviews suggest that response rates are not a strong indicator of non-response bias in surveys that use probability sampling. 187 Unfortunately, because ethnicity is not extracted from medical records for those sent a questionnaire it is impossible to determine the response rates in different ethnic groups. However, it is known that respondents from output areas with increasing proportions of non-white people are less likely to respond. There remains the possibility that any differential response rates may introduce some bias that we are not able to allow for. If survey responders are more proficient in English, this may underestimate the communication difficulties experienced by certain minority ethnic groups.
Finally, as no objective measure of GP–patient communication exists for these data, our analyses are not able to provide insight into whether reported experience varies as a result of differences in actual experience or as a result of variations in expectations or survey response tendencies. For this, experimental approaches are required, as described in Chapter 6.
Implications for practice
The existence of marked differences in the experience of GP–patient communication underlines the need for a renewed focus on those groups at risk of poorer quality of care. For practitioners, an awareness of the particular difficulties and frustrations encountered on both sides in cross-cultural consultations is an important first step. For patients with limited English-language proficiency effective support for communication in the form of professional interpreters is important. 188 System-level as well as patient-targeted initiatives to improve health literacy are also likely to be important in reducing variations in care, although these inevitably require greater investment. 189
Chapter 6 How do white British and Pakistani patients rate communication during simulated general practitioner–patient consultations? Experimental vignette study
Parts of this chapter are based on Burt et al. 190 under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited (http://creativecommons.org/licenses/by/4.0/).
Abstract
Background
Although minority ethnic groups have consistently reported poorer care in patient surveys, it is not known whether this is because they receive worse care or because they respond differently to such surveys.
Methods
We conducted an experimental vignette study to investigate whether or not South Asian people rate simulated GP consultations differently from white British people. In total, 564 white British and 564 Pakistani adults were recruited using an in-home face-to-face approach. Trained fieldworkers completed computer-assisted personal interviews during which participants rated the communication within three video recordings of simulated GP–patient consultations. Consultations were shown in a random order, selected from a pool of 16. Mean differences in communication scores (on a scale of 0–100) between white British and Pakistani patients were estimated from linear regression.
Results
Pakistani participants, on average, scored consultations 9.8 points higher than white British participants (95% CI 8.0 to 11.7 points; p < 0.001) when viewing the same consultations. When adjusted for age, gender, deprivation, self-rated health and video, the difference increased to 11.0 points (95% CI 8.5 to 13.6 points; p < 0.001). The largest differences were seen in older participants (≥ 55 years) and when communication was scripted to be poor.
Conclusions
Substantial differences in ratings were found, with Pakistani respondents giving higher scores to videos showing the same care. If we take these findings at face value, they would suggest that the lower scores reported by Pakistani patients in national surveys such as the GP Patient Survey represent genuinely worse care.
Introduction and rationale for the study
As outlined in Chapter 5, some minority ethnic groups have reported consistently lower patient experience scores than the majority population in both the UK and the USA. 75,150–153 Of particular concern within the UK, and confirmed by the analyses undertaken for this programme grant, South Asian groups report significantly more negative experiences of GP–patient communication than their white British counterparts. 131,156 Potential explanations for these lower ratings focus on whether South Asian patients (1) receive lower quality care or (2) receive similar care but rate this more negatively. 75
A number of potential drivers of more negative ratings of similar standards of care exist. For example, it has been suggested that differences in the use of questionnaire response scales (e.g. Elliott et al. 80) may lead to South Asian groups being less likely to endorse the most positive options when asked to evaluate a doctor’s communication skills. Our analysis of GP Patient Survey data, drawing on item response theory to explore whether or not items receive systematically different responses from South Asian and white British groups, suggested that this was unlikely to be the case. 159 Yet there are also other, alternative drivers of poorer ratings of similar care, most notably that the evaluation of consultations by South Asian respondents is influenced by systematic variations in their expectations of, or preferences for, care.
Fundamentally, these concerns centre on a well-recognised and long-standing problem with surveys: that individuals may interpret and respond to the ‘same’ question in many different ways. 191 Potential solutions to this problem arose first within the field of political science, where the use of standardised scenarios, or vignettes, was proposed to evaluate the disparity in responses to survey items. 82 Such approaches are particularly relevant to understanding minority ethnic experiences: as already described, alongside potential variations in scale use by individuals from various ethnic backgrounds, we also need to consider systematic cultural variations in expectations of or preferences for care, as well as the potential for systematic variations in actual experience. A recent US study81 adopted King et al. ’s vignette methodology to examine the extent of cross-cultural incomparability in survey responses, using predominantly written vignettes. This online survey concluded that score variations observed on national surveys among African American, Latino and white respondents were likely to reflect true differences in real-life experiences, at least for items in the survey that used an ‘always to never’ response scale. 81
The aim of this strand of work was to build on previous vignette approaches to examine whether or not people from a Pakistani background rate the communication within simulated GP consultations differently from white British people. If these groups rate simulated consultations similarly when viewing identical video vignettes, then we would be able to conclude that it is more likely that the lower scores previously reported by South Asian respondents in national patient experience surveys reflect real differences in quality of communication within consultations.
Changes to study methods from the original protocol
This strand of work, as stated in the original protocol, formed part of our wider aim of exploring in more detail the experiences of minority ethnic groups, together with the GP Patient Survey analyses reported in Chapter 5: to understand the reasons why minority ethnic groups, especially South Asians, give lower scores on patient surveys than the white British population (aim 5).
In our original protocol, to undertake this study we envisaged developing a DVD containing short clips (3–4 minutes) of four simulated patient consultations and asking respondents to rate these using the GP–patient communication items of the GP Patient Survey. These DVDs would be sent out, with questionnaires and instructions, to patients registered with practices with a high proportion of South Asian patients. We suggested using SANGRA (South Asian names and group recognition algorithm)192 to identify South Asian patients. In practice, we first devised a more robust and efficient approach to recruiting participants, using targeted face-to-face recruitment in partnership with the market research agency, Ipsos MORI. This enabled us to effectively reach a rigorously sampled set of participants of known Pakistani ethnicity. Second, participants rated simulated consultations during face-to-face computer-assisted interviews conducted by trained fieldworkers. This enabled us to collect high-quality and consistent ratings of consultations. Our recruitment and rating approach is detailed in full in Methods.
As we acknowledged in our original protocol, the requirement of the vignettes approach to show identical consultations to all participants meant that all videos had to be in English. However, we had stated that, although we would therefore have to exclude patients who could not understand English, we would make study questionnaires and documentation available in four Asian languages. As we employed face-to-face computer-assisted interviews in the study, this requirement was no longer necessary once we had screened for those who were confident in their ability in spoken English. This therefore represents a further improvement on our original study design.
Methods
In this experimental vignette study we showed videos of simulated GP–patient consultations to white British and Pakistani respondents, who were asked to rate the quality of the communication within each consultation that they viewed. The study advisory group was particularly involved in consideration of the nature of the vignettes to be shown and the study materials.
Simulated consultations
To ensure generalisability and to avoid the chance inclusion of a characteristic or event that, unknown to us, might systematically be rated differently by the two participating ethnic groups, we produced a series of 16 vignettes. We set out to manipulate the vignettes on three key domains:
-
the presenting complaint depicted within each consultation
-
the quality of the communication within each consultation (poor or good)
-
the ethnic background of the actors playing the doctor and patient (South Asian or white British).
Published recommendations for the production of vignettes emphasise the importance of developing a valid script and considering how best to manipulate this on the domains of interest. 193 We therefore based our vignettes on real-life consultations that were video recorded as part of another workstream (the association between patients’, raters’ and GPs’ assessments of communication in a consultation, for which we recorded > 500 real-life consultations). We undertook an extensive process of script development, role playing and rating prior to filming the vignettes with professional actors (Figure 14).
FIGURE 14.
Development of the vignettes. Reproduced from Burt et al. 190 under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited (http://creativecommons.org/licenses/by/4.0/).
The vignettes that we produced covered four different clinical scenarios: persistent cough, perforated ear drum, painful elbow and generalised numbness. We developed two different scripts for each clinical scenario: one designed to illustrate poor communication by the doctor and one designed to illustrate good communication. We formulated ‘poor’ and ‘good’ standards of communication according to the GCRS. 126 This observer-rated measure of communication competence (derived from the widely used Calgary–Cambridge guide to the medical interview127,128) was developed as part of our workstream on patients’ and raters’ assessments of communication competence within a consultation. The GCRS instrument covers 12 domains including initiating the session, gathering information, building the relationship and achieving a shared understanding (see Appendix 1 for the full instrument). We then used both the ‘poor’ and the ‘good’ version of the four clinical scenarios to film two sets of vignettes. The first set of vignettes had white British actors playing the GP and the patient, whereas the second repeated the same scripts but with South Asian actors playing the GP and the patient. The GP role was acted throughout by either one white British or one South Asian actor; eight different actors (four white British and four South Asian) role-played patients, each participating in one clinical scenario. The final 16 videos were each scored by three trained clinical raters using the GCRS to assess communication quality in relation to professionally defined norms. 126 Mean GCRS scores for the ‘poor’ communication vignettes ranged from 0.6 to 2.4 (out of 10), whereas mean GCRS scores for the ‘good’ communication vignettes ranged from 5.1 to 8.4.
Data collection
Ipsos MORI fieldworkers conducted data collection in collaboration with our team. As per the original protocol, we aimed to recruit 1120 respondents, each of whom was asked to rate three simulated GP–patient consultations. Our original sample size calculation was based on data from the General Practice Assessment Questionnaire (which includes some identical items to those in the GP Patient Survey); we repeated this using more recent GP Patient Survey data. This confirmed that the inclusion of 560 Pakistani respondents and 560 white British respondents would give > 80% power to detect a 3.1-point difference (on a 0–100 scale) seen between these two groups after controlling for age, gender, deprivation, self-rated health and practice. As our analyses of GP Patient Survey data had identified that ethnic disparities were largest among older age groups, we set out to recruit equal numbers above and below the age of 55 years within each ethnic group. 156
Following consultation with Ipsos MORI, we used different recruitment strategies for the different ethnic groups. To recruit Pakistani respondents, we selected output areas (geographically confined areas of approximately 130 households) in which at least 35% of the population was identified as Pakistani in 2011 census data. 173 These were then ranked according to the proportion of the population aged > 50 years (the cut-off point of 50 years of age used for sampling reflects available census categories; for our recruitment we specifically used a cut-off point of 55 years of age). Trained fieldworkers then recruited participants within these areas using an in-home face-to-face approach, starting in the output areas with the highest proportion of residents aged > 50 years. Fieldworkers were also provided with one or two output areas neighbouring the area sampled and were able to recruit from these if necessary. Snowball recruitment (e.g. known neighbours suggested to fieldworkers) and additional household interviews were allowed.
To recruit white British participants, we first excluded output areas with low proportions of white British residents (< 90%) and residents aged > 50 years. The remaining output areas were ranked by social grade (the percentage of people who were social grade A/B according to 2011 census data194) and geography. Ipsos MORI then selected output areas to approach using proportional systematic sampling.
Fieldworkers screened potential participants for ethnicity (using the ONS 18-group categorisation143) and for English-language competency (using a screening question regarding self-reported confidence in understanding short videos in English). Eligible respondents who consented then completed a computer-administered personal interview during which the fieldworker used a standardised script. Each participant viewed three of the sixteen simulated consultation videos that we had produced. Following each video, the participant was asked to rate the consultation using five GP–patient communication items taken from the most recent national GP Patient Survey (Table 23). We assigned videos so that each participant saw three different presenting conditions (and, therefore, videos), with two of the videos featuring South Asian–South Asian and white British–white British ethnic GP–patient pairs and at least one of the videos for each condition featuring either the ‘good’ or ‘poor’ communication script. The selection of videos shown to each participant was such that approximately equal numbers of all possible combinations were used, given the restrictions that we have described. Participants also completed basic sociodemographic questions (age, self-rated health, whether or not born in the UK, language spoken most often at home). An area-based measure of socioeconomic deprivation (IMD) was recorded based on the participants’ postcode.
| Thinking about the doctor you have just seen in the video, how good was the doctor at each of the following? Please put a ✗ in one box for each row | Very good | Good | Neither good nor poor | Poor | Very poor | Doesn’t applya |
|---|---|---|---|---|---|---|
| Giving enough time | □ | □ | □ | □ | □ | □ |
| Listening to the patient | □ | □ | □ | □ | □ | □ |
| Explaining tests and treatments | □ | □ | □ | □ | □ | □ |
| Involving the patient in decisions about his or her care | □ | □ | □ | □ | □ | □ |
| Treating the patient with care and concern | □ | □ | □ | □ | □ | □ |
Analysis
As in our previous analyses of GP Patient Survey data, we scored each participant’s rating of each consultation by linearly scaling the response options between 0 (very poor) and 100 (very good) and averaging all informative answers when at least three of the five items were completed. We used linear regression to model the mean difference between white British and Pakistani participants’ ratings of GP–patient communication. We estimated the unadjusted difference in ratings as well as the difference adjusting for patient age, gender, self-rated health, deprivation and a set of 15 indicator variables for the video. We did not originally plan to conduct any analysis of interaction terms. However, the effect size found was much larger than that anticipated in our original power calculations and so we investigated interactions between participant ethnicity and the following variables:
-
relating to the video: ethnicity of GP/patient and quality of GP–patient communication
-
relating to the participant: age, gender and deprivation.
When modelling interactions, we used only variables for the video attributes, rather than using indicator variables for all videos. For interactions involving age, the oldest two age groups were combined and a continuous version of the age groups was used in the interaction term only. CIs and p-values were estimated using bootstrapping with 500 replications (given non-normal data), clustered by participant (with each participant supplying three communication scores). We conducted a sensitivity analysis that clustered the bootstrap resampling by output area rather than by participant to account for multiple sampling in households and small geographical areas; however, this made only trivial changes to standard errors and we consequently do not report this here.
Results
Participants
We recruited a total of 1128 participants: 564 (50%) self-identified as white British and 564 (50%) self-identified as Pakistani. The sociodemographic profile of the participants is shown in Table 24. Although the sampling restriction that half of participants in each group be aged ≥ 55 years increased the similarity of the groups’ age distribution, Pakistani participants were younger than the white British participants within the sampled age strata. Pakistani participants were also more likely to be male (58% vs. 45%), to be in fair or poor health (38% vs. 26%) and to live in the most deprived areas (82% vs. 14%). Figure 15 shows the geographical locations from where participants were recruited. White British participants were recruited from a wide range of geographical locations, whereas, as a result of our sampling approach, Pakistani participants were located from a small number of geographically confined locations. Between 202 and 222 participants scored each of the video vignettes for GP–patient communication (Table 25).
| Characteristic | All, n (%) | White British, n (%) | Pakistani, n (%) |
|---|---|---|---|
| Age (years) | |||
| 18–24 | 88 (7.8) | 40 (7.1) | 48 (8.5) |
| 25–34 | 154 (13.7) | 56 (9.9) | 98 (17.4) |
| 35–44 | 151 (13.4) | 70 (12.4) | 81 (14.4) |
| 45–54 | 175 (15.5) | 118 (20.9) | 57 (10.1) |
| 55–64 | 267 (23.7) | 94 (16.7) | 173 (30.7) |
| 65–74 | 179 (15.9) | 109 (19.3) | 70 (12.4) |
| 75–84 | 95 (8.4) | 63 (11.2) | 32 (5.7) |
| ≥ 85 | 19 (1.7) | 14 (2.5) | 5 (0.9) |
| Gender | |||
| Male | 583 (51.7) | 255 (45.2) | 328 (58.2) |
| Female | 545 (48.3) | 309 (54.8) | 236 (41.8) |
| Self-rated health | |||
| Excellent | 132 (11.7) | 82 (14.5) | 50 (8.9) |
| Very good | 289 (25.6) | 181 (32.1) | 108 (19.1) |
| Good | 348 (30.9) | 157 (27.8) | 191 (33.9) |
| Fair | 207 (18.4) | 86 (15.2) | 121 (21.5) |
| Poor | 152 (13.5) | 58 (10.3) | 94 (16.7) |
| Mean deprivation quintile | |||
| 1 (least deprived) | 108 (9.6) | 100 (17.7) | 8 (1.4) |
| 2 | 137 (12.1) | 137 (24.3) | 0 (0.0) |
| 3 | 122 (10.8) | 111 (19.7) | 11 (2.0) |
| 4 | 221 (19.6) | 138 (24.5) | 83 (14.7) |
| 5 (most deprived) | 540 (47.9) | 78 (13.8) | 462 (81.9) |
FIGURE 15.
Geographical locations of the census-based output areas from where participants were recruited: (a) white British participants; and (b) Pakistani participants. Reproduced from Burt et al. 190 under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited (http://creativecommons.org/licenses/by/4.0/).
| Video number | Clinical scenario | Scripted communication quality | Ethnicity of GP and patient | Number of times video scored |
|---|---|---|---|---|
| 1 | Persistent cough | Bad | White | 220 |
| 2 | Asian | 202 | ||
| 3 | Good | White | 202 | |
| 4 | Asian | 212 | ||
| 5 | Perforated ear drum | Bad | White | 210 |
| 6 | Asian | 206 | ||
| 7 | Good | White | 217 | |
| 8 | Asian | 207 | ||
| 9 | Painful elbow | Bad | White | 206 |
| 10 | Asian | 210 | ||
| 11 | Good | White | 210 | |
| 12 | Asian | 215 | ||
| 13 | Generalised numbness | Bad | White | 216 |
| 14 | Asian | 222 | ||
| 15 | Good | White | 212 | |
| 16 | Asian | 214 |
Main results
The distribution of communication scores for white British and Pakistani participants was skewed in both groups; however, the communication scores from Pakistani participants were typically higher than those from white British participants (Figure 16). The mean communication score from Pakistani participants was 67.3 out of 100, 9.9 points higher (95% CI 8.0 to 11.7 points; p < 0.001) than the mean score from white British participants (57.4 out of 100). In a regression model (full output shown in Table 26) adjusting for participant age, gender, self-rated health, deprivation and video there was a slightly larger difference between the two ethnicities of 11.0 points (95% CI 8.5 to 13.5 points; p < 0.001).
FIGURE 16.
Box plots showing the distribution of GP–patient communication scores recorded by white British and Pakistani participants. Reproduced from Burt et al. 190 under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited (http://creativecommons.org/licenses/by/4.0/).
| Characteristic | Adjusted difference (95% CI) | p-value |
|---|---|---|
| Ethnicity | ||
| White British | Reference | < 0.001 |
| Pakistani | 11.01 (8.53 to 13.49) | |
| Age (years) | ||
| 18–24 | –5.55 (–8.94 to –2.16) | < 0.001 |
| 25–34 | –4.96 (–7.99 to –1.93) | |
| 35–44 | –1.67 (–4.67 to 1.33) | |
| 45–54 | –1.86 (–4.60 to 0.87) | |
| 55–64 | Reference | |
| 65–74 | 4.01 (1.20 to 6.83) | |
| 75–84 | 6.70 (3.26 to 10.13) | |
| ≥ 85 | 3.66 (–3.66 to 10.97) | |
| Gender | ||
| Male | Reference | 0.115 |
| Female | 1.41 (–0.34 to 3.16) | |
| Self-rated health | ||
| Excellent | Reference | 0.866 |
| Very good | –1.15 (–4.05 to 1.74) | |
| Good | –1.65 (–4.71 to 1.41) | |
| Fair | –1.77 (–5.12 to 1.58) | |
| Poor | –1.41 (–5.21 to 2.38) | |
| Mean deprivation quintile | ||
| 1 (least deprived) | Reference | 0.505 |
| 2 | –0.92 (–4.10 to 2.27) | |
| 3 | 1.08 (–2.26 to 4.42) | |
| 4 | –1.45 (–4.57 to 1.68) | |
| 5 (most deprived) | 0.13 (–3.32 to 3.58) | |
| Video number | ||
| 1 | Reference | < 0.001 |
| 2 | –3.90 (–6.79 to –1.01) | |
| 3 | –56.51 (–60.51 to –52.50) | |
| 4 | –49.57 (–53.77 to –45.37) | |
| 5 | –4.09 (–7.06 to –1.12) | |
| 6 | –7.45 (–10.58 to –4.33) | |
| 7 | –48.08 (–51.81 to –44.34) | |
| 8 | –49.70 (–53.53 to –45.87) | |
| 9 | –3.24 (–6.33 to –0.14) | |
| 10 | –7.40 (–10.48 to –4.33) | |
| 11 | –52.19 (–56.03 to –48.34) | |
| 12 | –48.94 (–52.80 to –45.08) | |
| 13 | –9.59 (–12.89 to –6.29) | |
| 14 | –9.36 (–12.45 to –6.27) | |
| 15 | –54.23 (–58.07 to –50.38) | |
| 16 | –46.63 (–50.52 to –42.73) | |
Analysis of interactions
As the difference in scores between Pakistani and white British participants was considerably larger than that expected at the design stage, we were able to explore interactions between ethnicity and other variables. We found no evidence that the difference in scores between Pakistani and white British participants varied by patient gender (p = 0.92), deprivation (p = 0.68) or ethnicity of the doctor/patient actor pairing in the videos (p = 0.53). There was, however, strong evidence that the difference in scores between Pakistani and white British participants was larger for older participants (p = 0.001) and for consultations scripted to contain poorer levels of GP–patient communication (p < 0.001). Table 27 shows the mean difference in age by good/poor scripted communication strata, estimated from a model containing all of the main effects plus (1) ethnicity and age interactions, (2) ethnicity and good/poor communication interactions and (3) the three-way interaction between those variables (p < 0.001 for the three-way interaction).
| Age (years) | Scripted communication: adjusted difference (95% CI) | |
|---|---|---|
| Good | Poor | |
| 18–24 | –1.31 (–5.38 to 2.76) | 10.29 (5.00 to 15.57) |
| 25–34 | –0.15 (–3.58 to 3.27) | 13.32 (9.10 to 17.54) |
| 35–44 | 1.01 (–1.96 to 3.97) | 16.34 (12.91 to 19.77) |
| 45–54 | 2.17 (–0.62 to 4.95) | 19.37 (16.24 to 22.50) |
| 55–64 | 3.33 (0.39 to 6.27) | 22.40 (18.94 to 25.86) |
| 65–74 | 4.49 (1.11 to 7.87) | 25.42 (21.16 to 29.69) |
| ≥ 75 | 5.65 (1.64 to 9.66) | 28.45 (23.11 to 33.79) |
The difference between scores given by younger (< 55 years of age) white British and Pakistani participants to consultations containing ‘good’ levels of communication was small and not statistically significant. However, larger and statistically significant differences were seen for older patients and for consultations portraying ‘poor’ communication at all ages. In these ‘poor’ consultations, the difference in scores increased with rising age of participants. For example, ratings of consultations portraying poor communication were 10.29 points higher (95% CI 5.00 to 15.57 points) for Pakistani participants aged 18–24 years than for white British participants of the same age. This difference increased to 28.45 points (95% CI 23.11 to 33.79 points) for those aged ≥ 75 years.
Discussion
This experimental study found that respondents from a Pakistani background rated simulated GP consultations substantially more positively than their white British counterparts. These differences were largest for consultations depicting poor GP–patient communication and for older respondents. The differences that we observed were in the opposite direction to those in the national GP Patient Survey, which relates to a patients’ most recent consultation with a GP, for which Pakistani respondents give significantly lower scores for communication than their white British counterparts.
Strengths and limitations
We used an in-home face-to-face recruitment approach to ensure access to a wide range of respondents, independent of the GP practice that they were registered with. However, it is possible that respondents who agreed to participate in this research may differ in a number of unidentified ways from the population as a whole. For example, to ensure efficient recruitment to the study, we focused our efforts on areas with a high density of Pakistani people, which also have high levels of deprivation (82% of our Pakistani participants were living in areas in the most deprived quintile, whereas only 51% of Pakistani patients nationally live in the most deprived quintile). The sampled population may therefore differ from the Pakistani population as a whole; for example, recent research suggests that minority ethnic populations in lower ethnic density areas may report higher satisfaction with health care. 195 Ratings of consultations by ‘analogue patients’ (members of the public asked to rate care received by a third party), such as our participants, are commonly more critical than patients commenting on their own care. 196 In our study, negative response options were used more often than in the national GP Patient Survey; for example, only 2.6% of responses to the GP communication questions in the most recent GP Patient Survey were given as poor or very poor,197 compared with 26.6% of responses in this study. We deliberately set out to create a wider than typical range of communication quality within our vignettes to enable us to test the hypothesis that differential response tendencies between groups may exist only at one end of the communication range, for example that South Asian respondents tend to be more negative about the best care but rate the poorest care in the same way. In fact, we found that there were differences in ratings (with Pakistani respondents more positive) at both ends of the communication spectrum, reinforcing our interpretation that the disparities in real-life surveys are not to do with differential response tendencies. To enable the same vignettes to be viewed by all participants, the study was conducted in English, limiting our ability to understand evaluations by those with low levels of English language proficiency (and who might, for example, respond to the GP Patient Survey in other languages). In the USA, minority ethnic groups preferring languages other than English generally show response tendencies that are in the same direction as English-preferring members of the same minority ethnic groups, but to a greater extent, perhaps reflecting a continuum of acculturation. 150 However, it was not possible to produce vignettes that would remain equivalent in other languages and, as 99.8% of respondents to the GP Patient Survey respond in English, our ability to extrapolate to the wider population remains high.
Previous examinations of inequalities in patient experience between ethnic groups (including our own) have commonly relied on real-world data, in which it is difficult to distinguish whether differences are attributable to variations in care or variations in the reporting of that care. 75,131,150–153,156 The experimental design that we used in this workstream enabled us to control the content of the consultations being rated by respondents to explore how differences in reporting may explain the disparities in minority ethnic experience in real-life surveys. It builds on previous vignette research by using multiple video vignettes manipulating several key attributes. 81,82 Video vignettes have so far been little employed in this field, in spite of evidence of viewers perceiving them as realistic and enabling immersion in the situation at hand. 193 In the USA, Weinick et al. 81 reported no evidence of differences among white, African American and Latino evaluations of doctor–patient communication in vignettes when using an ‘always to never’ response scale; they concluded that variations within national surveys on such items for these groups were likely to reflect differences in real-life experiences. In this study, however, we found substantially more positive ratings by Pakistani respondents than by white British respondents.
Implications for practice
We designed this workstream to explore whether or not people from a Pakistani background rate the communication within simulated GP consultations differently from white British people. Similar ratings of simulated consultations from both ethnic groups would have suggested that the low scores observed in national surveys for Pakistani and other South Asian respondents reflect real differences in the quality of communication experienced by these patients in comparison to the quality of communication experienced by white British patients. The substantially more positive ratings from Pakistani respondents that we observed in our experimental study suggest that not only are there differences in the quality of communication in real-life consultations, but also that these differences are even greater than those identified in real-life surveys. We suggest that Pakistani patients receive genuinely worse standards of communication within a consultation. However, although we can be confident that differences in experience exist, it is difficult to extrapolate our vignette-derived data to estimate the magnitude of the difference in real life. Poor communication for these groups may arise from system-level, provider-level and/or patient-level factors. 198 For example, language barriers within consultations may lead to more negative experiences of care for both doctors and patients. 178 Levels of acculturation may be linked with a patient’s ability to navigate the health-care system, with consequent impacts on patient experiences of care. 180 System- and provider-level issues, including discrimination and bias, are sensitive and challenging topics, but ones to which more recent dialogue has looked to as likely key contributors to inequalities in care. 185
Our findings add substantial weight to the likelihood that inequalities affecting South Asian people in national surveys reflect systematic variations in the quality of communication within consultations. Although there is a body of research into the drivers of inequalities in care, we suggest that further research in this area now needs to focus on how factors including language barriers, health literacy, provider-side discrimination and system-level failures combine to inhibit good communication within individual consultations.
Section C Using data on patient experience for quality improvement
Chapter 7 Attitudes to receiving feedback from patient experience surveys: focus groups with practice staff
Parts of this chapter are based on Boiko et al. 199 under the terms of the Creative Commons Attribution Non-Commercial No Derivatives License (CC BY-NC-ND), which permits non-commercial use, distribution and reproduction in any medium, without alteration.
Abstract
Background
Despite widespread adoption of patient feedback surveys in the NHS, evidence of a demonstrable impact of surveys on service improvement is sparse. The objective of this study was to explore the views of primary care practice staff regarding the utility of patient experience surveys.
Methods
We conducted focus groups with staff from 14 practices following the receipt of feedback from a recent patient experience survey.
Results
Although participants engaged with feedback from patient experience surveys, they routinely questioned its validity and reliability. Participants identified surveys as having a number of useful functions: for patients, as a potentially therapeutic way of getting their voice heard; for practice staff, as a way of identifying areas of improvement; and for GPs, as a source of evidence for professional development and appraisal. Areas of potential change stimulated by survey feedback included redesigning front-line services, managing patient expectations and managing the performance of GPs. Despite this, practice staff struggled to identify and action changes based on survey feedback alone.
Conclusions
Although surveys may be used to endorse existing high-quality service delivery, their use in informing changes in service delivery is more challenging for practice staff. Drawing on the Utility Index framework, we identified concerns relating to reliability and validity, cost and feasibility, acceptability and educational impact that combine to limit the utility of patient survey feedback. Feedback from patient experience surveys has great potential; however, without a specific and renewed focus on how to translate feedback into action, this potential will remain incompletely realised.
Introduction and rationale
As outlined in the introduction to this report (see Chapter 1), feedback from patients is intended to inform quality improvements by increasing the responsiveness of the health-care system to the needs of service users and by identifying areas of poor performance or organisation that might be susceptible to change. 85,86,137 Although policy initiatives such as the introduction of the QOF or revalidation highlight feedback on patient experience as a key driver of quality improvement, evidence suggests that patient experience has had only a limited impact on service delivery45 and GPs and other health-care professionals may experience difficulties in making sense of survey-generated information. 119,200
In this strand of work, we drew on qualitative data to examine how primary health-care practitioners and their teams view and act on feedback from patient experience surveys. We examined the role that patient feedback plays in both assessing and improving standards of care. To assist our consideration, we adopted van der Vleuten’s201 Utility Index model as the basis for considering potential drivers of the gap between receiving and acting on patient feedback in primary care practices. Originally developed as a framework for assessment design and evaluation in educational settings, reports of the use of the Utility Index model have been extensive, although such reports have nearly always emanated from educational settings. We felt that the six domains of the model (educational impact, validity, reliability, cost, acceptability and feasibility) also had potential relevance when considering issues relating to the introduction and use of surveys of patients’ experience of care in routine clinical settings. 202
Changes to study methods from the original protocol
The aim of this strand of work, as stated in the original protocol, was to understand how general practices respond to low patient survey scores, testing a range of approaches that could be used to improve patients’ experience of care (aim 1).
In our application, we envisaged drawing on GP Patient Survey scores to facilitate a discussion with participating practices on their most recent results, their responses to their scores and any intention to change as a result. However, as part of our overall programme of work we conducted an individual GP-level postal survey with 25 practices (aim 2; see Chapter 9). This gave us the opportunity to feed back results to practices and individual GPs from our own, very recent survey and subsequently explore their responses to both practice-level feedback and the potential for individual feedback within focus groups (reported in this chapter) and interviews (reported in Chapter 8).
Methods
We conducted a postal survey of patients who had recently seen a doctor at one of a stratified random sample of 25 practices in Cornwall, Devon, Bristol, Bedfordshire, Cambridgeshire and North London (see Chapter 9 for details of sampling, recruitment and survey conduct). The patient experience survey used was based on items from the national GP Patient Survey11 and asked patients about access, waiting times, opening hours and continuity and interpersonal aspects of care. We reported the results back to practice staff at aggregate practice level (reported to all staff) and at individual GP level (confidential reports to each participating GP).
We purposively approached practices that had participated in the survey to take part in focus groups, aiming to reflect a diversity of practice size, geographical location and practice-level survey scores for communication. We undertook focus groups in 14 practices. Groups (with between four and 15 participants in each) were conducted following the completion of practice surveys and feedback of the findings to staff. Overall, 127 professionals from a range of backgrounds (38 GPs, 19 practice managers, 18 nurses, 21 receptionists, 13 administrators and secretaries and 18 other staff members including dispensers and health-care assistants) took part. In reporting, all practices were assigned a practice pseudonym; real practice names were not used (Table 28).
| Practice pseudonym | 2009/10 national GP Patient Survey scores for communication | Location | Number of practising GPs | Number of focus group participants |
|---|---|---|---|---|
| Highfields | High | Rural | 4 | 5 |
| Church Road | High | Urban | 8 | 15 |
| Fieldview | High | Rural | 5 | 9 |
| Town Road | Medium | City | 3 | 11 |
| Meadow | Medium | Rural | 5 | 13 |
| Pilkington | Medium | Urban | 3 | 9 |
| The Towers | Low | Urban | 2 | 4 |
| Brentwell | Low | City | 5 | 4 |
| Crossways | Low | City | 7 | 6 |
| White Road | Low | Urban | 2 | 7 |
| Torch Street | Low | City | 6 | 10 |
| The Maples | Low | Urban | 5 | 13 |
| Fallowfield | Low | City | 4 | 6 |
| Beeches | Low | Urban | 5 | 15 |
Focus groups, lasting around 1 hour, were held on practice premises and were facilitated by experienced qualitative researchers. A second researcher was present at each group to take notes. We piloted a topic guide (Box 2) at two non-study practices before beginning fieldwork. Key areas of discussion included attitudes to patient surveys, past experiences of surveys and practice procedures for dealing with survey feedback. All group discussions were transcribed verbatim. To maintain anonymity, participants were assigned pseudonyms.
-
What do you think of patient surveys in general? What do you think the survey results are saying to your practice?
-
Are the results of patient surveys circulated within your practice and if so, to whom? Have the scores encouraged you or your colleagues in wanting to change anything?
-
Do you think that individual GP scores following a patient experience survey could have an impact on the practice as a whole?
-
Do you think that over time, surveys of patient experience that focus on individual doctors’ skills, might affect the attitude of doctors towards their patients – or the attitude of patients towards their doctors?
-
To further explore the impact of individual GP performance on practice functioning, focus group participants were also invited to comment on two hypothetical situations where some doctors within the practice received less favourable scores from patient surveys than other doctors.
Reproduced from Boiko et al. 199 under the terms of the Creative Commons Attribution Non-Commercial No Derivatives license (CC BY-NC-ND) which permits non-commercial use, distribution and reproduction in any medium, without alteration (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/).
We drew upon framework approaches to organise and analyse our data, which allowed for themes to be assigned both from a priori research questions and from the narratives of focus group participants. NVivo 10 software was used for organising and examining the data. Analysis was undertaken by two researchers (OB and JB) and broadly took place over five stages: familiarisation (reading transcripts and listening to recordings in detail to gain an overview of content), thematic analysis (developing a coding scheme), indexing (applying the codes systematically to the data), charting (re-arranging the data according to the thematic content to allow comparative analysis), and mapping and interpretation (defining key concepts, delineating the range and nature of phenomena, creating typologies, findings, associations, providing explanations and developing strategies). 203
Guided by this approach, we drew on transcripts from the first focus groups to develop an initial coding framework, which included 48 codes grouped loosely into headings including validity of surveys, interpretation of survey feedback, organisational changes and performance comparison. Our coding framework went through a process of application, discussion and revision until all transcripts were coded using the final agreed version. Codes were subsequently grouped into four overarching analytical themes: survey validity and interpretation, practice dynamics, leadership and interprofessional decision making, and improvement strategies. The coding of each theme and subtheme was further triangulated by two researchers against a selected number of transcripts and discussed within the wider research team. The study was guided by our advisory panel including four PPI members, who provided input into the study design and conduct and interpretation of findings.
Findings
We focus in our findings on the organisational response of practice staff towards patient surveys. First, we consider how practice staff understand and engage with surveys and survey feedback. Then, we consider three dimensions of potential and actual change that appear to have been driven, in full or part, by surveys: redesigning front-line services, managing patient expectations and managing the performance of GPs. In the discussion, we place our findings within the context of the Utility Index model to consider how the utility of surveys to practice staff might influence their uptake as either quality assurance or quality improvement mechanisms. 201
Understanding of, and engagement with, surveys
All practice teams had extensive, first-hand involvement in surveying their patients and in receiving feedback from the GP Patient Survey. Attitudes to patient surveys were markedly contradictory. Recent experiences of payments linked to survey results under the QOF had caused resentment for many, particularly those who had lost out financially. Overall, practice staff found it difficult to trust surveys to reflect ‘reality’. Yet their expressed ambivalence about surveys was often mixed with an interest in, and engagement with, the findings. We explore these ideas in more detail below.
Credibility of surveys
Practice teams spoke broadly about the perceived weaknesses of survey methods, singling out issues around their design, administration, representativeness, reliability, sample size, bias and the political ends which they were intended to serve:
The surveys only take a snapshot.
Nurse, Torch Street
Only people with strong views complete them.
Receptionist, Crossways
You need to have sufficient sample size and a meaningful way of comparing across different GPs in order for someone to get some useful knowledge out of it.
GP, Fallowfield
Practice staff sometimes struggled with the concept of quantifying patient experience, voicing concerns that the complex reality of health-care interactions could not be measured using such rigid methods:
And a lot of this data that’s collected in a measurable kind of way doesn’t really represent reality. There’s kind of a fixation on measurable outcomes, but they don’t really tell us what’s going on, they’re just measuring that thing.
GP, The Maples
Discussions often distinguished between the utility and the relevance of different types of surveys, from in-house surveys conducted by receptionists handing out questionnaires, to the national survey programme. Local surveys were highlighted as enabling practice staff and patients to have greater control over the perceived relevance of the questions, but teams were often cynical about their robustness:
And some practices can manipulate their patients that they survey, so they will only hand out the questionnaire to nice patients and patients they know, they won’t do it on duty day when doctor is maybe running behind or very busy.
GP, Church Road
Criticisms levelled at the current national GP Patient Survey included its distribution to a sample of all patients registered with a practice regardless of whether or not they have consulted recently, the focus on feedback at practice rather than individual practitioner level, and the lack of inclusion of free text comments. Surveys that encompassed these elements were frequently regarded more positively:
We want to see data tailored to individual practitioner, because we all practice differently.
GP, Town Road
Other sources of patient feedback, such as complaints, were often framed as a more useful source of information to understand where the problems lie:
And I think we learn a lot more from patients that write to us individually with complaints.
Administrator, Town Road
Engaging with surveys
Despite these concerns, the importance attached to patient feedback via surveys in today’s health-care system was well recognised and broadly accepted:
I think we must not be too negative about surveys because they are part of the way we do things nowadays . . . I think if you look at how general practice changed particularly over the last 20 years, it has become a lot more patient focused and those things did not happen by accident, they have happened by design, and patient surveys have been a tool to drive that.
GP, Highfields
However, although participants (in particular GPs and practice managers) paid attention to and positively engaged with survey findings from year to year, contradictions and tensions were still evident, for example in relation to the validity of patient’s reports:
I think it is the only way to find out exactly what’s going on is to do a survey. The only way you really find out what the patients think. They are not always honest. Well, they are not always honest on the survey either.
Nurse, Beeches
I think it is useful for the extremes, but personally, I don’t think it is particularly useful for any middle ground. [Later in focus group] I think it’s very useful, when it compares against national average. I find that really, really helpful.
GP, Beeches
For practices that scored below national benchmarks, engaging with survey findings was often an emotional experience for staff:
It can be a bit disheartening at times though, if you feel that you’re really doing your best and then you get negative feedback.
Receptionist, Torch Street
The functions of surveys
In general, practice staff valued feedback from surveys as a source of information about their performance. Participants suggested that patients, individual GPs, and the practice as a whole could all benefit from surveys: for patients, for example there may be a therapeutic function, ‘the chance to get something off their chest and . . . to then move on’ (GP, Highfields). For GPs, the function of surveys was often to fulfil the requirements for appraisal. For practice staff, surveys could have a clear ‘improvement’ message, including the potential to highlight under-performing GPs:
It helps to highlight areas of improvement, to make sure that we’re continuing to do as well as we think we’re doing and it prevents us becoming complacent and assuming that you’re doing well. I mean if we are doing well, then it confirms that we are doing well, if we’re not doing well then it identifies areas that hopefully we can change. But not always.
GP, Highfields
You can argue over the validity of surveys but if over 3/4 years someone is consistently scoring low in certain areas, you can start making assumptions about the doctor performing not very well in the practice.
GP, Brentwell
Changes driven by survey feedback
The processing of survey feedback by practice staff was the essential first step in making any changes, which could encompass redesigning frontline services, managing patient expectations, and managing the performance of GPs. However, variation was evident in how transparent practice staff were in sharing survey information within the team, and in whether practice-level feedback was circulated between GP partners, to just a few practice decision-makers, or to all of the staff. In a small number of practices, results had been fed back promptly by staff to their patient participation groups (comprised usually of patients, the practice manager, and one or more GPs, such groups are convened by practices to discuss and review the services offered and how improvements may be made to these). Inevitably, the level of transparency impacted on the understanding of and engagement with patient feedback by practice staff.
Redesigning front-line services
Practice staff often described changes they had made to front-line services and systems as a result of patient preferences, including modifications to their facilities, appointment systems, and to staffing issues such as staff training. For example, car parks had been extended, GP triage had been introduced and new call management programmes had been installed. Staff in three practices clearly articulated the incorporation of suggestions from patient surveys into an annual action plan. However, in most practices changes were rarely attributable directly to survey feedback, the survey having provided a ‘nudge’ to action in areas practice staff had already been considering:
We did a change to open extended hours Thursdays, so that is a good thing – a benefit from last year’s I think, or was it the year before?’
Yeah, a year now.
Although it wasn’t really a response to a survey, that, it was a response to an initiative from . . . It was a response to the fact that there was funding available from the PCT for extended hours.
Torch Street
Managing patient expectations
For staff in some practices, survey feedback raised issues about how to communicate change to patients, how to shape expectations, and how to raise patient responsibility. Practice staff often felt they struggled to respond to patient demands and to increase understanding among their patients about how the system worked:
Was there anything in the feedback where you kind of, you thought maybe you wouldn’t respond?
Opening Sundays.
[Laughter]
I think another thing that was highlighted, for instance, is the question of marketing. I think we probably haven’t, in spite of having additional extended hours on Saturdays, and I think that was, was one of the things we had a big conversation about the MORI survey. At that point, we were offering all sorts of extended hours, but patients didn’t seem aware of it.
The Maples
Practice staff often felt that a perceived lack of understanding of systems and services was evident in ‘demanding’ patients, whatever effort was made. Furthermore, issues that suited one group of patients (music in the waiting room, telephone consultations) ran the risk of provoking dissatisfaction in others. As for relationships with GPs, individual patient preferences for doctors were not always fulfilled because many GPs worked part-time.
Practice staff felt that patients had a role to play in smooth and efficient functioning of primary care services. Staff spoke about increasing patient accountability and engaging patients in the feedback process through patient participation groups.
Managing the performance of general practitioners
Individual GP performance was regarded as an important factor in determining overall practice scores. Several managers in low-scoring practices admitted that, practically, it is very hard to tackle individual doctor’s (poor) performance:
If the survey results are between [the survey providers] and the doctor, and he knows that or she knows it, there’s absolutely no reason for them to change their ways, is there? What is the motivation to change, what is the driver to change when they have been rude or pretty lazy? Nobody knows that, let’s get on and continue as before. It is only when this information becomes available to, perhaps, the practice, that things could start to change. And when I say practice, who in that practice I don’t know, it could be the executive partner. But I think somebody ought to know and somebody ought to discuss these issues.
What’s the point in doing the survey anyway? If nothing is going to happen, is no point in doing that if doctor . . .
Nothing is going to change.
. . . got the bad score and they keep it to themselves.
Brentwell
The idea of having an ‘outlier’ doctor, whether it was a high or a low performer, was familiar to practice staff. Both scenarios could have an effect on the running of the practice, for example when patients found it difficult to obtain an appointment with a particularly popular doctor. In addition, the complexity and interlinking of factors influencing patients’ responses was highlighted: patients’ overall impression of the surgery and of the appointment system was perceived to influence their reports regarding consultations, and possibly the performance of the doctor too:
Looking at the way people have access, the way the practice is organised, that they have access to facilities within the practice, the hours that the practice is open, the stage of the practice, the receptionist, how the admin is done, virtually how the sort of machinery of the practice works . . . I would not be surprised that where you had a poorly organised practice, poor machinery, if you like, you also had poor doctors, because I think doctors are influenced by the machinery in which they work, as well as influencing the machinery themselves.
GP, Pilkington
The majority of teams stressed that they would support a doctor who consistently received negative patient feedback, although they did raise concerns about the difficulty of having an ‘unmanageable’ GP in the practice. Suggested internal mechanisms of support ranged from mentoring from a team member, role-plays and peer support sessions, to interventions by a partner and/or manager. Creating a supportive environment was described as an important enabler, although it was not always clear what the concept of ‘supportive environment’ actually meant for the participants. There were no doubts that doctors who were put ‘at the bottom of the pile’ by survey results could perceive any intervention as threatening. In three low-scoring urban practices, staff were supportive of making the doctors’ scores publicly available, identifying a responsibility to maintain patient safety.
Barriers to improvement
Discussions on potential improvements most commonly focused on changes to the practice premises and organisational aspects of the delivery of care. Even for such changes, which may have been at least in part precipitated by patient survey feedback, staff in most practices felt there was little long-term impact on patient opinion:
We’ve done a number of things and the MORI poll results have been remarkably stubborn in terms of the change in perception by patients. That’s been quite slow.
Manager, Beeches
As one respondent highlighted, survey fatigue and the feasibility of being able to make relevant, meaningful changes was a persistent problem:
The cynicism that [Dr Ahmed], has quite rightly identified as being the problem with the surveys, is the fact that we have been surveying, and patients have been surveyed, for several years, the questionnaires are inevitably similar, the responses are inevitably similar, but the consequences of the survey are depressingly zero. So there may be a request from patients, for example, that old chestnut, the Saturday morning surgery, but that has never been, and never will be, as far as I’m aware . . . funded to take place. So, you then question the validity, the point of actually having the survey.
GP, Church Road
Staff highlighted a wide range of barriers to implementing changes which may have been requested by patients, most particularly expressing concerns around funding and staff capacity. A distinction was made between patient ‘needs’ and patient ‘wants’, with identification of an on-going struggle to meet unrealistic expectations:
It is a bit like opening on Saturday issue. Would you like the surgery to be open on Saturday? Yeah. Would you like us to go 24 hours? Yeah. Are you going to pay more taxes to have it open on Saturday? No. Are you going to use appointments during the week when you are able to make it? Mmm, not sure. But if the question is would you like to have it open on Saturday? Yeah. Consumerist.
GP, Church Road
There was far less discussion and agreement on how to effect changes to interpersonal aspects of care, if survey feedback highlighted issues relating to a particular GP. Issues included confidentiality and the ‘unlikely’ situation of GP feedback being shared with other practice staff [‘self-learning and training, then I think that’s more of a personal issue rather than being shared with the practice’ (Practice Manager, Highfields)], and the idea that practice staff may need to recognise a balance in a GP’s interpersonal abilities and other aspects of their professional practice [‘maybe that doctor is not a great communicator but they are great at doing something else, you know’ (GP, Church Road)].
Ultimately, staff in many practices felt there was little external support for making changes in response to patient feedback:
. . . we need more support in this area . . . one of my concerns up until now is that sometimes services have come out and there has been very little support from anyone to say, right this is how you can improve things that might help, or we understand why you might be having problems, which ways we can help you with that. It has always been: here is your survey results, it is up to you how you sort it.
GP, Highfields
Discussion
We suggest there are two primary purposes of large scale surveys of patient experience. First, surveys may be used to endorse and affirm good clinical practice or service organisation. Second, in line with the aspirations of policy-makers, surveys may provide evidence to inform improvements in health-care provision. 85 Our findings suggest that staff in general practice broadly view the role of patient feedback as one of quality assurance, providing evidence of whether or not they are offering an acceptable level of care to their patients. However, the role of surveys in quality improvement appeared less certain among participants. Although we identified potential dimensions of change (including front-line service improvements, management of patient expectations and management of GPs’ performance) which could be informed by survey feedback, actual changes were usually confined to ‘easy targets’ for modification such as décor or playing music. Practice staff frequently oscillated between questioning the credibility of survey findings and taking them at face value; as we observed, respondents could be critical of survey methods while being pleased that their practice had ‘done well’. For those who had performed less well, pathways to change were not often clear. These organisational responses to patient experience surveys were, inevitably, dominated by GPs and practice managers; within our focus groups, receptionists and administrative staff were far less vocal. Although not reported within this paper, our analyses suggest important variations in the extent of the influence of practice managers, and the dynamics between practice managers and GPs, on how practice staff as a whole reflect and act upon patient feedback.
Strengths and limitations and implications for future research
This study benefits from drawing on a large sample of primary care practitioners providing care in a range of practice settings in England. Participants represented a range of primary health-care professionals. Fourteen focus groups, of varying size, acted, we believe, as an effective means of capturing a range of participant views. The topic appeared of interest to participants. Participants were drawn from socio-demographically and geographically diverse areas, although all in England. Future similar research might usefully explore approaches to the impact of more immediate feedback, determining the extent of bias in response associated with varying response rates, and exploring motivations associated with changing (or not changing) practice in response to patient survey feedback.
The Utility Index
Van der Vleuten’s201 Utility Index was originally developed to consider assessments within an educational context (for example, the provision of feedback on progress to medical trainees or the conduct of examinations for specialist training), yet this model also has value in exploring the utility of patient surveys in service contexts. Any expectation of quality improvement from patient surveys is framing feedback from such undertakings as an intervention aimed at stimulating action. Examining our emerging findings through the utility lens, which we undertook as a post hoc exercise, suggested that the overall value of patient feedback from surveys (and thus its potential to drive significant quality improvements) is undermined by a combination of variable attitudes to its credibility, and challenges for practice staff in identifying and bringing about meaningful changes (Figure 17).
FIGURE 17.
The ‘Utility Index’ of patient experience surveys in primary care: perspectives of practice staff. Reproduced from Boiko et al. 199 under the terms of the Creative Commons Attribution Non-Commercial No Derivatives License CC BY-NC-ND and permits non-commercial use, distribution and reproduction in any medium, without alteration.
Drawing on both our work and others’ work, we suggest that the notion that survey feedback alone will stimulate major changes in care is an unrealistic expectation. 83,204 Although we saw evidence of changes to minor modifications such as car parking, décor and (slightly more challengingly) appointments systems, issues such as the management of GPs with evidence of poor communication skills, or responding to other ‘interpersonal’ aspects of professional practice, were much harder to tackle. Although patient experience will no doubt be improved by making general practices more accessible and more pleasant, significant aspects of experience linked to better clinical outcomes, including the quality of nurse- and GP-patient communication and trust and confidence in clinical staff, risk being left outside the focus of improvement work undertaken by practice staff.
There are six dimensions of the Utility Index (reliability, validity, cost, feasibility, educational impact and acceptability) which may determine the potential utility of an intervention, including patient experience survey feedback. All have relevance for how general practice staff view the current role of patient surveys.
Our identification of issues over the credibility of surveys, and difficulties in the interpretation of feedback, clearly echoes previous work in both primary and secondary care, which suggests widespread scepticism about the robustness of patient surveys. 119,200,205,206 Practice staff were more likely to view results positively if their scores were stable over time, were above average, and corroborated other sources of feedback such as complaints and compliments.
Although respondents felt that national patient surveys were perfectly feasible, there were concerns about the challenges of undertaking local practice surveys. Issues included the time taken to undertake such work and how best to ensure that in-house surveys were conducted robustly. There were also mixed attitudes about the cost-effectiveness of national survey programmes, in part due to the perceived difficulties in acting on feedback. We are not aware of any studies that have explored the cost-effectiveness of large-scale patient feedback surveys. This reflects recent discussions among GP leaders calling on national surveys to be banned on account of generating irrelevant and overly expensive data.
We found a consistent lack of impact of surveys at practice level, driven by factors including an absence of coordinated action and difficulties in making sense of survey feedback. 207,208 Benchmarking data were seen to be useful, although it was not always easy to make sense of. 67 Likewise, practice staff welcomed free text comments from patients as providing more specific information about their opinions. 209,210 Most commonly, when change did happen, survey findings were only one of the spurs to action to address an already-acknowledged problem. Changes, however, usually focused on service organisation or facilities and not on individual practitioner behaviour. There remains little evidence that patient feedback alone has any impact on the behaviour or skills of medical practitioners, with a number of controlled trials of the impact of patient feedback having little demonstrable influence on subsequent patient feedback. 83,204,211 The provision of facilitated feedback of results may be more effective in engendering engagement and action, as recent evidence in the secondary care setting demonstrates. 212 However, the emotional toll of negative patient feedback on staff is also relevant here: staff reported how disheartening it could be to receive consistently poor comments. The potential to see patient feedback as threatening and harmful, both at individual clinician level and at practice level, is an additional barrier to acting on such data, and further suggests the potential for facilitated reflection in assimilating feedback.
Practice staff worried that an endless cycle of surveys was inconvenient and burdensome for their patients. Nevertheless, surveys appeared to be broadly accepted as part of the new paradigm of patient-centred care, and welcomed in that role. However, lingering concerns over the linking of patient feedback to pay-for-performance and the external imposition of surveys on general practice (along with a long list of other activities) tempered the acceptance of current surveying practices, particularly for GPs.
Drawing these components together, we suggest that key drivers of the gap between conducting surveys and implementing changes relate to the difficulties of practice staff in trusting and making sense of survey findings, coupled with a lack of support for identifying and making changes to practice.
Implications for practice
Although practice staff predominantly view feedback from patient experience surveys as a mechanism for affirming good or detecting poor service delivery (i.e. as a quality assurance mechanism), the current direction of policy targets a higher aspiration of providing evidence to inform changes in practice (a quality improvement mechanism). The question remains as to how patient experience survey data can become a key driver of service improvement. Evidence suggests that securing feedback alone is insufficient to stimulate change,204 and our findings point to primary care practices being left to be responsible for developing their own implementation mechanisms. GP contractual arrangements prior to 2009 offered incentives to primary care practices to discuss the findings of patient feedback surveys with patient representatives, for example through the use of patient participation groups. Although now withdrawn, such an approach may have substantial merits in facilitating change, as well as acting as a means of responding to the need for active patient and public participation in informing the design and configuration of services.
Recent work in secondary care highlights the potentially important role of facilitators in enabling staff to review survey results and, most importantly, act on them. 212 Within primary care, such initiatives are lacking. Practice staff need to be supported to reflect on patient feedback; this will need dedicated resources on top of those committed to collecting patient experience data. Quality assurance of survey development, data collection and reporting of results is of vital importance if the findings of surveys are not to be dismissed out-of-hand on the grounds of credibility, or to become the subject of discussion aimed at diverting rather than promoting action and change.
Where surveys highlight the need for change, formal processes for planning and delivering change are required, covering both minor modifications and more challenging problems such as reported problems with the quality of clinician–patient communication. In the current climate of scarce resources, a commitment to developing patient experience surveys as quality improvement mechanisms would therefore displace other competing priorities, and policy-makers and practitioners must be realistic about what can be achieved. However, until then, it is our view that the full potential of patient feedback will not be achieved.
Conclusions
We have identified a number of key reasons for the gap between the receipt of patient feedback and acting on that feedback. Addressing the concerns of primary care providers across all aspects of patient surveys – reliability, validity, cost, feasibility, impact and acceptability – and supporting them to reflect on the meaning of such data will be important if we are to draw on such evidence in quality improvement programmes. Alongside this, however, we need to develop a realistic understanding of where surveys may be expected to drive change, and where they may not.
Chapter 8 Attitudes to receiving feedback from patient experience surveys: interviews with general practitioners
Parts of this chapter are based on Farrington et al. 213 under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited (http://creativecommons.org/licenses/by/4.0/).
Abstract
Background
To date, little research has focused on doctors’ attitudes to patient experience surveys that give them personalised feedback. Although national surveys, such as the GP Patient Survey, report results at a practice level, GPs are additionally required to reflect on individual-level patient feedback for the purposes of appraisal and revalidation. This chapter examines doctors’ perceptions of patient experience surveys, and the receipt of personal feedback from these, in primary care settings.
Methods
We analysed data from 21 interviews conducted with GPs across 14 practices. Participants were sampled from doctors who had participated in our patient experience survey (reported in Chapter 9) and had recently received individual-level survey feedback.
Results
General practitioners expressed commitment to incorporating patient feedback in quality improvement efforts. However, they also expressed negative views about the credibility of survey findings and patients’ motivations and competence in terms of providing feedback. As a result, they found it challenging to make sense of and take action as a result of the feedback that they received from patient experience surveys.
Conclusions
General practitioners’ ambivalence towards patient experience surveys is likely to limit their impact on the success of quality improvement initiatives. In response, this chapter highlights the need for initiatives to address doctors’ concerns about the credibility of surveys.
Introduction and rationale
A number of recent policy initiatives have emphasised the utility of patient feedback for quality improvement. 38 In the UK, a series of initiatives has established and expanded the role of patient experience surveys in the NHS, leading to the recent NHS Outcomes Framework,214 which features patient experience as one of five key domains on which NHS performance is judged. In addition to national surveys, such as the GP Patient Survey, numerous surveys of various kinds are undertaken at the local level by health-care providers. 215 In 2012, the General Medical Council (GMC) introduced a revalidation programme requiring individual doctors to collect patient feedback on the care that they provide. 216 Such feedback is subsequently used as supporting information in a 5-yearly procedure through which doctors ‘revalidate’, that is, retain their licence to practise, and is also intended to facilitate reflective improvements in the quality of individual doctors’ practice. 137 Nevertheless, most national survey programmes continue to be conducted and reported at the organisational level. Likewise, existing research has tended to focus on doctors’ engagement with reports of patient experience at the level of the hospital ward, primary care practice or similar organisational units within primary or secondary care.
Existing research highlights the importance that doctors place on patient experience in principle and the potential for positive improvements based on patient feedback. 119 This body of work has also explored challenges surrounding the incorporation of patient feedback into medical practice. Doctors commonly express a range of negative views about the plausibility of survey findings, including concerns about sample size and representativeness; respondent bias and subjectivity; reliability and validity of survey instruments; lack of clarity on the purpose of surveys; contextual sensitivity; and the challenges of interpreting patient feedback when lacking contextual information, with numerical scores viewed by many doctors as ‘a simplistic reduction from a complex range of factors’ (p. e160)200 (see also Coulter et al. 45 and Asprey et al. 119). These challenges relate to long-standing critiques of quantitative surveys that highlight issues such as the lack of self-evident meaning in numerical findings (see, for example, Williams217) in a range of contexts including special educational services and health-care provider performance. 218,219 These and other concerns have tended to limit the impact of patient feedback in terms of quality improvement. 215
Many of the challenges associated with patient experience surveys relate to standard features of survey administration and so are also likely to be relevant to surveys administered at the individual doctor level. With some exceptions (e.g. Hill et al. 67), few researchers have focused directly on doctors’ engagement with patient experience surveys at the individual doctor level. Although such engagement is largely unexplored, it is of considerable significance given the well-established role of patient experience surveys in contemporary health care (and the NHS in particular) and the recent introduction of mandatory individual doctor-level surveys.
This chapter draws on qualitative data to explore attitudes towards patient survey feedback on the part of individual GPs. By exploring attitudes towards the plausibility of surveys, this chapter demonstrates the generally contested, problematic and inconsistent nature of doctors’ current engagement with patient experience surveys and points towards the need for additional investment in training and relevant resources.
Changes to study methods from the original protocol
The aim of this strand of work, as stated in the original protocol, was to understand how general practices respond to low patient survey scores, testing a range of approaches that could be used to improve patients’ experience of care (aim 1).
The interviews reported here took place alongside the focus groups with practice staff, reported in Chapter 7. In our original application, we set out plans to interview each doctor in between five and eight low-scoring practices. These interviews would cover their accounts of what contributed to their practice score, considering their recent GP Patient Survey feedback. However, as with the focus groups, the conduct of our own patient experience survey (reported in Chapter 9) at individual doctor level meant that we were able to feed back to GPs their own patient experience scores. Interviews thus considered attitudes to both practice- and individual-level feedback. We also altered our sampling strategy, deciding instead to incorporate a wider range of practices (14 practices) to reflect a greater diversity of practice cultures within which the GPs were working.
Methods
Data collection
We conducted 40 semistructured face-to-face interviews with GPs in practices across Cornwall, Devon, Bristol, North London, Bedfordshire and Cambridgeshire. These practices were part of a larger group of 25 practices participating in our patient experience survey (see Chapter 9 for details of sampling, recruitment and survey conduct). From the sample of 25 practices, two doctors were interviewed from practices with low GP Patient Survey scores and one doctor was interviewed from each medium- and high-scoring practice. Individual GPs were identified randomly within each practice and approached one by one for consent to participate. Each GP had received an individual report from our patient experience survey, focused on patient responses to communication items and including summary statistics and free-text comments. An interview topic guide was developed in light of existing literature to focus on individual-level patient experience surveys and was revised in relation to policy changes on revalidation that occurred during the conduct of the study. Interviews lasted between 20 and 60 minutes.
For the purposes of this report we excluded 19 interviews conducted with GPs prior to the introduction of revalidation in December 2012, as this changed the nature of the topic guide and issues covered in the interviews in relation to the conduct and implications of individual doctor-level patient surveys. We thus include data from 21 GP interviews conducted across 14 practices.
Data analysis
The interviews were digitally recorded with written consent and transcribed verbatim. NVivo 10 software was used to organise and categorise the data. Transcripts from four GP interviews (not included in the final analysis) were used to develop an initial coding framework, which included 44 codes grouped into headings including survey experience and survey-related change. A thematic analysis approach was used220 involving six distinct stages: familiarisation with the data; generating initial codes; searching for themes; reviewing themes; defining and naming themes; and producing a final analysis, which was discussed among the research team before being revised and finalised.
Results
Dimensions of ambivalence
Our analysis found that GPs demonstrated profound ambivalence regarding the purpose and plausibility of patient surveys, leading to complex, varied and problematic engagement with patient feedback. The Oxford English Dictionary221 defines ambivalence as ‘having mixed feelings or contradictory ideas about something’, a definition that was interpreted in this study as a spectrum from mixed feelings about something to holding ideas that directly contradict each other. Two main dimensions of ambivalence were identified. The first relates to doctors’ views of patients’ motivations and competence as responders in surveys. The second relates to doctors’ views of surveys from the perspective of enabling quality improvement (or otherwise) – views that may diverge from what is intended by the managers responsible for introducing and administering surveys.
Interviewees rarely situated themselves consistently with regard to these two dimensions of ambivalence; indeed, it was common for GPs to express inconsistent and contradictory views on both dimensions of ambivalence, often within the same interview (see following sections). Consequently, although some themes (e.g. a greater emphasis on negative rather than positive views of patients) were more to the fore than others, ambivalence is the dominant and unifying feature of the findings in this area.
Patients and surveys
General practitioners emphasised the centrality of the doctor–patient relationship and the utility of receiving feedback from their patients. For example, one interviewee described the doctor–patient relationship as an ‘adult to adult’ relationship in which patients know more about some things than doctors and in which doctors need to listen to patient feedback:
[T]he only way you’re going to know whether you’re doing your job properly . . . it’s listening to what the patients are telling you [in their feedback].
GP4
Against this backdrop, many GPs discussed patients’ motivation and competence to provide feedback in more detail. One GP discussed how patients’ feedback showed that they were reflecting in depth on their experience before communicating it through free-text comments:
They’re . . . thinking ‘Well, actually, what do we think of the [practice]?’. . . rather than just at the time when they’re desperate for an appointment and frustrated, you know, to think actually . . . what things at the [practice] do they actually value.
GP9
More widely, several doctors noted that patients were used to responding to surveys in other spheres of their lives, potentially (although not inevitably) increasing their willingness to provide feedback on their health-care experiences. As such, many doctors saw patients as motivated to reflect on and communicate their experiences (although this was also raised as a concern in terms of raising patient expectations; see following section).
Similarly, some doctors expressed the view that patients are competent to judge their care. Patients’ ability to evaluate doctors was sometimes endorsed because it aligned with the doctors’ pre-existing positive views of their own professional skill. However, despite this, many interviewees expounded fundamentally ambiguous views of patients considered as survey respondents, often combining in the same interview seemingly positive views of patients’ motivation and competence with more negative views. For example, one GP emphasised the utility of patient surveys in terms of patients’ capacity to identify specific problems:
I think the patient feedback is really important . . . You’ve got to actually listen to what are patients saying, [e.g.] they are telling us through this [feedback] that the system currently in place for booking appointments . . . is not working for them.
GP2
The same GP also stressed, however, the ways in which patients’ comments were often of little use for improving care quality, especially at the individual doctor level:
When I read the comments it was just a diatribe of accusations against the practice as a whole . . . [I]n terms of my individual practice it gives me no feedback at all . . . [The] majority of the comments on the appointment system and on lack of [relational] continuity [were] all on the issues that we are totally aware of.
GP2
Doctors often questioned patients’ motivations, first, by viewing patients who provided negative feedback as doing so because they had specific grievances to express [‘if they’ve got an axe to grind’ (GP10)] and, second, by suggesting that patients participate in surveys to gain leverage over doctors. Many interviewees also questioned patient competence or patients’ ability to provide accurate and relevant feedback. Overall, GPs advanced six principal characteristics of patients that singly and/or collectively undermined their ability to provide accurate feedback:
-
Positive bias – the tendency of patients to give strongly positive feedback regarding doctors, linked to patients’ well-documented reluctance to criticise doctors in general and their own GPs in particular. In this context, one GP described the patients as ‘quite reluctant to talk the doctor down, because we’ve got a good ongoing relationship’ (GP10).
-
Negative halo effects – patients ascribe negative characteristics to consultations because of other negative experiences during their visit to the practice. As one GP described, patients may carry an ‘initial bad experience’ with the practice reception ‘all the way through . . . into the consulting room as well . . . it affects all of your feedback’ (GP6).
-
Failure to understand surveys – for example one GP noted that ‘because [patients] don’t understand the questionnaire, they might tick whatever box they think; and that’s the reason we don’t get true results’ (GP19).
-
Subjectivity – several doctors emphasised that different individual patients could give different feedback despite having experienced similar consultations concerning similar medical problems. More widely, one GP highlighted patient subjectivity by suggesting that strongly negative patient feedback could ‘reflect more on the person [patient] than it does on you [the doctor]’ (GP6).
-
Inability to evaluate clinical competence – GPs highlighted patients’ inability to judge doctors’ clinical competence. As one noted, patients ‘don’t know about my clinical ability . . . [or] how much I know’ (GP8).
-
‘Good doctors, bad feedback’ – doctors felt that good care may result in negative feedback because it differs from patients’ preferences. Common examples included doctors refusing to prescribe antibiotics or write ‘sick notes’ for patients with depression. GPs referred to situations in which patients were unhappy with treatments recommended (or withheld) by doctors and often saw themselves as having a responsibility to protect NHS resources rather than pleasing patients: ‘pleasing a patient isn’t the same thing as being a good doctor . . . I see part of my role as a GP [as] gatekeeping NHS resources, including my own time’ (GP8).
Thus, although doctors’ views often combined positive and negative views of patient feedback, negative views tended to dominate, resulting in a sceptical attitude that questioned patients’ motivations and competence vis-à-vis the provision of feedback (Table 29).
| Category | Doctors’ attitude | |
|---|---|---|
| Positive | Negative | |
| Patient motivation | Willing to take time to provide feedback | Axe grinding |
| Used to providing feedback in other spheres | Desire to influence doctors | |
| Patient competence | Able to recognise good-quality care/improvements | Positive bias |
| Negative halo effects of clinic/survey experiences | ||
| Unable to understand survey instruments | ||
| Subjective judgements | ||
| Lack of clinical knowledge | ||
| Good doctor/bad feedback | ||
Patient experience surveys and quality improvement
This section focuses on a second dimension of ambivalence, relating to GPs’ perceptions of the potential for patient experience surveys to drive quality improvement. Doctors identified benefits in reflecting on patient feedback and encouraging competition between doctors through comparison of patient feedback scores. However, they also presented a number of concerns that undermined the potential of surveys to facilitate quality improvement. As with doctors’ attitudes towards patient feedback, the overall impression was more negative than positive.
Positive attitudes
Doctors emphasised the potential for patient experience surveys to facilitate quality improvement in a variety of ways. One GP, for instance, emphasised that they ‘actually took on board things which people were saying’, as ‘there’s no point doing a survey . . . unless you’re actually going to take notice of what the results say’ (GP1). Numerous participants described negative feedback as having more utility for change than positive feedback. Furthermore, a number of doctors discussed the potential for quality improvement to be driven by doctors’ competitiveness with regard to colleagues’ performance and/or benchmarked data (i.e. data supplied alongside comparative figures for comparable surveys undertaken in the past or elsewhere). One GP, for example, noted that surveys are:
[A]ll about comparing yourself with other GPs who do the same job . . . Because, I think, you want to know that you’re in the best group, compared with other people.
GP5
Overall, interviewees saw the potential for survey-based quality improvement in three main areas:
-
Reminders of core proficiencies, especially communication skills and basic tasks such as introducing themselves to patients and ensuring that patients are satisfied with the consultation before they leave. Several doctors remarked on the utility of repeated surveys for highlighting the importance of such issues, with one GP saying, ‘I think it flags up . . . the initial consultation tips that you think you do that perhaps you don’t always’ (GP5).
-
Reinforcements of known problems (and providing evidence to support change), often at the practice level: ‘the [survey] was useful because [it] really reinforced the impressions that we were beginning to form as . . . colleagues, and it was a bit more evidence that we could actually say, “Well, look, this isn’t personal, because look at this, and this is random and anonymised data coming in” ’ (GP15).
-
Unexpected issues documented in free-text comments. These were often seen as providing more useful material for reflection and change than numerical feedback, which was seen as overly positive about the care that patients had received. Thus, one GP stated that ‘I actually took more from the free-text comments . . . because I think the figures were . . . all pretty good really . . . [R]eading through the comments I think is really quite helpful . . . just having it there makes you think about it and think “Well, why do I do that?” ’ (GP1).
Negative attitudes
Doctors’ positive attitudes towards the potential of patient experience surveys in facilitating quality improvement, noted in the previous section, were paralleled and undermined by a plethora of sceptical views. For interviewees, this led to an ambiguous but overall decidedly negative picture in which the value of surveys for quality improvement purposes was placed in severe doubt, in line with previous research in other fields that emphasises the challenges involved in interpreting survey data. 218,219 As well as negative views of patient motivations and competence, outlined above, GPs added several more reasons for discounting surveys as quality improvement tools. Broadly, these concerns fell into five categories:
-
Concerns about the validity and reliability of surveys on the basis of factors including low response numbers, biased samples and problematic administration methods. GPs expressed concerns about response numbers despite having high numbers of respondents for their individual feedback (with a mean of 71, double the usual number required for adequate reliability). One GP linked what they saw as low response rates to patients’ fatigue regarding surveys: ‘There is a little bit of questionnaire overload . . . And I think it’s reflected in a very poor response rate’ (GP2).
-
Difficulties surrounding interpretation, including the separation of statistics from free-text comments and thus the difficulty of interpreting patients’ rationale for specific responses in a given survey. As one GP remarked, ‘if there was a problem there [in the numbers] I’d look towards addressing that, but I couldn’t really find a comment which was associated with that . . . so I found it quite difficult’ (GP1). As research has found in other contexts,212 feedback presented to health-care professionals without expert facilitation can be difficult to interpret and act on.
-
Issues of context. Doctors raised concerns about specific features of clinical encounters or patient characteristics that could influence patient feedback and thus undermine the value of patient feedback as a foundation for quality improvement. For example, some GPs who worked in deprived areas felt that surveys did not take sufficient account of the possibility of some population groups giving systematically more negative feedback than other groups: ‘sometimes I think you have a survey and I don’t think it’s a true reflection of where you are, your demographics. And I think that can be a problem’ (GP11).
-
Anxiety about negative feedback. A number of GPs discussed actual or potential anxiety arising as a result of negative feedback. This could problematise doctors’ engagement with survey findings, impact on their confidence and make them less likely to adopt a positive and constructive attitude towards improving their care. One GP, for instance, described feeling upset and worried following negative feedback – feelings that were shared by many other GP interviewees: ‘I find it quite difficult, because I’ll always take it quite personally’ (GP3).
-
The risk of raising patient expectations. The fifth and final area of concern relates to the risk of raising patient expectations through surveys by introducing a consumerist element previously more associated with customer relations than medicine. As one GP noted, ‘it’s like TripAdvisor, everything, everybody’s being rated’ (GP8). As several doctors noted, it is not always possible to meet these rising expectations, especially with regard to resource-related issues such as out-of-hours appointments; consequently, surveys may encourage patients to expect changes that are impossible to implement in practice, leading in turn to negative patient feedback. Thus, if quality improvement is evaluated at least in part on the basis of patient experience surveys, then surveys themselves may render evidence of improvement less likely.
Overall, negative views of the potential contribution of patient surveys to quality improvement agendas dominated the findings (Box 3).
-
Value of reflecting on patient feedback.
-
Value of competition between doctors on the basis of survey feedback.
-
Reminders of core proficiencies.
-
Reinforcements of known problems (and providing evidence to support change).
-
Unexpected issues documented in free-text comments.
-
Discounting of patient motivations and competence.
-
Concerns about the validity and reliability of surveys.
-
Difficulties surrounding interpretation.
-
Issues of context.
-
Anxiety about negative feedback.
-
Risk of raising patient expectations.
Reproduced from Farrington et al. 213 under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited (http://creativecommons.org/licenses/by/4.0/).
Discussion
This study explored GPs’ engagement with patient experience surveys and our findings show that they express mixed and contradictory opinions, despite long-standing experience of such surveys. We have discussed doctors’ views with regard to two key dimensions of ambivalence: patients as responders to surveys and the potential of patient feedback to facilitate quality improvement agendas. Interviewees’ vacillation between different standpoints problematised attempts to generate a clear impression of engagement with patient experience surveys. Nevertheless, it is possible to draw some important conclusions. Although GPs endorsed patients’ motivations for participating in surveys and their competence to provide accurate and relevant feedback, these notions were outweighed by the numerous ways in which doctors emphasised what they saw as patients’ questionable motivations and lack of competence vis-à-vis surveys. Consequently, doctors appear to view patients, as survey respondents, in a deeply ambiguous fashion, that is, as being simultaneously competent and incompetent at evaluating doctors, as being both accurate reporters of experience and inevitably biased commentators, as disinterested contributors to quality improvement and axe grinders. Likewise, although participants appeared to emphasise the potential utility of patient feedback for quality improvement, they also presented numerous factors that individually and collectively undermined this agenda. Overall, GPs’ engagements with patient experience surveys were highly contested, problematic and inconsistent.
From a GP’s perspective, surveys themselves exhibit varied properties and capacities in multiple dimensions, including the different ways that patients are perceived to interact with survey instruments, the different purposes for which surveys can be undertaken and the different conceptualisations that doctors can generate about them, reflecting the wider challenges inherent in interpreting numerical data. 212 Additionally, the varied ways in which survey feedback is disseminated in different settings generates another tier of properties: a feedback report that is e-mailed to a doctor with no benchmarking or interpretative guidance is effectively a different kind of feedback from a benchmarked report discussed with a facilitator. As a result of these varied properties, ambiguity is a strong feature of surveys as currently administered. GPs appeared to make sense of this ambiguity by drawing on their identities and frames to arrive at a widely shared yet ‘internal’ ambivalence. In this context, we understand ‘internal’ ambivalence as a kind of ambivalence that takes place not so much across different doctors (although this was seen at times) but rather within doctors, such that individuals tended to express multiple and mutually contradictory ideas. From this perspective, doctors appeared to consider more than one interpretation of patient experience surveys as plausible at the same time.
Nevertheless, doctors did not see all interpretations as equally plausible. As discussed earlier, they tended to settle on negative views of patients and patient experience surveys, thus undermining the potential for reflective change and quality improvement in response to patient feedback (in line with previous research119,200). The numerous specific reasons that doctors gave in support of their standpoints – ranging from patients’ lack of clinical expertise to surveys’ lack of contextual sensitivity (Table 30) – suggest that plausibility in this context is a complex, multilayered and largely ‘negative’ phenomenon. As such, patient experience surveys can be seen as an important instance of a wider problematic identified by May et al. :222 ‘what to do with the patient’s subjective experience of illness, and how to connect it with medical knowledge and practice’ (p. 1023). Research in other domains, such as teachers’ responses to pupils’ feedback, illustrates that this problem is not specific to health-care contexts, but rather characterises more universal responses to feedback and criticism. 223
| Factors inhibiting plausibility of interpretations favouring quality improvement | Solutions to increase plausibility |
|---|---|
| Views of patients | |
| Not dispassionate evaluators | Facilitate doctors’ personal engagement with patients; training for doctors regarding psychometric bases of validity |
| Incompetent evaluators | Facilitate doctors’ personal engagement with patients; include clearer instructions to patients on survey instruments |
| Views of surveys | |
| Difficulties of interpreting feedback | Provide facilitated feedback for individual doctors/groups of doctors embedded within wider local change programmes; provide additional information on feedback material (e.g. benchmarking data) |
| Lack of contextual sensitivity | Explore potential for development/validation of tailored survey instruments for different care settings |
| Anxiety regarding negative feedback | Provide support for individual doctors concerned about negative feedback |
| Risk of raising patient expectations | Limit frequency of survey administration to minimum necessary, except when raising patient expectations is intended |
Implications for practice
A particular aim of this strand of work was to consider what approaches might be used to improve patients’ experience of care. Our findings suggest that some basic steps are first required to improve the credibility of survey findings in the minds of GPs and increase their engagement with them. Although the ‘internal’ ambiguity exhibited by GPs, that is, the coexistence of positive and negative views of patient experience surveys, demonstrates the problematic nature of doctors’ engagement with patient experience surveys, it also suggests the possibility of positive change in the future by building on some of the positive views that doctors already hold regarding patients and surveys. In the patient survey context, opportunities exist for managers and lead clinicians to engage in processes aimed at strengthening the plausibility of patient feedback surveys. For GPs to see quality improvement on the basis of patient feedback as plausible, these findings suggest that they would need to be persuaded simultaneously of patients’ evaluative competence and disinterestedness; the possibility of interpreting feedback meaningfully; the ability of survey instruments to take account of contextual factors; the provision of support for doctors receiving negative feedback; and assurance of measures to limit the risk of raising patient expectations (except when it is intended to raise patient expectations). In each of these arenas, as presented in Table 30, potential exists for measures to be taken. By doing so, relevant stakeholders can help to shape GPs’ engagements with patient surveys in more positive directions.
Conclusions
This chapter has explored the ambiguities in GPs’ attitudes to patient experience surveys and has focused on the plausibility of survey findings. Although policy developments over the past decade have increasingly emphasised the importance of patient experience surveys in terms of quality improvement, these findings suggest that this agenda faces significant challenges in terms of doctors’ inconsistent and highly critical engagements with patient feedback. GPs discount patients’ motivations and competence at the same time as emphasising patient-centred care, and undermine the potential for survey-based quality improvement while also highlighting the importance of patient feedback. GPs demonstrated complex and ambivalent attitudes towards the plausibility of patient experience – attitudes that are likely to constrain the potential impact of patient experience surveys on care delivery. In response, we highlight the need for initiatives on the part of managers and lead clinicians to address doctors’ plausibility concerns.
Chapter 9 Understanding high and low patient experience scores: analysis of patients’ survey data for general practices and individual general practitioners
Parts of this chapter are based on Roberts et al. 129 under the terms of the Creative Commons Attribution licence (CC BY 4.0), which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited (http://creativecommons.org/licenses/by/4.0).
Abstract
Background
There is increasing interest in collecting and potentially publishing performance data at an individual practitioner level, in part to enable patients to make informed choices about their care provider. However, UK general practice performance data remain at the practice level, potentially masking important differences between individual practitioners. The aim of this strand of work was to determine the extent to which practice-level scores mask variation in individual performance between doctors within a practice. Additionally, we aimed to determine the test–retest reliability of core items derived from the GP Patient Survey.
Methods
Patient experience surveys were sent to patients who had recently had a face-to-face GP consultation in one of a stratified sample of general practices. In addition, a subsample of patients returning questionnaires were sent a retest questionnaire.
Results
For the main survey, 7721 patients consulting one of 105 GPs across 25 practices returned a questionnaire (response rate 50.9%). The proportion of variance in communication scores attributable to differences between doctors (6.4%) was considerably more than that attributable to practices (1.8%). Higher-performing practices usually included only higher-performing doctors, but lower-performing practices may include doctors with a wide range of communication scores. In the test–retest analysis, 348 patients consulting one of 20 GPs from five practices returned a retest questionnaire (response rate 58.3%). The percentage agreement for categorical items between test and retest ranged from 66% to 100% (kappa coefficients ranged from 0.00 to 1.00). The intraclass correlations for ordinal items averaged 0.67 (range 0.44–0.77).
Conclusion
Aggregating doctors’ communication scores at practice level can mask considerable variation in individual doctor performance, particularly in lower-performing practices. Most of the items derived from the GP Patient Survey have moderate to almost perfect reliability, with performance-related items achieving substantial reliability.
Introduction and rationale
Public reporting of performance at either provider or individual level is increasingly becoming the norm in health care. The approach is proposed to increase accountability, transparency and public engagement. 85,86 It is proposed that offering users the potential to compare their primary care provider with other similar providers may enable patients to make more informed decisions about their care, although evidence in this area is limited. 224 The major source of compiled and published patient feedback and general practice performance scores, the English GP Patient Survey,11 is currently available on websites such as NHS Choices12 and Compare. 225 Making such data publicly available may provide a comprehensive overview of NHS primary care performance, although it is not without controversy. 5
The GP Patient Survey collects patients’ views on the quality of care that they receive from their local GPs, dentists and out-of-hours doctor services. It includes a series of items on the interpersonal skills of the last GP who they saw at their practice (within the previous 6 months). A significant limitation of GP Patient Survey ratings, however, is that items relating to the doctor–patient relationship are reported at practice level, possibly masking considerable performance variation among individual GPs within that practice. Aggregation of ratings may offer both inaccurate reporting of patient views of individual doctor performance and little scope for reflection on the part of GPs about their personal strengths and weaknesses. Current indicators may consequently fail to provide users, providers or commissioners with an accurate assessment of performance within a practice.
Changes to study methods from the original protocol
The aim of this strand of work, as stated in the original protocol, was to estimate the extent to which aggregation of scores to practice level in the national study masks differences between individual doctors (aim 2).
In this chapter, we address two of the original four main objectives for the study:
-
to provide scores for individual doctors, to allow us to estimate the extent to which aggregation of scores to practice level in the national survey masks differences between individual doctors within practices
-
to explore the extent to which patient responses to items used in the GP Patient Survey show stability over time (7–10 days).
In our original application, we envisaged using this strand of work to additionally (1) identify patients for cognitive interviewing (as reported in Chapter 2) and (2) identify patients of South Asian ethnicity for our work on variations in patient experience in minority ethnic groups (as reported in Chapters 2, 5 and 6). In practice, these two objectives were moved to stand-alone studies as a result, in part, of our switch of survey distribution from face-to-face, as originally planned, to postal mode. Pilot work showed that the distribution of a post-consultation survey by our research team in participating practices over a defined data collection period risked a high proportion of ‘missed’ eligible patients and would place a large burden on research staff to accomplish this for > 7000 respondents. Instead, after further pilot work, we undertook a postal survey, working with practices to identify patients who had undergone a face-to-face consultation within the previous 3 weeks (see Methods for full details).
Methods
Twenty-five general practices in Cornwall, Devon, Bristol, Bedfordshire, Cambridgeshire and North London were invited to participate. The aim was to recruit 15 practices from the lowest-scoring 25% of all practices in the 2009/10 GP Patient Survey on a composite case mix adjusted score for the doctor–patient communication items in the questionnaire and five practices scoring in each of the middle and highest quartiles (i.e. 37.5th–62.5th or > 75th percentile). Linear regression models were used to adjust for patient age, gender, ethnicity, deprivation score and self-rated health for case mix. Practices had to have at least two registered GPs and the sample was stratified by practice-level communication score and by GP head count, deprivation index and geographical location. All GPs (working at least four sessions per week and not trainees, short-term locums or currently on extended leave) within each practice were required to be willing to participate. Practices were approached in a randomised order until the quota for each stratum was achieved.
Data collection took place from October 2011 to June 2013. A list of face-to-face GP–patient consultations conducted in the 3 weeks prior to the specified date was extracted from electronic practice records. Practice staff screened lists for recent deaths, terminal illness, those aged < 18 years and mental incapacity. Once the extracted list was screened the remaining patients were sent a patient experience survey accompanied by a letter from the practice, a study information sheet and a prepaid envelope. Repeat consulting patients were sent one questionnaire only, which related to their most recent consultation at the time of extraction. Non-responders were sent one reminder within 3 weeks of the initial mail-out and questionnaires returned up to 100 days after the initial mail-out were accepted.
Fifty completed questionnaires were judged sufficient for obtaining reliable mean communication scores for comparable patient feedback instruments. 226,227 The survey cycle was thus repeated until 50 completed questionnaires for each participating GP were received or until three cycles were complete. Patient consent to take part in the study was inferred by receipt of a questionnaire.
Questionnaire used in the study
The questionnaire used in the study was based on the instrument used in the national GP Patient Survey and asked patients about access, waiting times, opening hours, continuity and interpersonal aspects of care, as well as demographic details, including self-rated health. The advisory group had particular input into the design of the study materials, including the questionnaire. Patients were asked to recall and report on a consultation with a specified GP on a specified date (corresponding to details extracted from practice records) when completing the seven communication items and one confidence and trust item. A mean communication score for the GP from each respondent was calculated from the seven communication items (questions 22a–g; see Appendix 2 for the full questionnaire) for patients providing four or more informative responses.
Test–retest reliability
General practitioners within the five participating practices with the highest response rates from the initial mail-out were selected to take part in the retest phase. Patients returning the test phase questionnaire within 3 weeks of mail-out were sent a retest pack, containing a differently coloured questionnaire, a covering letter and an information sheet. Only retest questionnaires returned within 4 weeks of their initial mail-out were accepted. The gap between completion of the first (test) questionnaire and completion of the retest questionnaire varied between 3 and 49 days; the gap between the consultation and completion of the retest questionnaire varied between 30 and 76 days.
Analysis
Main analysis
The gender balance, proportion of doctors who trained in the UK and mean time since registration in the practice sample, together with questionnaire response rates and intervals between patient consultations and mail out and receipt of questionnaires were described. A two-sample t-test was used to test whether or not intervals between consultation and mail-out were associated with questionnaire responses.
In our study design, groups of individual patient scores are associated with (nested within) individual GPs and groups of GPs are associated with individual practices. Although some variance in patient scores could be attributed to individual experiences, some of the variance was likely to be attributed to GPs, as well as to other aspects of the practices (e.g. reception staff, opening hours). Three-level mixed-effects hierarchical linear models were used to estimate the extent to which variance for each outcome measure was attributable to the differences between practices, between doctors within each practice and between the patients and other residual scores.
The models were adjusted for four self-reported patient attributes shown to be important predictors of reported patient experience: gender, age, ethnicity and self-reported health status. 131 The practice-, doctor- and patient-related variance components from each model were expressed as percentages of the total variance. The ‘best linear unbiased predictors’ of the practice and doctor effects were used to provide estimates of the mean score for each doctor on each of the outcome measures. 228 Corresponding estimates of the mean scores for each practice were elicited from additional models, omitting random effects for doctors. The variation in GP and practice mean scores were described and simple correlation analysis investigated the association between the practices’ mean score and the within-practice standard deviation (SD) of the GPs’ mean scores. The variance components from each model were used to estimate the number of patient scores per doctor needed to achieve a reliability of at least 0.7 or 0.8 for the doctor’s mean score (see Appendix 3 for the formula). Whereas a reliability of ≥ 0.8 is desirable for moderate- to high-stakes assessments,229 a threshold of 0.7 is regarded as acceptable in patients’ assessments of doctors’ performance in some contexts. 230 Stata 10.1 was used for data analysis.
Analysis of test–retest reliability
The response rate and response timings for both test and retest phases were described and the demographic profiles of three groups of patients were compared: those who were sent but did not return a test questionnaire within 3 weeks of mail-out (not eligible for retest), those who were sent but did not return a retest questionnaire within 4 weeks of mail-out and those who returned both test and retest questionnaires within the deadlines. The proportions of non-response by patients eligible to answer each of the 54 separate items were compared between the test and the retest phases using chi-squared tests with a Holm–Bonferroni correction for multiple comparisons. 231 For the 33 categorical response items, the test–retest reliability was measured using raw agreement rates and Cohen’s kappa statistic. 232 Integer scores were assigned to meaningful response options (ignoring ‘don’t know’ or ‘not applicable’ options) for the 21 ordinal response items and intraclass correlation coefficients (ICCs) were calculated. Both ICCs and kappa statistics were interpreted as follows: < 0.00 was judged as ‘poor’, 0.00–0.20 as ‘slight’, 0.21–0.40 as ‘fair’, 0.41–0.60 as ‘moderate’, 0.61–0.8 as ‘substantial’ and 0.81–1.00 as ‘almost perfect’. The mean score on each item in the test and retest phases was calculated and paired sample t-tests using the Holm–Bonferroni correction were used to test possible changes in the mean scores. Data analysis was conducted using SPSS (Statistical Product and Service Solutions) version 18 (SPSS Inc., Chicago, IL, USA).
Results
Of 59 practices initially approached, six were ineligible, nine declined participation and 19 did not respond by the time that the quota (n = 25) was achieved. In total, 105 doctors participated [mean 4.2 (range 2–8) doctors per practice] (Table 31), of whom 46% were female and 80% trained in the UK. The doctors had an average of 19.5 (range 4–38) years’ experience since registration with the GMC. Table 32 provides an overview of responders’ demographics. The mean interval between the patients’ consultation date and questionnaire mail-out was 16.6 (SD 6.0) days and there was no evidence that the interval length was related to the likelihood of a completed questionnaire being returned (two sample t-test, p = 0.157). The overall questionnaire response rate was 50.9% (7721/15,172), with a range of 23.6–80.7% for individual GPs and 24.1–75.5% for practices. In total, 92 out of 105 (87.6%) GPs achieved 50 returned questionnaires. The mean interval between a patient’s consultation and receipt of the completed questionnaire was 35.3 (SD 15.5) days. Questionnaires with fewer than four informative responses to the seven communication items were excluded and scores for the 7429 (96.2%) remaining patients were calculated, with a mean communication score of 87.5 (SD 17.8) on a 0–100 scale.
| Setting | Banding on 2009/10 GP Patient Survey communication scorea | GP head count | Participating doctors | List size (×1000) | Deprivation indexb | Overall response rate (%) |
|---|---|---|---|---|---|---|
| Inner city | Low | 2 | 2 | 6.9 | 26.6 | 37.9 |
| Inner city | Low | 3 | 3 | 5.1 | 48.5 | 36.8 |
| Inner city | Low | 4 | 4 | 5.1 | 36.6 | 37.8 |
| Inner city | Low | 5 | 4 | 7.8 | 26.1 | 50.5 |
| Inner city | Low | 8 | 6 | 8.7 | 32.4 | 43.5 |
| Inner city | Middle | 2 | 2 | 2.5 | 30.1 | 47.0 |
| Inner city | Middle | 3 | 3 | 5.4 | 13.7 | 67.7 |
| Inner city | Middle | 6 | 6 | 8.0 | 39.4 | 32.0 |
| Urban | Low | 2 | 2 | 3.5 | 15.2 | 71.0 |
| Urban | Low | 2 | 2 | 2.9 | 22.2 | 58.9 |
| Urban | Low | 2 | 2 | 3.2 | 29.6 | 24.1 |
| Urban | Low | 3 | 3 | 6.6 | 15.1 | 55.8 |
| Urban | Low | 4 | 4 | 4.1 | 18.3 | 59.3 |
| Urban | Low | 5 | 5 | 12.0 | 27.6 | 58.9 |
| Urban | Low | 5 | 5 | 6.0 | 19.3 | 52.6 |
| Urban | Low | 7 | 6 | 9.7 | 20.0 | 53.8 |
| Urban | Low | 8 | 7 | 16.5 | 14.4 | 45.1 |
| Urban | Low | 9 | 8 | 11.8 | 16.4 | 48.1 |
| Urban | Middle | 3 | 3 | 5.3 | 20.8 | 67.8 |
| Urban | High | 6 | 5 | 8.5 | 22.1 | 47.2 |
| Urban | High | 8 | 8 | 14.2 | 18.9 | 64.4 |
| Rural | Middle | 5 | 4 | 5.1 | 23.1 | 60.5 |
| Rural | High | 3 | 2 | 2.4 | 18.9 | 49.8 |
| Rural | High | 4 | 4 | 5.4 | 11.5 | 75.5 |
| Rural | High | 5 | 5 | 9.1 | 4.8 | 71.7 |
| All | 114 | 105 | 50.9 |
| Characteristic | n (% non-missing) |
|---|---|
| Gender | |
| Female | 4785 (62.4) |
| Male | 2882 (37.6) |
| Missing | 54 |
| Age (years) | |
| < 18 | 5 (0.1) |
| 18–24 | 249 (3.2) |
| 25–34 | 786 (10.3) |
| 35–44 | 983 (12.8) |
| 45–54 | 1150 (15.0) |
| 55–64 | 1474 (19.2) |
| 65–74 | 1550 (20.2) |
| 75–84 | 1171 (15.3) |
| ≥ 85 | 299 (3.9) |
| Missing | 54 |
| Ethnicity | |
| White British | 6138 (81.5) |
| White Irish | 132 (1.8) |
| Any other white background | 459 (6.1) |
| Mixed white and black Caribbean | 23 (0.3) |
| Mixed white and black African | 10 (0.1) |
| Mixed white and Asian | 18 (0.2) |
| Any other mixed background | 19 (0.3) |
| Asian or Asian British – Indian | 169 (2.2) |
| Asian or Asian British – Pakistani | 55 (0.7) |
| Asian or Asian British – Bangladeshi | 71 (0.9) |
| Any other Asian background | 72 (1.0) |
| Black or black British – Caribbean | 95 (1.3) |
| Black or black British – African | 161 (2.1) |
| Any other black background | 9 (0.1) |
| Chinese | 45 (0.6) |
| Any other ethnic group | 57 (0.8) |
| Missing | 188 |
| Health | |
| Poor | 714 (9.5) |
| Fair | 1827 (24.3) |
| Good | 2502 (33.2) |
| Very good | 1961 (26.1) |
| Excellent | 523 (6.9) |
| Missing | 194 |
| All | 7721 |
Twenty doctors from five practices took part in the test–retest substudy. In the test phase 2877 patients who had recently consulted one of the participating GPs from the five practices were sent questionnaires. Retest questionnaires were sent to 597 patients who had returned a completed test questionnaire within 3 weeks of mail-out and 58% (n = 348/597) returned a completed retest questionnaire within 4 weeks. The mean time from mail-out to receipt of a completed questionnaire was 8.7 days in the test phase and 10.1 days in the retest phase. There were no gender differences between test and retest respondents, but retest responders tended to be older and white British (Table 33). No significant differences in item non-response rates between the test phase and the retest phase were found for any of the 54 items.
| Characteristic | Patients sent but not returning a test questionnaire within 3 weeks of mail-out | Patients sent but not returning a retest questionnaire within 4 weeks of mail-out | Patients returning both a test and a retest questionnaire within the deadlines | p-value |
|---|---|---|---|---|
| n | 2009 | 249 | 348 | NA |
| n (%) male | 807 (40.2) | 89 (35.7) | 138 (39.7) | 0.404 |
| n (%) white Britisha | 404 (85.4) | 204 (89.1) | 326 (95.6) | 0.001 |
| Age (years), mean (SD) | 46.2 (18.5) | 59.4 (18.8) | 65.3 (15.1) | < 0.001 |
Main results
For the six outcome measures of interest, most of the variance in patient-level scores resulted from differences in ratings of the same doctor by different patients (Table 34). For both GP communication and trust and confidence in the doctor, the variance resulting from differences between doctors was greater than that attributable to differences between practices; however, the reverse was true for the other four non-doctor-specific measures. Table 35 shows the number of patient ratings required to achieve the 0.7 and 0.8 reliability thresholds for each outcome measure, judged by authorities to represent minimum acceptable thresholds in postgraduate assessment settings. A substantial majority of doctors received sufficient scores to achieve reliable estimates of performance in communication – 103 out of the 105 GPs received at least 27 patients’ communication scores and 95 GPs received ≥ 46 (overall mean 71 scores per doctor).
| Outcome measure | Source of variance (%) | ||
|---|---|---|---|
| Practice | Doctor | Patients and residual error | |
| Communication score | 1.8 | 6.4 | 91.9 |
| Confidence and trust | 0.8 | 5.2 | 94.0 |
| Overall satisfaction with surgery | 6.0 | 1.1 | 92.9 |
| Helpfulness of receptionists | 7.3 | 0.5 | 92.2 |
| Cleanliness of health centre | 10.6 | 0.3 | 89.1 |
| Ease of getting into building | 1.9 | 0.4 | 97.6 |
| Level of reliability | Communication score | Confidence and trust | Overall satisfaction with surgery | Helpfulness of receptionists | Cleanliness of health centre | Ease of getting into building |
|---|---|---|---|---|---|---|
| Reliability of raw mean score | ||||||
| 0.7 | 21 | 30 | 23 | 25 | 15 | 78 |
| 0.8 | 36 | 51 | 38 | 42 | 26 | 133 |
| Reliability of adjusted mean scorea | ||||||
| 0.7 | 27 | 37 | 31 | 28 | 20 | 97 |
| 0.8 | 46 | 63 | 53 | 48 | 33 | 167 |
The estimated mean communication scores for individual doctors and for practices as a whole are shown in Figure 18. This shows the extent to which the variation in mean communication scores between individual doctors (within practices) was greater than the variation between practices and suggests that within-practice variability in doctors’ scores was greater in the lower-scoring practices. We conducted further analysis to confirm this: the within-practice SD of GPs’ mean communication score was negatively correlated with the practices’ mean communication score (Pearson’s r = −0.505; p = 0.010).
FIGURE 18.
Mean communication score (best estimate) by practice and doctor. Note: practices (n = 25) are sorted by their mean communication score. Horizontal shading serves only as a visual separation of the results for different practices. Reliability calculations using variance components showed that achieving acceptable reliability (> 0.7) for GPs’ adjusted mean communication scores with 27 patients’ scores and good reliability (> 0.8) with 46 patients’ scores per doctor is feasible (see Appendix 3 for formula). All but 10 of the 105 participating doctors had > 46 scores; two received < 27 scores (mean 71 scores per doctor). Data for these doctors were retained in the subsequent modelling, as use of best linear unbiased predictors to estimate doctors’ mean scores has a ‘conservative’ effect. When sample sizes are smaller, estimated mean scores are drawn closer to the practice mean. Reproduced from Roberts et al. 129 under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. (http://creativecommons.org/licenses/by/4.0/).
In contrast to Figure 18, Figure 19 highlights the adjusted doctor-level and practice-level mean scores for ‘cleanliness of the practice buildings’, demonstrating the minimal within-practice variability between GPs for this non-doctor-specific measure.
FIGURE 19.
Mean score for cleanliness of practice building (best estimate) by practice and doctor. Note: practices (n = 25) are sorted by their mean score for cleanliness. Horizontal shading serves only as a visual separation of the results for different practices. Reproduced from Roberts et al. 129 under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. (http://creativecommons.org/licenses/by/4.0/).
Results of the test–retest reliability analysis
The percentage agreement in response to the 33 categorical items ranged from 66% to 100% (mean 88%), whereas the kappa coefficients ranged from 0.00 to 1.00 (mean 0.53). Only one item, relating to booking an appointment by fax, achieved a perfect agreement (κ = 1.00) (Table 36). The raw agreement rates were ≥ 80% for 27 of these items. ICCs for the 21 ordinal items averaged 0.67 and ranged from 0.44 for question 9 (‘How easy do you find it to get into the building at this GP surgery or health centre?’) to 0.77 for question 25 (‘Would you recommend this GP surgery or health centre to someone who has just moved to your local area?’). The ICCs for 20 of these items (all items except question 9) were > 0.6, representing substantial test–retest reliability. Mean scores in the retest phase were higher for eight and lower for 12 of the 21 items (Table 37). After applying the Holm–Bonferroni procedure, question 9 was the only item for which a significant difference was found between the mean scores in the test and retest phases (p = 0.001).
| Topic/item | n | Raw agreement (%) | κ |
|---|---|---|---|
| Making an appointment | |||
| Q1a Normally book an appointment in person | 348 | 82 | 0.63 |
| Q1b Normally book an appointment by telephone | 348 | 95 | 0.69 |
| Q1c Normally book an appointment by fax | 348 | 100 | 1.00 |
| Q1d Normally book an appointment online | 348 | 99 | 0.93 |
| Q1e Normally book an appointment by digital TV | 348 | 100 | a |
| Q1f Booking doesn’t apply | 348 | 99 | 0.00 |
| Q2a Prefer to book in person | 348 | 81 | 0.62 |
| Q2b Prefer to book by telephone | 348 | 85 | 0.44 |
| Q2c Prefer to book by fax | 348 | 99 | 0.50 |
| Q2d Prefer to book online | 348 | 93 | 0.79 |
| Q2e Prefer to book by digital TV | 348 | 100 | a |
| Q2f No preference in booking an appointment | 348 | 98 | 0.39 |
| Access to a doctor | |||
| Q4 In the past 6 months, have you tried to see the doctor quickly? | 334 | 82 | 0.49 |
| Q5 Were you able to see the doctor quickly? | 234 | 83 | 0.46 |
| Q6a If you couldn’t be seen quickly was this because there were no appointments? | 348 | 83 | 0.39 |
| Q6b If you couldn’t be seen quickly was this because the times did not suit you? | 348 | 97 | 0.46 |
| Q6c If you couldn’t be seen quickly was this because the appointment was with a doctor you didn’t want to see? | 348 | 94 | 0.44 |
| Q6d If you couldn’t be seen quickly was this because the appointment offered was with a nurse and you wanted to see a doctor? | 348 | 99 | 0.46 |
| Q6e If you couldn’t be seen quickly was this because you were offered an appointment at a different branch? | 348 | 98 | 0.44 |
| Q6f If you couldn’t be seen quickly was this because there was a different reason? | 347 | 98 | 0.43 |
| Q6g Can’t remember why you were unable to be seen quickly | 348 | 97 | 0.43 |
| Q7 In the past 6 months, have you tried to book ahead for an appointment with a doctor? | 339 | 79 | 0.44 |
| Q8 Were you able to get an appointment with a doctor more than 2 weekdays ahead? | 239 | 73 | 0.40 |
| Arriving at the appointment | |||
| Q11 In the reception area, can other patients overhear what you say to the receptionist? | 339 | 80 | 0.59 |
| Continuity of care | |||
| Q15 Is there a particular doctor you prefer to see? | 338 | 91 | 0.68 |
| Q17 Was your consultation with your preferred doctor? | 254 | 89 | 0.55 |
| Opening hours | |||
| Q19a As far as you know is the surgery open before 08.00? | 330 | 75 | 0.46 |
| Q19b As far as you know is the surgery open at lunchtime? | 309 | 71 | 0.49 |
| Q19c As far as you know is the surgery open after 18.30? | 307 | 66 | 0.47 |
| Q19d As far as you know is the surgery open on Saturdays? | 309 | 80 | 0.42 |
| Q19e As far as you know is the surgery open on Sundays? | 308 | 85 | 0.38 |
| Q20 Would you like the surgery to be open at additional times? | 313 | 83 | 0.57 |
| Q21 Which additional time would you most like your surgery to be open? | 111 | 77 | 0.49 |
| Topic/item | n | ICC (95% CI) | Mean difference (95% CI) | p-valuea |
|---|---|---|---|---|
| Telephone access | ||||
| Q3a How easy have you found getting through on the telephone? | 333 | 0.73 (0.67 to 0.78) | –2.40 (–4.91 to 0.11) | 0.061 |
| Q3b How easy have you found speaking to a doctor on the telephone? | 191 | 0.68 (0.59 to 0.75) | –4.01 (–7.64 to –0.39) | 0.030 |
| Q3c How easy have you found speaking to a nurse on the telephone? | 82 | 0.63 (0.48 to 0.75) | –2.85 (–8.62 to 2.93) | 0.330 |
| Q3d How easy have you found getting test results on the telephone? | 131 | 0.62 (0.51 to 0.72) | 0.25 (–3.88 to 4.39) | 0.903 |
| Arriving at the appointment | ||||
| Q9 How easy do you find it to get into the building at this GP surgery or health centre? | 345 | 0.44 (0.35 to 0.52) | 2.32 (0.94 to 3.70) | 0.001 |
| Q10 How clean is this GP surgery or health centre? | 344 | 0.60 (0.53 to 0.66) | 1.16 (–0.10 to 2.42) | 0.070 |
| Q12 How helpful do you find the receptionists at this GP surgery or health centre? | 335 | 0.69 (0.63 to 0.74) | –0.60 (–2.39 to 1.20) | 0.514 |
| Q13 How long after your appointment time do you normally wait to be seen? | 315 | 0.67 (0.60 to 0.73) | –0.95 (–2.60 to 0.70) | 0.257 |
| Q14 How do you feel about how long you normally have to wait? | 308 | 0.70 (0.64 to 0.75) | –2.11 (–4.43 to 0.21) | 0.074 |
| Continuity of care | ||||
| Q16 How often do you see the doctor you prefer? | 255 | 0.71 (0.64 to 0.77) | –0.78 (–3.49 to 1.92) | 0.568 |
| Opening hours | ||||
| Q18 How satisfied are you with the hours that this GP surgery or health centre is open? | 325 | 0.65 (0.59 to, 0.71) | 2.23 (0.40 to 4.06) | 0.017 |
| Doctor–patient communication and trust | ||||
| Q22a How good was the doctor at giving you enough time? | 337 | 0.62 (0.55 to 0.68) | 0.45 (–0.96 to 1.85) | 0.532 |
| Q22b How good was the doctor at asking about your symptoms? | 317 | 0.70 (0.64 to 0.75) | –0.47 (–1.84 to 0.90) | 0.498 |
| Q22c How good was the doctor at listening to you? | 331 | 0.72 (0.66 to 0.77) | 0.38 (–0.88 to 1.63) | 0.554 |
| Q22d How good was the doctor at explaining tests and treatments? | 275 | 0.72 (0.65 to 0.77) | –1.27 (–2.81 to 0.26) | 0.104 |
| Q22e How good was the doctor at involving you in decisions about your care? | 275 | 0.68 (0.61 to 0.73) | –1.00 (–2.65 to 0.65) | 0.233 |
| Q22f How good was the doctor at treating you with care and concern? | 326 | 0.67 (0.61 to 0.73) | 0.23 (–1.16 to 1.62) | 0.745 |
| Q22g How good was the doctor at taking your problems seriously? | 324 | 0.72 (0.67 to 0.77) | –0.08 (–1.46 to 1.31) | 0.913 |
| Q23 Did you have confidence and trust in the doctor you saw? | 340 | 0.70 (0.64 to 0.75) | –0.15 (–1.86 to 1.57) | 0.866 |
| Overall satisfaction | ||||
| Q24 In general, how satisfied are you with the care you get at this surgery or health centre? | 344 | 0.74 (0.69 to 0.78) | –0.58 (–1.81 to 0.65) | 0.353 |
| Q25 Would you recommend this GP surgery or health centre to someone who has just moved to your local area? | 333 | 0.77 (0.73 to 0.81) | 0.00 (–1.51 to 1.51) | 1.000 |
Discussion
Our findings show that the measurement of patient experience at the practice level may mask considerable variation between doctors within the same practice. These findings are in line with other studies showing that the proportion of variance due to doctors is greater than that due to practices in the case of doctor-specific measures and is less than that due to practices in the case of non-doctor-specific measures. 22,233,234 For indicators that are more likely to be under the control of the doctor (e.g. doctor–patient communication), more variance is explained by doctors than by practices; this may be taken as a validation of the use of these indicators to measure individual GP performance. Our findings additionally demonstrate that higher-performing practices usually include higher-performing doctors, but lower-performing practices may include doctors with a range of communication scores. This has important implications for evaluating practice performance, as GPs requiring support to improve their communication skills are unlikely to be identified using current practice-level approaches. As such, the current practice-based performance indicators may not provide meaningful information to commissioners, providers or users for key domains such as communication skills. However, other indicators observed to have more variance at practice level (e.g. cleanliness of a practice) are more suitable for evaluating performance at an organisational level.
Our test–retest reliability results demonstrated good to almost perfect agreement on a number of items used in the GP Patient Survey and included within our patient experience questionnaire. Patients’ willingness to recommend their practice to a friend or family member showed substantial reliability and items orientated to staff performance also had substantial stability. Items regarding the physical environment of the practice, such as ease of access and cleanliness, demonstrated moderate to substantial reliability.
The results suggested that, despite the high proportion of patient-level variance in communication scores, a reliable (> 0.8) adjusted mean score for individual doctors can be obtained with 46 patient scores per GP using this instrument, so little variance in reported doctor-level scores was attributable to patients and residual sources, which is in line with other published work. 226 With sample sizes smaller than this, a trade-off must be made between reliability and the utility of conducting individual- rather than group-level evaluations.
Communication is a key driver of overall patient satisfaction102 and ensuring patients’ ability to access accurate information on performance is important if they are expected to make informed choices among providers, as current policy aspires to. In compiling performance indicators to inform patients’ choice of provider, it would therefore be preferable to report communication scores at the individual practitioner level or to reliably report the range of individual practitioner scores within an organisation. If the aspiration is to use quality indicators to identify poor performance, rather than to inform patient choice, an alternative to the potentially costly option of obtaining communication scores for all individual practitioners could be to use organisation-level assessments (such as the current GP Patient Survey) to screen for lower-performing practices. Individual-level assessments could then be targeted only to organisations in which performance concerns were identified. Further research to explore users’, providers’ and commissioners’ perceptions about the feasibility of alternative approaches to generating performance data on doctor–patient communication would be useful. Furthermore, clarity about the association between the publication of performance data and quality improvement, including the mechanisms underpinning the instigation of any personal or organisational changes, is needed.
Strengths and limitations
This was a large study, including responses from 7721 patients relating to 105 doctors across 25 practices and resulting in the production of a first report on the stability of patient responses on items used within the GP Patient Survey over time. The stratified sampling strategy ensured participation from doctors with a range of summary scores for interpersonal skills, after adjusting for case mix, which improves generalisability to wider primary care contexts. The use of the postal survey resulted in an average delay of just over 2 weeks between patients’ consultation and receipt of their questionnaire, which is substantially less than for the national GP Patient Survey. The 2-week delay is unlikely to produce significant recall bias and would be expected to affect all participating doctors equally.
The response rate for the main study was considerably higher (51%) than response rates achieved in the national GP Patient Survey (which have ranged in recent years from 38% in 2009/10 to 35% in 2012/13), although there was substantial variation in response rates between participating GPs. The response rate for the test–retest substudy was similar to that observed in other primary care test–retest exit surveys. 118,226 Non-response tends to inflate doctors’ and practices’ scores, but this inflation is largest when non-response is highest. 235 As the lowest response rates were inclined to occur for lower-scoring practices, any non-response bias tended to attenuate the extent of variance between both doctors and practices, rather than inflate it. The estimated magnitudes of such effects were small and it was not expected that the resulting variance at the practice and doctor levels, or the conclusions regarding the comparison of doctor- and practice-level variances, would alter.
Sampling practices from different quartiles of the GP Patient Survey practice-level communications scores may mean that the estimate of the total practice-level variance could differ slightly from that of the full population. However, this is not expected to affect the conclusions regarding the relationship between practice-level scores and the extent of within-practice variation. The analysis was not adjusted for neighbourhood-level deprivation, as the research team was blinded to patients’ postcodes. This limitation is unlikely to have biased the results as deprivation has only a very small association with patients’ experience after controlling for gender, age, ethnicity and health status. 131 Although the sample size of this study was considerably larger than that used for the General Practice Assessment Survey,236 responding patients were not fully representative of the general patient population of England and Wales.
Conclusions
Currently, evaluations of GPs’ communication performance most commonly report indicators at a practice level, rather than enabling patients and stakeholders to evaluate individual practitioners directly. Reporting communication-related performance indicators at practice level may mask large variation between individual practitioners. Practice-level surveys could potentially act as an initial screen for concerns about performance, with subsequent data gathering focusing on individual doctor-level surveys in lower-performing practices.
Chapter 10 Exploratory trial of a real-time feedback intervention to improve patient experience in general practice
Parts of this chapter are based on Wright et al. 237 and Carter et al. 238
Abstract
Background
Our early findings and the published research evidence suggested that an intervention seeking to improve patient experience in general practice should consider the level at which feedback from patients might be provided to practice teams, whether or not such feedback should be facilitated and the need for timeliness of feedback. The aim of this project was to conduct a feasibility study and an exploratory trial of an intervention that might inform change and improve patient experience in general practice.
Methods
We designed a feasibility and pilot clinical trial. RTF touch screens were installed in practice waiting areas for 12 weeks. Practices or individual doctors received fortnightly patient feedback summaries. Some teams attended a facilitated reflection session. We undertook a multimethod evaluation of the intervention.
Results
In total, 2.5% of consulting patients provided RTF (range 0.7%–8.0% across eight practices). Men and patients aged > 65 years were under-represented among responders. Reception staff often interacted with patients but rarely encouraged touch screen use. When staff did encourage patients to use the touch screen, 36 out of 60 (60%) patients attempted to start the survey. Most patients were positive about RTF but identified a range of barriers. Staff views of and engagement with RTF varied. Within-team communication influenced perceptions, and the successful implementation and use, of RTF. Costs ranged from £1125 (unfacilitated/team-level feedback) to £1887 (facilitated/team- + practitioner-level feedback).
Conclusions
The successful implementation of RTF requires team engagement, shared responsibility and careful communication. Future studies need to make RTF accessible to a wider range of patients and ensure that questions presented to support RTF are relevant to practices. Shorter, repeated episodes of RTF collection may be of greater utility to practices and to researchers seeking to evaluate the approach than sustained and ongoing RTF.
Introduction and rationale
In the original outline of this programme of research we planned to undertake initial feasibility testing and piloting of an intervention seeking ultimately to improve patient experience of care. We anticipated that we would draw on our earlier research findings to inform the design and implementation of this project, conducted towards the end of the programme of work. In exploring what might constitute a suitable intervention, we prioritised the area of doctor communication as being one of vital importance to patients. Cheraghi-Sohi et al. 57 had previously highlighted communication between doctors and their patients as being a central priority for patients in their assessment of what high-quality care might look like. Furthermore, our preliminary research had identified that communication, rather than access, was a key driver of patients’ overall satisfaction with care. 102 Given these observations, we undertook a review of the literature seeking to identify potentially promising interventions that had targeted doctor communication as a primary consideration and that might inform the design of an intervention study that would also incorporate findings arising from our early research in this programme.
Review of the evidence on interventions to improve communication skills in primary care
In 2008, a systematic review assessed the efficacy of feedback of real patient assessments of interpersonal care skills or brief training focused on the improvement of interpersonal care. 83 Of the nine RCTs found (two patient-based feedback studies and seven brief training studies), only one feedback study (involving trainee GPs) and one training study (conducted in 1987) reported a significant positive effect. The review concluded that:
[T]he interventions to be tested in future research should consider using insights from the wider literature on communication outside primary care, might benefit from a clearer theoretical basis, and should examine the use of combined brief training and feedback to improve physicians’ interpersonal skills.
Cheraghi-Sohi and Bower, p. 17983
We updated this review by repeating the search strategy in the Cochrane Central Register of Controlled Trials (CENTRAL) from 2007 onwards (see Appendix 4). We searched for studies that fit the following criteria:
-
RCTs involving primary care practitioners and their patients
-
involving one or both of the following interventions:
-
feedback of assessments of patients on the interpersonal skills of clinicians
-
‘brief’ (up to 1 working week) training focused on interpersonal care
-
-
with a patient-based assessment of change in interpersonal skills as an outcome.
Of 1610 studies returned in the search, only one met all of the criteria (Table 38). 239 Haskard et al. 239 assessed the effect of a communication skills training programme for both patients and doctors. The study involved 156 doctors from three primary care specialties (obstetrics/gynaecology, family medicine and internal medicine) in the USA. Data were collected between 1996 and 1998. The clinical training programme involved three 6-hour interactive workshops conducted on a monthly basis, covering core communication skills and concepts including recognising interpersonal difficulties and tensions in doctor–patient relationships. Additionally, clinicians received three 30- to 45-minute coaching sessions involving the review of videotaped consultations. The patient training programme was a 20-minute preconsultation intervention involving an audio CD and booklet concerning planning and organising concerns and questions to ask the doctor. There were four experimental groups:
-
neither doctors nor patients trained (doctors, n = 39, control group)
-
doctors only trained (doctors, n = 41)
-
patients only trained (doctors, n = 38)
-
doctors and patients trained (doctors, n = 38).
| Study and setting | Target population and n | Intervention | Patient satisfaction measure and other outcomes | Summary of findings |
|---|---|---|---|---|
| Haskard 2008239 USA; West Coast University medical centre (n = 93); Veterans Affairs clinic (n = 5); staff model health maintenance organisation (n = 58) |
Physicians from three primary care specialties (obstetrics/gynaecology, family medicine, internal medicine) Physicians, n = 156; patients in interaction, n = 2196 |
3 months of physician workshops and coaching with assessments (6 hours at each time point), with previsit intervention at preceding time 2 | Patient satisfaction and perceptions of choice, decision-making, information and lifestyle counselling; physicians’ satisfaction and stress; global ratings of the communication process (all composite measures) No other secondary measures reported |
Physician training significantly improved patients’ satisfaction with information and overall care; increased willingness to recommend the physician; increased physicians’ counselling (as reported by patients) about weight loss, exercise and quitting smoking and alcohol consumption; increased physician satisfaction with physical examination detail; increased independent ratings of physicians’ sensitive, connected communication with their patients; and decreased physician satisfaction with interpersonal aspects of professional life. Patient training improved physicians’ satisfaction with data collection. If only physician or patient was trained, physician stress increased and physician satisfaction decreased |
| Reinders 2010211 Vrije Universiteit University Medical Centre, Amsterdam, the Netherlands |
First-year general practice trainees (GPTs) n = 53: intervention, n = 23; control, n = 30 |
Patient feedback training programme (how to acquire relevant patient feedback to improve communication skills). Instructions to staff delivered in a 2-hour meeting followed by half a day’s instruction meeting, and then 3 months to obtain feedback from 20 patients | Patient feedback questionnaire on consultation skills (PFC) Primary outcome measure MAAS-Global assessment used by trained assessors (five behavioural scientists and three GPs) to assess video-taped consultations with three SPs (standardised patients) in six consultation scenarios Process outcomes: intensity of GPT participation in the programme based on the number of PFCs that the GPT collected; number of learning points formulated, etc. In addition, GPTs completed an evaluative questionnaire and National Knowledge Test in General Practice Medicine |
Consultation skills in the entire cohort of participants improved, with a small to moderate effect size within the 3-month observation period. Consultation skills in the intervention group did not improve any more than those in the control group. A subgroup of GPTs who participated ‘actively’ (i.e. high intensity) in the programme showed a greater improvement in consultation skills than those who did not actively participate |
Overall, doctor training improved doctors’ information giving and lifestyle health behaviour counselling, and increased patients’ quality of care ratings and their willingness to recommend the physician. However, doctors’ satisfaction with the interpersonal aspects of their professional life decreased significantly more among trained than among untrained physicians. Training both doctors and patients had complex effects on doctors’ satisfaction and stress: interaction effects reflected a relative increase in stress and decrease in doctor satisfaction when only one (either doctor or patient) was trained. The authors noted that the intervention was intensive and may have placed additional stress on doctors, some of whom were also experiencing organisational changes at the time. 239
One other study assessed the effect of patient feedback on communication competencies, but used expert raters’ (rather than patients’) assessments of skills as the outcome211 (see Table 38). This was a trial of a patient feedback training programme in first-year GP trainees in the Netherlands. The intervention group (n = 23) received instruction in how to obtain patient feedback in daily practice using the patient feedback questionnaire on consultation skills (PFC), which focuses on GP–patient communication. Following training, GP trainees in the intervention group were asked to obtain feedback using the PFC from 20 patients over a period of 3 months; they also completed a self-assessment version and compared this with the patient version and ‘formulated learning points which they discussed with their GP trainers’ (p. 158). The control group attended the regular doctor–patient communication skills training. For the purposes of this study, simulated patients, trained to enact six consultations of moderate complexity, visited the 53 GP trainees’ practices and video-taped consultations with the GP trainees. Video-taped consultations were then assessed by eight raters (five behavioural scientists; three GPs) using the MAAS-Global instrument. Data on 50 GP trainees were available for analysis. Both the control group and the intervention group improved their consultation skills between baseline (when scores were already high) and the post-intervention assessments, but there were no significant differences in improvement between the control group and the intervention group. However, there was a trend for intensity of participation in the patient feedback programme to predict greater improvement in MAAS-Global scores.
One of the above studies211 was included in a systematic review of the effect of patient feedback on physicians’ consultations skills. 204 This searched for all empirical studies involving practising doctors (including postgraduate trainees) that incorporated feedback from real patients; assessed physicians’ general consultation skills; incorporated feedback on communicative aspects in general health care; and evaluated physicians who received formal, individually directed feedback from patients (e.g. by means of aggregated patient reports or educator-mediated coaching sessions).
Of 15 studies included in the review (from 1980 to 2010), 10 were in primary care and five in other specialties. A variety of study designs were included (RCTs, quasi-experimental studies, cross-sectional studies and qualitative studies). Twelve studies observed a positive effect of patient feedback on physicians’ consultation skills. In an assessment of the outcomes of studies against the Kirkpatrick hierarchy240–243 (four levels at which educational interventions can have an effect), they found that:
-
All nine studies that evaluated level 1 effects (valuation or views of the learning experience) reported positive effects.
-
All four studies that evaluated level 2 effects (change in knowledge or skills) reported positive effects.
-
All three studies that evaluated level 3 effects (change in intended behaviour) reported positive effects.
-
Four of seven studies that evaluated level 4 effects (change in actual performance or outcomes) reported positive results.
Despite the apparently positive results, the authors argued that ‘consulting skills’ need to be much better defined in studies of this type. Additionally, observed effects cluster at the lower end of Kirkpatrick’s hierarchy, mostly in qualitative, non-randomised studies. Actual change in performance was rarely observed. Three possible reasons were offered for the observed heterogeneity of findings:
-
Assessing actual change in general consultation skills or clinical performance may be difficult because of the lack of precision in defining ‘consulting skills’ and the lack of responsiveness of the assessment instruments.
-
Patients who have poor experiences might not report a poor outcome, limiting the effect of patient-reported outcome measures (witness the ceiling effects in many patient feedback questionnaires).
-
There may be a true absence of effect – interventions were not sufficient to drive behavioural change or doctors were not susceptible to change.
They concluded that there is a dearth of evidence showing that patient feedback has any effect on actual behaviour.
Modelling the intervention
One of the striking conclusions of our review of the previous empirical work in this area was the inconsistency in the findings of the major studies in primary care. Two of the biggest trials report opposite results, with Greco et al. 118 reporting positive effects of feedback on GP registrars and Vingerhoets et al. 207 reporting no benefits in established doctors. In considering these contradictory findings, we identified two potentially important contextual factors that may moderate the relationship between intervention and outcome: (1) the training and experience of the doctors receiving the intervention and (2) motivators to change (Figure 20). There is an assumption that GP communication skills can be developed at some point in the medical career, but that change becomes less likely as doctors develop a routine way of consulting, as reflected in our conceptualisation. 244
FIGURE 20.
Typology of GPs’ potential responses to communication training.
Discussions within the team highlighted other potential moderators, particularly the context within which any intervention might be introduced. Notably, at the time of the original programme grant application, the major contextual factor was the priority being afforded to survey results by the then PCTs, with the expectation that they would be engaged in project management of practices against GP Patient Survey scores. With the reorganisation of commissioning and care, this was no longer the case, and our experience with practices during the previous phases of work suggested that, in the absence of external drivers, there was little support or capacity for the kind of intensive communication interventions that had been previously trialled.
We identified a number of other contextual factors that might impact on response:
-
case mix (practices serving certain patient populations or in certain areas may respond differently)
-
incentivisation of communication training and patient feedback
-
previous experience and engagement with patient feedback at a practice level
-
length of consultations and the organisation of practices in response to the QOF may facilitate or hinder change
-
the priority placed on access as opposed to continuity
-
practice culture and communication.
Additionally, a further issue might be that educational interventions often assume that the individual practitioner is the correct ‘unit of intervention’, but it is possible that the practice may be a more important unit. However, little is known about how practices understand variation in communication quality between practitioners or how they respond. Response may relate to complex issues around ‘sense making’, identity and the perception of legitimate work and to clinical etiquette around acceptable topics for discussion and learning among practitioners. 245,246
The impact of any intervention linked to communication also needed to be seen in the context of the large number of other QOF and non-QOF issues that provide competing priorities for GP time and attention.
In reflecting on the evidence to date and our own emerging findings from previous phases of work, we identified three key questions that would shape our intervention:
-
the level at which patient feedback takes place: for practices or for individual doctors
-
the requirement for feedback to be facilitated or not
-
the timeliness of feedback.
We explore these in more detail in the following sections.
The level of feedback: practices or individuals?
Findings from the patient survey at the practice and individual GP level (see Chapter 9) identified the importance of the issue of whether any suitable potential intervention should be targeted at the practice level or at the level of the individual doctor. Through our survey, we had identified that patient feedback aggregated to practice level may mask a range of individual levels of doctor performance within practices, especially in those practices that scored at the lower end of overall practice scores (e.g. in respect of patients’ experiences of communication). In addition, it was unclear to us whether summarised patient feedback for clinicians was most effectively given in group or individual doctor settings. Given our developing interest in whether doctors or their teams were the important units for any potential intervention, we drew on our knowledge of the research literature relating to the effective functioning of teams.
One team-level attribute that can be measured and potentially manipulated to change behaviour and bring about improvements in an organisation’s effectiveness is ‘team climate’. 247,248 Anderson and West249 define team climate as ‘a team’s shared perceptions of organisational policies, practices and procedures’ (from Schneider250). Four group processes (or facets) of team climate have been proposed to be important prerequisites for improved quality of health care:249,251 (1) team vision and objectives – team members’ views on the clarity, sharedness and attainability and value of the team’s objectives; (2) participatory safety – team members’ participation in information sharing and decision making and psychological safety and support (e.g. in trying out new ideas); (3) task orientation –team members’ emphasis on reflection on appraisal, feedback and performance monitoring of their work; and (4) support for innovation – perceptions of articulated and enacted support in applying new ideas and change. Studies in general practice in the UK252,253 and Australia254 have reported that a favourable team climate is associated with improved standards of care for a range of long-term conditions, better access, higher patient satisfaction, higher staff satisfaction and greater perceived team effectiveness. A Finnish study of team climate in hospital settings reported that a favourable team climate was associated with a lower turnover of health-care staff,255 although two more recent UK studies in general practice256,257 questioned the previously observed relationship between team climate and quality of care, with the authors arguing that further research focusing on the associations between team functioning and quality of care is needed.
Supported feedback to practices: facilitated or unfacilitated?
Findings from our earlier qualitative research (see Chapter 7) identified a further important consideration – the perceived lack of support for doctors or practice teams in making sense of findings emanating from patient feedback, suggesting the potential benefit of facilitation in respect of the process of reviewing patient feedback.
Timely feedback: the potential of real-time data acquisition and reporting
Collecting patient feedback is insufficient on its own to improve services and best practice guidance47,92,258 suggests that organisations need to reflect and act appropriately on the feedback while it is still ‘fresh’. Such guidance also suggests that (1) by implementing change based on continuous real-time data, organisations can monitor whether or not the changes that they implement have an effect on patients’ experience; (2) organisations should be prepared to show patients how their RTF has been used to change services; and (3) sharing this information suggests that the organisation is willing to listen and respond to patient views and that this, in turn, may mean that patients will be more willing to give their views in the future. Quite apart from the published guidance, some doctors in our early qualitative research (see Chapter 7) had identified concerns regarding the timeliness of data capture and reporting as being a potential impediment to doctors and their teams in taking action in response to patient feedback.
Recent years have seen a substantial move towards the incorporation of real-time technologies to support the acquisition of patient feedback data. RTF involves the systematic collection, analysis and reporting of information from patients who have recently used a health-care service. The approach typically uses kiosks or hand-held electronic devices (e.g. tablets) at the point of care to capture patients’ feedback about their experiences on a continuous basis. The information collected is regularly collated and reported back to service providers to inform and support service improvement. RTF offers organisations an opportunity to improve their services by designing and delivering services to meet patients’ preferences in terms of quality and content; it also enables patients to shape the services that they use.
According to best practice guidance,47,92,258 the collection of RTF requires careful planning, co-ordination and monitoring to ensure that response rates are maximised, assess whether or not the patients who provide feedback are representative of the practice population as a whole and ensure that patients are kept informed of the purpose of RTF and receive adequate practical support with the process of feeding back. One US study259 used electronic touch screen kiosks to obtain feedback, with primary care clinic staff directing patients to the kiosk after their consultation. This approach achieved a 50% response rate and did not adversely affect waiting times or other aspects of the practice routine. Male patients were as likely as female patients to use the kiosk, but older people and ethnic minority groups were less likely to use the facility. In 2009–10, a 6-month pilot study47 was carried out across 22 general practices in England to determine whether or not real-time patient feedback could be used to help practices to understand their patients’ views on services, identify opportunities to improve services and evaluate whether or not any changes that the practices made were effective. Three devices were piloted to collect patient feedback (tablet PC, kiosk and desktop device) and participating practices varied in size, patient list, staffing levels, geography and demography. The key findings were that RTF could be implemented successfully in most general practices, that RTF could drive performance improvement in this setting and that RTF has the potential to complement findings from the national GP Patient Survey but needs to be actively promoted to fully engage patients and staff.
Step-by-step guidance informed by the pilot study47 has since been produced to provide practical advice to general practices that wish to gather and use real-time patient feedback effectively. A number of experienced real-time technology suppliers exist in the UK, including Dr Foster’s (London, UK; Patient Experience Tracker or PET), the Picker Institute (Oxford, UK; Frequent Feedback service) and Customer Research Technology (CRT) Limited (Coventry, UK; ViewPoint system). Previous work highlights the need to monitor response rates and the representativeness of patients who provide RTF. For example, if hand-held devices are being handed out, some patients may be intentionally excluded, including those who are perceived as being likely to provide negative feedback or those who need extra help to feed back because of language barriers or disability. Some patient groups may find kiosks or hand-held devices less user-friendly and, therefore, decline to provide feedback if there is no assistance readily available. Others may be in a rush to leave the practice and/or reluctant to queue to leave their feedback.
Towards a clinical trial
Our early work therefore suggested testing an intervention focusing on practices in the first instance, as these were the unit of reporting for GP Patient Survey scores. But within that ambition, there appeared to be a clear need to examine issues relating to the level at which to target feedback discussions (group or individual) and to consider whether such feedback should be facilitated or unfacilitated. As we had originally been commissioned on the basis of undertaking a trial, internal discussions within the team identified a further important consideration – the need to consider incorporating a control group – in this case, practices who would not, as part of our trial, receive any intervention that we were testing.
Taken together, these observations suggested the potential of undertaking an intervention at practice level, using real-time data collection as the means of capturing patient feedback, and exploring within the same study the potential for group or individual feedback of results, using facilitated or unfacilitated modes of feedback delivery. Given the timing of the research – coming at the end of seven projects – we were pragmatic in our consideration of the number of practices with which we could reasonably work. We felt that a reasonable target was to undertake feasibility and pilot testing using a randomised design in a total of 10 practices, as outlined in Practice sampling and recruitment.
Changes to study methods from the original protocol
The aim of this strand of work, as stated in the original protocol, was to carry out an exploratory RCT of an intervention to improve patient experience, using tools developed in earlier parts of the programme (aim 6).
Within the original protocol, the exact nature of the exploratory trial was therefore undefined, although we outlined three key objectives of this workstream:
-
to develop a model based on theory and published empirical evidence that relates patient assessments of interpersonal care to professional behaviours and outcomes
-
to use that model and the views of key stakeholders from earlier workstreams to develop an intervention to improve interpersonal behaviour that is feasible and acceptable in UK primary care
-
to conduct an exploratory trial of that intervention to:
-
test methods for the recruitment of practices and patients
-
test the implementation of the combined feedback and training in practice
-
provide estimates of the effect of the intervention for sample size calculations.
-
The exact nature of the exploratory trial that we devised as a result has been broadly outlined already in this chapter and is detailed in the following sections.
Aims and objectives
The aim of this workstream was to conduct, in a small number of general practices, a feasibility study and an exploratory trial of a RTF based intervention that might inform change and improve patient experience in general practice. Although neither phase of this work was sufficiently powered to investigate the effectiveness of the RTF based intervention or its various components, the workstream sought to:
-
pilot a RTF intervention for general practices
-
evaluate, within the context of a pragmatic survey embedded in routine practice, and from the perspective of practice teams, the feasibility and acceptability of collecting and receiving RTF with/without a facilitated reflection session
-
evaluate, from the perspective of patients, the feasibility and acceptability of providing RTF, including (a) estimating the number and proportion of patients using RTF touch screens when these are available in practice waiting areas; (b) describing the characteristics of consulting patients who use RTF touch screens and contrasting these with the characteristics of all consulting patients during the same time period; and (c) obtaining the views of patients who used/did not use RTF touch screens
-
estimate the costs associated with the RTF intervention from the perspectives of the NHS (cost of touch screen equipment, training and staff time).
The workstream had two phases:
-
feasibility study (January to June 2014) involving two general practices and designed to develop the RTF intervention and research methods
-
exploratory trial (July 2014 to February 2015) involving 10 general practices to address the above objectives.
Methods
Practice sampling and recruitment
A similar approach for sampling practices was used in both phases of the study. Practices that fell in the lowest 50% of scores on the national GP Patient Survey communication items (year 7 data11) were eligible to participate. To facilitate fieldwork, practices within reasonable travelling distance of the research centres were prioritised in the initial sampling frame. In the feasibility study, two practices were purposively recruited from the South West of England to represent contrasting geographical contexts (urban/rural).
In the exploratory trial, invitations were posted to 16 practices in the South West and 11 practices in Cambridgeshire and were followed up by telephone calls from the local researcher. Detailed briefing sessions were organised with staff at practices that expressed an interest. The practice manager or lead GP provided written consent on behalf of the practice team and a practice profile questionnaire was completed (providing background information about the practice).
Staff surveys
All practice staff were invited to complete a postal survey at two time points: (1) before RTF touch screens were installed (‘baseline’) and (2) after the 12-week RTF implementation period (‘follow-up’). Each team member was allocated a unique study ID number so that his or her completed questionnaires could be matched. Reminder packs were sent to non-responders approximately 2 weeks after delivery of the initial survey pack.
The baseline and follow-up questionnaires included demographic and contextual information (age, gender, ethnic origin, role within the practice) and the Value of Patient Feedback (VOP) scale. The VOP scale was developed within the IMPROVE programme specifically to measure staff attitudes towards patient feedback within this exploratory trial. The availability of a robust approach to evaluating perceptions of the utility and impact of patient feedback was central to our assessment of the engagement of health-care professionals with patient experience data and the likely impact of such information on professional practice. However, a search of the literature found no suitable approach to achieving this. We therefore drew on standard scale development processes to derive and test a new instrument. We give a brief overview of this process below but for full details of the development of the instrument see Appendix 5.
First, we derived key constructs using qualitative data that we had previously collected, both within the programme and prior to the programme in other patient experience research. This gave us a body of data to draw on consisting of interviews with 40 GPs and 14 focus groups with primary care practice staff concerning the impact and utility of patient experience surveys, which we supplemented with a review of relevant literature in the area. From this, we developed a pool of 56 potential items. Following expert panel review (n = 6), 52 items were retained for further consideration. We undertook cognitive testing through interviews with clinicians (n = 7) and items were further reduced to 43, with textual amendments. Pretesting of all 43 items took place using an online survey of doctors and nurses (n = 215). Item reduction was undertaken on the basis of participant feedback and performance of the items in polychoric correlation matrices. We also undertook exploratory factor analysis resulting in further item reduction. A proposed 16-item version of the VOP scale was pilot tested in a survey of hospital doctors (n = 108) and GPs and practice nurses (n = 78) to inform confirmatory factor analysis. The final version of the scale used within the exploratory trial described here consisted of 16 items with five-point Likert-type rating scales (‘strongly agree’ to ‘strongly disagree’) (Table 39). Possible scores on the VOP range from 0 to 80.
| 1. Have you ever received structured patient feedback (such as through patient surveys)? □ At an individual level (e.g. through a report of patient feedback specific to the care you have provided) □ At an organisational level (e.g. through a report of patient feedback aggregated for your practice or clinic) □ I have never received structured patient feedback (such as through a patient survey) |
|||||
| 2. Please put an ✗ in one box for each row to indicate your attitude towards each statement: | |||||
| Strongly agree | Agree | Neither agree nor disagree | Disagree | Strongly disagree | |
|---|---|---|---|---|---|
| 1. Patient feedback is an important mechanism of quality improvement | □ | □ | □ | □ | □ |
| 2. Making patient feedback publicly available is beneficial to other patients | □ | □ | □ | □ | □ |
| 3. I have reservations about patient feedback received via complaints | □ | □ | □ | □ | □ |
| 4. I have reservations about patient feedback currently received via patient forums or participant groups | □ | □ | □ | □ | □ |
| 5. I have reservations about patient feedback currently received via surveys | □ | □ | □ | □ | □ |
| 6. Patient surveys help identify areas for service improvement | □ | □ | □ | □ | □ |
| 7. I can make good use of patient feedback | □ | □ | □ | □ | □ |
| 8. Responders to patient surveys are representative of my patient population | □ | □ | □ | □ | □ |
| 9. Feedback from current patient surveys is usually reliable | □ | □ | □ | □ | □ |
| 10. It is beneficial to receive patient feedback via complaints | □ | □ | □ | □ | □ |
| 11. It is beneficial to receive patient feedback via patient forums or participant groups | □ | □ | □ | □ | □ |
| 12. It is beneficial to receive patient feedback via surveys | □ | □ | □ | □ | □ |
| 13. I am likely to make changes to my individual practice as a result of patient feedback | □ | □ | □ | □ | □ |
| 14. Patients are able to provide useful feedback on organisational issues, such as appointment systems | □ | □ | □ | □ | □ |
| 15. I am concerned about my individual reputation as a result of patient feedback being made public | □ | □ | □ | □ | □ |
| 16. Patient feedback can improve the clinical quality of care I provide | □ | □ | □ | □ | □ |
Practice allocation to intervention groups
In the feasibility study, both practices piloted an intervention involving facilitated feedback and feedback reports provided at team and individual practitioner levels.
In the exploratory trial, participating practices were randomised to one of four intervention groups (eight practices) or to a control group (two practices). The level of RTF reporting and the provision of a facilitated team reflection session varied among intervention groups (A–D) (Table 40). Control group practices did not collect RTF during the implementation phase but could do so at the end of the project. RTF was reported at team and individual practitioner levels for control practices but no facilitated session was offered.
| Facilitated reflection? | Level of feedback reporting | |
|---|---|---|
| Practice level only | Practice level plus practitioner level | |
| Yes | Group A (two practices) | Group B (two practices) |
| No | Group C (two practices) | Group D (two practices) |
After completion of the baseline staff survey, practices were randomised by a University of Exeter Medical School statistician (otherwise unconnected to the project). Randomisation occurred in two blocks of five practices using a simple randomisation approach based on random number generation. Given the small number of practices involved, stratification by variables such as practice size or GP Patient Survey score was not attempted.
After randomisation, the trial allocation was confirmed with the practice and a timeline for data collection agreed. RTF collection began at both feasibility practices in February 2014. In the exploratory trial, intervention group practices began RTF collection between July and August 2014. For the two control group practices, RTF collection began in November 2014.
Figure 21 shows the schedule of study activities at exploratory trial practices.
FIGURE 21.
Overview of the practice pathway (exploratory trial).
Description of the real-time feedback intervention
Installation of touch screens
In each practice, touch screens were installed in the surgery waiting area after completion of the baseline staff survey. The installation of hardware was supported by CRT Limited. A short training session for practice staff was provided, explaining the purpose and day-to-day management of the touch screens and the need to encourage patients to provide feedback as well as giving a practical, interactive demonstration of the touch screens. A ‘run-in’ period of up to 5 days allowed for any set-up issues (such as the positioning of the touch screen) to be resolved before ‘live’ RTF collection began.
Practices were provided with leaflets and posters advertising the touch screen to patients and were encouraged to use other means (such as the practice website or newsletter) to promote the RTF devices. Exploratory trial practices were provided with a large banner and a supply of postcards for clinical staff to hand to consulting patients.
Collection of real-time feedback from attending patients
Patients visiting the surgery over the 12-week implementation period were eligible to provide feedback using the touch screens, including those attending for consultations or for other reasons (e.g. to book an appointment). Patients activated and navigated the survey by touching the screen.
The core survey items and response options are summarised in Table 41. In line with NHS guidance, the Friends and Family Test appeared first, followed by items selected from the national GP Patient Survey. Practices could add up to two questions of their own choice. Because of limited funding, survey items were presented in English only. A parent/guardian or carer (‘proxy’) could complete the survey on behalf of the patient if necessary.
| Question source/type | Wording of item | Response options presented |
|---|---|---|
| NHS Friends and Family Test | How likely are you to recommend our GP surgery to friends and family? | Extremely likely/likely/neither likely nor unlikely/unlikely/extremely unlikely/don’t know |
| GPPS – telephone access | How easy is it to get through on the telephone to this practice? | Very easy/fairly easy/not very easy/not at all easy/have not tried or don’t know |
| GPPS – access to appointments | How easy is it to get an appointment for a time that suits you? | Very easy/fairly easy/not very easy/not at all easy/have not tried or don’t know |
| GPPS – receptionists | How helpful do you find the receptionists at this GP surgery or health centre? | Very helpful/fairly helpful/not very helpful/not at all helpful/don’t know |
| GPPS – overall experience and satisfaction | Overall, how satisfied are you with the care you get at this GP surgery or health centre? | Very satisfied/fairly satisfied/neither satisfied nor dissatisfied/fairly dissatisfied/very dissatisfied |
| Filter question | Have you had an appointment with a health professional at the practice today? | Yes/no |
| Filter question | If ‘yes’, which of the following health professionals did you see? | Doctor/nurse/health-care assistant/phlebotomist (for a blood test)/practice counsellor/other health professional |
| Filter question | If doctor or nurse, which doctor or nurse did you see today? | List and photographs of individual staff at the practice plus another doctor/another nurse/don’t know |
| GPPS – confidence and trust | If seen doctor or nurse, do you have confidence and trust in the doctor or nurse you saw today? | Yes, definitely/yes, to some extent/no, not at all/don’t know or can’t say |
| GPPS – clinician communication skills | How good was the health professional at each of the following:
|
Very good/good/neither good nor poor/poor/very poor/doesn’t apply |
| Practice-specific items | Up to two items (with relevant response options) on topics selected by the practice team were included after the clinician communication skills items or after the overall experience/satisfaction item (for patients who had not consulted a health professional) | |
| Respondent information | Are you . . .? | The patient/parent or guardian of the patient/spouse or partner of the patient/another relative or friend of the patient/other |
| Patient’s gender | Are you/is the patient . . .? | Male/female |
| Patient’s age group | How old are you/how old is the patient? | < 18 years/18–25 years/26–45 years/46–65 years/> 65 years |
| Patient’s ethnic group | What is your ethnic group/what is the patient’s ethnic group? | White/mixed/Asian or Asian British/black or black British/Chinese or other |
| Free-text comments | If you would like to leave any further comments, please type below | – |
To reduce the survey length, and following discussions with our advisory group, only four of the seven GP Patient Survey communication skills items were included. Three items loading most strongly plus one item loading least strongly onto overall communication scores for GPs and nurses were selected. 130 Filter questions were included to ensure that respondents were presented with items relevant to their visit. For example, patients who had not had a consultation were not asked to rate the communication skills of a health professional.
Practice feedback reports
All practices received a fortnightly summary of team-level feedback (six reports per practice in total). Patient feedback was transmitted from the touch screens to CRT Limited by Wi-Fi or 3G connection. When no reliable signal was available, data were manually downloaded (approximately fortnightly) by the researcher.
Data were ‘quarantined’ if, for example, the respondent had not answered a minimum number of survey items, if response options appeared to be randomly selected or if a response had been provided in a time frame that suggested that the question could not have been read. Otherwise, data were considered to be ‘valid’.
Cumulative feedback reports were generated by CRT Limited including all valid feedback collected since touch screen installation. Reports contained frequency tables and graphs and patients’ free-text comments. Free-text comments were screened by the local researcher and details that might identify individual patients were removed. Negative comments about a clinician’s practice or standards of care were discussed on an individual basis with the chief investigator and a course of action proportionate to the risk to patients was agreed.
General practitioners, nurses and health-care assistants from the two feasibility and six exploratory trial practices (intervention groups B and D and the control group) were provided with personalised reports if they accumulated valid feedback from ≥ 20 respondents. These were similar in format to the team-level reports but summarised feedback only from patients who reported consulting a particular practitioner.
Team-level reports were e-mailed to the practice manager every fortnight for dissemination to the wider practice team. Personalised reports were e-mailed or posted directly to the individual practitioner.
Facilitated team reflection session
Two feasibility and four exploratory trial practices (intervention groups A and B) were offered a facilitated team reflection session. Facilitated reflection sessions took place at the surgery approximately half-way through RTF implementation (weeks 6–7) and lasted 45–60 minutes. Clinical and administrative staff were invited to attend and participants were provided with a printed copy of the practice’s most recent team-level feedback report.
The session was led by one of four experienced GP appraisers/trainers based in Exeter or Cambridge. Facilitators were briefed in advance about the study and the aims of the session and were provided with information about the practice and their most recent RTF report.
The facilitator and practice team explored the feedback and identified aspects of service that were well received by patients, as well as areas with potential for improvement.
Following experience in the two feasibility study practices, structured action-planning paperwork was used in the facilitated session for the exploratory trial. One member of staff was nominated to complete an action plan sheet during the session, summarising the team’s reflections, discussions and agreed action points. With the practice team’s permission, the session was observed by a researcher who took brief field notes.
Details of data collection
A multimethod approach (Figure 22) was adopted to investigate the feasibility and acceptability of the RTF intervention. This included focused ethnographic methods to explore how the new technology could be introduced into a complex system with ‘multiple human actors’. 260
FIGURE 22.
Multimethod approach to data collection.
Practice visits
During the 12-week implementation period, researchers visited participating practices every fortnight to observe patients’ interactions with practice staff and RTF devices. The visits took place on varying weekdays and at a range of times, to ensure that different staff were on duty and to capture workload variations and a range of activities at the practice. All data were recorded in anonymised form to protect patient and staff confidentiality. The practice visits were divided into shorter sessions (approximately 1 hour), each focusing on different types of data collection.
Unstructured observations
Researchers took detailed, contemporaneous field notes describing the practice environment as well as interactions between patients and staff and between patients and RTF devices. These included descriptions of specific events, as well as the researchers’ own impressions and interpretations. Observation notes were periodically shared and discussed within the research team to develop the methodology and maximise the richness of data collected.
Structured observations
Researchers used checklists, including yes/no tick boxes,261 to systematically record interactions between patients and practice staff and patients’ use of the touch screens and publicity materials (Box 4).
When some interaction occurred, did the receptionist:
-
tell the patient about the opportunity to leave RTF?
-
point to or take the patient to the touch screen?
-
offer to demonstrate how to use the touch screen?
When some interaction occurred, did the health professional:
-
tell the patient about the opportunity to leave RTF?
-
point to or take the patient to the touch screen?
-
offer to demonstrate how to use the touch screen?
During his or her visit, did the patient:
-
pick up a RTF leaflet/flyer or look at a RTF poster?
-
spend time reading RTF information in detail?
-
Was a touch screen free when the patient left the surgery?
-
What level of interaction was observed?
-
patient looked at or walked up to a touch screen
-
patient stopped to read the first screen
-
patient touched the first screen to begin the survey
-
patient stopped using the touch screen without answering any items
-
patient answered some or all RTF items.
-
-
Patient asked (staff or researcher) what the touch screen was for.
-
Patient required help to use the touch screen (from staff or researcher).
During observation sessions, a poster was displayed in the waiting area to explain the researcher’s presence in the practice. Individual patient and staff consent was not sought in case this significantly altered behaviour relating to the touch screens. 262
Patient exit surveys
Researchers conducted brief face-to-face exit surveys with a convenience sample of patients as they left the practice, whether or not they had used the RTF device. The purpose of the survey was explained to patients and their verbal consent to participate was sought. Participants responded verbally to a series of structured questions about the touch screens and their views of RTF (Table 42). Brief demographic details were also recorded.
| Patient exit survey items | Attending patients who: | |
|---|---|---|
| Had used a touch screen | Had not used a touch screen | |
| Was the patient aware of the opportunity to leave feedback using a touch screen device? | NA | ✓ |
| How did the patient find out about the opportunity to leave feedback? | ✓ | ✓ |
| Reasons for not using the touch screen today | NA | ✓ |
| How easy did the patient find it to use the touch screen? | ✓ | NA |
| Did the patient have any difficulty understanding the RTF questions? | ✓ | NA |
| How long did it take to answer the RTF questions? | ✓ | NA |
| The patient’s overall view of touch screens as a way of collecting patient feedback | ✓ | ✓ |
| Patient’s gender | ✓ | ✓ |
| Patient’s age group | ✓ | ✓ |
Extraction of appointment statistics
Researchers used practice appointment systems to determine the number of appointments attended (with any health professional) during the 12-week RTF implementation period and to collect anonymised age/gender information about the patients who had consulted. This information was used, combined with consultation information from the RTF survey, to calculate the percentage of consulting patients who had used a touch screen and to explore the extent to which those patients were representative of the consulting patient population with respect to age and gender. No records were kept of the number of patients who attended the practice for other reasons so it was not possible to calculate a ‘feedback rate’ for this group.
Practice team focus groups or interviews
At the end of RTF implementation, researchers conducted either semistructured interviews with a purposive sample recruited from among all practice staff or focus groups to which all practice staff were invited. Interviews and focus groups explored aspects of RTF implementation within the practices, including training and technical support, processes involved in the collection of RTF, reports from the devices, learning from and acting on patient RTF and how much staff valued the feedback that they received. Interviews and focus groups lasted approximately 40–45 minutes and were audio recorded with the participants’ permission and transcribed verbatim. Participants provided individual written informed consent prior to the focus group or interview. Lessons from discussions with the feasibility practice teams were used to refine the RTF intervention and wider study processes prior to the exploratory trial phase.
Interviews with facilitators
Semistructured interviews were conducted with facilitators to explore aspects of the team reflection sessions, including the facilitators’ general approach to the sessions; perceptions about practice teams’ engagement with the facilitation process; views about the practice teams’ or individuals’ understanding of and reflections about RTF (including individualised feedback) and any plans the teams had for acting on it; and their assessment of the value of facilitated sessions in general. Facilitators provided individual written informed consent and the discussion was audio recorded with their permission and transcribed verbatim.
Data analysis
Quantitative analysis
The proportion of consulting patients who used the touch screens during the RTF implementation period was calculated (overall and for each practice) using the following equation: number of patients who provided valid RTF and reported having a consultation with a health professional (ascertained from each practice’s final RTF data set) divided by the number of patients who consulted a health professional in the same period (ascertained from each practice’s computerised appointments system).
The age, gender and ethnic origin of consulting patients who provided valid feedback were ascertained from each practice’s final RTF report, along with the type of health professional (GP/nurse/health-care assistant/other) consulted.
To identify whether or not particular patient groups were more likely to use the touch screens than others, the proportion of patients who provided valid RTF by age group and gender was compared with the respective proportions of all patients who consulted in the same time period (ascertained from the appointments statistics) using z-tests.
Not all patients who provided RTF disclosed their age or gender. To derive more accurate response rates and proportions of responders by gender and age bands, the number of responders in each demographic subgroup per practice was increased in proportion to the number of missing values expected in that subgroup, based on the proportions in the practice’s consulting population. For example, if a practice had 10 respondents who had not provided their gender and 60% of appointments at the practice were for female patients, the number of responding female patients for that practice was increased by six and the number of responding male patients was increased by four.
Data derived from completed structured observation checklists were summarised descriptively to determine the frequency of a range of prespecified interactions occurring during the observation periods, for example the number (%) of patients who were encouraged by reception staff to use the touch screen. Patient exit survey responses were also summarised descriptively: the number (%) of patients endorsing each response option.
To determine whether or not the RTF intervention was associated with staff attitudes to patient feedback, mean scores on the VOP scale before and after the study were compared using a paired-samples t-test.
We explored whether or not changes over time in VOP scale scores varied across trial arms and health professional groups using analysis of variance (ANOVA). The ANOVA model included change in VOP scale score as the dependent variable, derived by subtracting baseline scores from follow-up scores and, therefore, included responses only from staff who returned baseline and follow-up questionnaires. The model had a 5 (groups A, B, C, D and control) × 2 (clinical, non-clinical) design, with trial arm and practice role as between-subjects factors. To provide more balanced ‘practice role’ groups, GPs, nurses and other health professionals were categorised as having a clinical role, whereas receptionists, administrators and managers were categorised as having a non-clinical role.
Cost analysis
This analysis sought to estimate the cost of providing a RTF intervention in general practices over a 12-week period. Such costs could potentially be compared with outcomes in a cost–consequences analysis. Cost items are listed in Box 5.
Publicity (leaflets and poster).
Kiosk rental.
Touch screen rental.
Kiosk collection.
Reporting.
Training/set-upTime for:
-
GP
-
practice manager
-
practice nurse
-
receptionist
-
health-care assistant
-
administrator.
Facilitator fee.
Staff time (categories as per training/set-up).
Data for the hire of equipment and provision of team- and individual-level reports were provided in aggregate by the RTF provider. Time inputs for practice staff and facilitators were collected from each of the eight intervention practices. Unit costs for staff (Table 43) were extracted from standard UK sources. 263 The price year for the analysis was 2014 and costs included value-added tax (VAT) when applicable. The cost in the two control practices was assumed to be zero. Given the pilot nature of the study and the small sample size, summary costs only were reported and no attempt was made to draw comparisons between trial arms.
| Role | Hourly rate (£) | Source/notes |
|---|---|---|
| GP | 109.00 | PSSRU 2014a (p. 195) – per hour of General Medical Services activity, excluding direct care staff costs and qualification costs |
| Nurse practitioner | 51.00 | PSSRU 2014a (p. 193) – nurse advanced per hour (excluding qualification costs) |
| Practice nurse | 34.00 | PSSRU 2014a (p. 192) – per hour (excluding qualification costs) |
| Health-care assistant | 10.06 | PSSRU 2014a (p. 266) – mean pay for health-care assistants: £16,600, assumed 37.5 hours per week and 44 weeks per annum |
| Physiotherapist | 32.00 | PSSRU 2014a (p. 179) – per hour |
| Pharmacist | 51.00 | PSSRU 2014a (p. 184) – per hour |
| Practice manager | 21.54 | Midpoint (point 30) Agenda for Change band 7.b PSSRU 2014a (p. 197) states that practice manager for a dentist is typically on Agenda for Change band 7. Assumed same cost for a general practice and 37.5 hours per week, 44 weeks per annum |
| Administrator | 10.78 | Assumed Midpoint Agenda for Change band 3 (£17,794), 37.5 hours per week and 44 weeks per annumb |
| Receptionist | 9.35 | Midpoint Agenda for Change band 2 (£15,432). PSSRU 2014a (p. 197) states that receptionist for a dentist is typically on Agenda for Change band 2. Assumed same cost for a general practice and 37.5 hours per week, 44 weeks per annum |
Qualitative analysis
NVivo 10 software was used to facilitate the organisation, coding, linking and retrieval of the qualitative data from the sources described above. After initial independent reading of a sample of the transcripts, two qualitative researchers (MC and AD) discussed preliminary themes, eliminated any duplication and resolved any differences. MC developed a coding framework, underpinned by normalisation process theory (NPT) constructs,264 into which the refined themes were organised. Using this framework, the remaining transcripts were each analysed and coded by MC and a subset of transcripts was analysed and coded by AD. Both researchers ensured that data that did not appear to fit within the NPT constructs were also included in their analysis. The progress of the analysis was discussed in regular group sessions with a third researcher (CW), whose role as academic lead afforded a comprehensive view of qualitative and quantitative aspects of the RTF research.
Normalisation process theory provided an analytical tool with which the RTF implementation was explored. NPT is a theory of implementation originally developed to understand the embedding of new technologies into health systems,264 so was judged to be particularly appropriate to the processes involved in RTF implementation. The theoretical framework includes four constructs: coherence/sense making, cognitive participation/relationships, collective action/enacting and reflexive monitoring/appraisal (Figure 23). Although presented as discrete, linear categories, in reality the NPT constructs often operate and are experienced simultaneously.
FIGURE 23.
Normalisation process theory framework: qualitative analysis.
Results: feasibility study
Summary of findings
The characteristics of the two practices recruited for the feasibility study are summarised in Table 44. Across the two practices, 607 out of 14,372 (4.2%) consulting patients provided valid feedback, with a similar rate of touch screen use in each practice [178/4355 (4.1%) in practice A and 429/10,017 (4.3%) in practice B].
| Characteristic | Practice A | Practice B |
|---|---|---|
| List sizea | 4122 | 6555 |
| Number of practice staff | ||
| GPs | 5 | 9 |
| Nurses | 1 | 1 |
| Health-care assistants | 3 | 4 |
| Reception/administrative staff | 11 | 22 |
| Managerial staff | 1 | 5 |
| Setting | Urban | Rural |
| GP Patient Survey centile score (%) | 46.7 | 16.7 |
| Deprivation decileb | 7 | 6 |
| Proportion of telephone consultations (%) | 76–100 | 26–50 |
Observation of interactions in the reception/waiting areas revealed that, although staff interacted with patients (100/185; 54% observations), they were rarely encouraged to leave feedback or directed to the touch screens (4/87; 5% of interactions with reception staff). RTF publicity materials were rarely noticed by patients (2/185; 1% of observations) but were competing with a large volume of health and social care information displayed in waiting areas. This suggested that more conspicuous publicity materials and a greater emphasis in set-up training on the need to encourage patients to use touch screens were required. Large pull-up banners and postcards for clinicians to hand to consulting patients were therefore introduced for the exploratory trial phase.
Patients were divided in their views of RTF as a way to provide feedback. Many were positive about the touch screens (51/60; 85%), finding their immediacy and anonymity advantageous, and welcomed the opportunity to provide feedback. However, some patients highlighted potential problems for other patients who may not be comfortable with computers, who are rushed or who have consecutive appointments. Patients who did not use the touch screens commented on the positioning of the devices and feeling ‘like you are on show’. These comments were taken into consideration when advising practices in the exploratory trial phase.
Staff from both practices were enthusiastic about the touch screens, but confirmed that the publicity materials did not adequately attract patients’ attention. Receptionists found juggling their normal workload with encouraging patients to use the touch screens difficult. Although staff found the facilitation sessions useful, circulation of RTF reports to the wider team and identifying an action plan at the end of the session were problematic. It was suggested, for the exploratory trial, that one individual within each practice should be responsible for completing an action plan sheet for circulation to the team after the facilitation session.
Results: exploratory trial
Ten practices were recruited for the exploratory trial: eight from the South West and two from Cambridgeshire. Table 45 summarises the characteristics of participating practices.
| Characteristic | Intervention Aa | Intervention Ba | Intervention Ca | Intervention Da | Control group | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Practice 1 | Practice 2 | Practice 3 | Practice 4 | Practice 5 | Practice 6 | Practice 7 | Practice 8 | Practice 9 | Practice 10 | |
| List sizeb | 4114 | 4568 | 3618 | 8005 | 13,000 | 15,189 | 10,998 | 9500 | 11,727 | 6675 |
| Number of practice staff | ||||||||||
| GPs | 3 | 4 | 3 | 6 | 11 | 6 | 12 | 6 | 6 | 4 |
| Nurses | 2 | 3 | 1 | 5 | 3 | 7 | 7 | 3 | 8 | 2 |
| Health-care assistants | 2 | 1 | 1 | 1 | 3 | 2 | 2 | 2 | 2 | 1 |
| Reception/administrative staff | 6 | 8 | 7 | 12 | 12 | 12 | 17 | 14 | 16 | 8 |
| Managerial staff | 1 | 3 | 2 | 2 | 3 | 2 | 3 | 2 | 2 | 1 |
| Setting | Rural | Urban | Urban | Inner city | Rural | Urban | Urban/rural | Inner city | Urban | Urban |
| GP Patient Survey centile score (%) | 33.4 | 39.5 | 28.9 | 14.3 | 34.0 | 31.9 | 21.9 | 32.7 | 27.1 | 14.5 |
| Deprivation decilec | 8 | 2 | 10 | 2 | 6 | 2 | 9 | 7 | 7 | 7 |
| Consultations per week, mean | 441.6 | 707.0 | 181.3 | 620.3 | 1809.5 | 474.6 | 434.0 | 636.8 | 1040.2 | 250.3 |
Proportion and characteristics of patients providing real-time feedback
Altogether, 1941 out of 79,145 (2.5%) consulting patients provided valid feedback (‘responders’), with a 95% CI of 2.3% to 2.6%. Patient use of the touch screens varied across practices (Table 46), with a range of 0.7% (95% CI 0.6% to 0.9%) to 8.0% (95% CI 7.3% to 8.8%). The mean practice-level response rate was 3.2% (SD 2.2%).
| Demographic variable | RTF responses/appointments | % (95% CI) |
|---|---|---|
| Overall | 1941/79,145 | 2.5 (2.3 to 2.6) |
| Practice (group)a | ||
| 1 (intervention A) | 231/5299 | 4.4 (3.8 to 4.9) |
| 2 (intervention A) | 201/8484 | 2.4 (2.1 to 2.7) |
| 3 (intervention B) | 110/2175 | 5.1 (4.2 to 6.1) |
| 4 (intervention B) | 168/7443 | 2.3 (1.9 to 2.6) |
| 5 (intervention C) | 162/21,764 | 0.7 (0.6 to 0.9) |
| 6 (intervention C) | 64/5695 | 1.1 (0.9 to 1.4) |
| 7 (intervention D) | 416/5208 | 8.0 (7.0 to 8.8) |
| 8 (intervention D) | 102/7642 | 1.3 (1.1 to 1.6) |
| 9 (control) | 386/12,482 | 3.1 (2.8 to 3.4) |
| 10 (control) | 101/3003 | 3.4 (2.7 to 4.1) |
| Genderb | ||
| Male | 531/23,739 | 2.2 (2.1 to 2.4) |
| Female | 859/34,226 | 2.5 (2.3 to 2.7) |
| Age band (years)b | ||
| < 18 | 150/6747 | 2.2 (1.9 to 2.6) |
| 18–25 | 78/3998 | 2.0 (1.5 to 2.4) |
| 26–45 | 315/12,383 | 2.5 (2.3 to 2.8) |
| 46–65 | 469/15,190 | 3.1 (2.8 to 3.4) |
| ≥ 65 | 377/19,647 | 1.9 (1.7 to 2.1) |
Data on patient ethnicity were not available from the appointments system at any of the exploratory trial practices and, at three practices, appointment data could not be broken down by age and gender. Table 46 shows the response rate broken down by gender and age bands using data from seven of the 10 exploratory trial practices. For these practices, the mean percentage of responders who did not provide their gender was 6.7% (range 1.9–13.7%) and the mean percentage of responders who did not provide their age was 6.7% (range 2.2–13.7%).
The age and gender of consulting patients who provided RTF (at seven of the 10 practices) are summarised in Table 47, together with the characteristics of all patients from these practices who consulted during the study period.
| Characteristic | Responders, n/N (%) | All patients, n/N (%) | p-valuea,b |
|---|---|---|---|
| Womenb | 859/1390 (61.8) | 34,226/57,965 (59.0) | 0.039 |
| Age band (years)c | |||
| < 18 | 150/1390 (10.8) | 6747/57,965 (11.6) | 0.329 |
| 18–25 | 78/1390 (5.6) | 3998/57,965 (6.9) | 0.061 |
| 26–45 | 315/1390 (22.7) | 12,383/57,965 (21.4) | 0.243 |
| 46–65 | 469/1390 (33.7) | 15,190/57,965 (26.2) | < 0.001 |
| ≥ 65 | 377/1390 (27.1) | 19,647/57,965 (33.9) | < 0.001 |
| Ethnicityd | |||
| White | 1724/1941 (88.8) | NA | – |
| Mixed | 28/1941 (1.4) | NA | – |
| Asian | 52/1941 (2.7) | NA | – |
| Black | 27/1941 (1.4) | NA | – |
| Chinese | 8/1941 (0.4) | NA | – |
| Missing | 102/1941 (5.3) | NA | – |
There was a higher proportion of female responders (61.8%) than in the consulting population (59.0%) (z = 2.063, p = 0.039). The proportion of responders in the < 18, 18–25 and 26–45 years age bands did not differ significantly from the corresponding proportions in the consulting population. There were significantly more responders aged 46–65 years (33.7% of responders compared with 26.2% of the population; z = 6.300, p < 0.001) and significantly fewer responders aged ≥ 65 years (27.1% of responders compared with 33.9% of the population; z = –5.277, p < 0.001).
Observed patient and staff interactions
Researchers conducted structured observation sessions only at the eight intervention group practices in the exploratory trial. Observations were not conducted at the control group practices.
In total, 873 of 1205 (72.5%) attending patients were observed to have some form of verbal interaction with a receptionist, but there were fewer interactions with health professionals in the waiting area (0.8%). Across 1199 observed staff–patient interactions, 60 (5.0%) patients were encouraged to use the touch screens by a receptionist, but never by a health professional. When staff encouraged patients to use the touch screen, 36 out of 60 (60.0%) patients attempted to start the survey. In contrast, only 28 out of 1114 (2.5%) patients attempted the survey without encouragement. Few patients (78/1199, 6.5%) were observed to read the publicity materials in the waiting area.
Patient views of real-time feedback
In total, 375 patients participated in exit surveys at the eight intervention arm practices in the exploratory trial. Of those surveyed, 103 (27.5%) had used the touch screen in the waiting area and 272 (72.5%) had not.
Of the patients who had used a touch screen, 87 out of 101 (86.1%) had positive views of RTF as a way of leaving feedback for the practice. All responders reported that they had found it easy to complete the RTF survey and that they had answered all questions. The majority (79/98, 80.6%) of responders reported completing the survey in ≤ 2 minutes.
Patients who had not used a touch screen gave a range of reasons for this. Over half (149/268, 55.6%) were not aware of the touch screens or the opportunity to leave feedback. Of those who were aware of the touch screens, 29 out of 84 (34.5%) said that they did not have time to use them; 5 out of 84 (6.0%) felt that their feedback would not be relevant (e.g. because it was positive); 4 out of 84 (4.8%) had concerns about anonymity or how the feedback would be used; 15 out of 84 (17.9%) had concerns about technology; and 12 out of 84 (14.3%) reported completing RTF before but were not aware that they could leave feedback on each visit. Despite not using the touch screens during their current visit, 178 out of 260 (68.5%) patients thought that the idea of RTF was good.
Real-time feedback and staff attitudes to patient feedback
Across the 10 exploratory trial practices, 162 out of 247 (65.6%) members of staff returned a baseline questionnaire, 123 out of 247 (49.8%) returned a follow-up questionnaire and 107 out of 247 (43.3%) returned both questionnaires. Of these, 92 out of 107 (86.0%) completed all items on the VOP scale at both time points and were included in the analysis of pre- and post-intervention scores. Table 48 presents the mean VOP scale scores at the two time points for each intervention group and by staff group. The results suggest that staff perceptions of the value of patient feedback did not change significantly from baseline (mean 42.9, SD 8.44) to follow-up (mean 41.7, SD 8.20; t91 = 1.703, p = 0.092).
| Groupa | n | Pre-intervention mean (SD) | Post-intervention mean (SD) | Difference mean (SD) | 95% CI for the difference |
|---|---|---|---|---|---|
| Group A | |||||
| Clinical | 4 | 45.25 (13.6) | 41.00 (14.02) | –4.25 (11.03) | –21.8 to 13.30 |
| Non-clinical | 8 | 40.25 (6.18) | 41.63 (5.85) | 1.38 (2.88) | –1.03 to 3.78 |
| Overall | 12 | 41.92 (8.99) | 41.42 (8.69) | –0.50 (6.79) | –4.81 to 3.81 |
| Group B | |||||
| Clinical | 6 | 41.17 (8.28) | 44.50 (9.27) | 3.33 (10.27) | –7.44 to 14.11 |
| Non-clinical | 6 | 45.17 (3.31) | 39.33 (3.98) | –5.83 (6.62) | –12.78 to 1.11 |
| Overall | 12 | 43.17 (6.37) | 41.92 (7.32) | –1.25 (9.53) | –7.30 to 4.80 |
| Group C | |||||
| Clinical | 14 | 42.29 (10.23) | 40.86 (11.44) | –1.43 (4.47) | –4.01 to 1.15 |
| Non-clinical | 8 | 42.63 (6.09) | 40.75 (4.37) | –1.88 (4.26) | –5.43 to 1.68 |
| Overall | 22 | 42.41 (8.79) | 40.82 (9.35) | –1.59 (4.29) | –3.50 to 0.31 |
| Group D | |||||
| Clinical | 5 | 45.80 (8.20) | 45.80 (9.04) | 0.00 (5.24) | –6.51 to 6.51 |
| Non-clinical | 12 | 43.50 (5.60) | 43.50 (7.24) | 0.00 (6.4) | –4.06 to 4.06 |
| Overall | 17 | 44.18 (6.29) | 44.18 (7.59) | 0.00 (5.92) | –3.04 to 3.04 |
| Control | |||||
| Clinical | 16 | 40.94 (11.89) | 40.63 (9.08) | –0.31 (8.15) | –4.65 to 4.03 |
| Non-clinical | 13 | 45.15 (7.10) | 41.62 (6.84) | –3.54 (5.33) | –6.76 to –0.32 |
| Overall | 29 | 42.83 (10.09) | 41.07 (8.03) | –1.76 (7.10) | –4.46 to 0.94 |
The ANOVA examining the difference between follow-up and baseline VOP scale scores assessed change over time between trial arms and staff groups. There was no significant effect of trial arm (F4,77 = 0.301; p = 0.877), indicating that change over time in VOP scores did not vary reliably across the intervention and control groups. Similarly, there was no significant effect of staff role (F2,77 = 2.351; p = 0.102), suggesting that neither the VOP scores of clinical staff nor those of non-clinical staff changed over time. No significant interaction between trial arm and staff group was apparent (F8,77 = 1.549; p = 0.154).
Cost analysis
Costs by RTF intervention groups A–D are shown in Table 49. The mean cost per practice of providing RTF was approximately £1117 over the 12-week intervention period. The largest component was rental of the RTF touch screens (total £972 per practice). The practice manager and administrative staff attended the set-up session in most practices. At practices allocated to a facilitated feedback arm (groups A and B), GPs and nurses also attended (Figure 24). A nurse was the only attendee at the set-up session in practice 6 and this practice had the lowest feedback response rate by the end of RTF implementation. Training time was assumed to be 15 minutes per staff member, estimated at £27 per practice (SD £22).
| Item | Details | Groupa | All groups, mean (SD) | Groups | |||
|---|---|---|---|---|---|---|---|
| A | B | C | D | ||||
| Feedback level | P | P&I | P | P&I | |||
| Facilitated session? | Yes | Yes | No | No | |||
| Number of practices | 2 | 2 | 2 | 2 | 8 | ||
| RTF equipment: hire and provision (£) | |||||||
| Publicity (posters and leaflets) | 750 postcards + one poster per practice | 107 | |||||
| Touch screen (kiosk) rental | 12-week hire | 630 | |||||
| Touch screen (desktop) rental | 12-week hire | 342 | |||||
| Kiosk collection | 38 | ||||||
| Reporting | 75b | ||||||
| Total | 1117 | A–D | |||||
| Practice staff set-up session | 43 | 34 | 8 | 22 | 27 (22) | A–D | |
| Total | 1144 (22) | A–D | |||||
| Facilitated reflection (£) | |||||||
| Facilitator fees | 250 | 250 | 250 (58) | A, B | |||
| Practice staff to attend facilitation | 477 | 378 | 428 (180) | A, B | |||
| Total | 727 | 628 | 678 (227) | A, B | |||
| Total cost | 1887 | 1779 | 1125 | 1139 | |||
FIGURE 24.
Attendance at RTF set-up sessions by intervention arm, practice and staff role.
The total cost to a practice of the system for the 12-week implementation period was £1144 (including fees paid to the RTF provider). There was no difference in cost between the team-level and individual-level feedback as the processing fee for the report (£75) was assumed to be the same for both.
Facilitated feedback (groups A and B) cost an estimated £678 per practice (SD £227). This consisted of £250 (SD £58) in fees to the facilitator and £428 (SD £180) in practice staff time to attend facilitation.
Qualitative evaluation
Staff from four of the participating 10 practices (intervention, n = 21; control, n = 7) took part in focus groups and various staff members from the remaining six practices participated in interviews (n = 24). Table 50 summarises the characteristics of participating staff.
| Practice | Intervention group | Number of interviews by staff type | Number of focus group attendees by staff type |
|---|---|---|---|
| 1 | Facilitated reflection and practice-level feedback | GPs, n = 2; administrative staff including receptionists, n = 5; nurses, n = 2 | |
| 2 | Facilitated reflection and practice-level feedback | GPs, n = 1; administrative staff including receptionists, n = 6; nurses, n = 1 | |
| 3 | Facilitated reflection and practice-level and individual practitioner-level feedback | Deputy practice managers, n = 1; administrative staff including receptionists, n = 2 | |
| 4 | Facilitated reflection and practice-level and individual practitioner-level feedback | GPs, n = 1; administrative staff including receptionists, n = 2; nurses, n = 1 | |
| 5 | Unfacilitated reflection and practice-level feedback | GPs, n = 1; administrative staff including receptionists, n = 2; nurses, n = 1 | |
| 6 | Unfacilitated reflection and practice-level feedback | GPs, n = 1; administrative staff including receptionists, n = 1; nurses, n = 1 | |
| 7 | Unfacilitated reflection and practice-level and individual practitioner-level feedback | Practice managers, n = 1; deputy practice managers, n = 1; administrative staff including receptionists, n = 2 | |
| 8 | Unfacilitated reflection and practice-level and individual practitioner-level feedback | Practice managers, n = 1; GPs, n = 1; administrative staff including receptionists, n = 2; nurses, n = 1 | |
| 9 | Control | GPs, n = 2; administrative staff including receptionists, n = 3; nurses, n = 2 | |
| 10 | Control | Practice managers, n = 1; GPs, n = 2; administrative staff including receptionists, n = 1; GP registrars, n = 1 |
Staff from four practices attended facilitated sessions either during or after the data collection phase and three facilitators were interviewed once data collection was complete.
Researchers visited practices 57 times to conduct observations and patient exit interviews. Out of 375 patients approached for an exit interview, 300 (80%) provided additional comments about their experience of providing RTF. Qualitative data extracted from the RTF devices (patient free-text comments) were excluded from the analysis as this information related to patient experience of the practice and not to RTF implementation.
Results from the qualitative study are presented under the headings of the four NPT constructs (coherence, cognitive participation, collective action and reflexive monitoring), subdivided and illustrated by our findings from the RTF implementation.
Conventions
Sources of quotations are denoted as follows:
-
practice staff: six-digit numerical code denoting individual participant/FG (focus group) or Int (interview)/staff category (PN, practice nurse; GP; ADM, administrative staff; REC, receptionist; PM, practice manager; DPM, deputy practice manager)
-
patients: six-digit numerical code/PAT
-
facilitators: three-digit numerical code/FAC.
Editing is indicated by . . . when some words are missing or [. . .] when a larger fragment is missing.
Making sense of the real-time feedback implementation (‘coherence’)
Practice staff used a range of ways to make sense of RTF: by comparing it with other feedback methods; by adapting normal routines to absorb RTF into their practice organisation; by seeing it as part of a programme of communication with their patients; by considering how it may be received by their patient population(s). They drew on their own experience of technological initiatives in other contexts and of past participation in research studies. Views about the content of the RTF survey influenced overall attitudes towards the concept of receiving immediate feedback.
Feedback and methods
Many members of staff mentioned that they were well used to receiving feedback from their patients and that their patients were accustomed to giving feedback. Many made sense of RTF by comparing it with other feedback methods with which they were more familiar and some favoured the convenience of RTF to traditional paper-based surveys and mentioned the difficulty of collecting responses:
I think it’s the way you’ve done it immediately that is much better. Which is if you give them something maybe to take away and bring back or post, they aren’t gonna do that . . . and that’s the end of it . . . With the touch screen it’s . . . much easier.
007021/Int/ADM
Others highlighted problems with obtaining feedback from patients who had not had time to reflect before inputting their responses:
You’re gonna get some hotheaded responses aren’t you? I mean people are going to come out and get really cross, there’ll be some emotion going on there that if they cooled down for 5 minutes you wouldn’t get those responses.
015009/FG/PN
Some staff voiced concern about access for specific groups of patients. Elderly patients were characterised by some as being less willing or able to use technology. Staff also mentioned patients with low literacy levels and patients whose first language is not English.
Motivation
Some practices were motivated to participate in the study by the requirements of other schemes, such as the Friends and Family Test (CQC) and annual Royal College of General Practitioners appraisals.
Context
Many practice staff viewed the RTF implementation purely as part of a research study and, therefore, being limited in its impact on the practice and workloads:
I suppose the key is . . . in the nicest possible way . . . having as little impact on sort of patients and staff as possible but gathering enough information for the study to be worthwhile.
069027/Int/PM
For others, RTF was part of an overall strategy for obtaining feedback from patients and staff valued it as an additional means of staying in touch with their patients’ concerns.
Many participants mentioned that people in general are asked for feedback about a range of services and were able to place RTF within a familiar context:
I think one in four people have got smartphones and . . . I’ve seen that sort of survey used at airports, it’s sort of in . . . people’s lives now really.
017019/Int/PM
Real-time feedback content
Staff voiced varying views about the content of the RTF survey. Several were critical about the number of demographic questions. Some were positive about including a couple of their own questions, making the implementation particularly relevant to their practice.
Practice organisation/roles
Practice organisation, including both physical configuration and staff arrangements, affected individual and group perceptions about RTF. Most practices had an electronic check-in system and/or notification screen to call patients to consultations. Sometimes this meant that there was little interaction between receptionists and patients and reduced opportunities to promote the use of the RTF devices. The position of the RTF devices, often dictated by the physical limitations of the practice building, also influenced levels of use and the degree to which RTF became embedded within the normal routines of the surgery:
It was a bit difficult where it was placed, because we have a booking-in machine and it was next to that, and I think people thought it was another booking-in machine.
015019/FG/REC
Although practices were given materials for involving all staff in implementation, some viewed it as predominantly a task for the reception team and depended on receptionists to promote the devices to patients:
I think we particularly focused on getting reception staff to try and remind patients . . . as they checked in for their appointment rather than to do it afterwards, cos otherwise it is a lot to ask the clinical staff to remember.
011017/FG/PM
The degree to which practices involved clinical staff varied. In some practices, all staff were familiar with the implementation, so efforts to engage patients were coherent and in tune with the general practice ethos:
If they [patients] made a comment about the service, I said, ‘please can you feed it back’ . . . it was really nice that I could give them something definite to do immediately.
011001/FG/GP
An individual member of staff, such as the practice manager, or the research champion often took responsibility for managing the implementation. The effectiveness of this individual influenced how RTF was received by both staff and patients. In some practices, when RTF had been explained and promoted successfully, members of staff adapted their existing roles to embrace the new initiative, but in other practices some individuals voiced discontent and disenfranchisement:
As far as I knew it was a kind of if they want to use it, so was it up to me to actually ask them – it wasn’t really made that clear.
069013/Int/REC
Practice knowledge of their patients
Many practice staff used their knowledge of their patients to anticipate how RTF may be received by them. Many felt that particular groups would not be comfortable using a touch screen device: most often mentioned were elderly patients, patients with literacy problems and patients whose first language is not English.
Staff also mentioned that patients’ use of the devices was influenced by processes and the volume of work within the practice, both generally and at particular times during the day, week or year:
If we’re running late, then people were often in a hurry to leave as quickly as they could, having been . . . held up by us, so that was also an issue I think, for some of the patients.
010003/FG/GP
In some practices staff concluded that, as their patients were well used to using touch screen devices, they would be happy to leave feedback in this way. Others said that patients were not confident or competent with touch-screen technology:
I think . . . giving people access to a computer even if it’s a very simple touch-screen computer . . . is sometimes quite challenging. Just like touch-screen check-in, there’s a whole group which . . . hasn’t checked in because they’ve missed one of the buttons.
021001/Int/GP
Working together and with patients to establish real-time feedback (‘cognitive participation and collective action’)
The ways in which individuals and groups of staff worked together, and with patients, with regard to RTF varied among practices. The findings within these two related NPT constructs (cognitive participation and collective action) are closely linked with some of the sense-making aspects detailed in the previous section. Differences in style and methods of communication both within and between staff groups, and with patients, were more pronounced in some practices.
Communication/differences/engagement
Staff relationships with each other and with patients were a crucial part of RTF implementation. Often an open, inclusive approach to communication between staff members coincided with the way in which they related to their patients:
There’s nothing that’s kept away from us . . . whether it be good or bad . . . if we have to do something to either make it better or keep up what we’re doing then they tell us so being told is the only . . . way you’re gonna provide a service that the patients actually want.
007021/Int/ADM
By contrast, some administrative staff felt that their knowledge of patients’ concerns was ignored and they were not given the opportunity to make suggestions:
I mean we can have queues out of the door and it’s not noticed by the managers to think, oh right we need more staff there . . . We are always saying about confidentiality, the patients complain to us a lot about it, but we might mention it and nothing is ever done.
069013/Int/REC
There were differences in style between individual team members in the same practice:
If you want anything filled in you get [name removed] to give it to the patients because she just goes out and says, would you mind filling it in? [. . .] Whereas the others are not quite so interactive with the patients. They’ll say something to them over the desk but they won’t actually go out and interact with them.
068015/Int/DPM
Differences between individual patients were often mentioned and several receptionists admitted that they selected patients to encourage to leave RTF on the basis of their perceived abilities or level of sickness.
Some clinicians were used to asking their patients for feedback and felt very comfortable doing so, but others were more reticent:
I think it’s a bit embarrassing though, to say, well could you give me some feedback, I mean I find it quite embarrassing, so I wouldn’t ask them.
010003/FG/GP
One GP suggested that the process of requesting feedback may hint at a hidden agenda during an otherwise positive consultation:
It can feel awkward . . . if the conversation has gone really well, it sometimes slightly undermines the goodness of the conversation or the help that you’ve given.
007002/Int/GP
The way in which RTF had been introduced in practices had a profound effect on how engaged individual members of staff felt with the implementation. In practices in which communication was inclusive, staff felt part of an important initiative and understood their roles. In other practices, often when the research team had communicated solely with an individual practice contact, some staff felt remote from decision-making and so were not fully engaged with RTF:
I haven’t been involved. I don’t know what the plan is from here.
016021/Int/REC
Timing and workload
Timing, in many senses, influenced the embedding of RTF in practices, including timing of the request for feedback (after consultation), timing of the implementation itself (in the context of other practice activities) and variations in work volume according to particular times and days of the week:
It’s much easier to get patients to fill things in whilst they’re waiting than when they’ve finished . . . because they’re sitting down waiting, they’re almost sitting and looking for something to do.
011017/FG/PM
Reception staff found fluctuations in workload and demand from patients greatly influenced their ability to play a part in the RTF implementation:
Especially on a Monday and a Tuesday when the phones are ringing and the queue is long. It’s hard to . . . explain to them what it’s about.
068007/Int/REC
Involving patients
Most practices voiced their intentions to share the results from the RTF devices with their patients through posters or the practice website and several were keen to let their patients know that their responses had led to change.
One practice had involved its Patient Participation Group (PPG) in devising additional, practice-specific questions to include in the RTF survey and another had shared results with the PPG before discussing them as a practice team.
Feedback reports
On the whole, practice staff appreciated the regular feedback reports from the study team and drew favourable comparisons with other surveys, which often entailed in-house data analysis and assembly of results. Many scanned the reports for data that confirmed what they already knew and some were surprised by the volume of positive feedback.
Several practices were concerned about low RTF response rates and suggested possible reasons for this, including feedback ‘fatigue’ among both patients and practice staff. Some staff believed that patients would not bother to leave feedback unless their experience at the practice was either strongly positive or strongly negative.
Appraising and learning from real-time feedback (‘reflexive monitoring’)
Practice staff viewed the RTF implementation and results in a variety of ways. Some believed that giving their patients an opportunity to leave immediate feedback was a valuable addition to existing ways of communicating with them. Others viewed RTF responses as confirming what they knew already and possibly could not act on. Many mentioned plans for acting on the RTF and a few had already instigated changes within the practice by the time that they were interviewed or attended focus groups. The degree to which staff trusted the credibility of the results had an inevitable effect on their learning from it and several staff had suggestions for improving the usefulness of the questions and responses.
Trust/learning/suggestions
Some staff believed that the immediacy of a response added strength to it, but, conversely, many felt that a period of reflection was important and could greatly change how a patient viewed his or her consultation:
You might have been denied a medication at the time which might deeply upset you but then 2 weeks down the line you kind of realise that that’s [the] right thing and the feedback might be different.
007002/Int/GP
Many practice staff noted the low RTF completion rates and felt that the majority of their patients had not been given the chance to participate:
I suppose in the back of your mind you know it’s only a small percentage of your whole population, the people who are coming through the door.
068015/Int/DPM
Some members of staff felt that the RTF devices were used by patients to record two extremes of response and excluded the middle ground:
It attracts two types of people doesn’t it, the people who love you and tell you they love you and the people that just had a really bad experience that day and want to take it out on the system, really.
015016/FG/PM
Many staff found the free text left by patients more useful than the quantitative responses, but some staff found these comments frustrating as they could not follow them up with the individuals concerned:
It feels a bit like people might leave feedback if they’re unhappy, and so there’s very little positive and when you read it back certainly I found it difficult to reflect on and learn from it and to improve my practice because it wasn’t very specific.
0210017/Int/GP
Some suggested that an optional facility for respondents to provide their name would be helpful.
Some reception staff mentioned that individual feedback would be helpful as a learning tool for their staff group (the devices were not able to provide individual feedback for receptionists):
I take offence at that, cos I think, don’t tar us all with the same brush.
018011/FG/REC
Many staff expressed surprise about the positive feedback that they received – in both comments and quantifiable data. Negative responses were expected and in many cases confirmed previous feedback from patients.
Action
Several members of staff, predominantly practice managers, mentioned that they had taken or were intending to take action based on RTF. Some were keen to amalgamate the results with data from other initiatives before formulating a plan:
What we’ve historically done is . . . when we’ve had a survey, we’ve published the results of that survey, along with our action plan of how we are going to respond to different aspects of it and over what timescale . . . and who would be involved in that . . . we do do that on a sort of an annual basis [. . .] if you do it in dribs and drabs, it sort of doesn’t quite have the same impact.
017019/Int/PM
Some mentioned involving their PPG in discussions about the results and action planning, whereas others intended to inform their patients by publicising the results in the practice. Some individuals were not sure about what was planned and did not see it as affecting their own work.
Some staff said that the RTF responses had been expected and covered issues already familiar to the practice, which they had addressed or were addressing.
Many staff welcomed confirmation of previously held views, but some felt that their patients’ expectations were unrealistic and it was not possible to satisfy them.
Facilitation
Practice staff had not previously explored patient feedback at sessions guided by an external facilitator. Several factors, occurring before and during the facilitated session, influenced the success of these sessions. These can be summarised as:
-
prior to the session – communication of the aims and objectives of the session to the entire practice team; dissemination of patient feedback results so that staff could highlight areas for review; and protected time built into staff schedules for reviewing patient feedback
-
during the session – skill of the individual facilitator; provision of a clear agenda at the outset including expected outcomes; time to celebrate positive results; and an agreed action plan for staff to work to.
Facilitators found it difficult to get staff to agree and commit to a set of actions during the facilitation session. This was partly because of time limitations. Some staff were reluctant to implement changes suggested during the facilitation session, possibly because of previous experience of external pressures:
. . . a lot of GPs are fairly conservative and don’t want to change much, maybe that’s because they’ve got so much change imposed on them, they’re reluctant to change.
300/FAC
The facilitation sessions were flexible to allow staff to explore the results, but facilitators suggested that an action plan may have been compiled had it been explicitly included as an intended outcome within the agenda.
Some practices were more engaged with the facilitation process than others and this was demonstrated by their willingness to set a time for staff to meet and discuss RTF. Clear communication of the purpose of the session allowed for staff to contribute to the discussion and share ownership of the decisions being made, especially in practices in which facilitation was a novel approach:
But I think that was down to ground setting and me being clear from the start what we are doing. Also explaining the rules of the group and giving them ownership really of what was discussed.
002/FAC
In some practices the reception staff were more engaged with the feedback than clinical staff and provided suggestions for changes to their work routines more readily:
[T]he reception staff seemed fairly willing to contribute . . ., a lot of the feedback, was very pertinent to them.
100/FAC
Yeah, the reception staff are actually better at talking than the GPs because I think they were more enthusiastic [laughter]. They were the ones that came up with the ideas.
300/FAC
As mentioned above, in some practices reception and administrative staff made a greater contribution to the discussion than their clinical colleagues. The skills of the individual facilitator had an impact on the discussion; staff from one practice felt that ‘their’ facilitator did not effectively encourage contributions from all staff, but this was not experienced in other practices. Facilitators found it difficult to fully discuss sensitive issues and did not have sufficient time to work through them properly during the session:
You have opened up this box of really quite difficult stuff and then gone away again. What the surgery does with it now is really up to them.
200/FAC
All facilitators found that the session was not long enough to enable full discussion of the feedback and development of a clear action plan. The timing of the facilitation session was important: the majority took place half-way through the data collection, but some practices commented that it may have been useful at different time intervals during the data collection:
. . . maybe if we were running it for 6 or 12 months, you could have them at like quarterly intervals . . . but I think kind of 6 weeks into it, we were only just getting into it.
018015/FG/DPM
Patient perspectives
Although patients acknowledge the important role of technology, and recognise its value for providing feedback, some highlighted technology as a potential barrier. They mentioned other patients who may be intimidated and deterred from leaving feedback by technology, for example the elderly and those with literacy difficulties. Some patients who chose not to use the RTF device said that they would prefer to feed back directly to the GP or receptionist and were sceptical about the possibility of changes being made as a result of feedback if not voiced in person to the practice.
The lack of publicity and information about the purpose of the RTF device was a potential barrier for patients and affected their trust that RTF would result in change within the practice. The location of the touch screen device was an important factor, particularly in terms of privacy. If it was overlooked by reception staff or other patients, individuals often did not use it. Equally, if it was installed in an unobtrusive location, many patients did not notice it.
The timing of the request for feedback was also an important consideration. Some patients said that being asked to stay behind to provide feedback after a consultation was unacceptable, particularly if they had waited a long time for their appointment:
If you are ill you just want to go home after.
010001/PAT
Discussion
Real-time feedback is a relatively novel approach to the collection of patient feedback in general practice. In this research, the idea of RTF appeared broadly acceptable to both patients and staff in participating practices. However, communication within the practice team, and between staff and patients, was a key factor that influenced the level of acceptability, and the feasibility, of embedding RTF in practice routine.
Effective communication underpinned the successful implementation of RTF, not only in encouraging patients to use touch screens in the waiting area, but in the organisations’ use of collected feedback. Communication within the practice team influenced staff perceptions of RTF as a useful learning tool and the success of a facilitated reflection session as a means of discussing and planning service change.
In the context of this pragmatic, essentially unfacilitated survey, practice staff and patients viewed RTF positively, but engagement with the touch screens was lower than reported in other studies from the USA. 259,266 In absolute terms, the majority of practices in the current study collected feedback from ≥ 100 patients. However, the proportion of consulting patients who used the touch screens varied across practices (range 0.7–8.0%) and, overall, feedback represented the views of a relatively small proportion (mean 3.2%) of consulting patients.
The absolute number of patient responses using RTF was comparable to that achieved by the same practices in the most recently published national GP Patient Survey134 but, overall, practices’ response rates in the national GP Patient Survey were much higher (range 27–53%).
The difference in response rates between the current study and the US studies may reflect the greater number of items in our survey. It may also reflect the lower level of direct encouragement and support provided by staff to help patients use the touch screens. At many practices, receptionists were given responsibility for encouraging patients to use touch screens rather than clinicians. Receptionists were observed to interact with a significant proportion of patients who attended the surgery but they were rarely observed to encourage the use of the touch screens. Although a number of reasons were given by patients in the exit surveys for not using the touch screens, over half of these patients had been unaware of the opportunity to leave feedback; others may have provided feedback if clearer information had been provided about the purpose of the touch screens. When staff encouragement to use touch screens did occur, patients were more likely to start the survey. Direct encouragement was more effective than publicity materials displayed in the waiting areas, which went largely unnoticed by patients.
Practices accustomed to collecting and using patient feedback viewed RTF as part of their ongoing dialogue with patients and the immediacy of feedback helped offset the risk of ‘feedback fatigue’ for both staff and patients. However, practices and patients were concerned about patient groups who might be excluded from feedback processes that involve the use of touch screens, specifically older patients and those for whom English is not a first language. Others felt that the RTF screen was easy to read and acknowledged that people of all ages are well used to using touch screen devices in other areas of life. Our analysis suggested that some age groups (those aged 46–65 years) were over-represented among RTF users, whereas others were under-represented (those aged ≥ 65 years). In our study, female patients were more likely to provide RTF than male patients (62% vs. 38%, respectively), in contrast to the most recently published national GP Patient Survey data in which approximately even proportions of male (49%) and female (51%) patients responded. 134 The observation of lower rates of feedback in older age groups is in line with the study by Dirocco and Day,259 in which more intensive staff support for RTF had been available. Dirocco and Day259 also reported lower feedback rates among minority ethnic groups. Our study was unable to investigate this as appointment data could not be broken down by patient ethnicity at any of the participating practices.
Our findings with regard to levels of staff engagement with RTF and effective communication within practices and with patients are broadly in line with an earlier UK 6-month pilot study. 47 Practices’ physical configuration and flexible assignment of roles can either help or hinder participation and collective action among staff (and with patients) with regard to a new system or process. Good communication about RTF fosters involvement and buy-in from both clinical and administrative staff, including shared reasons for participation, the roles of different staff groups, ongoing progress with RTF collection and the content of feedback reports. Our findings suggest that information was not always communicated effectively to individuals and some felt remote from the process. Wofford et al. 266 suggested that RTF (collected using tablets) had minimal impact on working routines when implemented in a primary care setting. Our findings are more mixed about this: some practices and individuals suspended their involvement with RTF implementation during busy times or with particular patients, whereas others (particularly clinicians) reported that RTF did not impinge on their daily routines.
Practice staff identified potential benefits of using a facilitated session for discussion of patient feedback and having protected time for the celebration of achievements. Effective communication about patient feedback with all members of staff prior to and during a facilitated session encouraged constructive debate and all-practice engagement with any changes agreed at the session. Some practices saw advantages in the immediacy of feedback and the potential for quick action, in line with existing guidance from the Department of Health92 and the NHS Practice Management Network. 47 However, other practices preferred to combine their RTF results with other information before considering action or even action planning. Many patients commented on the importance of their practice taking account of and acting on feedback, but the degree to which any plans or changes resulting from RTF were communicated to or shared with patients varied greatly.
The costs of RTF need to be compared with outcomes to judge whether RTF represents a good investment for a general practice. Outside the context of a research project, the cost of hiring touch screens may be borne directly by the practice alongside staff time invested in set-up briefings and team meetings to reflect on patient feedback. GPs and nurses tended only to attend set-up briefing sessions in practices allocated to facilitated feedback, suggesting that clinician engagement was higher in those practices. This may be worthy of more detailed investigation in future studies, as it might be a mediator of any observed outcomes. To maximise patient use of touch screens, consistent effort and time from practice staff (particularly receptionists) is required to directly encourage and support feedback from patients. However, this could be seen as time well spent if it leads to the collection of RTF from a sizeable and representative group of the patient population.
The ability to achieve change in practice is a major issue highlighted in this study. Participants in the qualitative research identified an inertia – perhaps even an unwillingness or a resistance to implement change – following patient feedback. Such an observation concurs with findings from Deming267 (p. 81) who reported on such resistance and inertia:
I fill out a report when anything goes wrong. Someone from management, I was told, would come and take a look at the problem. No one has ever come.
What good comes of making a suggestion to your foreman? He just smiles and walks away.
And the telling comment:
What else could he do? He does not understand the problem, and could get nothing done if he did.
Strengths and limitations
Our investigation of the acceptability and feasibility of RTF was enhanced by a multimethod approach. A better understanding of the obstacles and drivers associated with embedding RTF in general practices was achieved by organising data from interviews, focus groups and observations according to NPT constructs. Although it is important to note that all four NPT constructs operated and were experienced concurrently, the NPT framework enabled a coherent view of the processes involved in RTF implementation, including the ways in which practice staff and patients understood RTF, teamwork and collective action within practices and reflection, learning and actions arising from the feedback. Focus groups were attended by a range of staff and individuals were encouraged to share their views about RTF. When focus groups were not possible, a range of staff participated in one-to-one interviews.
A range of general practices was recruited to the study, including those in urban, inner-city and rural settings, with varying deprivation scores and list sizes. However, practices were drawn from two broad geographical areas (South West and Cambridgeshire), which may not be representative of the UK as a whole. Participating practices may also have been those with an interest in research or service improvement.
The implementation of RTF in this strand of work had inherent limitations compared with other means of collecting feedback. For example, the survey items were presented only in English and patients who did not visit the surgery during the implementation period were unable to provide feedback. In some practices, it proved difficult to extract demographic information about consulting patients from the practice system and there was some evidence that appointment data were not consistently recorded within systems, limiting reliable assessment of the response rate and the representativeness of patients who used touch screens compared with the consulting population. It was not possible to calculate response rates for patients who attended the surgery for reasons other than a consultation. The work undertaken was preliminary in nature and not intended to address issues relating to the overall effectiveness of the RTF intervention or the related issues pertaining to the timeliness or mode of feedback to practices. Such research would require both considerable additional time and considerable resources to allow for definitive studies to be undertaken.
The implementation in each practice lasted for one 12-week period. In some cases staff noted that they had felt better able to engage with the process because they knew that it was time limited, whereas others believed that more time was needed for RTF to become part of the normal routine of the practice. Future studies would need to consider the optimum time period for collecting RTF in general practice, perhaps favouring a more intensive effort to collect feedback for a shorter period of time with the process being repeated after a suitable interval to assess the impact of any resulting service changes on patient experience.
Although a key, responsive contact within the practice is an important factor influencing the success of a time-limited research study, spreading information and motivation throughout the practice is crucial. This requires good communication between staff groups and individuals, to foster a sense of involvement at all stages of implementation and thereby achieve ‘buy-in’ from the whole practice.
Real-time feedback content also needs to be relevant to the concerns of the practice and patients. Some staff were critical of the volume of demographic details required from respondents. Although such information was necessary to address the research objectives, it did not reflect the interests of all practice teams. Greater practice and patient involvement with the design and content of the RTF survey may achieve a greater sense of ownership and involvement.
Many of the challenges involved in successfully implementing RTF within practices revolve around the issues of timing. The issues involved include avoiding ‘feedback fatigue’ (in staff and patients) and duplication of effort by blending RTF with other feedback initiatives and ensuring that teams make contingency plans that take account of busy times within the practice. Consideration also needs to be given to making the best use of patients’ time, for example patients may have more time and be more willing to use touch screens to provide feedback about practice services while they are waiting to see a health professional than after their consultation.
Conclusions
Despite the low RTF response rate observed when touch screens were located in general practice waiting areas, patients and practice staff were broadly positive about the concept of RTF. Enhanced buy-in from practice staff and patients might be achieved in a number of ways. This includes involving practices in the design and content of RTF surveys and addressing language barriers and patient concerns about the use of technology. A shared responsibility within practices to promote and support RTF may result in more proactive encouragement and support of patients to use touch screen equipment in the waiting area. A longer overall implementation period may be required, during which shorter ‘bursts’ of RTF collection and reporting occur, thus allowing a more thorough assessment of the degree to which RTF can become embedded into general practice and used to improve the patient experience. Our reflections on how this work might inform a future trial are outlined in Box 6.
-
Recruiting and randomising practices to take part in such a study is feasible.
-
Engaging the whole practice team is of vital importance for the successful implementation of RTF in practice; in particular, this requires ensuring engagement and ‘buy-in’ from staff involved in supporting the day-to-day delivery of RTF, most commonly reception staff.
-
It is possible, over time, to attain an acceptable sample size of participating patients, even when full staff ‘buy-in’ has not been achieved.
-
There is a need to focus effort on securing participation from younger and older patients, patients from ethnic minority groups and those with English-language difficulties.
-
Real-time feedback-based interventions may be costly to implement in practice, at approximately £5 per participant recruited.
-
Attention needs to be paid to the physical configuration and context of the RTF process.
-
There is a need for flexible assignment of roles to support RTF implementation and secure patient participation.
-
Facilitated feedback is desirable to support RTF in practice and is welcomed by practice staff.
-
Multimethods approaches to evaluation are advantageous.
Chapter 11 The validity and use of patient experience survey data in out-of-hours care
Parts of this chapter are based on Warren et al. 268 under the terms of the Creative Commons CC-BY-NC license, which permits non-commercial use, distribution and reproduction in any medium, provided the original work is properly cited.
Abstract
Background
In England, out-of-hours GP services provide urgent medical care to patients when their GP surgeries are closed. National Quality Requirement 5 (NQR5) requires out-of-hours services to routinely audit patient experiences, but provides no guidance on methods. In the absence of comparable data from providers, the out-of-hours items from the national GP Patient Survey have been used to monitor patient experience.
Aims
The aims of this study were to (1) explore whether variation in service users’ experiences of care were driven by user or provider characteristics, (2) document the validity of out-of-hours GP Patient Survey items and (3) understand how providers collect/use patient feedback to drive service improvements.
Methods
This was a multimethod study, analysing out-of-hours items from the GP Patient Survey data set (2012/13; 971,232 service users) and a bespoke survey (six providers; 1396 services users) and including a qualitative interview study with staff (11 providers and 31 staff).
Findings
Service users provided less positive ratings of out-of-hours care provided by commercial organisations than for out-of-hours care provided by either NHS or not-for-profit providers. Service users whose ethnic origin was ‘non-white’ or those finding it difficult to take time off work to attend their general practice also reported poorer experiences. GP Patient Survey data, subject to minor modifications, appeared valid and thus suitable for benchmarking. However, the items need updating to reflect the changes made to accessing out-of-hours services by telephone. Patient feedback (including GP Patient Survey data) has a limited role in driving changes to out-of-hours service provision, in part because of the lack of clarity of NQR5.
Conclusions
Out-of-hours items on the GP Patient Survey require refinement but appear suitable for benchmarking purposes. NQR5 is ambiguous and requires revision to assist providers in collecting and acting on patient feedback.
Introduction and rationale
Defining out-of-hours GP care
In England, out-of-hours GP services provide urgent medical care to patients when their GP surgeries are closed, that is, between 18.30 and 08.00 on weekdays and at weekends and bank holidays. Although medical care is largely provided by GPs, nurses and emergency care practitioners may also provide clinical care. Out-of-hours services are provided to manage health problems that cannot wait until the next working day; these services are not intended as an alternative route to health care for non-urgent problems for those patients who cannot attend during practice opening hours. Recent English national audit data reported that out-of-hours GP services handled around 5.8 million contacts during 2013–14, of which 3.3 million were face-to-face patient consultations. 269
The provision of out-of-hours GP care has changed significantly in England over the last decade. In 2004, responsibility for out-of-hours services transferred from local GPs to NHS primary care commissioners. Commissioners are now responsible for purchasing care from provider organisations and, in some regions, very different models of care have emerged. In England there are currently 211 clinical commissioning groups269 commissioning out-of-hours services, although the number of providers is smaller, as many providers contract with two or more neighbouring commissioners. Out-of-hours services are also provided by different types of organisations, including NHS trusts, not-for-profit providers (e.g. social enterprises) and commercial health-care providers. 65,270 Such services continue to evolve; since the phased introduction of the NHS 111 service, completed in February 2014,269 different providers may provide different aspects of care, for example call handling and delivery of clinical care, in the same geographical area.
Ensuring quality and safety of out-of-hours care
Although the reorganisation of out-of-hours GP care has the potential to bring about new approaches and increased efficiency of service provision, such reconfiguration may also generate reduced service coverage or quality. To tackle concerns regarding the quality of care provided, national standards were published with which all out-of-hours GP care providers were expected to comply. 66 Providers are required to report their performance to their commissioners across a range of NQR recommendations. Of particular relevance to the IMPROVE programme is recommendation 5 (NQR5), which mandates out-of-hours providers to regularly audit a random sample of patients’ experiences and to take appropriate action based on the results.
Despite the introduction of the NQRs, criticism of the quality and safety of out-of-hours care persists. 65,271 Prompted by the death of a patient in 2008, the CQC investigated the case and produced additional recommendations for commissioners and providers of out-of-hours services regarding performance assessment. 270 More widely, urgent care provision in England has been criticised regarding service accessibility, the lack of continuity of care and concerns about patient safety. 272–275 Within this context, the CQC has recently assumed responsibility for regulating and inspecting the quality and safety of out-of-hours primary care services. 276 With CQC inspections commencing in October 2014, the latest CQC overview reported that the majority of service provision was of high quality, but that there were some areas in which improvements could be made. 277
Role of patient experience surveys in quality assessment
Since 2015, service commissioners are expected to publish annual data on provider performance against the NQRs. 278 Such a requirement is problematic for NQR5, as there is no agreed methodology for conducting patient experience audits. Without reliable and valid methods of assessing patient experience, it is impossible for providers to accurately assess their own performance and to subsequently use this information to guide service improvement. Providers may also use different tools and survey methods and the resultant data cannot be used for the purposes of benchmarking to assess variations in service quality between providers. Although a number of standardised patient questionnaires are available to assess patient experiences of out-of-hours primary care services,279 these tools have not been widely adopted in routine practice.
Although it is not possible to benchmark out-of-hours providers using the patient experience data collected for NQR5, the 2014 national audit of GP out-of-hours care269 and the CQC both analysed patient experience data from the English GP Patient Survey. The GP Patient Survey includes six items relating to out-of-hours care (two ‘access’ and four ‘evaluative’ items). As the only large-scale population survey of patients’ understanding, use and experiences of out-of-hours care, benchmarking of GP Patient Survey data is potentially possible. Establishing the validity of the GP Patient Survey out-of-hours items is, however, an important prerequisite to using this data to document variation in scores between out-of-hours services and for benchmarking. We have previously published evidence to support the reliability of the GP Patient Survey (including out-of-hours items). 141 Using a range of different methods and analytical approaches, we have also demonstrated the validity of GP Patient Survey items evaluating in-hours primary care services,131,280 but this has yet to be established for out-of-hours care items.
Once the causes of poor patient experience of out-of-hours care have been understood, interventions to improve care can then be designed. However, the current literature on the effects of feedback of patient assessments is insufficient in scope, quality and consistency to design effective interventions targeting service delivery and organisation or the performance of clinicians. 21,83,281
Rationale for the out-of-hours research
This research was designed to address these gaps in our knowledge to enable managers, patients and professionals to have confidence in the meaning of patient assessments of out-of-hours primary care services recorded in the national GP Patient Survey. The work package addressed three important areas.
The first workstream built on earlier analysis of the GP Patient Survey, which reported that important sociodemographic variations exist in patient experiences of in-hours primary care services,131 but did not examine if such variations existed for out-of-hours items. Given that the CQC and National Audit Office have both used the GP Patient Survey to monitor service users’ experiences of out-of-hours care, it is important to understand whether or not variation in service users’ experiences of care is driven by user characteristics, as opposed to differences in the care provided by different types of providers.
The second workstream sought to explore the validity of the out-of-hours items from the GP Patient Survey. The Out-of-hours Patient Questionnaire (OPQ) is a complementary tool to the GP Patient Survey, which collects more detailed information on patient experience of out-of-hours care and has undergone more extensive testing and validation. 8,90,282 The second workstream tested the performance of GP Patient Survey out-of-hours questions against data derived from the OPQ to examine the validity of GP Patient Survey items.
The third workstream examined how out-of-hours GP services make sense of the information provided by patient questionnaires and, when possible, use this information to design interventions to improve patient experience through service reconfiguration and development.
Structure of the out-of-hours work package
The out-of-hours work package consisted of three workstreams, each of which used different data sets and methods. The remainder of this chapter describes the study aims and objectives, methods, results and discussion arising from each of the three workstreams in turn, before summarising the key conclusions that arose from the work programme.
Stakeholder advisory group
A stakeholder advisory group composed of three representatives from out-of-hours service providers, two primary care academics and a service user was convened to support workstreams 2 and 3. The group met to review study methods and procedures in light of the findings of preliminary piloting and testing of the methods (see Workstream 2) and to comment on topic guides supporting interviewing in workstream 3. Because of the logistical challenges of organising face-to-face meetings around staff availability, after an initial face-to-face meeting most advisory group input was secured by e-mail communication and telephone.
The original aim was to recruit two service users through our links with local service providers and using methods recommended by our Exeter University-supported PPI groups [see http://clahrc-peninsula.nihr.ac.uk/patient-and-public-involvement-in-research and www.folkus.org.uk (accessed 13 December 2016)]. Potential service user participants were provided with a brief information sheet regarding what would be involved in advisory group membership and were informed that any costs incurred in preparing for or attending advisory group meetings would be reimbursed. Despite significant efforts to secure lay stakeholder participation, it proved difficult to recruit service users with relevant, lived experience to the advisory board. Although this was problematic to the research, provider staff members indicated that their services experienced similar problems, probably because of the nature by which patients consulted (i.e. relatively infrequent consulters seeking care for an urgent problem) and the lack of continuity between provider and service user.
Changes to study methods from the original protocol
The overall aim of this strand of work, as stated in the original protocol, was to investigate how the results of the GP Patient Survey can be used to improve patients’ experience of out-of-hours care (aim 7).
In our original application we specified four objectives within this work package, three of which were successfully addressed within this programme (objective 1: cognitive testing of GP Patient Survey out-of-hours items; objective 2: establishing GP Patient Survey item validity and reliability; and objective 3: identifying how data from the GP Patient Survey can be effectively used to inform out-of-hours service reconfiguration). Objective 4, undertaking preliminary piloting of an intervention to improve patient experiences of out-of-hours care, was not achieved. The qualitative research undertaken to address aim 3 identified significant heterogeneity in terms of how providers collected and acted on patient feedback and in terms of the perceived utility of the GP Patient Survey as a platform on which to mount quality improvement. It was clear on completion of the qualitative work with service providers that more research was needed to design and then test the feasibility and acceptability of an intervention to embed patient feedback within quality improvement cycles.
For the three objectives that were achieved, some minor modifications to the study methods were implemented as the full protocols were developed. For example, it was initially proposed to interview up to 45 patients to test user responses to out-of-hours GP Patient Survey items. In reality, only 20 service users underwent cognitive interviewing, as this proved sufficient for testing the validity of the items. Similarly, to address objective 3, a more ambitious, qualitative interview study was undertaken with staff members from out-of-hours services. Here, 11 English providers (rather than six) were sampled and interviewed to ensure greater diversity in the types of provider organisation and the populations served.
Workstream 1: exploring variations in national GP Patient Survey out-of-hours items
Study aims and objectives
This workstream investigated:
-
potential associations between service users’ evaluations of out-of-hours GP care and individual-level sociodemographic factors
-
whether or not variations in evaluations were related to ‘clustering’ of service users reporting poorer experience within providers reporting poorer performance overall
-
whether or not there was an association between service users’ evaluations and type of provider organisation (NHS, commercial or not-for-profit organisations).
To address these aims, an analysis of service users’ ratings of out-of-hours GP care from GP Patient Survey data was undertaken.
Methods
Patient questionnaires
GP Patient Survey data (July–September 2012 and January–March 2013) were analysed [overall response rate of 35% (971,232/2,750,000)]. 283 The GP Patient Survey included four evaluative questions on out-of-hours provision, three of which were analysed: ‘timeliness’ of receiving care (‘about right’, ‘took too long’ or ‘don’t know/doesn’t apply’), ‘confidence and trust’ in the out-of-hours clinician (‘yes, definitely’, ‘yes, to some extent’, ‘no, not at all’ or ‘don’t know/can’t say’) and ‘overall experience’ of the out-of-hours GP service (five-point Likert scale from ‘very good’ to ‘very poor’). These questions were completed only by service users who had attempted to contact an out-of-hours GP service within the preceding 6 months.
Service user characteristics
Five sociodemographic variables derived from GP Patient Survey responses were analysed: gender (male as reference), ethnicity (white as reference vs. five categories derived from ONS data284), age in eight categories (18–24 years as reference), parent status (non-parent as reference) and whether the service user was able to take time away from work to attend his or her practice during working hours (individuals ‘not in paid work’ as reference vs. ‘paid work, can take time away’ or ‘paid work, could not take time away’). A sixth sociodemographic variable, deprivation (national IMD fifths; ‘least deprived’ as reference), was determined based on the respondents’ residential postcode. 285
Practice and out-of-hours general practitioner service providers
Each service user was mapped to the out-of-hours GP provider responsible for providing clinical care for the service user’s practice during the 6-month period prior to sending the questionnaire. Mapping was achieved for 96% (934,931/971,232) of service users in the data set; 7886 practices were mapped to 91 out-of-hours GP providers, of which 86 had an identifiable provider organisation type (not-for-profit as reference vs. NHS or commercial).
Statistical methods
Analyses were performed using Stata 12. Sociodemographic data are described for all service users contacting an out-of-hours GP provider in the previous 6 months (for themselves or on behalf of another person). To facilitate comparison between measures on different scales, outcomes were linearly rescaled from 0 to 100,131 with a difference of < 3 points considered ‘small’ in respect of practical significance. 102 Missing data at the level of service users or providers (including ‘don’t know’/’does not apply’) were excluded from the analysis. It was assumed that service user responses would be ‘clustered’ by out-of-hours provider (not practice), with clustering adjusted for as a random effect.
Three statistical models were employed. Model A was a fixed-effect multivariable linear regression model including individual sociodemographic factors as covariates and generated mean differences in outcome scores for comparator sociodemographic groups compared with reference categories, without accounting for differences in outcome across providers. Model B was a mixed-effects model that extended model A by incorporating a random intercept for provider. Model B therefore adjusted for differences in outcome between providers and estimated the mean difference between the comparator group and the reference group in outcome scores within providers. Comparing models A and B identified the extent to which any overall difference between service users of specific sociodemographic groups was due to clustering of service users within providers achieving a low outcome score. 131
Model C extended model B by adding ‘provider type’ as a covariate. This model estimated the effect of provider type, with adjustment for service user characteristics, for each outcome. Comparing the between-provider variance from models B and C quantified the degree of between-provider variation attributable to provider type. The effect of provider type, analogous to an effect size such as Cohen’s d, was expressed as the standardised mean difference (mean difference between comparator provider type and not-for-profit providers divided by the between-provider SD derived from model C).
Results
The sociodemographic characteristics of 106,513 service users (from 7492 practices) who had contacted an out-of-hours provider and were mapped to a provider of a known organisation type are shown in Table 51. Service users’ overall evaluations of out-of-hours GP services were generally positive (Table 52): 71% (73,983/103,523) of participants reported a ‘very good’ or ‘fairly good’ overall experience, although 31% (31,966/104,145) felt that it took too long to receive care.
| Characteristics | Service users, n (%) |
|---|---|
| Gender | |
| Male | 38,553 (36.6) |
| Female | 66,879 (63.4) |
| Total | 105,432 |
| Age (years) | |
| 18–24 | 4850 (4.6) |
| 25–34 | 14,745 (14.0) |
| 35–44 | 20,066 (19.0) |
| 45–54 | 18,699 (17.7) |
| 55–64 | 16,760 (15.9) |
| 65–74 | 14,704 (13.9) |
| 75–84 | 11,201 (10.6) |
| ≥ 85 | 4509 (4.3) |
| Total | 105,534 |
| Ethnic group | |
| White | 90,034 (85.5) |
| Mixed/multiple ethnic groups | 860 (0.8) |
| Asian/Asian British | 7985 (7.6) |
| Black/African/Caribbean/black British | 2471 (2.3) |
| Other ethnic group | 3934 (3.7) |
| Total | 105,284 |
| Mean deprivation quintile | |
| 1 (least deprived) | 19,537 (18.4) |
| 2 | 20,672 (19.4) |
| 3 | 21,633 (20.3) |
| 4 | 21,486 (20.2) |
| 5 (most deprived) | 23,028 (21.7) |
| Total | 106,356 |
| Parent/guardian of children aged < 16 years? | |
| No | 61,276 (62.8) |
| Yes | 36,277 (37.2) |
| Total | 97,553 |
| Can you take time away from work to see a GP during your typical working hours?b | |
| Not relevantc | 51,027 (51.3, NA) |
| Yes | 31,298 (31.5, 64.7) |
| No | 17,057 (17.2, 35.3) |
| Total | 99,382 |
| Total relevant | 48,355 |
| Question | Response frequency,a n (%) |
|---|---|
| How do you feel about how quickly you received care from the out-of-hours GP service? | |
| It was about right | 65,298 (62.7) |
| It took too long | 31,966 (30.7) |
| Don’t know/doesn’t apply | 6881 (6.6) |
| Total | 104,145 |
| Did you have confidence and trust in the out-of-hours clinician you saw or spoke to? | |
| Yes, definitely | 42,264 (40.7) |
| Yes, to some extent | 42,938 (41.3) |
| No, not at all | 12,222 (11.8) |
| Don’t know/can’t say | 6490 (6.2) |
| Total | 103,914 |
| Overall, how would you describe your experience of out-of-hours GP services? | |
| Very good | 33,662 (32.5) |
| Fairly good | 40,321 (38.9) |
| Neither good nor poor | 15,638 (15.1) |
| Fairly poor | 8140 (7.9) |
| Very poor | 5762 (5.6) |
| Total | 103,523 |
Data were included for 86 providers: 44 not-for-profit, 21 NHS and 21 commercial providers. Provider type was associated with all three outcomes (global p-value < 0.001 for confidence and trust and overall experience, p-value = 0.013 for timeliness). No statistically significant differences were observed between NHS and not-for-profit organisations with regard to any of the outcomes, whereas commercial providers scored lower than not-for-profit organisations for all three outcomes (Table 53). The magnitude of these differences was approximately 3 points (model C) for all outcomes.
| Provider typea | Mean differenceb (95% CI) | p-valuec | Standardised mean difference |
|---|---|---|---|
| Timeliness of out-of-hours GP care | |||
| Model Cd,e (providers, n = 86; service users, n = 83,176); between-provider SD 5.19 | |||
| NHS | 1.28 (–1.61 to 4.17) | 0.013 | 0.25 |
| Commercial | –3.52 (–6.40 to –0.64) | –0.68 | |
| Confidence and trust in out-of-hours clinician | |||
| Model Cd,e (providers, n = 86; service users, n = 83,316); between-provider SD 3.14 | |||
| NHS | 1.00 (–0.79 to 2.79) | < 0.001 | 0.32 |
| Commercial | –3.25 (–5.03 to –1.46) | –1.04 | |
| Overall experience of out-of-hours GP care | |||
| Model Cd,e (providers, n = 86; service users, n = 88,423); between-provider SD 3.33 | |||
| NHS | 1.07 (–0.77 to 2.90) | < 0.001 | 0.32 |
| Commercial | –3.13 (–4.96 to –1.30) | –0.94 | |
A comparison of the between-provider variance (model B vs. model C) for overall experience of care observed that 18.6% of the between-provider variability was the result of provider type (Table 54 and see Table 53); the equivalent values for timeliness and confidence and trust were 11.3% (Table 55 and see Table 53) and 20.9% (Table 56 and see Table 53). The standardised mean difference for commercial providers compared with not-for profit providers was –0.68 SDs for timeliness, –1.04 SDs for confidence and trust and –0.94 SDs for overall experience (see Table 53). This equates to a moderate (timeliness) or large (confidence and trust and overall experience) effect size attributable to commercial provider type.
| Sociodemographic covariate | Model Aa (n = 88,423), overall difference | Model Ba,b (providers, n = 86; service users, n = 88,423), within out-of-hours provider difference (between-provider SD 3.69) | Percentage of overall difference (if negative) attributable to clustering of sociodemographic group in lower-scoring providers | ||
|---|---|---|---|---|---|
| Mean differencec (95% CI) | p-valued | Mean differencec (95% CI) | p-valued | ||
| Ethnic groupe | |||||
| Mixed | –3.44 (–5.47 to –1.41) | < 0.001 | –2.01 (–4.03 to 0.01) | < 0.001 | 42 |
| Asian | –5.61 (–6.32 to –4.90) | –3.62 (–4.36 to –2.89) | 35 | ||
| Black | –2.14 (–3.40 to –0.89) | 0.13 (–1.14 to 1.40) | > 100 | ||
| Other | –0.75 (–1.78 to 0.27) | 1.29 (0.25 to 2.32) | > 100 | ||
| Able to take time away from work during typical working hoursf | |||||
| Yes | 1.30 (0.82 to 1.78) | < 0.001 | 1.29 (0.81 to 1.76) | < 0.001 | Not applicable |
| No | –4.79 (–5.36 to –4.23) | –4.73 (–5.29 to –4.17) | 1 | ||
| Sociodemographic covariate | Model Aa (n = 83,176), overall difference | Model Ba,b (providers, n = 86; service users, n = 83,176), within out-of-hours provider difference (between-provider SD 5.51) | Percentage of overall difference (if negative) attributable to clustering of sociodemographic group in lower-scoring providers | ||
|---|---|---|---|---|---|
| Mean differencec (95% CI) | p-valued | Mean differencec (95% CI) | p-valued | ||
| Ethnic groupe | |||||
| Mixed | –4.78 (–8.34 to –1.23) | < 0.001 | –3.45 (–6.99 to 0.09) | < 0.001 | 28 |
| Asian | –13.27 (–14.51 to –12.03) | –11.08 (–12.37 to –9.79) | 17 | ||
| Black | –7.64 (–9.86 to –5.42) | –5.67 (–7.92 to –3.42) | 26 | ||
| Other | –8.44 (–10.24 to –6.64) | –6.57 (–8.40 to –4.75) | 22 | ||
| Able to take time away from work during typical working hoursf | |||||
| Yes | 3.45 (2.62 to 4.27) | < 0.001 | 3.48 (2.65 to 4.30) | < 0.001 | Not applicable |
| No | –6.58 (–7.56 to –5.61) | –6.48 (–7.45 to –5.51) | 2 | ||
| Sociodemographic covariate | Model Aa (n = 83,316), overall difference | Model Ba,b (providers, n = 86; service users, n = 83,316), within out-of-hours provider difference (between-provider SD 3.53) | Percentage of overall difference (if negative) attributable to clustering of sociodemographic group in lower-scoring providers | ||
|---|---|---|---|---|---|
| Mean differencec (95% CI) | p-valued | Mean differencec (95% CI) | p-valued | ||
| Ethnic groupe | |||||
| Mixed | –3.02 (–5.58 to –0.46) | < 0.001 | –1.72 (–4.27 to 0.84) | < 0.001 | 43 |
| Asian | –5.95 (–6.85 to –5.05) | –3.92 (–4.86 to –2.99) | 34 | ||
| Black | –2.62 (–4.22 to –1.02) | –0.33 (–1.95 to 1.29) | 88 | ||
| Other | –1.18 (–2.48 to 0.13) | 0.87 (–0.46 to 2.19) | > 100 | ||
| Able to take time away from work during typical working hoursf | |||||
| Yes | 2.24 (1.64 to 2.84) | < 0.001 | 2.23 (1.63 to 2.82) | < 0.001 | Not applicable |
| No | –5.35 (–6.05 to –4.64) | –5.27 (–5.97 to –4.57) | 1 | ||
Service users of mixed ethnicity and Asian ethnicity reported poorer care for all three outcomes than white respondents; a more variable pattern of care was evident for service users of black ethnicity and other ethnicity (see Tables 54–56). In general, the mean differences in scores between white service users and service users from the mixed, black and other ethnic groups tended to be of lower magnitude that those between Asian and white service users.
A comparison of models A and B indicated that, with regard to timeliness, only 17% of the mean difference in scores between Asian and white service users derived from model A (–13.27, 95% CI –14.51 to –12.03; see Table 55) was due to clustering of Asian service users within providers that scored lower overall (vs. 28%, 26% and 22% for mixed, black and other ethnicity service users, respectively). For overall experience of care, 35% of the mean difference between Asian and white service users derived from model A (–5.61, 95% CI –6.32 to –4.90; see Table 54) was attributable to clustering of Asian service users within a lower-scoring provider.
Service users who could not take time away from work to attend their practice reported lower mean scores across all three outcomes than those for whom this was not applicable, whereas service users who could take time away from work reported higher mean scores (see Tables 54–56).
Other individual-level sociodemographic characteristics (gender, age, deprivation and parent status) were also associated with the three outcomes measures (deprivation was associated only with trust and confidence and overall experience) but the effects were not explored further because of the small magnitude of the mean differences when compared with the relevant reference category or because of more positive scores in the comparator category (i.e. potentially more disadvantaged) than in the reference group.
Discussion
Analysis of GP Patient Survey data identified that commercial provider organisations were associated with poorer reports of care across all three outcome measures when compared with not-for-profit organisations after controlling for patient-level sociodemographic characteristics. The lower scores associated with commercial providers is consistent with observations from US data showing that for-profit hospitals were associated with worse patient experiences than non-profit hospitals. 286,287 However, the reasons underlying the lower scores for commercial organisations, even after controlling for individual sociodemographic variables, are unclear. This may reflect a genuinely poorer experience of care provided by commercial providers or the willingness of commercial providers to operate in areas deemed less attractive to NHS or not-for-profit organisations. It may also be that service users’ perceptions of provider type influenced their ratings, although it is questionable whether or not service users are aware whether their provider was a commercial organisation as opposed to a NHS or not-for-profit organisation, except perhaps in areas where media attention has focused on their local service.
Service users from minority ethnic groups tended to report less favourable care than white service users, with some variation observed across out-of-hours providers. This finding was in part attributable to clustering of minority ethnic service users in out-of-hours GP services with lower overall scores. Previous analysis of GP Patient Survey data regarding ‘in-hours’ care has indicated that minority ethnic patients reported generally lower experience scores131 and that patients of different ethnic backgrounds may differ with regard to drivers of satisfaction. 102 In our analyses, although Asian service users reported lower mean scores than white service users for all three experience outcomes, the greatest difference was in the timeliness of care. Similar differences were seen for other ethnic groups, but of a lesser magnitude, suggesting that service users from minority ethnic groups, and Asian service users in particular, place substantial value on the timeliness of out-of-hours care. The ability of an out-of-hours GP service to meet service users’ expectations has previously been argued to be a strong driver of satisfaction with care,288 although this cross-sectional analysis cannot definitively answer this question.
Those who were unable to attend their practice because of work commitments were significantly associated with lower scores across all three outcomes than those not in paid work, whereas individuals who reported being able to take time off work reported somewhat better experiences. One explanation is that out-of-hours providers, who do not provide routine ‘non-urgent’ care, may not meet the expectations of service users who find it difficult to attend their practice during regular hours. However, as no information on the nature or the urgency of the service users’ health conditions was available this question cannot be addressed definitively.
Strengths and limitations
Unlike CQC and national audit data, this analysis of GP Patient Survey data was the first to map the majority of practices (and hence service users) to a specified out-of-hours GP provider and to determine the organisational provider type. The large sample available enabled sophisticated modelling to test the associations between provider and service user sociodemographic characteristics and service user evaluations of care.
Several limitations were evident regarding the data available from the GP Patient Survey. Service users were invited to provide feedback on their experiences of out-of-hours care in the preceding 6 months. Recall bias cannot be discounted, as previous research has found that older patients may not accurately report health service resource use over the short time frame of 3 months. 289 No data were collected regarding the nature/urgency of the service users’ complaints, the time/date of the contacts or how the contacts were managed. Although data on ethnicity were collected, the GP Patient Survey did not ask about service users’ English language ability, nor about educational attainment, both of which may be related to experience of care. 80 The lack of detailed response options regarding whether or not the service user was able to take time away from work and the timeliness of care also restricted our ability to interpret these data.
The GP Patient Survey response rate of 35% is also problematic. However, no evidence of an adverse association between response rate and non-response bias has been found for the GP Patient Survey and previous research using rigorous probability sampling methods (as used in the GP Patient Survey) has observed only a weak association between non-response rates and non-response bias. 133,187,290 An analysis of data on out-of-hours care in the Netherlands suggested that non-response bias was small in respect of overall satisfaction with out-of-hours care. 291
Workstream 2: establishing the validity of GP Patient Survey out-of-hours items
Study aims and objectives
The overarching aim of this workstream was to establish the validity of the GP out-of-hours care items within the GP Patient Survey to inform its suitability for benchmarking providers. This was achieved through a multimethod project composed of two stages. In the first stage, preliminary psychometric testing of the out-of-hours items was undertaken through cognitive interviews, combined with a pilot survey of out-of-hours users to test survey methods. The second stage tested the hypothesis that the GP Patient Survey items (modified after piloting) would demonstrate construct validity if together the GP Patient Survey items were correlated with the two known subscales of the OPQ (an established, valid and reliable measure of patient experience8,282). Concurrent validity would be established if the thematically relevant OPQ items were found to be associated with each of the GP Patient Survey items in linear regression modelling.
Methods
Settings
Six out-of-hours providers across England were recruited for a cross-sectional survey of service users. Data from year 5, quarter 2 (July to September 2010) of the GP Patient Survey11 were used to sample providers to ensure that there was variation in respect of performance (high/medium/low scoring) on respondents’ overall ratings of care received by GP out-of-hours services, as well as the type of provider (NHS, commercial, social enterprise) and the geographical area covered by the service (inner city/suburban, rural). Two participating service providers were operated by NHS trusts, three were operated by commercial companies and one was a not-for-profit social enterprise.
Survey piloting and cognitive interviews
A pilot study was conducted with two providers, with study questionnaires distributed to 500 service users (n = 250 per provider). Cognitive interviews with out-of-hours service users were conducted to explore the cognitive challenges faced by service users when completing the GP Patient Survey out-of-hours items and establish the validity of the item set. Twenty service users (predominantly female and aged ≥ 65 years) from two out-of-hours providers were interviewed using a think-aloud and four-stage verbal probing approach. 292 Interviews were audio recorded, transcribed verbatim and analysed using protocol analysis. 292
This preliminary work highlighted issues with the GP Patient Survey questions and with sampling of service users. The GP Patient Survey filters respondents to the out-of-hours items if they report having tried to make contact with a GP out-of-hours service in the previous 6 months, either for themselves or for someone else. As the respondents in this study were sampled from known users of out-of-hours providers, respondents were requested to evaluate their experience of the last time they made contact with a GP out-of-hours service. Minor modifications to the wording of the GP Patient Survey out-of-hours items (one item) and/or response options (Table 57) and sampling exclusion criteria were suggested by the study team. These changes were reviewed and approved by the study advisory group prior to commencing data collection.
| GPPS item wording | GPPS response options | Revised item wording | Revised response options |
|---|---|---|---|
| Q38. How easy was it to contact the out-of-hours GP service by telephone? | Very easy; fairly easy; not very easy; not at all easy; don’t know/didn’t make contact | No changes made | Very easy; fairly easy; not very easy; not at all easy; don’t know/didn’t make contact by telephone |
| Q39. How do you feel about how quickly you received care from the out-of-hours GP service? | It was about right; it was too long; don’t know/doesn’t apply | No changes made | It was quicker than expected; it was about right; it was too long; don’t know/doesn’t apply |
| Q40. Did you have confidence and trust in the out-of-hours clinician you saw or spoke to? | Yes, definitely; yes, to some extent; no, not at all; don’t know/can’t say | Did you have confidence and trust in the out-of-hours health-care professional you consulted with? | No changes made |
| Q41. Overall, how would you describe your experience of the out-of-hours GP service? | Very good; fairly good; neither good nor poor; fairly poor; very poor | No changes made | No changes made |
Description of the questionnaire
The questionnaire had two sections. Section 1 contained the four modified GP Patient Survey evaluative stem items (applicable to all participants). These four items assessed service users’ ratings of the ‘entry access’ to the service, the ‘timeliness of care’ received, their ‘confidence and trust’ in the health professional who they consulted with and their ‘overall experience’ of the out-of-hours service. Section 2 consisted of the OPQ, which is composed of seven sections designed to capture information on the entirety of service users’ experience of out-of-hours care. The composition of the OPQ has been detailed elsewhere8 and was found to be both valid and reliable. Participants’ ratings on 14 evaluative items were analysed (Table 58); these were not management specific and assessed users’ experience of entry to the service, the outcome of their call and the consultation with a health professional.
| Questionnaire section | Item | Response scale |
|---|---|---|
| Making contact with the service | How do you rate [how long it took your call to be answered, excluding any introductory message]? | Five-point scale: ‘very poor’ to ‘excellent’ |
| Please rate the helpfulness of the call operator | Five-point scale: ‘very poor’ to ‘excellent’ | |
| Please rate the extent to which you felt the call operator listened to you | Five-point scale: ‘very poor’ to ‘excellent’ | |
| Making contact with the service | How do you rate [how long it took for a health professional to call you back]? | Five-point scale: ‘very poor’ to ‘excellent’ |
| Were you happy with the type of care you received? | Yes/no | |
| How do you rate [the length of your consultation with the health professional]? | Five-point scale: ‘very poor’ to ‘excellent’ | |
| [Please rate] the thoroughness of the consultation | Five-point scale: ‘very poor’ to ‘excellent’, plus NA | |
| Outcome of your call | [Please rate] the accuracy of the diagnosis | Five-point scale: ‘very poor’ to ‘excellent’, plus NA |
| Consultation with the health professional | [Please rate] the treatment you were given | Five-point scale: ‘very poor’ to ‘excellent’, plus NA |
| [Please rate] the advice and information you were given | Five-point scale: ‘very poor’ to ‘excellent’, plus NA | |
| [Please rate] the warmth of the health professional’s manner | Five-point scale: ‘very poor’ to ‘excellent’, plus NA | |
| [Please rate] the extent to which you felt listened to | Five-point scale: ‘very poor’ to ‘excellent’, plus NA | |
| [Please rate] the extent to which you felt things were explained to you | Five-point scale: ‘very poor’ to ‘excellent’, plus NA | |
| [Please rate] the respect you were shown | Five-point scale: ‘very poor’ to ‘excellent’, plus NA |
Sampling
Sampling took place within 2 weeks of the service user contacting the out-of-hours service. The contact and demographic details for a random sample of 2000 service users were extracted from the electronic records at each site. Exclusion criteria were age 12–17 years, because of the risk of breaching patient confidentiality if a questionnaire was sent to a patient’s home address and because the GP Patient Survey targets those aged ≥ 18 years; admission to hospital as a result of the contact; palliative care needs; or a temporary/incomplete address. After all exclusions were applied, a questionnaire, accompanied by covering letters from the research team and service provider, an information sheet and a prepaid envelope, was sent to a consecutive sample of the first eligible 850 service users (or a parent or a guardian if the service user was a child) from the sampling frame at each site. In one area, only 818 service users were sampled because of logistical constraints in the screening process. The total sample approached therefore totalled 5068 service users. A reminder was sent 2 weeks after the initial mailing to non-respondents. Implicit consent was assumed if a completed questionnaire was received by the research team; no reminder was sent to service users who returned a blank questionnaire. Data collection took place between September 2013 and July 2014.
Data analysis
Respondents were compared with non-respondents with respect to their age, gender, deprivation quintile (using service users’ postcodes to derive their IMD285) and management option received as a result of the last recorded contact (from the service provider record: telephone advice, treatment centre attendance, home visit) using a multilevel logistic regression model, clustering respondents by the provider from which they were sampled.
Construct validity
Construct validity of the four modified GP Patient Survey items was assessed by ascertaining how well they summarised the OPQ. First, a confirmatory factor analysis was conducted to establish whether or not the OPQ possessed the same two-factor structure reported in the paper detailing its development. 8 The standardised factor loadings with 95% CIs for this model are reported. As Hu and Bentler293 suggest, goodness of fit of the model was assessed through a two-index strategy using the standardised root-mean-squared residual supplemented with the comparative fit index (CFI),294 neither of which are adversely affected by large sample sizes. 295
A principal component analysis (PCA) of the four modified GP Patient Survey items was then conducted to establish their latent structure, using the polychoric correlation matrix to account for the ordinal nature of these items. 296 Inspection of eigenvalues and component loadings were used to explore the underlying structure of the responses. Based on this PCA, the construction of scales using the modified GP Patient Survey items and their internal consistency (Cronbach’s alpha) was explored. Finally, the correlations between the scales constructed above and the factor scores from the confirmatory factor analysis of the OPQ were investigated to assess the extent to which the modified GP Patient Survey item set summarised the OPQ.
Consultation satisfaction scale
The OPQ includes nine items rating service users’ satisfaction with their consultation with an out-of-hours clinician (see Table 58). These items were combined into a ‘consultation satisfaction’ scale, as suggested by the paper validating the OPQ,8 to avoid issues of multicollinearity in the regression models. To achieve this, each item was linearised to a 0–100 scale and respondents’ mean scores from the nine items were derived as their consultation satisfaction scale score, provided that they had answered at least four of the items. Finally, the scale was standardised so that the regression modelling would produce standardised coefficients.
Concurrent validity
To investigate the concurrent validity of the modified GP Patient Survey items, four multilevel linear regression models were constructed, with a separate model for each evaluative outcome. The covariates were the management non-specific items from the OPQ (see Table 58), including the consultation satisfaction scale. Concurrent validity was considered to be established if each modified GP Patient Survey outcome was found to be significantly associated with thematically related items from the OPQ. Univariate analyses were undertaken first, with covariates being excluded from the final models if they were not associated (p < 0.10) with any of the four outcomes. All models controlled for service users’ age, gender, deprivation quintile and management option, as well as the type of provider contacted (NHS, commercial, not-for-profit), and were clustered by provider. Missing data were accounted for using multiple imputations. To ensure that the regression coefficients of the covariates were comparable across models, the four modified GP Patient Survey outcomes, which originally had differing response scales (see Table 57), were standardised. Sensitivity analyses were conducted to test for a linear trend over the covariate rating length of time taken for a health professional to call back, modelling the data while excluding those who answered ‘not applicable’ (n = 192). All analyses were performed using Stata 13.
Results
Response rate and sample
Completed questionnaires were received from 1396 out of 5068 (27.5%) sampled service users. The multilevel logistic regression assessing response indicated that responders were older and more affluent (lower IMD score), but did not differ with respect to gender. Differences in response rates were also evident across the management options (Table 59). The response distributions for all variables of interest are displayed in Appendix 6 (see Table 81).
| Characteristic | Responders | Non-responders | p-valuea |
|---|---|---|---|
| Patients, n (%) | 1396 (27.6) | 3672 (72.4) | |
| Age (years), mean (SD) | 46.0 (28.2) | 32.5 (26.2) | < 0.001 |
| Gender female, n (%) | 877 (62.8) | 2208 (71.6) | 0.081 |
| IMD score, mean (SD) | 19.0 (14.0) | 23.9 (15.9) | < 0.001 |
| Management option, n (%) | |||
| Telephone advice | 492 (35.2) | 1143 (31.1) | 0.001 |
| Treatment centre | 647 (46.3) | 1765 (48.1) | |
| Home visit | 172 (12.3) | 301 (8.2) | |
| Other | 85 (6.1) | 193 (5.3) | |
Construct validity
Confirmatory factor analysis of the Out-of-hours Patient Questionnaire
The confirmatory factor analysis revealed that the data fit the proposed entry access and consultation satisfaction two-factor structure reported by Campbell et al. 8 moderately well (Table 60), with a standardised root-mean-squared residual of 0.06 (values of < 0.08 represent good fit) and a CFI of 0.89, which is just short of the suggested cut-off of 0.90 for good fit. 293 In line with Campbell et al. ,8 the two latent variables were moderately correlated (r = 0.54, p < 0.001).
| OPQ item | Coefficienta | 95% CI | p-value |
|---|---|---|---|
| Entry access | |||
| How do you rate [how long it took your call to be answered]? | 0.65 | 0.61 to 0.70 | < 0.001 |
| Please rate the helpfulness of the call operator | 0.91 | 0.89 to 0.93 | < 0.001 |
| Please rate the extent to which you felt the call operator listened to you | 0.90 | 0.88 to 0.92 | < 0.001 |
| How do you rate [how long it took for a health professional to call you back]? | 0.66 | 0.62 to 0.70 | < 0.001 |
| Consultation satisfaction | |||
| Were you happy with the type of care you received? [no/yes] | 0.47 | 0.41 to 0.52 | < 0.001 |
| How do you rate [the length of your consultation with the health professional]? | 0.80 | 0.77 to 0.83 | < 0.001 |
| [Please rate] the thoroughness of the consultation | 0.88 | 0.86 to 0.89 | < 0.001 |
| [Please rate] the accuracy of the diagnosis | 0.84 | 0.81 to 0.86 | < 0.001 |
| [Please rate] the treatment you were given | 0.86 | 0.84 to 0.88 | < 0.001 |
| [Please rate] the advice and information you were given | 0.90 | 0.88 to 0.91 | < 0.001 |
| [Please rate] the warmth of the health professional’s manner | 0.87 | 0.85 to 0.89 | < 0.001 |
| [Please rate] the extent to which you felt listened to | 0.93 | 0.92 to 0.94 | < 0.001 |
Principal component analysis of the modified GP Patient Survey items
In the PCA of the four modified GP Patient Survey items there was a single component with an eigenvalue exceeding 1.0 (eigenvalue of 2.78), which accounted for 69.5% of the variance in the data. Observed component loadings were 0.44 for entry access, 0.47 for timeliness of care, 0.51 for confidence and trust and 0.57 for overall experience. This component can be interpreted as overall satisfaction with out-of-hours care. A rotation was unnecessary, as a simple structure was obtained.
Informed by the PCA, we investigated the construction of an overall satisfaction scale using all four items. This scale was derived by summing the standardised items (to account for differing response scales) if responses were given to all items. The scale had acceptable internal consistency (α = 0.772). Excluding the entry access item suggested a very minor improvement in alpha (α = 0.777; see Appendix 6, Table 82).
How well do the modified GP Patient Survey items summarise the Out-of-hours Patient Questionnaire?
The overall satisfaction scale was highly correlated with the factor scores of both OPQ domains for entry access (r = 0.63, p < 0.001, r2 = 0.397) and consultation satisfaction (r = 0.66, p < 0.001, r2 = 0.440). These correlations were both stronger than the correlation reported between the two OPQ domains. When combined into a scale, the four modified GP Patient Survey items explained 39.7% of the variation in entry access factor scores and 44.0% of the variation in consultation satisfaction factor scores, summarising both scales moderately well. Table 60 reveals that the entry access domain of the OPQ was most related to service users’ experience of the call operator, for which there is no equivalent GP Patient Survey item, perhaps explaining the lower correlation between the overall satisfaction scale and the entry access factor scores.
Concurrent validity
Multiple imputation of missing data allowed for inclusion of all 1396 respondents in the four mixed-effects multilevel linear regressions. A divergent pattern of associations across the covariates was evident between the models for each of the four GP Patient Survey outcomes (Table 61).
| Covariate | Entry access | Timeliness of care | Confidence and trust | Overall experience | ||||
|---|---|---|---|---|---|---|---|---|
| Coefficient (95% CI) | p-value | Coefficient (95% CI) | p-value | Coefficient (95% CI) | p-value | Coefficient (95% CI) | p-value | |
| Call answer time | 0.13 (0.06 to 0.21) | 0.001 | 0.09 (0.03 to 0.15) | 0.006 | 0.00 (–0.06 to 0.05) | 0.945 | 0.01 (–0.05 to 0.07) | 0.808 |
| Helpfulness of operator | 0.14 (0.04 to 0.24) | 0.008 | 0.06 (–0.03 to 0.15) | 0.204 | 0.04 (–0.04 to 0.12) | 0.345 | 0.12 (0.04 to 0.20) | 0.003 |
| How operator listened | 0.15 (0.05 to 0.25) | 0.003 | 0.05 (–0.04 to 0.14) | 0.268 | 0.00 (–0.08 to 0.09) | 0.954 | 0.07 (–0.01 to 0.15) | 0.068 |
| Health professional call-back timea | 0.09 (0.03 to 0.16) | 0.007 | 0.45 (0.39 to 0.52) | < 0.001 | 0.05 (–0.02 to 0.11) | 0.140 | 0.13 (0.08 to 0.19) | < 0.001 |
| Very poor/poor | Reference group | Reference group | Reference group | Reference group | ||||
| Acceptable | 0.16 (–0.02 to 0.34) | 0.089 | 0.70 (0.54 to 0.86) | < 0.001 | 0.07 (–0.09 to 0.23) | 0.376 | 0.38 (0.24 to 0.52) | < 0.001 |
| Good | 0.34 (0.15 to 0.53) | 0.001 | 1.05 (0.87 to 1.22) | < 0.001 | 0.18 (0.02 to 0.35) | 0.030 | 0.51 (0.37 to 0.66) | < 0.001 |
| Excellent | 0.35 (0.14 to 0.56) | 0.001 | 1.41 (1.22 to 1.60) | < 0.001 | 0.10 (–0.08 to 0.29) | 0.271 | 0.48 (0.31 to 0.64) | < 0.001 |
| Not applicable | 0.29 (0.07 to 0.52) | 0.011 | 0.98 (0.79 to 1.17) | < 0.001 | –0.04 (–0.23 to 0.15) | 0.706 | 0.35 (0.17 to 0.53) | < 0.001 |
| Happy with treatment option | ||||||||
| Yes | Reference group | Reference group | Reference group | Reference group | ||||
| No | –0.21 (–0.39 to –0.02) | 0.030 | –0.32 (–0.49 to –0.15) | < 0.001 | –0.58 (–0.73 to –0.44) | < 0.001 | –0.70 (–0.83 to –0.56) | < 0.001 |
| Consultation satisfaction | 0.05 (–0.01 to 0.12) | 0.105 | 0.06 (0.01 to 0.12) | 0.025 | 0.56 (0.51 to 0.61) | < 0.001 | 0.43 (0.38 to 0.47) | < 0.001 |
Discussion
This study sought to determine the construct and concurrent validity of four items from the GP Patient Survey evaluating service users’ experience of out-of-hours care through comparisons with an established, valid and reliable measure, the OPQ. 8,282 Preliminary work highlighted the need to make minor modifications to three of the four GP Patient Survey items to improve comprehension by service users’ and response options. The modified GP Patient Survey item set (entry access, timeliness of care, confidence and trust and overall experience) formed a single scale, which summarised the two-domain structure of the OPQ moderately well. Therefore, given minor modifications, these findings indicate that the GP Patient Survey item set evaluating out-of-hours care has potentially acceptable construct validity as a scale of overall satisfaction.
Each of the four outcomes was strongly associated with a distinct set of thematically related items from the OPQ, demonstrating their concurrent validity. Evaluations of entry access were related to ratings of the length of time before service users’ calls were answered, the helpfulness of the call operator and the extent to which the operator listened, which is supported by these items loading onto the same construct in PCAs in the present study and elsewhere. 8,130 Similarly, evaluations of timeliness of care were significantly associated with the time taken for the call to be answered, but were not related to ratings of the helpfulness of the call operator. Instead, timeliness was most strongly associated with the length of time taken for a call back from a health professional, an association also observed in a recent study of patient satisfaction with out-of-hours care from the Netherlands. 291
Croker et al. 148 found that patients’ confidence and trust in a health professional with whom they consulted in an in-hours primary care setting was highly influenced by interpersonal aspects of the care delivered as reported by patients. Important characteristics included having been given enough time, having felt listened to, having been given explanations about tests and treatments, having been treated with care and concern and having been taken seriously. In the present study, analogous items from the OPQ, combined into the consultation satisfaction scale, were strongly associated with service users’ ratings of confidence and trust in the out-of-hours health professional they consulted with. Confidence and trust were not related to items evaluating entry access.
Respondents’ ratings of their overall experience were strongly related to items from all three included sections of the OPQ: entry access, the result of the user’s call and the consultation with a health professional. The consultation satisfaction scale included an item rating the length of the consultation, which has also been shown to be a factor related to confidence and trust. 297 Patients’ evaluations of their overall experience of in-hours primary care have been shown to be most associated with doctor communication and the helpfulness of receptionists. 102 In the present study, service users’ ratings of their overall experience (the item unmodified from the GP Patient Survey) were strongly associated with their ratings of consultation satisfaction, which included elements of doctor communication as well as the helpfulness of the call operator.
Strengths and limitations
A strength of this study is the large number of service users included, which facilitated reliable statistical analyses using a large number of variables. When using factor analysis, best practice is to have five to 10 participants per measure,295 with a higher participant-to-measure ratio yielding more reliable results; upwards of 64 participants per measure were used in these analyses.
The overall response rate was low and responders tended to be older and living in less deprived areas; the final respondent sample also had a higher proportion of males than the non-respondent sample. This threat to the representativeness of the study sample is unlikely to have affected the analyses reported here. Specifically, this analysis aimed to determine the structure of users’ experience items and associations between them, rather than providing incidence/prevalence rates of conditions or similar outcomes that might be more affected by response bias issues. The methods employed controlled for these factors when possible and the findings are corroborated by the existing literature, as discussed above.
Minor modifications to either the word stems or response categories for three of the four GP Patient Survey items were made after careful piloting with service users that included the use of cognitive testing. Furthermore, the GP Patient Survey asks questions to respondents about making contact with a GP out-of-hours service in the past 6 months, whereas this study’s respondents were asked to answer questions relating to the last time that they made contact with a GP out-of-hours service, having been sampled from out-of-hours providers’ databases within 2 weeks of having made contact. Although this may limit the degree to which these findings apply to the existing GP Patient Survey items, this piloting was essential as early feedback from service users identified problems interpreting the items and changes to two items were designed to minimise missing data through blank responses (e.g. missing response categories). Implications for practice based on these findings are therefore contingent on the adjustment of current GP Patient Survey items.
Workstream 3: exploring how out-of-hours services use patient feedback
Study aims and objectives
This study aimed to identify how out-of-hours GP providers routinely collect patient experience feedback (including GP Patient Survey data) to inform their practice, with a particular focus on how it can be used to inform service reconfiguration and improve patient experiences of out-of-hours care. This was achieved by undertaking qualitative interviews with staff from out-of-hours service providers.
Methods
Sampling and data collection
The aim was to recruit an additional six out-of-hours providers as six (n = 12 total) were already recruited and had taken part in the survey study (see report on the conduct of workstream 2). Provider and staff recruitment ceased when data saturation was achieved. To achieve diversity of high-, medium- and low-scoring services, providers were first sampled on the basis of their scores on the GP Patient Survey item for care received from the service (question 40, April to September 2010 national GP Patient Survey data set). Once categorised into these groupings, information on organisation type and geographical location was considered. The final sample of providers ensured diversity across these three domains (GP Patient Survey score, organisation type and location), although no comparison of different subgroups of providers was planned. Up to three potential interviewees who had some involvement in conducting patient experience surveys were identified and approached to be interviewed at each provider. Participants were provided with an information pack consisting of a covering letter and participant information sheet. A mutually convenient time was organised to conduct the interview.
A week before the interview participants were sent a copy of a ‘feedback report’ containing patient ratings of their provider organisation based on the July 2012–March 2013 wave of the GP Patient Survey. Benchmarking data (generated by matching general practice postcodes to provider localities) were produced to allow providers to compare their performance with that of the 91 other English out-of-hours services for whom scores were able to be generated. Reports for the six services that had participated in the survey study (workstream 2) also included a summary of the their ratings derived from the research survey.
Face-to-face interviews, conducted at the participants’ workplace, took place between April and July 2014, each lasting between 39 and 88 minutes (mean 59 minutes). Topic guides were developed from a literature review, discussion between researchers and providers and previous findings with comments provided by the study advisory group. The topic guide included questions on how providers collected patient experience data and how this was used to make service changes; on awareness and views of the GP Patient Survey and out-of-hours items within it; and on the use of GP Patient Survey benchmarking provided in the feedback report.
Analysis
Interviews were digitally recorded and transcribed verbatim and transcripts were checked against the original recording for accuracy. Transcripts were coded in NVivo 10 software and analysis was independently coded using an iterative approach by one researcher (HB). A sample of five transcripts was independently analysed by a second coder (AA) to ensure that agreement was reached on the coding frame and codes. A deductive, framework approach with preliminary codes reflecting the content of the topic guide was used to construct the coding framework. However, a more inductive approach with additional thematic coding was undertaken using the ‘constant comparison’ method297 to capture new themes emerging from the data set. The initial coding frame was discussed within the team and when possible the codes were tested through seeking negative cases and/or divergent data. The data were then reorganised and collapsed into overarching themes. This process took place on two occasions until the main categories were agreed. All participants were sent a summary of the findings with a structured feedback form inviting comments on the veracity of the interpretation of the study findings. Final themes were reviewed and agreed between the research team to enhance reliability.
Results
Study participants
Five of the six providers approached took part (in addition to the six who participated in the survey study). A total of 31 staff from the 11 providers (NHS organisations, n = 2; social enterprises, n = 4; commercial organisations, n = 5) were interviewed, at which point data saturation was judged to have been achieved. Most participants were female (n = 23); 18 were service managers, seven were clinicians (GPs) and six were administrators. Participants who completed the feedback form (n = 2) on the findings were satisfied with the accuracy of the summary. Three main themes emerged: using surveys as a method of obtaining patient feedback; the utility of patient feedback; and the value of benchmarking.
Surveys as the most common method of obtaining patient feedback
Most participants focused on survey methods for collecting patient feedback, as 10 of the 11 providers undertook regular surveys to audit their patients’ experiences. Participants also discussed the ambiguities of operationalising NQR5, the desire for qualitative feedback to supplement survey data and the role of alternative methods in addition to surveys. It was evident from discussions that each provider interpreted the sampling for NQR5 differently, for example the range of patients being routinely audited varied from 1% to 20%:
We send out approximately 250 a week. Our National Quality Requirements require us to survey 1% – we actually do considerably more than that because we have taken our own interpretation on it.
11_4001, manager
Audits were undertaken on either a weekly or a monthly basis, using survey instruments developed by the organisation. Some participants reported that weekly audits were useful in terms of maximising patient response rates:
[T]hey’ve [out-of-hours service] worked out that the sooner the patient gets the questionnaire the more likely it is that they will complete it because it’s still fresh in their minds, so they try to do it as quickly as possible.
14_4003, GP
Most participants placed great importance on qualitative feedback from free-text comments provided by patients, which helped to interpret the quantitative findings, identify actions and provide a more personalised response from patients:
If they have got a real issue they can put it down, can’t they? Just doing the survey itself is just a way you test the water . . . The free text allows someone who has got a very bad experience the opportunity to write to us.
10_4001, manager
I’m dealing with people, I’m not dealing with robots. I mean, it’s their experiences, their feelings and they need to have a place to feed that back . . . they absolutely need to have a place to express their opinions – that’s giving people a voice.
14_4003, GP
Although it was agreed by all but one of the participating providers that patient surveys were a necessity, this was not a sufficient resource to drive change within services. A wide variety of alternative methods used by providers were reported, such as comment cards, ‘complaint and compliment systems’ and new technologies:
At the moment we’re thinking of going more electronically, so as soon as you have your consultation in the base, you come out and there’s a tablet so you can actually do your surveys straight after . . . that way you can get more accurate feedback of how people are feeling.
19_4002, administrator
Utility of patient feedback
Many participants cited examples of ways in which patients’ reported experiences had been used to make changes to service provision, although most changes tended to be ‘low level’. Because of the lack of observed trends within the data, most participants reported that patient survey data were insufficient to instigate service-wide changes:
In the main the results are stable and pretty good, but there’s not enough that’s consistent that I think we could use around wholesale service change.
12_4003, manager
Participants reported that patients’ expectations of the out-of-hours service were often unrealistic and difficult to manage and this made patient feedback difficult to deal with:
You often get patients who are very unhappy about the service they got and when you drill down into it it’s because they didn’t get antibiotics for their cold. Its expectations.
16_4003, GP
The changing landscape of the urgent care system was also confusing to patients. Some staff participants questioned the validity of patient experience data as the patients might be unaware of the different elements of the care pathway. Another barrier identified was the low-level engagement by commissioners. Despite the fact that patient experience audits are part of NQR5, many participants reported that commissioners treated them as a ‘tick-box’ exercise:
They [the commissioners] don’t come across to me as particularly engaged in this at all, and never really ask us too many questions around it.
18_4003, manager
Although acknowledging the identified barriers, some participants discussed how engaging with patient feedback had subtly changed the culture within their organisation and highlighted the importance of transparency and being responsive to change. In addition, participants reported the benefits of being able to compare patient feedback with other areas of reporting within the NQRs.
Value of benchmarking
Most participants acknowledged the benefits of having access to benchmarking data and felt that these data were a facilitator to enabling change. Notwithstanding this, many staff interviewees placed greater importance on their own surveys over the GP Patient Survey data, largely because their own surveys were more detailed.
Some staff expressed concerns about the reluctance of some providers to share with and learn from other providers, an issue mainly arising from commercialisation taking place within the NHS:
It’s terrible isn’t it, when everybody’s competing and not collaborating? That’s the system we’re living with, we’ve had to get used to it.
18_4001, GP
The benchmarking provided using the GP Patient Survey out-of-hours patient ratings was seen as useful, although many identified weaknesses with set items as they felt that the questions did not reflect the current urgent care system and lacked detail:
It is [GP Patient Survey out-of-hours evaluative items] just four questions, you get asked in McDonalds. It’s not detail is it?
10_4001, manager
Discussion
In the UK out-of-hours primary care providers are mandated to regularly audit patients’ experiences as part of the NQRs and services routinely meet this requirement by conducting patient surveys as well as by obtaining feedback using a variety of other methods. However, NQR5 is ambiguous and the resultant data cannot be used to compare services as providers are undertaking audits of varying scale, frequency and methodology. Staff reported a strong preference for qualitative patient feedback, which is echoed in other settings, as it yields richer, more detailed feedback than quantitative survey scores. For example, hospital staff have found that qualitative data from patients added a more patient-centred aspect to patient satisfaction measurements. 210,298 Research has shown that health-care leaders place great importance on complaints, comments and compliments as sources of patient feedback,299 as do general practice staff (see Chapter 7).
Patient feedback appeared to have a limited role as a driver for service change and effective change was hindered by modifications taking place in the urgent care landscape, which confused patients with regard to how care was organised. Some staff also reported that commissioners appeared uninterested in patient experience audit findings. In some settings audit and feedback have been shown to have small to moderate effects on health-care professionals’ practice,204,300 although in other settings it can have a wider impact. 301 For change to occur, the organisational culture must be supportive of change and be patient focused. 206,209,302 Most of the changes reported by staff were ‘low level’ and unlikely to drive system-wide reconfiguration because of the lack of consistent patterns observed in the data. There was a preference for qualitative feedback as patient free-text comments could potentially identify specific areas of actionable change or contribute to wider data-gathering audits, for example critical incident techniques. 303 However, to be useful, patients’ attention must be focused to provide qualitative feedback on the out-of-hours service.
Staff valued the GP Patient Survey patient experience benchmarking data and the GP Patient Survey presents an opportunity for benchmarking of all out-of-hours services. NHS England has recently recommended that NHS commissioners use the GP Patient Survey results to monitor patient experiences of out-of-hours providers278 and the CQC has published GP Patient Survey provider performance at commissioner level. 277 Despite the strengths of the GP Patient Survey (regularly and independently collected data that is publicly available), participants were reluctant to use GP Patient Survey data in its present form because of concerns about the face validity of out-of-hours items and the absence of free-text comments, a limitation found in previous studies. 119,199 In addition, the current out-of-hours items are not reflective of the recent changes that have taken place within the urgent care system (e.g. introduction of the NHS ‘111’ telephone portal). Most staff did not believe that the limited number of GP Patient Survey items would drive change by themselves.
Strengths and limitations
This is the first qualitative study to explore the views of out-of-hours staff who have an in-depth knowledge of patient feedback processes within their organisation. Sampling ensured that staff from a variety of different types of provider (e.g. not-for-profit or commercial enterprises), serving diverse populations across England, were included. Although sampling diversity was achieved, it is acknowledged that participating organisations may be more interested in the patient experience agenda than non-participants and thus findings may not reflect the views of the wider population. The views of commissioners were not sought in this study and thus the widespread perception that some commissioners were apathetic towards patient feedback data must be interpreted cautiously. Because of logistical constraints it was not possible to interview commissioners and obtain their perspective on the perceived role and value of patient feedback data.
Conclusions from the out-of-hours research
Implications for practice and future research
An analysis of national GP Patient Survey data (see Workstream 1) identified that commercial providers were associated with poorer patient experiences of out-of-hours GP care than NHS or not-for-profit providers. It is not possible to derive simple explanations regarding the drivers of these lower ratings in this observational data set and further research is required to understand what drives these differences. Although some insight might be gained from an understanding of patient differences (e.g. nature or urgency of requests for care) at the level of the provider, such data are not routinely collected in the GP Patient Survey for out-of-hours service evaluations. It is unknown whether or not factors such as user awareness of the provider type may also be of importance in interpreting service users’ ratings.
Further research, possibly involving qualitative approaches or a vignette study, is required to investigate the reasons for the generally lower scores from service users from minority ethnic backgrounds (see Chapter 6 for vignette work conducted as part of the wider IMPROVE programme). Similarly, research investigating the reasons why service users who were unable to take time off from work to attend their practice during regular hours reported poorer scores across all three evaluative questions is needed. Finally, as for in-hours GP care,131 investigation of the extent to which variations between sociodemographic groups in respect of care ratings might be attributable to the clustering of servicer users belonging to sociodemographic groups reporting relatively lower scores within providers with lower overall scores is required. This analysis would help inform the development and targeting of interventions aimed at improving service users’ experiences of out-of-hours GP care for specific population subgroups.
National standards (NQR5) require out-of-hours providers to routinely audit patient experiences, although no specific survey tools or methods are recommended to achieve compliance. In the absence of data collected directly by providers, both the National Audit Office and the CQC have recently used the GP Patient Survey as an alternative data source to monitor patient experiences of GP out-of-hours care. However, an important prerequisite to using GP Patient Survey data to benchmark services is that its psychometric properties are established. The reliability of GP Patient Survey out-of-hours items have been previously reported,141 but there was no evidence regarding their validity. The second workstream demonstrated that, although our survey was composed of only four of the GP Patient Survey evaluative items (after minor but essential modifications identified through cognitive testing and piloting), the GP Patient Survey out-of-hours items that we used had both construct and concurrent validity. These findings provide support for the use of the GP Patient Survey for national benchmarking purposes.
Whereas workstreams 1 and 2 examined the technical performance of the GP Patient Survey out-of-hours items, the third workstream examined how out-of-hours staff use patient feedback and their views on the utility of GP Patient Survey items. This qualitative study found that, although NQRs are intended to promote transparency and allow comparisons between out-of-hours providers, NQR5 was ambiguous and in its current form does not support benchmarking or service improvement. A critical review of the NQRs is required to help providers to engage with patient feedback and drive service improvement effectively.
In the absence of clear NQR guidance, providers were inventive in the ways in which they engage with patients. Qualitative feedback was highly valued as it provided detailed information that could lead to actionable changes. However, services struggled to find ways to use patient feedback to drive anything other than low-level service change. Future research should explore how out-of-hours services managing patients with urgent care needs, and particularly those delivering services to diverse populations, can be assisted in engaging more fully with patient feedback. Evidence is also needed on whether or not comprehensive guidance on how to collect, interpret and act on patient feedback has the potential to drive quality improvement initiatives. 45,206,302
In the context of the rapidly changing landscape of UK urgent care services, although participating providers could see the potential of using the GP Patient Survey for benchmarking purposes, its out-of-hours items need urgent revision as they do not reflect current telephone access arrangements (NHS 111) for out-of-hours care. This qualitative finding supports our preliminary survey piloting work and cognitive interviews with service users (see Workstream 2). Minor but essential amendments to the GP Patient Survey out-of-hours items are required to improve the comprehension of items and improve data quality.
Patient feedback currently has a limited role in driving changes to out-of-hours service provision and the utility of feedback may be hindered, in part, by recent modifications to the urgent care system and the ambiguity of NQR5 in relation to gathering and acting on patient feedback. English GP Patient Survey data may be used to benchmark and compare service providers. However, the out-of-hours items need to be updated to reflect the changes made to accessing out-of-hours services by telephone, so that providers can be confident that ratings reflect their services’ performance. A greater understanding of how variations in patient and provider characteristics drive variations in patient experiences of out-of-hours care is needed to support the development and targeting of quality improvement initiatives.
Chapter 12 Conclusions, implications for practice and recommendations for future research
Conclusions
In Chapter 1 we outlined how, following the introduction of a wide range of quality improvement strategies as part of an overarching ‘clinical governance’ strategy in the late 1990s, there had been step changes in the management of major chronic diseases in the NHS. However, the ways in which patients experienced health care had not been given such a priority and the need for a rebalancing was seen by increasing attention to patient experience in policy documents, the routine publication of patient experience data, benchmarking of hospitals in relation to patient experience and even an (ill-fated) attempt to attach payments to patients’ assessments of their GP’s care.
There has therefore been widespread acceptance that good patient experience is an important outcome of care in its own right and our work304 and that of others305 has shown that patient experience is a domain of quality that is distinct from, but complementary to, the quality of clinical care. Although an increasing number of surveys have been developed to measure patient experience, there has been equally widespread acceptance that these measures have not been very effective at actually improving care. 45 This is the background to our programme of work. Entitled IMPROVE, we aimed to find better ways of both measuring and using information on patient experience that would lead to improvements in patient care in both in-hours and out-of-hours primary care settings.
In the introduction, we described a range of ways of obtaining patient feedback on their care, including surveys, focus groups and analysis of complaints. In this programme, we have focused on the use of patient surveys as they are the dominant method currently used in the UK. However, in Chapter 10 we describe an exploratory trial of RTF, which moves away from the paper-based questionnaires that still dominate the measurement of patient experience in the NHS.
This programme had seven aims, each of which was tied closely to one work package of research. These aims were to:
-
understand how general practices respond to low patient survey scores, testing a range of approaches that could be used to improve patients’ experience of care
-
estimate the extent to which aggregation of scores to practice level in the national study masks differences between individual doctors
-
investigate how patients’ ratings on questions in the GP Patient Survey relate to actual behaviour by GPs in consultations
-
understand better patients’ responses to questions on communication and seeing a doctor of their choice
-
understand the reasons why minority ethnic groups, especially South Asian populations, give lower scores on patient surveys than the white British population
-
carry out an exploratory RCT of an intervention to improve patient experience, using tools developed in earlier parts of the programme
-
investigate how the results of the GP Patient Survey can be used to improve patients’ experience of out-of-hours care.
The aims of the programme did not change during the 5 years of our research, although some details of the research were modified as the work progressed (we have summarised any changes in each individual chapter). We presented the results of our research under three broad headings and also use these headings in this conclusions chapter, namely:
-
understanding patient experience data (aims 3 and 4)
-
understanding patient experience in minority ethnic groups (aim 5)
-
using data on patient experience for quality improvement (aims 1, 2, 6 and 7).
Understanding patient experience data
Patient surveys are now widely used in many countries, yet still comparatively little is known about what experiences lead patients to respond in particular ways in these surveys. What drives them to tick particular boxes and how do those responses relate to the care that they have actually received? We approached this in two main studies, one in which we asked patients directly about how they chose certain items on the questionnaire while showing them a video of their consultation (see Chapter 2) and one in which we compared their responses with those of expert raters using two standard instruments for assessing videos of consultations (see Chapter 3). The results of these studies have important implications for the interpretation of survey data, particularly data focused on patient evaluations of specific encounters with health-care professionals.
The first study (see Chapter 2) showed that, although patients readily criticised their care when reviewing GP consultations on video, they had been reluctant to be critical when completing a questionnaire after the consultation. Reasons for this included the need to maintain a relationship with the GP (including uncertainty about how confidential survey results would be) and their gratitude for the care that they had received from the NHS in the past. In addition, perceived power asymmetries made people reluctant to criticise their doctor. Patients were also disinclined to be critical when completing a questionnaire if they had actually received the treatment that they wanted. Overall, we concluded that patients find that questionnaires administered at the point of care may be limited tools for being able to feed back concerns about primary care consultations.
The second study (see Chapter 3) reinforced our conclusion from Chapter 2 that patient evaluations of consultations in surveys may present an uncritical view of the actual consultations. In this study videotapes of GP–patient consultations were assessed by four independent clinical raters. The results were striking. When trained raters rated communication within a consultation to be of a high standard, patients did the same (with one single exception). However, when trained raters judged the communication during a consultation to be of a poor standard, patients’ assessments varied from poor to very good. This finding again points to the reluctance of patients to criticise their doctor in questionnaire surveys. In the previous study the ‘gold standard’ was the patient’s own account of the consultation and in this study the standard was that of a trained external GP rater.
We do not think that these results mean that patient surveys cannot be used to assess the quality of general practice care. However, they do point to clear limitations. One of the concerns that GPs have about surveys (see Chapters 7 and 8) is that they are selectively completed by critical or grumpy patients and that survey results will therefore give a negative and biased view of their care. The results of the two studies described here suggest that the opposite is the case. Patients’ reluctance to criticise their doctor means that survey responses using evaluative type of questions are likely to give an overly positive view of their doctor’s care. This is one reason why there has been a move towards using report items in some survey instruments (though we do not know whether or not these suffer from similar problems). Because of this tendency for patients to choose the most positive response options, we suggest that absolute scores should be treated with some caution, as they may present an overly optimistic view of their care. However, this does not mean that surveys cannot be used to look at relative scores: scores from a GP that are lower than those of his or her colleagues and from GPs in other practices are likely to indicate a problem, even though high scores from other doctors or practices may conceal deficiencies in care in those practices too.
We also looked at how GPs rated their own consultations. GPs completed a form immediately after each consultation, using the same scale as the patients. GPs were certainly more inclined to criticise themselves than the patients were to criticise the care that they had received. This is entirely consistent with the findings from our subsequent interviews with patients. However, we found absolutely no correlation between patient scores and GP scores. Neither did we find any correlation between GPs’ own scores and those of expert raters who reviewed the consultations on video. GPs are clearly using different parameters when assessing their own performance, but we were not able to investigate this in more detail in this study.
When we spoke to GPs about their survey results (see Chapters 7 and 8), through both focus groups and face-to-face interviews, they reported how, although positive about the concept of patient feedback, they struggled to engage with and make changes under the current approaches to measurement. They also commonly expressed concern that patients would be critical of their care if they did not get what they wanted (e.g. an antibiotic prescription). This concern was borne out to some extent by our results. In our analysis of the assessment of nurses (see Chapter 4), a strong predictor of survey scores was whether or not patients wanted to see a nurse when they first contacted the practice. If they had wanted to see a GP but saw a nurse, the scores given to those nurses were much lower. We have no reason to think that the nurses’ communication was worse in those consultations and the low scores may therefore indicate a more general dissatisfaction of patients because of not having their original expectations met.
It is important to understand that, in line with the overall aims of the programme, the work in these two chapters focused on the assessment of communication in the primary care consultation (such as giving the patient enough time and explaining tests and treatments). Our conclusion that survey scores have more value in assessing relative performance than absolute performance of doctors may or may not hold true for other aspects of practice performance commonly assessed in surveys, such as difficulty in getting appointments, getting through on the telephone and waiting times. Patients’ reasons for not wanting to criticise their doctor may be less important when they assess what they regard as management aspects of the practice.
A second aspect of care that we identified as part of our programme of work relates to patients’ ability to see a GP of their choice. Although most of our research focused on communication, the results that we report in Chapter 4 have some important findings in relation to patient choice. 142 The results show that most patients have a particular GP who they prefer to see. It is sometimes suggested that this matters only for some population groups (e.g. not young people) but we found that this is not the case. Even among those aged 18–24 years, > 50% of respondents to the GP Patient Survey have a particular doctor who they prefer to see, rising to > 80% in those aged > 75 years. Disturbingly, a large percentage of people who have such a preference are unable to see the doctor of their choice. This percentage has risen from 30% to 40% from 2010 to 2015. One possible impact of this change comes from our analysis of data from patients who saw a nurse when they had originally wanted to see a doctor; they expressed considerable dissatisfaction with their subsequent consultation with the nurse. However, these data do not reflect what would have happened if patients had seen another doctor, just not the one of their choice.
Overall, patients express more negative opinions about choice of doctor than in any other part of the GP Patient Survey, something that may in part have got worse as a result of government policies to improve access. There is a clear tension between the ability of practices to provide rapid access and the ability of practices to provide continuity of care and data from our studies suggest that patients’ inability to see a doctor of their choice is a significant quality issue for the NHS.
Understanding patient experience in minority ethnic groups
In this part of our research, we focused our main work on survey responses from minority ethnic groups and on South Asian groups in particular. The general interest in minority ethnic groups is because they tend to report worse experiences using surveys in most countries studied, including in the UK. Our research on out-of-hours care in this programme (see Chapter 11) replicated this result, with Asian and mixed ethnic groups reporting worse experiences than the white majority.
Our specific focus in the major strand of this research was on South Asian respondents because of the size of this group in England and the consistently low scores generated by this group in English surveys across both primary and secondary care settings. We focused on questionnaires competed in English; although the GP Patient Survey is available in 15 languages, a tiny minority of surveys are completed in languages other than English (typically < 0.2% of returns).
A number of potential explanations have been suggested for the lower ratings given by South Asian and other minority ethnic groups. Broadly, these relate to whether these groups of patients (1) receive lower-quality care or (2) receive the same care but rate this more negatively. 75 For example, such respondents might rate the same care more negatively if they have higher expectations or because they interpret the survey items and response options in different ways (such as being culturally less likely to check extreme options).
The last of these options was potentially the simplest to explore. Taking advantage of the large numbers of respondents available in the GP Patient Survey to examine the responses of South Asian groups using item response theory and allowing for a wide range of other sociodemographic characteristics (see Chapter 5, Workstream 3), we found no evidence that South Asian respondents used the scales in a different way from white British respondents. Although these results do not provide conclusive evidence of equivalence in the way in which different respondents use the survey scales, they increase the likelihood that the worse experience reported by South Asian respondents reflects either differences in expectations or genuinely worse care. Our previous work75 suggested that, for one aspect of care (waiting times), South Asian respondents might have higher expectations of care, implying that their lower scores on surveys might not be associated with worse care. We were able to advance our understanding of this complex issue considerably as a result of the research in this programme.
First, we showed that South Asian respondents to the GP Patient Survey tend to be registered in practices with generally low scores. This explained about half of the difference in reported experience between South Asian and white British patients (see Chapter 5, Workstream 1) and identified that some practice effects were related to the ethnicity of the doctor (with minority ethnic doctors receiving lower scores for doctor–patient communication; see Chapter 5, Workstream 4). However, these practice effects did not account for the low scores among South Asian patients, even though the differences were reduced when practices offered consultations in a South Asian language172 (PhD project allied to our programme). Next, we showed that, far from being uniform across all population groups, the lower scores from South Asian patients were much more marked among older female respondents. It was therefore important in our subsequent work to ensure that these patients were represented in our research (see Chapter 5, Workstream 2).
In video elicitation interviews with South Asian patients (see Chapter 2), we identified the same issues driving evaluations of communication in South Asian as in white British patients: their relationship with their GP (and others within the practice), their expectations of the consultation and a reluctance to criticise their doctor’s performance. The finding that South Asian patients are assessing broadly similar issues when completing questionnaires therefore still leaves unanswered the question of why scores from South Asian patients are low.
The final and most original part of this work provides insight into this (see Chapter 6). Here, we filmed 16 simulated consultations based on transcripts of real consultations using various combinations of white and Asian doctors and patients, with half scripted to be ‘good’ and half scripted to be ‘poor’. We showed three randomly sampled videos to each of 1120 people (half of whom were white British and half of whom were Pakistani, equally split between those aged < 55 years and those aged ≥ 55 years) and asked them to score the consultations using the communication items from the GP Patient Survey.
If the low scores reported by South Asian patients in real-life settings were the result of higher expectations on their part, then we would expect them to give lower scores in the experimental vignette situation. However, quite the reverse happened. When viewing the same consultations, South Asian respondents gave scores that were higher, indeed much higher when adjusted for sociodemographic characteristics, than those of the white British respondents. This suggests that the low scores given by South Asian patients in surveys such as the GP Patient Survey reflect care that is genuinely worse, and possibly much worse, than that experienced by their white British counterparts. This is consistent with the only previous study of this type in which predominantly written consultations were shown to people from different ethnic groups in the USA, with the conclusion being that differences in ratings were more likely to represent differences in care than differences in expectations or scale use. 81
There is a clear practice implication of this result: low scores from South Asian patients should be investigated as possible indicators of poor care. This is relevant to all settings, not just primary care.
Using data on patient experience for quality improvement
The results that we have discussed so far indicate that the results of patient experience surveys such as the GP Patient Survey can identify areas where there are important gaps in care that the NHS provides, such as patients being able to see a doctor of their choice. However, although patients tend to give very high scores for doctor–patient communication, these conceal significant negative experiences that patients describe when shown, and which independent observers can see in, recorded primary care consultations. These issues extend to minority ethnic patients and our research suggests that the negative scores that South Asian patients record (compared with those of white British patients) do represent genuine problems with care. This therefore brings us to the important issue of how data from patient surveys can be used to improve care.
Current national approaches to measuring patient experience, including communication, rely on practice-level assessments of care. In Chapter 9, we outline the results of a patient experience survey that we conducted across 25 general practices, asking patients specifically about their experience of a particular consultation with a named GP. We found that practice-level scores for communication mask considerable variation between GPs within each practice, notably for those practices receiving poorer communication scores overall. Such ‘poorly performing’ practices, which may be identified as such through the national GP Patient Survey, may in fact contain GPs with communication skills ranging from very poor to very good. This has important implications for the use of national survey data to identify primary care practices and practitioners in need of improvement.
In Chapters 7 and 8 we describe the two studies in which we sought the views of GPs and practice staff on survey results, seeking to understand how they could better be used as quality improvement tools. Chapter 7 describes focus groups with practice staff following feedback of practice-level scores for patient experience and Chapter 8 describes interviews with GPs after we had conducted a survey in which they received individual feedback from surveys returned by patients whom they had seen in the surgery. In Chapter 11, we describe how out-of-hours providers use data from patient surveys.
Broadly, staff in different primary care settings neither believed nor trusted patient surveys. Concerns were expressed about the validity and reliability of surveys (some practices have very low rates of response) and about the likely representativeness of those who responded. Some practice groups mentioned recent negative experiences with pay linked to survey scores as part of the QOF (a technicality of the payment schedule meant that payments could be reduced even though practice performance had improved). There was also a view expressed that some patients had unreasonable expectations: staff worked as hard as they could and could not be expected to respond to all patients’ ‘wants’. Some practices did describe improvements that they had made as a result of survey results. Those that were easiest to engage with related to practices’ office functions such as appointment systems and telephone answering systems. Addressing an individual doctor’s performance (e.g. communication skills) was much more difficult. Out-of-hours service staff were also concerned that service users did not understand the complex care pathways within urgent care settings and that this might lead to unrealistic expectations of what individual services were expected to deliver. Staff viewed surveys as necessary, but not sufficient. Clear preferences for more qualitative feedback to supplement survey scores were expressed as this provided more actionable data on which to mount quality improvement initiatives.
The doctors who we interviewed expressed markedly ambivalent views in discussing feedback from surveys. Although they had a number of concerns about individual doctor surveys (credibility, reliability, concerns about patient motivation), they also expressed positive views about the importance of patient feedback in monitoring and improving services.
These results led us to consider how patient feedback might be obtained in a way that would engage doctors more actively with patient survey results to stimulate quality improvement. We conducted a preliminary evaluation of RTF, using touch screens that patients could use to leave feedback following a primary care consultation. RTF was selected to address some of the problems identified by our research, such as providing practice feedback on a much more regular basis (e.g. fortnightly) and allowing practices the opportunity to add questions of their own to the RTF survey to increase the relevance of the results to their service.
As RTF has not been widely used, an exploratory RCT and qualitative study were conducted to answer questions about the feasibility of using RTF in real-world general practice, estimate likely response rates, obtain patient and staff views on providing feedback in this way and estimate the costs to a practice of introducing RTF. We also included facilitated feedback in one arm of the exploratory trial.
In the exploratory trial, only 2.5% of consulting patients left any RFT without prompting; however, if encouraged to leave RTF by staff, as many as 60% of patients did so. Encouragement was rare, with such encouragement provided in only 5% of > 1100 patient–staff interactions that we observed in reception areas. Of patients who used a touch screen to leave RTF, 86% found it easy to use and were positive about it as a feedback method. Lack of awareness of the screens and lack of time were the most common reasons given for not providing feedback.
Staff were broadly positive about using RTF and practices valued the ability to include their own questions in the survey. Practices that had open communication between staff members tended to be more positive about using patient feedback. Practice staff identified clear benefits from having a facilitated session for discussion of patient feedback and having protected time to discuss the results.
Had practices not been taking part in a research study, the cost of RTF to practices would have been substantial at > £1000 for the 12 weeks, with the bulk of the cost relating to provision of the equipment and analysis and feedback of the data collected from the touch screens.
Although the absolute number of patients providing RTF to each practice (> 100) was comparable to the number of respondents per practice in the national GP Patient Survey, we do now know how the considerably lower response rate in our RTF study (2.5%) would have affected the outcome of the patient experience surveys (it was not part of our study design to find this out). We do not know how representative or valuable the views of a small proportion of patients who respond are, just as we do not know how representative are the views of the very small numbers of patients providing the narrative feedback that is recorded on NHS Choices.
Considering these results together, we have been able to identify some clear learning to take forward into a future clinical trial examining the potential utility and effectiveness of RTF in informing service delivery in primary care.
Implications for practice
The work that we have carried out over the 5 years of the programme grant has clear implications for practice. We summarise these here.
The importance of patient experience
Our research supports the continuing emphasis on obtaining patient experience feedback as an important means of informing NHS care. Although continuing effort should be invested in refining the most effective and meaningful mechanism to capture high-quality patient feedback, the key challenge is to provide primary care staff with the support and means to enable them to act on patient feedback.
The need for action on the quality of care for minority ethnic groups
There has been much speculation whether the lower scores reported by minority ethnic groups on numerous patient experience surveys are ‘real’, reflecting poorer quality of care, or are an artefact of the questionnaires used or higher expectations of care. We have now conducted a series of studies to progressively examine this issue to understand with greater certainty the major drivers of reported variations in care. Examinations of survey responses, interviews with patients and an innovative experimental vignette study combine to strongly suggest that it is the former: patients from South Asian backgrounds experience considerably poorer communication with GPs than their white British counterparts. It is of concern that survey results may be dismissed as artefactual when, in fact, they are likely to point to real areas of concern. Effort should be invested to ensure that lower scores from such groups on patient experience surveys in both primary care and secondary care are investigated as markers of poorer quality of care.
Patients give overly positive responses when rating their care
Our results show the difficulty that patients have in feeding back negative experiences in questionnaire surveys. This suggests that there is more work to be done in improving patient experience than might be suggested by the high scores that are commonly seen in patient surveys. However, patients’ reluctance to criticise a doctor or provider with whom they have to maintain an ongoing relationship will not be addressed simply by changing the survey method. Efforts should be made to ensure that providers and managers understand that absolute scores paint an optimistic picture of patients’ true views.
Surveys are not sufficient to fully capture patient feedback
Across primary and out-of-hours care settings, staff view patient surveys as necessary, but not sufficient. Alternative methods for gaining more qualitative feedback were commonly used to supplement survey scores, with free text often viewed as providing more actionable data than responses to standard survey questions. Taken alongside our findings on patients’ reluctance to criticise doctors through surveys and staff challenges to the credibility of surveys, we suggest that additional approaches are therefore needed to better capture aspects of patient experience that can be used to improve the quality of care.
The need for valid, reliable individual-level feedback for doctors
Despite the comments above, we have shown that there is substantial variation in performance within practices for aspects of care related to individual doctors (e.g. doctor–patient communication). Reporting patient experience at practice level masks this variation and makes it more difficult for doctors to relate to feedback. However, we have also shown that, if a practice has overall high scores for doctor–patient communication, it is very unlikely that such a practice contains a low-scoring doctor. In contrast, when a practice is low scoring, individual doctors may be high or low scoring. Therefore, if there are additional requirements for individual-level surveys, they could be focused on practices with low overall scores. Additionally, robust mechanisms are required to help practices, particularly lower-scoring practices, identify and support individual doctors whose patient feedback identifies areas of potential improvement.
We note that, at present, data are provided at practice level for the GP Patient Survey, scores are produced at practice level for the Friends and Family Test and GPs have to provide individual-level surveys to meet GMC requirements for revalidation. This results in considerable overlap and duplication and adds to the sense that these are ‘boxes to be ticked’ rather than sources of information that are valuable for improving care.
Patient surveys need to become more meaningful to staff
Our research shows that primary care staff in different settings are ambivalent about the value of patient surveys. Although believing in general about the importance of issues such as doctor–patient communication, they use every opportunity to challenge the credibility and reliability of scores produced by national surveys. This is not helped by their recent experiences, for example of a poorly conceived attempt to tie financial incentives to patient reports of waiting times to get an appointment306 and the imposition of the Friends and Family Test, which is even regarded by NHS England as being of limited value for comparing health-care organisations. 60
On the whole, practices found it easier to engage with items on surveys that related to practice management (e.g. availability of appointments, ability to get through on the telephone) than to items that related to issues around communication between patients and clinical staff. Staff viewed surveys as necessary, but not sufficient, and expressed a clear preference for qualitative feedback to supplement survey scores as this provided more actionable data on which to mount quality improvement initiatives.
Immediacy of feedback, regularity of feedback and having some control over the questions asked were all aspects of our experiment with RTF that were valued by practices and had the potential to make feedback more useful. However, a number of important questions remain before RTF could be recommended as a replacement for postal questionnaires. We outline these in the next section on research recommendations.
The value of surveys in monitoring national trends
Despite some reservations about the value of national surveys as vehicles for stimulating quality improvement in general practices and out-of-hours services, they can be important for monitoring national trends. For example, the GP Patient Survey is the only source of data which demonstrates that, year on year, from 2010 to 2015, patients report that they have had increasing difficulty in seeing a doctor of their choice. Indeed, for out-of-hours services the GP Patient Survey is the only way to monitor such trends as individual services use very different tools and approaches, precluding comparisons. Additionally, patient feedback – particularly in secondary care – is used for organisational risk assessment and regulatory monitoring. However, when national surveys are used to monitor trends in care it is important that the questions stay the same. In contrast to questions in the GP Patient Survey related to whether or not patients are able to see a doctor of their choice, questions in the survey on access have undergone major changes, making it difficult to follow long-term trends. However, it should be noted that much smaller sample sizes are required to monitor national trends and comparable national surveys often include tens of thousands of participants rather than millions. Our work on out-of-hours care suggests some ways in which the current questions in the GP Patient Survey could be improved.
Development of surveys in out-of-hours care
Our work on the use of patient experience surveys in out-of-hours care highlights a number of areas requiring consideration. National quality requirements (NQR5) state that all out-of-hours services must audit patient experience but provide no information on how to do this. 307 In the absence of clear guidance on tools and approaches, many services are taking different tacks to both collect and act on patient feedback. As well as being inefficient in approach, with little consistency or shared learning, this also precludes national comparisons being made between providers. We suggest that NQR5 should be reviewed and tightened to avoid the duplication of effort occurring in different services.
Second, out-of-hours items from the GP Patient Survey are now being used for the purposes of CQC and National Audit Office monitoring of out-of-hours care. Our research in this area commenced prior to the launch of the CQC and providers knew little about the GP Patient Survey and expressed concern about the relevance of the out-of-hours items. Our research suggests that, subject to minor amendments, the GP Patient Survey is suitable for this kind of national monitoring of out-of-hours care; indeed, it is the only current approach suitable for monitoring, given the variation in approaches to patient feedback currently taken by service providers. However, although the GP Patient Survey enables the use of benchmarking, it is not sufficiently detailed to support quality improvement and as such is unlikely to replace the in-house methods and tools being used by providers. We also note that current presentations of GP Patient Survey data for out-of-hours care are at ‘commissioner’ level; as providers often cover more than one commissioner level, such analyses may not highlight problems occurring at the larger organisational level. Finally, to look at the performance of different out-of-hours providers on key patient experience measures, it is important that NHS England maintains a list of such providers to ensure oversight, which it currently does not.
Overall, large-scale postal surveys are likely to remain the dominant approach for gathering patient feedback for the time being, although refinements to this approach as well as the development of other modes are required to address the weaknesses that we have identified. We are aware that providers are experimenting with a wide range of other approaches, one of which (RTF) has been part of our research. Other methods include interviews and focus groups, online feedback, analysis of complaints, practice participation groups and social media. In the following section, we outline recommendations for research and identify the criteria that any new methods will need to meet to become useful quality improvement tools.
Recommendations for research
The world of patient feedback is becoming increasingly diverse and complex, with standard patient survey approaches being supplemented by the use of tablets, kiosks, online feedback, including that provided by the NHS and by commercial organisations, analysis of complaints, the use of interviews and focus groups and practice participation groups. In addition, social media may come to play an important part in how patients choose their doctor and how they feed back on their experiences. Some of these new approaches are being evaluated in terms of their ability to provide more detailed information on what is needed to improve services, for example using patient narratives308 and through the analysis of internet-based feedback. 309,310 However, despite the plethora of approaches to gathering patient feedback, our research demonstrates that there is a major deficit in taking action as a result of such feedback. Enabling and supporting providers to engage with and plan changes may require complex whole-system approaches, and our knowledge of what is most effective in this area is currently sparse.
Research is therefore needed into how gathering and acting on patient feedback may be best supported, across five key areas:
-
How patient experience can be captured so that it more effectively identifies areas of performance that could be improved – this should include investigation of diverse methods of obtaining patient feedback to support patients to highlight poor care when necessary. An additional important area of work is how some of the issues highlighted within this report, such as patients’ reluctance to criticise, apply to different approaches to assessing patient experience using either rating-type or report-type questionnaire items.
-
The system, practitioner and patient factors that influence poorer reported experiences of care in South Asian patient groups and how these may be addressed – this should include a particular focus on the impact of cross-cultural consultations.
-
How information from patients can be fed back to clinicians and services in a way that appears credible to them – this should include evaluations of approaches to increase the plausibility of patient surveys, such as greater use of benchmarking and innovative ways of presenting and interpreting findings, as well as assessment of varying, tailored ways of presenting feedback to the different health-care professionals who might receive feedback on their care. Of additional relevance here is how clinicians are encouraged to reflect on their own performance and others’ assessments of this, with the aim of understanding where and how gaps in evaluations may occur.
-
How services can be organised and managed in such a way that patient feedback is seen as a positive opportunity for improving services.
-
What interventions are most effective in improving care when deficiencies in care are identified – the area where there is the greatest gap here is in doctor–patient communication, with our results showing that clinicians have great difficulty in even discussing deficiencies among their colleagues and that few effective interventions exist.
Our finding in the research on out-of-hours care that commercial providers had lower ratings for patient experience than services provided by the NHS is consistent with previous work suggesting that practices working under Alternative Provider Medical Services contracts, which are sometimes provided by the private sector, may provide worse care. 311 However, the circumstances in which commercial providers gain contracts for primary care services may be very different from those in other areas. The way in which the primary care workforce is configured is changing rapidly, with an increase in the proportion of salaried GPs, the development of GP federations and super-practices and an increase in the number of large-scale provider groups (owned both by commercial companies and by GPs). It is important that these changes should be monitored so that we understand their impact on quality of care.
Acknowledgements
We would like to express our particular thanks to the lay members of the IMPROVE advisory group for their commitment over the 5 years of the programme and for sharing their experiences and views and providing guidance throughout this time. Thanks, too, to Dr Sahadev Swain for his contribution to the group and, in particular, his expertise on the issues facing patients from minority ethnic backgrounds.
Contributions of authors
The report authors’ contributions are listed for each workstream in the tables below, alongside the contributions from the many others who made this programme of work possible. Chapters 1 and 12 were drafted by John Campbell and Martin Roland, with additional input from Jenni Burt. The practice recommendations outlined by Martin Roland, John Campbell and Jenni Burt were discussed at a full IMPROVE team meeting, with particular contributions from Georgios Lyratzopoulos, Gary Abel, Suzanne Richards, Marc N Elliott, Charlotte Paddison and Jenny Newbould. Julia Beckwith conducted a review of the literature that supported Chapter 1. Details of the overall contributions of the authors are listed in Table 62.
| Name and role | Chapters contributed to | Nature of contribution |
|---|---|---|
| Contributions of authors | ||
| Jenni Burt (Senior Research Associate, University of Cambridge) | All | Programme manager: contributed to the design and oversaw the conduct of the programme, reviewing of the literature and analysis and interpretation of the data and writing of individual project reports. Drafted and edited the final report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | All | Co-chief investigator: contributed to the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| Gary Abel (Senior Research Associate, University of Cambridge) | Chapters 2–6 and 9 | Lead statistician: contributed to the design of the study, conceived and conducted data analysis and interpretation and drafted and edited the report as necessary |
| Ahmed Aboulghate (PhD student, University of Cambridge) | Chapter 4 | Statistical analysis and interpretation of the data |
| Faraz Ahmed (PhD student, University of Cambridge) | Chapter 2 | Study researcher: contributed to study set-up (ethics and research management and governance), participant recruitment, data collection and analysis of the data for the South Asian workstream |
| Anthea Asprey (Associate Research Fellow, University of Exeter) | Chapters 7 and 9 | Study researcher: conducted focus groups and analysed the data |
| Heather Barry (Associate Research Fellow, University of Exeter) | Chapter 11 | Study researcher: contributed to study design, organisation, data collection, analysis and interpretation of the findings and drafting of the report |
| Julia Beckwith (Research Assistant, University of Cambridge) | Chapters 2 and 6 | Study researcher: contributed to reviewing the literature and analysis of the data and drafting of the report |
| John Benson (Senior Lecturer, University of Cambridge) | Chapters 3 and 6 | Study advisor: contributed to the design of the study, the development of the GCRS and data interpretation |
| Olga Boiko (Associate Research Fellow, University of Exeter) | Chapters 7 and 8 | Study researcher: contributed to the design of the study and study set-up, conducted focus groups, analysed the data and edited the report as necessary (Chapter 7 only) |
| Pete Bower (Professor, University of Manchester) | Chapter 10 | Study advisor: contributed to study design and reviewing the literature |
| Raff Calitri (Research Fellow, University of Exeter) | Chapter 11 | Study researcher: assisted with organisation, data collection and data analysis and interpretation |
| Mary Carter (Associate Research Fellow, University of Exeter) | Chapters 9 and 10 | Study researcher: collected qualitative and quantitative data, conducted the majority of the qualitative analysis and contributed to the interpretation of the results and drafted the qualitative section of the report |
| Antoinette Davey (Research Fellow, University of Exeter) | Chapters 2, 3 and 7–10 | Study researcher: responsible for reviewing the literature, contributed to study organisation (Chapter 9), participant recruitment and data collection and analysis and interpretation of the data and commented on the report |
| Marc N Elliott (Distinguished Chair in Statistics, RAND Corporation) | Chapters 3–6 and 9 | Statistical advisor: contributed to the design of the study, data analysis and interpretation and drafting of the report |
| Natasha Elmore (Research Assistant, University of Cambridge) | Chapters 2, 3 and 6–10 | Study researcher: contributed to the design of the study, study set-up (ethics and research management and governance) and reviewing the literature and assisted with participant recruitment and analysis of the data. Contributed to drafting of individual project reports and editing of the final report |
| Conor Farrington (Research Associate, University of Cambridge) | Chapters 7 and 8 | Study researcher: contributed to the design of the study, study set-up, data collection, analysed and interpreted the data and wrote the project report (Chapter 8 only) |
| Hena Wali Haque (Research Assistant, University of Cambridge) | Chapter 2 | Study researcher: contributed to participant recruitment and data collection |
| William Henley (Professor of Medical Statistics, University of Exeter) | Chapter 11 | Study contributor: assisted with data analysis and interpretation |
| Val Lattimer (Professor of Health Services Research and Dean of Health Sciences, University of East Anglia) | Chapter 4 | Study advisor: formulated aspects of the research question, contributed to the design of the analysis and interpretation of the data and commented on the report |
| Nadia Llanwarne (Academic Clinical Fellow/GP, University of Cambridge) | Chapters 2 and 6 | Study researcher: responsible for reviewing the literature, participant recruitment and data collection, analysis and interpretation of the data and drafting of the report |
| Cathy Lloyd (Professor of Health Studies, The Open University) | Chapters 5 and 6 | Study advisor: contributed to the interpretation of the ethnicity interaction analysis |
| Georgios Lyratzopoulos (Reader in Cancer Epidemiology, University College London) | Chapter 5 | Lead for the first ethnicity analysis: designed and oversaw the analysis, interpreted the data and contributed to drafting of the report |
| Inocencio Maramba (Associate Research Fellow, University of Exeter) | Chapters 2, 3 and 7–9 | Study researcher: assisted with study setup and practice and participant recruitment and was responsible for local data entry systems and storage (Chapter 3 only) |
| Luke Mounce (Associate Research Fellow, University of Exeter) | Chapter 10 | Study statistician: conducted quantitative analysis and contributed to the drafting of the project report |
| Jenny Newbould (Honorary Research Fellow, University of Cambridge) | Chapters 2 and 7–10 | Contributed to the design of the study, reviewing the literature, participant recruitment and data collection, analysis and interpretation of the data and drafting the report. Contributed to qualitative data collection (Chapter 10) |
| Charlotte Paddison (Senior Lecturer, Anglia Ruskin University) | Chapters 3 and 9 | Study advisor: contributed to the design of the study, study set-up (Chapter 9 only) and data interpretation |
| Richard Parker (Research Assistant, University of Cambridge) | Chapter 4 | Statistical analysis and interpretation of the data |
| Suzanne Richards (Senior Lecturer in Primary Care, University of Exeter) | Chapter 11 | Workstream lead: responsible for protocol design, study organisation, data collection, analysis and interpretation of the findings, drafting the report and critiquing all outputs for important intellectual content |
| Martin Roberts (Senior Psychometrician, University of Plymouth) | Chapter 9 | Statistician: conducted data analysis and contributed to the drafting of the report |
| Claude Setodji (Senior Statistician, RAND Corporation) | Chapter 5 | Study statistician: designed and conducted the DIF analysis, interpreted the data and contributed to the drafting of the report |
| Jonathan Silverman (Associate Dean, School of Clinical Medicine, University of Cambridge) | Chapter 3 | Study advisor: contributed to the design of the study, the development of the GCRS and data interpretation |
| Fiona Warren (Lecturer in Medical Statistics, University of Exeter) | Chapter 11 | Study contributor: assisted with data analysis and interpretation |
| Ed Wilson (Senior Research Associate, University of Cambridge) | Chapter 10 | Study researcher: responsible for the economic analysis |
| Christine Wright (Research Fellow, University of Exeter Medical School) | Chapter 10 | Workstream lead: contributed to the study design (development of the protocol), study organisation and set-up (recruitment of practices, ethics and research management and governance), reviewing the literature, supervision of data collection and collation of cost analysis data and contributed to the draft quantitative analysis plan and interpretation and drafting of the project report |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | All | Co-chief investigator and principal investigator: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| Contributions of others | ||
| Emily Taylor (Research Associate, University of Cambridge) | Chapters 2, 3 and 7–9 | Study researcher: assisted with study set-up, participant recruitment and data collection |
Chapter 2
We would like to thank the 13 general practices that participated in the project, particularly the 45 GPs who so kindly agreed for their consultations with patients to be video recorded. We would particularly like to thank the 52 patients who so generously gave up their time to take part in a video elicitation interview, without which this study would not have been possible. We also give our thanks to the three practices and eight GPs who took part in the additional interviews with South Asian respondents and the 23 patients from these three practices who so generously gave up their time to take part in a video elicitation interview, without which this study would not have been possible.
Finally, we would like to thank the IMPROVE advisory group who gave their advice on study design and interpretation of the data.
| Name and role | Nature of contribution |
|---|---|
| Contributions of authors | |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | Co-chief investigator and principal investigator for this work stream: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | Co-chief investigator: contributed to the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| Jenni Burt (Senior Research Associate, University of Cambridge) | Programme manager: contributed to the design and oversaw the conduct of the study, reviewing of the literature and analysis and interpretation of the data. Drafted and edited the report as necessary |
| Faraz Ahmed (PhD student, University of Cambridge) | Study researcher: contributed to study set-up (ethics and research management and governance), participant recruitment, data collection and analysis of the data for the South Asian workstream |
| Julia Beckwith (Research Assistant, University of Cambridge) | Study researcher: contributed to reviewing the literature and analysis of the data |
| Antoinette Davey (Research Fellow, University of Exeter) | Study researcher: responsible for reviewing the literature, participant recruitment and data collection and analysis and interpretation of the data and commented on the report |
| Natasha Elmore (Research Assistant, University of Cambridge) | Study researcher: contributed to the design of the study, study set-up (ethics and research management and governance) and reviewing of the literature and assisted with participant recruitment and analysis of the data |
| Hena Wali Haque (Research Assistant, University of Cambridge) | Study researcher: contributed to participant recruitment and data collection |
| Nadia Llanwarne (Academic Clinical Fellow/GP, University of Cambridge) | Study researcher: responsible for reviewing the literature, participant recruitment and data collection, analysis and interpretation of the data and drafting of the report |
| Inocencio Maramba (Associate Research Fellow, University of Exeter) | Study researcher: assisted with participant recruitment |
| Jenny Newbould (Honorary Research Fellow, University of Cambridge) | Workstream lead: contributed to the design of the study, reviewing the literature, participant recruitment and data collection, analysis and interpretation of the data and drafting the report |
| Contributions of others | |
| Emily Taylor (Research Associate, University of Cambridge) | Study researcher: assisted with participant recruitment and data collection |
Chapter 3
We would like to thank the 13 general practices that participated in the project, particularly the 45 GPs who so kindly agreed for their consultations with patients to be video recorded and the receptionists and administrative staff who assisted with patient identification and recruitment. We would also like to thank the 529 patients who completed a study questionnaire following their consultation with a GP. The rating of the consultations would not have been possible without our trained GCRS raters, to whom we extend our thanks for their expertise. Pete Bower gave important advice on study design. Finally, we would like to thank the IMPROVE advisory group who gave advice on study design and interpretation of the data.
| Name and role | Nature of contribution |
|---|---|
| Contributions of authors | |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | Co-chief investigator and principal investigator for this workstream: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | Co-chief investigator: contributed to the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| Jenni Burt (Senior Research Associate, University of Cambridge) | Programme manager and workstream lead: contributed to the design of the study, oversaw the conduct of the study, assisted with the analysis, interpreted the data and drafted the report |
| Gary Abel (Senior Research Associate, University of Cambridge) | Lead study statistician: contributed to the design of the study, conducted data analysis, interpreted the data and contributed to the drafting of the report |
| John Benson (Senior Lecturer, University of Cambridge) | Study advisor: contributed to the design of the study, the development of the GCRS and data interpretation |
| Antoinette Davey (Research Fellow, University of Exeter) | Study researcher: responsible for participant recruitment and data collection and assisted with data entry, storage and management |
| Marc N Elliott (Distinguished Chair in Statistics, RAND Corporation) | Statistical advisor: contributed to the design of the study, data analysis and interpretation and drafting of the report |
| Natasha Elmore (Research Assistant, University of Cambridge) | Study researcher: contributed to the design of the study and study set-up (ethics and research management and governance), assisted with participant recruitment, co-ordinated the rating of videos and responsible for local data entry systems and storage |
| Nadia Llanwarne (Academic Clinical Fellow/GP, University of Cambridge) | Study researcher: responsible for participant recruitment and data collection, assisted with data entry, storage and management and contributed to data interpretation |
| Inocencio Maramba (Associate Research Fellow, University of Exeter) | Study researcher: assisted with participant recruitment and responsible for local data entry systems and storage |
| Jenny Newbould (Honorary Research Fellow, University of Cambridge) | Study researcher: responsible for participant recruitment and data collection, assisted with data entry, storage and management and contributed to data interpretation |
| Charlotte Paddison (Senior Lecturer, Anglia Ruskin University) | Study advisor: contributed to the design of the study and data interpretation |
| Jonathan Silverman (Associate Dean, School of Clinical Medicine, University of Cambridge) | Study advisor: contributed to the design of the study, the development of the GCRS and data interpretation |
| Contributions of others | |
| James Brimicombe (Data Manager, University of Cambridge) | Data manager: designed data entry systems and storage |
| Emily Taylor (Research Associate, University of Cambridge) | Study researcher: assisted with participant recruitment and data collection |
Chapter 4
This chapter represents a series of analyses of GP Patient Survey data conducted by members of the IMPROVE team during the course of the programme. We would like to thank Ipsos MORI for assisting with various enquiries during this time, as well as NHS England for their continued support for our analytical work in this area.
| Name and role | Nature of contribution |
|---|---|
| Contributions of authors | |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | Co-chief investigator and principal investigator for this workstream: oversaw the design and conduct of the analyses and the interpretation of the data and edited the report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | Co-chief investigator: contributed to the design and conduct of the analysis and interpretation of the data and edited the report as necessary |
| Jenni Burt (Senior Research Associate, University of Cambridge) | Programme manager: contributed to the design and conduct of the analysis and the interpretation of the data and edited the report as necessary |
| Gary Abel (Senior Research Associate, University of Cambridge) | Lead study statistician: conceived and conducted data analyses, interpreted the data and drafted the report |
| Ahmed Aboulghate (PhD student, University of Cambridge) | Statistical analysis and interpretation of data |
| Marc N Elliott (Distinguished Chair in Statistics, RAND Corporation) | Statistical analysis and interpretation of data |
| Val Lattimer (Professor of Health Services Research and Dean of Health Sciences, University of East Anglia) | Study advisor: formulated aspects of the research question, contributed to the design of the analysis and the interpretation of the data and commented on the report |
| Richard Parker (Research Assistant, University of Cambridge) | Statistical analysis and interpretation of data |
Chapter 5
This chapter describes further analyses of GP Patient Survey data conducted by members of the IMPROVE team during the course of the programme. We would like to thank Ipsos MORI for assisting with various enquiries during this time, as well as NHS England for their continued support for our analytical work in this area.
| Name and role | Nature of contribution |
|---|---|
| Contributions of authors | |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | Co-chief investigator and co-principal investigator for this workstream: oversaw the design and conduct of the data analysis and interpretation of the data and edited the report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | Co-chief investigator and co-principal investigator for this workstream: oversaw the design and conduct of the data analysis and interpretation of the data and edited the report as necessary |
| Jenni Burt (Senior Research Associate, University of Cambridge) | Programme manager: contributed to the design of a number of the ethnicity analyses and interpretation of the data and drafted the report |
| Gary Abel (Senior Research Associate, University of Cambridge) | Study statistician: designed and conducted the ethnicity interaction analysis and the practice-level analysis and interpreted the data and drafted the report |
| Marc N Elliott (Distinguished Chair in Statistics, RAND Corporation) | Statistical advisor: contributed to the design, conduct and interpretation of the analyses and contributed to drafting of the report |
| Cathy Lloyd (Professor of Health Studies, The Open University) | Study advisor: contributed to the interpretation of the ethnicity interaction analysis |
| Georgios Lyratzopoulos (Reader in Cancer Epidemiology, University College London) | Lead for the first ethnicity analysis: designed and oversaw the analysis, interpreted the data and contributed to the drafting of the report |
| Claude Setodji (Senior Statistician, RAND Corporation) | Study statistician: designed and conducted the DIF analysis, interpreted the data and contributed to the drafting of the report |
Chapter 6
This complex study would not have been possible without the contributions of many people. We would particularly like to thank Steve Attmore for his assistance with the development and recording of the study vignettes and the Media Studio at Cambridge University Hospitals NHS Foundation Trust for its assistance with the recording and editing of the vignettes. We would also like to thank all of the actors who took roles in the vignettes. A wide range of staff at Ipsos MORI contributed to developing and refining the study design: we would particularly like to thank Anna Carluccio and Lara Sarson for their professional oversight of the project and their colleagues James Wilks and Victoria Hough for their input. Pete Bower gave important advice on study design. Special thanks go to the team of Ipsos MORI fieldworkers who so diligently worked to recruit participants and conduct interviews. Special thanks, too, to the 1124 participants who gave up their time to view and rate the simulated GP consultation vignettes – without them this project would not have been possible. Finally, many thanks to the IMPROVE advisory group for its assistance with study design and data interpretation.
| Name and role | Nature of contribution |
|---|---|
| Contributions of authors | |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | Co-chief investigator and principal investigator for this workstream: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | Co-chief investigator: contributed to the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| Jenni Burt (Senior Research Associate, University of Cambridge) | Programme manager and workstream lead: designed the study, designed and produced the study vignettes, oversaw the study conduct, assisted with the analysis and interpretation of the data and drafted the report |
| Gary Abel (Senior Research Associate, University of Cambridge) | Lead study statistician: designed the study, conducted data analysis and interpreted the data and drafted the report |
| Julia Beckwith (Research Assistant, University of Cambridge) | Study researcher: co-ordinated the rating of vignettes by GCRS raters |
| John Benson (Senior Lecturer, University of Cambridge) | Study advisor: contributed to the development and production of the study vignettes |
| Marc N Elliott (Distinguished Chair in Statistics, RAND Corporation) | Statistical advisor: contributed to the design of the study, data analysis and interpretation and drafting of the report |
| Natasha Elmore (Research Assistant, University of Cambridge) | Study researcher: conducted the review of the literature and contributed to the development and production of the study vignettes |
| Nadia Llanwarne (Academic Clinical Fellow/GP, University of Cambridge) | Study advisor: contributed to the production of the study vignettes |
| Cathy Lloyd (Professor of Health Studies, The Open University) | Study advisor: contributed to the study design and interpretation of the data |
| Contributions of others | |
| Steve Attmore (Simulated Patient Co-ordinator, University of Cambridge) | Study contributor: assisted with the design and development of the vignettes, recruited all vignette actors and oversaw the vignette role plays |
| Anna Carluccio (Research Director, Ipsos MORI) | Project leader at market research agency: contributed to the study design and oversaw data collection |
| Lara Sarson (Research Manager, Ipsos MORI) | Project manager at market research agency: contributed to the study design and managed data collection |
Chapter 7
This study, involving focus groups with practice staff, was linked to the survey reported in Chapter 9. We would like to thank all those practices that made our research team so welcome throughout the data collection period. We would particularly like to thank all of the practice staff who gave up their time to take part in focus groups and reflect on the way in which they engage with patient feedback: we are most grateful for their important contribution to this work. Thanks, too, to the IMPROVE advisory group who supported the research team throughout.
| Name and role | Nature of contribution |
|---|---|
| Contributions of authors | |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | Co-chief investigator: contributed to the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | Co-chief investigator and principal investigator for this workstream: oversaw the design and conduct of the study and the analysis and interpretation of the data and drafted the report |
| Jenni Burt (Senior Research Associate, University of Cambridge) | Programme manager: contributed to the design of the study, oversaw the conduct of the study, analysed the data and drafted the report |
| Anthea Asprey (Associate Research Fellow, University of Exeter) | Study researcher: conducted focus groups and analysed the data |
| Olga Boiko (Associate Research Fellow, University of Exeter) | Study researcher: contributed to the design of the study and study set-up, conducted focus groups, analysed the data and edited the report as necessary |
| Antoinette Davey (Research Fellow, University of Exeter) | Study researcher: assisted with study set-up, practice recruitment and data entry |
| Natasha Elmore (Research Assistant, University of Cambridge) | Study researcher: assisted with study set-up, practice recruitment, data collection and data entry |
| Conor Farrington (Research Associate, University of Cambridge) | Study researcher: assisted with data collection |
| Inocencio Maramba (Associate Research Fellow, University of Exeter) | Study researcher: assisted with study set-up and practice recruitment |
| Jenny Newbould (Honorary Research Fellow, University of Cambridge) | Study researcher: assisted with study set-up, practice recruitment and data collection |
| Contributions of others | |
| Emily Taylor (Research Associate, University of Cambridge) | Study researcher: assisted with study set-up, practice recruitment and data collection |
Chapter 8
This study, involving interviews with GPs, was linked to the survey reported in Chapter 9. We would like to thank all of those practices who made our research team so welcome as we conducted the survey. The study would not have happened without those GPs who were willing to participate in interviews with our research team, in which they reflected honestly and openly about their views on patient feedback and, in particular, their personal experiences of receiving patient feedback. Thanks go additionally to the IMPROVE advisory group who supported the research team throughout.
| Name and role | Nature of contribution |
|---|---|
| Contributions of authors | |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | Co-chief investigator and principal investigator for this workstream: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | Co-chief investigator and principal investigator for this workstream: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| Jenni Burt (Senior Research Associate, University of Cambridge) | Programme manager: contributed to the design of the study, oversaw the conduct of the study, contributed to data collection and the analysis of the data and edited the report as necessary |
| Olga Boiko (Associate Research Fellow, University of Exeter) | Study researcher: contributed to the design of the study and study set-up, carried out data collection and contributed to the analysis of the data |
| Antoinette Davey (Research Fellow, University of Exeter) | Study researcher: assisted with study set-up, practice recruitment and data entry |
| Natasha Elmore (Research Assistant, University of Cambridge) | Study researcher: assisted with study set-up and practice recruitment |
| Conor Farrington (Research Associate, University of Cambridge) | Study researcher: contributed to the design of the study and study set-up, carried out data collection, analysed and interpreted the data and drafted the report |
| Inocencio Maramba (Associate Research Fellow, University of Exeter) | Study researcher: assisted with study set-up and practice recruitment |
| Jenny Newbould (Honorary Research Fellow, University of Cambridge) | Study researcher: assisted with study set-up and practice recruitment |
| Contributions of others | |
| Emily Taylor (Research Associate, University of Cambridge) | Study researcher: assisted with study set-up and practice recruitment |
Chapter 9
The survey reported within this chapter involved a large-scale data collection effort to which many people contributed. We would like to extend particular thanks to the 25 general practices that participated in the study and the staff within these who went out of their way to ensure that the survey was conducted efficiently and to the highest standards. Thanks, too, to all those patients who took the time to respond to the questionnaire. The IMPROVE advisory group contributed particular help with devising study documentation, for which we are very grateful. Finally, Pete Bower gave important advice on study design.
| Name and role | Nature of contribution |
|---|---|
| Contributions of authors | |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | Co-chief investigator: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | Co-chief investigator and principal investigator for this workstream: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| Jenni Burt (Senior Research Associate, University of Cambridge) | Programme manager: contributed to the design of the study, oversaw study set-up, practice recruitment and the conduct of the study, contributed to data collection, analysis and interpretation and drafted and edited the report as necessary |
| Gary Abel (Senior Research Associate, University of Cambridge) | Statistician: contributed to the study design, sampled practices for the study, contributed to the analysis plan, analysis and data visualisation and interpretation of the study and helped to draft and edit the report |
| Anthea Asprey (Associate Research Fellow, University of Exeter) | Study researcher: contributed to study set-up |
| Mary Carter (Associate Research Fellow, University of Exeter) | Study researcher: contributed to study set-up, practice recruitment and data collection |
| Antoinette Davey (Research Fellow, University of Exeter) | Study researcher: contributed to the organisation of the study, practice recruitment, data collection and drafting of the report |
| Marc N Elliott (Distinguished Chair in Statistics, RAND Corporation) | Statistical advisor: contributed to the study design, analysis plan and analysis and data visualisation and interpretation of the study and helped to draft and critique the report |
| Natasha Elmore (Research Assistant, University of Cambridge) | Study researcher: contributed to study set-up, practice recruitment, data collection and data entry |
| Inocencio Maramba (Associate Research Fellow, University of Exeter) | Study researcher: contributed to data collection |
| Jenny Newbould (Honorary Research Fellow, University of Cambridge) | Study researcher: contributed to study set-up, practice recruitment and data collection |
| Charlotte Paddison (Senior Lecturer, Anglia Ruskin University) | Study advisor: assisted with study design and set up |
| Martin Roberts (Senior Psychometrician, University of Plymouth) | Statistician: conducted data analysis and contributed to the drafting of the report |
| Contributions of others | |
| Amy Gratton (Administrator, University of Exeter) | Study administrator: assisted with study organisation and data entry |
| Dawn Swancutt (Project Manager, University of Exeter) | Project manager: assisted with study set-up |
| Emily Taylor (Research Associate, University of Cambridge) | Study researcher: contributed to study set-up, practice recruitment and data collection |
| Emma Whitton (Administrator, University of Exeter) | Study administrator: assisted with study organisation and data entry |
Chapter 10
The completion of this complex workstream was possible only with the contribution of a great many people. We would particularly like to thank the patients and staff from the 12 general practices that participated in the project. We thank also the facilitators who delivered team feedback reflection sessions at six practices during the feasibility and exploratory trial phases. We are grateful to staff from CRT Limited (particularly Richard Farrell, Toby Knight and Nicky Allen) who provided the touch screen equipment, organised data cleaning and summarising, prepared RTF reports for practices and provided technical assistance and advice before and during the RTF implementation period. Antoinette Davey and Mary Carter conducted all of the fieldwork in the South West of England. Natasha Elmore, Jenny Newbould and Jenni Burt conducted fieldwork at the two Cambridge practices during the exploratory trial phase. Ed Wilson designed, analysed and reported the cost analysis elements. Luke Mounce conducted and reported the quantitative analysis for the exploratory trial phase. John Campbell, Martin Roland, Jenni Burt and Gary Abel provided helpful comments during the development of the study protocol. Christine Wright managed the overall conduct and delivery of the workstream.
The development of the value of patient feedback tool as part of this workstream rested on the contribution of a wide range of people. We would particularly like to thank Nadia Llanwarne, John Benson, Felix Greaves and Pete Bower for their critical input into the construction of the scale; the GPs who took part in cognitive interviews; and all of the clinicians who participated in the piloting phases of development by completing the scale in its various developmental incarnations. James Brimicombe was instrumental in co-ordinating online piloting of the tool and we extend our particular thanks to him for all his support with this.
| Name and role | Nature of contribution |
|---|---|
| Contributions of authors | |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | Co-chief investigator: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | Co-chief investigator and principal investigator for this workstream: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| Jenni Burt (Senior Research Associate, University of Cambridge) | Programme manager: contributed to the design of the study and the qualitative analysis plan, conducted qualitative data collection, assisted with interpretation of the data and edited the report as necessary |
| Pete Bower (Professor, University of Manchester) | Study advisor: contributed to study design and reviewing the literature |
| Mary Carter (Associate Research Fellow, University of Exeter) | Study researcher: collected quantitative and qualitative data, conducted the majority of the qualitative analysis and contributed to the interpretation of the results and; drafted the qualitative section of the report |
| Antoinette Davey (Research Fellow, University of Exeter) | Contributed to the study design, organisation of the study, recruitment of practices, data collection and input, analysis and interpretation of the qualitative data and drafting of the qualitative results for the report |
| Natasha Elmore (Research Assistant, University of Cambridge) | Study researcher: responsible for local study set-up (research management and governance and recruitment), collected the data for the Cambridge sites, entered the data for the Cambridge sites and commented on the report |
| Luke Mounce (Associate Research Fellow, University of Exeter) | Study statistician: conducted quantitative analysis and contributed to the drafting of the report |
| Jenny Newbould (Honorary Research Fellow, University of Cambridge) | Study researcher: contributed to qualitative data collection |
| Ed Wilson (Senior Research Associate, University of Cambridge) | Study researcher: responsible for the economic analysis |
| Christine Wright (Research Fellow, University of Exeter Medical School) | Workstream lead: contributed to study design (development of the protocol), study organisation and set-up (recruitment of practices, ethics and research management and governance), reviewing of the literature, supervision of data collection, collation of cost analysis data, the draft quantitative analysis plan, data interpretation and drafting of the report |
Chapter 11
The out-of-hours workstream involved contributions from a wide variety of organisations and people. We would particularly like to thank all of the out-of-hours service providers and their staff who participated in the research. We also thank all those service users who took the time to respond to the questionnaire. We are grateful to Ipsos MORI for their assistance in providing the GP Patient Survey data for analysis; Jonathan Jackson from the Health and Social Care Information Centre for providing the data to map out-of-hours GP providers to associated practices and for analytical input; and James Wallis from NHS England for analytical input. Finally, we thank Martin Roberts for statistical assistance.
| Name and role | Nature of contribution |
|---|---|
| Contributions of authors | |
| Martin Roland (Professor of Health Services Research, University of Cambridge) | Co-chief investigator: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| John Campbell (Professor of General Practice and Primary Care, University of Exeter) | Co-chief investigator and principal investigator for this workstream: oversaw the design and conduct of the study and the analysis and interpretation of the data and edited the report as necessary |
| Anthea Asprey (Associate Research Fellow, University of Exeter) | Study researcher: secondary analysis of qualitative data |
| Heather Barry (Associate Research Fellow, University of Exeter) | Study researcher: contributed to study design, organisation, data collection and analysis and interpretation of the findings and drafting of the report |
| Raff Calitri (Research Fellow, University of Exeter) | Study researcher: assisted with organisation, data collection, data analysis and interpretation |
| Antoinette Davey (Research Fellow, University of Exeter) | Study researcher: contributed to data collection and data entry |
| William Henley (Professor of Medical Statistics, University of Exeter) | Study contributor: assisted with data analysis and interpretation |
| Luke Mounce (Associate Research Fellow, University of Exeter) | Study researcher: contributed to study design, organisation, data collection and analysis and interpretation of the findings and drafting of the report |
| Suzanne Richards (Senior Lecturer in Primary Care, University of Exeter) | Workstream lead: responsible for protocol design, study organisation, data collection, analysis and interpretation of the findings, drafting of the report and critiquing all outputs for important intellectual content |
| Fiona Warren (Lecturer in Medical Statistics, University of Exeter) | Study contributor: assisted with data analysis and interpretation |
Publications
Articles
Chapter 4
Aboulghate A, Abel G, Elliott MN, Parker RA, Campbell J, Lyratzopoulos G, Roland M. Do English patients want continuity of care, and do they receive it? Br J Gen Pract 2012;62:e567–75.
Chapter 5
Lyratzopoulos G, Elliott M, Barbiere JM, Henderson A, Staetsky L, Paddison C, et al. Understanding ethnic and other socio-demographic differences in patient experience of primary care: evidence from the English General Practice Patient Survey. BMJ Qual Saf 2012;21:21–9.
Setodji CM, Elliott MN, Abel G, Burt J, Roland M, Campbell J. Differential item functioning in the English General Practice Patient Survey: comparison of South Asian and white British subgroups. Med Care 2015;53:809–17.
Burt J, Lloyd C, Campbell J, Roland M, Abel, G. Variations in GP–patient communication by ethnicity, age, and gender: evidence from a national primary care patient survey. Br J Gen Pract 2016;66:e47–52.
Chapter 7
Boiko O, Campbell JL, Elmore N, Davey AF, Roland M, Burt J. The role of patient experience surveys in quality assurance and improvement: a focus group study in English general practice. Health Expect 2015;18:1982–94.
Chapter 9
Roberts MJ, Campbell JL, Abel GA, Davey AF, Elmore NL, Maramba I, et al. Understanding high and low patient experience scores in primary care: analysis of patients’ survey data for general practices and individual doctors. BMJ 2014;349:g6034.
Maramba ID, Davey A, Elliott M, Roberts M, Roland M, Brown F, et al. Web-based textual analysis of free-text comments from patients in primary care. JMIR Med Inform 2015;3:e20.
Chapter 11
Warren FC, Abel G, Lyratzopoulos G, Elliott MN, Richards S, Barry HE, et al. Characteristics of service users and provider organisations associated with experience of out of hours general practitioner care in England: population based cross sectional postal questionnaire survey. BMJ 2015;350:h2040.
Additional references
Ahmed F, Burt J, Roland M. Measuring patient experience: concepts and methods. Patient 2014;7:235–41.
Burt J, Abel G, Elmore N, Campbell J, Roland M, Benson J, Silverman J. Assessing communication quality of consultations in primary care: initial reliability of the Global Consultation Rating Scale, based on the Calgary–Cambridge guide to the medical interview. BMJ Open 2014;6:e004339.
Ahmed F, Abel GA, Lloyd CE, Burt J, Roland M. Does the availability of a South Asian language in practices improve reports of doctor–patient communication from South Asian patients? Cross sectional analysis of a national patient survey in English general practices. BMC Fam Pract 2015;16:55.
Barry HE, Campbell JL, Asprey A, Richards SH. The use of patient experience survey data by out-of-hours primary care services: a qualitative interview study. BMJ Qual Saf 2016;25:851–9.
Brodie K, Abel G, Burt J. Languages spoken at home and the association between ethnicity and doctor-patient communication in primary care: analysis of survey data for South Asian and white British patients. BMJ Open 2016;6:e010042.
Burt J, Abel G, Elmore N, Lloyd C, Benson J, Sarson L, Roland M. Understanding negative feedback from South Asian patients: an experimental vignette study. BMJ Open 2016;6:e011256.
Burt J, Abel G, Elmore N, Newbould J, Davey A, Llanwarne N, et al. Rating communication in GP consultations: the association between ratings made by patients and trained clinical raters [published online ahead of print 3 October 2016]. Med Care Res Rev 2016.
Carter M, Davey A, Wright C, Elmore N, Newbould J, Roland M, et al. Capturing patient experience: a qualitative study of implementing real-time feedback in primary care. Br J Gen Pract 2016;66:e786–93.
Davey AF, Roberts MJ, Mounce L, Maramba I, Campbell JL. Test–retest stability of patient experience items derived from the national GP patient survey. Springerplus 2016;5:1755.
Elmore N, Burt J, Abel G, Maratos FA, Montague J, Campbell J, Roland M. Investigating the relationship between consultation length and patient experience: a cross-sectional study in primary care. Br J Gen Pract 2016;66:e896–e903.
Farrington C, Burt J, Boiko O, Campbell J, Roland M. Doctors’ engagements with patient experience surveys in primary and secondary care: a qualitative study [published online ahead of print 28 April 2016]. Health Expect 2016.
Mounce LTA, Barry HE, Calitri R, Henley WE, Campbell J, Roland M, Richards S. Establishing the validity of National GP Patient Survey items evaluating out-of-hours care. BMJ Qual Saf 2016;25:842–50.
Wright C, Davey A, Elmore N, Carter M, Mounce L, Wilson E, et al. Patients’ use and views of real-time feedback technology in general practice [published online ahead of print 28 April 2016]. Health Expect 2016.
Burt J, Newbould J, Abel G, Elliot MN, Beckwith J, Llanwarne N, et al. Investigating the meaning of ‘good’ or ‘very good’ patient evaluations of care in English general practice: a mixed methods study. BMJ Open 2017; in press.
Llanwarne N, Newbould J, Burt J, Campbell JL, Roland M. Wasting the doctor’s time? A video-elicitation interview study with patients in primary care. Soc Sci Med 2017;176:113–22.
Conference presentations
Chapter 2
Newbould J. Patient Questionnaires – a Useful Reflection of Patient Experience? Society of Academic Primary Care London & South East Regional Conference, Cambridge, UK, January 2015.
Newbould J. The Tick and the Talk: do Patients’ Survey Responses Relate to their Narrated Experience of Primary Care Consultations? Society of Academic Primary Care Annual Scientific Meeting, Oxford, UK, July 2015.
Chapter 3
Abel G. Rating Communication in GP Consultations: Do Patients and Experienced Trained Raters Agree? Society for Academic Primary Care London & South East Regional Conference, Cambridge, UK, January 2015.
Abel G. Rating Communication in GP Consultations: Do Patients and Experienced Trained Raters Agree? Health Services Research Network (HSRN), Nottingham, UK, July 2015.
Abel G. Rating Communication in GP Consultations: Do Patients and Experienced Trained Raters Agree? Society of Academic Primary Care Annual Scientific Meeting, Oxford, UK, July 2015.
Chapter 5
Burt J. Do Ethnic Disparities in Patient Reported GP–Patient Communication Vary by Age and Gender? Evidence from a National Patient Survey. Society of Academic Primary Care Annual Scientific Meeting, Oxford, UK, July 2015.
Chapter 6
Burt J. How Do White British and Pakistani People Rate Communication within Simulated GP–Patient Consultations? A National Experimental Vignette Study. Society of Academic Primary Care Annual Scientific Meeting, Oxford, UK, July 2015.
Burt J. How Do White British and Pakistani People Rate Communication within Simulated GP–Patient Consultations? A National Experimental Vignette Study. Health Services Research Network (HSRN), Nottingham, UK, July 2015.
Burt J. Using Vignettes to Understand Differences in Patient Experiences for White and Pakistani Adults in England. North American Primary Care Research Group Annual Meeting, Cancun, Mexico, October 2015.
Chapter 7
Boiko O. Acting on Patient Feedback: Managing Change and Improving Service in Response to Patient Surveys in Primary Care. Society for Academic Primary Care South West meeting, Southampton, UK, March 2013.
Chapter 8
Farrington C. Dimensions of Ambivalence: Doctor and Patient Experience Surveys in Primary and Secondary Care. British Sociological Association Medical Sociology Group Annual Conference, University of York, UK, September 2015.
Chapter 9
Davey A. Investigating the Stability of Patient Responses to GP Patient Survey Items using Test Retest Methodology. Society of Academic Primary Care Annual Scientific Meeting, Oxford, UK, July 2013.
Burt J. My GP Practice Scores Well on Doctors’ Consultations Skills – So Why Doesn’t My Doctor Listen to Me? Society of Academic Primary Care Annual Scientific Meeting, Oxford, UK, July 2013.
Maramba I. Researching with our Head in the Clouds? Using Tag Clouds to Analyse Free Text Patient Feedback. Medicine 2.0, London, UK, September 2013.
Chapter 10
Wright C. Feasibility and Acceptability of a Real Time Feedback Intervention to Improve Patient Experience in General Practice: Preliminary Results. Society of Academic Primary Care Annual Scientific Meeting, Glasgow, UK, July 2014.
Carter M. Using Touch Screens in GP Surgery Waiting Areas to Collect Real-Time Patient Feedback: Practice Staff Views of Feasibility and Acceptability. Society of Academic Primary Care South West meeting, Birmingham, UK, March 2015.
Chapter 11
Barry H. Understanding Patient Experience of Out-of-Hours Primary Care in England. Society for Academic Primary Care South West meeting, Southampton, UK, March 2013.
Barry H. Understanding Patient Experience of Out-of-Hours Primary Care: a Pilot Study. Society of Academic Primary Care Annual Scientific Meeting, Nottingham, UK, July 2013.
Barry H. A Cross-Sectional Survey Study of Service Users’ Experiences of Out-of-Hours Primary Medical Care in England. Society for Academic Primary Care South West meeting, Bristol, UK, March 2014.
Barry H. A Cross-Sectional Survey Study of Service Users’ Experiences of Out-of-Hours Primary Medical Care in England. Society of Academic Primary Care Annual Scientific Meeting, Glasgow, UK, July 2014.
Mounce L. Establishing the Potential Validity of English GP Patient Survey Items Evaluating Out-of-Hours Care. Society of Academic Primary Care South West meeting, Birmingham, UK, March 2015.
Richards S. How do staff from GP out-of-hours services use patient feedback to drive quality improvement? A qualitative interview study (POSTER). Health Services Research Network, Annual Scientific Meeting, Nottingham, UK, July 2015.
Richards S. How Do Staff from GP Out-of-Hours Services USE Patient Feedback to Drive Quality Improvement? A Qualitative Interview Study. Society of Academic Primary Care Annual Scientific Meeting Oxford, UK, July 2015.
Data sharing statement
The data arising from the various strands of this programme of work have been archived in accordance with the agreements covered by ethical approvals for the research. For Chapters 2 and 3, consent to participate was given on condition of video recording of consultations and associated data being accessible only to members of the immediate IMPROVE research team; therefore, the data generated in these chapters are not suitable for sharing. The GP Patient Survey data used in Chapters 4 and 5 are accessible by request to NHS England; please contact the author for further information. Chapter 6 data – rating of simulated consultations – are available by request to the author, as are the survey data set out in Chapter 9. The data used in Chapters 7, 8, 10 and 11, including interviews, observations and focus groups, are not currently available to ensure the confidentiality of the practices that participated.
Disclaimers
This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, CCF, NETSCC, PGfAR or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the PGfAR programme or the Department of Health.
References
- Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 1990.
- Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff 2008;27:759-69. http://dx.doi.org/10.1377/hlthaff.27.3.759.
- Scoring Healthcare: Navigating Customer Experience Rating. New York, NY: PwC; 2013.
- NHS England . Improving General Practice: A Call to Action 2013. www.england.nhs.uk/ourwork/com-dev/igp-cta/ (accessed 3 May 2016).
- NHS GP Practices Indicators and Methodology. London; 2015.
- Rozenblum R, Bates DW. Patient-centred healthcare, social media and the internet: the perfect storm?. BMJ Qual Saf 2013;22:183-6. http://dx.doi.org/10.1136/bmjqs-2012-001744.
- Health and Social Care Information Centre . Quality and Outcomes Framework (QOF) 2014. www.hscic.gov.uk/qof (accessed 12 April 2016).
- Campbell S, Reeves D, Kontopantelis E, Middleton E, Sibbald B, Roland M. Quality of primary care in England with the introduction of pay for performance. N Engl J Med 2007;357:181-90. http://dx.doi.org/10.1056/NEJMsr065990.
- High Quality Care for All. NHS Next Stage Review Final Report. London: DH; 2008.
- NHS England . NHS Outcome Framework: Domain 4: Ensuring That People Have a Positive Experience of Car n.d. www.england.nhs.uk/resources/resources-for-ccgs/out-frwrk/dom-4/ (accessed 3 May 2016).
- NHS England . GP Patient Survey 2015. https://gp-patient.co.uk/ (accessed 3 May 2016).
- NHS Choices n.d. www.nhs.uk (accessed 5 December 2016).
- University of Cambridge . General Practice Assessment Questionnaire (GPAQ) n.d. www.phpc.cam.ac.uk/gpaq/ (accessed 3 May 2016).
- Quality Improvement Made Simple: What Everyone Should Know about Healthcare Quality Improvement. London: Health Foundation; 2013.
- Campbell SM, Reeves D, Kontopantelis E, Sibbald B, Roland M. Effects of pay for performance on the quality of primary care in England. N Engl J Med 2009;361:368-78. http://dx.doi.org/10.1056/NEJMsa0807651.
- Rodriguez HP, Rogers WH, Marshall RE, Safran DG. Multidisciplinary primary care teams: effects on the quality of clinician–patient interactions and organizational features of care. Med Care 2007;45:19-27. https://doi.org/10.1097/01.mlr.0000241041.53804.29.
- Rodriguez HP, Marshall RE, Rogers WH, Safran DG. Primary care physician visit continuity: a comparison of patient-reported and administratively derived measures. J Gen Intern Med 2008;23:1499-502. http://dx.doi.org/10.1007/s11606-008-0692-z.
- Bower P, Roland M, Campbell J, Mead N. Setting standards based on patients’ views on access and continuity: secondary analysis of data from the general practice assessment survey. BMJ 2003;326. https://doi.org/10.1136/bmj.326.7383.258.
- Care Quality Commission . Adult Inpatient Survey 2015 n.d. www.cqc.org.uk/content/adult-inpatient-survey-2015 (accessed 13 December 2016).
- Sitzia J, Wood N. Patient satisfaction: a review of issues and concepts. Soc Sci Med 1997;45:1829-43. https://doi.org/10.1016/S0277-9536(97)00128-7.
- Russell S. Patients’ Experiences: Top Heavy with Research. A Literature Review. Melbourne, VIC: Bayside Medicare Local; 2013.
- Salisbury C, Wallace M, Montgomery AA. Patients’ experience and satisfaction in primary care: secondary analysis using multilevel modelling. BMJ 2010;341. http://dx.doi.org/10.1136/bmj.c5004.
- Buetow SA. What do general practitioners and their patients want from general practice and are they receiving it? A framework. Soc Sci Med 1995;40:213-21. https://doi.org/10.1016/0277-9536(94)E0074-3.
- Wensing M, Jung HP, Mainz J, Olesen F, Grol R. A systematic review of the literature on patient priorities for general practice care. Part 1: description of the research domain. Soc Sci Med 1998;47:1573-88. https://doi.org/10.1016/S0277-9536(98)00222-6.
- Wofford MM, Wofford JL, Bothra J, Kendrick SB, Smith A, Lichstein PR. Patient complaints about physician behaviors: a qualitative study. Acad Med 2004;79:134-8. https://doi.org/10.1097/00001888-200402000-00008.
- Coulter A. Trends in Patients’ Experience of the NH. Oxford: Picker Institute Europe; 2006.
- Benson J, Britten N. Patients’ decisions about whether or not to take antihypertensive drugs: qualitative study. BMJ 2002;325. https://doi.org/10.1136/bmj.325.7369.873.
- Griffith S. A review of the factors associated with patient compliance and the taking of prescribed medicines. Br J Gen Pract 1990;40:114-16.
- Pound P, Britten N, Morgan M, Yardley L, Pope C, Daker-White G, et al. Resisting medicines: a synthesis of qualitative studies of medicine taking. Soc Sci Med 2005;61:133-55. http://dx.doi.org/10.1016/j.socscimed.2004.11.063.
- Kim SS, Park BK. Patient-perceived communication styles of physicians in rehabilitation: the effect on patient satisfaction and compliance in Korea. Am J Phys Med Rehabil 2008;87:998-1005. http://dx.doi.org/10.1097/PHM.0b013e318186babf.
- Stewart M, Brown JB, Donner A, McWhinney IR, Oates J, Weston WW, et al. The impact of patient-centered care on outcomes. J Fam Pract 2000;49:796-804.
- Orth JE, Stiles WB, Scherwitz L, Hennrikus D, Vallbona C. Patient exposition and provider explanation in routine interviews and hypertensive patients’ blood pressure control. Health Psychol 1987;6:29-42. https://doi.org/10.1037/0278-6133.6.1.29.
- Kuzel AJ, Woolf SH, Gilchrist VJ, Engel JD, LaVeist TA, Vincent C, et al. Patient reports of preventable problems and harms in primary health care. Ann Fam Med 2004;2:333-40. https://doi.org/10.1370/afm.220.
- Vincent C, Davy C, Esmail A, Neale G, Elstein M, Cozens JF, et al. Learning from litigation. The role of claims analysis in patient safety. J Eval Clin Prac 2006;12:665-74. http://dx.doi.org/10.1111/j.1365-2753.2006.00634.x.
- Vincent CA, Coulter A. Patient safety: what about the patient?. Qual Saf Health Care 2002;11:76-80. https://doi.org/10.1136/qhc.11.1.76.
- Koutantji M, Davis R, Vincent CA, Coulter A. The patient’s role in patient safety: engaging patients, their representatives, and health professionals. Clin Risk 2005;11:99-104.
- Macdonald W, Rogers A, Blakeman T, Bower P. Practice nurses and the facilitation of self-management in primary care. J Adv Nurs 2008;62:191-9. http://dx.doi.org/10.1111/j.1365-2648.2007.04585.x.
- Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open 2013;3. http://dx.doi.org/10.1136/bmjopen-2012-001570.
- Kennedy A, Nelson E, Reeves D, Richardson G, Roberts C, Robinson A, et al. A randomised controlled trial to assess the impact of a package comprising a patient-orientated, evidence-based self-help guidebook and patient-centred consultations on disease management and satisfaction in inflammatory bowel disease. Health Technol Assess 2003;7. https://doi.org/10.3310/hta7280.
- Zolnierek KB, Dimatteo MR. Physician communication and patient adherence to treatment: a meta-analysis. Med Care 2009;47:826-34. http://dx.doi.org/10.1097/MLR.0b013e31819a5acc.
- Carcaise-Edinboro P, Bradley CJ. Influence of patient–provider communication on colorectal cancer screening. Med Care 2008;46:738-45. http://dx.doi.org/10.1097/MLR.0b013e318178935a.
- Hsiao CJ, Boult C. Effects of quality on outcomes in primary care: a review of the literature. Am J Med Qual 2008;23:302-10. http://dx.doi.org/10.1177/1062860608315643.
- Patient Experience . Understanding the Patient Experience n.d. www.patientexperience.co.uk/en/understanding-the-patient-experience/ (accessed 22 July 2015).
- Robert G, Cornwell J. Rethinking policy approaches to measuring and improving patient experience. J Health Serv Res Pol 2013;18:67-9. https://doi.org/10.1177/1355819612473583.
- Coulter A, Locock L, Ziebland S, Calabrese J. Collecting data on patient experience is not enough: they must be used to improve care. BMJ 2014;348. http://dx.doi.org/10.1136/bmj.g2225.
- Locock L, Robert G, Boaz A, Vougioukalou S, Shuldham C, Fielden J, et al. Using a national archive of patient experience narratives to promote local patient-centered quality improvement: an ethnographic process evaluation of ‘accelerated’ experience-based co-design. J Health Serv Res Policy 2014;19:200-7. https://doi.org/10.1177/1355819614531565.
- A Best Practice Guide to Using Real-Time Patient Feedback. London: NHS Practice Management Network; 2010.
- Anderson J, Gigg S, Saunders K, Davis A. Working to Improve Patient Experience: Real-Time Monitoring. London: Imperial College Healthcare Trust; 2011.
- Airey C, Bruster S, Erens B, Lilley S, Pickering K, Pitson L. National Surveys of NHS Patients: General Practice, 1998. London: Picker Europe; 1999.
- CFEP UK Surveys . Improving Practice Questionnaire. (IPQ) n.d. www.cfepsurveys.co.uk/OrganisationalAndPersonalDevelopment/PrimaryCare/IPQ (accessed 3 May 2016).
- Roland M. Linking physicians’ pay to the quality of care – a major experiment in the United kingdom. N Engl J Med 2004;351:1448-54. http://dx.doi.org/10.1056/NEJMhpr041294.
- Sainsbury R, Ditch J, Hutton S. Computer assisted personal interviewing. Soc Res Update 1993;3:1-12.
- Picker Institute Europe . About NHS Patient Surveys n.d. www.nhssurveys.org/ (accessed 3 May 2016).
- Cleary PD, Crofton C, Hays RD, Horner R. Advances from the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Project. Introduction. Med Care 2012;50. http://dx.doi.org/10.1097/MLR.0b013e31826ec0cb.
- NHS Patient Survey Programme . National Surveys of NHS Patients 1998. http://webarchive.nationalarchives.gov.uk/20130107105354/http://dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/documents/digitalasset/dh_4035999.pdf (accessed 3 May 2016).
- Hall JA, Dornan MC. What patients like about their medical care and how often they are asked: a meta-analysis of the satisfaction literature. Soc Sci Med 1988;27:935-9. https://doi.org/10.1016/0277-9536(88)90284-5.
- Cheraghi-Sohi S, Bower P, Mead N, McDonald R, Whalley D, Roland M. What are the key attributes of primary care for patients? Building a conceptual ‘map’ of patient preferences. Health Expect 2006;9:275-84. http://dx.doi.org/10.1111/j.1369-7625.2006.00395.x.
- NHS Patient Experience Framework. London: DH; 2012.
- Review of the Friends and Family Test. London: Department of Health; 2014.
- Friends and Family Test in General Guidance: Data Submission Guidance. London: NHS Employers; 2014.
- Kmietowicz Z. Friends and family test ‘unfit’ for comparing NHS services, finds research. BMJ 2014;348. http://dx.doi.org/10.1136/bmj.g4355.
- Sizmur S, Graham C, Walsh J. Influence of patients’ age and sex and the mode of administration on results from the NHS Friends and Family Test of patient experience. J Health Serv Res Policy 2015;20:5-10. http://dx.doi.org/10.1177/1355819614536887.
- Raising Standards for Patients. New Partnerships in Out-Of-Hours Care; an Independent Review of GP Out-of-Hours Services in England. London: DH; 2000.
- House of Commons Health Committee . GP Out-of-Hours Services. Fifth Report of Session 2003–04 n.d. www.publications.parliament.uk/pa/cm200304/cmselect/cmhealth/697/69702.htm (accessed 3 May 2016).
- The Provision of Out-of-Hours Care in England. London: The Stationery Office; 2006.
- UK Department of Health . National Quality Requirements in the Delivery of Out-Of-Hours Services 2004. http://webarchive.nationalarchives.gov.uk/20080313140813/http://dh.gov.uk/en/publicationsandstatistics/publications/publicationspolicyandguidance/dh_4091213 (accessed 3 May 2016).
- Hill JJ, Asprey A, Richards SH, Campbell JL. Multisource feedback questionnaires in appraisal and for revalidation: a qualitative study in UK general practice. Br J Gen Prac 2012;62. http://dx.doi.org/10.3399/bjgp12X641429.
- Ipsos MORI. GP Patient Survey Access Survey Final Technical Report 2007 2007. http://gp-survey-production.s3.amazonaws.com/archive/2007/June%202007%20Technical%20Annex.pdf (accessed 3 May 2016).
- El Turabi A, Abel GA, Roland M, Lyratzopoulos G. Variation in reported experience of involvement in cancer treatment decision making: evidence from the National Cancer Patient Experience Survey. Br J Cancer 2013;109:780-7. https://doi.org/10.1038/bjc.2013.316.
- Saunders CL, Abel GA, Lyratzopoulos G. Inequalities in reported cancer patient experience by socio-demographic characteristic and cancer site: evidence from respondents to the English Cancer Patient Experience Survey. Eur J Cancer Care (Engl) 2015;24:85-98. https://doi.org/10.1111/ecc.12267.
- Paddison C, Saunders C, Abel G, Payne R, Adler A, Graffy J, et al. How do people with diabetes describe their experiences in primary care? Evidence from 85,760 patients with self-reported diabetes from the English General Practice Patient Survey. Diabetes Care 2015;38:469-75. https://doi.org/10.2337/dc14-1095.
- Abel G, Mavaddat N, Elliott M, Lyratzopoulos G, Roland M. Primary care experience of people with longstanding psychological problems: evidence from a national UK survey. Int Rev Psychiatry 2011;23:2-9. https://doi.org/10.3109/09540261.2010.545985.
- Saunders CL, Abel GA, Lyratzopoulos G. What explains worse patient experience in London? Evidence from secondary analysis of the Cancer Patient Experience Survey. BMJ Open 2014;4. https://doi.org/10.1136/bmjopen-2013-004039.
- Howie JG, Heaney DJ, Maxwell M, Walker JJ, Freeman GK. Developing a ‘consultation quality index’ (CQI) for use in general practice. Fam Pract 2000;17:455-61. https://doi.org/10.1093/fampra/17.6.455.
- Mead N, Roland M. Understanding why some ethnic minority patients evaluate medical care more negatively than white patients: a cross sectional analysis of a routine patient survey in English general practices. BMJ 2009;339. http://dx.doi.org/10.1136/bmj.b3450.
- Kontopantelis E, Roland M, Reeves D. Patient experience of access to primary care: identification of predictors in a national patient survey. BMC Fam Pract 2010;11. http://dx.doi.org/10.1186/1471-2296-11-61.
- Ahmed F, Burt J, Roland M. Measuring patient experience: concepts and methods. Patient 2014;7:235-41. http://dx.doi.org/10.1007/s40271-014-0060-5.
- Taira DA, Safran DG, Seto TB, Rogers WH, Inui TS, Montgomery J, et al. Do patient assessments of primary care differ by patient ethnicity?. Health Serv Res 2001;36:1059-71.
- Murray-Garcia JL, Selby JV, Schmittdiel J, Grumbach K, Quesenberry CP. Racial and ethnic differences in a patient survey: patients’ values, ratings, and reports regarding physician primary care performance in a large health maintenance organization. Med Care 2000;38:300-10. https://doi.org/10.1097/00005650-200003000-00007.
- Elliott MN, Haviland AM, Kanouse DE, Hambarsoomian K, Hays RD. Adjusting for subgroup differences in extreme response tendency in ratings of health care: impact on disparity estimates. Health Serv Res 2009;44:542-61. http://dx.doi.org/10.1111/j.1475-6773.2008.00922.x.
- Weinick RM, Elliott MN, Volandes AE, Lopez L, Burkhart Q, Schlesinger M. Using standardized encounters to understand reported racial/ethnic disparities in patient experiences with care. Health Serv Res 2011;46:491-509. http://dx.doi.org/10.1111/j.1475-6773.2010.01214.x.
- King G, Murray CJ, Salomon JA, Tandon A. Enhancing the validity and cross-cultural comparability of measurement in survey research. Am Polit Sci Rev 2004;98:191-207. https://doi.org/10.1017/S000305540400108X.
- Cheraghi-Sohi S, Bower P. Can the feedback of patient assessments, brief training, or their combination, improve the interpersonal skills of primary care physicians? A systematic review. BMC Health Serv Res 2008;8. http://dx.doi.org/10.1186/1472-6963-8-179.
- Berwick DM, James B, Coye MJ. Connections between quality measurement and improvement. Med Care 2003;41:I30-8. https://doi.org/10.1097/00005650-200301001-00004.
- Fung CH, Lim YW, Mattke S, Damberg C, Shekelle PG. Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med 2008;148:111-23. https://doi.org/10.7326/0003-4819-148-2-200801150-00006.
- Contandriopoulos D, Champagne F, Denis JL. The multiple causal pathways between performance measures’ use and effects. Med Care Res Rev 2014;71:3-20. http://dx.doi.org/10.1177/1077558713496320.
- Canadian Broadcasting Corporation (CBC) . Rate Your Hospital: Online Tool a 1st for Canadians 2013. www.cbc.ca/news/health/rate-your-hospital-online-tool-a-1st-for-canadians-1.1336081 (accessed 3 May 2016).
- Patient Opinion Australia . Patient Opinion n.d. www.patientopinion.org.au (accessed 3 May 2016).
- The Leapfrog Group . Compare Hospitals Now n.d. www.leapfroggroup.org/cp (accessed 3 May 2016).
- Richards SH, Pound P, Dickens A, Greco M, Campbell JL. Exploring users’ experiences of accessing out-of-hours primary medical care services. Qual Saf Health Care 2007;16:469-77. http://dx.doi.org/10.1136/qshc.2006.021501.
- Campbell J, Roland M, Richards S, Dickens A, Greco M, Bower P. Users’ reports and evaluations of out-of-hours health care and the UK national quality requirements: a cross sectional study. Br J Gen Pract 2009;59. http://dx.doi.org/10.3399/bjgp09X394815.
- Understanding What Matters: A Guide to Using Patient Feedback to Transform Services. Leeds: DH; 2009.
- Robert Wood Johnston Foundation . Developing a Public Report for the CAHPS® Clinician & Group Survey: A Decision Guide 2013. www.rwjf.org/content/dam/farm/reports/issue_briefs/2013/rwjf407929 (accessed 3 May 2016).
- University Hospital of South Manchester . UHSM Launches Pioneering Transparency System n.d. www.uhsm.nhs.uk/news/Pages/transp.aspx (accessed 3 May 2016).
- Collins K, O’Cathain A. The continuum of patient satisfaction – from satisfied to very satisfied. Soc Sci Med 2003;57:2465-70. https://doi.org/10.1016/S0277-9536(03)00098-4.
- Hanley B, Bradburn J, Barnes M, Evans C, Goodare H, Kelson M, et al. Involving the Public in NHS, Public Health and Social Care Research: Briefing Notes for Researchers. Eastleigh: INVOLVE Support Unit; 2004.
- Edwards C, Staniszweska S, Crichton N. Investigation of the ways in which patients’ reports of their satisfaction with healthcare are constructed. Sociol Health Illn 2004;26:159-83. http://dx.doi.org/10.1111/j.1467-9566.2004.00385.x.
- Williams B, Coyle J, Healy D. The meaning of patient satisfaction: an explanation of high reported levels. Soc Sci Med 1998;47:1351-9. https://doi.org/10.1016/S0277-9536(98)00213-5.
- Lupton D, Donaldson C, Lloyd P. Caveat emptor or blissful ignorance? Patients and the consumerist ethos. Soc Sci Med 1991;33:559-68. https://doi.org/10.1016/0277-9536(91)90213-V.
- Lupton D. Consumerism, reflexivity and the medical encounter. Soc Sci Med 1997;45:373-81. https://doi.org/10.1016/S0277-9536(96)00353-X.
- Henry SG, Fetters MD. Video elicitation interviews: a qualitative research method for investigating physician–patient interactions. Ann Fam Med 2012;10:118-25. http://dx.doi.org/10.1370/afm.1339.
- Paddison CA, Abel GA, Roland MO, Elliott MN, Lyratzopoulos G, Campbell JL. Drivers of overall satisfaction with primary care: evidence from the English General Practice Patient Survey. Health Expect 2015;18:1081-92. https://doi.org/10.1111/hex.12081.
- Lofland J, Snow D, Anderson L, Lofland LH. Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Belmont, CA: Wadsworth; 2006.
- The NHS Plan: A Plan for Investment, A Plan for Reform. London: DH; 2000.
- Goodyear-Smith F, Buetow S. Power issues in the doctor–patient relationship. Health Care Anal 2001;9:449-62. http://dx.doi.org/10.1023/A:1013812802937.
- Pilnick A, Dingwall R. On the remarkable persistence of asymmetry in doctor/patient interaction: a critical review. Soc Sci Med 2011;72:1374-82. http://dx.doi.org/10.1016/j.socscimed.2011.02.033.
- Kravitz RL, Callahan EJ, Paterniti D, Antonius D, Dunham M, Lewis CE. Prevalence and sources of patients’ unmet expectations for care. Ann Intern Med 1996;125:730-7. https://doi.org/10.7326/0003-4819-125-9-199611010-00004.
- Marple RL, Kroenke K, Lucey CR, Wilder J, Lucas CA. Concerns and expectations in patients presenting with physical complaints. Frequency, physician perceptions and actions, and 2-week outcome. Arch Intern Med 1997;157:1482-8. https://doi.org/10.1001/archinte.1997.00440340122012.
- Coyle J. Exploring the meaning of ‘dissatisfaction’ with health care: the importance of ‘personal identity threat’. Sociol Health Illn 1999;21:95-123. https://doi.org/10.1111/1467-9566.00144.
- Campbell JL, Richards SH, Dickens A, Greco M, Narayanan A, Brearley S. Assessing the professional performance of UK doctors: an evaluation of the utility of the General Medical Council patient and colleague questionnaires. Qual Saf Health Care 2008;17:187-93. http://dx.doi.org/10.1136/qshc.2007.024679.
- Campbell J, Wright C. GMC Multi-Source Feedback Questionnaires. Interpreting and Handling Multisource Feedback Results: Guidance for Appraisers. Exeter: Peninsula College of Medicine and Dentistry; 2012.
- Elliott MN, Edwards C, Angeles J, Hambarsoomians K, Hays RD. Patterns of unit and item nonresponse in the CAHPS Hospital Survey. Health Serv Res 2005;40:2096-119. http://dx.doi.org/10.1111/j.1475-6773.2005.00476.x.
- Ipsos MORI. GP Patient Survey – Technical Annex. 2013–14 Annual Report. London: NHS England; 2014.
- Klein DJ, Elliott MN, Haviland AM, Saliba D, Burkhart Q, Edwards C, et al. Understanding nonresponse to the 2007 Medicare CAHPS survey. Gerontologist 2011;51:843-55. http://dx.doi.org/10.1093/geront/gnr046.
- Martino SC, Weinick RM, Kanouse DE, Brown JA, Haviland AM, Goldstein E, et al. Reporting CAHPS and HEDIS data by race/ethnicity for Medicare beneficiaries. Health Serv Res 2013;48:417-34. http://dx.doi.org/10.1111/j.1475-6773.2012.01452.x.
- O’Malley AJ, Zaslavsky AM, Elliott MN, Zaborski L, Cleary PD. Case-mix adjustment of the CAHPS Hospital Survey. Health Serv Res 2005;40:2162-81. http://dx.doi.org/10.1111/j.1475-6773.2005.00470.x.
- McKinstry B, Walker J, Blaney D, Heaney D, Begg D. Do patients and expert doctors agree on the assessment of consultation skills? A comparison of two patient consultation assessment scales with the video component of the MRCGP. Fam Pract 2004;21:75-80. https://doi.org/10.1093/fampra/cmh116.
- Greco M, Spike N, Powell R, Brownlea A. Assessing communication skills of GP registrars: a comparison of patient and GP examiner ratings. Med Educ 2002;36:366-76. https://doi.org/10.1046/j.1365-2923.2002.01175.x.
- Asprey A, Campbell JL, Newbould J, Cohn S, Carter M, Davey A, et al. Challenges to the credibility of patient feedback in primary healthcare settings: a qualitative study. Br J Gen Pract 2013;63. http://dx.doi.org/10.3399/bjgp13X664252.
- Kenny DA, Veldhuijzen W, Weijden TV, Leblanc A, Lockyer J, Légaré F, et al. Interpersonal perception in the context of doctor–patient relationships: a dyadic analysis of doctor–patient communication. Soc Sci Med 2010;70:763-8. http://dx.doi.org/10.1016/j.socscimed.2009.10.065.
- Roberts MJ, Campbell JL, Richards SH, Wright C. Self-other agreement in multisource feedback: the influence of doctor and rater group characteristics. J Contin Educ Health Prof 2013;33:14-23. http://dx.doi.org/10.1002/chp.21162.
- Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006;296:1094-102. http://dx.doi.org/10.1001/jama.296.9.1094.
- Eva KW, Regehr G. ‘I’ll never play professional football’ and other fallacies of self-assessment. J Contin Educ Health Prof 2008;28:14-9. http://dx.doi.org/10.1002/chp.150.
- Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med 1991;66:762-9. https://doi.org/10.1097/00001888-199112000-00012.
- Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol 1999;77:1121-34. https://doi.org/10.1037/0022-3514.77.6.1121.
- Burt J, Abel G, Elmore N, Campbell J, Roland M, Benson J, et al. Assessing the communication quality of consultations in primary care: initial reliability assessment of the Global Consultation Rating Scale, based on the Calgary–Cambridge Guide to the Medical Interview. BMJ Open 2014;4. https://doi.org/10.1136/bmjopen-2013-004339.
- Kurtz SM, Silverman JD. The Calgary–Cambridge Referenced Observation Guides: an aid to defining the curriculum and organizing the teaching in communication training programmes. Med Educ 1996;30:83-9. https://doi.org/10.1111/j.1365-2923.1996.tb00724.x.
- Kurtz S, Silverman J, Benson J, Draper J. Marrying content and process in clinical method teaching: enhancing the Calgary–Cambridge guides. Acad Med 2003;78:802-9. https://doi.org/10.1097/00001888-200308000-00011.
- Roberts MJ, Campbell JL, Abel GA, Davey AF, Elmore NL, Maramba I, et al. Understanding high and low patient experience scores in primary care: analysis of patients’ survey data for general practices and individual doctors. BMJ 2014;349. http://dx.doi.org/10.1136/bmj.g6034.
- Campbell J, Smith P, Nissen S, Bower P, Elliott M, Roland M. The GP Patient Survey for use in primary care in the National Health Service in the UK – development and psychometric characteristics. BMC Fam Pract 2009;10. http://dx.doi.org/10.1186/1471-2296-10-57.
- Lyratzopoulos G, Elliott M, Barbiere JM, Henderson A, Staetsky L, Paddison C, et al. Understanding ethnic and other socio-demographic differences in patient experience of primary care: evidence from the English General Practice Patient Survey. BMJ Qual Saf 2012;21:21-9. http://dx.doi.org/10.1136/bmjqs-2011-000088.
- Rodriguez HP, Crane PK. Examining multiple sources of differential item functioning on the Clinician & Group CAHPS® survey. Health Serv Res 2011;46:1778-802. http://dx.doi.org/10.1111/j.1475-6773.2011.01299.x.
- Elliott MN, Zaslavsky AM, Goldstein E, Lehrman W, Hambarsoomians K, Beckett MK, et al. Effects of survey mode, patient mix, and nonresponse on CAHPS hospital survey scores. Health Serv Res 2009;44:501-18. http://dx.doi.org/10.1111/j.1475-6773.2008.00914.x.
- Ipsos MORI. GP Patient Survey National Summary Report: July 2015. London: NHS England/Ipsos MORI; 2015.
- Davoll S, Kowalski C, Kuhr K, Ommen O, Ernstmann N, Pfaff H. ‘Tendency to excuse’ and patient satisfaction of those suffering with breast cancer. Int J Public Health 2013;58:385-93. http://dx.doi.org/10.1007/s00038-012-0405-6.
- Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: a systematic review. Adv Health Sci Educ Theory Pract 2009;14:595-621. https://doi.org/10.1007/s10459-007-9090-2.
- Colleague and Patient Feedback for Revalidation. London: GMC; 2015.
- Gillard S, Benson J, Silverman J. Teaching and assessment of explanation and planning in medical schools in the United Kingdom: cross sectional questionnaire survey. Med Teach 2009;31:328-31. http://dx.doi.org/10.1080/01421590801953018.
- Essers G, van Dulmen S, van Es J, van Weel C, van der Vleuten C, Kramer A. Context factors in consultations of general practitioner trainees and their impact on communication assessment in the authentic setting. Patient Educ Couns 2013;93:567-72. http://dx.doi.org/10.1016/j.pec.2013.08.024.
- Essers G, Dielissen P, van Weel C, van der Vleuten C, van Dulmen S, Kramer A. How do trained raters take context factors into account when assessing GP trainee communication performance? An exploratory, qualitative study. Adv Health Sci Educ Theory Pract 2015;20:131-47. http://dx.doi.org/10.1007/s10459-014-9511-y.
- Lyratzopoulos G, Elliott MN, Barbiere JM, Staetsky L, Paddison CA, Campbell J, et al. How can health care organizations be reliably compared?: lessons from a national survey of patient experience. Med Care 2011;49:724-33. http://dx.doi.org/10.1097/MLR.0b013e31821b3482.
- Aboulghate A, Abel G, Elliott MN, Parker RA, Campbell J, Lyratzopoulos G, et al. Do English patients want continuity of care, and do they receive it?. Br J Gen Pract 2012;62:e567-75. http://dx.doi.org/10.3399/bjgp12X653624.
- Office for National Statistics . Harmonised Concepts and Questions for Social Data Sources. Primary Principles: Ethnic Group 2015. www.ons.gov.uk/ons/guide-method/harmonisation/primary-set-of-harmonised-concepts-and-questions/ethnic-group.pdf (accessed 15 December 2016).
- Freeman GK. Continuity of Care Toolkit: Helping Clinicians and Practices Maximise Relationship Continuity. London: Royal College of General Practitioners; 2013.
- Roland M, Paddison C. Better management of patients with multimorbidity. BMJ 2013;346. http://dx.doi.org/10.1136/bmj.f2510.
- Campbell JL, Ramsay J, Green J. Age, gender, socioeconomic, and ethnic differences in patients’ assessments of primary health care. Qual Health Care 2001;10:90-5. https://doi.org/10.1136/qhc.10.2.90.
- Venn S, Fone DL. Assessing the influence of socio-demographic factors and health status on expression of satisfaction with GP services. Clin Govern Int J 2005;10:118-25. https://doi.org/10.1108/14777270510594290.
- Croker JE, Swancutt DR, Roberts MJ, Abel GA, Roland M, Campbell JL. Factors affecting patients’ trust and confidence in GPs: evidence from the English national GP patient survey. BMJ Open 2013;3. http://dx.doi.org/10.1136/bmjopen-2013-002762.
- Dyson B. Improving General Practice; a Call to Action – Phase 1 Report. NHS England; 2014.
- Weech-Maldonado R, Elliott MN, Adams JL, Haviland AM, Klein DJ, Hambarsoomian K, et al. Do racial/ethnic disparities in quality and patient experience within Medicare plans generalise across measures and racial/ethnic groups?. Health Serv Res 2015;50:1829-49. https://doi.org/10.1111/1475-6773.12297.
- Price RA, Haviland AM, Hambarsoomian K, Dembosky JW, Gaillot S, Weech-Maldonado R, et al. Do Experiences with Medicare managed care vary according to the proportion of same-race/ethnicity/language individuals enrolled in one’s contract?. Health Serv Res 2015;50:1649-87. http://dx.doi.org/10.1111/1475-6773.12292.
- Goldstein E, Elliott MN, Lehrman WG, Hambarsoomian K, Giordano LA. Racial/ethnic differences in patients’ perceptions of inpatient care using the HCAHPS survey. Med Care Res Rev 2010;67:74-92. http://dx.doi.org/10.1177/1077558709341066.
- Weech-Maldonado R, Morales LS, Elliott M, Spritzer K, Marshall G, Hays RD. Race/ethnicity, language, and patients’ assessments of care in Medicaid managed care. Health Serv Res 2003;38:789-808. https://doi.org/10.1111/1475-6773.00147.
- Weinick R, Haviland A, Hambarsoomian K, Elliott MN. Does the racial/ethnic composition of Medicare Advantage plans reflect their areas of operation?. Health Serv Res 2014;49:526-45. http://dx.doi.org/10.1111/1475-6773.12100.
- The English Indices of Deprivation 2010. London: Department for Communities and Local Government; 2010.
- Burt J, Lloyd C, Campbell J, Roland M, Abel G. Variations in GP–patient communication by ethnicity, age, and gender: evidence from a national primary care patient survey. Br J Gen Pract 2016;66:e47-52. http://dx.doi.org/10.3399/bjgp15X687637.
- Lyratzopoulos G, Neal RD, Barbiere JM, Rubin GP, Abel GA. Variation in number of general practitioner consultations before hospital referral for cancer: findings from the 2010 National Cancer Patient Experience Survey in England. Lancet Oncol 2012;13:353-65. http://dx.doi.org/10.1016/S1470-2045(12)70041-4.
- EuroQol Group . EuroQol – a new facility for the measurement of health-related quality of life. Health Polic 1990;16:199-208. https://doi.org/10.1016/0168-8510(90)90421-9.
- Setodji CM, Elliott MN, Abel G, Burt J, Roland M, Campbell J. Evaluating differential item functioning in the English General Practice Patient Survey: comparison of South Asian and white British subgroups. Med Care 2015;53:809-17. http://dx.doi.org/10.1097/MLR.0000000000000397.
- Rasch G. Probabilistic Models for Some Intelligence and Attainment Tests. Copenhagen: Danmarks Paedogogiske Institut; 1960.
- Hambleton RK, Swaminathan H, Rogers H. Fundamentals of Item Response Theory. Newbury Park, CA: Sage Press; 1991.
- Lord FM. Applications of Item Response Theory to Practical Testing Problem. Hillsdale, NJ: Lawrence Erlbaum; 1980.
- Thissen D, Steinberg L, Wainer H, Holland P, Wainer H. Differential Item Functioning. Hillsdale, NJ: Erlbaum; 1993.
- Holland P, Dorans N, Pommerich M, Holland P. Linking and Aligning Scores and Scales. New York, NY: Springer; 2007.
- Kopf J, Zeileis A, Strobl C. A framework for anchor methods and an iterative forward approach for DIF detection. Appl Psychol Meas 2015;39:83-103. https://doi.org/10.1177/0146621614544195.
- Cai L. flexMIRT: flexible multilevel item factor analysis and test scoring. Seattle, WA: Vector Psychometric Group, LLC; 2012.
- Cudeck R, Henly SJ. Model selection in covariance structures analysis and the ‘problem’ of sample size: a clarification. Psychol Bull 1991;109:512-19. https://doi.org/10.1037/0033-2909.109.3.512.
- Samejima F. Estimation of Latent Ability Using a Response Pattern of Graded Scores. Richmond, VA: Psychometric Society; 1969.
- Hays RD, Morales LS, Reise SP. Item response theory and health outcomes measurement in the 21st century. Med Care 2000;38:28-42. https://doi.org/10.1097/00005650-200009002-00007.
- Public Health England . Metadata for National General Practice Profile n.d. www.apho.org.uk/resource/item.aspx?RID=95729 (accessed 3 May 2016).
- Office for National Statistics . 2011 Rural Urban Classification n.d. http://webarchive.nationalarchives.gov.uk/20160105160709/http://www.ons.gov.uk/ons/guide-method/geography/products/area-classifications/2011-rural-urban/index.html (accessed 13 December 2016).
- Ahmed F, Abel GA, Lloyd CE, Burt J, Roland M. Does the availability of a South Asian language in practices improve reports of doctor–patient communication from South Asian patients? Cross sectional analysis of a national patient survey in English general practices. BMC Fam Pract 2015;16. http://dx.doi.org/10.1186/s12875-015-0270-5.
- Language in England and Wales, 2011. London; 2014.
- Ahmed N. Language, gender and citizenship: obstacles in the path to learning English for Bangladeshi women in London’s East End. Sociol Res Online 2008;13. https://doi.org/10.5153/sro.1813.
- Meischke HW, Calhoun RE, Yip MP, Tu SP, Painter IS. The effect of language barriers on dispatching EMS response. Prehosp Emerg Care 2013;17:475-80. http://dx.doi.org/10.3109/10903127.2013.811565.
- John-Baptiste A, Naglie G, Tomlinson G, Alibhai SM, Etchells E, Cheung A, et al. The effect of English language proficiency on length of stay and in-hospital mortality. J Gen Intern Med 2004;19:221-8. https://doi.org/10.1111/j.1525-1497.2004.21205.x.
- Sarver J, Baker DW. Effect of language barriers on follow-up appointments after an emergency department visit. J Gen Intern Med 2000;15:256-64. https://doi.org/10.1111/j.1525-1497.2000.06469.x.
- Parsons JA, Baker NA, Smith-Gorvie T, Hudak PL. To ‘get by’ or ‘get help’? A qualitative study of physicians’ challenges and dilemmas when patients have limited English proficiency. BMJ Open 2014;4. http://dx.doi.org/10.1136/bmjopen-2013-004613.
- Hunt LM, Schneider S, Comer B. Should ‘acculturation’ be a variable in health research? A critical review of research on US Hispanics. Soc Sci Med 2004;59:973-86. http://dx.doi.org/10.1016/j.socscimed.2003.12.009.
- Hasnain M, Schwartz A, Girotti J, Bixby A, Rivera L. UIC Experiences of Care Project Group . Differences in patient-reported experiences of care by race and acculturation status. J Immigr Minor Health 2013;15:517-24. http://dx.doi.org/10.1007/s10903-012-9728-x.
- Lopez-Class M, Castro FG, Ramirez AG. Conceptions of acculturation: a review and statement of critical issues. Soc Sci Med 2011;72:1555-62. http://dx.doi.org/10.1016/j.socscimed.2011.03.011.
- Zambrana RE, Carter-Pokras O. Role of acculturation research in advancing science and practice in reducing health care disparities among Latinos. Am J Public Health 2010;100:18-23. http://dx.doi.org/10.2105/AJPH.2008.138826.
- Smith WR, Betancourt JR, Wynia MK, Bussey-Jones J, Stone VE, Phillips CO, et al. Recommendations for teaching about racial and ethnic disparities in health and health care. Ann Intern Med 2007;147:654-65. https://doi.org/10.7326/0003-4819-147-9-200711060-00010.
- Betancourt JR. Cross-cultural medical education: conceptual approaches and frameworks for evaluation. Acad Med 2003;78:560-9. https://doi.org/10.1097/00001888-200306000-00004.
- Burgess DJ. Addressing racial healthcare disparities: how can we shift the focus from patients to providers?. J Gen Intern Med 2011;26:828-30. http://dx.doi.org/10.1007/s11606-011-1748-z.
- Torres S. Expanding the gerontological imagination on ethnicity: conceptual and theoretical perspectives. Ageing Soc 2015;35:935-60. https://doi.org/10.1017/S0144686X14001330.
- Groves RM, Peytcheva E. The impact of nonreponse rates on nonresponse bias. Public Opin Q 2008;72:167-89. https://doi.org/10.1093/poq/nfn011.
- Karliner LS, Jacobs EA, Chen AH, Mutha S. Do professional interpreters improve clinical care for patients with limited English proficiency? A systematic review of the literature. Health Serv Res 2007;42:727-54. http://dx.doi.org/10.1111/j.1475-6773.2006.00629.x.
- Greenhalgh T. Health literacy: towards system level solutions. BMJ 2015;350. http://dx.doi.org/10.1136/bmj.h1026.
- Burt J, Abel G, Elmore N, Lloyd C, Benson J, Sarson L, et al. Understanding negative feedback from South Asian patients: experimental vignette study. BMJ Open 2016;6. http://dx.doi.org/10.1136/bmjopen-2016-011256.
- Brady HE. The perils of survey research: inter-personally incomparable responses. Polit Methodol 1985;11:269-91.
- Nanchahal K, Mangtani P, Alston M, dos Santos Silva I. Development and validation of a computerized South Asian Names and Group Recognition Algorithm (SANGRA) for use in British health – related studies. J Public Health 2001;23:278-85. https://doi.org/10.1093/pubmed/23.4.278.
- Hillen MA, van Vliet LM, de Haes HC, Smets EM. Developing and administering scripted video vignettes for experimental research of patient–provider communication. Patient Educ Couns 2013;91:295-309. http://dx.doi.org/10.1016/j.pec.2013.01.020.
- Office for National Statistics . 2011 Census: Detailed Characteristics on Approximated Social Grade for Middle Layer Super Output Areas and 2011 Census Merged Wards in England and Wales 2014. www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/datasets/2011censusdetailedcharacteristicsonapproximatedsocialgradeformiddlelayersuperoutputareasand2011censusmergedwardsinenglandandwales (accessed 6 May 2016).
- Bécares L, Das-Munshi J. Ethnic density, health care seeking behaviour and expected discrimination from health services among ethnic minority people in England. Health Place 2013;22:48-55. http://dx.doi.org/10.1016/j.healthplace.2013.03.005.
- van Vliet LM, van der Wall E, Albada A, Spreeuwenberg PM, Verheul W, Bensing JM. The validity of using analogue patients in practitioner–patient communication research: systematic review and meta-analysis. J Gen Intern Med 2012;27:1528-43. http://dx.doi.org/10.1007/s11606-012-2111-8.
- NHS England . GP Patient Survey: Survey and Report n.d. https://gp-patient.co.uk/surveys-and-reports (accessed 3 May 2016).
- Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. Washington DC: National Academic Press; 2003.
- Boiko O, Campbell JL, Elmore N, Davey AF, Roland M, Burt J. The role of patient experience surveys in quality assurance and improvement: a focus group study in English general practice. Health Expect 2015;18:1982-94. https://doi.org/10.1111/hex.12298.
- Edwards A, Evans R, White P, Elwyn G. Experiencing patient-experience surveys: a qualitative study of the accounts of GPs. Br J Gen Pract 2011;61:e157-66. http://dx.doi.org/10.3399/bjgp11X567072.
- van der Vleuten CP. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract 1996;1:41-67. http://dx.doi.org/10.1007/BF00596229.
- Developing and Maintaining an Assessment System – a PMETB Guide to Good Practice. London: PMETB; 2007.
- Ritchie J SL, Bryman A, Burgess RG. Qualitative Data Analysis for Applied Policy Research. Analysing Qualitative Data. London: Routledge; 1994.
- Reinders ME, Ryan BL, Blankenstein AH, van der Horst HE, Stewart MA, van Marwijk HW. The effect of patient feedback on physicians’ consultation skills: a systematic review. Acad Med 2011;86:1426-36. http://dx.doi.org/10.1097/ACM.0b013e3182312162.
- Sargeant J, Mann K, Ferrier S. Exploring family physicians’ reactions to multisource feedback: perceptions of credibility and usefulness. Med Educ 2005;39:497-504. http://dx.doi.org/10.1111/j.1365-2929.2005.02124.x.
- Davies EA, Meterko MM, Charns MP, Seibert ME, Cleary PD. Factors affecting the use of patient survey data for quality improvement in the Veterans Health Administration. BMC Health Serv Res 2011;11. http://dx.doi.org/10.1186/1472-6963-11-334.
- Vingerhoets E, Wensing M, Grol R. Feedback of patients’ evaluations of general practice care: a randomised trial. Qual Health Care 2001;10:224-8. https://doi.org/10.1136/qhc.0100224.
- Wensing M, Vingerhoets E, Grol R. Feedback based on patient evaluations: a tool for quality improvement?. Patient Educ Couns 2003;51:149-53. https://doi.org/10.1016/S0738-3991(02)00199-4.
- Reeves R, Seccombe I. Do patient surveys work? The influence of a national survey programme on local quality-improvement initiatives. Qual Saf Health Care 2008;17:437-41. http://dx.doi.org/10.1136/qshc.2007.022749.
- Boyer L, Francois P, Doutre E, Weil G, Labarere J. Perception and use of the results of patient satisfaction surveys by care providers in a French teaching hospital. Int J Qual Health Care 2006;18:359-64. http://dx.doi.org/10.1093/intqhc/mzl029.
- Reinders ME, Blankenstein AH, van der Horst HE, Knol DL, Schoonheim PL, van Marwijk HW. Does patient feedback improve the consultation skills of general practice trainees? A controlled trial. Med Educ 2010;44:156-64. http://dx.doi.org/10.1111/j.1365-2923.2009.03569.x.
- Reeves R, West E, Barron D. Facilitated patient experience feedback can improve nursing care: a pilot study for a Phase III cluster randomised controlled trial. BMC Health Serv Res 2013;13. http://dx.doi.org/10.1186/1472-6963-13-259.
- Farrington C, Burt J, Boiko O, Campbell J, Roland M. Doctors’ engagements with patient experience surveys in primary and secondary care: a qualitative study [published online ahead of print 28 April 2016]. Health Expect 2016. http://dx.doi.org/10.1111/hex.12465.
- The NHS Outcomes Framework 2013–14. London: DH; 2012.
- Fitzpatrick R, Graham C, Gibbons E, King J, Flott K, Jenkinson C. Development of New Models for Collection and Use of Patient Experience Information in the NHS. Oxford: Picker Institute Europe; 2014.
- General Medical Council . Revalidation 2017. www.gmc-uk.org/doctors/revalidation.asp (accessed 5 January 2017).
- Williams TR. A critique of some assumptions of social survey research. Public Opin Q 1959;23:55-62. https://doi.org/10.1086/266845.
- Elbaum B. Challenges in interpreting accountability results for schools’ facilitation of parent involvement under IDEA. J Disabil Policy Stud 2014;24:206-17. https://doi.org/10.1177/1044207312461947.
- Hibbard JH, Greene J, Daniel D. What is quality anyway? Performance reports that clearly communicate to consumers the meaning of quality of care. Med Care Res Rev 2010;67:275-93. http://dx.doi.org/10.1177/1077558709356300.
- Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006;3:77-101. https://doi.org/10.1191/1478088706qp063oa.
- Oxford English Dictionary. Oxford: Oxford University Press; 2012.
- May C, Rapley T, Moreira T, Finch T, Heaven B. Technogovernance: evidence, subjectivity, and the clinical encounter in primary care medicine. Soc Sci Med 2006;62:1022-30. http://dx.doi.org/10.1016/j.socscimed.2005.07.003.
- Boyd D, Molloy E. Feedback in Higher and Professional Education: Understanding it and Doing it Well. Abingdon: Routledge; 2013.
- Iacobucci G. General practices could be subject to ‘Ofsted style’ ratings within two years. BMJ 2013;346. http://dx.doi.org/10.1136/bmj.f1941.
- Compare n.d. www.locallyhealthy.co.uk/perf (accessed 23 January 2016).
- Wright C, Richards SH, Hill JJ, Roberts MJ, Norman GR, Greco M, et al. Multisource feedback in evaluating the performance of doctors: the example of the UK General Medical Council patient and colleague questionnaires. Acad Med 2012;87:1668-78. http://dx.doi.org/10.1097/ACM.0b013e3182724cc0.
- Roland M, Roberts M, Rhenius V, Campbell J. GPAQ-R: development and psychometric properties of a version of the general practice assessment questionnaire for use for revalidation by general practitioners in the UK. BMC Fam Pract 2013;14. http://dx.doi.org/10.1186/1471-2296-14-160.
- Robinson GK. That BLUP is a good thing: the estimation of random effects. Stat Sci 1991;6:15-32. https://doi.org/10.1214/ss/1177011926.
- Downing SM. Reliability: on the reproducibility of assessment data. Med Educ 2004;38:1006-12. http://dx.doi.org/10.1111/j.1365-2929.2004.01932.x.
- Lockyer J. Multisource feedback in the assessment of physician competencies. J Contin Educ Health Prof 2003;23:4-12. http://dx.doi.org/10.1002/chp.1340230103.
- Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat 1979;6:65-70.
- Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 1968;70:213-20. https://doi.org/10.1037/h0026256.
- Safran DG, Karp M, Coltin K, Chang H, Li A, Ogren J, et al. Measuring patients’ experiences with individual primary care physicians. Results of a statewide demonstration project. J Gen Intern Med 2006;21:13-21. http://dx.doi.org/10.1111/j.1525-1497.2005.00311.x.
- Haggerty JL, Pineault R, Beaulieu MD, Brunelle Y, Gauthier J, Goulet F, et al. Practice features associated with patient-reported accessibility, continuity, and coordination of primary health care. Ann Fam Med 2008;6:116-23. http://dx.doi.org/10.1370/afm.802.
- Mazor KM, Clauser BE, Field T, Yood RA, Gurwitz JH. A demonstration of the impact of response bias on the results of patient satisfaction surveys. Health Serv Res 2002;37:1403-17. https://doi.org/10.1111/1475-6773.11194.
- Ramsay J, Campbell JL, Schroter S, Green J, Roland M. The General Practice Assessment Survey (GPAS): tests of data quality and measurement properties. Fam Pract 2000;17:372-9. https://doi.org/10.1093/fampra/17.5.372.
- Wright C, Davey A, Elmore N, Carter M, Mounce L, Wilson E, et al. Patients’ use and views of real-time feedback technology in general practice [published online ahead of print 28 April 2016]. Health Expect 2016. http://dx.doi.org/10.1111/hex.12469.
- Carter M, Davey A, Wright C, Elmore N, Newbould J, Roland M, et al. Capturing patient experience: a qualitative study of implementing real-time feedback in primary care. Br J Gen Pract 2016;66:e786-93. http://dx.doi.org/10.3399/bjgp16X687085.
- Haskard KB, Williams SL, DiMatteo MR, Rosenthal R, White MK, Goldstein MG. Physician and patient communication training in primary care: effects on participation and satisfaction. Health Psychol 2008;27:513-22. http://dx.doi.org/10.1037/0278-6133.27.5.513.
- Kirkpatrick DL. Techniques for evaluating training programs. J AST 1959;75:3-9.
- Kirkpatrick DL. Techniques for evaluating training programs: part 2 – learning. J AST 1959;13:21-6.
- Kirkpatrick DL. Techniques for evaluating training programs: part 3 – behavior. J AST 1960;14:13-8.
- Kirkpatrick DL. Techniques for evaluating training programs: part 4 – results. J AST 1960;14:28-32.
- Byrne P, Long B. Doctors Talking to Patients. London: HMSO; 1976.
- Checkland K. Understanding general practice: a conceptual framework developed from case studies in the UK NHS. Br J Gen Pract 2007;57:56-63.
- Armstrong D, Ogden J. The role of etiquette and experimentation in explaining how doctors change behaviour: a qualitative study. Sociol Health Illn 2006;28:951-68. http://dx.doi.org/10.1111/j.1467-9566.2006.00514.x.
- King EB, de Chermont K, West M, Dawson JF, Hebl MR. How innovation can alleviate negative consequences of demanding work contexts: the influence of climate for innovation on organizational outcomes. J Occup Organ Psychol 2007;80:631-45. https://doi.org/10.1348/096317906X171145.
- West MA, Borrill C, Cox J, King J, Hutchinson A, McAvoy P. Understanding Doctor’s Performance. Oxford: Radcliffe Publishing; 2006.
- Anderson NR, West MA. Measuring climate for work group innovation: development and validation of the team climate inventory. J Organ Behav 1998;19:235-58. https://doi.org/10.1002/(SICI)1099-1379(199805)19:3<235::AID-JOB837>3.0.CO;2-C.
- Schneider B, Schneider B. Organizational Climate and Culture. San Francisco, CA: Jossey Bass; 1990.
- Anderson NR, West MA. The team climate inventory: development of the TCI and its application in teambuilding for innovativeness. Eur J Work Organ Psychol 1996;5:53-66. https://doi.org/10.1080/13594329608414840.
- Bower P, Campbell S, Bojke C, Sibbald B. Team structure, team climate and the quality of care in primary care: an observational study. Qual Saf Health Care 2003;12:273-9. https://doi.org/10.1136/qhc.12.4.273.
- Campbell SM, Hann M, Hacker J, Burns C, Oliver D, Thapar A, et al. Identifying predictors of high quality care in English general practice: observational study. BMJ 2001;323:784-7. https://doi.org/10.1136/bmj.323.7316.784.
- Proudfoot J, Jayasinghe UW, Holton C, Grimm J, Bubner T, Amoroso C, et al. Team climate for innovation: what difference does it make in general practice?. Int J Qual Health Care 2007;19:164-9. http://dx.doi.org/10.1093/intqhc/mzm005.
- Kivimäki M, Vanhala A, Pentti J, Länsisalmi H, Virtanen M, Elovainio M, et al. Team climate, intention to leave and turnover among hospital employees: prospective cohort study. BMC Health Serv Res 2007;7. http://dx.doi.org/10.1186/1472-6963-7-170.
- Goh TT, Eccles MP, Steen N. Factors predicting team climate, and its relationship with quality of care in general practice. BMC Health Serv Res 2009;9. http://dx.doi.org/10.1186/1472-6963-9-138.
- Hann M, Bower P, Campbell S, Marshall M, Reeves D. The association between culture, climate and quality of care in primary health care teams. Fam Pract 2007;24:323-9. http://dx.doi.org/10.1093/fampra/cmm020.
- Brown H, Davidson D, Ellins J. NHS West Midlands Investing for Health: Real-time Patient Feedback Project – Final Report. Birmingham: University of Birmingham and NHS West Midlands, Health Services Management Centre; 2009.
- Dirocco DN, Day SC. Obtaining patient feedback at point of service using electronic kiosks. Am J Manag Care 2011;17.
- Greenhalgh T, Swinglehurst D. Studying technology use as social practice: the untapped potential of ethnography. BMC Med 2011;9. http://dx.doi.org/10.1186/1741-7015-9-45.
- Bentley ME, Boot MT, Gittelsohn J, Stallings RY. The Use of Structured Observations in the Study of Health Behaviour (Occasional Paper 277). The Hague: IRC International Water and Sanitation Centre/London School of Hygiene and Tropical Medicine; 1994.
- Srigley JA, Furness CD, Baker GR, Gardam M. Quantification of the Hawthorne effect in hand hygiene compliance monitoring using an electronic monitoring system: a retrospective cohort study. BMJ Qual Saf 2014;23:974-80. http://dx.doi.org/10.1136/bmjqs-2014-003080.
- Curtis L. Unit Costs of Health and Social Care 2014. Canterbury: PSSRU, University of Kent; 2014.
- May C, Finch T. Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology 2009;43:535-54. https://doi.org/10.1177/0038038509103208.
- Public Health England . National General Practice Profile n.d. https://fingertips.phe.org.uk/profile/general-practice/data (accessed 6 January 2017).
- Wofford JL, Campos CL, Jones RE, Stevens SF. Real-time patient survey data during routine clinical activities for rapid-cycle quality improvement. JMIR Med Inform 2015;3. http://dx.doi.org/10.2196/medinform.3697.
- Deming WE. Out of the Crisis. Cambridge, MA: MIT Press; 2000.
- Warren FC, Abel G, Lyratzopoulos G, Elliott MN, Richards S, Barry HE, et al. Characteristics of service users and provider organisations associated with experience of out of hours general practitioner care in England: population based cross sectional postal questionnaire survey. BMJ 2015;350. http://dx.doi.org/10.1136/bmj.h2040.
- Out-of-Hours GP Services in England. London: Controller and Auditor General; 2014.
- Colin-Thomé D, Field S. General Practice Out-of-Hours Services: Project to Consider and Assess Current Arrangement. London: Care Quality Commission; 2010.
- Not Just a Matter of Time. A Review of Urgent and Emergency Care Services in England. London: Healthcare Commission; 2008.
- Cosford PA, Thomas JM. Safer out of hours primary care. BMJ 2010;340. http://dx.doi.org/10.1136/bmj.c3194.
- Egbunike JN, Shaw C, Bale S, Elwyn G, Edwards A. Understanding patient experience of out-of-hours general practitioner services in South Wales: a qualitative study. Emerg Med J 2008;25:649-54. http://dx.doi.org/10.1136/emj.2007.052001.
- Poole R, Gamper A, Porter A, Egbunike J, Edwards A. Exploring patients’ self-reported experiences of out-of-hours primary care and their suggestions for improvement: a qualitative study. Fam Pract 2011;28:210-19. http://dx.doi.org/10.1093/fampra/cmq090.
- O’Cathain A, Knowles E, Turner J, Nicholl J. Acceptability of NHS 111 the telephone service for urgent health care: cross sectional postal survey of users’ views. Fam Pract 2014;31:193-200. http://dx.doi.org/10.1093/fampra/cmt078.
- A Fresh Start for the Regulation and Inspection of GP Practices and GP Out-of-Hours Service. Newcastle upon Tyne: Care Quality Commission; 2013.
- Our New Approach to the Inspection of NHS GP Out-of-Hours Services: Findings from the First Comprehensive Inspections. Newcastle upon Tyne: Care Quality Commission; 2014.
- Primary Medical Care Functions Delegated to Clinical Commissioning Groups: Guidance. London: NHS England; 2014.
- Garratt AM, Danielsen K, Hunskaar S. Patient satisfaction questionnaires for primary care out-of-hours services: a systematic review. Br J Gen Pract 2007;57:741-7.
- Paddison C, Elliott M, Parker R, Staetsky L, Lyratzopoulos G, Campbell JL, et al. Should measures of patient experience in primary care be adjusted for case mix? Evidence from the English General Practice Patient Survey. BMJ Qual Saf 2012;21:634-40. http://dx.doi.org/10.1136/bmjqs-2011-000737.
- Overeem K, Faber MJ, Arah OA, Elwyn G, Lombarts KM, Wollersheim HC, et al. Doctor performance assessment in daily practise: does it help doctors or not? A systematic review. Med Educ 2007;41:1039-49. http://dx.doi.org/10.1111/j.1365-2923.2007.02897.x.
- Allemann Iseli M, Kunz R, Blozik E. Instruments to assess patient satisfaction after teleconsultation and triage: a systematic review. Patient Prefer Adherence 2014;8:893-907. http://dx.doi.org/10.2147/PPA.S56160.
- Ipsos MORI. GP Patient Survey National Summary Report: June 2013. London: NHS England/Ipsos MORI; n.d.
- Office for National Statistics . Quality and Methodology Information for Population Estimates by Ethnic Group (PEEG 2011. www.ons.gov.uk/ons/taxonomy/index.html?nscl=Population+Estimates+by+Ethnic+Group (accessed 3 May 2016).
- McLennan D, Barnes H, Noble M, Davies J, Garratt E, Dibben C. English Indices of Deprivation 2010 2011. www.gov.uk/government/uploads/system/uploads/attachment_data/file/6320/1870718.pdf (accessed 3 May 2016).
- Jha AK, Orav EJ, Zheng J, Epstein AM. Patients’ perception of hospital care in the United States. N Engl J Med 2008;359:1921-31. http://dx.doi.org/10.1056/NEJMsa0804116.
- Lehrman WG, Elliott MN, Goldstein E, Beckett MK, Klein DJ, Giordano LA. Characteristics of hospitals demonstrating superior performance in patient experience and clinical process measures of care. Med Care Res Rev 2010;67:38-55. http://dx.doi.org/10.1177/1077558709341323.
- McKinley RK, Stevenson K, Adams S, Manku-Scott TK. Meeting patient expectations of care: the major determinant of satisfaction with out-of-hours primary medical care?. Fam Pract 2002;19:333-8. https://doi.org/10.1093/fampra/19.4.333.
- Richards SH, Coast J, Peters TJ. Patient-reported use of health service resources compared with information from health providers. Health Soc Care Community 2003;11:510-18. https://doi.org/10.1046/j.1365-2524.2003.00457.x.
- Groves RM. Nonresponse rates and nonresponse bias in household surveys. Public Opin Q 2006;70:646-75. https://doi.org/10.1093/poq/nfl033.
- van Uden C, Ament A, Hobma S, Zwietering P, Crebolder H. Patient satisfaction with out-of-hours primary care in the Netherlands. BMC Health Serv Res 2005;5. https://doi.org/10.1186/1472-6963-5-6.
- Willis G. Cognitive Interviewing: a Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications; 2005.
- Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modeling 1999;6:1-55. https://doi.org/10.1080/10705519909540118.
- Bentler PM. Comparative fit indexes in structural models. Psychol Bull 1990;107:238-46. https://doi.org/10.1037/0033-2909.107.2.238.
- Russell DW. In search of underlying dimensions: The use (and abuse) of factor analysis in Personality and Social Psychology Bulletin. Pers Soc Psychol Bull 2002;28:1629-46. https://doi.org/10.1177/014616702237645.
- Kolenikov S, Angeles G. The Use of Discrete Data in PCA: Theory, Simulations, and Applications to Socioeconomic Indices. Chapel Hill, NC: Carolina Population Center MEASURE Evaluation, University of North Carolina at Chapel Hill; 2004.
- Freeman GK, Horder JP, Howie JG, Hungin AP, Hill AP, Shah NC, et al. Evolving general practice consultation in Britain: issues of length and context. BMJ 2002;324:880-2. https://doi.org/10.1136/bmj.324.7342.880.
- Riiskjær E, Ammentorp J, Kofoed PE. The value of open-ended questions in surveys on patient experience: number of comments and perceived usefulness from a hospital perspective. Int J Qual Health Care 2012;24:509-16. http://dx.doi.org/10.1093/intqhc/mzs039.
- Measuring Patient Experience. Evidence Scan. London: Health Foundation; 2013.
- Jamtvedt G, Young JM, Kristoffersen DT, Thomson O’Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2003;2. https://doi.org/10.1002/14651858.cd000259.
- Nielsen J, Riiskjaer E. From patient surveys to organizational change: rational change processes and institutional forces. J Change Manag 2013;13:179-205. https://doi.org/10.1080/14697017.2012.745584.
- Luxford K, Safran DG, Delbanco T. Promoting patient-centered care: a qualitative study of facilitators and barriers in healthcare organizations with a reputation for improving the patient experience. Int J Qual Health Care 2011;23:510-15. http://dx.doi.org/10.1093/intqhc/mzr024.
- Manary MP, Boulding W, Staelin R, Glickman SW. The patient experience and health outcomes. N Engl J Med 2013;368:201-3. http://dx.doi.org/10.1056/NEJMp1211775.
- Llanwarne NR, Abel GA, Elliott MN, Paddison CA, Lyratzopoulos G, Campbell JL, et al. Relationship between clinical quality and patient experience: analysis of data from the English Quality and Outcomes Framework and the National GP Patient Survey. Ann Fam Med 2013;11:467-72. http://dx.doi.org/10.1370/afm.1514.
- Chang JT, Hays RD, Shekelle PG, MacLean CH, Solomon DH, Reuben DB, et al. Patients’ global ratings of their health care are not associated with the technical quality of their care. Ann Intern Med 2006;144:665-72. https://doi.org/10.7326/0003-4819-144-9-200605020-00010.
- Roland M, Elliott M, Lyratzopoulos G, Barbiere J, Parker RA, Smith P, et al. Reliability of patient responses in pay for performance schemes: analysis of national General Practitioner Patient Survey data in England. BMJ 2009;339. http://dx.doi.org/10.1136/bmj.b3851.
- National Quality Requirements in the Delivery of Out-of-Hours Service. London: DH; 2006.
- Tsianakas V, Maben J, Wiseman T, Robert G, Richardson A, Madden P, et al. Using patients’ experiences to identify priorities for quality improvement in breast cancer care: patient narratives, surveys or both?. BMC Health Serv Res 2012;12. http://dx.doi.org/10.1186/1472-6963-12-271.
- Greaves F, Pape UJ, King D, Darzi A, Majeed A, Wachter RM, et al. Associations between Internet-based patient ratings and conventional surveys of patient experience in the English NHS: an observational study. BMJ Qual Saf 2012;21:600-5. http://dx.doi.org/10.1136/bmjqs-2012-000906.
- Greaves F, Millett C, Nuki P. England’s experience incorporating ‘anecdotal’ reports from consumers into their national reporting system: lessons for the United States of what to do or not to do?. Med Care Res Rev 2014;71:65-80. http://dx.doi.org/10.1177/1077558714535470.
- Greaves F, Laverty AA, Pape U, Ratneswaren A, Majeed A, Millett C. Performance of new alternative providers of primary care services in England: an observational study. J R Soc Med 2015;108:171-83. http://dx.doi.org/10.1177/0141076815583303.
- Department of Health . The Operating Framework for the NHS in England 2012 13 2011. www.gov.uk/government/uploads/system/uploads/attachment_data/file/216590/dh_131428.pdf (accessed 6 May 2016).
- The Picker Institute . Using Patient Feedback 2009. www.nhssurveys.org/Filestore/documents/QIFull.pdf (accessed 6 May 2016).
- Collins D. Pretesting survey instruments: an overview of cognitive methods. Qual Life Res 2003;12:229-38. https://doi.org/10.1023/A:1023254226592.
- Tucker LR, Lewis C. The reliability coefficient for maximum likelihood factor analysis. Psychometrika 1973;38:1-10. https://doi.org/10.1007/BF02291170.
Appendix 1 Global Consultation Rating Scale
Appendix 2 Chapter 9: patient questionnaire
Appendix 3 Chapter 9: calculation of reliability
Reliability of doctors’ mean scores
Unit-level reliability is defined as the proportion of variance in reported unit sample means (e.g. practice means or physician means) attributable to true variation between units. 30 Owing to the nesting of doctors within practices, the reliability of the mean score for a randomly chosen doctor is calculated from the variance components of the three-level hierarchical model using the formula:
where var(P) = variance due to practices; var(D) = variance due to doctors; var(E) = variance due to patients and random error; and N = number of patient scores per doctor.
By inserting the values of the variance components and manipulating this formula, the number of patient scores per doctor that are required to achieve a given level of reliability can be calculated. These calculations can be performed using variance components obtained from a model with no fixed effects where the resulting reliability is that pertaining to the raw mean scores. Alternatively, we can take into account the fact that some of the variation (at all levels) may occur due to different patient demographics and calculate a reliability of an adjusted mean score such as those shown in Figures 20 and 21. Both raw and adjusted reliabilities estimated from our data are shown in Table 29.
Appendix 4 Cochrane Central Register of Controlled Trials (CENTRAL) search strategy
#1 MeSH descriptor Family Practice, this term only
#2 MeSH descriptor Primary Health Care, this term only
#3 MeSH descriptor Community Health Services explode all trees
#4 MeSH descriptor Physicians, Family, this term only
#5 MeSH descriptor Comprehensive Health Care, this term only
#6 MeSH descriptor Patient Care Team, this term only
#7 MeSH descriptor Ambulatory Care, this term only
#8 “shared care”:ti,ab
#9 “integrated care”:ti,ab
#10 “family practice”:ti,ab
#11 “family practitioner”:ti,ab
#12 “general practice”:ti,ab
#13 “general practitioner”:ti,ab
#14 “community care”:ti,ab
#15 “family medicine”:ti,ab
#16 “family physician”:ti,ab
#17 “family physicians”:ti,ab
#18 “primary care”:ti,ab
#19 “primary health care”:ti,ab
#20 “primary healthcare”:ti,ab
#21 “family doctor”:ti,ab
#22 “family doctors”:ti,ab
#23 “primary medical care”:ti,ab
#24 “general physician”:ti,ab
#25 “general physicians”:ti,ab
#26 “general practices”:ti,ab
#27 “general practitioners”:ti,ab
#28 “primary care practitioners”:ti,ab
#29 “primary care practitioner”:ti,ab
#30 (community next health):ti,ab
#31 (community next healthcare):ti,ab
#32 “health care”:ti,ab
#33 GP:ti,ab
#34 GPs:ti,ab
#35 “primary healthcare team”:ti,ab
#36 “primary healthcare teams”:ti,ab
#37 “primary medical care”:ti,ab
#38 “general internist”:ti,ab
#39 “general internists”:ti,ab
#40 obstetric*:ti,ab
#41 paediatric*:ti,ab
#42 pediatric*:ti,ab
#43 (#1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 OR #10 OR #11 OR #12 OR #13 OR #14 OR #15 OR #16 OR #17 OR #18 OR #19 OR #20 OR #21 OR #22 OR #23 OR #24 OR #25 OR #26 OR #27 OR #28 OR #29 OR #30 OR #31 OR #32 OR #33 OR #34 OR #35 OR #36 OR #37 OR #38 OR #39 OR #40 OR #41 OR #42)
#44 MeSH descriptor Practice Management, Medical, this term only
#45 MeSH descriptor Quality Assurance, Health Care, this term only
#46 MeSH descriptor Quality Indicators, Health Care, this term only
#47 MeSH descriptor Quality of Health Care explode all trees
#48 MeSH descriptor Education, Professional, this term only
#49 MeSH descriptor Attitude of Health Personnel, this term only
#50 MeSH descriptor Patient Acceptance of Health Care, this term only
#51 MeSH descriptor Cooperative Behavior, this term only
#52 MeSH descriptor Professional-Patient Relations, this term only
#53 MeSH descriptor Professional Competence, this term only
#54 MeSH descriptor Physician's Practice Patterns, this term only
#55 MeSH descriptor Professional Practice, this term only
#56 MeSH descriptor Patient-Centered Care, this term only
#57 MeSH descriptor Education, Medical, Continuing, this term only
#58 MeSH descriptor Professional Role, this term only
#59 MeSH descriptor Physician-Patient Relations, this term only
#60 “quality assurance”:ti,ab
#61 “professional behaviour”:ti,ab
#62 cpd:ti,ab
#63 “continuing professional development”:ti,ab
#64 “patient centered care”:ti,ab
#65 “patient centred care”:ti,ab
#66 “continuing medical education”:ti,ab
#67 (training next program*):ti,ab
#68 (training next intervention*):ti,ab
#69 (training next meeting*):ti,ab
#70 (training next session*):ti,ab
#71 (training next strateg*):ti,ab
#72 (training next workshop*):ti,ab
#73 (education* next program*):ti,ab
#74 (education* next intervention*):ti,ab
#75 (education* next meeting*):ti,ab
#76 (education* next session*):ti,ab
#77 (education* next strateg*):ti,ab
#78 “professional behavior”:ti,ab
#79 (#44 OR #45 OR #46 OR #47 OR #48 OR #49 OR #50 OR #51 OR #52 OR #53 OR #54 OR #55 OR #56 OR #57 OR #58 OR #59 OR #60 OR #61 OR #62 OR #63 OR #64 OR #64 OR #66 OR #67 OR #68 OR #69 OR #70 OR #71 OR #72 OR #73 OR #74 OR #75 OR #76 OR #77 OR #78)
#80 MeSH descriptor Feedback explode all trees
#81 MeSH descriptor Interpersonal Relations, this term only
#82 MeSH descriptor Communication explode all trees
#83 MeSH Descriptor Patient Satisfaction explode all trees
#84 (interpersonal next skill*):ti,ab
#85 (consultation next skill*):ti,ab
#86 (communication next skill*):ti,ab
#87 (client next feedback*):ti,ab
#88 (patient next feedback*):ti,ab
#89 (user next feedback*):ti,ab
#90 (consumer next feedback*):ti,ab
#91 (carer next feedback*):ti,ab
#92 (client next evaluation*):ti,ab
#93 (patient* next evaluation*):ti,ab
#94 (user next evaluation*):ti,ab
#95 (consumer next evaluation*):ti,ab
#96 (customer next evaluation*):ti,ab
#97 (carer next evaluation*):ti,ab
#98 (interpersonal next care):ti,ab
#99 feedback:ti,ab
#100 (patient next derived):ti,ab
#101 (patient next mediated):ti,ab
#102 (patient next illicited):ti,ab
#103 (patient next initiated):ti,ab
#104 (#99 OR #100 OR #101 OR #102)
#105 (#98 AND #103)
#106 (feedback near/25 change):ti,ab
#107 (feedback near/25 effect*):ti,ab
#108 (feedback near/25 impact):ti,ab
#109 (feedback near/25 evaluat*):ti,ab
#110 (feedback near/25 compar*):ti,ab
#111 (feedback near/25 modif*):ti,ab
#112 (problem-based next learning):ti,ab
#113 (problem-based next teaching):ti,ab
#114 (problem-based next skill):ti,ab
#115 (problem-based next training):ti,ab
#116 (motivational next interview*):ti,ab
#117 (doctor next patient* next relation*):ti,ab
#118 (doctor next client* next relation*):ti,ab
#119 (physician* next patient* next relation*):ti,ab
#120 (physician* next client* next relation*):ti,ab
#121 (practitioner* next patient* next relation*):ti,ab
#122 (practitioner* next client* next relation*):ti,ab
#123 (doctor next consumer* next relation*):ti,ab
#124 (physician* next consumer* next relation*):ti,ab
#125 (practitioner* next consumer* next relation*):ti,ab
#126 (doctor* next patient* next interaction*):ti,ab
#127 (doctor* next client* next interaction*):ti,ab
#128 (physician* next patient* next interaction*):ti,ab
#129 (physician* next client* next interaction*):ti,ab
#130 (practitioner* next patient* next interaction*):ti,ab
#131 (practitioner* next client* next interaction*):ti,ab
#132 (doctor* next consumer* next interaction*):ti,ab
#133 (physician* next consumer* next interaction*):ti,ab
#134 (practitioner* next consumer* next interaction*):ti,ab
#135 (patient next survey*):ti,ab
#136 (patient next questionnaire*):ti,ab
#137 (#80 OR #81 OR #82 OR #83 OR #84 OR #85 OR #86 OR #87 OR #88 OR #89 OR #90 OR #91 OR #92 OR #93 OR #94 OR #95 OR #96 OR #97 OR #104 OR #105 OR #106 OR #107 OR #108 OR #109 OR #110 OR #111 OR #112 OR #113 OR #114 OR #115 OR #116 OR #117 OR #118 OR #119 OR #120 OR #121 OR #122 OR #123 OR #124 OR #125 OR #126 OR #127 OR #128 OR #129 OR #130 OR #131 OR #132 OR #133 OR #134 OR #135 OR #136)
#138 (#43 AND #79 AND #137)
Appendix 5 Chapter 10: the development of the Value of Patient Feedback scale
The Value of Patient Feedback scale: a report on the development of a new scale
This report sets out the development to date of the Value of Patient Feedback scale, a new scale developed to assess the perceived value to clinicians of receiving patient feedback.
Conceptualisation and scope of the scale
Recent policy initiatives have highlighted the importance for NHS service providers of inviting and reacting to patient feedback. The NHS Operating Framework for 2012/13 requires that ‘NHS organisations must actively seek out, respond positively and improve services in line with patient feedback. This includes acting on complaints, patient comments, local and national surveys and results from ‘real time’ data techniques’. 312 Although patient feedback is now routinely collected from a multitude of sources, as listed above, the impact of this on health-care professionals remains poorly understood. One recent study found that, although the attitudes of GPs towards the concept of patient feedback were often broadly positive, there remained concerns about the credibility of patient experience surveys as a foundation for changes to practice. 119
To assist with studies of the impact of patient feedback on health-care professionals, we have developed a new instrument – the VOP scale – to measure health-care professionals’ attitudes towards receiving feedback from patients. The availability of a robust approach to evaluating perceptions of the utility and impact of patient feedback is an important step in assessing the engagement of health-care professionals with patient experience data and the likely impact of such information on professional practice.
In developing this scale, we defined ‘patient feedback’ as ‘the views and opinions of patients and service users on the health care they experience, as reported to or sought out by service providers through a range of mechanisms and modes, both solicited and unsolicited’ (adapted from The Picker Institute313).
Patient feedback therefore consists of:
-
comments from patients or services users concerning their actual experience of health care, such as a recent GP or outpatient appointment or hospital inpatient stay
-
the communication of these comments to the relevant health-care provider or the seeking out of these comments by the relevant health-care provider, for example through the scanning of online repositories of patient opinion.
Patient feedback may relate to a service as whole (e.g. a general practice or clinic) or to a particular individual (e.g. a doctor or a nurse). Mechanisms of patient feedback are multiple and include solicited opinions (e.g. those that are specifically sought through surveys, RTF, focus groups and interviews) and unsolicited opinions (e.g. those that are made in response to a particular experience, e.g. complaints, compliments, comments through online routes such as NHS Choices or Patient Opinion and comments through social media including Facebook [Facebook, Inc., Menlo Park, CA, USA) and Twitter (Twitter, Inc., San Francisco, CA, USA)].
A literature review of the area located no scale developed specifically to address this area. We therefore initiated the construction of a new instrument.
Drawing on our definition of patient feedback, we used qualitative data collected in previous studies conducted by the research team to derive key constructs to cover in the scale. These data were:
-
interviews with 40 GPs and 14 focus groups with primary care practice staff concerning the impact and utility of patient experience surveys, conducted as part of this NIHR programme grant
-
interviews with 18 GPs concerning their attitudes to patient experience surveys, conducted as part of this NIHR programme grant examining various aspects of the GP Patient Survey
We therefore used an inductive approach to identifying relevant constructs, drawing on existing qualitative data, supplemented with a review of relevant literature in the area.
Consideration of the data and literature suggested that the core construct that we wished to evaluate was multidimensional, covering the following key domains:
-
the right of patients to give feedback
-
responsibility for organising patient feedback
-
preferred mode of patient feedback
-
credibility of patient feedback
-
utility of patient feedback
-
impact of receiving patient feedback
-
changes to individual practice as a result of patient feedback
-
changes to overall quality of health care as a result of patient feedback
-
overall value of patient feedback value.
Item generation
Based on the key domains outlined above, the research team developed a pool of potential items. Items were all positively worded; however, we aimed to create a balanced pool by constructing items to evaluate either positive or negative aspects of domains when possible. We chose to use a five-point Likert response scale (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree). Table 73 provides the full list of potential items. We then undertook initial content validation for the scale by mapping each potential item onto the key domains to ensure that each was represented by at a number of items and that no items were irrelevant to the construct under consideration (see Annex A).
| Number | Item |
|---|---|
| 1 | Patients should have the opportunity to provide feedback on their experiences of care provided |
| 2 | It is important to listen to patients about their experiences of care |
| 3 | It is a clinician’s responsibility to gather evidence of patients’ experience of care |
| 4 | It is the responsibility of clinical commissioning groups and hospitals to gather evidence of patients’ experience of care |
| 5 | It is the responsibility of NHS England to gather evidence of patient’ experience of care |
| 6 | I have reservations about patient feedback received via complaints and compliments |
| 7 | I have reservations about patient feedback received via surveys |
| 8 | I have reservations about patient feedback received via patient forums or participant groups |
| 9 | I have reservations about patient feedback received via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) |
| 10 | It is beneficial to receive patient feedback via complaints and compliments |
| 11 | It is beneficial to receive patient feedback via surveys |
| 12 | It is beneficial to receive patient feedback via patient forums or participant groups |
| 13 | It is beneficial to receive patient feedback via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) |
| 14 | I am interested in learning more about receiving patient feedback through social media such as Twitter or Facebook |
| 15 | Responders to patient surveys are representative of my patient population |
| 16 | Data from patient surveys are valid and reliable |
| 17 | Patients always have grounds for the complaints they make |
| 18 | Patient feedback via patient forums or participant groups is a reliable indicator of patient concerns |
| 19 | Patients who use online patient feedback mechanisms (such as NHS Choices or Patient Opinion) are representative of my patient population |
| 20 | I trust patient feedback received via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) |
| 21 | Patients are able to provide useful feedback on my clinical skills |
| 22 | Patients are able to provide useful feedback on my interpersonal skills |
| 23 | Patients are able to provide useful feedback on administrative and organisational issues |
| 24 | Patient survey data are more valuable if they include benchmarking |
| 25 | Free-text comments are the most useful aspect of patient surveys |
| 26 | I find it difficult to interpret the results of patient surveys |
| 27 | I find it easy to understand patient feedback |
| 28 | Receiving supported feedback (such as with a facilitator) following surveys of patients’ experience of care would help me gain a better understanding of the results |
| 29 | Feedback from patient surveys is presented in a timely way |
| 30 | Patient feedback on specific identifiable care experiences is more helpful than general opinions |
| 31 | Patient feedback is useful for individual performance assessment, such as in appraisal or revalidation |
| 32 | Patient surveys indicate what needs to be done to improve |
| 33 | Patient anonymity limits the usefulness of most patient feedback |
| 34 | I know how to act on anonymous patient feedback |
| 35 | I know how to act on feedback received from a named patient |
| 36 | I can make good use of patient feedback |
| 37 | I have doubted my competence after receiving feedback from patients |
| 38 | Receiving feedback from patients has improved my confidence at work |
| 39 | Receiving patient feedback via patient surveys is a positive experience |
| 40 | Receiving a complaint from a patient can impact on my ability to work effectively |
| 41 | Engaging with patient feedback requires a lot of energy |
| 42 | I worry about my workplace’s reputation as a result of patient feedback being made public |
| 43 | I worry about my individual reputation as a result of patient feedback being made public |
| 44 | Acting on patient feedback can improve the clinical quality of care I provide |
| 45 | Acting on patient feedback can improve the interpersonal quality of care I provide |
| 46 | Acting on patient feedback can improve the organisation and administration of the care I provide |
| 47 | Receiving patient feedback can improve my relationship with patients |
| 48 | I have made changes to my individual practice as a result of patient feedback |
| 49 | I am likely to make changes to my individual practice as a result of patient feedback |
| 50 | It is necessary to have patient feedback to improve the overall quality of health care |
| 51 | Listening to patients will lead to useful changes to health care |
| 52 | Patient feedback is an important mechanism of quality improvement |
| 53 | Gathering patient feedback is beneficial to the health service |
| 54 | Data collection for large, representative patient surveys is cost-effective |
| 55 | Online patient feedback mechanisms (such as NHS Choices or Patient Opinion) are beneficial to the health service |
| 56 | I value feedback from colleagues more than feedback from patients |
Expert panel review
Following the generation of the item pool, we asked six experts in the field to critically analyse the proposed scale to review (a) content validity – whether or not the items fully cover the concept adequately – and (b) the clarity, readability and content of each item. Experts were all academics and clinicians involved in the evaluation of patient experience of care. Draft item statements were sent to all six panel members who were asked to assess, comment on and suggest changes to the items. We subsequently drew on these evaluations to further review the item pool, removing, retaining, adding or rewording items as required. Annex B sets out the responses to each item from each reviewer: either items were retained as is or amendments were suggested by reviewers. Following the receipt of all comments, the research team met and by group consensus derived a final list of 52 items to take forward to the next stage.
Cognitive interviewing
Methods
Following the collation of the revised item pool, we undertook cognitive testing through interviews with clinicians. We used a combination of cognitive interviewing techniques (rephrasing, thinking aloud and probing) to evaluate respondents’ approaches to answering items, the wording of items and particular items or words that were problematic. 314 The revised items (n = 52) were divided into three groups, two groups of 18 items and one group of 19 (see Annex C).These groups were selected to cover each of the major domains of interest and to ensure that a number of respondents considered each item.
Participants (n = 7) were recruited from the General Practice Education Group at the University of Cambridge Institute of Public Health. Interviews took place between December 2013 and January 2014. Each participant completed one of the subscales at the start of the interview (Table 74). Participants were free to discuss items as they went along or wait until the longer discussion after completing all of the items in full. The interviewer also noted any hesitation in response that took place for specific items. This was discussed in more detail after completion of the scale and participants were asked to describe how they found the scale overall and if there were any particular items of concern. An interview guide with a number of suggested prompts was used to help guide the discussion (see Annex D). Broadly, interview probes examined retrieval, comprehension, confidence and response. 314 Interviews lasted approximately 20 minutes and were recorded and transcribed verbatim. NVivo 10 software was used to assist with interview coding and interviews were analysed thematically. An initial coding framework was developed that incorporated each of the scale items. Additional codes were added when participants discussed issues that did not relate to a specific scale item. When a new code was identified, transcripts were rechecked for instances of this new code.
| Participant ID | VOP version assignment |
|---|---|
| VOP1_GP1 | 1 |
| VOP2_GP1 | 2 |
| VOP3_GP2 | 3 |
| VOP3_GP1 | 3 |
| VOP1_GP2 | 1 |
| VOP2_GP2 | 2 |
| VOP1_GP3 | 1 |
Analysis
Overall, responses to the scale were found to be positive. In terms of phrasing and comprehension, no items were found to be difficult to understand. In more general terms, some participants had concerns that the overall methodological approach of establishing clinicians’ attitudes towards patient feedback may not be best obtained through a questionnaire. Additionally, for some participants, the term ‘patient feedback’ was problematic. For example, some interpreted this term as encompassing all types of feedback received from patients, inclusive of verbal feedback, comments and complaints, online feedback and practice-level or national surveys. However, others (and this appeared to be older GPs) viewed feedback more as surveys. Although some items related to specific types of feedback, many of the participants responded verbally with ‘it depends’. There were also issues relating to the organisational level (individual vs. practice vs. national) that we referred to.
After the first three interviews, we added an introductory ‘blurb’ to the start of the scale to try to provide greater clarity on the term ‘patient feedback’. However, when subsequent participants were asked about how useful the blurb was at the start of the questionnaire, none of the participants reported having read it. This may have been a result of the layout and design of the scale rather than the blurb; however, even after having read it when prompted, it did not seem to assist participants in answering the items.
One item in particular seemed to cause some concern among several of the participants: ‘It is a clinician’s responsibility to gather evidence of patients’ experience of care’. This item caused ambiguity. One GP suggested that the wording implied that the clinician should be the one to be physically administering questionnaires, etc. (VOP2_GP1). ‘To gather’ caused ambiguity (VOP3_GP1). ‘I think, on reflection, it is my responsibility. It doesn’t necessarily mean to say I have to do it, but I have a responsibility to ensure it happens’ (VOP2_GP1). A second ambiguity related to the ‘responsibility’, as another GP pointed out that this may vary depending on position, for example hospital doctor, salaried GP, GP partner and so on (VOP1_GP2).
Other items that participants provided specific feedback on were as follows:
-
‘I have reservations about patient feedback received via patient forums or participant groups’: similar question to ‘It is beneficial to receive patient feedback via patient forums or participant groups’ – see previous comments by VOP3_GP1; can you say something is beneficial without being representative (VOP3_GP2) or biased (VOP3_GP1)?
-
‘It is beneficial to receive patient feedback via patient forums or participant groups’: same comments as above (VOP3_GP1, VOP3_GP2)
-
Free-text comments are better than replies to closed questions: probably, but not always convenient and, therefore, tick boxes win over (VOP1_GP1, VOP1_GP2); not a straightforward response.
-
‘I worry about my workplace’s reputation as a result of patient feedback being made public’: distinctions need to be made about sharing accurate, representative, unbiased data with the public. If that feedback is then negative then this should be shared, but if not it provides an unfair view (VOP2_GP1, VOP2_GP2).
-
Patient feedback can improve the clinical quality of care I provide: consider making the distinction in the question about ‘detailed and specific’ patient feedback (VOP1_GP1). Similar (although different perspective) question to ‘. . . quality of care I provide . . .’ (VOP1_GP2).
-
‘I have made changes to my clinical practice as a result of patient feedback’: tension between wanting to say yes to justify other responses (patient feedback is good, etc.) but not wanting to say strongly agree, as some types of patient feedback are viewed negatively (NHS Choices, etc.) (VOP2_GP1, VOP2_GP2).
-
I find it difficult to make tangible changes to my practice as a result of patient survey results: ambiguity from ‘tangible’ in the context – changes made all the time, some without realising, therefore difficult to respond to (VOP3_GP1, VOP3_GP2).
-
I have a good idea of how satisfied my patients are with the care I provide: difficult to answer as ‘you don’t know what you don’t know’. Similar to the representativeness issues as likely to be only the people who like you who will come back and provide positive feedback (VOP3_GP1, VOP3_GP2). Easier to answer on a practice level. Consider having this question at the beginning? (VOP3_GP2).
-
‘I have reservations about patient feedback received via complaints and compliments’: very similar issues raised for this item as for other ‘beneficial’ vs. ‘reservation’ items (see previous comments) (VOP1_GP1, VOP1_GP2).
Conclusions
Following cognitive interviews, the 52 items were further reduced to 43, with textual amendments made to some items.
Pretesting
To undertake pretesting of the scale, we conducted an online survey of doctors and nurses using the established web interface QuestionPro (QuestionPro Inc., San Francisco, CA, USA). Consent was indicated by completion of the items. The sample was recruited using snowball methods. We initially approached participants by e-mail, drawing on those we were already in contact with through the IMPROVE programme (e.g. those who participated in previous projects within the IMPROVE NIHR programme grant) and colleagues (e.g. the General Practice Education Group at the University of Cambridge; nursing and midwifery units). Respondents were asked to forward details of the survey to clinical colleagues. We also used social media, including Twitter, to ask for respondents to complete the questionnaire. As an incentive, all respondents who completed the questionnaire were offered the opportunity to enter a prize draw to win a Kindle Paperwhite. The survey was conducted from February to May 2014. It included the 43 VOP items alongside some brief sociodemographic questions (see Annex E). An initial analysis of the first 30 responses found no problems with missing data and we continued the survey as planned. By the closing date, we had obtained 215 responses from doctors and nurses.
A stacked bar chart showing the proportion of respondents selecting each response for each item is shown in Figure 25. We can see a range of response tendencies across questions. Table 75 shows the polychoric correlation matrix for all items, with green indicating a positive correlation and red indicating a negative correlation (colour coding has been allowed to saturate at ±0.5 to illustrate variation). What is quite clear from this table is that some items are negatively correlated with most other items (indicated by bands of red). These items tend to be those for which strongly agreeing with a statement conveys a lack of support for patient feedback, for example ‘I have reservations about patient feedback currently received via surveys’. Such negative questions had been identified prior to the analysis. We then reversed the scoring of these questions and the resulting stacked bar chart and correlation matrix are shown in Figure 26 and Table 76, respectively. In the correlation matrix the vast majority of correlations are positive, except those for question 19, ‘Patient survey data are more valuable if they include comparison with how others are doing’. On reflection, it was not clear whether agreeing with this statement is supportive of patient feedback or not. As we were trying to develop a scale to measure the value that responders place on patient feedback rather than to gather opinions, this question was not seen as helping and so was dropped from further analysis.
FIGURE 25.
Percentage endorsing each response category for each question (including missing when question not answered rather than page missing), with ‘strongly positive’ indicating a view in favour of patient feedback.
| Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 | Q11 | Q12 | Q13 | Q14 | Q15 | Q16 | Q17 | Q18 | Q19 | Q20 | Q21 | Q22 | Q23 | Q24 | Q25 | Q26 | Q27 | Q28 | Q29 | Q30 | Q31 | Q32 | Q33 | Q34 | Q35 | Q36 | Q37 | Q38 | Q39 | Q40 | Q41 | Q42 | Q43 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Q1 | 1.00 | ||||||||||||||||||||||||||||||||||||||||||
| Q2 | 0.88 | 1.00 | |||||||||||||||||||||||||||||||||||||||||
| Q3 | 0.44 | 0.35 | 1.00 | ||||||||||||||||||||||||||||||||||||||||
| Q4 | −0.28 | −0.22 | −0.05 | 1.00 | |||||||||||||||||||||||||||||||||||||||
| Q5 | −0.13 | −0.25 | −0.08 | 0.49 | 1.00 | ||||||||||||||||||||||||||||||||||||||
| Q6 | −0.19 | −0.30 | −0.20 | 0.52 | 0.50 | 1.00 | |||||||||||||||||||||||||||||||||||||
| Q7 | −0.39 | −0.40 | −0.08 | 0.41 | 0.54 | 0.60 | 1.00 | ||||||||||||||||||||||||||||||||||||
| Q8 | 0.31 | 0.30 | 0.21 | −0.48 | −0.14 | −0.32 | −0.29 | 1.00 | |||||||||||||||||||||||||||||||||||
| Q9 | 0.41 | 0.37 | 0.14 | −0.28 | −0.50 | −0.35 | −0.40 | 0.56 | 1.00 | ||||||||||||||||||||||||||||||||||
| Q10 | 0.23 | 0.31 | 0.33 | −0.26 | −0.21 | −0.55 | −0.35 | 0.66 | 0.52 | 1.00 | |||||||||||||||||||||||||||||||||
| Q11 | 0.34 | 0.33 | 0.17 | −0.22 | −0.29 | −0.27 | −0.68 | 0.43 | 0.51 | 0.54 | 1.00 | ||||||||||||||||||||||||||||||||
| Q12 | 0.32 | 0.27 | 0.04 | −0.26 | −0.05 | −0.14 | −0.27 | 0.30 | 0.20 | 0.31 | 0.44 | 1.00 | |||||||||||||||||||||||||||||||
| Q13 | 0.10 | 0.08 | 0.11 | −0.25 | −0.29 | −0.26 | −0.31 | −0.01 | 0.10 | 0.05 | 0.19 | 0.04 | 1.00 | ||||||||||||||||||||||||||||||
| Q14 | 0.12 | 0.10 | 0.10 | −0.30 | −0.51 | −0.30 | −0.36 | −0.06 | 0.31 | 0.12 | 0.18 | 0.02 | 0.55 | 1.00 | |||||||||||||||||||||||||||||
| Q15 | 0.17 | 0.15 | 0.25 | −0.39 | −0.23 | −0.42 | −0.35 | 0.31 | 0.37 | 0.44 | 0.32 | 0.10 | 0.14 | 0.25 | 1.00 | ||||||||||||||||||||||||||||
| Q16 | 0.32 | 0.29 | 0.22 | −0.22 | −0.05 | −0.25 | −0.16 | 0.13 | 0.21 | 0.15 | 0.13 | 0.09 | 0.22 | 0.12 | 0.22 | 1.00 | |||||||||||||||||||||||||||
| Q17 | 0.10 | 0.31 | 0.23 | 0.00 | 0.04 | −0.13 | 0.02 | 0.33 | 0.25 | 0.27 | 0.12 | 0.06 | 0.11 | 0.11 | 0.24 | 0.24 | 1.00 | ||||||||||||||||||||||||||
| Q18 | 0.17 | 0.36 | 0.21 | −0.12 | 0.14 | −0.11 | −0.03 | 0.42 | 0.15 | 0.39 | 0.21 | 0.26 | 0.14 | 0.07 | 0.27 | 0.29 | 0.59 | 1.00 | |||||||||||||||||||||||||
| Q19 | −0.15 | −0.04 | 0.08 | 0.20 | 0.05 | 0.18 | 0.29 | −0.11 | 0.00 | −0.09 | −0.10 | 0.01 | −0.14 | 0.02 | −0.09 | −0.17 | 0.13 | 0.06 | 1.00 | ||||||||||||||||||||||||
| Q20 | −0.13 | −0.10 | −0.04 | 0.27 | 0.43 | 0.36 | 0.27 | −0.10 | −0.22 | −0.23 | −0.11 | −0.12 | −0.16 | −0.34 | −0.07 | 0.06 | −0.04 | 0.13 | 0.02 | 1.00 | |||||||||||||||||||||||
| Q21 | 0.30 | 0.29 | 0.19 | −0.24 | −0.29 | −0.42 | −0.28 | 0.09 | 0.23 | 0.29 | 0.22 | 0.24 | 0.31 | 0.36 | 0.22 | 0.19 | 0.24 | 0.20 | −0.04 | −0.45 | 1.00 | ||||||||||||||||||||||
| Q22 | 0.05 | 0.12 | 0.09 | −0.09 | −0.24 | −0.26 | −0.15 | 0.07 | 0.17 | 0.15 | 0.04 | −0.02 | 0.14 | 0.32 | 0.20 | 0.13 | 0.13 | 0.09 | 0.22 | −0.08 | 0.09 | 1.00 | |||||||||||||||||||||
| Q23 | 0.22 | 0.30 | 0.23 | −0.22 | −0.21 | −0.38 | −0.22 | 0.32 | 0.37 | 0.36 | 0.20 | 0.18 | 0.29 | 0.21 | 0.25 | 0.35 | 0.32 | 0.29 | 0.13 | −0.25 | 0.39 | 0.21 | 1.00 | ||||||||||||||||||||
| Q24 | 0.36 | 0.45 | 0.10 | −0.39 | −0.34 | −0.41 | −0.33 | 0.30 | 0.42 | 0.29 | 0.30 | 0.35 | 0.18 | 0.40 | 0.16 | 0.39 | 0.27 | 0.42 | 0.06 | −0.21 | 0.34 | 0.28 | 0.27 | 1.00 | |||||||||||||||||||
| Q25 | 0.15 | 0.17 | 0.07 | −0.35 | −0.35 | −0.37 | −0.23 | 0.22 | 0.41 | 0.16 | 0.22 | 0.23 | 0.26 | 0.41 | 0.15 | 0.24 | 0.26 | 0.26 | −0.05 | −0.22 | 0.31 | 0.18 | 0.26 | 0.74 | 1.00 | ||||||||||||||||||
| Q26 | −0.12 | −0.05 | 0.14 | 0.33 | 0.30 | 0.19 | 0.24 | −0.24 | −0.33 | −0.09 | −0.28 | −0.09 | −0.18 | −0.19 | −0.15 | 0.08 | −0.10 | 0.09 | 0.01 | 0.23 | −0.02 | −0.05 | −0.04 | −0.14 | −0.22 | 1.00 | |||||||||||||||||
| Q27 | 0.31 | 0.43 | 0.25 | −0.27 | −0.23 | −0.34 | −0.34 | 0.30 | 0.36 | 0.32 | 0.27 | 0.17 | 0.20 | 0.29 | 0.24 | 0.40 | 0.34 | 0.29 | 0.02 | −0.26 | 0.33 | 0.27 | 0.43 | 0.61 | 0.44 | −0.04 | 1.00 | ||||||||||||||||
| Q28 | 0.25 | 0.26 | 0.17 | −0.21 | −0.01 | −0.23 | −0.18 | 0.36 | 0.34 | 0.39 | 0.33 | 0.27 | 0.00 | 0.07 | 0.26 | 0.29 | 0.33 | 0.40 | 0.13 | −0.19 | 0.11 | 0.19 | 0.34 | 0.28 | 0.27 | −0.03 | 0.53 | 1.00 | |||||||||||||||
| Q29 | 0.00 | 0.08 | 0.06 | −0.26 | −0.25 | −0.31 | −0.29 | 0.19 | 0.13 | 0.16 | 0.20 | 0.20 | 0.28 | 0.15 | 0.15 | 0.16 | 0.16 | 0.18 | −0.15 | −0.18 | 0.23 | 0.18 | 0.42 | 0.23 | 0.27 | −0.06 | 0.52 | 0.37 | 1.00 | ||||||||||||||
| Q30 | −0.05 | −0.02 | 0.16 | −0.24 | −0.48 | −0.35 | −0.21 | 0.18 | 0.36 | 0.20 | 0.27 | 0.15 | 0.35 | 0.43 | 0.08 | 0.05 | 0.05 | 0.05 | 0.00 | −0.39 | 0.39 | 0.17 | 0.35 | 0.33 | 0.49 | −0.21 | 0.35 | 0.11 | 0.57 | 1.00 | |||||||||||||
| Q31 | −0.05 | −0.15 | −0.04 | 0.35 | 0.27 | 0.34 | 0.45 | −0.19 | −0.17 | −0.15 | −0.30 | −0.26 | −0.30 | −0.27 | −0.12 | 0.00 | −0.08 | −0.07 | 0.17 | 0.25 | −0.20 | −0.02 | −0.05 | −0.22 | −0.28 | 0.33 | −0.16 | −0.05 | −0.19 | −0.25 | 1.00 | ||||||||||||
| Q32 | −0.17 | −0.16 | −0.16 | 0.38 | 0.30 | 0.37 | 0.40 | −0.18 | −0.23 | −0.20 | −0.26 | −0.23 | −0.20 | −0.24 | −0.18 | −0.02 | −0.06 | −0.08 | 0.14 | 0.23 | −0.24 | 0.03 | −0.08 | −0.21 | −0.16 | 0.30 | −0.18 | −0.03 | −0.23 | −0.18 | 0.87 | 1.00 | |||||||||||
| Q33 | 0.28 | 0.37 | 0.29 | −0.25 | −0.25 | −0.34 | −0.28 | 0.32 | 0.34 | 0.32 | 0.31 | 0.24 | 0.10 | 0.16 | 0.31 | 0.41 | 0.28 | 0.28 | −0.15 | −0.12 | 0.20 | 0.34 | 0.43 | 0.46 | 0.47 | −0.09 | 0.56 | 0.47 | 0.43 | 0.33 | −0.25 | −0.22 | 1.00 | ||||||||||
| Q34 | 0.15 | 0.25 | 0.15 | −0.24 | −0.17 | −0.29 | −0.24 | 0.24 | 0.22 | 0.27 | 0.30 | 0.22 | 0.02 | 0.14 | 0.31 | 0.20 | 0.11 | 0.29 | −0.04 | −0.05 | 0.10 | 0.22 | 0.29 | 0.36 | 0.30 | −0.01 | 0.41 | 0.61 | 0.44 | 0.29 | −0.11 | −0.15 | 0.65 | 1.00 | |||||||||
| Q35 | −0.07 | −0.23 | −0.18 | 0.32 | 0.24 | 0.57 | 0.34 | −0.25 | −0.23 | −0.44 | −0.26 | −0.20 | −0.27 | −0.22 | −0.19 | −0.16 | −0.10 | −0.18 | 0.09 | 0.25 | −0.29 | −0.27 | −0.24 | −0.38 | −0.37 | 0.10 | −0.40 | −0.35 | −0.41 | −0.25 | 0.24 | 0.25 | −0.45 | −0.38 | 1.00 | ||||||||
| Q36 | 0.09 | 0.14 | 0.10 | −0.23 | −0.17 | −0.20 | −0.07 | 0.12 | 0.03 | 0.25 | 0.18 | 0.10 | 0.05 | 0.10 | 0.26 | 0.23 | 0.05 | 0.21 | −0.07 | −0.11 | 0.20 | 0.14 | 0.21 | 0.27 | 0.28 | 0.10 | 0.29 | 0.44 | 0.20 | 0.23 | 0.07 | 0.05 | 0.48 | 0.66 | −0.26 | 1.00 | |||||||
| Q37 | 0.31 | 0.38 | 0.17 | −0.26 | −0.37 | −0.38 | −0.31 | 0.24 | 0.32 | 0.36 | 0.38 | 0.19 | 0.12 | 0.20 | 0.29 | 0.26 | 0.17 | 0.29 | −0.06 | −0.26 | 0.30 | 0.24 | 0.36 | 0.37 | 0.41 | −0.06 | 0.57 | 0.57 | 0.44 | 0.41 | −0.11 | −0.06 | 0.62 | 0.70 | −0.51 | 0.61 | 1.00 | ||||||
| Q38 | 0.50 | 0.45 | 0.12 | −0.45 | −0.32 | −0.37 | −0.37 | 0.41 | 0.40 | 0.36 | 0.35 | 0.37 | 0.15 | 0.21 | 0.26 | 0.22 | 0.27 | 0.24 | −0.08 | −0.16 | 0.29 | 0.19 | 0.40 | 0.49 | 0.36 | −0.12 | 0.59 | 0.50 | 0.42 | 0.29 | −0.31 | −0.28 | 0.54 | 0.40 | −0.40 | 0.23 | 0.45 | 1.00 | |||||
| Q39 | 0.42 | 0.52 | 0.23 | −0.38 | −0.39 | −0.43 | −0.41 | 0.44 | 0.57 | 0.45 | 0.49 | 0.30 | 0.12 | 0.32 | 0.41 | 0.26 | 0.39 | 0.37 | 0.05 | −0.19 | 0.30 | 0.33 | 0.49 | 0.55 | 0.47 | −0.15 | 0.64 | 0.55 | 0.40 | 0.40 | −0.40 | −0.32 | 0.67 | 0.45 | −0.38 | 0.22 | 0.55 | 0.83 | 1.00 | ||||
| Q40 | 0.42 | 0.57 | 0.14 | −0.36 | −0.39 | −0.36 | −0.34 | 0.53 | 0.57 | 0.50 | 0.48 | 0.35 | 0.07 | 0.23 | 0.28 | 0.18 | 0.37 | 0.37 | 0.12 | −0.23 | 0.30 | 0.28 | 0.47 | 0.56 | 0.41 | −0.23 | 0.62 | 0.50 | 0.43 | 0.40 | −0.39 | −0.39 | 0.56 | 0.32 | −0.34 | 0.14 | 0.48 | 0.79 | 0.94 | 1.00 | |||
| Q41 | −0.11 | −0.20 | −0.14 | 0.24 | 0.44 | 0.23 | 0.18 | −0.15 | −0.31 | −0.29 | −0.23 | −0.12 | 0.01 | −0.23 | −0.15 | −0.07 | −0.13 | −0.10 | 0.03 | 0.24 | −0.21 | −0.11 | −0.18 | −0.19 | −0.26 | 0.22 | −0.18 | −0.28 | −0.16 | −0.28 | 0.15 | 0.12 | −0.35 | −0.29 | 0.27 | −0.16 | −0.33 | −0.30 | −0.49 | −0.49 | 1.00 | ||
| Q42 | −0.22 | −0.37 | −0.21 | 0.31 | 0.24 | 0.41 | 0.24 | −0.34 | −0.40 | −0.38 | −0.38 | −0.33 | −0.29 | −0.25 | −0.34 | −0.25 | −0.30 | −0.21 | 0.18 | 0.17 | −0.32 | −0.14 | −0.42 | −0.32 | −0.35 | 0.21 | −0.35 | −0.19 | −0.44 | −0.32 | 0.25 | 0.28 | −0.41 | −0.19 | 0.42 | −0.11 | −0.37 | −0.43 | −0.52 | −0.50 | 0.30 | 1.00 | |
| Q43 | 0.26 | 0.33 | 0.22 | −0.22 | −0.30 | −0.36 | −0.29 | 0.16 | 0.23 | 0.19 | 0.28 | 0.29 | 0.19 | 0.25 | 0.29 | 0.29 | 0.25 | 0.32 | 0.00 | −0.02 | 0.07 | 0.23 | 0.22 | 0.34 | 0.27 | −0.05 | 0.33 | 0.29 | 0.27 | 0.15 | −0.29 | −0.34 | 0.46 | 0.32 | −0.25 | 0.25 | 0.33 | 0.37 | 0.59 | 0.54 | −0.21 | −0.34 | 1.00 |
FIGURE 26.
Percentage endorsing each response level for each question (including missing when question not answered rather than page missing), with ‘strongly positive’ indicating a view in favour of patient feedback.
| Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 | Q11 | Q12 | Q13 | Q14 | Q15 | Q16 | Q17 | Q18 | Q19 | Q20 | Q21 | Q22 | Q23 | Q24 | Q25 | Q26 | Q27 | Q28 | Q29 | Q30 | Q31 | Q32 | Q33 | Q34 | Q35 | Q36 | Q37 | Q38 | Q39 | Q40 | Q41 | Q42 | Q43 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Q1 | 1.00 | ||||||||||||||||||||||||||||||||||||||||||
| Q2 | 0.88 | 1.00 | |||||||||||||||||||||||||||||||||||||||||
| Q3 | 0.44 | 0.35 | 1.00 | ||||||||||||||||||||||||||||||||||||||||
| Q4 | 0.28 | 0.22 | 0.05 | 1.00 | |||||||||||||||||||||||||||||||||||||||
| Q5 | 0.13 | 0.25 | 0.08 | 0.49 | 1.00 | ||||||||||||||||||||||||||||||||||||||
| Q6 | 0.19 | 0.30 | 0.20 | 0.52 | 0.49 | 1.00 | |||||||||||||||||||||||||||||||||||||
| Q7 | 0.39 | 0.39 | 0.08 | 0.41 | 0.53 | 0.59 | 1.00 | ||||||||||||||||||||||||||||||||||||
| Q8 | 0.31 | 0.30 | 0.21 | 0.48 | 0.14 | 0.32 | 0.29 | 1.00 | |||||||||||||||||||||||||||||||||||
| Q9 | 0.41 | 0.37 | 0.14 | 0.28 | 0.50 | 0.35 | 0.40 | 0.56 | 1.00 | ||||||||||||||||||||||||||||||||||
| Q10 | 0.23 | 0.31 | 0.33 | 0.26 | 0.21 | 0.55 | 0.35 | 0.66 | 0.52 | 1.00 | |||||||||||||||||||||||||||||||||
| Q11 | 0.34 | 0.33 | 0.17 | 0.22 | 0.29 | 0.27 | 0.68 | 0.43 | 0.51 | 0.54 | 1.00 | ||||||||||||||||||||||||||||||||
| Q12 | 0.32 | 0.27 | 0.04 | 0.26 | 0.05 | 0.14 | 0.26 | 0.30 | 0.20 | 0.31 | 0.44 | 1.00 | |||||||||||||||||||||||||||||||
| Q13 | 0.10 | 0.08 | 0.11 | 0.25 | 0.29 | 0.26 | 0.31 | −0.01 | 0.10 | 0.05 | 0.19 | 0.04 | 1.00 | ||||||||||||||||||||||||||||||
| Q14 | 0.12 | 0.10 | 0.10 | 0.29 | 0.51 | 0.30 | 0.35 | −0.06 | 0.31 | 0.12 | 0.18 | 0.02 | 0.55 | 1.00 | |||||||||||||||||||||||||||||
| Q15 | 0.17 | 0.15 | 0.25 | 0.39 | 0.23 | 0.42 | 0.35 | 0.31 | 0.37 | 0.44 | 0.32 | 0.10 | 0.14 | 0.25 | 1.00 | ||||||||||||||||||||||||||||
| Q16 | 0.32 | 0.29 | 0.22 | 0.22 | 0.05 | 0.25 | 0.16 | 0.13 | 0.21 | 0.15 | 0.13 | 0.09 | 0.22 | 0.12 | 0.22 | 1.00 | |||||||||||||||||||||||||||
| Q17 | 0.10 | 0.31 | 0.23 | 0.00 | −0.04 | 0.13 | −0.02 | 0.33 | 0.25 | 0.27 | 0.12 | 0.06 | 0.11 | 0.11 | 0.24 | 0.24 | 1.00 | ||||||||||||||||||||||||||
| Q18 | 0.17 | 0.36 | 0.21 | 0.12 | −0.14 | 0.11 | 0.03 | 0.42 | 0.15 | 0.39 | 0.21 | 0.26 | 0.14 | 0.07 | 0.27 | 0.29 | 0.59 | 1.00 | |||||||||||||||||||||||||
| Q19 | −0.15 | −0.04 | 0.08 | −0.20 | −0.05 | −0.18 | −0.29 | −0.11 | 0.00 | −0.09 | −0.10 | 0.01 | −0.14 | 0.02 | −0.09 | −0.17 | 0.13 | 0.06 | 1.00 | ||||||||||||||||||||||||
| Q20 | 0.13 | 0.10 | 0.03 | 0.27 | 0.43 | 0.36 | 0.26 | 0.10 | 0.22 | 0.23 | 0.11 | 0.12 | 0.16 | 0.34 | 0.07 | −0.06 | 0.05 | −0.13 | −0.02 | 1.00 | |||||||||||||||||||||||
| Q21 | 0.30 | 0.29 | 0.19 | 0.24 | 0.29 | 0.42 | 0.28 | 0.09 | 0.23 | 0.29 | 0.22 | 0.24 | 0.31 | 0.36 | 0.22 | 0.19 | 0.24 | 0.20 | −0.04 | 0.45 | 1.00 | ||||||||||||||||||||||
| Q22 | 0.05 | 0.12 | 0.09 | 0.09 | 0.24 | 0.26 | 0.15 | 0.07 | 0.17 | 0.15 | 0.04 | −0.02 | 0.14 | 0.32 | 0.20 | 0.13 | 0.13 | 0.09 | 0.22 | 0.08 | 0.09 | 1.00 | |||||||||||||||||||||
| Q23 | 0.22 | 0.30 | 0.23 | 0.21 | 0.20 | 0.38 | 0.22 | 0.32 | 0.37 | 0.36 | 0.20 | 0.18 | 0.29 | 0.21 | 0.25 | 0.35 | 0.32 | 0.29 | 0.13 | 0.25 | 0.39 | 0.21 | 1.00 | ||||||||||||||||||||
| Q24 | 0.36 | 0.45 | 0.10 | 0.39 | 0.34 | 0.41 | 0.33 | 0.30 | 0.42 | 0.29 | 0.30 | 0.35 | 0.18 | 0.40 | 0.16 | 0.39 | 0.27 | 0.42 | 0.06 | 0.21 | 0.34 | 0.28 | 0.27 | 1.00 | |||||||||||||||||||
| Q25 | 0.15 | 0.17 | 0.07 | 0.34 | 0.35 | 0.37 | 0.23 | 0.22 | 0.41 | 0.16 | 0.22 | 0.23 | 0.26 | 0.41 | 0.15 | 0.24 | 0.26 | 0.26 | −0.05 | 0.22 | 0.31 | 0.18 | 0.26 | 0.74 | 1.00 | ||||||||||||||||||
| Q26 | 0.12 | 0.05 | −0.14 | 0.33 | 0.30 | 0.19 | 0.24 | 0.24 | 0.33 | 0.09 | 0.28 | 0.09 | 0.19 | 0.20 | 0.15 | −0.08 | 0.10 | −0.09 | −0.01 | 0.23 | 0.02 | 0.06 | 0.04 | 0.15 | 0.22 | 1.00 | |||||||||||||||||
| Q27 | 0.31 | 0.43 | 0.25 | 0.27 | 0.23 | 0.34 | 0.34 | 0.30 | 0.36 | 0.32 | 0.27 | 0.17 | 0.20 | 0.29 | 0.24 | 0.40 | 0.34 | 0.29 | 0.02 | 0.26 | 0.33 | 0.27 | 0.43 | 0.61 | 0.44 | 0.04 | 1.00 | ||||||||||||||||
| Q28 | 0.25 | 0.26 | 0.17 | 0.21 | 0.01 | 0.23 | 0.18 | 0.36 | 0.34 | 0.39 | 0.33 | 0.27 | 0.00 | 0.07 | 0.26 | 0.29 | 0.33 | 0.40 | 0.13 | 0.19 | 0.11 | 0.19 | 0.34 | 0.28 | 0.27 | 0.03 | 0.53 | 1.00 | |||||||||||||||
| Q29 | 0.00 | 0.08 | 0.06 | 0.26 | 0.25 | 0.31 | 0.29 | 0.19 | 0.13 | 0.16 | 0.20 | 0.20 | 0.28 | 0.15 | 0.15 | 0.16 | 0.16 | 0.18 | −0.15 | 0.18 | 0.23 | 0.18 | 0.42 | 0.23 | 0.27 | 0.06 | 0.52 | 0.37 | 1.00 | ||||||||||||||
| Q30 | −0.05 | −0.02 | 0.16 | 0.24 | 0.48 | 0.35 | 0.21 | 0.18 | 0.36 | 0.20 | 0.27 | 0.15 | 0.35 | 0.43 | 0.08 | 0.05 | 0.05 | 0.05 | 0.00 | 0.38 | 0.39 | 0.17 | 0.35 | 0.33 | 0.49 | 0.21 | 0.35 | 0.11 | 0.57 | 1.00 | |||||||||||||
| Q31 | 0.06 | 0.15 | 0.04 | 0.35 | 0.27 | 0.34 | 0.45 | 0.19 | 0.17 | 0.15 | 0.30 | 0.26 | 0.30 | 0.27 | 0.12 | 0.00 | 0.08 | 0.07 | −0.17 | 0.25 | 0.20 | 0.02 | 0.05 | 0.22 | 0.28 | 0.33 | 0.16 | 0.05 | 0.19 | 0.25 | 1.00 | ||||||||||||
| Q32 | 0.17 | 0.16 | 0.16 | 0.38 | 0.30 | 0.37 | 0.40 | 0.18 | 0.23 | 0.20 | 0.26 | 0.23 | 0.20 | 0.24 | 0.18 | 0.02 | 0.07 | 0.08 | −0.14 | 0.24 | 0.24 | −0.03 | 0.08 | 0.21 | 0.16 | 0.30 | 0.18 | 0.03 | 0.23 | 0.18 | 0.87 | 1.00 | |||||||||||
| Q33 | 0.28 | 0.37 | 0.29 | 0.25 | 0.25 | 0.34 | 0.27 | 0.32 | 0.34 | 0.32 | 0.31 | 0.24 | 0.10 | 0.16 | 0.31 | 0.41 | 0.28 | 0.28 | −0.15 | 0.12 | 0.20 | 0.34 | 0.43 | 0.46 | 0.47 | 0.09 | 0.56 | 0.47 | 0.43 | 0.33 | 0.25 | 0.22 | 1.00 | ||||||||||
| Q34 | 0.15 | 0.25 | 0.15 | 0.24 | 0.17 | 0.29 | 0.24 | 0.24 | 0.22 | 0.27 | 0.30 | 0.22 | 0.02 | 0.14 | 0.31 | 0.20 | 0.11 | 0.29 | −0.04 | 0.05 | 0.10 | 0.22 | 0.29 | 0.36 | 0.30 | 0.01 | 0.41 | 0.61 | 0.44 | 0.29 | 0.11 | 0.15 | 0.65 | 1.00 | |||||||||
| Q35 | 0.07 | 0.24 | 0.18 | 0.33 | 0.24 | 0.57 | 0.34 | 0.25 | 0.23 | 0.45 | 0.26 | 0.20 | 0.27 | 0.22 | 0.20 | 0.17 | 0.10 | 0.19 | −0.10 | 0.25 | 0.29 | 0.27 | 0.24 | 0.38 | 0.37 | 0.11 | 0.40 | 0.36 | 0.42 | 0.25 | 0.24 | 0.26 | 0.45 | 0.38 | 1.00 | ||||||||
| Q36 | 0.09 | 0.14 | 0.10 | 0.23 | 0.17 | 0.20 | 0.07 | 0.12 | 0.03 | 0.25 | 0.18 | 0.10 | 0.05 | 0.10 | 0.26 | 0.23 | 0.05 | 0.21 | −0.07 | 0.11 | 0.20 | 0.14 | 0.21 | 0.27 | 0.28 | −0.10 | 0.29 | 0.44 | 0.20 | 0.23 | −0.07 | −0.05 | 0.48 | 0.66 | 0.26 | 1.00 | |||||||
| Q37 | 0.31 | 0.38 | 0.17 | 0.26 | 0.37 | 0.38 | 0.31 | 0.24 | 0.32 | 0.36 | 0.38 | 0.19 | 0.12 | 0.20 | 0.29 | 0.26 | 0.17 | 0.29 | −0.06 | 0.26 | 0.30 | 0.24 | 0.36 | 0.37 | 0.41 | 0.06 | 0.57 | 0.57 | 0.44 | 0.41 | 0.11 | 0.06 | 0.62 | 0.70 | 0.51 | 0.61 | 1.00 | ||||||
| Q38 | 0.50 | 0.45 | 0.12 | 0.45 | 0.32 | 0.37 | 0.37 | 0.41 | 0.40 | 0.36 | 0.35 | 0.37 | 0.15 | 0.21 | 0.26 | 0.22 | 0.27 | 0.24 | −0.08 | 0.16 | 0.29 | 0.19 | 0.40 | 0.49 | 0.36 | 0.13 | 0.59 | 0.50 | 0.42 | 0.29 | 0.31 | 0.28 | 0.54 | 0.40 | 0.41 | 0.23 | 0.45 | 1.00 | |||||
| Q39 | 0.42 | 0.52 | 0.23 | 0.38 | 0.39 | 0.43 | 0.41 | 0.44 | 0.57 | 0.45 | 0.49 | 0.30 | 0.12 | 0.32 | 0.41 | 0.26 | 0.39 | 0.37 | 0.05 | 0.19 | 0.30 | 0.33 | 0.49 | 0.55 | 0.47 | 0.15 | 0.64 | 0.55 | 0.40 | 0.40 | 0.40 | 0.33 | 0.67 | 0.45 | 0.38 | 0.22 | 0.55 | 0.83 | 1.00 | ||||
| Q40 | 0.42 | 0.57 | 0.14 | 0.36 | 0.39 | 0.37 | 0.34 | 0.53 | 0.57 | 0.50 | 0.48 | 0.35 | 0.07 | 0.23 | 0.28 | 0.18 | 0.37 | 0.37 | 0.12 | 0.23 | 0.30 | 0.28 | 0.47 | 0.56 | 0.41 | 0.23 | 0.62 | 0.50 | 0.43 | 0.40 | 0.39 | 0.39 | 0.56 | 0.32 | 0.35 | 0.14 | 0.48 | 0.79 | 0.94 | 1.00 | |||
| Q41 | 0.11 | 0.20 | 0.14 | 0.24 | 0.44 | 0.23 | 0.18 | 0.15 | 0.31 | 0.30 | 0.23 | 0.12 | −0.01 | 0.23 | 0.15 | 0.07 | 0.13 | 0.10 | −0.03 | 0.24 | 0.21 | 0.11 | 0.18 | 0.19 | 0.26 | 0.22 | 0.19 | 0.28 | 0.16 | 0.28 | 0.15 | 0.12 | 0.35 | 0.29 | 0.27 | 0.16 | 0.33 | 0.31 | 0.49 | 0.49 | 1.00 | ||
| Q42 | 0.22 | 0.37 | 0.21 | 0.31 | 0.24 | 0.41 | 0.24 | 0.34 | 0.40 | 0.38 | 0.38 | 0.33 | 0.29 | 0.25 | 0.34 | 0.25 | 0.30 | 0.21 | −0.18 | 0.16 | 0.32 | 0.14 | 0.42 | 0.31 | 0.35 | 0.21 | 0.34 | 0.19 | 0.44 | 0.32 | 0.25 | 0.28 | 0.41 | 0.19 | 0.42 | 0.11 | 0.37 | 0.43 | 0.52 | 0.50 | 0.30 | 1.00 | |
| Q43 | 0.26 | 0.33 | 0.22 | 0.22 | 0.30 | 0.36 | 0.28 | 0.16 | 0.23 | 0.19 | 0.28 | 0.29 | 0.19 | 0.25 | 0.29 | 0.29 | 0.25 | 0.32 | 0.00 | 0.02 | 0.07 | 0.23 | 0.22 | 0.34 | 0.27 | 0.05 | 0.33 | 0.29 | 0.27 | 0.15 | 0.29 | 0.34 | 0.46 | 0.32 | 0.26 | 0.25 | 0.33 | 0.37 | 0.59 | 0.54 | 0.21 | 0.34 | 1.00 |
Item reduction was performed by identifying pairs or triplets of questions with high correlation coefficients (> 0.7). When this was the case, item redundancy led to such items being dropped (n = 7):
-
Q2 It is important to listen to patients about their experiences of care.
-
Q3 It is a clinician’s responsibility to gather evidence of patient’s experience of care.
-
Q25 Patient surveys help identify actions that might be taken to improve services.
-
Q31 I am concerned about my workplace’s reputation as a result of the kinds of patient feedback currently being made public.
-
Q34 I have made changes to my clinical practice as a result of patient feedback.
-
Q38 It is necessary to have patient feedback to improve the overall quality of health care.
-
Q40 Gathering feedback is beneficial to the health service.
Two items were then removed from the analysis as they were specific to a UK setting and thus would not allow the scale to be used in an international context. These items were:
-
Q7 I have reservations about patient feedback currently received via online patient feedback mechanisms (such as NHS Choices, Patient Opinion or I Want Great Care).
-
Q11 It is beneficial to receive patient feedback via online patient feedback mechanisms (such as NHS Choices or Patient Opinion or I Want Great Care).
A further seven items were dropped from the analysis as they required responders to have received patient feedback to answer the question. There was a fear that if these questions were left in the resulting score would tell us about whether or not a responder had received feedback from patients rather than how they valued it. The dropped items were:
-
Q20 I find it difficult to interpret the results of patient surveys.
-
Q21 I find it easy to understand patient feedback.
-
Q28 I have questioned the way I do things after receiving feedback from patients.
-
Q29 Receiving feedback from patients has improved my confidence at work.
-
Q30 Receiving patient feedback via patient surveys is a positive experience.
-
Q35 I find it difficult to make changes to my practice as a result of patient survey results.
-
Q36 I have made changes to the way I consult as a result of patient surveys.
Finally one further item, ‘I feel it is a clinician’s responsibility to ensure evidence of patients’ experience of care is collected’, was dropped as cognitive testing had highlighted some concerns and we continued to have reservations about its relevance and performance. This left a final set of 26 items (Table 77), on which we performed an exploratory factor analysis.
| Question number | Question |
|---|---|
| Q1 | Patients should have the opportunity to provide feedback on their experiences of health care provided |
| Q4 | I have reservations about patient feedback received via complaints |
| Q5 | I have reservations about patient feedback currently received via surveys |
| Q6 | I have reservations about patient feedback currently received via patient forums or participant groups |
| Q8 | It is beneficial to receive patient feedback via complaints |
| Q9 | It is beneficial to receive patient feedback via surveys |
| Q10 | It is beneficial to receive patient feedback via patient forums or participant groups |
| Q12 | I think social media such as Twitter or Facebook are a useful route for receiving patient feedback |
| Q13 | Responders to patient surveys are representative of my patient population |
| Q14 | Feedback from current patient surveys is usually reliable |
| Q15 | Patients usually have grounds for the complaints they make |
| Q16 | Patients are able to provide useful feedback on my clinical skills |
| Q17 | Patients are able to provide useful feedback on my communication skills |
| Q18 | Patients are able to provide useful feedback on organisational issues, such as appointment systems |
| Q22 | Receiving supported feedback (such as with a facilitator) following surveys of patients’ experience of care would help me gain a better understanding of the results |
| Q23 | I feel patient feedback is useful for individual performance assessment, such as in appraisal or revalidation |
| Q24 | Patient surveys help identify areas for service improvement |
| Q26 | Patient anonymity limits the usefulness of most patient feedback |
| Q27 | I can make good use of patient feedback |
| Q32 | I am concerned about my individual reputation as a result of patient feedback being made public |
| Q33 | Patient feedback can improve the clinical quality of care I provide |
| Q37 | I am likely to make changes to my individual practice as a result of patient feedback |
| Q39 | Patient feedback is an important mechanism of quality improvement |
| Q41 | Data collection for large, representative patient surveys is not a good use of resources |
| Q42 | I value feedback from colleagues more than feedback from patients |
| Q43 | Making patient feedback publicly available is beneficial to other patients |
The scree plot from the exploratory factor analysis is shown in Figure 27. There is not an obvious elbow from which to select the most appropriate number of factors but it suggests something in the range 2–5. Using a cut-off point of eigenvalues of > 1 would suggest using three factors. The Akaike information criterion suggests a seven-factor model whereas the Bayesian information criterion suggests a two-factor model. With no clear-cut solution we explored two-, three-, four- and five-factor models. Examination of these models suggested that the four-factor model gave the most interpretable solution, with the four factors covering the themes of changes in response to patient feedback, benefits of patient feedback, validity of patient feedback mechanisms and reservations about patient feedback.
FIGURE 27.
Scree plot of eigenvalues from exploratory factor analysis.
The factor loadings from the four-factor model are shown in Table 78. Here, for each of the factors we list the items that loaded on that factor with a factor loading of at least 0.3. The items are ordered in decreasing order of loading, with shaded items having a factor loading of ≥ 0.4. Two items did not load onto any factor (and thus are not listed in the table). To produce a final set of questions following this factor analysis we employed the following logic. We started with all items with a factor loading of ≥ 0.4. Q9 was also included because it formed a run of questions asking about the benefits of different modes of feedback and this run mirrored similar questions in the reservations group. We also removed Q26, ‘Patient anonymity limits the usefulness of most patient feedback’, as we continued to have reservations about its overall performance, considering all sources of evidence from testing. The final proposed set of questions is shown in Box 7.
| Item | Wording | Factor loadings | |||
|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | ||
| Factor 1: changes in response to patient feedback | |||||
| Q33 | Patient feedback can improve the clinical quality of care I provide | 0.7764 | –0.0317 | –0.0606 | –0.0233 |
| Q39 | Patient feedback is an important mechanism of quality improvement | 0.7159 | 0.1149 | –0.0706 | 0.1222 |
| Q37 | I am likely to make changes to my individual practice as a result of patient feedback | 0.6678 | –0.0213 | 0.0245 | –0.0156 |
| Q27 | I can make good use of patient feedback | 0.6377 | 0.0508 | 0.1077 | –0.1083 |
| Q43 | Making patient feedback publicly available is beneficial to other patients | 0.5051 | –0.0479 | 0.0635 | 0.1011 |
| Q24 | Patient surveys help identify areas for service improvement | 0.4613 | 0.1329 | 0.232 | –0.0677 |
| Q41 | Data collection for large, representative patient surveys is not a good use of resources | 0.3702 | –0.0862 | –0.198 | 0.382 |
| Q16 | Patients are able to provide useful feedback on my clinical skills | 0.3543 | 0.1009 | 0.1944 | –0.2482 |
| Factor 2: benefits of patient feedback | |||||
| Q8 | It is beneficial to receive patient feedback via complaints | 0.0104 | 0.7204 | –0.1479 | 0.0961 |
| Q10 | It is beneficial to receive patient feedback via patient forums or participant groups | 0.0523 | 0.6435 | –0.0004 | 0.0478 |
| Q18 | Patients are able to provide useful feedback on organisational issues, such as appointment systems | 0.1279 | 0.4559 | 0.187 | –0.351 |
| Q9 | It is beneficial to receive patient feedback via surveys | 0.1952 | 0.3754 | 0.1049 | 0.1693 |
| Q15 | Patients usually have grounds for the complaints they make | 0.1344 | 0.3666 | 0.1476 | 0.078 |
| Q17 | Patients are able to provide useful feedback on my communication skills | 0.131 | 0.3187 | 0.2275 | –0.2768 |
| Factor 3: validity of patient feedback mechanisms | |||||
| Q13 | Responders to patient surveys are representative of my patient population | –0.0818 | 0.0177 | 0.6805 | 0.0132 |
| Q14 | Feedback from current patient surveys is usually reliable | 0.1037 | –0.1032 | 0.6413 | 0.1678 |
| Q5 | I have reservations about patient feedback currently received via surveys | 0.1722 | –0.1078 | 0.3378 | 0.5236 |
| Factor 4: reservations about patient feedback | |||||
| Q4 | I have reservations about patient feedback received via complaints | 0.0244 | 0.2591 | 0.0995 | 0.5434 |
| Q5 | I have reservations about patient feedback currently received via surveys | 0.1722 | –0.1078 | 0.3378 | 0.5236 |
| Q32 | I am concerned about my individual reputation as a result of patient feedback being made public | 0.0141 | 0.0942 | 0.0604 | 0.4994 |
| Q26 | Patient anonymity limits the usefulness of most patient feedback | –0.1337 | 0.1633 | 0.0671 | 0.4706 |
| Q6 | I have reservations about patient feedback currently received via patient forums or participant groups | 0.1118 | 0.2575 | 0.2215 | 0.4081 |
| Q41 | Data collection for large, representative patient surveys is not a good use of resources | 0.3702 | –0.0862 | –0.198 | 0.382 |
Patient feedback can improve the clinical quality of care I provide.
Patient feedback is an important mechanism of quality improvement.
I am likely to make changes to my individual practice as a result of patient feedback.
I can make good use of patient feedback.
Making patient feedback publicly available is beneficial to other patients.
Patient surveys help identify areas for service improvement.
Benefits of patient feedbackIt is beneficial to receive patient feedback via complaints.
It is beneficial to receive patient feedback via patient forums or participant groups.
It is beneficial to receive patient feedback via surveys.
Patients are able to provide useful feedback on organisational issues, such as appointment systems.
Validity of patient feedback mechanismsResponders to patient surveys are representative of my patient population.
Feedback from current patient surveys is usually reliable.
Reservations about patient feedbackI have reservations about patient feedback received via complaints.
I have reservations about patient feedback currently received via patient forums or participant groups.
I have reservations about patient feedback currently received via surveys.
I am concerned about my individual reputation as a result of patient feedback being made public.
We fitted a preliminary confirmatory factor analysis model to these data using our proposed final questionnaire. It had only a moderately good fit with RMSEA = 0.84, CFI = 0.85 and Tucker–Lewis Index315 = 0.82.
Annex A: content validation of initial item pool
| Number | Wording of item | Domain of construct: ‘value of patient feedback’ | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Right of patients to give feedback | Responsibility for organising patient feedback | Preferred mode of patient feedback | Credibility of patient feedback | Utility of patient feedback | Impact of receiving patient feedback | Changes to individual practice | Changes to overall quality of health care | Overall value of patient feedback | ||
| 1 | Patients should have the opportunity to provide feedback on their experiences of care provided | ✗ | ||||||||
| 2 | It is important to listen to patients about their experiences of care | ✗ | ||||||||
| 3 | It is a clinician’s responsibility to gather evidence of patients’ experience of care | ✗ | ||||||||
| 4 | It is the responsibility of clinical commissioning groups and hospitals to gather evidence of patients’ experience of care | ✗ | ||||||||
| 5 | It is the responsibility of NHS England to gather evidence of patients’ experience of care | ✗ | ||||||||
| 6 | I have reservations about patient feedback received via complaints and compliments | ✗ | ||||||||
| 7 | I have reservations about patient feedback received via surveys | ✗ | ||||||||
| 8 | I have reservations about patient feedback received via patient forums or participant groups | ✗ | ||||||||
| 9 | I have reservations about patient feedback received via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | ✗ | ||||||||
| 10 | It is beneficial to receive patient feedback via complaints and compliments | ✗ | ||||||||
| 11 | It is beneficial to receive patient feedback via surveys | ✗ | ||||||||
| 12 | It is beneficial to receive patient feedback via patient forums or participant groups | ✗ | ||||||||
| 13 | It is beneficial to receive patient feedback via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | ✗ | ||||||||
| 14 | I am interested in learning more about receiving patient feedback through social media such as Twitter or Facebook | ✗ | ||||||||
| 15 | Responders to patient surveys are representative of my patient population | ✗ | ||||||||
| 16 | Data from patient surveys are valid and reliable | ✗ | ||||||||
| 17 | Patients always have grounds for the complaints they make | ✗ | ||||||||
| 18 | Patient feedback via patient forums or participant groups is a reliable indicator of patient concerns | ✗ | ||||||||
| 19 | Patients who use online patient feedback mechanisms (such as NHS Choices or Patient Opinion) are representative of my patient population | ✗ | ||||||||
| 20 | I trust patient feedback received via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | ✗ | ||||||||
| 21 | Patients are able to provide useful feedback on my clinical skills | ✗ | ||||||||
| 22 | Patients are able to provide useful feedback on my interpersonal skills | ✗ | ||||||||
| 23 | Patients are able to provide useful feedback on administrative and organisational issues | ✗ | ||||||||
| 24 | Patient survey data are more valuable if they include benchmarking | ✗ | ||||||||
| 25 | Free-text comments are the most useful aspect of patient surveys | ✗ | ||||||||
| 26 | I find it difficult to interpret the results of patient surveys | ✗ | ||||||||
| 27 | I find it easy to understand patient feedback | ✗ | ||||||||
| 28 | Receiving supported feedback (such as with a facilitator) following surveys of patients’ experience of care would help me gain a better understanding of the results | ✗ | ||||||||
| 29 | Feedback from patient surveys is presented in a timely way | ✗ | ||||||||
| 30 | Patient feedback on specific identifiable care experiences is more helpful than general opinions | ✗ | ||||||||
| 31 | Patient feedback is useful for individual performance assessment, such as in appraisal or revalidation | ✗ | ||||||||
| 32 | Patient surveys indicate what needs to be done to improve | ✗ | ||||||||
| 33 | Patient anonymity limits the usefulness of most patient feedback | ✗ | ||||||||
| 34 | I know how to act on anonymous patient feedback | ✗ | ||||||||
| 35 | I know how to act on feedback received from a named patient | ✗ | ||||||||
| 36 | I can make good use of patient feedback | ✗ | ||||||||
| 37 | I have doubted my competence after receiving feedback from patients | ✗ | ||||||||
| 38 | Receiving feedback from patients has improved my confidence at work | ✗ | ||||||||
| 39 | Receiving patient feedback via patient surveys is a positive experience | ✗ | ||||||||
| 40 | Receiving a complaint from a patient can impact on my ability to work effectively | ✗ | ||||||||
| 41 | Engaging with patient feedback requires a lot of energy | ✗ | ||||||||
| 42 | I worry about my workplace’s reputation as a result of patient feedback being made public | ✗ | ||||||||
| 43 | I worry about my individual reputation as a result of patient feedback being made public | ✗ | ||||||||
| 44 | Acting on patient feedback can improve the clinical quality of care I provide | ✗ | ||||||||
| 45 | Acting on patient feedback can improve the interpersonal quality of care I provide | ✗ | ||||||||
| 46 | Acting on patient feedback can improve the organisation and administration of the care I provide | ✗ | ||||||||
| 47 | Receiving patient feedback can improve my relationship with patients | ✗ | ||||||||
| 48 | I have made changes to my individual practice as a result of patient feedback | ✗ | ||||||||
| 49 | I am likely to make changes to my individual practice as a result of patient feedback | ✗ | ||||||||
| 50 | It is necessary to have patient feedback to improve the overall quality of health care | ✗ | ||||||||
| 51 | Listening to patients will lead to useful changes to health care | ✗ | ||||||||
| 52 | Patient feedback is an important mechanism of quality improvement | ✗ | ||||||||
| 53 | Gathering patient feedback is beneficial to the health service | ✗ | ||||||||
| 54 | Data collection for large, representative patient surveys is cost-effective | ✗ | ||||||||
| 55 | Online patient feedback mechanisms (such as NHS Choices or Patient Opinion) are beneficial to the health service | ✗ | ||||||||
| 56 | I value feedback from colleagues more than feedback from patients | ✗ | ||||||||
Annex B: comments from expert reviewers
| Item pool following initial generation of items | Comments from NL | Comments from PB | Comments from JC | Comments from FG | Revised item (round 1) |
|---|---|---|---|---|---|
| 1. Patients should have the opportunity to provide feedback on their experiences of care provided | Patients should have the opportunity to provide feedback on their experiences of care provided | Patients should have the opportunity to provide feedback on their experiences of care | Patients should have the opportunity to provide feedback on their experiences of care provided | Patients should have the opportunity to provide feedback on their experiences of care provided | Patients should have the opportunity to provide feedback on their experiences of health care provided |
| 2. It is important to listen to patients about their experiences of care | It is important to listen to patients about their experiences of care | It is important to listen to patients about their experiences of care | It is important to listen to patients about their experiences of care | It is important to listen to patients about their experiences of care | It is important to listen to patients about their experiences of care |
| 3. It is a clinician’s responsibility to gather evidence of patients’ experience of care | It is a clinician’s responsibility to gather evidence of patients’ experience of care | It is a clinician’s responsibility to gather evidence of patients’ experience of care | It is a clinician’s responsibility to gather evidence of patients’ experience of care | It is a clinician’s responsibility to gather evidence of patients’ experience of care | It is a clinician’s responsibility to gather evidence of patients’ experience of care |
| 6. I have reservations about patient feedback received via complaints and compliments | I have reservations about patient feedback received via complaints and compliments | I have reservations about patient feedback received via complaints and compliments | I have reservations about patient feedback received via complaints and compliments | I have reservations about patient feedback received via complaints and compliments | I have reservations about patient feedback received via complaints and compliments |
| 7. I have reservations about patient feedback received via surveys | I have reservations about patient feedback received via surveys | I have reservations about patient feedback received via surveys | I have reservations about patient feedback received via surveys | I have reservations about patient feedback received via surveys | I have reservations about patient feedback received via surveys |
| 8. I have reservations about patient feedback received via patient forums or participant groups | I have reservations about patient feedback received via patient forums or participant groups | I have reservations about patient feedback received via patient forums or participant groups | I have reservations about patient feedback received via patient forums or participant groups | I have reservations about patient feedback received via patient forums or participant groups | I have reservations about patient feedback received via patient forums or participant groups |
| 9. I have reservations about patient feedback received via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | I have reservations about patient feedback received via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | I have reservations about patient feedback received via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | I have reservations about patient feedback received via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | I have reservations about patient feedback received via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | I have reservations about patient feedback received via online patient feedback mechanisms (such as NHS Choices, Patient Opinion or I Want Great Care) |
| 10. It is beneficial to receive patient feedback via complaints and compliments | It is beneficial to receive patient feedback via complaints and compliments | It is beneficial to receive patient feedback via complaints and compliments | It is beneficial to receive patient feedback via complaints and compliments | It is beneficial to receive patient feedback via complaints and compliments | It is beneficial to receive patient feedback via complaints and compliments |
| 11. It is beneficial to receive patient feedback via surveys | It is beneficial to receive patient feedback via surveys | It is beneficial to receive patient feedback via surveys | It is beneficial to receive patient feedback via surveys | It is beneficial to receive patient feedback via surveys | It is beneficial to receive patient feedback via surveys |
| 12. It is beneficial to receive patient feedback via patient forums or participant groups | It is beneficial to receive patient feedback via patient forums or participant groups | It is beneficial to receive patient feedback via patient forums or participant groups | It is beneficial to receive patient feedback via patient forums or participant groups | It is beneficial to receive patient feedback via patient forums or participant groups | It is beneficial to receive patient feedback via patient forums or participant groups |
| 13. It is beneficial to receive patient feedback via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | It is beneficial to receive patient feedback via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | It is beneficial to receive patient feedback via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | It is beneficial to receive patient feedback via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | It is beneficial to receive patient feedback via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) | It is beneficial to receive patient feedback via online patient feedback mechanisms (such as NHS Choices or Patient Opinion) |
| 14. I am interested in learning more about receiving patient feedback through social media such as Twitter or Facebook | I am interested in learning more about receiving patient feedback through social media such as Twitter or Facebook | I am interested in learning more about receiving patient feedback through social media such as Twitter or Facebook | I am interested in learning more about receiving patient feedback through social media such as Twitter or Facebook | I am interested in learning more about receiving patient feedback through social media such as Twitter or Facebook | I think social media such as Twitter or Facebook are a useful route for receiving patient feedback |
| 15. Responders to patient surveys are representative of my patient population | Responders to patient surveys are representative of my patient population | Responders to patient surveys are representative of my patient population | Responders to patient surveys are representative of my patient population | Responders to patient surveys are representative of my patient population | Responders to patient surveys are representative of my patient population |
| 16. Data from patient surveys are valid and reliable | Data from patient surveys are valid and reliable | Data from patient surveys are valid and reliable | Data from patient surveys are valid and reliable | Data from patient surveys are valid and reliable | Feedback from patient surveys is usually accurate |
| 17. Patients always have grounds for the complaints they make | Patients always have grounds for the complaints they make | Patients always have grounds for the complaints they make | Patients always have grounds for the complaints they make | Patients always have grounds for the complaints they make | Patients usually have grounds for the complaints they make |
| 21. Patients are able to provide useful feedback on my clinical skills | Patients are able to provide useful feedback on my clinical skills | Patients are able to provide useful feedback on my clinical skills | Patients are able to provide useful feedback on my clinical skills | Patients are able to provide useful feedback on my clinical skills | Patients are able to provide useful feedback on my clinical skills |
| 22. Patients are able to provide useful feedback on my interpersonal skills | Patients are able to provide useful feedback on my interpersonal skills | Patients are able to provide useful feedback on my interpersonal skills | Patients are able to provide useful feedback on my interpersonal skills | Patients are able to provide useful feedback on my interpersonal skills | Patients are able to provide useful feedback on my communication skills |
| 23. Patients are able to provide useful feedback on administrative and organisational issues | Patients are able to provide useful feedback on administrative and organisational issues | Patients are able to provide useful feedback on administrative and organisational issues | Patients are able to provide useful feedback on administrative and organisational issues | Patients are able to provide useful feedback on administrative and organisational issues | Patients are able to provide useful feedback on organisational issues, such as appointment systems |
| 24. Patient survey data are more valuable if they include benchmarking | Patient survey data are more valuable if they include benchmarking | Patient survey data are more valuable if they include benchmarking | Patient survey data are more valuable if they include benchmarking | Patient survey data are more valuable if they include comparison with how others are doing | Patient survey data are more valuable if they include comparison with how others are doing |
| 25. Free-text comments are the most useful aspect of patient surveys | Free-text comments are the most useful aspect of patient surveys | Free-text comments are the most useful aspect of patient surveys | Free-text comments are the most useful aspect of patient surveys | Free-text comments are the most useful aspect of patient surveys | Free-text comments are the most useful aspect of patient surveys |
| 26. I find it difficult to interpret the results of patient surveys | I find it difficult to interpret the results of patient surveys | I find it difficult to interpret the results of patient surveys | I find it difficult to interpret the results of patient surveys | I find it difficult to interpret the results of patient surveys | I find it difficult to interpret the results of patient surveys |
| 27. I find it easy to understand patient feedback | I find it easy to understand patient feedback | I find it easy to understand patient feedback | I find it easy to understand patient feedback | I find it easy to understand patient feedback | I find it easy to understand patient feedback |
| 28. Receiving supported feedback (such as with a facilitator) following surveys of patients’ experience of care would help me gain a better understanding of the results | Receiving supported feedback (such as with a facilitator) following surveys of patients’ experience of care would help me gain a better understanding of the results | Receiving supported feedback (such as with a facilitator) following surveys of patients’ experience of care would help me gain a better understanding of the results | Receiving supported feedback (such as with a facilitator) following surveys of patients’ experience of care would help me gain a better understanding of the results | Receiving supported feedback (such as with a facilitator) following surveys of patients’ experience of care would help me gain a better understanding of the results | Receiving supported feedback (such as with a facilitator) following surveys of patients’ experience of care would help me gain a better understanding of the results |
| 31. Patient feedback is useful for individual performance assessment, such as in appraisal or revalidation | Patient feedback is useful for individual performance assessment, such as in appraisal or revalidation | Patient feedback is useful for individual performance assessment, such as in appraisal or revalidation | Patient feedback is useful for individual performance assessment, such as in appraisal or revalidation | Patient feedback is useful for individual performance assessment, such as in appraisal or revalidation | Patient feedback is useful for individual performance assessment, such as in appraisal or revalidation |
| 32. Patient surveys indicate what needs to be done to improve | Patient surveys indicate what needs to be done to improve | Patient surveys indicate what needs to be done to improve | Patient surveys indicate what needs to be done to improve | Patient surveys indicate what needs to be done to improve | Patient surveys help identify areas for service improvement |
| Patient surveys help identify actions that might be taken to improve services | |||||
| 33. Patient anonymity limits the usefulness of most patient feedback | Patient anonymity limits the usefulness of most patient feedback | Patient anonymity limits the usefulness of most patient feedback | Patient anonymity limits the usefulness of most patient feedback | Patient anonymity limits the usefulness of most patient feedback | Patient anonymity limits the usefulness of most patient feedback |
| 34. I know how to act on anonymous patient feedback | I know how to act on anonymous patient feedback | I know how to act on anonymous patient feedback | I know how to act on anonymous patient feedback | I know how to act on anonymous patient feedback | I usually know how to act on anonymous patient feedback |
| 35. I know how to act on feedback received from a named patient | I know how to act on feedback received from a named patient | I know how to act on feedback received from a named patient | I know how to act on feedback received from a named patient | I know how to act on feedback received from a named patient | I usually know how to act on feedback received from a named patient |
| 36. I can make good use of patient feedback | I can make good use of patient feedback | I can make good use of patient feedback | I can make good use of patient feedback | I can make good use of patient feedback | I can make good use of patient feedback |
| 37. I have doubted my competence after receiving feedback from patients | I have doubted my competence after receiving feedback from patients | I have doubted my competence after receiving feedback from patients | I have doubted my competence after receiving feedback from patients | I have doubted my competence after receiving feedback from patients | I have doubted my competence after receiving feedback from patients |
| 38. Receiving feedback from patients has improved my confidence at work | Receiving feedback from patients has improved my confidence at work | Receiving feedback from patients has improved my confidence at work | Receiving feedback from patients has improved my confidence at work | Receiving feedback from patients has improved my confidence at work | Receiving feedback from patients has improved my confidence at work |
| 39. Receiving patient feedback via patient surveys is a positive experience | Receiving patient feedback via patient surveys is a positive experience | Receiving patient feedback via patient surveys is a positive experience | Receiving patient feedback via patient surveys is a positive experience | Receiving patient feedback via patient surveys is a positive experience | Receiving patient feedback via patient surveys is a positive experience |
| 40. Receiving a complaint from a patient can impact on my ability to work effectively | Receiving a complaint from a patient can impact on my ability to work effectively | Receiving a complaint from a patient can impact on my ability to work effectively | Receiving a complaint from a patient can impact on my ability to work effectively | Receiving a complaint from a patient can impact on my ability to work effectively | Receiving a complaint from a patient can impact on my ability to work effectively |
| 41. Engaging with patient feedback requires a lot of energy | Engaging with patient feedback requires a lot of energy | Engaging with patient feedback requires a lot of energy | Engaging with patient feedback requires a lot of energy | Engaging with patient feedback requires a lot of energy | Engaging with patient feedback requires a lot of time |
| 42. I worry about my workplace’s reputation as a result of patient feedback being made public | I worry about my workplace’s reputation as a result of patient feedback being made public | I worry about my workplace’s reputation as a result of patient feedback being made public | I worry about my workplace’s reputation as a result of patient feedback being made public | I worry about my workplace’s reputation as a result of patient feedback being made public | I worry about my workplace’s reputation as a result of patient feedback being made public |
| 43. I worry about my individual reputation as a result of patient feedback being made public | I worry about my individual reputation as a result of patient feedback being made public | I worry about my individual reputation as a result of patient feedback being made public | I worry about my individual reputation as a result of patient feedback being made public | I worry about my individual reputation as a result of patient feedback being made public | I worry about my individual reputation as a result of patient feedback being made public |
| 44. Acting on patient feedback can improve the clinical quality of care I provide | Acting on patient feedback can improve the clinical quality of care I provide | Acting on patient feedback can improve the clinical quality of care I provide | Acting on patient feedback can improve the clinical quality of care I provide | Acting on patient feedback can improve the clinical quality of care I provide | Patient feedback can improve the clinical quality of care I provide |
| 45. Acting on patient feedback can improve the interpersonal quality of care I provide | Acting on patient feedback can improve the interpersonal quality of care I provide | Acting on patient feedback can improve the interpersonal quality of care I provide | Acting on patient feedback can improve the interpersonal quality of care I provide | Acting on patient feedback can improve the interpersonal quality of care I provide | Patient feedback can improve my communication skills |
| 46. Acting on patient feedback can improve the organisation and administration of the care I provide | Acting on patient feedback can improve the organisation and administration of the care I provide | Acting on patient feedback can improve the organisation and administration of the care I provide | Acting on patient feedback can improve the organisation and administration of the care I provide | Acting on patient feedback can improve the organisation and administration of the care I provide | Patient feedback can improve organisational issues, such as appointment systems |
| 47. Receiving patient feedback can improve my relationship with patients | Receiving patient feedback can improve my relationship with patients | Receiving patient feedback can improve my relationship with patients | Receiving patient feedback can improve my relationship with patients | Receiving patient feedback can improve my relationship with patients | Receiving patient feedback can improve the way I relate to my patients |
| 48. I have made changes to my individual practice as a result of patient feedback | I have made changes to my individual practice as a result of patient feedback | I have made changes to my individual practice as a result of patient feedback | I have made changes to my individual practice as a result of patient feedback | I have made changes to my individual practice as a result of patient feedback | I have made changes to my clinical practice as a result of patient feedback |
| I find it difficult to make tangible changes to my practice as a result of patient survey results | |||||
| I have made changes to the way I consult in response to feedback from patient surveys | |||||
| 49. I am likely to make changes to my individual practice as a result of patient feedback | I am likely to make changes to my individual practice as a result of patient feedback | I am likely to make changes to my individual practice as a result of patient feedback | I am likely to make changes to my individual practice as a result of patient feedback | I am likely to make changes to my individual practice as a result of patient feedback | I am likely to make changes to my individual practice as a result of patient feedback |
| 50. It is necessary to have patient feedback to improve the overall quality of health care | It is necessary to have patient feedback to improve the overall quality of health care | It is necessary to have patient feedback to improve the overall quality of health care | It is necessary to have patient feedback to improve the overall quality of health care | It is necessary to have patient feedback to improve the overall quality of health care | It is necessary to have patient feedback to improve the overall quality of health care |
| 52. Patient feedback is an important mechanism of quality improvement | Patient feedback is an important mechanism of quality improvement | Patient feedback is an important mechanism of quality improvement | Patient feedback is an important mechanism of quality improvement | Patient feedback is an important mechanism of quality improvement | Patient feedback is an important mechanism of quality improvement |
| 53. Gathering patient feedback is beneficial to the health service | Gathering patient feedback is beneficial to the health service | Gathering patient feedback is beneficial to the health service | Gathering patient feedback is beneficial to the health service | Gathering patient feedback is beneficial to the health service | Gathering feedback is beneficial to the health service |
| 54. Data collection for large, representative patient surveys is cost-effective | Data collection for large, representative patient surveys is cost-effective | Data collection for large, representative patient surveys is cost-effective | Data collection for large, representative patient surveys is cost-effective | Data collection for large, representative patient surveys is cost-effective | Data collection for large, representative patient surveys is not a good use of resources |
| 56. I value feedback from colleagues more than feedback from patients | I value feedback from colleagues more than feedback from patients | I value feedback from colleagues more than feedback from patients | I value feedback from colleagues more than feedback from patients | I value feedback from colleagues more than feedback from patients | I value feedback from colleagues more than feedback from patients |
| New item. Making patient feedback publicly available is beneficial to other patients | Making patient feedback publicly available is beneficial to other patients | ||||
| I have a good idea of how satisfied my patients are with the care I provide |
Annex C: item groupings for cognitive interviews
Patients should have the opportunity to provide feedback on their experiences of health care provided.
It is a clinician’s responsibility to gather evidence of patients’ experience of care.
I have reservations about patient feedback received via complaints and compliments.
I have reservations about patient feedback received via online patient feedback mechanisms (such as NHS Choices, Patient Opinion or I Want Great Care).
It is beneficial to receive patient feedback via complaints and compliments.
It is beneficial to receive patient feedback via online patient feedback mechanisms (such as NHS Choices or Patient Opinion).
Feedback from patient surveys is usually accurate.
Patients are able to provide useful feedback on my communication skills.
Free-text comments are better than replies to closed questions.
Receiving supported feedback (such as with a facilitator) following surveys of patients’ experience of care would help me gain a better understanding of the results.
Patient surveys help identify actions that might be taken to improve services.
I usually know how to act on feedback received from a named patient.
Receiving feedback from patients has improved my confidence at work.
Engaging with patient feedback requires a lot of time.
Patient feedback can improve the clinical quality of care I provide.
Receiving patient feedback can improve the way I relate to my patients.
I have made changes to the way I consult in response to feedback from patient surveys.
Patient feedback is an important mechanism of quality improvement.
I value feedback from colleagues more than feedback from patients.
Patients should have the opportunity to provide feedback on their experiences of health care provided.
It is a clinician’s responsibility to gather evidence of patients’ experience of care.
I have reservations about patient feedback received via surveys.
It is beneficial to receive patient feedback via surveys.
I think social media such as Twitter or Facebook are a useful route for receiving patient feedback.
Patients usually have grounds for the complaints they make.
Patients are able to provide useful feedback on organisational issues, such as appointment systems.
I find it difficult to interpret the results of patient surveys.
Patient feedback is useful for individual performance assessment, such as in appraisal or revalidation.
Patient anonymity limits the usefulness of most patient feedback.
I can make good use of patient feedback.
Receiving patient feedback via patient surveys is a positive experience.
I am concerned about my workplace’s reputation as a result of patient feedback being made public.
Patient feedback can improve my communication skills.
I have made changes to my clinical practice as a result of patient feedback.
I am likely to make changes to my individual practice as a result of patient feedback.
Gathering feedback is beneficial to the health service.
Making patient feedback publicly available is beneficial to other patients.
It is important to listen to patients about their experiences of care.
It is a clinician’s responsibility to gather evidence of patients’ experience of care.
I have reservations about patient feedback received via patient forums or participant groups.
It is beneficial to receive patient feedback via patient forums or participant groups.
Responders to patient surveys are representative of my patient population.
Patients are able to provide useful feedback on my clinical skills.
Patient survey data are more valuable if they include comparison with how others are doing.
I find it easy to understand patient feedback.
Patient surveys help identify areas for service improvement.
I usually know how to act on anonymous patient feedback.
I have doubted aspects of my competence after receiving feedback from patients.
Receiving a complaint from a patient can impact on my ability to work effectively.
I am concerned about my individual reputation as a result of patient feedback being made public.
Patient feedback can improve organisational issues, such as appointment systems.
I find it difficult to make tangible changes to my practice as a result of patient survey results.
It is necessary to have patient feedback to improve the overall quality of health care.
Data collection for large, representative patient surveys is not a good use of resources.
I have a good idea of how satisfied my patients are with the care I provide.
Annex D: Value of Patient Feedback scale – cognitive interview topic guide
Preamble
(Obtain written consent.)
(Thank participant for taking part.)
The IMPROVE research team at the University of Cambridge and the University of Exeter is developing a scale to assess the value of patient feedback to clinicians. The aim of the interview today is to help develop the scale by identifying any items in the scale that might be difficult to answer or understand. We will work through the questionnaire, focusing on questions within the scale that the research team anticipate being problematic, and ask you questions about how you went about answering it. We will be audio recording the interview for later transcription but all information collected today will be held confidentially.
Probes (to be used as appropriate for 15–20 pre-identified problematic items, after each item answered)
-
Retrieval: I noticed you hesitated before you answered the question, can you explain why?
-
Comprehension: What does the term ‘e.g. patient feedback mechanisms’ mean to you?
-
Comprehension: How would you rephrase this question in your own words? (seek clarification when required)
-
Comprehension: What do you understand by ‘e.g. patient feedback mechanisms’?
-
Confidence judgement: How sure/confident of your answer are you?
-
Response: Were you able to find your first answer to the question from the response options shown? If not, what additional response option would you like to see?
At end of scale
Finally, were there any other questions on the scale that you had difficulty answering?
If yes, then probe:
-
Can you explain why do you think that was?
-
How would you rephrase this question in your own words?
Thank participant for taking part again.
Annex E: pilot survey – outline of questions
What is the purpose of this research?
To better understand health-care professionals’ attitudes to receiving feedback from patients, we are developing a new measurement instrument, the Value of Patient Feedback (VOP) scale. This questionnaire will be used to evaluate how doctors and nurses engage with the concept of patient feedback and how helpful and relevant they feel it is both to their practice and to the wider health-care system.
The research is being undertaken by the School of Clinical Medicine at the University of Cambridge and the University of Exeter Medical School (see end for full contact details).
What does taking part involve?
To help us develop a robust instrument, we are running a pilot survey of our questionnaire. We are inviting UK registered doctors and nurses to participate in the study by completing a series of online questions. We will use these responses to refine the VOP scale, ready for use in research into the impact of patient feedback on doctors and nurses.
As a thank you, we are running a prize draw offering participants the chance to win a Kindle Paperwhite. If you wish to enter into the competition you will need to provide us with your GMC or NMC number, name and e-mail address when requested (see end for full terms and conditions).
How will my information be used?
All information collected during the course of this study will be kept strictly confidential. The anonymous questionnaires will be analysed by the research team. All data will be stored securely by the University of Cambridge and Exeter Medical School.
You are free to complete the survey giving no contact details. However, should you wish to be entered into the prize draw you will need to complete the contact details section, including your GMC or NMC number to confirm that you are a registered doctor or nurse.
Research team
For queries relating to this survey, please contact:
[Researcher name]
Lead Researcher
[Researcher contact details]
Survey questions
Part 1: VOP scale items
[Full list of VOP items to be taken forward to pilot testing.]
Part 2: some questions about you
-
Your professional role:
□ Nurse
□ Doctor
-
What is your grade, band or job title? (e.g. doctor grade FY2, nursing band 6)? Please write in:
-
Do you have direct clinical contact with patients?
□ Yes
□ No
-
Are you male or female?
□ Male
□ Female
-
How old are you?
□ Under 18
□ 18 to 24
□ 25 to 34
□ 35 to 44
□ 45 to 54
□ 55 to 64
□ 65 to 74
□ 75 or over
-
What is your ethnic group?
-
White:
□ English/Welsh/Scottish/Northern Irish/British
□ Irish
□ Gypsy or Irish Traveller
□ Any other white background, please describe
-
-
Mixed/multiple ethnic groups
□ White and black Caribbean
□ White and black African
□ White and Asian
□ Any other mixed/multiple ethnic background, please describe
-
Asian/Asian British
□ Indian
□ Pakistani
□ Bangladeshi
□ Chinese
□ Any other Asian background, please describe
-
Black/African/Caribbean/black British
□ African
□ Caribbean
□ Any other black/African/Caribbean background, please describe
-
Other ethnic group
□ Arab
□ Any other ethnic group, please describe
Appendix 6 Chapter 11: supplementary tables
| Questionnaire item | n (total = 1396) | % |
|---|---|---|
| How easy was it to contact the out-of-hours GP service by telephone? | ||
| Very easy | 884 | 63.32 |
| Fairly easy | 404 | 28.94 |
| Not very easy | 42 | 3.01 |
| Not at all easy | 15 | 1.07 |
| Don’t know/didn’t make contact by telephone | 38 | 2.72 |
| Not answered | 13 | 0.93 |
| How do you feel about how quickly you received care from the out-of-hours GP service? | ||
| It was quicker than expected | 605 | 43.34 |
| It was about right | 613 | 43.91 |
| It took too long | 152 | 10.89 |
| Don’t know/doesn’t apply | 13 | 0.93 |
| Not answered | 13 | 0.93 |
| Did you have confidence and trust in the out-of-hours health-care professional you consulted with? | ||
| Yes, definitely | 928 | 66.48 |
| Yes, to some extent | 347 | 24.86 |
| No, not at all | 75 | 5.37 |
| Don’t know/can’t say | 29 | 2.08 |
| Not answered | 17 | 1.22 |
| Overall, how would you describe your experience of the out-of-hours GP service? | ||
| Very good | 772 | 55.3 |
| Good | 417 | 29.87 |
| Neither good nor poor | 91 | 6.52 |
| Poor | 44 | 3.15 |
| Very poor | 39 | 2.79 |
| Not answered | 33 | 2.36 |
| How do you rate [how long it took your call to be answered]? | ||
| Very poor | 30 | 2.15 |
| Poor | 36 | 2.58 |
| Acceptable | 349 | 25.00 |
| Good | 460 | 32.95 |
| Excellent | 432 | 30.95 |
| Not answered | 89 | 6.38 |
| Please rate the helpfulness of the call operator | ||
| Very poor | 27 | 1.93 |
| Poor | 20 | 1.43 |
| Acceptable | 215 | 15.40 |
| Good | 554 | 39.68 |
| Excellent | 496 | 35.53 |
| Not answered | 84 | 6.02 |
| Please rate the extent to which you felt the call operator listened to you | ||
| Very poor | 18 | 1.29 |
| Poor | 24 | 1.72 |
| Acceptable | 212 | 15.19 |
| Good | 549 | 39.33 |
| Excellent | 513 | 36.75 |
| Not answered | 80 | 5.73 |
| How do you rate [how long it took for a health professional to call you back]? | ||
| Very poor | 34 | 2.44 |
| Poor | 105 | 7.52 |
| Acceptable | 320 | 22.92 |
| Good | 353 | 25.29 |
| Excellent | 364 | 26.07 |
| Not applicable | 192 | 13.75 |
| Not answered | 28 | 2.01 |
| Were you happy with the type of care you received? | ||
| Yes | 1187 | 85.03 |
| No | 149 | 10.67 |
| Not answered | 60 | 4.30 |
| How do you rate [the length of your consultation with the health professional]? | ||
| Very poor | 43 | 3.08 |
| Poor | 54 | 3.87 |
| Acceptable | 302 | 21.63 |
| Good | 481 | 34.46 |
| Excellent | 452 | 32.38 |
| Not answered | 64 | 4.58 |
| Please rate the thoroughness of the consultation | ||
| Very poor | 34 | 2.44 |
| Poor | 50 | 3.58 |
| Acceptable | 213 | 15.26 |
| Good | 519 | 37.18 |
| Excellent | 525 | 37.61 |
| Not applicablea | 9 | 0.64 |
| Not answered | 46 | 3.30 |
| Please rate the accuracy of the diagnosis | ||
| Very poor | 35 | 2.51 |
| Poor | 66 | 4.73 |
| Acceptable | 202 | 14.47 |
| Good | 486 | 34.81 |
| Excellent | 461 | 33.02 |
| Not applicablea | 71 | 5.09 |
| Not answered | 75 | 5.37 |
| Please rate the treatment you were given | ||
| Very poor | 44 | 3.15 |
| Poor | 58 | 4.15 |
| Acceptable | 181 | 12.97 |
| Good | 424 | 30.37 |
| Excellent | 450 | 32.23 |
| Not applicablea | 161 | 11.53 |
| Not answered | 78 | 5.59 |
| Please rate the advice and information you were given | ||
| Very poor | 42 | 3.01 |
| Poor | 64 | 4.58 |
| Acceptable | 197 | 14.11 |
| Good | 498 | 35.67 |
| Excellent | 513 | 36.75 |
| Not applicablea | 16 | 1.15 |
| Not answered | 66 | 4.73 |
| Please rate the warmth of the health professional’s manner | ||
| Very poor | 32 | 2.29 |
| Poor | 53 | 3.8 |
| Acceptable | 173 | 12.39 |
| Good | 438 | 31.38 |
| Excellent | 647 | 46.35 |
| Not applicablea | 4 | 0.29 |
| Not answered | 49 | 3.51 |
| Please rate the extent to which you felt listened to | ||
| Very poor | 34 | 2.44 |
| Poor | 50 | 3.58 |
| Acceptable | 163 | 11.68 |
| Good | 473 | 33.88 |
| Excellent | 624 | 44.70 |
| Not applicablea | 3 | 0.21 |
| Not answered | 49 | 3.51 |
| Please rate the extent to which you felt things were explained to you | ||
| Very poor | 32 | 2.29 |
| Poor | 63 | 4.51 |
| Acceptable | 183 | 13.11 |
| Good | 463 | 33.17 |
| Excellent | 583 | 41.76 |
| Not applicablea | 17 | 1.22 |
| Not answered | 55 | 3.94 |
| Please rate the respect you were shown | ||
| Very poor | 29 | 2.08 |
| Poor | 23 | 1.65 |
| Acceptable | 147 | 10.53 |
| Good | 418 | 29.94 |
| Excellent | 724 | 51.86 |
| Not applicablea | 10 | 0.72 |
| Not answered | 45 | 3.22 |
| Item | n | Item–test correlationa | Item–rest correlationb | Average inter-item correlation | Alpha |
|---|---|---|---|---|---|
| Ease of access | 1345 | 0.697 | 0.451 | 0.538 | 0.777 |
| Timeliness of care | 1370 | 0.732 | 0.500 | 0.505 | 0.754 |
| Confidence and trust | 1350 | 0.790 | 0.598 | 0.442 | 0.704 |
| Overall experience | 1363 | 0.875 | 0.743 | 0.348 | 0.615 |
| Scale | 0.458 | 0.772 |
Appendix 7 Chapters 2 and 3: practice information sheet
Appendix 8 Chapters 2 and 3: general practitioner information sheet
Appendix 9 Chapters 2 and 3: general practitioner consent form
Appendix 10 Chapters 2 and 3: patient full information sheet
Appendix 11 Chapters 2 and 3: patient summary information sheet
Appendix 12 Chapters 2 and 3: patient video consent form
Appendix 13 Chapters 2 and 3: general practitioner questionnaire
Appendix 14 Chapters 2 and 3: patient questionnaire
Appendix 15 Chapter 2: patient interview information sheet
Appendix 16 Chapter 2: patient interview consent form
Appendix 17 Chapter 2: video elicitation interview topic guide
Appendix 18 Chapter 6: computer-assisted personal interview schedule
Appendix 19 Chapter 7: participant information sheet
Appendix 20 Chapter 7: participant consent form
Appendix 21 Chapter 7: focus group topic guide
Improving patient experience in primary care (individual general practitioner feedback)
(Researcher 1 leads the focus group session while researcher 2 assists and scribes)
Focus group topic guide: practice teams
Researchers answer any questions and ensure each participant has had an opportunity to read the information leaflet and has signed the consent form.
Introduction
Thank you for agreeing to take part in our study. The purpose of this focus group is to listen to your thoughts and ideas so that we can get an understanding of what you know about surveys of patient experience of general practice/primary care – such as the national GP Patient Survey, local surveys of patient experience or patient surveys for appraisal or revalidation. We are interested in how your practice might respond to scores based on patient surveys and how these surveys and/or scores may be used to inform any changes your practice is considering. We are also interested in how practices might respond to surveys carried out of patients’ experiences of individual doctors – in general, not focusing on doctors from this particular practice.
We would like the discussion to be as informal as possible. There are no right or wrong answers and everything you say will be in confidence. Feel free to ask if something is not clear or if you want to add anything or even change your mind about something – this is more of a conversation to explore your views rather than a formal meeting. Before we start may I ask if everyone is happy to have this focus group audio recorded?
(Turn on recorder)
Please would everyone introduce themselves by giving their name and role in the practice.
-
Are the results of patient surveys (such as the GP Patient Survey or surveys you have done in your own practice) circulated within your practice and, if so, to whom?
-
GPs
-
Nurses
-
Receptionists
-
Other members of staff (health-care assistants, etc.)
-
-
Which results are circulated – summary/good/bad/nothing?
-
Overall, what do you think the survey results are saying to your practice?
-
Are you aware of the score of your practice on any particular questions in any surveys that have been carried out?
-
Which questions?
-
Why did you focus on these?
-
Were there any surprises?
-
Where did you access this information?
-
Was the information easy to understand/interpret? For you? For your patients?
-
Have you compared these with other practices? If so, which practices, over which questions and why?
-
-
Have the scores made you or your colleagues want to change anything?
-
Do you currently intend to change anything in response to scores on the GP Patient Survey or surveys you have done in your own practice?
-
What would make you want to change?
-
This research is about how practices may respond to patient feedback about individual doctors, so the next questions focus on this issue:
-
Do you think that individual GP scores following a patient experience survey could have an impact on the practice as a whole – perhaps on the way things are done?
-
In a positive or a negative way? Why?
-
-
I’d like you to try and think of an average type of practice – say four or five doctors and 7000 patients. Imagine that a patient experience survey carried out for individual doctors in the practice had identified that two of the doctors had scores that were not very good on communication skills with patients.
-
Can you think of any ways that the practice as a whole in that type of situation might address the issues raised by these results?
-
Do you think it would have an impact on the rest of the practice staff if a GP had a low individual score in such an area?
-
If yes, in what way?
-
-
Now I’d like you to try and think again of an average type of practice – say four or five doctors and 7000 patients. Imagine that a patient experience survey carried out for individual doctors in the practice showed that patients found it really hard to get to see one or two of the doctors, that is, patients were not able to see the doctor of their choice.
-
Can you think of any ways that the practice as a whole in that type of situation might address the issues raised by these results?
-
Do you think it would have an impact on the rest of the practice staff if the practice had a low score in such an area?
-
If yes, in what way?
-
In your experience, what priority do patients give to seeing the same person when they come to the surgery?
-
-
Do you think that, over time, surveys of patient experience that focus on individual doctors’ skills might affect the attitude of doctors towards their patients – or the attitude of patients towards their doctors?
-
If so, in what ways?
-
Do patients ever talk to you about any of the patient feedback surveys?
-
-
In general, what do you think about surveys of patient experience?
-
At practice level
-
At individual doctor level
-
What might be the strengths of such surveys?
-
And weaknesses?
-
-
Can you think of anything else that we have not discussed about this topic that you would like to raise?
-
Thanks
-
Reassurance of confidentiality
-
Reassurance of removal of all identifying material in records and publications
-
Any further questions: use details on information sheet
-
Thank you very much for your time.
Appendix 22 Chapter 8: participant information sheet
Appendix 23 Chapter 8: participant consent form
Appendix 24 Chapter 8: interview topic guide
Improving patient experience in primary care (individual practitioner feedback)
Interview topic guide: individual general practitioners
(Researcher answers any questions and ensures that the participant has had an opportunity to read the information leaflet and has signed the consent form)
Thank you for agreeing to take part in our study. The purpose of this interview is to listen to your thoughts and ideas so that we can get an understanding of how you feel about getting feedback from your patients as a result of using patient surveys – such as the national GP Patient Survey, local surveys of patient experience or patient surveys for appraisal or revalidation. We are interested in how you might respond to such feedback and how it may be used to inform any changes you might consider. We would like the discussion to be as informal as possible. There are no right or wrong answers and everything you say will be in confidence. Feel free to ask if something is not clear or if you want to add anything or even change your mind about something – remember, this is more of a conversation to explore your views rather than a formal interview. Before we start may I ask if you are happy to have this interview audio recorded?
-
What experience have you had of patient feedback obtained through surveys of patient experience?
-
For example, how closely have you been involved with the practice results from the national GP Patient Survey or surveys you have done in your own practice?
-
-
Thinking about the feedback that your practice has had from patients, can you tell me your thoughts and ideas/reflections about the feedback?
-
Did anything surprise you?
-
What things stood out particularly for you?
-
-
You have recently taken part in an individual-level patient survey and had a chance to have a look at your personal results.
-
Can you tell me your thoughts and ideas/reflections about the feedback?
-
Did anything surprise you?
-
What things stood out particularly for you?
-
Have your own scores made you want to change anything?
-
As an individual – what sorts of things?
-
In terms of the practice – what sorts of things?
-
If there was something you wanted to change, how would you go about it? What sort of help might you need?
-
-
Do you think it is different to get feedback as an individual GP rather than for the practice as a whole?
-
In what ways?
-
Do you prefer one method over the other?
-
What are the strengths and weaknesses of each approach?
-
-
In your opinion, how do you think feedback from your patients could best be used to develop care?
-
Have any patients ever mentioned their participation in (any) patient surveys to you – either about completing the survey or about the results?
-
Do you think it is possible or likely that over time the survey (at an individual level) might affect your relationship with your patients?
-
If so, in what ways?
-
-
Do you think it is possible or likely that over time the survey (at an individual level) might affect the way you might do things?
-
If so, in what ways?
-
-
Would you mind if a member of the research team contacts you in a few months’ time to find out how you have changed your practice or behaviour as an individual?
-
If so, what would be the best method of contacting you?
-
-
Can you think of anything else that we have not discussed about this topic that you would like to raise?
Thank you for talking to me today. Remember, not only will all your details be kept confidential, but any identifying information relating to you or anyone else you spoke about will be deleted from the record that we will keep and use. If there is anything you would like to ask after you have left today, remember you can contact us using the details on the information sheet. Thank you very much for your time.
Appendix 25 Chapter 9: practice information sheet
Appendix 26 Chapter 9: patient information sheet
Appendix 27 Chapter 10: practice information sheet
Appendix 28 Chapter 10: practice consent form
Appendix 29 Chapter 10: practice profile questionnaire
Appendix 30 Chapter 10: The Value of Patient Feedback scale
Appendix 31 Chapter 10: structured observation sheets
Appendix 32 Chapter 10: patient exit surveys
Appendix 33 Chapter 10: example focus group/interview topic guide
Appendix 34 Chapter 10: facilitator interview topic guide
Appendix 35 Chapter 11: provider information sheet
Appendix 36 Chapter 11: patient survey information sheet
Appendix 37 Chapter 11: patient questionnaire
Appendix 38 Chapter 11: patient interview information sheet
Appendix 39 Chapter 11: patient interview consent form
Appendix 40 Chapter 11: patient interview topic guide
Appendix 41 Chapter 11: service provider information sheet
Appendix 42 Chapter 11: service provider consent form
Appendix 43 Chapter 11: service provider interview topic guide
List of abbreviations
- ANOVA
- analysis of variance
- CFI
- comparative fit index
- CI
- confidence interval
- CQC
- Care Quality Commission
- CRT
- Customer Research Technology
- DIF
- differential item functioning
- EQ-5D-3L
- EuroQol-5 Dimensions three-level version
- GCRS
- Global Consultation Rating Scale
- GMC
- General Medical Council
- GP
- general practitioner
- ICC
- intraclass correlation coefficient
- IMD
- Index of Multiple Deprivation
- IMPROVE
- improving patient experience in primary care
- IQR
- interquartile range
- NIHR
- National Institute for Health Research
- NPT
- normalisation process theory
- NQR
- National Quality Requirement
- NRES
- National Research Ethics Service
- ONS
- Office for National Statistics
- OPQ
- Out-of-hours Patient Questionnaire
- OR
- odds ratio
- PCA
- principal component analysis
- PCT
- primary care trust
- PFC
- patient feedback questionnaire on consultation skills
- PPG
- Patient Participation Group
- PPI
- patient and public involvement
- QOF
- Quality and Outcomes Framework
- RCT
- randomised controlled trial
- RMSEA
- root-mean-square error of approximation
- RTF
- real-time feedback
- SD
- standard deviation
- SHA
- strategic health authority
- VOP
- Value of Patient Feedback