Developing and evaluating a tool to measure general practice productivity: a multimethod study

Overall performance measurement

The measurement of performance in health care has long been a contentious issue. The pressures of competing priorities mean that there is often no consensus over the definition of what constitutes high-quality performance. 28 In health systems that operate for profit, profitability of a unit may provide one suitable measure of performance; however, in the NHS and other systems that do not operate on a for-profit basis, the measurement of financial performance is both more complex and less appropriate.

To capture overall performance in health care (whether of a single organisation or the system as a whole), a range of different indicators is undoubtedly necessary. Often these will take the form of a ‘balanced scorecard’ – a set of measures designed to capture all the main areas of performance. For example, areas covered may include indicators relating to patient health, mortality, safety, patient satisfaction and the extent to which targets are met. The precise types of measures will depend on the context and nature of the units being studied; however, there should certainly be alignment between the objectives of the unit and the measurements used. 29,30

Productivity is a particularly difficult area of performance measurement. It is always a challenge for health-care providers and administrators to be able to produce as much as possible with the resources available. In the NHS, for example, the Wanless report31 identified that in future years the NHS would need to be able to make better use of its resources in order to maintain the same level of service – and that was at a time of relative prosperity and growth in the NHS. In times of relative austerity and uncertainty, the necessity becomes even greater.

A classical definition of productivity is simply the ratio of outputs to inputs, and is often defined in simple financial terms. However, in health care this definition is not sufficient: the simple measurement of financial outputs does not usually take account of the quality of care delivered. The most common definition used in health care is the ratio of outputs to inputs, adjusted for quality.

Although attempts have been made to measure quality-adjusted outputs directly, these have tended to focus on secondary care, and do not generally account for the wide range of potential data, but instead prefer routinely collected outcome data. 18,19 Quality can refer to a mixture of things, including health outcomes, safety and patient experience. It seems evident that, particularly for primary care, the full extent of quality cannot be measured without taking into account the views of patients. 20

Performance measurement in primary care

Primary care in general (and general practice in particular) covers a huge range of activity, with practitioners needing a wide enough scope of knowledge to be able to deal with all presenting problems, whether these are dealt with directly within the primary care setting or referred on to secondary care or other services. 2,32 Some models of overall primary care effectiveness do exist, although they do not focus on productivity, and are not geared towards the NHS situation in particular. However, the dimensions identified by Kringos et al. 33 in particular (i.e. primary care processes determined by access in addition to continuity, co-ordination and comprehensiveness of care, and outcomes determined by quality of care, efficiency of care and equity in health) give a useful benchmark for comparison.

The role of the GP within this setting is key to its success, and the nature of the consultation between GP and patient has itself been the subject of much scrutiny, particularly with regard to its potential to explore broader health concerns than that initially presented by the patient. For example, Stott and Davis34 presented a four-point framework to help GPs achieve greater breadth in consultations, covering management of presenting problems, modification of help-seeking behaviour, management of continuing problems and opportunistic health promotion. By engaging in all four of these aspects, rather than merely dealing with the primary issue, GPs can seek to improve general health and avoid future concerns. Mehay35 went further than this, describing 15 separate models of doctor–patient consultation, and Pawlikowska et al. 36 undertook an analysis of a variety of consultation types. Although they36 did not advocate any one particular model, their analysis suggested some key themes including establishing a rapport, appropriate questioning style, active listening, empathy, summarising, reflection, appropriate language, silence, responding to cues, patient’s ideas, concerns and expectations, sharing information, social and psychological context, clinical examination, partnership, honesty, safety netting/follow-up and housekeeping. In particular, Pawlikowska et al. 36 concluded that excellent communication skills alone are not enough. In general, GPs also have a key role in managing patients’ uncertainty, and act as a key operator at the boundary between different agencies. 37

With this in mind, the measurement of performance in general practice needs to embrace this complexity and reflect the broad activity undertaken by GPs and other practice staff, including the work of practice nurses and other clinicians working under the umbrella of the general practice; however, this is far from straightforward. One attempt to do so is the provision of profiles of general practices by Public Health England. However, these include only certain areas of performance and are updated infrequently. A 2015 review38 conducted by the Health Foundation examined indicators that were then in use in the NHS. Although there were a multiplicity of sources of indicators available, the over-riding conclusion was that the accessibility of these indicators was poor, particularly from the patients’ perspective. Considering that the rationale for publication of indicators may include improvement, patient choice/voice, and accountability, it recommended that a single web location be developed to provide access to these indicators (rather than relying on very different web sources, such as NHS Choices, CQC ratings, MyNHS, Public Health England and NHS Digital), but, even then, there would be significant areas of performance that were not covered by reliable indicators. The review also suggested that composite indicators should not be developed and gave six reasons for this: (1) aggregation masks aspects of quality of care, (2) it was suggested that a composite index would provide little value over and above the CQC ratings, (3) patients and service users are not a homogeneous group, (4) any selection and weighting of indicators would be highly contentious, (5) the number of robust indicators available is extremely small and not comprehensive and (6) there would not be enough detail for professionals to pinpoint areas for improvement. 38

Some of these arguments are more persuasive than others. It is certainly true that aggregation can mask specific aspects of care, but this does not imply that overall performance is not a meaningful concept. In particular, if an overall performance index can be developed that also allows examination of specific areas within it, it could service both needs. We disagree that no value can be added to the CQC ratings with other composite indicators; as discussed later in this section, CQC inspections (leading to ratings) are made too infrequently and do not offer the detail that could be given by a more comprehensive index. It is also true that patients and service users are a very varied group, and any attempt to measure overall performance of a practice should not imply that the performance is the same for all groups. The contentious nature of the choice of indicators, particularly in the light of the small number of robust indicators, is a critical issue. This suggests that any specific choice of indicators might favour one type of practice over another. The difficulty of balance in performance measurement is something discussed by multiple authors. 39–41 For this reason, any overall performance index may not be so useful as a direct comparison between practices, but instead as a longitudinal tracker within practices – more as an improvement tool than a performance management tool.

Baker and England42 have presented a framework covering many of these outcomes: both final outcomes (including mortality, morbidity, disease episodes, quality of life, adverse incidents, equity, patient satisfaction, costs and time off work/school) and intermediate outcomes (e.g. clinical outcomes, such as immunisation/screening, health behaviours, resource utilisation and patient experience, and practitioner-related outcomes, such as work satisfaction). For each outcome (or type of outcome) there is both a degree of importance of the outcome to the overall perception of effectiveness, and a degree of influence that the general practice can have over it. For example, mortality is a very important final outcome, and, although primary care can influence mortality, the influence is small, with the characteristics of patients being very much the most powerful predictors of mortality rates. On the other hand, patient satisfaction with care is increasingly seen as an important outcome, and the delivery of primary care will certainly influence it much more directly.

One of the greatest challenges for general practice is in the effective allocation of resources. A 2017 study by Watson et al. 43 used a value-based health-care framework to propose how resources can be allocated more effectively. As well as quality, safety, efficiency and cost-effectiveness, it includes optimality (balancing of improvements in health against cost of improvement). Optimality requires evidence and shared decision-making with individual patients. The authors argue that primary care has an essential role in delivering optimality and, therefore, value. However, they also point to the lack of readiness currently within the primary care system: ‘primary care measurement systems need to be developed to generate data that can assist with the identification of optimality’. 43 Of interest, among its suggestions for things to do more/less of, it suggests fewer health checks and fewer unnecessary appointments. It does, however, recommend more social prescribing, more patient self-care, better integration of services and a higher overall allocation of resources into primary care.

This is closely tied to the area of efficiency, which has had its own section of literature within health care generally and primary care in particular. Of particular note here is the work using data envelopment analysis, which uses multidimensional geometric methods to compare the inputs and outputs of a unit and to establish a comparative index of efficiency. In particular, Pelone et al. 44–46 have undertaken to do this in different primary care settings. However, this method is simply one of using other indicators to create a composite measure of efficiency (or productivity); it does not calculate a specific index, but instead depends on the comparability of units. Therefore, the choice of appropriate indicators is still of paramount importance.

The nature of primary care means that a certain number of clinical and patient activity indicators are inevitable, and these should cover as many of the major conditions and patient types as possible, including public health priorities. However, there are other, more general areas that also need to be considered. One of these is patient safety. However, patient safety within primary care is not something that is measured in any standard form. Lydon et al. 47 conducted a systematic review of measurement tools for the proactive assessment of patient safety in general practice. Of the 56 studies identified by this systematic review, 34 used surveys/interviews, 14 used a form of patient chart audit and 7 used practice assessment checklists; there were a handful of other tools that were not repeated across studies. Nothing was discovered that was either commonplace or appropriately sophisticated. Similarly, Hatoun et al. 48 undertook a systematic review of patient safety measures in adult primary care. They found 21 articles, including a total of 182 safety measures, and classified these into six dimensions: (1) medication management, (2) sentinel events, (3) care co-ordination, (4) procedures and treatment, (5) laboratory testing and monitoring and (6) facility structures/resources. However, the types of measures were not dissimilar to those found by Lydon et al. 47 An earlier review by Ricci-Cabello et al. 49 undertook a similar exercise and reached similar conclusions (albeit on a smaller scale). Therefore, the inclusion of patient safety within a broader index will pose a significant challenge.

It also seems important, following on from the Baker and England42 and Dixon et al. 38 frameworks, that the experience of the patient is given a key place within any overall index. Patient satisfaction and experience measures are commonplace in health care, but are used to a different extent in different scenarios; in the NHS, routine data collection is common, with the annual GP patient survey and all practices using a (minimal) Friends and Family Test (FFT) on an ongoing basis, as well as gathering qualitative feedback via patient reference groups (PRGs). Moreover, the advent of technology has started to change how these data are collected. A recent National Institute for Health Research (NIHR) Programme Grants for Applied Research study examined patient survey scores in detail, aiming to understand the data and how general practices respond to low scores, looked at some specifics [e.g. black and minority ethnic (BME) patient scores and OOH care] and carried out a randomised controlled trial (RCT) of an intervention to improve patient experience. This intervention involved real-time feedback via a touchscreen on exit; encouragement of patients boosted response rates hugely. The major conclusions were that a variety of feedback mechanisms should be used, and certainly not reliance on postal surveys. In addition to satisfaction with the care provided, access is a key issue for patients, but it is increasingly under pressure. 50

Overall, a wide variety of measures of performance have been used in general practice settings (both in the NHS and elsewhere), and these continue to evolve without there yet being much in the way of definitive best practice. However, it is important to consider next what is currently used in the NHS at a national stakeholder level, and to determine in what ways these do and do not satisfy the needs of practices, patients and other stakeholders.

General practice performance in the NHS

As with much of the NHS, there has been significant emphasis on measuring performance in primary care, even though this is sometimes less straightforward than for other parts of the service (such as the acute sector). Since the introduction of the GMS and PMS contracts for general practice in 2004, the main vehicle for measuring the performance of practices, and for determining at least part of the payments due to practices, has been the QOF.

Under the QOF, each practice needs to submit data annually as evidence of how they are meeting various targets. Detailed rules are provided about how each score should be calculated, with most being derived directly from clinical information systems. For example, one of the indicators under the area ‘secondary prevention of coronary heart disease (CHD)’ is the percentage of patients with CHD in whom the last blood pressure reading (measured in the preceding 12 months) was ≤ 150/90 mmHg. This percentage, which can be extracted using a specific query from the practice’s clinical system, is worth up to 17 points (out of a possible 45 points for the domain, or 558 QOF points in total) and is maximised when the percentage is between 53% and 93%. The sum for all 77 indicators (across 25 areas in two domains) is calculated for a practice and this is converted to a payment made to the practice as part of their overall funding. Thus, the financial incentive for practices to perform well on QOF is strong.

In one sense, therefore, this already provides an overall index of performance for a practice. However, the content of the QOF indicators is a matter of some contention. Table 1 summarises the domains, areas and indicators in use in the NHS year 2017/18, although this detail has changed somewhat over the years. Originally, the QOF indicators were developed by the Department of Health and Social Care, with a total of 146 indicators across four domains: clinical, organisational, patient experience and additional services. Since 2009, the changes made to the QOF have been the responsibility of NICE. As with much of NICE’s work, this was supposed to ensure that decisions were based on a strong evidence base. However, within general practice this is not always straightforward, and the evidence (or lack of evidence) underlying changes to QOF scores has been criticised as relying too much on expert opinion. 52

In particular, there have been major changes to the structure of the QOF. Of the initial four domains, the clinical domain has been substantially expanded, but the organisational and patient-experience domains have been removed, principally because of a lack of good-quality data in these areas, rather than because of a perceived lack of importance. This, therefore, represents a significant weakness within the current QOF; it is often perceived as being imbalanced (either in terms of what is perceived as most important or in terms of the actual work of general practices). 53 In addition, and probably less controversially, the additional services domain has been subsumed within a broader public health domain.

Even more contentious, however, is the effectiveness of the QOF as an index and its usefulness as a methodology. This has attracted a substantial level of research and comment in recent years. Forbes et al. 54 undertook a review of research examining the effects of the QOF. They found that the introduction of the QOF was associated with a modest slowing of both the increase in emergency admissions and the increase in consultations in severe mental illness, and modest improvements in diabetes mellitus care. However, there was no clear evidence of causality. Furthermore, there was no evidence of any effect on mortality, on integration or co-ordination of care, on holistic care, self-care or patient experience. The work of this research team in the field of CHD confirms the lack of evidence of an effect on mortality and offers a potential explanation in that the QOF has concentrated clinical attention on the management of patients with diagnosed conditions instead of a population approach to primary care that actively identifies patients with undiagnosed conditions. 55 Specifically, levels of detection of hypertension predict premature mortality, greater detection being associated with lower mortality, whereas QOF indicators did not predict mortality. 56

Marshall and Roland57 argued that the QOF is highly divisive and has become increasingly unpopular with GPs. Using financial incentives has diverted focus from the interpersonal elements that are important in consultations, and care for single diseases has been prioritised over holistic care. Their review57 of observational studies suggested that there had been modest improvements in some areas and a small decrease in emergency admissions in the incentivised areas, but no overall effect on patient mortality. Counterbalancing the limited evidence of improvements is a rising administrative workload associated with the QOF. They conclude that although the QOF may have had some benefits, it has failed to achieve what was intended in terms of driving improvements in health. 57 In a similar vein, Ryan et al. 58 performed a longitudinal population study on data from 1994 to 2010 using a difference-in-difference analysis, studying mortality in areas that had been prioritised by the QOF. They found that the introduction of the QOF was not associated with any improvement in mortality, either in specific conditions or overall. This is perhaps unsurprising given the marginal link between general practice and mortality overall.

Thorne10 revealed that health inequalities had not been reduced by the introduction of the QOF. Ashworth and Gulliford59 used findings from QOF-based research to argue that an increase in funding for general practice was needed, and that a salaried GP workforce could assist in improving the situation.

Ruscitto et al. 60 studied the issue of payments for individual diseases by modelling possible scenarios in which comorbidities attracted different payments. They found that, although there were substantial differences in the resulting payments, the existing system favours more deprived areas, because of the higher number of patients with multiple morbidities, which therefore attracts multiple payments. Kontopantelis et al. 61 examined a different problematic area within the QOF: that of exemptions (patients who could be excluded from QOF calculations). They found that, across 644 practices, there was evidence that the odds of exemption increased with age, deprivation and multimorbidity. At the same time, exempted patients were more likely to die in the following year. Thus, combining these findings with those of Ruscitto et al. ,60 it appears that the QOF system may favour practices in more deprived areas, but not necessarily to the benefit of the patients in those areas. The fidelity of QOF reporting has also been called into question: Martin et al. 62 uncovered differences in the monitoring of physical health conditions for patients with major mental illness.

Overall, the research into the effects of the QOF is somewhat inconclusive. There is certainly little clear evidence that it is effective in improving population health, and yet there is substantial evidence about unintended consequences. Combined with the concerns about the coverage of the QOF (in particular its current lack of any patient experience, access or organisational indicators), the research points to a system that is, at best, imperfect and, at worst, a wasteful exercise. Indeed, even Simon Stevens, Chief Executive of NHS England, stated in October 2016 that the usefulness of QOF was nearing its end, and would be phased out in new GP contracts. 63 However, as yet, there have been no firm plans announced about how QOF will be replaced or altered in future years.

Given that the QOF cannot be relied on to measure the effectiveness of practices, it is unsurprising that national bodies, particularly regulators, have turned to different measures. In 2014, the CQC embarked on a series of in-depth inspections of general practices. Inspections are conducted by independent teams that always include a GP, as well as various other people, often including an ‘expert by experience’ (patient/service user). As of May 2017, on the most recent inspection, 86% of practices were rated as ‘good’ and 4% as ‘outstanding’; 8% were rated as ‘requires improvement’ and 2% as ‘inadequate’.

Within this overall rating, however, there are five dimensions rated: safe, effective, caring, responsive to people’s needs and well led. Six different population groups are considered separately: older people; people with long-term conditions; families, children and young people; working-age people; people whose circumstances make them vulnerable; and people experiencing poor mental health. Safety is the main concern: on this dimension, 13% of practices were rated as ‘requires improvement’ and 2% as ‘inadequate’ (although this is a substantial improvement from the first inspection, when these figures were 27% and 6%, respectively). Often, these failings were found to stem from poor systems, processes or governance. 9

Although CQC inspections may be viewed as broader than QOF (especially given the focus on safety, more holistic care, and good leadership), they cannot provide regular feedback to (or about) practice teams, as in normal circumstances there would be multiple years between inspections. Practices wanting more regular feedback therefore require other tools. Examples of these have been provided by the Royal College of General Practitioners (RCGP), which introduced a suite of quality improvement (QI) tools. In particular, it described a QI ‘wheel’ for general practice. The hub of the wheel represents context and culture in QI. The inner wheel comprises QI tools (diagnosis, planning and testing, implementing and embedding, sustaining and spreading), and the supporting rings represent patient involvement, engagement and improvement science. However, although tools for diagnosis are suggested, and other techniques such as plan, do, study, act (PDSA) cycles, run charts and statistical process control charts are discussed, the implementation is largely left to practices’ own priorities, and some of the outer aspects of the wheel are less well developed. 64

However, practices are not operating in a vacuum. Efforts to ameliorate the continually increasing pressures on the NHS in general, and general practice in particular, create specific difficulties. The 2014 NHS Five Year Forward View65 presented a vision for the NHS that required a transformation in the way services are delivered, and primary care is a key aspect of this. 65–67 The evolving sustainability and transformation partnerships (STPs) (formerly known as sustainability and transformation plans) for each of 44 localities in England, and the forthcoming integrated care systems (ICSs) that will be formed from these, are an effort to create more efficient, integrated, patient-focused services that respond to the needs of a particular local population, and, unsurprisingly, most place general practice at their heart. 68 Health Education England has produced a report69 on developing the skill mix of primary care teams and allowing GPs to concentrate on more complex clinical problems. Ongoing reforms mean that practices need to be more productive, serving increasing numbers of patients with more complex health needs while making better use of resources; therefore, efficiencies are needed across the service, and the General Practice Forward View11 presented a vision of how this would be achieved, with five major elements:

1. Increasing investment – accelerating funding of primary care by investing an additional £2.4B per year, representing a 14% real-terms increase on 2015/16 levels by 2020/21.

2. Increasing the workforce – expanding and supporting GPs and wider primary care staffing, with plans to double the rate of growth of the medical workforce, leading to an overall net growth of 5000 GPs by 2020; this would also require additional international recruitment.

3. Reducing workload – lowering practice burdens and releasing staff time.

4. Improving practice infrastructure – developing primary care estate and investing in better technology [including an increase of > 18% in allocations to CCGs for provision of information technology (IT) services/technology for general practice].

5. Improving care design – providing a major programme of improvement support to practices (including greater integration and, in particular, involvement in ICSs).

The extent to which these objectives are achievable is unclear; in particular, the increase in workforce is not currently supported by increases in training, and the specific approaches to increasing the number of GPs are ambitious. Reform in other areas, such as expanding the roles of pharmacists, will need careful examination also. Because of this, greater attention is needed towards initiatives to increase efficiency and lower waste. In some regions, groups of practices have joined together as federations – effectively acting as one organisation, but still providing care across the same locations. 63 In other areas, looser networks have been formed. A report published by the NHS Alliance in 2015, Making Time in General Practice,70 identified 10 high-impact actions that practices could take to improve the time available for care. These are summarised as follows:

1. Active signposting – provides patients with a first point of contact who directs them to the most appropriate source of help. Web- and app-based portals can provide self-help and self-management resources as well as signposting to the most appropriate professional.

2. New consultation types – introduce new communication methods for some consultations, such as telephone and e-mail, improving continuity and convenience for the patient and reducing clinical contact time.

3. Reduce did not attends (DNAs) – maximise the use of appointment slots and improve continuity by reducing DNAs. Changes may include redesigning the appointment system, encouraging patients to write appointment cards themselves, issuing appointment reminders by text message and making it quick for patients to cancel or rearrange an appointment.

4. Develop the team – broaden the workforce in order to reduce demand for GP time and connect the patient directly with the most appropriate professional.

5. Productive work flows – introduce new ways of working that enable staff to work smarter, not harder.

6. Personal productivity – support staff to develop their personal resilience and learn specific skills that enable them to work in the most efficient way possible.

7. Partnership working – create partnerships and collaborations with other practices and providers in the local health and social care system.

8. Social prescribing – use referral and signposting to non-medical services in the community that increase wellbeing and independence.

9. Support self-care – take every opportunity to support people to play a greater role in their own health and care with methods of signposting patients to sources of information, advice and support in the community.

10. Develop QI expertise – develop a specialist team of facilitators to support service redesign and continuous QI.

However, most of these are entirely unrelated to what is captured by the QOF and, although some overlap with CQC inspection domains (particularly the ‘well-led’ domain), it seems appropriate that any overall effort to capture the effectiveness and/or productivity of general practices should include the areas mentioned in these 10 high-impact actions.

Overall, therefore, there is much room for improvement in the measurement of performance in general practice. One of the principal objectives of this study was to rectify this, which the research team have attempted to do using a specific method of measure development, ProMES. 27 In the following section, the literature on this method is reviewed.

Defining the map parameters

A mapping review is typically used for broad questions; therefore, an important first step to ensure that the topic and definitions are matched to the available resources is to define the map parameters.

The map parameters used were set using the methods outlined in the Social Care Institute for Excellence (SCIE) guidance. 87 These involved (1) discussion of the review topic between members of the research team, (2) pre-map scoping of the literature, (3) operationalising the parameters into inclusion and exclusion criteria and (4) developing a search strategy. These steps were used iteratively so that, with each process, the findings were used to make suitable changes to parameters.

Scoping searches were conducted using MEDLINE. Initially, these consisted of free-text terms, then corresponding terms were found using a search engine for medical subject headings (MeSHs). Numbers of hits for these searches were recorded and titles scanned to check their relevance.

Inclusion and exclusion criteria were developed and trialled with title and abstract screening on the results from scoping searches. This was also used to check the search terms were retrieving studies pertinent to the review question. In early searches, this was done by simply asking whether or not the paper fitted with the broad review question. Reasons why papers did not fit the review question were recorded and these recordings were used to make a more comprehensive list for inclusion and exclusion criteria. The screening was conducted by one author, but another author also screened a random 10% sample of the articles to assess inter-rater agreement.

This process was iterated three times before the relevant search terms and eligibility criteria were finalised. Finally, an information specialist was consulted to maximise the sensitivity of the search strategy while maintaining a manageable volume of hits.

Search strategy

Databases searched were MEDLINE, EMBASE, the Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Emerald Insight. No date range was specified (i.e. it was unlimited) and the final search was conducted on 1 August 2017. These databases were chosen as research in the fields of medicine, nursing and management was considered relevant.

Full searches were carried out in August 2017. The full list of terms can be found in Appendix 1. The agreed search strategy used thesaurus and free-text entries with Boolean logic, combining terms relating to the general practice setting, practice-level phenomena, and quality, effectiveness or productivity. Search terms were added with the Boolean phrase ‘NOT’ to exclude phenomena relating to policy or QOF, as these would be beyond the scope of the review. Searches were limited to English-language articles.

Reference tracking was performed on included studies identified through database searching. Hand-searching was used on the most recent issues of a select range of journals, to capture any relevant articles that may not yet have entered the database records. These were the British Journal of General Practice, the Canadian Family Physician, the New Zealand Family Physician, the Scandinavian Journal of Primary Health Care and the Australian Family Physician.

As the review was focused on mapping the empirical evidence, grey literature searching was limited and focused on finding the referenced empirical evidence. This consisted of searching the NHS network for materials related to the 10 high-impact actions and compiling these,88 and tracking the references of the main report70 for these actions.

Study selection

The inclusion and exclusion criteria that were finalised after scoping searches are given in Table 3, alongside the justification for the parameters set.

Titles and abstracts were screened using the eligibility criteria. A random sample of 10% of those excluded at this stage were checked by a different member of the research team to ensure consistent application of the exclusion criteria. Full papers of those not yet excluded were then assessed for eligibility and reasons for excluding papers were provided.

Data extraction

A formalised data extraction form was used, which can be found in Appendix 2. This was designed using the examples given by James et al. 86 and following guidance provided for mapping reviews by the SCIE. 87 As the aim of a mapping review is to provide a description of the current body of research, and not to evaluate the results, data extraction focuses on metadata, such as methods, sample characteristics and country of study. Brief data regarding the results were extracted and quality assessment was recorded so that this could be taken into account when performing data synthesis. 86,87

Quality assessment

Commonly, the hierarchy of evidence scale alone is used to describe the level of quality of studies within mapping reviews. However, this is considered a rudimentary form of assessment in which study designs vary (e.g. a mixture of observational and experimental designs) in accordance with the phenomenon of interest or a broad-based question is applied. Although not an essential component of a systematic mapping review, use of a quality assessment tool helps provide a more detailed description of the validity of the current research body on the review topic. For this reason, criteria created by Kmet et al. 89 were used. These were first designed for a systematic review with a broad question, for assessing internal validity of diverse study designs. There are two checklists in the criteria: one for quantitative studies and one for qualitative studies. The criteria can be found as part of the data extraction form in Appendix 2. For each item, studies are given a score from 0–2 depending on whether or not the study answers the criterion (no, partial or yes). As there are a variable number of questions that apply to a given study, the total score is turned into a percentage of the maximum score from all of the applicable questions. This can then be used to compare the quality with that of other included studies. 86,89

Data synthesis

The generic data were summarised into a table of studies, including year of publication, country of study, design and methods, population of interest and sample size. Studies were then synthesised into a map, to demonstrate the current areas of research into the review topic.

The methods of evaluation were classified as concerning productivity, quality or effectiveness. As there is considerable crossover in terminology, three broad definitions were used to group outcomes as assessing effectiveness, quality or productivity. Productivity referred to any studies assessing how the practice was performing with consideration of resources (either monetary or human). Studies were ascribed as studying quality if they assessed standards of practice in terms of safety, adherence to best practice guidelines, or patient experience. Effectiveness was said to be assessed when measures of how a service was functioning (provision of care processes or achievement of outcomes) were used. The methods of evaluation were then summarised in accordance with whether they were processes, or intermediate or final outcomes as described by Baker and England. 42

Studies were grouped by the phenomena they investigated. These were grouped in accordance with their relationship to infrastructure, personnel or governance, and any patterns in topics that emerged were considered. A narrative overview of the findings in each of these areas was presented briefly, but with the caution that conclusion on effects could not be drawn.

Search results

Figure 2 depicts the results from the search strategy using a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram. Initially, database searches returned 2834 results. Once duplicates were removed, 1693 results were identified from database searches. A further 37 studies were identified using reference tracking, once duplicates were removed. Searching of the grey literature identified six relevant empirical titles. A total of 1736 records were screened at title and abstract level and 1521 were excluded at this stage. There were 215 full texts assessed for their eligibility. Of these, 39 papers met the criteria for inclusion in the systematic map, referring to 37 distinct studies. 90–128 The primary reasons for excluding the 176 studies are given in the PRISMA flow diagram (see Figure 2).

FIGURE 2.

The PRISMA flow diagram. (See text for explanation of papers that were eligible, but are not included in the map.)

In two cases, multiple papers referred to samples from the same study. Bower et al. 95 and Campbell et al. 96 report results from a mutual sample using the same independent variables, with Bower et al. 95 reporting an additional outcome: team effectiveness. Ludt et al. 117 and Petek and Mlakar. 120 both drew data from the European Practice Assessment of Cardiovascular risk management (EPA-Cardio) study, considering the same dependent and independent variables. In both cases, the more recent papers (2013117 and 2016120) were included in the map, as they reported in more detail. Klemenc-Ketis et al. 114 also drew data from the EPA-Cardio study, but examined a different population (those with established disease and healthy participants, in addition to those with a high cardiovascular risk), and reported on a different dependent variable (patient satisfaction). Therefore, it has been considered as a separate study.

Study designs

The characteristics of the included studies can be found in Appendix 3. The majority (n = 15) were based in the UK, with others being set in Canada (n = 6), Australia (n = 4), New Zealand (n = 3) and Europe (the Netherlands, n = 5; Slovenia, n = 2; multiple countries, n = 2). Publication dates ranged from 1978 to 2017.

A quantitative design was used by 26 studies,90–95,97,98,103,106–110,112–114,116,117,119–122,124–126,128 four studies used a qualitative design102,105,115,127 and five studies used mixed methods. 99–101,118,123

Two systematic reviews were included: Goh and Eccles104 focused on primary studies of any design, whereas Irwin et al. 111 used tertiary synthesis – reviewing systematic reviews. The individual systematic reviews in this tertiary synthesis were not considered as separate entities as they assessed interventions tailored for specific clinical groups or drew from populations wider than just primary care. However, at a tertiary level, this review drew together areas of QI that were highly relevant to this project and so justified inclusion.

Of those using quantitative data, the majority (n = 19) were cross-sectional. Three studies used a retrospective cohort design98,112,120 and four used experimental designs (one non-RCT,110 two RCTs113 and one pre- and post-intervention study97). This was anticipated, as the features and processes of interest were not likely to be pragmatic topics for RCTs or longitudinal assessment.

Of the qualitative studies, Grant et al. 105 and Swinglehurst et al. 127 used an ethnographic approach involving observations of practice, whereas Fisher et al. 102 and Lawton et al. 115 used face-to-face or telephone interviews.

Mixed-method designs were used for two studies examining a cross-section of general practice,99,101 and three studies100,118,123 in which phenomena were studied longitudinally, with the implementation of a change in practice being evaluated.

Quality assessment

The quality assessment scores for the studies can be found in Appendix 3. The checklist did not translate well for use when assessing systematic reviews so was not applied to Goh and Eccles104 or Irwin et al. 111 As quality assessment is not a requirement of a mapping review, this was considered an acceptable decision.

Scores for the remaining 35 studies ranged from 45% to 100%, of which 26 scored ≥ 75%. This suggests that, overall, the quality of the studies was quite high. See Appendix 3, Figures 17–19, for the areas in which the studies scored yes, no or partial on the quality criteria for quantitative, qualitative and mixed-methods studies.

Of 26 quantitative studies, 22 scored ≥ 75%. Areas in which studies scored well were the research question and aims, methods of analysis, results reporting and conclusions drawn. Areas in which quality was particularly poor were in the measurement of outcomes and exposures, controlling for confounding variables and sample size. All four qualitative studies scored > 75%. Each study lost points on a different component of quality, with all studies performing well on the question/objectives, design, context, data collection methods and analysis. Of those using mixed methods, none of the studies scored > 75%. They tended to score poorly on questions relating to the quantitative elements, especially regarding sample size and controlling for confounders, and on the reflexivity of the account, suggesting that the blending of the two methodologies was not, on the whole, sufficient.

Data synthesis: study mapping

Appendix 4 provides a summary of the phenomena adaptable by general practices that were investigated in the included studies. The methods used to evaluate their impact on practice outcomes are also summarised as effectiveness, quality or productivity, along with references to the corroborating papers. Appendix 4 represents all practice-adaptable features identified in the studies, regardless of results.

Means of evaluating productivity, quality and effectiveness

Figure 3 demonstrates the number of studies identified as relating to effectiveness, quality or productivity, and, of those, the number of studies that covered a single aspect of these outcomes compared with the number using a composite of objectives. As some studies used outcomes that were classified as effectiveness and quality, the sum of the bars in the graph is > 37. Of note is the fact that only Fisher et al. 102 identified an outcome related to productivity, and this was qualitatively evaluated. Most studies assessed a single objective relating to quality (n = 17), whereas five studies assessed a single objective relating to the effectiveness of a practice. Twelve studies measured effectiveness and/or productivity using multiple objectives. The review by Goh and Eccles104 was also concerned with both effectiveness or quality from multiple domains, whereas the review by Irwin et al. 111 examined quality in three categories: process improvement, physiological markers and other patient outcomes.

FIGURE 3.

Classification of study outcomes.

Features and processes adaptable at practice level

The independent variables studied that a practice could adapt are discussed in Infrastructure and Personnel. Beyond the remit of the study were many variables relating to features that a practice could not feasibly adapt. One grey area that emerged was the size of the practice. A variety of methods to measure this was used in the papers, but there was often overlap with features that the practice could not feasibly adapt, such as the size of the practice population. Although potentially relevant, these measures were excluded from the map as it was felt that selection bias could occur when dividing the measures into those that a practice could adapt and those that were not adaptable.

Stage 1a: ProMES-based workshops

The ProMES methodology was introduced in Chapter 2, Methods. In its traditional form, it involves a series of facilitated workshops with members of a team, in three broad stages: (1) definition/clarification of team objectives, (2) identification of indicators of these objectives and (3) the formation of contingencies to convert these indicators into effectiveness scores. In this study, the traditional ProMES methodology was adapted to be used across an entire sector (general practice) rather than a single team. This followed the approach used successfully in a different NIHR project,82 which used an equivalent design to develop a measure of effectiveness for CMHTs.

The design involved three phases of workshops.

Stage 1b: finalisation of the measure

There were two main elements to this phase: a consensus exercise, and the creation of an online version of the tool that could be used for piloting:

1. Consensus exercise. The aim of the consensus exercise was to build agreement on the weightings of the different indicators/objectives chosen, and to determine whether or not the measure appeared feasible (in terms of the data collection required) and valid (i.e. that it appears to measure something that is accepted as a measure of productivity). This included an initial meeting, to which representatives from relevant NHS, regulatory, professional and patient bodies were invited. This was followed by a series of online communications, including both e-mails and an online questionnaire, to finalise details.

2. Creation of an online tool. The tool used an online platform named Effecteev^® designed for ProMES applications by the specialist German company Feedbit Software GmbH (Nürnberg, Germany), to assist teams with the ProMES process. This software was used during stage 1a also, to help develop the contingencies. As part of this process, guidance (both online and in document form) was developed to instruct/assist practices in using the tool.

In reality, the precise methods for stage 1 deviated somewhat from the above for a combination of practical reasons. These variations, and their implications, are detailed in Chapter 4.

Stage 2 involved the piloting of this measure. The aim was to recruit 50 general practices from a variety of CCGs and regions to use the measurement tool over a 6-month period (stage 2a). This would then be evaluated (stage 2b) to test whether or not the tool worked in a way that was feasible for practices, whether or not it was perceived to be useful and whether or not there was any evidence of improvements being made with its use.

Stage 2a: piloting of the measure

A target was set for 50 practices to be recruited to trial the measure and online tool over a 6-month period. The aim was that the 50 practices should be spread across 8–10 CCGs, some of which would also have participated in stage 1 of the study. CCGs were recruited purposively, so that those involved in stage 1 were invited first, and then others selected to ensure a balance across regions of England and a balance between those that were more urban or more rural in nature. CCGs that agreed to participate sent out a request to their general practices for expressions of interest in the study. Practices that had participated in stage 1 were also contacted directly by the researchers.

A total of 51 practices agreed to participate. These were targeted to ensure that these practices had a good spread in terms of location, size and characteristics of the local population. Each of these 51 practices was visited by a member of the research team or a trained Clinical Research Network (CRN) representative, and at least one member of staff in the practice was trained in using the online tool, including details of the data collection required, and retrieving and interpreting monthly reports.

Each practice would then use the tool for a period of 6 months: collating and entering data on a monthly basis, and generating a report to indicate how indicators, objectives and overall effectiveness were changing over time. This would be followed by the evaluation. Following completion, practices were each paid £500 in compensation for the time spent piloting the tool.

Stage 2b: evaluation of the measure

At the end of the 6-month period, the measure was evaluated. The evaluation took three forms:

1. Analysis of effectiveness data. A statistical analysis of monthly practice data was conducted to determine whether or not there was any evidence of change over time overall, change over time for specific objectives or change over time for specific indicators, and whether the extent of change was related to characteristics of the practice.

2. Interviews with practice representatives. Telephone interviews were conducted with representatives from each practice. These were, in some cases, the same practice managers who had responded to the questionnaire, but, in other cases, GPs or other practice staff were interviewed (this was spread out to give a more balanced view). Interviews covered the perceived usefulness of the measure (including how widely results were discussed in practice meetings), whether or not there were any areas not covered, and what might be done to improve the usability of the tool. In some cases, interviews (and two focus groups) were conducted with patient representatives; these only covered the usefulness and coverage of the measure, particularly focusing on those sections most relevant to patient experience.

3. Practice manager questionnaire. An online survey was sent to all practice managers, covering three principal areas: (1) their experience of using the tool (including the amount of time it took, the mechanics of using the online platform and the availability of data required), (2) their perceptions of its usefulness and how well it could be used to improve practice effectiveness and (3) contextual information about the practice including major changes to staffing and overall practice expenditure over the period.

Data from the interviews and open-ended question data from the practice manager questionnaire were coded and analysed using thematic analysis. Findings from all three strands were then collated to draw overall conclusions about the tool.

This design is summarised in Figure 4.

FIGURE 4.

Overall summary of the research design.

Phase 1 workshops

Phase 2 workshops

Phase 3 workshops

Planned approach

Phase 3 was designed around two large, full-day workshops, with similar constitutions to phase 1, and recruitment was performed similarly. However, the nature of the workshops was very different from phase 1. The workshops were held during April and May 2016.

The day started with a description of the study and a summary of the first two phases, in particular the objectives identified.

The first exercise involved providing a weighting of the objectives. To achieve this, flip chart sheets representing each objective were placed around the room, and each participant was given 30 stickers. Participants were asked to place stickers on the flip chart sheets to represent how much of a team’s effort should be directed towards that objective compared with the others. If they perceived all objectives to be equally important, for example, this would mean putting three stickers on each of nine objectives, and leaving three unused. In an extreme situation, if they only considered one of the objectives worth any effort, they would place all their stickers on that objective.

Professional participants and public participants were given different coloured stickers, so that the views of these two groups could be compared. The findings were collated by the research team and fed back to the participants; some time was given for reflections on these before the rest of the day’s exercises began. These totals were fed into the overall weightings of the objectives in the final measure via the consensus exercise.

The main exercises for the day involved deriving contingencies for each of the indicators. Each workshop was split into six smaller groups, aligned along professional/patient groups (so that there would be a group of GPs, a group of practice managers, etc.), with a facilitator from the research team. Indicators were divided up between these groups and considered in turn.

For each indicator given a green rating in phase 2, and those given an amber rating that were subsequently chosen to be carried forward, at least one group undertook the following process:

• The definition and availability of data were considered further, as a check on the sense of using that indicator. (When multiple similar indicators were suggested in phase 2, each was considered and the most appropriate one chosen.)

• A discussion was held about whether or not the minimum, norm and maximum values chosen in phase 2 were appropriate, and these values were amended when necessary. (A norm value is one considered to be neither good nor bad, but where an average practice would expect to be on that indicator.)

• A discussion was held about the relative effects of achieving the minimum or maximum. This was to enable a starting point to be given for the number of effectiveness points available for these values. Under ProMES, a norm value is worth 0 effectiveness points, a maximum value up to 100 points and a minimum value up to –100 points. Therefore, if it were suggested that achieving the minimum would be twice as bad as achieving the maximum would be good, then the number of points initially set for achieving the minimum would be –100, and the number of points for achieving the maximum would be 50.

• A draft contingency was then shown on a screen, using the ProMES-based software, Effecteev (a custom-built programme to enable teams to use the ProMES process). At first, each contingency was a straight line, based on these minimum and maximum values. The precise shape of this contingency was then altered by discussion with the group. This was undertaken by choosing intermediate values of the indicator and discussing how good (or bad) this would be compared with the minimum, norm and maximum. This eventually formed a contingency such as that shown in Figure 5.

• The indicators for each objective were then given different weightings based on the perceived importance of that particular indicator in the context of the whole objective. For example, this might result in a contingency for one indicator being rescaled so that it had a maximum of 30 effectiveness points, rather than a maximum of 60 effectiveness points, if that indicator was judged to have only half the importance of another indicator with a maximum of 60 effectiveness points.

FIGURE 5.

An example of contingency in the GPET. GPET, General Practice Effectiveness Tool.

Part 1: general practitioners only

• Questions to elicit contingencies for 20 clinical indicators (estimates of low, medium and high levels).

• Questions about relative weightings of all indicators within an objective.

• Questions about relative weightings of all clinical objectives.

Part 2: all respondents

• Questions to elicit contingencies for six non-clinical indicators (that had been developed or amended since the phase 2/3 workshops).

• Questions about relative weightings of all non-clinical indicators within each objective.

• Questions about relative weightings of all objectives and performance areas.

• Questions about any other areas that were missing from the tool, or any other comments that respondents may have.

All participants from the consensus exercise meeting, along with other invitees who could not make that meeting, were invited to complete this online exercise. Responses were received from 27 participants, including 8 GPs (the other 19 respondents included at least 4 from national bodies with an interest in general practice, including NHS England, the Royal College of Nursing and the RCGP, at least 3 from patient organisations, including Healthwatch, and 12 who did not specify their organisation or role in their response).

Responses were examined for outliers; any extreme outliers were removed before findings were summarised by calculating mean levels. These mean levels were then used as the basis for final contingencies and weightings in the piloted version of the tool.

Clinical care

Four specific workshops focused on clinical care objectives and indicators, as well as subgroups of the two larger phase 3 workshops. The initial collection of suggested indicators included the following:

• healthier population [including reduced smoking, reduced alcohol consumption, reduced body mass index (BMI), increased activity]

• focus on wellness and prevention (including patient engagement regarding prevention, such as number of sessions offered by the practice to help patients learn about prevention)

• specific health outcomes – diabetes, dementia, other mental health, chronic obstructive pulmonary disease (COPD), heart disease, chronic pain

• significant events and complaints (including how are they used within the practice to improve care)

• medication reviews (used to promote ideal timely review and expected outcomes per review)

• prescription safety monitoring

• effective consultations (effective for patient, effective for practitioner)

• safeguarding (showing evidence of safe and effective safeguarding procedures and ability to care for vulnerable groups; evidence of contacting vulnerable groups specifically)

• prescribing errors

• referrals (appropriate referring to hospital, matching of symptom to outcome; proportion of referrals not out of area)

• availability of specialist services [including specialist staff to deal with chronic illness (e.g. diabetes, COPD, heart disease), children, mental health/learning disabilities, stroke]

• percentage of the population having NHS health checks

• percentage of the population not seeing a GP

• sick notes and fit notes issued

• immunisations performed (including influenza and age-appropriate immunisations for children)

• complaints (number upheld, reviews conducted)

• percentage of unnecessary consultations

• percentage of appointments not attended (DNAs).

As there were six separate workshops involved in developing these indicators, discussion at the earlier ones had more of a focus on becoming more specific about the indicators suggested, whereas discussions at later workshops took a more holistic view of the complete set of indicators. (It was during this later phase that the need to separate into three different objectives became clearer.)

The discussion around the clinical care indicators was particularly informed by existing indicators, such as those used in the QOF. Therefore, for each clinical indicator, there was a discussion around whether or not an existing QOF indicator would suffice, either in its existing form or adapted in some way. Several were adapted when it was felt that the QOF indicator(s) did not cover the most desirable clinical outcomes.

The general feeling was that the QOF indicators would cover many of the specific clinical conditions (particularly long-term conditions), as well as various public health/preventative medicine areas; however, the number of QOF indicators was far too large for this purpose. Instead, there should be (preferably) one indicator for each of a carefully selected number of specific clinical conditions (it was seen as crucial that common health conditions viewed as particularly important were measured by separate indicators). The most important aspects of preventative medicine should be covered and also it was important to cover aspects of clinical care that would not necessarily be included in individual patient records, but were reflective of the practice’s approach to clinical care as a whole.

One finding that became clear early on in these workshops was that, particularly for non-QOF indicators based on clinical data, practices would be unlikely to have the resources to design the queries to extract the data themselves; the majority of GPs and other practice staff who commented on this said that if they were asked to, that would probably rule them out of participating in the pilot phase. Therefore, it was decided that data extraction queries would have to be written centrally by the research team. In order to achieve this, the services of PRIMIS at the University of Nottingham were engaged; this is described further in Development of the online tool.

Overall, therefore, the indicators chosen here were designed to strike a balance between usable specificity (so that areas could be targeted and improvements could be made) and broad coverage of most of a practice’s clinical work, while not including too many separate indicators. Inevitably this means that some specific conditions are not directly covered (e.g. chronic kidney disease), but there was broad consensus that the conditions covered were the most relevant and important ones for general practice. (For most long-term conditions, a single indicator was chosen, although, for COPD, it was determined that a single indicator could not cover what was needed, and three indicators were formed – between them contributing the appropriate weight for COPD.)

An overall summary of the topics of the indicators chosen is presented in the following sections. This is merely indicative of the topics and number of indicators; full details of the descriptions of these indicators are provided in Final tool for piloting.

Practice management

Five specific workshops looked at practice management objectives and indicators, as well as subgroups at the two large phase 3 workshops. The initial collection of indicators suggested included the following.

Patient focus

Two specific workshops, as well as subgroups at the two large phase 3 workshops, concentrated on objectives and indicators with a patient focus. The initial collection of suggested indicators included the following.

External focus

Two specific workshops, as well as subgroups at the two large phase 3 workshops, focused on objectives and indicators with an external focus. In addition, the extra public health expert workshop contributed particularly to this performance area. As this workshop had a slightly specialised remit, it is described in more detail in this section.

Participants in this workshop understood that some aspects of medical public health were already included in the clinical care section of indicators. In addition, aspects of public health relating to community participation were covered already. Therefore, the group focused their discussion on health needs and root causes of ill health. A range of possible indicators was discussed, including social assessment and prescribing, setting out that the majority of consultations ideally include assessments of social circumstances and inequalities affecting health, as well as medical assessment and follow-up. Subsequently, this was reduced to social prescribing because of the requirements for ProMES indicators.

Another area considered desirable to include was related to the state of health in a practice area. It was suggested that an indicator could include a quarterly review of practice/neighbourhood population health, a well-being profile and, relating this, to the practice plan. Ultimately this was not included, as it was thought that the data would not be available on a regular enough basis (e.g. the Public Health England general practice profiles do provide this information, but only annually, which is not sufficient for this type of measurement).

Some aspects of the discussion were included in the indicators relating to the voluntary sector in partnership and engagement areas, although it was not possible to include activities undertaken by volunteers.

Overall, for this performance area, the initial collection of indicators suggested included the following.

Initial meeting

As described in Consensus exercise methods, there were four principal sets of discussion questions considered during this meeting:

1. What sort of tool will be most useful to practices (e.g. areas covered, frequency of data entry and feedback)? What unique selling point might make it most attractive for use in the future?

2. How should comprehensiveness and usability be weighed up? What is the right balance between covering core work and more aspirational performance?

3. How can it be ensured that the tool is best suited for use 5–10 years into the future? How can best use within federations be ensured (e.g. practice-level or federation-level data)? How can it be ensured that the Five Year Forward View65 and STPs are taken into account?

4. Regarding the content of the tool, which areas should be included/expanded? Are there any other areas that should be covered? How can best use be made of existing data?

A summary of the responses and discussion for each question is given in the following sections.

Online survey

The methods for the survey were given in Consensus exercise methods; the major output from this was the full contingencies for many of the indicators in the tool. These contingencies are shown in detail in Final tool for piloting. However, the other main aspect of the survey was to provide weightings for the different objectives within each performance area, and for the different performance areas overall.

Therefore, data from the 27 respondents were collated; the relative weightings for each performance area and objective were as shown in Table 7. It is notable that, within clinical care (which itself has the largest overall weighting, at 37% of the total), the three objectives were viewed as equally important by respondents. These weights superseded those found in the phase 3 workshops, because of the change in structure of the tool after that point (i.e. the introduction of performance areas and the splitting of clinical care into three objectives). However, the theme of the importance of both clinical care and practice management, which was evident from both professional and public participants in the phase 3 workshop, was retained in the consensus exercise also.

Therefore, these weightings were joined with the findings from the workshops (which balanced indicators within objectives) to create the total weighting of indicators within the tool to form an overall effectiveness index, using the following steps:

• The four performance areas were given a total number of points in accordance with the weights in column 2 of Table 7 (4920 points were available in total, so, for example, for patient focus, the number of points was approximately 18% of 4920 = 900 points).

• Within each performance area, the objectives were given a number of points in accordance with the weights in the final column of Table 7, using a similar mechanism.

• Within each objective, the weightings from the phase 3 workshops were used to divide the points available for each indicator.

Software: Effecteev

Effecteev is the name of a piece of software developed by BlackBox/Open, a German company, to assist teams with the ProMES process. It requires the entry of objectives and indicators, and then helps with the development of contingencies by enabling users to view them and alter them manually, directly on the graph itself. After being set up, it then prompts team members for data entry on a regular (e.g. monthly) basis and allows the download of a regular report for the team, showing progress overall, as well as on each objective and indicator.

The research team worked with the developers of this software to adapt it for the needs of this study. The development of the contingencies could be done in the workshops using the existing form of the software, but there were three areas for which a significant change was needed:

1. An extra level needed to be built into the software because of the adoption of performance areas in addition to objectives and indicators.

2. The system then also needed to be fixed for all teams in the sample, so that only the central research team could change the performance areas, objectives, indicators and contingencies, and only the central research team could view all of the data, but then individual ‘systems’ (teams) could be accessed by different practices. Therefore, for each practice, the process of entering data and receiving reports was as normal, but no changes to the central elements of the tool could be made by individual practices.

3. On the other hand, practices were free to select one or more of the optional indicators to include in their data collection if they so desired, even though these would not contribute to the overall effectiveness score.

As described in Chapter 5, each participating team was given access to the online tool on this basis, and trained by a member of the research team or a research nurse in using it.

Involvement of PRIMIS

As identified during the workshops and in the consensus exercise, practices would not have the resources to create the extraction of data from their clinical systems. The only feasible way of ensuring that a sufficient number of practices would be able to use the tool, therefore, was to provide them with the means to extract these clinical indicators automatically on a monthly basis.

To do this, the research team sought to develop Morbidity Information Query and Export Syntax (MIQUEST) queries (automated software to extract the correct information from practices’ clinical systems) that could be shared with the practices. However, the expertise to create these was not within the research team; therefore, PRIMIS (a business unit at the University of Nottingham, specialising in extraction of general practice data) was approached to explore ways to do this.

Having had initial discussions and obtained an estimate of the cost for PRIMIS to write these queries on our behalf, it was requested from NIHR that existing funds could be reallocated within the study to allow for this cost, which was agreed.

The research team subsequently worked closely with PRIMIS, via several telephone conversations and one half-day face-to-face workshop, to specify exactly what the indicators should be measuring, including precise definitions of populations, timescales and exceptions. This covered the 19 indicators that were measured from clinical systems.

At the time, all of the practices that had expressed an interest in participating in the pilot study used one of the two most common clinical data systems: EMIS (EMIS Health, Leeds, UK) or SystmOne [The Phoenix Partnership (TPP), Leeds, UK]. Therefore, we proceeded on the basis that queries written would be compatible with both these systems, and added the use of one of these systems as an inclusion criterion for participation in the pilot. PRIMIS then wrote and tested the queries on these two systems.

The net result was automated software that practices would be able to run, so that each month all of the necessary data would be outputted into a single document, for entering into the Effecteev system when prompted.

Although highly successful in one way, the added time taken meant that the queries were not finally ready until month 21 of the study. This had the net effect of compressing the remainder of the timescale for the study. The detail and implications of this are discussed in Chapter 5.

Pilot study in general practices

The aim of the pilot was to examine how practical and useful the GPET is, by testing it in 50 practices. The plan was that the 50 practices should be spread across 8–10 CCGs, some of which would also have participated in stage 1 of the study.

In total, 51 practices agreed to participate. After an initial opening invitation, follow-up requests for participation were targeted towards areas and types of practice that had lower representation in the sample, to ensure that the overall set of practices had a good spread in terms of location, size and characteristics of the local population. Each of these 51 practices was visited by a member of the research team or a trained CRN representative, and at least one member of staff in the practice was trained in using the online tool, including the details of the data collection required and retrieving and interpreting monthly reports.

Each practice would then use the tool for a period of 6 months: collating and entering data on a monthly basis, and getting a report to indicate how indicators, objectives and overall effectiveness were changing over time. This would be followed by the evaluation. Following completion, practices were each paid £500 in compensation for the time spent piloting the tool.

Interviews with practice staff and members of the public

The interviews with practice staff had three main aims. The first was to discover the perceived usefulness of the GPET. In particular, this was concerned with whether or not the feedback provided was useful; whether or not practices discussed this feedback in team meetings and what was done with the information; and whether or not practices would be interested in continuing to use the tool beyond the pilot. The second aim was to get views on the content of the tool using the following questions: ‘did it cover the areas that were perceived as important, useful and relevant?’; ‘was there anything missing or superfluous?’; and ‘were the indicators easy/feasible to collect?’. The third aim was to examine the usability of the tool, in terms of how easy the online platform was to use.

In total, 38 semistructured interviews were conducted with practice staff. Each practice was asked to nominate a member of staff who had been involved in the use of the tool; this included 26 practice managers, nine GPs, two practice or research nurses and one data manager. The interviews, typically of about 40 minutes’ duration, were audio-recorded, anonymised and transcribed. The interview schedule for these interviews is shown in Appendix 5.

It was also planned to interview patient representatives from 10–15 practices. The aim of these interviews was to assess the content of the tool in a similar way to the second aim of the interviews with practice staff. The interview schedule for these interviews is shown in Appendix 6.

It proved difficult to interview patients because, at the time of the interviews, the majority of participating practices had not yet shared information about the study with their PPG and practices felt that their patients would therefore be unable to answer the questions; consequently, only four interviews were arranged via this route. Therefore, to add to the four interviews, two focus groups were also arranged. These were held in London and Sheffield and were attended by seven members of the public. Each focus group lasted 2 hours; the session plan is shown in Appendix 7.

Focus group participants were sent a Microsoft Word (Microsoft Corporation, Redmond, WA, USA) version of the GPET at least 7 days prior to the focus group (see Appendix 8). They were asked to think about the following questions:

1. Do these objectives and indicators reflect what is important to the patient population?

2. Do you think the data held by the practice will give a true reflection of how the practice is performing?

3. Are there any other indicators that should be included?

4. Would the patient population understand this information (e.g. PPG/Healthwatch)?

Practice manager questionnaire

In addition to the interviews, an online survey of all practice managers was conducted at practices that had participated in the pilot (including those that had not submitted data independently). This survey offered an opportunity to cover some of the same ground as the interviews, but with a different emphasis. In particular, there was the opportunity for respondents to comment on each performance area, objective and indicator; it also asked for quantitative responses about Likert-type scales about views on ease of use, usefulness of the tool and usefulness of feedback.

In addition, there was a section in the survey on practice expenditure, which was designed to measure the practice ‘inputs’ for comparison with the outputs (the effectiveness measured by the GPET) to form a productivity measure. The main question in this section asked practices to identify expenditure on four categories (staff, premises, bills and other) for each of the 6 months of the pilot phase. They were also asked whether or not anything had changed during the pilot that may have had an impact on expenditure or productivity.

The full questionnaire used is shown in Appendix 9.

Practice managers were e-mailed with a link to the survey [which was hosted by Qualtrics (Provo UT and Seattle, WA, USA)] towards the end of the 6-month period. Up to three reminders were sent to non-responders; ultimately, responses were received from 41 practice managers.

Interviews with staff

A thematic approach was used to identify, analyse, and report the patterns (themes) that emerged based on a thorough and comprehensive coding of the interview data. 144 An exploratory method would not have been appropriate as there were some specific questions to be addressed that were relevant to the evaluation process.

One of the four interviewers developed an initial coding system that was then applied to a sample of the data. A colleague from one of the organisations conducting the study also analysed a sample of the transcripts for objectivity. The list of codes was modified after successive readings of the transcripts and was reapplied to the full set. The codes were arranged into an initial thematic map, which was reviewed and refined. A final structured representation of themes, and the relationships between them, that adequately reflected the whole data set is shown in Figure 15.

FIGURE 15.

Staff interview thematic map.

Feedback from patients and the public

In total, 11 members of the public were asked for feedback on the GPET. As explained in Interviews with practice staff and members of the public, four members of the public were interviewed and seven participated in two focus groups. Owing to the small sample size of interviews, these were not thematically analysed but responses from both were combined and are summarised in this section.

Overall, participants responded favourably to the tool. They particularly felt that it would be useful to share with the CQC during inspections to show that a practice is ‘thoughtful and willing to learn and develop’. They felt that it would be useful if the tool correlated with the CQC framework in looking for safe, responsive and effective practices.

Participants then reviewed the content of the tool; views are summarised here by each objective in turn. Those objectives with a greater relevance to patient experience were given greater weight in the discussion.

On average, how much time (in hours) would you estimate was spent on using the tool per month? This can include data entry, getting feedback, troubleshooting, etc.

This question was answered by 39 practices, stating that it took between 1 and 10 hours per month to use the tool (estimated mean 3.41 hours). Those who took less time were just entering the minimum data required. Several stated that the first few months took longer.

Two practices stated that this was much less time than anticipated, two a bit less than anticipated, 10 about the same as anticipated, 17 a bit more than anticipated and 10 much more than anticipated. Twenty practices highlighted gathering data as an aspect that had taken longer than anticipated, and 11 practices highlighted that entering data had taken longer. Other reasons were quoted by a small number of practices.

Usefulness and ease of use

Views on usefulness and ease of use were varied. On a scale from 0 to 10, the mean for ease of use was 5.6 (SD 2.1). For usefulness of the tool, the mean was 4.5 (SD 2.4). For usefulness of feedback, the mean was 4.7 (SD 2.6). This suggests that there was a substantial range of opinions. It is important to remember that this included responses from some practices that had not gone on to use the GPET after being trained in using it.

What, if any, additional resources were needed to use the tool?

This question was answered by 24 practices. Fourteen practices referred to the challenge of finding the time to use the tool. Most felt that it was a typical challenge of finding dedicated time for any non-day-to-day activity, whereas others referred specifically to the time needed to enter data required from practice records. Although the automated searches facilitated this, three practices stated that they had to amend the searches to be suitable for use within their practice.

Three practices commented specifically on the time taken to use the tool, referring to the need to have two screens to check the search results on the clinical system while inputting and that the tool may have been too long and overcomplicated.

Did you experience difficulties in using the software itself?

No difficulties were experienced by 25 practices. Fifteen said that they experienced a few difficulties and one said that it experienced considerable difficulties.

Please describe briefly the difficulties you encountered

Fourteen practices responded to the question asking about difficulties in using the software itself. Five practices felt that the tool was difficult to navigate, describing it as ‘clunky’, or ‘not intuitive’. One practice manager stated that it ‘felt like I was relearning it each month’. Two respondents referred to the data input process and offered suggestions for improvements, for example ‘It would have been better if having saved one [indicator response], it opened up the next automatically,’ and highlighted that activating each individual indicator could be a challenge that may best be done centrally.

Were there any performance areas missing from the General Practice Effectiveness Tool? If so, what were these?

According to 27 practices, there were no missing performance areas; it was stated that the tool was ‘very comprehensive’. One practice did think that ‘QOF indicators and enhanced services could be included’.

Were there any objectives missing from the General Practice Effectiveness Tool? If so, what were these, and under which performance area(s) should they fall?

This question was answered by 26 practices; 24 of which stated that no objectives were missing. Three practices felt that ‘an effective team is a productive team’ and that more measures on teamworking or workload management would have been beneficial.

Were any of the indicators problematic to gather? If so, please state which indicators these were

This question was answered by 36 practices, which provided useful suggestions for improvements. Fourteen practices highlighted problems with indicator 9.1 (hours of clinical appointments per 1000 patients per week). This was mainly because of the time it took to calculate. However, one practice stated that, ‘The results would only really be useful if this was split into doctor appointments, nurse appointments, HCA [health-care assistant] appointments, etc. to ascertain where any problems were’.

Nine practices also highlighted indicator 9.2 (percentage of patients waiting > 15 minutes past appointment time) for reasons similar to those for 9.1.

Six practices stated that gathering data for indicator 10.1 (percentage of attendance at MDT meetings) was difficult and/or time-consuming.

Three practices stated that data for indicator 3.5 (DNAs) was difficult to collect and that they could not complete indicator 6.5 [staff well-being (sick days)] as the information was not recorded in the way in which it was being asked for.

Some indicators were identified as needing to be collected less frequently than monthly, with quarterly collections being suggested. These, and all further information on improvements to be made for specific indicators, can be found in Appendix 11.

Were any of the indicators problematic to interpret? If so, please state which indicators these were

Responses to this question were received from 28 practices.

Twelve practices stated that the indicators were not difficult to interpret and one stated that they did not interpret the indicators. The remaining 15 practices commented on some ambiguity or subjectivity that was present; the mention of individual indicators is recorded in Appendix 11, as it highlights the importance of ensuring the clarity of the indicator descriptions.

Discussion of feedback

Seventeen practices said that they discussed feedback at team meetings, whereas 12 discussed it at individual meetings. Six practices said that they made other use of the feedback, including informal chats or sharing it with their patient representatives. In 13 of the practices, GPs had been involved in this discussion. Practice management staff had been involved in almost all discussions, and patient representatives had been involved in five practices (with other clinical and administrative staff involved in many practices also).

Did the feedback/data lead to any actions taken to improve effectiveness? If so, please summarise these briefly

Responses to this question were received from 29 practices. Sixteen practices stated that no action had been taken to improve effectiveness as a result of using the tool. One practice manager highlighted that, as there was no money attached to this, it was difficult to get GPs to make changes. One practice stated that they had not yet taken action but planned to and 12 highlighted how the feedback had led to actions within their practices. Four practices implemented more audits. Several practices amended templates to improve data capture and introduced care plans that had not previously been used effectively. The searches highlighted that some data were not Read coded properly by some clinical staff. Several practices stated that they found the ‘quality of team working’ questionnaire useful and would continue to use this (often it was referred to as the staff satisfaction survey).

If you have any other comments about the General Practice Effectiveness Tool, please state these here

Further comments about the tool were offered by 21 practices. Only three were predominantly negative:

It was clunky, took up a lot of time and didn’t give me any info[rmation] that was particularly of use to me as I already know what our focus/priorities are and how we are doing; at times it was demotivating because it highlighted we weren’t moving forward in some areas but then I had to remind myself they may not have been areas we needed to move forward in. I would not want to use this tool long term.

I wouldn’t use the tool voluntarily in its current format as it doesn’t serve to provide me with any information I don’t already have.

It was quite time-consuming and largely irrelevant to day-to-day work. An added pressure in an already busy day.

Six practices offered suggestions for improvements that would facilitate their use of the tool in future:

Could be a very useful benchmarking tool, but needs to have better report displays and options for manipulating the data to be more user friendly.

Entering the data manually is time-consuming, time is not something we have a lot of in general practice. There’s a lot of indicators that do not need to be asked every month; also some indicators would be better entered by clinical staff.

Financial part of this survey is not prescriptive enough and could be interpreted in different ways by different practices, e.g. ‘bills’ too loose a term.

I would not use the tool in its current form but can see a lot of benefits gathering some of the data. Collecting the data on a less frequent basis, e.g. 6 monthly, would make it more attractive and changes in results would be more apparent. I will use some of the categories in the future to monitor how various areas are progressing.

Nine practices offered comments that were partly or wholly positive:

It is quite a simple tool once you understand it but certain of the questions are difficult to get accurate answers for.

It offers opportunity for having a[n] overview from which practices can use to then consider areas for improvement and change. It enables discussions to take place with all levels of staff and with patients and provides a regular trend report on which any changes can begin to be measured against.

It was interesting to see another perspective on the surgery’s performances and is a good way to help motivate improvements where needed. Thank you.

Re[garding] my response to Q16, the tool would be useful but only if it replaced some of the existing tools. We are very stretched and adding yet another tool, regardless of how good it is, would be an additional burden on the team.

Something like this may be very helpful but having time to review results is absent.

The concept of the tool is very good and it helped to highlight and quantify some areas where we are below national averages. It also highlighted where we are doing well. To complete the same amount of data each month was more time-consuming than we had expected and, as some of it doesn’t change, I would like to have some more flexibility in how the tool or modules are set up so that you can track what is of interest to you and flag other items as unchanged or not priority to focus on for the practice.

We were able to see at a glance the areas that needed to improve and action them.

We would consider using again in the future but feel a much longer period of time needs to be looked at, especially for areas such as appraisals as these are done [at a] certain time of year, so much would be time specific.

Financial data

The financial data were asked for so that a productivity index could be derived, using financial resources as a denominator in a formula with effectiveness points used to calculate the numerator (as described in Chapter 3).

Unfortunately, however, many practices found it impossible to provide the data, despite agreeing in principle to do so before beginning the pilot. In total, only six practices were able to provide the monthly expenditure data that would have been used in this way. Clearly, this was insufficient to provide any useful productivity index development or analysis.

As described in Findings from the pilot study, the payment per registered patient was used as a practice characteristic variable to analyse alongside other practice characteristics. This was designed to pick up whether or not practices with a higher level of resources were more likely to complete, or perform better, on the GPET. However, there was no evidence that practices with greater financial resources per patient were more likely to participate, record higher effectiveness, or improve effectiveness than practices with lower financial resources per patient.

The overall effectiveness model

The ProMES has been well established as a method for generating tools to measure effectiveness. 73 The overlap and distinction between effectiveness and productivity is an important issue that will be discussed in more detail in How feasible is it to measure general practice productivity?, but for the purposes of this study, the ProMES exercise was designed to develop a measure of quality-adjusted outputs: that is, neither simply an assessment of activity nor of quality, but a combination of the two.

The large-scale ProMES process used in this study was based primarily on the approach taken by West et al. 82 in their study of CMHTs. The three-stage process consisted of two large workshops used to derive the objectives, six smaller workshops to formulate indicators and two further large workshops to refine, weight and assign contingencies to the indicators. This process proved to be very successful in CMHTs. However, in the context of general practices, it was more challenging for a variety of reasons.

First, the sheer range of clinical practice is vastly greater in general practices than in CMHTs, covering all types of patient and condition and with a huge volume of core activity. Thus, although nine objectives were generated initially, some of these, particularly in relation to clinical care, were found to be massive in scope and could not be measured by a small number of indicators (e.g. up to six), as is traditional in ProMES processes. As a result of this, the scheduled sessions proved insufficient to develop the measure, and several extra sessions needed to be added. It was important that a balance was struck between attempting to be comprehensive in what the measure would cover, and being realistic in terms of what practices could not only gather data on, but also respond to feedback on. Therefore it was imperative that this stage was not rushed.

Second, although there was much interest in the project in principle, because of the nature of GPs’ employment, it proved more difficult to attract GPs to the workshops, given that each workshop attended would have resulted in most of a day, or a whole day, lost from the workplace. Therefore, although many general practice staff did attend these workshops, only 15 GPs participated across the different workshops. As a result, the focus on clinical care, in particular, was left to a relatively small (albeit highly engaged and informed) group. In contrast to this, the level of interest from patients and public representatives was very high, with many very informed participants contributing to the workshops.

Third, the types of data that were suggested for the indicators were highly varied, and many needed very careful definition. This contrasted with the West et al. 82 study, in which the final measure was collected via employee questionnaires. The amount and range of objective data available within primary care means that there were many indicators that could be extracted from clinical systems (albeit with very careful definition and centrally constructed data queries). However, some other areas relied on practice records and, although there were some specific indicators that participants thought practices should have available, it was far from certain that practices would gather or store the information in the required form. Other areas were thought to be too difficult to measure.

The nine objectives that were derived from the first phase of workshops were considered to be valid at all stages of the study. In subsequent workshops, in the consensus exercise and in the implementation and evaluation of the pilot study, the over-riding impression was that these objectives successfully captured the broad activities of general practices. However, there was equal consensus that the clinical care objective was too big to be covered by a single objective and merited more of the content in an overall measure. Therefore, the separation of this one objective into three objectives, which relied on careful analysis of the suggestions made about clinical care indicators in the workshops, and the grouping together of similar objectives, produced an overall model with four performance areas and 11 objectives, as shown in Figure 16.

FIGURE 16.

Performance areas and objectives.

It was notable that, although there were concerns over some specific indicators, there was no significant dissent from this model when it was presented in the consensus exercise, or in the evaluation of the pilot itself. There was an acknowledgement that a tool such as this could never achieve both completeness and usability; therefore, the absence of some specific clinical areas was not seen as a major impediment. In addition, if the tool is to be further developed in the future, there would be flexibility to include some of these additional areas should it be deemed appropriate. There are, of course, some links with Kringos et al. ’s33 broader model for primary care effectiveness, in which they define the dimensions of processes as access, continuity of care, co-ordination of care and comprehensiveness of care (which broadly corresponds to the patient focus and external focus areas, with some elements of practice management, too), and outcomes determined by quality and efficiency of care and equity in health (broadly, the clinical care area).

It is to be expected that there would be some overlap with existing indicators of general practice performance. The clinical care performance area is closely related to the QOF, with several of the indicators overlapping (although sometimes defined slightly differently). The CQC inspection areas of ‘safe’, ‘effective’, ‘caring’, ‘responsive to people’s needs’ and ‘well-led’ clearly map on to the following performance areas: ‘safe’ primarily to clinical care, ‘effective’ to most (but particularly to clinical care), ‘caring’ to clinical care and patient focus, ‘responsive to people’s needs’ to patient focus and external focus and ‘well-led’ primarily to practice management. The advantage of this model over the CQC measure, however, is that it allows for far more regular feedback.

It is also notable how closely this model maps onto the 10 high-impact actions for making time in general practice. 13 These were described in full in Chapter 1, Overview of measurement of productivity and effectiveness in general practice. Table 14 demonstrates how well these align.

As discussed in Chapter 1, the fact that this is quite far removed from what is measured by the QOF suggests that there is a need to expand the model of general practice beyond what is included in the QOF. The fact that almost all sections of the effectiveness model produced by this research map closely onto either the QOF or these 10 high-impact actions further suggests that it is producing good and broad coverage of the most important areas of performance for general practice.

There are limitations with the ProMES approach, however, as there would be with any bottom-up approach. The precise formulation of the measure (both in terms of the specific indicators chosen, and the relative importance given to each) would probably vary depending on who was taking part in the exercise. It is also possible that some higher-level strategic viewpoints may be missed. The research team sought to minimise these limitations by including large enough numbers of participants to get a wide variety of viewpoints and by ensuring that enough experienced experts were included (e.g. several GPs and people who had held senior leadership positions within the NHS) among the workshop participants and contributors to the consensus exercise.

How feasible is it to measure general practice productivity?

The initial objective was to develop a standardised, comprehensive measure of general practice productivity, with productivity defined as a quality-adjusted ratio of outputs to inputs. Specifically, it was intended to develop a measure of quality-adjusted outputs (which might also be thought of as a measure of effectiveness), and use this measure as the numerator in a formula with practice inputs as the denominator.

In two respects, this objective was not fully met. First, the numerator measure (which became the GPET) was standardised to some extent and was judged to be comprehensive enough, but the standardisation was not sufficient to allow direct comparison between all practices. For example, some practices would inevitably score lower on some indicators, because local commissioning arrangements meant that certain services would not be provided in those practices. It was not possible to develop a model that would account for all possible commissioning arrangements in a fair way. Likewise, practices with higher proportions of older or unemployed patients, or patients from a lower socioeconomic background, would be disadvantaged on some indicators. The issue of comparability between practices is explored in greater depth later in this section. Therefore, although the measure would allow fair comparison between similar practices, or for tracking the same practice over time, it was not considered to be a measure of productivity that would be appropriate for performance measurement. In the researchers’ view, it would be almost impossible to create such a measure.

Second, an actual productivity metric could not be calculated because of the scarcity of financial data that practices were able to provide. Unlike the previous difficulty, this is one that should be surmountable given time and resources: practices should be able to record their monthly expenditure, although some clear definitions would have to be given. There was always likely to be a question of whether a straight ratio would be appropriate or whether some adjustment would be necessary (bearing in mind that the numerator of the equation is measured on a very different scale, ‘effectiveness points’, from the financial scale of the denominator). However, there was insufficient data even to examine this in any meaningful way. Instead, whether or not there was any link between effectiveness (the outputs) and resources was examined, as measured by payments per registered patient. There was no evidence of any link, although, of course, the sample size was small. It could be that a parallel ProMES exercise to determine measurements of resources would be an appropriate way to examine this more closely.

The primary outcome of this study, the GPET, should therefore not be considered a measure of productivity. Despite this, it is worth noting that the ProMES approach is based around productivity, as its full name suggests. However, ‘productivity’, as defined by the ProMES approach, is not the same as the classical definition of productivity, namely a ratio of outputs to inputs. Indeed, Pritchard et al. 27 define productivity as a mixture of effectiveness and efficiency. Although there are clear links between this and the often-used health-care-based definition of productivity (a quality-adjusted ratio of outputs to inputs), they are not one and the same. 16,27

Therefore, although it was not possible to get a full productivity measure, by some definitions the numerator alone – the effectiveness points as defined by the ProMES methodology, captured by the GPET – could be considered to capture one definition of productivity. 27 Indeed, examination of the specific indicators reveals a combination of those that are based around volume of activity, those that are based around quality of outcomes and those that combine both. Therefore, it could be that, in the absence of a true measure of productivity, this could be considered as a viable alternative.

An important question, however, is about the extent to which it allows comparability between practices. This is a more complex and nuanced question than it might at first appear. First, there is the question of whether or not all of the indicators are available at all practices. The selection criteria used for the indicators (described in Chapter 4, Detailed workshop methods) were such that this should indeed be the case, although some rely on completeness of practice records on areas as diverse as waiting times, missed appointments, staff training and sickness absence. Although, in principle, all practices should be able to measure and retain data for all of these, the fidelity with which they can do so varies. Second, some indicators may favour larger practices. For example, the ‘availability of enhanced services’ indicator includes 12 specialist services, many of which are more likely to be provided by larger practices with more resources. Does this make those practices more effective or productive? It could be argued that it does, but equally it could be argued that this results in an unfair comparison between two practices of very different sizes (at least, if a smaller practice is not part of a federation or other grouping of practices). Third, many of the indicators, particularly those in the clinical care performance area, will be affected by the specific local population. Although care was taken during the indicator generation phase to make the indicators as fair as possible (e.g. for ‘smoking cessation’, the indicator is the percentage of smokers who have been offered smoking cessation advice or treatment in the previous 12 months, rather than the percentage of patients who are smokers), there are still likely to be significant differences in such an indicator between a practice in a more deprived area where a greater proportion of patients are smokers, and a practice in a wealthier area where fewer patients smoke.

Overall, therefore, although the tool does give an effectiveness score that represents a real difference between practices, that is not to say that a practice with a higher score is necessarily a better practice, all things considered, as this may, in part, be a function of the practice size, local population and other external variables (although little evidence of this was seen in the data analysis). Thus, although such a score can provide useful information, it does not provide a fair comparison of the effectiveness of practices in different regions or of different sizes; to some extent, this could be overcome by ensuring that any comparisons are made only between practices of similar types, although that may be more limiting. Just as previous attempts to create a measure of overall performance have been limited for reasons such as narrow data (e.g. QOF) or irregular measurement (e.g. CQC inspections), a limitation in the tool produced here is that it cannot provide a fair comparison across all practices.

However, the GPET certainly does produce data that is comparable within a practice over time. Therefore, its use as a measurement of effectiveness to be tracked longitudinally could offer real value for practices, and may serve as a contribution to the call from the RCGP for QI tools. 64 However, this is not a QI tool in the traditional sense: to derive such a tool would have required a different approach and framework.

The extent to which this would happen in practice depends on the willingness of practices to use the tool to its full extent. From the evaluation, it appeared that willingness varied: some practices were not convinced of the benefit of the tool; far more could see its potential usefulness but could not spare the resources to make full use of it; whereas others found it very useful and were keen to continue using the tool if they could.

Successful use of the tool

The majority of practices participating in the study managed to use the tool successfully, although the extent of successful use was mixed. Fifty-one practices began the process, being trained in using the measure, and 38 practices independently proceeded to use it to enter their own data. Of these, 10 practices did not fulfil 5 months’ full data entry (and a further 10 did not fulfil the sixth month, but this was largely because of delays in starting its use, taking the start point to beyond April 2017).

Thus, 28 of 51 practices engaged sufficiently (i.e. included at least 5 months’ worth of data) for them to be considered to have used the GPET to its full extent. This is obviously lower than would have been desired, and suggests that there were particular problems in some circumstances with being able to use it effectively. Some of this related to specific data collection methods, which are explored in Specific areas for future development. However, many practices indicated that the time taken was a major factor in this, with 27 of out 39 practices who answered this question saying that it took more time than had been anticipated.

The ease of use was also considered mixed: the mean of 5.6 on a scale from 0 to 10 suggests that there were only slightly more positive than negative views on this (although this included practices that had not gone on to use the tool independently after being trained). In terms of usefulness, this figure crept below the mid-point to 4.5, suggesting that some practices did not see the value in investing the time necessary, although, as will be discussed in subsequent sections, those who did often saw more benefit.

Overall, therefore, it was found that not all practices were able to put in the time and effort to make best use of the tool, but a majority still did comply on the whole. It is certainly the case that, with the learning from this study, the time and effort involved could be reduced; therefore, the fact that this was the major barrier for many is somewhat encouraging.

It is important to note that this assessment was based on a relatively straightforward evaluation that was not based on any particular theoretical framework. This was designed to elicit relatively basic responses. A more thorough, theoretically grounded evaluation could have produced a more rigorous understanding of the issues faced by practices when using the GPET.

Changes in effectiveness

An interesting angle of the use of the GPET by practices in the pilot study was what would happen to their effectiveness over the course of the study. The analysis showed that the overall effectiveness score increased significantly over the months that practices participated. Even if the first month was ignored (to allow for practices getting used to gathering and entering data), there was an average monthly increase of 64.7 effectiveness points [95% confidence interval (CI) 13.1 to 116.3 points]. This represents 1.3% of total effectiveness points that were available, or a standardised estimate (change divided by SD) of 0.09. Although this is a fairly modest change – even over the course of 3-monthly changes, this represents a difference of 0.27 SDs, which is still close to what is traditionally thought of as a small effect (a change of 0.2) – the fact that there is any significant change at all deserves attention.

It should be noted that, not only is the change small in standardised terms, it is small in comparison with other studies that have used the ProMES approach. Pritchard et al. ’s73 meta-analysis found, across 83 field studies, an average improvement of 1.16 SDs. Although this was, typically, over a far larger number of periods (a mean of 19.8 periods across different studies), it was also the case that the effects tended to dissipate over time. 73 Thus, a relatively small change, such as the one witnessed in this study, is a long way short of what has been shown elsewhere. There are probably two main reasons for this. One is the added complexity of general practice (requiring 52 indicators in the GPET, compared with 8–12 indicators, typically, in the studies in the meta-analysis); this means improvements are averaged over far more scores, and it is not realistic for practices to improve on that many at once, especially as many indicators are actually measured over the course of a year, and some are unlikely to change much at all in a short period. Thus, the short time scale for the study was far from ideal for making an evaluation of the extent to which scores could be improved.

The other reason is that, in a typical ProMES exercise, many team members will be involved in the creation of objectives and indicators and this will increase the team members’ motivation, allowing the processes of feedback, goal-setting and participation to have a greater effect. 73 Most of the practice members involved in the pilot study had not themselves been involved in the development of the tool, and even though some representatives from practices had been, there was no evidence that this involvement was related to improvements in effectiveness (although there was some evidence that practices participating in the development of the tool were more likely to submit more months’ data).

If anything, therefore, it is remarkable that there was any significant improvement, and substantial caution is needed in its interpretation. Although the theory behind the ProMES process, that goal-setting, feedback and participation drive improvements, may explain this in part (or in full), it is worth considering other possible explanations of this. Of course, practices participating in the pilot were self-selecting, meaning that they may have been particularly geared towards improvement, or some may have chosen to be involved because they were aware that their performance was poor. It is also notable that of the 38 practices to enter data, over one-quarter (n = 10) did not complete 5 months’ data entry. Although this in itself should not have led to the differences seen, as the changes quoted are average within-practice changes, it does remain possible that those practices that dropped out were those where improvements were less likely. The analysis showed no association between number of months’ data entered and deprivation, practice size or initial effectiveness points, but there was no systematic method to capture a practice’s ability or willingness to improve.

It is important to note also that this was a straightforward longitudinal study with no control group. We cannot be certain how practices would have performed over this time period if they had not been using the GPET. Importantly, data were collected over the period of a few months, primarily between February and September 2017. There could easily be seasonal changes during this period that would affect the outcomes, either because of comparatively lower activity over the summer months, or because of the impact of needing to submit QOF data at the end of March. In addition, it could be that some practices collected (or entered) data with more fidelity and accuracy as the study went on. Therefore, the improvements seen need to be viewed with some caution. Further insight may be gained by looking at specific performance areas or objectives in this way.

There was no evidence of changes to the clinical care or external focus performance areas. Within the clinical care area, it is worth noting that most of the indicators relate to activity over the previous 12 months, and some over longer periods, with only a small number relating to shorter periods; specifically, one over 6 months, one over 3 months, one over 1 month (DNAs), and two that relate to current policies (availability of enhanced services and safeguarding, both of which require more significant effort to change). Therefore, to make substantial improvements within a month on these seems a very tall order, and it is not surprising that no significant change was seen. In addition, because of the overlap of several of these indicators with those in the QOF, it is likely that practices are already well aware of data in these areas and are trying to maximise their performance on these.

For external focus, there were far fewer indicators, and given the nature of these (requiring working together with external organisations, and/or the PPG/PRG), it is also not surprising that change would be slow. Whether practices would be sufficiently motivated to change these based on feedback from the GPET remains an unanswered question. This was the performance area for which the development of indicators was most problematic, largely owing to the availability of relevant data, and so a greater focus on capturing important indicators of externally facing work would be beneficial if improvement in this area is to be achieved.

For the other two performance areas (practice management and patient focus), however, there were significant improvements, not just overall, but on five out of the six objectives. The only objective that did not show significant change was good physical environment; again, this is something that would be likely to require considerable investment and/or time to change substantially. However, there were significant improvements in effective use of IT systems, motivated and effective practice team and good overall practice management among the practice management objectives, and both of the patient focus objectives (high levels of patient satisfaction with services and ease of access and ability to book appointments). This was particularly promising in the light of recent research showing that public satisfaction with GP services has been decreasing. 14

Many of these areas can be addressed by more modest changes; for example among the motivated and effective practice team objective, the proportion of staff with training needs met increased significantly, suggesting that, by examining this indicator, practices were able to identify and make improvements in this area. However, it is also notable that the amount of sickness absence among staff decreased; this could either be because of improvements in weather (and subsequent decreases in illnesses) over the summer months or because action was taken to try to reduce this in practices (or a combination of both). Likewise, improvements in patient satisfaction may be due to responsiveness to prior feedback, but may also be due to less pressure on services and more availability of appointments later in the process.

Overall, therefore, there is evidence that improvements were made, but it would need a more rigorous, controlled trial in order to determine whether or not these were as a result of the use of the GPET.

Performance area: clinical care

The content of this performance area included many indicators that were extracted automatically from clinical systems (using MIQUEST queries), and several that overlapped with the QOF. Participants found this unsurprising and helpful in many ways, although several commented that a more joined-up, automated system would be helpful. Some of the indicators in this area were similar to indicators that had previously been part of QOF and these were found to be particularly useful by some participants, as these were no longer automatically monitored.

It was noted that there are some conditions that are not explicitly covered by the indicators included in the GPET. For example, asthma, epilepsy and cancer were all brought up as conditions with no explicit indicators. Nevertheless, it was acknowledged that a tool of this nature could not reasonably cover every condition, even QOF does not do that, and that the conditions included would cover the majority of the important activity. Conditions such as those mentioned above (i.e. asthma, epilepsy and cancer) certainly have substantial overlaps with existing indicators; therefore, the coverage of the tool appears to meet the needs of practices.

One specific area that was suggested for inclusion, which does not substantially overlap with existing indicators, is women’s health (e.g. smear updates and breast screening). Consideration may be given as to whether or not this should be included in the future. Another area that was brought up by patient respondents was that of continuity of care. In particular, in the case of long-term conditions, it was felt there should be some measurement of whether or not a GP has a holistic overview of a patient.

Some of the services included did not apply to all practices. For example, several practices were not commissioned to provide a health check for patients aged > 75 years, resulting in a zero score for that indicator (meaning a negative effectiveness score: –30 effectiveness points). Consideration will need to be given as to whether or not all indicators belong as part of this performance area (or, at least, whether it should be optional), although it should be noted that if the comparisons made are largely to be within-practice, or between practices commissioned for the same services, this should not matter hugely.

One area that was considered by patients to be of less importance was the indicator of BMI reduction. This view was not shared by practitioners, however. There were four other indicators for which the extent of data entry was low: ‘alcohol consumption’ (31% of entries missed), ‘initial care of mental health conditions’ (69% of entries missed), ‘COPD care (care plan)’ (72% missed), and ‘lifestyle of people with long-term conditions’ (44% missed).

This may, in part, be due to the specific clinical system codes (Read codes) that were used, resulting in the queries being captured inconsistently in different practices; one practice respondent indicated that they ran their own searches to get a more accurate answer.

In any case, any future use of the tool will require an adjustment of the MIQUEST queries because of the change in clinical system codes from Read to SNOMED, which is now in progress. The adjustment of the indicators above, as well as more general changes, would need to be reviewed as new queries are written.

Performance area: practice management

This performance area is another for which it was generally considered that the GPET covers the main areas well. In particular, within the ‘motivated and effective practice team’ objective, the issues of staff retention and sickness absence were mentioned as useful indicators.

Only one indicator was flagged as being particularly problematic during the evaluation: this was the ‘staff well-being’ indicator, measured as the proportion of working-time lost due to sickness absence. Some respondents indicated that they did not previously collect the data in this way, and there were significant discrepancies in the apparent values that practices entered; although they were given instructions for calculating the percentage of working time lost in a month, there were several practices for which values of ≥ 20% were entered each month. This seems unrealistic, although there is no easy way of verifying this. (It should be stated that the significant changes in effectiveness scores persisted even if this indicator was excluded.)

The inclusion of the staff questionnaire on teamworking was viewed very positively by many practices, and was thought to illuminate areas of concern that had not previously been made known. However, it was questioned whether or not this needed to be conducted on a monthly basis.

Performance area: patient focus

Three of the five indicators for this performance area were gathered using questions asked as part of the FFT feedback exercise. Although some practices found this difficult to gather (25% of responses to the main FFT question were left blank and 31% of responses to each of the other two questions were left blank), many practices found this a very positive experience; in general, these were felt to be good indicators (with practices saying that they thought that these were more useful questions than the original FFT question). The fact that there was a significant improvement over time in this performance area (and in both objectives within it) further suggests that this area was useful. Some patient respondents felt that the FFT question offered diminishing returns, however, and wondered about having a staff FFT question instead.

One indicator was found to be slightly problematic: the hours of clinical appointments per thousand patients per week. This was found to be difficult to calculate, and a future iteration of the tool could ask for the elements separately (i.e. total hours of appointments and number of patients) and perform the calculation automatically. One practice thought that this would be more useful if split into different types of appointments (e.g. GP, practice nurse, health-care assistant appointments).

Although the indicator ‘percentage of patients waiting > 15 minutes past appointment time’ was generally viewed positively by practices, some patients indicated that communication about waiting times would be more relevant. However, nine practices indicated that this information was difficult or impossible to collect accurately.

Performance area: external focus

In many ways this performance area was the most difficult, in terms of generating appropriate indicators for the objectives and in terms of gathering accurate data.

One of the key indicators chosen for external focus was ‘attendance at MDT meetings’. Given that this involved partners from different organisations, it was felt by some participants that this was out of a practice’s control and, therefore, seemed unfair. To a lesser extent, the same can be said for the ‘working with different partners’ indicator. A counterview here could be that a proactive practice might be able to forge and encourage links, and attendance at meetings, with external organisations. However, the ‘attendance at MDT meetings’ indicator was not completed in 25% of cases, suggesting that record-keeping may be a problem in this context.

Likewise, there also appeared to be a record-keeping issue for the ‘number of meetings with PPG/PRG in the last quarter’ (25% incomplete), ‘amount of time staff spend in face-to-face contact with the public at appropriate external groups’ (81% incomplete), and ‘outreach and partnerships with local population and community’ (28% incomplete). However, in some cases, the lack of records could be indicative of little or no activity in the area.

With all of the problematic indicators discussed here, it is not the case that such information should be difficult to gather; however, the main problem is gathering the data retrospectively. If sufficient forewarning is given in advance of future use of the tool, the acquisition of such information would be less difficult.

Patients’ views on the PPG/PRG indicators were particularly interesting. The definition of a ‘virtual’ meeting could be problematic, for example respondents were adamant that an e-mail sent to a couple of people should not count, whereas the ‘resourcing of the PPG/PRG’ indicator could benefit from asking about the seniority of staff present and how much staff time was given to this. Another suggestion for inclusion was asking about the extent of representativeness of the PPG/PRG in terms of population demographics, although this would be difficult in practice. In addition, it was suggested that these indicators should be completed by patient representatives rather than staff. This would represent a departure from the approach used in the rest of the tool, but would be worth considering for future administrations, although it would provide other challenges, such as ensuring that there were sufficient members of the PPG (or other representatives) available to complete this section of the tool.

Another issue that was brought up with this performance area was the lack of good public health indicators. The public health responsibilities of general practices are varied, and the clinical care performance area includes many indicators that GPs considered (during the workshops) to be a measure of some aspects of public health. However, during the specific public health workshop, it was made clear that the public health remit of practices should go beyond this individualised, medical-based approach. Two indicators that were suggested were ‘social assessment and prescribing’ (the proportion of consultations including an assessment for social and economic issues affecting well-being, as well as clinical issues) and ‘health and work’ (the proportion of consultations that pick up on occupational or underemployment related sickness or ill health and provide support, treatment and/or referral to other agencies). Unfortunately, no common data sources could be found for these, and it became clear during subsequent workshops that most practices would not be able to gather such data. Therefore, these were left as optional indicators, and only two and five practices, respectively, chose to use them (with no guarantee about the quality of the data).

Therefore, an important issue for this performance area in the future would be the development of indicators that did capture this side of the public health responsibility of general practice.

The online General Practice Effectiveness Tool

The online GPET was adapted from an existing tool for ProMES systems, but designed so that the indicators (and objectives and performance areas) were fixed centrally, as were the contingencies which mapped indicator scores to effectiveness points, and could then be used across multiple practices. Each month, participants would get e-mails asking them to enter data on the relevant indicators, which would include a hyperlink to take them to the relevant page of the tool.

Overall, views about the online tool were positive, with many comments about its usefulness. However, there were a number of issues. One was the nature of data entry – there was a disconnect between the output from the MIQUEST searches and the data that needed to be entered (the ordering not being obvious); some respondents wished that it were possible to automate the procedure of entering the MIQUEST search results into the online tool.

All practices stated that it took them some time to get used to entering data into the tool and interpreting the reports. Until they were confident that the data collection was accurate, they did not share it with the wider team or patient groups.

Views on the tool itself were mixed, with some appreciating the ease of data entry, but others described it as clunky and not intuitive, with some relearning being necessary each month. A system whereby data enterers could move directly from one indicator to the next would have been an improvement, as would clearer on-screen descriptions of what was required.

There were some technical teething problems in some practices; these mainly related to the access to the tool (both getting past firewalls and in use of passwords). Important learning here was that initial e-mails need to be followed up to ensure that they do get through to practices; in some cases, formal access to the tool had to be granted by the network administrator.

Many participants suggested that monthly data entry was too frequent for some indicators and, for things that change less often, a quarterly or biannual data gathering would be more appropriate. This would ease the burden on data gathering, certainly, but would offer less opportunity (or less frequent opportunity) for genuine improvement and may decrease the level of familiarity with the process.

Use of feedback

Feedback from the tool was provided to practices in a number of ways, primarily visually, in the form of charts. For overall effectiveness, and for each performance area, each objective and each indicator, line charts showing how performance had changed over time were produced. In addition, pie charts indicating the percentage of available points achieved for each indicator, as well as charts showing how each score compared with the contingency, were produced. In particular, these latter charts helped practices identify on which indicators they would be able to make gains with relatively small changes. Furthermore, data were available to download in tabular format. Feedback was produced electronically via the online portal and could then be copied into other formats if desirable.

The longer that practices used the tool, and the more they had used the tool’s facilities to feed back progress to the team, the more positive they found the experience. When teams found that they had made an improvement in overall effectiveness, this contributed to their positive view of the tool.

Practices commented on the benefits of having all of the information in one place to identify key issues to work on. In many cases, practices had not yet managed to make the changes that would lead to improvements, but did plan to do so. The people involved in discussing the feedback varied significantly: 17 practices said that they discussed it at team meetings, although in only 13 of these had GPs been involved in the discussion. Most discussion had involved practice management staff, in either team or individual meetings. In five practices, at least some feedback had been shared with patient representatives.

Twelve practices said that they had taken some specific actions as a result of feedback, including increases of audits, amendments of templates for data capture and introduction of new care plans. Practices also liked the idea of being able to use the reports for other purposes (e.g. CQC) to show improvement, as well as being able to evidence their willingness to learn, develop and improve.

Some practices, however, said that they took the decision not to spend more time using the tool (including using and discussing feedback) as they considered it a lower priority than other issues, such as QOF, which would be more important to them in terms of financial incentives.

There were relatively few comments about the nature of the feedback provided, although some commented on the simple visual display being effective. One practice did suggest that an improvement would be to allow more manipulation of data and feedback, as well as an easier way of printing the graphs.

Overall, the experience of using the feedback was generally positive, and was more limited by available time than by the software itself.

Search strategy

1. exp General Practice/

2. exp Family Practice/

3. Primary Health Care/

4. (primary adj (healthcare or care)).mp.

5. (general adj practice$).mp.

6. (family adj practice$).mp.

7. 1 or 2 or 3 or 4 or 5 or 6

8. (practice$ adj3 character$).mp.

9. (practice$ adj3 level$).mp.

10. (practice$ adj3 modif$).mp.

11. 8 or 9 or 10

12. “Outcome and Process Assessment (Health Care)”/

13. effective$.mp.

14. Efficiency, Organizational/ or Efficiency/

15. productiv$.mp.

16. Quality Control/ or Quality Assurance, Health Care/ or “Quality of Health Care”/

17. qualit$.mp.

18. 12 or 13 or 14 or 15 or 16 or 17

19. 7 and 11 and 18

20. Policy Making/ or Fiscal Policy/ or Policy/

21. (quality adj outcome$ adj framework$).mp.

22. QOF.mp.

23. (pay adj3 performance$).mp.

24. 20 or 21 or 22 or 23

25. 19 not 24

26. limit 25 to english

Search strategy

1. exp general practice/

2. primary medical care/

3. (general adj practice$).mp.

4. (family adj practice$).mp.

5. (primary adj (healthcare or care)).mp.

6. (practice$ adj3 character$).mp.

7. (practice$ adj3 level$).mp.

8. (practice$ adj3 modif$).mp.

9. outcome assessment/

10. productivity/

11. quality control/or health care quality/

12. qualit$.mp.

13. productiv$.mp.

14. effective$.mp.

15. 1 or 2 or 3 or 4 or 5

16. 6 or 7 or 8

17. 9 or 10 or 11 or 12 or 13 or 14

18. 15 and 16 and 17

19. fiscal policy/or policy/

20. (quality adj outcome$ adj framework$).mp.

21. QOF.mp.

22. (pay adj3 performance$).mp.

23. 19 or 20 or 21 or 22

24. 18 not 23

25. limit 24 to english

Search strategy

(MH “Family Practice”) OR (MH “Primary Health Care”) OR “general practice*” OR “family practice*” OR “primary healthcare” OR “primary care”

AND

(practice* N3 level) OR (practice* N3 character*) OR (practice* N3 modif*)

AND

(MH “Quality of Health Care”) OR (MH “Outcomes (Health Care)”) OR (MH “Quality Assurance”) OR (MH “Quality Control (Technology)”) OR (MH “Organizational Efficiency”) OR (MH “Productivity”) OR qualit* OR productiv* OR effective*

NOT

(MH “Policy Making”) OR “quality outcomes framework” or QOF or (pay N3 performance)

Search strategy

general practice* or family practice* or primary care or primary healthcare

AND

practice* level or practice* modif* or practice character*

AND

productiv* or qualit* or effective*

NOT

policy or policies or QOF or “quality outcomes framework” or “pay for performance”

AND

limit to english