A qualitative and quantitative evaluation of the Advancing Quality pay-for-performance programme in the NHS North West

Provider capacity and pay for performance

The introduction of a quality measurement and reward initiative requires reliable technical support and a capable administrative staff. At its most basic, this requires a sufficient data system to gather and (if necessary) analyse data, and requires individuals responsible for advocating and implementing the scheme internally. A number of papers recognise electronic medical records as a key element in implementing both a successful P4P scheme and, in turn, an efficient health-care system. 23,24 In summarising the effectiveness of best practice tariffs, McDonald et al. 25 noted reduced performance on the quality measures that required a large amount of additional data collection and a restructuring of electronic data systems. Additionally, providers identified insufficient capacity in relation to the built environment (e.g. day-case surgery facilities) and workforce (e.g. orthogeriatrician capacity as a result of a failure to invest locally but also as part of a more general scarcity workforce issue). The built environment may help or hinder quality improvement (QI) initiatives. Additionally, where such initiatives require the shifting of responsibilities across professions, a rebalancing of the provider workforce may be needed.

Provider support for pay for performance

McDonald et al. 25 also identified a lack of support for certain aspects of a P4P programme in areas where clinicians disputed the content of best practice pathways. This suggests that obtaining provider support for a P4P scheme is vital for its success. Furthermore, an evaluation of the national CQUIN initiative in a later study by McDonald et al. 26 found that, although support was often present among managers involved in negotiating quality goals and related payment rules, such support was often absent with regard to frontline clinicians. There were various reasons for this, including a failure by managers to engage and inform frontline clinicians in the process.

Additionally, in some cases, commissioners sought to impose quality goals, which led to friction and less collaborative forms of working in addition to a tick box approach in some cases. Other studies highlight how a lack of engagement and ownership reduces the impact of such initiatives. 27 This suggests that scheme content needs to be informed by close discussion with clinicians and experts in order to ensure relevance, feasibility and commitment. Clear and robust communication and support mechanisms are needed between relevant parties during all phases of a scheme’s design and development, in order to avoid implementing a scheme of which clinicians are unaware or unsupportive. 28

The tendency of financial incentives to crowd out intrinsic motivation29 raises concerns that P4P may compromise altruistic desires to maximise the welfare of patients. Such effects may be minimised by ensuring that tangible rewards (such as money) are complemented by symbolic rewards (such as praise or public recognition). Provider support is likely to be more readily forthcoming when P4P initiatives involve applying incentives to standardised, simple processes, rather than more complex processes requiring greater cognitive application; limit commissioner use of coercive methods, such as surveillance and threats, when promoting quality measures; and acknowledge – if not ensure – that the level of incentive reflects the cost of additional effort required by participants. 30 Crowding out of intrinsic motivation may also be reduced by targeting the scheme at teams and groups rather than individuals and by involving clinicians in the development of indicators and exclusion criteria. 31

Pay for performance and targeting at individual health-care professionals versus an organisation/team level

In an aggregated (i.e. team as opposed to individual) measure, poor performance of one contributor may be hidden by high performance from other individuals. 32 Through performance monitoring of distinct individuals, there exists a more direct link between behaviour and output and a greater visibility of free-riding participants. Moreover, some procedures and care practices, such as the provision of discharge information or smoking cessation guidance, rely on the autonomy of single individuals, which may be undermined by team-level measures. 33

Evaluating individual-level performance for providers that see a low relevant caseload in the scheme timeframe may be overly complex when individual contributions to care are unclear, and misleading when individual physicians see only a small relevant caseload. 34 It may be unfitting to reward just one individual involved in a patient’s care where such care has spanned a number of health-care professionals. Rewards across clinical teams place greater emphasis on incentive schemes as a form of altruism and a way of improving welfare for patients, rather than as a source of additional income for individuals. 35

A team-level incentive scheme may still be supplemented with individual-level performance reporting, minimising issues of accountability and free-riding in group schemes. However, setting a sufficiently challenging minimum target for performance within a team may be preferable because this can also substantially reduce free-riding, by allowing peer effects to increase motivation.

Pay-for-performance and single indicators versus whole-pathway incentives

It may be inappropriate to incentivise single-indication quality measures without consideration of the patient’s other conditions, which may advance the risk of drug adverse events or treatment contraindications. 36 Furthermore, providing health care often involves the expertise of several individuals, across a substantial time period, and this is particularly the case for patients with chronic diseases or multiple comorbidities.

Many health conditions lack a clear care plan, particularly when the patient remains undiagnosed or with an unclear course of treatment. 37 Patient interactions with primary and secondary care services – hospitalisations, repeat prescriptions, rehabilitation, and so on – are often unpredictable in chronic health conditions such as diabetes or mental health disorders. This complicates the development of incentivised measures significantly, as they need to span the whole care pathway.

Pay-for-performance schemes based on care plans have been introduced, for example in Germany and Australia, and designed around a package of care spanning prevention and treatment, with several co-ordinated providers incorporated within one payment contract. 38 Rather than externally assessing the relative contributions of clinical teams or individuals, bonuses are often received at the system level, with teams and individuals rewarded at the provider’s discretion. However, these care plan P4P schemes have yet to be evaluated for effectiveness. 38

Voluntary versus mandatory participation

Voluntary P4P schemes allow providers with a sufficient level of infrastructure to join the scheme at their discretion, without the need to wait for lesser-resourced providers to develop their labour or IT capabilities. Voluntary P4P schemes may only attract those who are already performing highly, excluding those most in need of the programme. Furthermore, highly resourced providers may reap benefits from early participation, further widening the performance gap between providers. 39 Mandatory participation in P4P would therefore remove inequity between providers resulting from self-selection. However, coercing participation is likely to impact adversely on intrinsic motivation. 27

Evidence of over-representation of high performers in voluntary P4P schemes is inconclusive. One study of a Hawaiian quality initiative suggests that low- and high-performing providers were uniformly represented in the scheme. 40 A study of HQID, the US-based predecessor of AQ, suggests that those who volunteered for the scheme exhibited significantly different levels of patient volume and mortality. 9 Overall, conclusive evidence on the extent to which voluntary or mandatory participation affects performance is relatively absent from the literature.

Structuring bonuses

There are four principal structures for rewards under P4P:

1. a predefined, single threshold of achievement (benchmarking/target – e.g. completion of a given measure among x% of eligible patients)

2. multiple thresholds of achievement at predefined intervals (e.g. a small incentive for completion of a given measure among x% of eligible patients, followed by a larger incentive for completion among x + 10% of patients)

3. continuous reward systems, with incentive provided on a per-completed measure/patient basis

4. relative performance (a tournament – e.g. incentives shared among the top x performing providers).

These are often used in combination to reward a mixture of absolute, relative, and improvement in, performance.

Size of the incentives

Historical evidence suggests that incentive values in P4P schemes are rarely correlated with subsequent improvements in care quality. 20 Kristensen et al. 22 note that many systematic reviews of the effectiveness of P4P often only touch on the notion that bonus values and subsequent performance may be correlated or that bonuses should be equated with the marginal costs and benefits of the scheme.

Provider costs of pay for performance

Increased effort results in a cost to providers. This can involve communication, administration and data-collection costs, as well as the cost of the behaviour change itself. Some quality indicators will be more straightforward and/or less costly to implement than others: persuading individuals to receive a vaccination, for example, is generally easier than convincing them to quit smoking; and, conversely, provision of either service would be cheaper than administering magnetic resonance imaging or computerised tomography for a patient.

In converting the cost savings of QIs into provider tariffs, rewards reflect the impact of high performance on cost savings in the system – a tariff design used in the first major Medicare demonstration48 but yet to be seen in the UK. However, many improvements in quality are difficult to quantify in financial terms, such as programmes to raise patient satisfaction or to target long-term primary prevention. Furthermore, using expected savings to set incentive levels would represent a significant administrative burden and require frequent re-evaluation as clinical evidence evolves. Cost savings-based incentives that do not explicitly refer to the provider and commissioner costs of implementing and maintaining the programme when estimating returns result in flawed and/or partial evaluations of cost-effectiveness.

Many P4P schemes (such as AQ) have been introduced with an intentionally minimal direct focus on cost savings; the principal objective is often instead to improve and standardise care quality for patients, with any subsequent cost saving superfluous to the general cause. Setting bonuses proportionate to cost savings automatically places emphasis on the scheme as a method for saving money, rather than for any altruistic reasons, which may lead to disengagement from participants (see Provider support for pay for performance).

Applying optimal bonus values to pay-for-performance schemes

Optimal service provision, from the perspective of a provider, is to continue implementing quality measures until the cost of doing so for one extra patient is equal to these marginal benefits. Hypothetically, providers would still choose to participate in the absence of a financial bonus, as long as the return from the altruistic component and marginal benefits to patients outweigh the costs. In reality, however, this perspective would be financially unsustainable, from the perspective of the commissioner at least; all NHS funds have an opportunity cost in terms of the potential health gains foregone in other areas.

Evaluating the benefit associated with increased service quality among providers is difficult: altruism as a concept would be complex to quantify and measure, and individual physicians and providers would obtain differing levels of personal satisfaction from improving patient care via P4P measures.

In practice, the cost savings and altruistic sentiments resulting from P4P are often unknown or unquantifiable. In a recent evaluation of the CQUIN scheme, although some attempt was made to mirror incentives to effort required or the level of priority assigned to a measure, decisions on reward weightings were taken on an ad hoc, localised basis, rather than with any formalised evidence to hand. 26

Pay for performance and penalties

Economic theory49 suggests that ‘people impute greater value to a given item when they give it up than when they acquire it’. 50 Although this implies that penalties may more effectively encourage behaviour changes among providers than would equivalent rewards, more recent evidence suggests an increased level of gaming of the system and a greater demotivating effect when providers are faced with potential losses. 30 Penalties may also reinforce a cycle of poor performance, often penalising providers most in need of investment that already lack the financial ability to innovate and implement change (see Werner et al. 39 for an example comparing safety-net and non-safety-net hospitals in the USA).

Contents of incentivised measure sets

A key consideration in scheme design is whether or not measures should focus on processes or outcomes. Patient outcomes are often driven by factors outside the control of providers; mortality and morbidity rates are likely to be higher in areas of high poverty, with greater disease prevalence and with lower patient education. Process or structure measures are less likely to be influenced by environmental factors outside the control of the provider and so may reduce provider concerns. Some specialties may be more ideally suited to specific measure types, owing to the way in which care is provided and care quality measured. 51

Some quality measures, such as smoking cessation, have applicability across a number of conditions; often, however, quality measures suffer from a lack of applicability past one or two similar clinical areas. Indicators are therefore often targeted at specific diseases, particularly those with a significant patient population, within which such schemes may have a greater potential impact. P4P is suited to standardised, well-defined procedures; it is less well suited when measures of performance are difficult to define, obtain or evaluate. 47

Process measures should be within the provider’s control to implement. For example, patient compliance with lifestyle changes such as diet or exercise would be difficult for a physician to monitor and verify. 52 P4P measures should be visible in terms of accountability, applicability and effectiveness; there may exist significant gaps between processes that can be viably quantified and subsequent patient health outcomes.

Quality measures will be most easily monitored and reported at the point at which providers collect appropriate data as part of standard procedure. Providers need to ensure that they are collecting the relevant data for measurement; calculating risk adjustments for patients, for example, may require non-standard, supplementary demographic or clinical information.

Measures should be designed such that at the point of evaluation there exists a clear distinction between high- and low-performing participants;34 a reduction in the spread of achievement across providers is a key identifier of improvement and a need to amend the scheme’s design. In the absence of baseline variability, evidence of provider improvement is less clear. 53

Rewarding care quality in a limited number of measures may lead to a more narrow focus on only incentivised measures or conditions, whereas larger bundles of measures may induce more general improvements in care quality. However, using a large number of measures increases the overall size and burden of the scheme and correlations between indicators are more likely to exist in a larger set, thereby increasing the risk of large gains and losses between participants. 54 Increasing the number of measures also dilutes the importance attached to each, which may result in a lack of effort and therefore improvement, if the influence of a measure on overall performance is perceived to be insignificant.

Locally developed measures have the potential benefit of greater relevance to local health needs. Providers can develop P4P schemes aligned to their own long-term QI efforts, with local schemes used to pilot indicators for later use at national level. In promoting CQUIN to providers, the NHS advocated the new framework as ‘an opportunity for commissioners and providers to focus on delivering higher levels of quality of care for their populations, rather than responding to centrally directed targets’. 55

National measures, however, allow participants to benchmark progress and achievement against other providers and are more likely to represent a cohesive, comparable set of quality standards. Evidence from the CQUIN evaluation suggests that, although local contribution to quality measures was considered vital to the success of the scheme, frontline clinicians were rarely encouraged to involve themselves in the development of CQUIN goals and the technical design may have been better suited as a centrally directed initiative. 22 A mixture of national- and local-level targets may be best suited – although clinicians may quickly tire of the multiple requests for data potentially resulting from schemes in place at both local and national levels, with a risk of overlap or conflict in quality measures as their volume increases.

Provider case mix issues

The significant resource requirements of unusual patient case mixes are likely to affect achievement potential for P4P participants, with slower improvement rates and lower achievement levels. For example, specialist providers may treat and manage patients with more severe diagnoses and may therefore have greater difficulty meeting certain process and outcomes indicators. Participants in areas with greater poverty will perform less well on outcomes indicators that measure risky behaviours such as smoking and drinking levels. Low levels of education are likely to affect patient adherence and co-operation, thereby slowing improvements in process measures relying on actions from the patient, such as attending support groups or adhering to medication. 56

Setting risk adjustments across providers creates a method for standardising performance across different population groups. However, risk-adjustment methods represent an additional administrative cost and, for the most part, remain underdeveloped and unsupported by clinicians. 57 Categorising participants into comparator groups based on patient demographics would limit benchmarking to a smaller number of similar providers and is therefore limited to situations where a sufficient number of comparator providers exist.

Pay-for-performance schemes often incorporate a system of exclusion reporting for patients considered to be ineligible for receipt of one or more quality indicators (such as a contraindication for a particular pharmacological product or opting for palliative care instead of treatment). Such criteria are seen as important in ensuring that patients are not given unsuitable care and allowing health-care professionals to exercise their clinical judgement without fear of being penalised. Eligibility criteria must, however, be clearly defined in order to avoid providers viewing exclusion reporting as an opportunity to exclude seriously ill or non-compliant patients who would otherwise reduce achievement in quality measures.

Unintended consequences and pay for performance

Positive spillover effects

Some QI measures have the potential to bring broader quality changes to non-incentivised conditions or aspects of care. A US P4P scheme incentivising completion of measures for diabetes patients under a managed care programme encouraged the rollout of the measures to patients across all health plans, despite the absence of an incentive provision for such patients. 52 Such positive spillover effects are likely to be amplified when participants are encouraged to communicate and collaborate with non-participating departments and providers, and when quality measures are applicable across a number of indications (such as discharge documentation) rather than strictly limited to incentivised conditions.

Supporting levers to accompany the financial incentives

Funding pay-for-performance initiatives

The source of funding for P4P schemes has implications on how providers view such programmes. Withhold schemes encourage the view that high-quality care should be standard practice, rather than an action warranting additional rewards. However, they may also encourage the view that P4P acts as a way for commissioners to hold back much-needed funds, a sentiment acknowledged in an evaluation of CQUINs. 73 There is also evidence that suggests that clinicians view withholds as unfair, coercive and contrary to the spirit of collaboration that should characterise P4P initiatives. 27 The use of a system whereby any unearned bonuses or capitation are paid out as bonuses to high performers on top of their original incentive value – termed a challenge pool in US literature74 – might encourage motivation when participants receive a reward greater than that of the original withheld funds, although this has yet to be used systematically. Similarly, in a combined reward–penalty scheme the income generated from fines to low performers would be used as bonus monies for high performers. Expected returns from penalties also need careful calculation in order to avoid owing bonuses greater than available funds (see Kahn et al. 75 for an example of this in HQID).

Designing achievement targets

Measure targets should be based on the capacity of a provider to improve in the subsequent period; although wealthier, highly resourced providers can be set a more challenging target than providers experiencing greater cost constraints, targets should maintain achievability among all participants. For tournament-style schemes, this could mean the grouping of providers based on resource availability, with rewards provided to winners in each group. For threshold-style schemes, participants may each work within their own set of performance thresholds or, again, may share targets with other similar providers. Alternatively, instead of benchmarking to competitor providers, targets may be set based on an individual participant’s performance in the preceding period (with a predetermined level of performance in the baseline period). Nevertheless, targets for all providers should lead to sufficient QIs (and, in turn, health benefits) so as to justify the opportunity cost of the scheme.

However, participants may become aware that targets in a subsequent period are based on current performance and may, therefore, reduce effort levels in the preceding period in order to maintain a lower future target. Furthermore, if targets do not actively encourage low performers to catch up with higher achievers, performance disparities may become institutionalised across providers. Targets must therefore be based on both the capacity and the relative need of the provider to improve relative to its peers.

Phasing in pay-for-performance schemes

In the early stages of a P4P scheme, when provider engagement is still in formation, the intense focus on specific procedures of care may be a shock for participants not yet acquainted with the nature of P4P. Larger schemes require the recruitment of, and support from, a greater number of health-care professionals, which may be difficult to achieve from the outset. The use of a pilot system has therefore been suggested as a valuable method for phasing in P4P initiatives. 65 This could involve testing the scheme in a limited geographical area or within a select number of indications or providers. In addition, baseline data during the piloting period can be used to set benchmarks and performance targets for when the scheme is fully implemented. Participation in the scheme may be voluntary for a set period, so that those providers requiring additional time to bring internal systems up to requirement need not be penalised in the interim.

Alternatively, participants could be incentivised on quality measure reporting levels alone (i.e. with no targets set for performance in the quality measures). Providers are then able to learn which areas of their infrastructure need investment prior to entering a full P4P scheme. Such a system of pay for reporting (P4R) was implemented in the USA prior to HQID, albeit not with the original intention of using it as a transition method into P4P. However, authors of a study into the Physician Quality Reporting Initiative concluded that P4R would act as a useful preparation technique for providers subsequently involved in P4P. 65

The process by which providers join a P4P scheme will affect the ease with which performance can be evaluated. As discussed previously, allowing providers to volunteer to participate in a scheme is likely to attract participation from those who expect to obtain a net benefit from opting in. In turn, extrapolating performance potential and applying to the (non-participating) remainder of the provider network may bias a priori estimates upwards. Mandatory participation of all providers, however, removes the ability to compare performance with a control group of non-participants, so that the counterfactual performance across the same time period may be determined. 76

Random selection into a compulsory scheme can instead ensure that the scheme maintains a mixture of providers operating at various levels of baseline performance. Subsequent achievement in P4P should therefore more accurately reflect the achievement potential of non-participating organisations. Alternatively, limiting performance to a small number of providers, selected on the basis of sharing some common characteristic, creates the ideal basis for a natural experiment. This is the case with our evaluation and enables us to compare performance in AQ providers in the north-west of England with those in the rest of the country. By observing performance across comparator organisations over the same time period, the relative effects of the P4P scheme may be isolated from national trends in performance. Ensuring that other contemporaneous shocks are accounted for when evaluating performance is vital, as is the ability to gather equivalent performance data from non-participating organisations from baseline onwards. In the case of AQ, although performance on process measures outside the NHS North West is not routinely available, we obtained outcomes (i.e. mortality rates) and use those for comparative purposes.

Quantifying achievement in pay for performance

Performance in P4P is often quantified by amalgamating achievement on individual indicators into a single composite score. Although methods for developing this score are numerous, two of the most frequent are the composite process score (CPS) and the appropriate care score (ACS). 77 The CPS represents the proportion of situations in which P4P measures have been appropriately administered. The ACS represents the proportion of patients receiving all care-quality measures for which they are eligible (Figure 1 shows an example calculation of each in AQ). Performance as measured by an ACS is likely to be poor if one or more measure is particularly difficult to complete, suggesting the use of a CPS in the initial stages of a scheme when measures are often re-evaluated for feasibility. The ACS is particularly useful when performance in the CPS has little variation and when there exists scope for additional measures: ‘it is more difficult to provide all the required measures of a large set than a small one’. 66 This makes the ACS more challenging as more measures are added. If measures complement one another, in the sense that the completion of a full set of measures leads to a superior outcome for the patient than completion of single measures alone, the ACS would be the appropriate score to use.

FIGURE 1.

Average quarterly hospital achievement for each condition across indicators. Note that the vertical line indicates start of long-term period (month 19 of the programme).

In amalgamating indicator-specific scores into a composite value, it may be desirable to weight each indicator relative to its clinical benefit or financial cost of completion. As a result, performance in a heavily weighted indicator will influence substantially a provider’s subsequent achievement. From the perspective of commissioners, however, individual weightings often represent additional analysis time and costs; from the perspective of clinicians and patients, they represent a complex obstacle to understanding performance. 78

Adjusting the pay-for-performance scheme over time

Some targets may require amending or retiring in order to reflect new evidence, such as those relating to clinical effectiveness or safety (see Reeves et al. 79 for an example in the QOF). Some quality measures may become more difficult or impractical to attain, owing to changes in either the internal provider environment, such as the supply of hospital beds (see CMS80 for an example in pneumonia), or the external environment, such as changes in socioeconomic conditions. 81 For most measures there exist natural ceiling or plateau effects and decreased returns to effort and investment, at which point financial incentives will no longer be sufficient to improve quality further. Finally, some incentivised behaviours may become so ingrained into standard practice that rewards are no longer necessary: a justification for re-evaluation again noted in a formal summary of QOF. 82 This may be seen as the ultimate success for a P4P programme.

If attainment in a particular measure is peaking at a suboptimal level, further investment or higher rewards may be required. As noted previously, certain internal and external barriers to achievement may mean that the marginal cost of initiating a quality measure on additional patients outweighs the marginal return in incentive. Inducing greater effort would therefore require a rebalancing of incentive value. Performance on a measure may be sufficiently high to warrant its discontinuation. Rather than an immediate removal of an indicator, however, it may be more appropriate to phase out the performance measure. If quality metrics are expected to change radically on a regular basis, support from providers may be more difficult to achieve, particularly for targets requiring significant investment, under the expectation that insufficient returns would be seen within such a short implementation period. 26

This idea of gradually evolving P4P measures has been used in QOF. Instead of requiring participating GPs to measure the blood pressure of patients with diabetes, for example, participants were subsequently required to ensure that the patient’s blood pressure was maintained below a predetermined threshold for the 15 months preceding evaluation. This has the benefit of maintaining focus on the same objective across both measures, while shifting the goalposts in the level of effort required from the provider. 79 Furthermore, a gradual shift from a process to outcomes measure should allow low-performing providers to continue towards the original (easier) quality measure, at the same time as encouraging continuous improvement from high performers.

Retired indicators should continue to be monitored in order to check for depletions in performance following their removal, as occurred in one process measure for the QOF. 79 Such occurrences may be a result of a lack of support from participants prior to incentive provision or because they represented a significant burden on workloads.

As we explain in this report, the AQ programme embodied many aspects of what might be seen as desirable design features of P4P schemes. However, this is no guarantee of success, as the implementation of P4P initiatives is likely to require substantial changes to practice. Evidence suggests that efforts to introduce such change frequently fall short of intentions, resulting in variable outcomes. 83

Commissioning for Quality and Innovation Payment Framework

From 1 April 2010, AQ became part of the NHS North West CQUIN payment framework, which meant that changes were made to payment rules. Whereas AQ was a tournament-style system that (in year 1) rewarded top performers, CQUIN involved setting provider-specific stretch targets that reward an agreed level of improvement from the previous year’s baseline. This meant that all providers that achieved the agreed performance were eligible for CQUIN payment. The local stretch targets for each provider for AQ year 3 were set by the SHA AQ team, which used year 1 results as baseline owing to the unavailability of year 2 data at the time CQUIN targets had to be disseminated to providers (March 2010). In some cases, low year 3 targets resulted in payments to providers that had performance below the average performance of year 2.

In October 2010, the SHA AQ team was incorporated into the Advancing Quality Alliance (AQuA). This is a membership organisation established in the north-west of England to support QI work in the region. AQuA is funded by member organisations, which were asked to pay £55,000 each for 1-year membership in the first year, but subsequently paid £25,000 each per year. All North West England providers joined AQuA in the year following its creation. AQuA is a membership health-improvement organisation, whose vision involves supporting members ‘to transform the health and quality of healthcare for the people they serve’. 99 (For more information about AquA, see www.aquanw.nhs.uk.) The AQ programme (initially funded by the PCTs) provides a large share (62%) of AQuA’s income (£5,190,222 in 2011), with 4% of this spent on AQuA staff and the remainder on external consultancy support. In practice, for AQ providers, AQuA organises and runs collaborative events (such as AQ leads meetings and collaborative meetings in each clinical area), as well as facilitating the sharing and dissemination of collected data. AQ is regularly described as the flagship programme of AQuA. The creation of Clinical Commissioning Groups (CCGs) means that AQuA now has to negotiate with CCGs on the subject of payment and focus of work.

Impact on mortality in the short term: methods

We obtained patient-level data from national Hospital Episode Statistics (HES) from the NHS Information Centre (now the Health and Social Care Information Centre) for all patients in England treated for three of the five conditions included in the scheme. We did not include hip and knee surgery because mortality following elective joint replacement is < 1%. We also did not consider CABG because only four hospitals out of the 24 in the North West region of England undertook this procedure.

Hospital Episode Statistics in England records deaths that occur within any hospital. We focused on all deaths that occurred within 30 days of admission. Published national statistics show that over 90% of deaths within 30 days for the incentivised conditions occur in a hospital. To check that there were no changes in discharge policies that might have led to more deaths outside hospitals, we also analysed changes in the proportions of patients discharged to care institutions rather than their own homes.

We obtained equivalent data for patients admitted with six primary diagnoses which were not incentivised. These conditions were chosen based on published statistics at national level13 to meet the following criteria: (1) not clinically linked to any incentivised condition; (2) sufficient volume (over 9000 admissions in England per year); (3) 30-day mortality over 6%; and (4) more than 80% of deaths within 30 days occurring in a hospital.

Six diagnoses met these four criteria and were treated as reference conditions: acute renal failure [International Classification of Diseases, Tenth Edition (ICD-10)100 codes beginning N17]; alcoholic liver disease (K70); intracranial injury (S06); paralytic ileus and intestinal obstruction without hernia (K56); pulmonary embolism (I26); and duodenal ulcer (K26). We excluded from the reference group all patients with a diagnosis included in the incentive scheme on any of their admissions over the 3-year study period. Our comparators therefore include two mutually exclusive sets of patients: one set with an admission covered by the scheme in hospitals not included in the scheme; and one set with an admission for a reference condition and no mention of any diagnosis covered by the scheme on any admission within the 3-year period.

Data were obtained for patients admitted over a 3-year period from 1 April 2007 to 31 March 2010. This 3-year period includes 18 months prior to the introduction of the scheme and the first 18 months of its operation. The data set includes patients treated at the 24 NHS hospitals in the North West region and the 130 NHS hospitals in all other regions of England which admitted more than 100 patients with each condition over the 3-year period. The final sample contains 410,384 patients with pneumonia, 201,003 patients with heart failure, 245,187 patients with AMI and 241,009 patients with non-incentivised conditions, treated at 154 different hospitals. Hospital characteristics were obtained from the websites of national regulators (e.g. Healthcare Commission and Monitor) and the NHS Information Centre.

We calculated expected risks of mortality using regressions at patient level which included sex and age (the primary ICD-10 diagnosis code); 31 Elixhauser comorbidities derived from secondary ICD-10 diagnosis codes; the type of admission (emergency or transfer from another hospital); and the location from which the patient was admitted (own home or institution). The analysis of risk-adjusted mortality was undertaken on data aggregated by the quarter of the year and by admitting hospital.

We tested whether or not the incentives had an impact on mortality in three ways: (1) a between-region difference-in-differences analysis comparing the changes in mortality over time between the North West region and the rest of England for incentivised conditions; (2) a within-region difference-in-differences analysis comparing the changes in mortality over time between the incentivised and non-incentivised conditions in the north-west of England; and (3) by estimating a triple difference, comparing the changes over time in mortality between the incentivised conditions in the North West region and the rest of England and between the incentivised and non-incentivised conditions. The triple-difference analysis captured the effect of the programme on mortality for the incentivised conditions in the North West region controlling for the effects of changes over time in mortality for the incentivised conditions owing to factors other than the initiative itself, in addition to changes over time in overall mortality in the North West region and differences in mortality between the incentivised and non-incentivised conditions between the North West region and the rest of England.

We estimated the effects for all three incentivised conditions combined and then separately for each condition. Each analysis allowed flexibly for time trends using a binary variable for each of the 12 quarter-years, and for hospital differences using a binary variable for each hospital, and includes an interaction term between the intervention group and the post-implementation period.

Impact on mortality in the short term: results

The characteristics of patient populations in the North West region and the rest of England before and after the scheme’s introduction are shown in Table 1.

For all conditions, patients in the North West region were slightly younger but had more comorbidities. Similar changes over time in patient volumes and patient characteristics are observed in both areas. The profile of hospitals in the North West region is similar to the rest of England (Table 2), with a slight tendency for fewer hospitals in the North West region to have received the lowest ratings by the national regulators for overall care quality and financial management in 2007.

Risk-adjusted mortality rates for all of the conditions we studied decreased over the study period in both the North West region and the rest of England. The reduction in mortality for incentivised conditions was greater in the North West region than in the rest of England, reducing from 21.9% to 20.1% in the North West region and from 20.2% to 19.3% in the rest of England (Table 3). Compared with non-incentivised conditions within the North West region (within-region difference-in-differences analysis, Table 3), there was a significant reduction in overall mortality for incentivised conditions of 0.9 percentage points (95% CI 0.1 to 1.7 percentage points), comprising a statistically significant reduction in pneumonia and non-significant reductions in the other two conditions. Comparing mortality for the incentivised conditions with mortality for the same conditions in other regions (between-region difference-in-differences analysis, Table 3), there was a significant reduction in overall mortality in the North West region of 0.9 percentage points (95% CI 0.4 to 1.4 percentage points), again made up of individually significant reductions in pneumonia and non-significant reductions for the other two conditions. Combining these two (triple-difference analysis, Table 3) suggested an overall reduction in mortality of 1.3 percentage points in the North West region (95% CI 0.4 to 2.1 percentage points), with a similar pattern for the individual conditions. The reduction in mortality over the period studied for non-incentivised conditions was not significantly different between the North West region and the rest of England.

Our finding that risk-adjusted mortality for the non-incentivised conditions reduced by similar amounts in the North West region and in the rest of England suggests that our findings are not explained by higher preintervention mortality or by a general improvement in quality or reduction in case-mix complexity in the study region. Nonetheless, we undertook a wide range of further analyses to test the robustness of our findings. There were no significant changes in the proportion of patients discharged to care homes and all differences were smaller than 0.3 percentage points. We verified that the trends in mortality were similar in the two areas prior to the scheme’s introduction. We also checked that our findings were unaffected when we controlled for changes in patient volumes and baseline mortality and when we compared the North West region with a subset of similar English regions.

Further examination of the additional mortality reductions in the North West region showed few differences by hospital type. Small hospitals, and hospitals rated as having excellent- or good-quality services by the Care Quality Commission prior to the scheme, showed the largest mortality reductions. North West region hospitals rated as having weak- or fair-quality services prior to the scheme did not reduce mortality more than similar hospitals in other regions.

Impact on mortality in the longer term: methods

Hospital-level data on quarterly performance on the metrics incentivised by the programme were obtained from the AQ programme. Patient-level data on patient characteristics, coexisting conditions and mortality were obtained from national HES. As in our 18-month analysis, we used data for all patients in England treated for the three conditions covered by the programme for which patients are admitted in an emergency and from which there is a substantial mortality rate within 30 days, that is, AMI, heart failure and pneumonia.

We obtained equivalent data for patients admitted in an emergency with five of the six non-incentivised conditions that we had considered in our initial work. The sixth condition was excluded as it had been the subject of a national QI programme. We considered the possibility that the quality of care provided for patients admitted with the remaining five conditions may have benefited from the improvements in the care offered in emergency departments of the participating hospitals in the longer term.

Data were obtained for patients admitted between 1 April 2007 and 31 March 2012. We divided the data into three periods: (1) before – 18 months prior to the introduction of the scheme; (2) short term – the first 18 months of its operation; and (3) long term – months 19–42 of the programme. The final sample contained 390,652 patients admitted for AMI, 338,921 for heart failure, 761,954 for pneumonia and 333,991 patients admitted for non-incentivised conditions, treated at 161 hospitals across England. We analysed mortality occurring in any hospital within 30 days of admission.

As before, we calculated expected risks of mortality using logistic regressions at patient level which included sex and age; the primary ICD-10 diagnosis code; 31 Elixhauser comorbidities derived from secondary ICD-10 diagnosis codes; the type of admission (emergency or transfer from another hospital); and the location from which the patient was admitted (own home or institution).

The analysis of risk-adjusted mortality was undertaken on data aggregated by 3-month period and by admitting hospital. We tested whether or not the incentives had an impact on mortality using a between-region difference-in-differences analysis comparing changes in mortality over time in the hospitals in the North West region with those in the rest of England.

We tested whether or not the incentives had an impact on mortality in two ways: a between-region difference-in-differences analysis comparing changes in mortality over time between the North West region and the rest of England for the incentivised and non-incentivised conditions; and a triple-difference analysis comparing the changes in mortality over time between the incentivised conditions in the North West region and the rest of England with the changes in mortality over time between the North West region and the rest of England for the non-incentivised conditions.

We estimated the effects for all three incentivised conditions combined, all five non-incentivised conditions combined, and then separately for each condition. We weighted the condition-specific mortality rates using total volumes over the entire period to ensure that the combined mortality series did not reflect changes in the relative volumes of patients admitted for different conditions. All analyses allowed flexibly for time trends using an indicator for each of the 20 quarter-years, and for hospital differences using an indicator for each hospital in the separate condition analyses and an indicator for each hospital–condition combination in the combined analyses. We estimated separate impacts for the short- and long-term periods by including interaction terms between the intervention group and each of the two post-implementation periods.

Impact on mortality in the longer term: results

The average performance reported by the participating hospitals on all of the quality measures improved in the first 18 months and improved further in the following 24 months, particularly for heart failure and pneumonia (Table 4). Analysis of performance by quarter (see Figure 1 and Appendix 4) shows that rates of improvement slowed over time and some quality measures, especially for AMI, plateaued at high levels of achievement towards the end of the period.

The characteristics of the patient populations in the North West region and the rest of England were similar before the scheme’s introduction, with a slight tendency for patients in the North West region to be younger and to have more coexisting conditions (Table 5). Similar changes over the short- and long-term periods in patient volumes and patient characteristics are observed in both areas.

Risk-adjusted mortality rates decreased over time in both the North West region and the rest of England for both incentivised and non-incentivised conditions (Table 6). The between-region difference-in-differences analyses show a significant impact of the programme on patient health in the short term (–0.9 percentage points, 95% CI –1.3 to –0.4 percentage points), comprising a statistically significant reduction in pneumonia (–1.5 percentage points, 95% CI –2.3 to –0.4 percentage points) and non-significant reductions in the other two conditions [AMI (–0.1 percentage points, 95% CI –0.9 to 0.6 percentage points) and heart failure (–0.2 percentage points, 95% CI –1.1 to 0.7 percentage points)]. The triple-difference analyses show an overall short-term effect of –1.5 percentage points (95% CI –2.6 to –0.5 percentage points), comprising a statistically significant reduction for pneumonia (–2.2 percentage points, 95% CI –3.4 to –1.0 percentage points) and non-significant reductions in the other two conditions.

Between the short- and the long-term periods, risk-adjusted mortality for the incentivised conditions fell by 2.3 percentage points in the rest of England and 1.8 percentage points in the North West region. This reduction in the rest of England is significantly larger (0.7 percentage points, 95% CI 0.3 to 1.2 percentage points) than in the North West region, and is again concentrated in pneumonia (1.1 percentage points, 95% CI 0.4 to 1.8 percentage points). However, the reductions in mortality were larger for the non-incentivised conditions in the North West region than in the rest of England between these periods. The triple-difference analysis shows a larger reduction in mortality (1.9 percentage points, 95% CI 1.0 to 2.8 percentage points) for the rest of England than in the North West region between the short-term and long-term periods.

We verified that the trends in mortality were similar in the two regions before the introduction of the programme (see Appendix 4) and confirmed that our findings were unaffected when we also included out-of-hospital deaths (which were less than 1 percentage point for all conditions) (see Appendix 4), and baseline mortality (see Appendix 4), excluded a small group of control hospitals for whom incentives for the same conditions were introduced in the long term (see Appendix 4), and when we examined 90-day rather than 30-day in-hospital mortality (see Appendix 4). We do not present these results in the main analysis, as the data were incomplete for the final 3 months of the study period.

In the long-term period of the study (19–42 months), two other regions outside the North West region (the South Central and South East Coast regions) adopted the AQ process measures which had already been used in North West region, although without the full supporting mechanisms in the AQ programme. For these two regions (which named their programmes ‘Enhancing Quality’ and ‘Improving Quality’), in the long term, there was a financial incentive for performance on the AQ process measures, with money being withheld from hospitals in these regions if they failed to perform to negotiated standards on the AQ process measures.

To test whether or not the adoption of the AQ measures had an effect on mortality in these two additional regions, we conducted a between-region difference-in-differences analysis similar to our main analysis, but identifying the South Central region and South East Coast region as a separate group (labelled ‘late-adopter regions’).

Table 7 shows that in the first 18-month period (the short term), when AQ was introduced in the North West region but before the process measures were introduced in the late-adopter regions, the reductions in mortality were similar in the rest of England and the late-adopter regions. This was as expected given the lack of incentives in the late-adopter regions at that time.

In the longer term (when the AQ process measures were used in the late-adopter regions), reductions in 30-day in-hospital mortality were greater in the late-adopter regions than in the rest of England, but this difference was statistically significant only for AMI (–0.7 percentage points, 95% CI –1.3 to –0.1 percentage points). This suggested that the adoption of the AQ indicators in the two late-adopting regions was associated with a reduction on mortality attributable to AMI.

To investigate the possibility that the apparent loss of effect of AQ on the incentivised conditions in the north-west of England in the long term was a result of improvements in care in the non-incentivised conditions in the intervention hospitals, we first examined the between-region difference in differences separately for each of the non-incentivised conditions. This was similar to the separate analysis of the incentivised conditions but carried out for non-incentivised conditions.

This analysis (reported in Table 8) shows that changes in mortality for non-incentivised conditions were similar in the North West region and the rest of England from before the programme to the short term. However, from the short term to the long term, reductions in mortality were greater for non-incentivised conditions in the North West region than in the rest of England, with statistically significant differences in acute renal failure (–2.7 percentage points, 95% CI –4.4 to –1.1 percentage points) and alcoholic liver disease (–2.0 percentage points, 95% CI –3.7 to –0.3 percentage points).

We then examined the extent to which patients with the non-incentivised conditions were treated in the same specialties as patients with the incentivised conditions in order to investigate the possibility of spillover of QI activities into the non-incentivised conditions in the intervention hospitals. We hypothesised that, if positive spillover was to occur, it would most likely affect conditions treated in the same specialties as those exposed to the QI measures in the AQ programme. We therefore examined the extent to which the non-incentivised conditions were treated in the same specialties and by the same specialists as the incentivised conditions. We did that by analysing two fields in the HES data that uniquely identify the specialist team responsible for each patient’s care and under which specialty the lead specialist was employed.

The results (Tables 9 and 10) show that patients with incentivised and some of the non-incentivised conditions were indeed being treated in the same specialties and by the same specialist teams. Specifically, Table 9 shows that, at specialty level, 41–53% of patients with the incentivised conditions were treated in general medicine (approximately equivalent to general internal medicine in the USA). This is also the specialty in which 48% of patients with acute renal failure and 57% of patients with alcoholic liver disease were treated. These are the two conditions that showed the greater reductions in mortality from the short term to the long term in the North West region. There was less overlap between the specialties treating the other non-incentivised conditions, which were also those with non-significant reductions in mortality in the later period.

We also found overlap at the lower level of individual specialists’ teams that treated patients with incentivised and non-incentivised conditions. Table 10 shows that 55% of specialists treating at least one AMI patient had also treated at least one patient with acute renal failure. It can be seen that, although this level of overlap is not uncommon across all incentivised and non-incentivised conditions, the proportion of specialists with a substantial workload of both patients with incentivised and patients with non-incentivised conditions [defined here as treating at least 20% of patients with either condition (of the total of any two conditions)] remains high for the two non-incentivised conditions with statistically significantly larger mortality reductions in the North West region and is lower for patients with the other non-incentivised conditions.

In summary, these findings can be interpreted as modest evidence of a potential mechanism through which AQ could have affected the care of patients for some of the non-incentivised conditions (in particular acute renal failure and alcoholic liver disease). Further investigation into such potential positive spillovers is an important area for further research.

Adapting a tried-and-tested initiative

Looking around for targeted ways of spending growth money, which would enable them to measure impact, the programme leaders came across the HQID programme. This was chosen as the preferred solution after the programme leaders had investigated a range of such initiatives. It was hypothesised that opting for an existing programme would be preferable to developing a local solution from scratch, but this theory assumed that adapting this to local contexts was important in order to increase the chances of success.

Piloting

The intention was to use piloting to test feasibility and identify any factors likely to hinder implementation to increase the likelihood of success. First-wave organisations were to receive twice as much in set-up costs as second-wave trusts (£60,000 vs. £30,000). It was envisaged that second-wave hospitals would experience fewer teething troubles, hence the reduced allocation. It is not clear whether or not the theory at this stage assumed that no additional resources would be required beyond this or whether or not there existed sufficient ‘organisational slack’85,103 to enable participating organisations to implement AQ.

Investing in dedicated support and infrastructure

Programme leaders believed that a supporting infrastructure would be required, possibly as a result of their extensive interaction with Premier, which had provided similar support to the US initiatives. Programme leaders hypothesised that dedicated support and infrastructure from people with direct experience of QI initiatives from outside the NHS were needed and undertook a tendering process to appoint a partner organisation.

They also established an AQ team comprising experienced NHS managers, based in the SHA and committed to developing a governance structure to underpin AQ. The theory here seems to have been that it was important to create places and spaces to conduct work and dialogue about AQ issues outside the day-to-day workings of the providers participating in AQ. In addition, the pledge to ensure transparency, making the workings of these various groups publicly available, appeared to be aimed at enhancing legitimacy and increasing participant commitment. It seems clear that the programme leaders reasoned that merely putting incentives out there and expecting people to respond was unlikely to achieve the desired effect.

However, this dedicated support was soon also required at a local level, with all organisations eventually investing some of their AQ resources (in some cases, the set-up funds) in dedicated local staff to support AQ within their own organisations. There was little initial explicit programme theory supporting these type of appointments, as evidenced by the staged approach and variation in the nature of the roles across the organisations.

Using data to get attention

The programme leaders hypothesised that it was necessary to use quantitative performance data (as opposed to anecdote or gut feeling) to get attention from providers about current performance, prior to the commencement of AQ. In the absence of a focusing event104 such as a crisis, the intention was to highlight the gap between perceived and actual service delivery as a means of alerting providers and commissioners to the problem of suboptimal service delivery. This was a means of addressing the political challenge for which it was felt that quantitative data were less likely to be debated politically than other types of data or anecdote.

Using voluntarism and peer pressure to drive participation

The political challenges associated with gaining commitment to change were also clear in the implicit theory, where it was hypothesised that a voluntary approach would result in higher levels of commitment compared with mandatory participation. However, it was hoped that peer pressure would act to encourage organisations to join the programme, leading to participation of all 24 eligible organisations.

Pragmatically using evidence to get buy-in, programme leaders drew on peer-reviewed studies of the HQID and also used Premier’s own publicity material (which had not been subject to peer review) to publicise the benefits of the programme and get participating organisations and key individuals on board.

Comparing apples with apples

The emphasis on data continued into the programme itself, in which leaders emphasised the importance of providing clear data definitions and compliance with these definitions to ensure comparisons were on a like-with-like basis. This was intended to enable valid benchmarking but also to ensure commitment of staff to what they saw as a fair and legitimate process. In addition to a data dictionary, clear written guidance and training sessions delivered and supported by Premier, the programme also incorporated an assurance process which subjected participants’ data to scrutiny by the Audit Commission, a national body external to the NHS North West. This also contributed towards addressing the cultural challenge of giving a shared meaning to quality throughout the organisations involved.

Providing strategic leadership by sustaining senior-level commitment

Sign-up to participate in AQ involved Chief Executive commitment to the process, as this was hypothesised as being likely to increase success through demonstration of the importance of quality within organisations – one of the cultural challenges. In a context in which well-intentioned initiatives quickly become yesterday’s news, this senior leadership was seen by programme leaders as important in sustaining participation.

Institutionalising behaviour change

For chief executives of AQ organisations, the implicit hypothesis appeared to be that, over a relatively short period of time, AQ would become embedded as part of routine practice, and dedicated AQ staff within the hospitals could be moved on to pastures new. The idea of replacing one set of practices and beliefs with another and for these to become taken for granted, institutionalised, was central to their theory and appeared to be addressing some of the cultural challenges.

Using feedback to inform learning

The theory here was that feeding back data to participants would enable them to act on that feedback by identifying poorly performing areas, investigate reasons and trial solutions. Feeding back comparative data was intended to encourage participants to improve care, but there were no correct solutions to accompany these data. Instead the theory seems to have been that data would encourage learning within organisations, as well as learning from good practice elsewhere, thus addressing one element of the educational challenge identified in QI.

Using competition to drive performance

From the outset, the programme leaders pledged to report publicly the results of the programme. This reflects a commitment to transparency generally, as part of a process of securing legitimacy, but it was also intended to act as a spur to improvement. In terms of other mechanisms, in addition to using annual public reporting, the intention was to feed comparative data back to all participating hospitals on a regular basis, tapping into the competitive spirit of participants and underpinned by the tournament-style AQ design. A combination of normative and coercive pressure embodied in the quarterly and annual performance ranking reports was hypothesised to prompt action in a context in which poor performance on AQ measures would be collectively frowned upon. This approach to addressing some of the emotional challenges identified was particularly important for clinical professionals for whom peer comparison is a fundamental part of their own professional development.

Using collaboration to drive improvement

Programme leaders were, however, anxious to avoid competition driving out collaboration. Alongside the competitive elements of AQ, it was also seen as important to encourage the development of an AQ community, which was hypothesised to facilitate collaborative learning and maintain improvement momentum. Implicitly, they were attempting to develop a movement for improvement, thus addressing another emotional challenge.

Using money to spur improvement

Underpinning all financial incentive initiatives is a relatively simple theory of cause and effect. Linking money to desired outcomes is hypothesised to improve performance compared with the absence of such monetary rewards. Whether this is viewed as addressing a political or an emotional challenge will depend on the context and the individuals and professions involved, but it was clear that programme leaders believed that the financial incentives were important.

In year 1, AQ rewarded trusts performing in the top two quartiles of all participating organisations in the North West region. In subsequent years, the intention was to reward improvement, as well as absolute levels of performance. There were no financial penalties for trusts whose performance was below average. The design that rewarded relative performance in year 1 and subsequently improvement too, was seen as fair and was hypothesised to get greater buy-in relative to other potential designs (e.g. low performers lose money, all participants win money). AQ leaders also stressed that rewards would flow to clinical teams to reinvest in services, in an effort to focus money and attention where action was needed most, as this was hypothesised to act as a motivator for the staff working in those areas and might be viewed as addressing an emotional challenge by obtaining frontline buy-in.

Facilitating standardisation

Promotion of AQ measures was intended to standardise processes of care, with the intention of improving outcomes. According to institutional theory, the pursuit of legitimacy leads to organisations yielding to pressures within the field. The various components of the AQ programme (e.g. standard metrics, data feedback, building a community and so on) were hypothesised to lead to compliance with the policy and convergence within the organisational field. 105 The latter refers to the way organisations within the field increasingly come to resemble one another as part of the process of securing legitimacy. Pressures towards this institutional isomorphism can be mimetic (organisations copy other organisations), coercive (responses to external pressures such as governmental action) and normative (reflecting accepted norms of the professional community). The emphasis on standardisation within the AQ programme theory appeared to involve a combination of all three of these types of pressures.

Adapting a tried-and-tested initiative

The programme was adapted to suit the local context throughout the first 3 years.

Although the AQ measures used were based on relatively good evidence of efficacy,106 their application was also intended to take account of local contexts and national policies. For example, the HQID scheme measure for pneumonia patients used antibiotics within 4 hours of arrival. But, in recognition of a national NHS 4-hour wait target for accident and emergency department patients, AQ used 6 hours. Although admitting patients a few minutes before they breached the 4-hour ED target was not ideal, it was common practice at the time and, rather than trying to change this practice, the leaders reasoned that 6 hours would be long enough to enable hospital staff to cope with a last-minute admission to meet the 4-hour target and give 2 hours thereafter to administer antibiotics. Taking account of other constraints in the system to produce measures that would be regarded as reasonable and feasible was seen as important in securing initial commitment and sustained participation therefore.

Early on in the AQ process, concerns were raised about the inclusion of beta-blocker at arrival for AMI patients, which led to this being dropped from the measures from July 2009 (i.e. 9 months after the start of the programme). Similarly, the requirement to document a CURB-65 score (a clinical indicator used to assess severity of pneumonia) was introduced with effect from April 2011 with agreement from the AQ community. This was intended to help participating organisations systematise care and identify patients early on in their journey, to facilitate prompt delivery of care in accordance with AQ metrics.

It is important that adaptation should not be viewed solely as a technical process of refining measures. A key aspect of this evolution and development involved listening to AQ participants and responding to their concerns. Changes to AQ were made on the basis of dialogue with participants. Reflecting the pragmatic approach to implementation, the requirement to take blood cultures prior to starting antibiotics was dropped with effect from November 2012. Many providers had experienced difficulties with this measure, in part because of policies to reduce taking blood cultures generally, which were designed to reduce hospital-acquired infection and, more recently, to cut costs. The need to listen resulted in what at times was a protracted process, but this was important in securing agreement for any changes made. Once agreement was reached, however, participants had to abide by decisions taken, albeit with the proviso that as with all aspects of the programme, the potential and mechanisms existed to review this at a future date if concerns were raised.

Objections about the requirement to provide smoking cessation advice to patients who had not smoked for many months (the target requires this for anybody who has smoked within the last 12 months) were a recurring theme, with this issue creating tensions among staff, many of whom regarded counselling non-smokers as a waste of time. However, this measure was not altered.

With regard to CABG, there were indications that insufficient adaptation had taken place in transferring HQID to AQ. Providers had little difficulty with compliance, scoring highly from the first quarter onwards. Given that there were only four providers within the region that performed this procedure, distinguishing between providers and allocating rewards on the basis of tiny differences in performance was perceived as problematic by some participants (see Bhattacharyya et al. 47 for similar issues with hip and knee measures in HQID). Furthermore, clinicians viewed the CABG measures as insufficiently challenging, particularly in a context in which collecting and reporting data on cardiac surgery performance on a range of indicators was introduced long before the AQ programme arrived in England. In some cases this resulted in clinicians feeling distanced from the programme, as well as rewards being allocated for relatively little additional effort.

Over the final 2 years the team supporting AQ extended its application to other clinical areas – starting with stroke, mental health and others. This is not a part of this evaluation but showed the enthusiasm for this type of approach (identifying common measures, reporting on them, comparing data, providing financial reward, etc.) based on the perceived success of the original five conditions.

Piloting

Seven trusts participated in a first-wave pilot. Site selection was informed by geographical location (bacause the intention was to obtain geographical spread), willingness to participate, readiness of systems, good evidence of partnership between primary and secondary care and organisation type. Among pilot sites, participation in a dry run AQ exercise was perceived as a shock to the system. Various problems were identified as part of this process. Rather than expressing their gratitude for the piloting opportunity, participants were anxious and risked being overwhelmed by the enormity of the task they had taken on. They were thankful for the additional resources, but at this stage many had chosen to keep those in reserve until they had worked out where best to invest them. The fact that these resources were non-recurrent was also a source of worry, because in most cases these were viewed as likely to be required on a recurring basis. Within participating organisations, there was very little in the way of organisational slack to allow organisations to adapt successfully to pressures for adjustment created by AQ and to initiate required changes in a sustainable manner.

Piloting also had a somewhat negative impact upon those not involved in the pilot. In a context in which rewards were based on competition and relative performance, these concerns were exacerbated. From their perspective, providers that has not participated in the pilot would receive less money and start the race from a position far behind the sites that had, which seemed unfair to both. Furthermore, the fact that not all organisations that volunteered to participate in the pilot were chosen was not well received by some participants.

The pilots revealed a number of challenges which persisted beyond the pilot and remain in many sites at the time of writing. These chime with the challenges identified by Bate et al. 84 and we discuss these in more detail.

Investing in dedicated support and infrastructure

Using data to get attention

Given that there were no baseline measures of performance on AQ measures, programme leaders used data from a small-scale audit of a sample of local hospitals to highlight deficiencies in the current system. In addition to presenting these as percentages for each measure (i.e. ‘x%’ of patients received ‘y’), the data highlighted that patients did receive care which followed pathways corresponding to HQID process measures (e.g. for pneumonia patients – oxygenation assessment within 24 hours prior to or after hospital arrival, blood cultures performed in the emergency department prior to initial antibiotic received in hospital, initial antibiotic consistent with current recommendations, receipt of antibiotics within 4 hours of arrival, smoking cessation advice/counselling provided). However, the percentage of patients who received all of these interventions was disappointing, particularly in relation to pneumonia and heart failure.

One of the programme leaders explained that, at a meeting for potential evaluators prior to implementation, the results shocked hospital staff who on seeing the indicators initially had responded that ‘we’re already doing this’. It certainly helps for a problem to be countable and indicators of performance can help move issues up the agenda. Yet, at the same meeting, the statement by this programme leader that there were no ‘antibodies’ in the system, suggests an underestimation of the amount of effort required to move from getting attention to changing behaviour. Having grabbed people’s attention, according to the theory, it was important to feed data back in a timely fashion and on a regular basis to sustain involvement and attention, although this was not what happened in practice, as we discuss in Data reporting and verification.

Using voluntarism and peer pressure to drive participation

The AQ community, which evolved over time, was helpful in maintaining voluntary participation via a process that involved normative pressures, together with support mechanisms for staff directly involved. However, as the programme developed, it was pressure to perform rather than to participate that became the dominant feature. Participation at a local level was also important and appeared to be affected by clinical/managerial relationships.

The introduction of AQ into CQUIN and the punitive (i.e. payment is withheld as opposed to being a bonus) aspect that AQ performance entailed, was seen by some as having the potential to contribute to an increase of clinical engagement. Given that AQ was now seen as part of a trust’s funding contract and missed AQ measures would result in financial penalties (i.e. reduced budgets), there has been a move over time, by several trusts to resort to naming and shaming in order to present more personalised versions of AQ results and performance to clinical teams. The aim was to appeal to professional competitiveness and engage a wider array of clinicians and it was reported by some leads as resulting in improvements.

Comparing apples with apples

Compliance with the guidance provided in the detailed data dictionary was intended to ensure standardisation within and between providers. The dictionary (or ‘the Bible’ as it was often referred to) was seen as very helpful and informative, and participants described having learned much during the process of working with the dictionary. In the early stages of implementation, despite training to supplement the process and e-mail advice and support from Premier and SHA AQ team staff, there were different interpretations of the rules and guidance, which led some staff to panic or feel disheartened. At a training session we observed in April 2009, for example, one attendee was horrified to learn that she had been incorrectly interpreting the guidance.

Another in an interview described confusing all the discharges for a particular month with all the admissions for that month at the start of the process because of a lack of understanding about what was required. Other examples included not collecting PROMs data for all eligible patients and general uncertainty around reasons for exclusion from the measures. The learning process around AQ data collection and submission was a lengthy process, although over time as providers built up a stock of knowledge around this it became less burdensome.

The result was that difficulties with the interpretation of AQ measurement definitions were, to a considerable degree, overcome.

The competitive nature of AQ may have contributed to some distrust in the early stages of the programme, with a small number of low performers suggesting that performance was more a reflection of inconsistency in data interpretation between trusts (with high performers being less stringent in their application of AQ rules) than of real differences in service delivery. Staff from the Audit Commission, who were responsible for assuring the data, confirmed that in some trusts the guidance had been interpreted too narrowly, leading these organisations to under-report their performance, although following feedback this has been largely rectified.

The involvement of the Audit Commission, which undertook a number of data assurance processes, helped reassure participants over time that all were playing by the rules of AQ and comparisons were valid. Despite welcoming the input, participants also complained that assurance processes were burdensome, but, as these evolved, a differential approach with greater focus on providers where problems had been identified was implemented. Many of the issues identified related to participants applying rules too strictly, as opposed to being too lax, which helped reassure doubters.

The process of comparing and assuring data highlighted the issue of data completeness. It would be possible for providers to declare success in meeting all measures for 100% of patients, if they failed to report on all eligible patients. Rules were developed, consulted on and implemented to ensure that only providers meeting minimum standards with regard to completeness and accuracy were eligible to be considered for rewards. These processes appeared to be important in securing and sustaining buy-in, with discussions increasingly being focused on strategies and tactics for improvement and sharing learning. Over time, distrust around data began to disappear from the AQ community discourse. Overcoming the cultural challenge by systematic ongoing attempts to align definitions of quality was one of the key aspects of the AQ programme, and one which led the approach to be applied to other areas.

Providing strategic leadership by sustaining senior-level commitment

In terms of ‘structuring, planning and co-ordinating’84 AQ, in all providers accountability structures were established identifying a member of the executive team with lead responsibility for AQ. Lead directors spanned a range of disciplines including finance, information, nursing and medicine. In many cases the choice of executive lead appeared to reflect the way in which AQ had been conceptualised by executive teams (e.g. an informatics project, a quality initiative), in terms of the nature of the programme and the relevant skills set required, although this was not always the case.

There were instances of AQ executive leadership responsibility being reassigned in a pragmatic way (who might be available to fill a gap, as opposed to who would be suited to the nature of the task) following the departure of an executive lead. In some cases, executive leadership was seen by those charged with delivering AQ on the ground as lacking, insufficiently engaged and/or integrated, with this adding to pressures on other members of staff involved in AQ. A small number of people reported that, despite having a designated executive lead and formal accountability structures, senior staff were too busy with other priorities to actively engage with AQ.

I was sat there as an information manager . . . And [other trust CEO] was sat there representing [other trust] . . . well which one of these is going to succeed? The one that’s got the Chief Executive sat there or the one that’s just sent the information manager?

Information manager, ID219, T15

Now there is a big push to say right let’s hit these targets . . . be in the top 25% . . . from the very top from the Chief Executive all the way down now there is a massive push to do it.

AQ lead, ID55, T2

Without the executives saying, yes we support you 100%, lots of projects die. You could put a lot of effort into it, but until somebody says, yeah okay, we’ll give you the money, it will never happen. So we very early on established that we had executive buy-in, certainly from the medical director that was interested in quality. We had a quick chat with the Chief Executive so yes, we really want to make sure that we get this going right . . . there was an executive direct lead who was interested, wanted to know what was going on but didn’t do any real hands on work.

AQ clinical information officer, ID123, T14

However, leadership meant different things to different people. In some sites this involved having a designated executive lead who responded positively to requests for resources, rather than to a nominated person at executive level driving forward QI initiatives. In some cases executive leadership was seen as lacking, insufficiently engaged and/or integrated, with this adding to pressures on other members of staff involved in AQ.

Get management and exec engagement straight away or as early on as possible and then it would work in the trust . . . [here] . . . suddenly panic, it missed out a load of people, it just suddenly went to somebody who had time to implement it and then onto somebody else who had time to do the data entry.

Clinical information analyst, ID26, T4

The Chief Exec doesn’t really get involved or the Trust Board doesn’t get involved.

Clinical audit co-ordinator, ID222, T12

Although the lack of formal structures for promoting executive leadership was reported as detracting from efforts to implement AQ, the mere presence of such structures was no guarantee of success. The extent to which executive leadership was sustained over time varied between providers, with executive leads in some sites only becoming actively involved when performance was poor. At one point, when a study was published which showed no impact, based on a large US data set,9 one of the Chief Executives who had initially been very vocal in his support for AQ, raised concerns at the AQ Reference Board meeting. The fact that this provider had not been performing as well as expected may have been a contributing factor here. This highlights that evidence can be used as part of a political process, a point that the rather underspecified nature of the initial theory tends to overlook. However, it also highlights how the normative pressures of AQ ensured that participants remained within the programme, despite any reservations they might have about doing so. It is clear that, although there are structural challenges in improving quality, addressing them is likely to be necessary but not sufficient for improvement.

Institutionalising behaviour change

Although not specified explicitly in the initial programme theory, in relation to institutional change, we might frame AQ as attempting to move to a new dominant logic. Institutional theory places great emphasis on legitimacy and shared norms and AQ can be seen as attempting to institutionalise AQ pathways, which are seen as good practice. This is the political challenge described by Bate et al. : ‘Negotiating the politics of change associated with implanting and sustaining the improvement process, including securing stakeholder buy-in and engagement, dealing with conflict and opposition, building change relationships, and agreeing and committing to a common agenda for improvement’. 84

At the same time, what counts as good practice and, more generally, knowledge are highly influenced by political and cultural factors (in other words, cognitive frames are hugely important, as we discuss in Chapter 6, Impact on mortality in the longer term). This makes knowledge brokering by AQ leads and/or other staff difficult and is a key cultural challenge. Part of the problem related to power differentials109 between core staff and clinicians. For example, many coding and administrative staff felt that they could not challenge clinicians. However, in common with Currie and White,110 we found that the challenge of knowledge brokering beyond one’s professional affiliation and hierarchical status was not insurmountable. This was related in part to professional roles, with some non-medical clinical staff less inhibited than their administrative counterparts. In some cases, specialist nurses were based relatively close to other groups of staff with whom they interacted on a regular basis. This facilitated the building of relationships, which enabled discussion and some modification of views. Even where this was not the case, the role undertaken by specialist nurses meant that they interacted regularly with staff involved in care delivery. In some sites medically qualified clinical champions acted to reinforce the AQ message in a way that allowed the nurses and/or AQ leads to punch above their hierarchical weight in terms of their knowledge brokerage role.

Clinical–managerial partnering was a key element of brokering. There were reports of clinician resistance to AQ from some quarters in all trusts. Often this conflict concerned the process of data recording, as opposed to the content of AQ quality metrics.

The issue of junior doctors failing to adhere to AQ treatment metrics was a recurring theme. The rotation of junior doctors across different sites at regular intervals as part of their training, in combination with the time lag in AQ results, was seen as contributing to the persistence of resistance to change among this group of staff. In all trusts there were reports of clinicians failing to record activity; however, although not easily resolved, some participants reported that improvements on this had been made on this over time. The most frequent obstacles in sustained clinical–managerial partnerships were reported to be the differences of perceived priorities between the two groups and the assumption that managers lack knowledge and direct experience of frontline services.

The actions of core AQ staff and, where relevant, medical champions suggested that knowledge brokering is as much a group process as an individual one. However, even where groups worked well together to perform this function, getting clinicians to change behaviour was often done by following them around, rather than trusting them to comply with AQ requirements. Junior (and often more senior) doctors failed to act in accordance with AQ requirements and at times complained that AQ was merely an audit that got in the way of patient care. More generally, workload pressures contributed to staff forgetting to take action. The absence of data-recording technologies, to prompt action and embed behaviour change, compounded the problem. Instead of creating institutionalised change in ways of doing things, often the old ways persisted, with a relatively small number of dedicated individuals participating in an ongoing process of educating, prompting and cajoling, as well as rolling up their sleeves and plugging gaps where necessary. It is therefore unclear whether or not this cultural challenge was addressed in a sustainable way.

Advancing Quality requires the spanning of boundaries between levels, professional groups and divisions within the organisation as well as internally and externally. In theory at least, the designated AQ programme lead in each trust could perform such a role. The extent to which leads acted as boundary spanners using influence and negotiation to achieve AQ goals varied between providers. For example, in one trust much of the time of the AQ lead was spent closely monitoring data collection and entry, ensuring that these activities happened and that they did so in a timely fashion. Covering for data collection and data entry when the staff members responsible (one for each clinical area) were on sick leave or overloaded with other duties was also part of their role. Alongside this, the AQ lead in this provider was directly accountable for performance and was often pressured (in one-to-one meetings or through electronic correspondence) by trust executives to improve AQ results and secure more substantial financial rewards for the organisation. The limited clinical engagement here contributed to a situation in which much of the implementation of AQ appeared to be sustained by four data collectors and an AQ lead.

This contrasts with another provider where the AQ lead gradually moved to a supervisory role, acting as a point of reference for clinical and executive staff but not being held directly and solely accountable for AQ results. Performance was discussed openly in bimonthly AQ meetings, with regular executive and clinical attendance, where each clinical department presented their AQ work and expected performance. Data collection and data entry were undertaken by several clinical staff members in each area allowing sufficient cover and sustained performance when staff members are absent or carrying a heavy clinical workload. The AQ lead in this organisation provided input as the person who has the most in-depth knowledge of the AQ programme (data requirements, financial incentive changes and future developments) as well as a comprehensive overview of the implementation process in the organisation (past and present). Although even in this case, it took a lot of time to get to this position.

In some cases challenges surfaced and were conceptualised in terms of a presenting problem,111 which turned out to be a symptom of a deeper issue. For example, in one trust where we observed meetings over a period of months, junior doctors were failing to record data required for AQ. Various steps were taken to improve recording including amending the data-collection tool (responding to a perceived physical and technological challenge), raising awareness among junior doctors (responding to a perceived educational challenge) and trying to build a shared understanding and commitment to AQ (responding to a perceived cultural challenge). However, despite attempts to address perceived challenges, and, although these staff were made aware of AQ and its requirements, they were largely resistant to it, which suggests that the challenge may also be about the politics of change surrounding AQ. Although it may be necessary to raise awareness and increase understanding and knowledge, where other challenges persist, such actions may not be sufficient to obtain engagement and improve AQ performance.

Nurses were also failing to take action on AQ measures. These staff were extremely unlikely to be unaware of the need to advise patients on smoking cessation, but a combination of work pressures and forgetfulness meant that many breaches occurred with regard to this measure. For many nurses, despite education on the importance of delivering this advice, recording this activity was not seen as important, which appeared to be indicative of a lack of a shared understanding and commitment to AQ. The fact that nurses needed constant reminders and nagging by AQ specialist nurses and other AQ staff is indicative of attitudes among many clinicians that AQ was not part of their core responsibilities.

A failure to record may sound like a minor transgression, on the grounds that delivering care is what matters. However, the act of recording involves interaction with technologies (electronic and paper medical records) in a way that helps change to become embedded and self-reproducing. Recording facilitates the institutionalisation of new ways of working so that over time (and this may be a relatively short time, e.g. see Checkland et al. 112) they become taken for granted. Rather than seeing data-recording tools and templates as helping to reinforce the systematic delivery of high-quality care, however, a frequent complaint from clinicians was that AQ was an audit and, therefore, a paper exercise of little value.

Bate et al. 84 suggest that an important part of the process of overcoming political challenges and institutionalising change is strong and close partnerships with relevant external stakeholders. In the early phase of interviews there were occasional reports of PCT commissioners helping to support AQ by providing financial support for activities relating to AQ, but this was rare. We also found reports of increased partnership working with PCT providers in the delivery of services (prior to Transforming Community Services) related to AQ metrics, particularly around smoking cessation.

Interviews undertaken with PCT commissioners in Autumn 2009 indicated that many had little involvement in and/or not much detailed knowledge of AQ. These interviews were undertaken following the receipt of reports within the PCT (normally by the Chief Executive) on provider AQ performance, but many interviewees had not seen these reports. Among those who had seen them, there were mixed views about their content, which in part appear to reflect the capacity of commissioners to understand and engage with AQ. These reports presented the first formal results on performance to PCTs and commissioners reported little involvement in the process up to this point. In interviews with commissioners during the second half of 2010, most participants reported that they had little long-term involvement with the programme in the implementation process, as they mostly considered it an acute trust initiative. Nevertheless, with hindsight commissioners accepted that future implementation of AQ in regions outside the North West region would require a more hands-on approach from a commissioner’s perspective in learning more about the internal processes and microcultures of the acute trusts to allow more effective monitoring and intervention.

The incorporation of AQ into the CQUIN payment framework might have been expected to encourage commissioners to play an active part in the monitoring of provider performance as part of regular meetings at which trust performance on CQUIN goals was discussed. However, there was little evidence that commissioners were more engaged following the introduction of CQUIN. Some participants suggested that introducing AQ into CQUIN in April 2010 did not allow enough time to assess the impact of AQ and learn lessons from the implementation process. Furthermore, although in 2009 some commissioning staff did start to attend AQ events, the announcement in July 2010 of large-scale NHS reforms to include the abolition of PCTs meant that PCT staff interest and participation was severely reduced.

In the final round of commissioner interviews with CCG leads, the high costs of AQ was mentioned repeatedly, with most interviewees suggesting that they were uncertain about their continued support for the programme. Shortly after these interviews, a small number of commissioners withdrew from AQ.

Using feedback to inform learning

Using competition to drive performance

Participants were acutely aware of the competitive nature of AQ. Although it was many months before initial first-quarter comparative data were made available, the commitment to public reporting and the tournament nature of the incentive system encouraged participants to compare their progress with that of other organisations. Striving to be near (or at) the top of the league table motivated people but it also placed them under pressure. Comparative data were presented at meetings where all AQ leads were present and the excitement, delight and disappointment were often palpable.

One participant described feeling ‘relieved but not complacent’ on seeing the figures because ‘it wouldn’t take much for you to slip just a tiny fraction and lose the incentive payment’. So, even among high performers, the results did nothing to diminish the feeling of being under constant pressure.

Advancing Quality programme leads used data sharing within their hospitals to try to use competition to motivate local clinicians, with varying degrees of success. As hypothesised, normative and coercive pressures prompted action in response to performance deficits.

Among those in each organisation who were responsible for delivering some aspect of the AQ programme on a day-to-day basis, AQ appeared to have its own momentum. The constant round of monthly data collection and reporting was a stressor. For AQ programme leads, participation in the process meant taking an emotional roller-coaster ride as they grappled with the highs and lows of the programme. None of the AQ leads had envisaged that the process would be so difficult and time-consuming, and poor performance on AQ metrics could leave participants feeling very low. Early on in the process, we interviewed one AQ programme lead whose office was full of mugs and pens that he was ‘too busy’ to distribute, and the practice of adding AQ to existing workload may explain such attitudes. In such cases, staff were replaced by others with a more positive orientation to AQ.

The competitive nature of AQ also resulted in AQ leads coming under increased pressure, as opposed to executive staff taking action, which translated into behaviour change among frontline clinicians. The use of evidence, combined with public reporting, was used to attempt to engage clinicians or, in extreme cases, to shame them. As one exasperated AQ lead recounted ‘And I’ve used phrases like, “Do you want to be viewed as the worst evidenced surgeons in this region?”’. However, the challenge of getting clinicians to follow evidence may also have been a presenting problem not solely amenable to resolution by such pragmatic approaches.

Achieving culture change and whole systems working takes time, yet the competitive nature of AQ, with results to be publicly reported, meant that short-term improvements were needed. This is one reason why AQ involved going around people rather than relying on persuading them to change their practice.

Using collaboration to drive improvement

Using money to spur improvement

Economic theory would suggest that providers need to be compensated for the costs of investing in service changes related to P4P,33 and at some sites it was necessary to make a business case for additional staff to demonstrate that this would be at least cost neutral. However, in many cases, investments were made without such information, largely on the grounds that poor performance was deemed unacceptable by participating organisations in a context in which the normative pressures surrounding AQ performance were strong.

The initial theory was based on a commitment to rewarding clinical teams, but ad hoc investment in coding, audit, information and clinicians (specialist nurses) meant that there was often no clear link between rewards and teams as envisaged in the theory. However, this did not appear to detract from the commitment of core AQ staff and a combination of evolving routines and practices together with feedback of relative performance ensured that they maintained their focus.

Facilitating standardisation

Although the implicit theory appears to assume compliance and convergence, with organisations coming to resemble each other in terms of standardised approaches to care, some organisational characteristics may be more open to change than others. Ashworth et al. 116 hypothesise that the impact of these pressures on organisational structures and processes is stronger than on strategy and culture (although their results are not entirely in accordance with this hypothesis). This resonates with the evidence on the difficulties of securing culture change in health organisations117 and with the AQ experience.

As the programme evolved, structures and processes were developed to plug gaps in care pathways and improve care coordination. The AQ measures represented requirements which, in the sense that they were non-negotiable (on a day-to-day basis), were coercive in nature. We observed two types of standardisation:

Pre-trends tests

We tested whether or not the risk-adjusted mortality rates in the North West region had a different trend to those in the rest of England prior to the introduction of the programme using a pre-trends test.

Using data from before the introduction of the programme, we estimated the following regression model for each condition:

in which t represents the quarter since the start of the data period, β is an estimate of the quarterly trend in the rest of England and ρ is the difference between the quarterly trend in the North West region of England and the quarterly trend in the rest of England.

We were able to accept the null hypothesis of equal pre-trends for each condition. The estimated values for ρ were as follows:

• AMI, –0.34 (95% CI –0.98 to 0.29)

• heart failure, 0.19 (95% CI –0.52 to 0.90)

• pneumonia, –0.23 (95% CI –0.81 to 0.36)

• non-incentivised conditions, –0.66 (95% CI –1.40 to 0.09).

Analysis using total (in-hospital and out-of-hospital) mortality rates

Effects of including baseline mortality

Analysis without hospitals with any financial incentives for the conditions incentivised by Advancing Quality or the non-incentivised conditions included in this paper

Effects of using 90-day mortality