Developing and evaluating packages to support implementation of quality indicators in general practice: the ASPIRE research programme, including two cluster RCTs

Stage 1: initial screening of candidate recommendations

We identified candidate recommendations and indicators from 147 NICE clinical guidelines (published December 2002–June 2012), 19 NICE quality standards (June 2010–June 2012) and 95 QOF clinical indicators (extracted June 2012). One researcher (BR) screened titles and summaries of NICE guidelines and quality standards for relevance to primary care; 20 were excluded as they were exclusively related to secondary care. We excluded a further 20 that had been superseded by a more recent update. We excluded four QOF indicators mainly related to secondary care. Together, these sources yielded a total of 2365 candidate recommendations.

Two clinical researchers (BR and RF) then independently screened candidate recommendations, discarding those judged irrelevant to primary care or not measurable using routine data. We grouped clearly linked sets of recommendations to form ‘composite’ recommendations (e.g. the nine recommended processes of care for patients with type 2 diabetes). 23 We resolved disagreements through discussion. The final ‘longlist’ of 102 candidate recommendations comprised 56 single and 46 composite recommendations (additional files published with Rushforth et al. 1).

Stage 2: online shortlisting by consensus panel

We used a modified RAND consensus process. 14 We convened an 11-member multidisciplinary consensus panel, which comprised five GPs (including two with responsibilities for commissioning services), a practice nurse, a practice manager, a consultant clinical advisor from NICE, a health informatics specialist and two patient representatives. We deliberately weighted the panel towards professionals who would typically act on clinical practice recommendations, recognising that a number of judgements required an in-depth, tacit understanding of the day-to-day realities of clinical practice. The panel was limited to 11 members as there are only marginal gains in reliability beyond this number. 14

Each panellist independently rated all 102 recommendations via an online survey according to three criteria: burden of illness (e.g. prevalence, severity, costs), potential for significant patient benefit (e.g. longevity, quality of life, safety of care) and scope for improvement on current levels of adherence (e.g. from perceived current low levels or large variations). All ratings were completed on a 9-point Likert scale (where 1 is low and 9 is high according to their perceptions of current practice) with ‘don’t know’ options available.

Panel ratings of recommendations were generally high for patient burden [mean ‘median’ score of 7.6; standard deviation (SD) 0.68] and potential for patient benefit (7.8; SD 0.81), with lower scores for scope for improvement (5.00; SD 0.88). We excluded 18 recommendations from further review at this stage because they scored ≤ 4 on scope for improvement (indicating that the panel perceived adherence to these recommendations to be relatively good).

A second online survey contained the top 62 recommendations (31 single and 31 composites) based on the highest aggregate rankings. Panellists rated shortlisted recommendations using three further criteria: the feasibility of measuring adherence (e.g. from routinely collected clinical data); the extent to which following a recommendation is directly within the control of practice teams or individual professionals; and the likelihood of cost savings without patient harm (all assessed on 1–9 Likert scales). Thresholds for disagreement were defined in advance as at least three panellists scoring a recommendation 1–3 on a particular criterion, and at least three scoring it 7–9. The panel disagreed on a total of 22 (11.8%) ratings; 20 disagreements concerned the feasibility of measuring adherence and two concerned the extent to which following a recommendation was directly within the control of practice teams or professionals.

Stage 3: face-to-face consensus panel meeting

The panel met for a facilitated, structured discussion, led by an experienced researcher. We presented summaries of the evidence for each recommendation, clarified any aspects of recommendations and discussed reasons for low or high rankings with a view to reaching, but not forcing, consensus. Panellists independently re-rated each recommendation immediately after discussing each recommendation. Following the panel meeting, there were disagreements for 12 (6.4%) ratings. Across the 62 shortlisted recommendations, the mean ‘median’ ratings were 6.8 (SD 1.57) for the feasibility of measuring adherence, 7.2 (SD 0.76) for the extent to which following a recommendation is directly within the control of individual practice teams or professionals and 7.3 (SD 0.73) for the likelihood of cost savings without patient harm. This process produced a ranked list of 50 recommendations for which consensus was achieved.

Stage 4: informal sense-checking

We added a sense-checking exercise to guide the final selection of recommendations, aiming for a list of around 20 that could be taken forward to the next stage of the programme. During the consensus and ranking exercise, we were struck by some unexpected anomalous rankings that appeared to lack face validity when considered against our rating criteria. For example, we doubted the feasibility of using routinely available data to measure adherence on recommended secondary prevention following myocardial infarction (MI), which had made the top 20 list after the panel rating. Current and past activity levels and preferences should be considered when advising on physical activity. An appropriately qualified health professional can help tailor advice on the benefits of exercise. 24 We also wanted to ensure that identified recommendations would be consistent with local priorities although their measurement was unlikely to face ceiling effects given known national and local initiatives.

We therefore identified a convenience sample of four GP commissioning leads and six academic GPs that we had existing working relationships with and who had practical experience of measuring primary care outcomes. We e-mailed and asked them to review the full ranked list of recommendations from the consensus panel. We invited them to select between 5 and 10 recommendations that they considered would best meet our aims and highlight any that they considered problematic to target. We then collated their selections and written comments. During this process, we amalgamated two similar recommendations (concerning initiation of insulin in type 2 diabetes) and replaced one recommendation concerning prescribing non-steroid anti-inflammatory drugs (NSAIDs) with a composite recommendation on risky prescribing. 25 Comments from this sense-checking exercise centred on concerns regarding perceived likelihoods of ceiling effects, difficulties in measurement or recommendations being outside the immediate control of the primary care team. This process produced 18 recommendations (Box 1), 11 of which had been ranked in the top 20 by our panel. These mainly covered chronic disease management and cardiovascular disease.

Stage 5: field testing of indicators

We assessed the extent to which it was feasible to operationalise the QIs in a sample of general practices [sampling described under work package (WP) 2]. Search algorithms were generated by a clinical researcher (BR) and a primary care data analyst. We applied and iteratively refined the algorithms with input from two external GP advisors. Appendix 1, Box 2, illustrates two examples. The full set of SystmOne (The Phoenix Partnership, Leeds, UK) searches for each of the 18 recommendations is available in Rushforth et al. 1

Study design and setting

We conducted a cross-sectional analysis of achievement against QIs using routinely collected, electronic data from randomly sampled general practices in West Yorkshire.

Participants

Of 334 general practices in West Yorkshire at the time of the study, 272 used the SystmOne clinical information system which permitted centralised data extraction. We sampled randomly from this group and stratified them by the then configured five NHS primary care trusts. We assumed a 30% decline rate based on earlier work44 and initially asked 78 practices to ‘opt in’ to the sharing of anonymised data to achieve a sample of 60 (see Sample size). After receiving several declines and a small number of acceptances, we approached and sampled an additional 36 practices, making 114 in total. At the same time, the local NHS Research Ethics Committee granted permission to change to ‘opt-out’ recruitment to reduce selection bias by facilitating general practices’ agreement to share anonymised patient data.

Variables

We selected seven of the QIs developed in WP1 based on likely scope for improvement and amenability to change (see table 1 of Willis et al. 33). Four indicators focused on processes of care (e.g. prescribing) and three focused on clinical outcomes (achievement of treatment goals).

We examined patient-level demographic variables (age, gender and ethnicity) and comorbidities as recorded for QOF disease registers. Practice-level variables comprised the number of GP partners (a proxy for practice size), the number of salaried GPs and training status. We used practice-level Index of Multiple Deprivation (IMD) scores. We used overall QOF clinical domain (2012–13) achievement as a proxy measure for overall quality of care. Patient data were remotely extracted and anonymised before transfer. We obtained data from National General Practice Profiles45 for two further practice-level variables: patient satisfaction (the proportion recommending the practice to others) and accessibility (the proportion reporting being able to speak with a GP or nurse within 48 hours of approach).

Data analysis

For each QI, we assessed the proportion of cases with documented receipt of appropriate care or attainment of treatment goals. Denominators were eligible patients, identified by diagnostic codes and prescribing records. Numerators were patients with evidence of a clinical intervention offered or received, or meeting defined treatment goals. In assessing the impact of practice and patient characteristics, we initially calculated unadjusted odds ratios (ORs) before adjusting for other variables associated with outcomes.

Data for almost all patients were complete. Data on age were missing from a small proportion (< 1%) of patients and excluded from analysis.

Sample size

Effect size calculations informed a recruitment target of 60 practices. With seven covariates, and a large effect size (defined by a difference of at least 0.8 SDs),46 60 practices would provide 94% power.

Participants

Eighty-nine practices (78.1% of those approached) shared patient data. Practices declining participation only differed from participating practices in having a smaller mean number of GPs (5 vs. 3.6; p = 0.05). The total number of patients for each indicator denominator ranged from 4773 (anticoagulation in AF) to 77,587 [blood pressure (BP) control]. Willis et al. 33 summarises patient demography (see table 2 of Willis et al. 33). Practice size was indicated by the number of practice partners (mean 3.7, SD 2.3) and salaried GPs (mean 1.3, SD 1.8). Mean practice-aggregated IMD score was 31.2 (SD 11.9, approximating to the highest quarter of deprivation). Mean QOF 2012–13 performance for clinical domains across practices was 637.4 (SD 27.6; approximating to the national mean47) and 20.2% were training practices.

Achievement of indicators

Median practice achievement of the indicators ranged from 43.2% (range 20.8–66.2%) for diabetes outcomes to 74.2% (range 50.7–100%) for BP control in chronic kidney disease (CKD) (see tables 3 and 4 in Willis et al. ). 33 Median achievement of the risky prescribing indicator was 8.7%, although lower scores were indicative of fewer instances of risky prescribing and were, therefore, desirable. Considerable between-practice variation in achievement existed for all indicators: the difference between the highest and the lowest achievers was 26.3% for risky prescribing and 100% for anticoagulation in AF and BP control in CKD. 33

Associations with achievement

The range of ORs associated with the random effects for practices demonstrate that the likelihood of achieving a specific indicator varied substantially as a consequence of the practice at which a patient was registered (see tables 3 and 4 in Willis et al. ). 33 These ORs were typically of a much greater magnitude than those for other variables, demonstrating strong practice effects. For process indicators, the impact of the practice attended was most pronounced for risky prescribing, with a sevenfold difference between the lowest and highest performing practices (OR range 0.40–3.51). Practice effects were least apparent for secondary prevention of MI (OR range 0.70–1.42). There were also sizeable practice effects for outcome indicators. For the achievement of target BP values in hypertension there was a greater than 10-fold difference between the highest and lowest performing practices (OR range 0.50–5.24) and a fourfold difference for diabetes control (OR range 0.51–2.05). Practice effects were less marked for the achievement of BP targets in CKD (OR range 0.54–1.60). Across the seven indicators, statistically significant associations were identified more frequently with patient than with practice characteristics (see tables 3 and 4 in Willis et al. ). 33 The amount of variance explained by these variables, however, was relatively low; practice characteristics explained less than 8% of variance across all seven models. Variance due to patient ethnicity typically explained a small amount of variance in achievement (< 10% of the variation due to practices).

Process indicators

Outcome indicators

Design and setting

We conducted semistructured interviews with primary care professionals in West Yorkshire, UK.

Indicator selection

We selected eight indicators from WP1 and WP2 for this interview study (one of which, advising on smoking cessation, we did not report under WP2 given doubts about its validity). Here, we focus on the four indicators subsequently targeted by our implementation package (Table 1):

1. risky prescribing, especially involving NSAIDs55

2. treatment targets in type 2 diabetes56

3. BP targets in treated hypertension57

4. anticoagulation in AF. 58

Sample

To gain a range of perspectives within practice teams, we aimed for a total sample comprising 30 GPs, 15 practice nurses and 15 practice managers. We invited staff from the 89 practices that shared data in WP2 to participate. Recruitment ran from September 2013 until June 2014.

Interview procedure

Each interview covered two indicators out of the eight under consideration. The topic guide drew on the TDF and was refined following piloting with three academic GPs54 (see table 10 of Lawton et al. 52).

Data analysis

Interviews were audio-recorded and transcribed verbatim. We used NVivo 10 (QSR International, Warrington, UK) software to facilitate analysis. Initial coding was completed by the same three researchers who conducted the interviews (GL, JH and EI).

Data saturation was considered at the indicator level. Interviewers held regular debriefing discussions after interviews and reached consensus on new or redundant content across the indicators. Redundancy typically coincided with the estimated 15 interviews per indicator, ranging approximately between 12 and 14 interviews. However, for each indicator, we also sought further interviews with any less well-represented participant group to maximise the diversity of data.

Our framework analysis comprised familiarisation, identification of a framework, indexing, charting and mapping, and interpretation. 59 The coding framework was developed through an iterative process that incorporated the study aims, the TDF and the detailed reading of interview transcripts. The coding framework included code definitions to ensure consistency.

As part of familiarisation, researchers read through each transcript before coding and wrote a brief summary outlining key themes and findings. Pieces of text were coded according to the iterative coding framework. At this stage, TDF determinants (primary codes) were coded at a broad level. Common additional codes and categories were organised into secondary codes for TDF determinants. When additional codes and categories were added to the framework, we revisited coded transcripts and applied the revised coding framework. Early-stage, face-to-face meetings ensured that there was agreement in coding. To promote reliability, six transcripts were coded independently by each researcher and disagreements were resolved through discussion.

We completed two stages of analysis. We first assessed determinants for individual indicators by examining the data coded against the TDF domains. To prioritise TDF determinants for each indicator, we focused on the key thematic content (e.g. the extent to which the TDF determinant was discussed across participant groups) and barriers and enablers. We then identified meta-themes across multiple indicators, including the additional codes and categories generated to produce the analytical framework.

Theoretical domains

Professional role and identity and environmental context and resources featured prominently across all indicators, whereas the importance of other domains (e.g. beliefs about consequences, social influences and knowledge) varied across indicators. Table 4 from Lawton et al. 52 describes all TDF domain content and specific barriers and enablers for each indicator and longer narrative accounts are also available in the published paper. We next focus on the meta-themes that emerged when synthesising data from multiple indicators.

Meta-themes spanning multiple indicators

We identified five meta-themes that potentially represent general influences on evidence-based practice: (1) perceived nature of the job and norms of practice, (2) internal and external sources of support, (3) communication pathways and interaction, (4) meeting the needs of patients and (5) perceptions of indicators. Tables 5–9 in Lawton et al. 52 illustrate interview excerpts.

Stage 1: selecting delivery mechanisms

We selected delivery mechanisms typically available within primary care and of known effectiveness: audit and feedback (A&F),67 educational outreach68 and computerised prompts and reminders. 69,70 We aimed to embed features associated with higher effectiveness (e.g. repeated feedback of audit data, requiring prescribers to select a reason for over-riding a computerised prompt). 67,69

Stage 2: identifying candidate behaviour change techniques

Team members (LG, RL, RM and RF) independently mapped BCTs (e.g. ‘feedback on behaviour’ or ‘action-planning’)66 to theoretical domains54 and resolved discrepancies by discussion. We thereby generated an inclusive list of ‘candidate’ change techniques.

Stage 3: prioritising determinants of behaviour

We convened a series of multidisciplinary panel meetings, one for each indicator. We invited 5–10 stakeholders with a range of perspectives and skills, including GPs, practice nurses, pharmacists, practice managers, quality improvement specialists and service commissioners. We presented them with emerging analyses from our earlier interviews (WP3a) with primary care professionals (frequency data and illustrative quotes for each determinant of achievement). 52 After reviewing the range of determinants, stakeholders contextualised our findings and suggested additional professional or organisational determinants. The panel considered the feasibility and acceptability of candidate BCTs and intervention delivery mechanisms and their potential enhanced features, taking primary care context and resources into account. We took field notes of discussions. We convened our patient and public involvement (PPI) panel in parallel and followed similar methods. The research team communicated key messages from one panel to another and reviewed suggestions from both groups. Following the stakeholder panels, we further analysed interview findings to identify the most prominent determinants and high-level themes. We grouped determinants into four categories: core, prominent, less evident and not identified. Determinants considered core to all four QIs (i.e. consistently raised regardless of QI) included ‘social and professional role’, and ‘environmental context and resources.’ Those considered prominent (i.e. determinants that varied in importance) included ‘beliefs about consequences’, ‘social influences’, ‘knowledge’ and ‘memory, attention and decision processes.’ ‘Skills’, ‘beliefs about capabilities’ and ‘motivation and goals’ were less evident, whereas ‘emotion’ and ‘behavioural regulation’ were not identified.

Stage 4: designing intervention content

We drew on stages 1–3 to create a prototype outline for each delivery mechanism (feedback report, educational outreach session, and prompts and reminders). We did not develop computerised prompts for diabetes or BP control because they were already widely used to support QOF. 10 Stakeholders also suggested patient-directed checklists to guide discussions around diabetes and BP control. We embedded candidate BCTs to target modifiable determinants of adherence. The prototype was adapted and tailored for each QI. We used the vocabulary and experiences expressed in interviews with health-care professionals and stakeholder panellists to tailor BCT content within delivery mechanisms. A graphic designer enhanced the intervention materials.

Stage 5: piloting and refining intervention content

We piloted each delivery mechanism for all QIs with five general practices involved in our earlier interview study (WP3a). 52 A researcher (EI) directly observed the delivery of each educational outreach session. She conducted brief, opportunistic semistructured interviews with practice staff (six GPs, two practice managers, and three practice nurses). Participants commented on the acceptability and feasibility of prototype feedback reports, patient-directed checklists and protocols for computerised prompts. We reviewed field notes and iteratively refined intervention content.

The implementation package adapted for four quality indicators

Audit and feedback aimed to give comparative feedback on achievement, inform and prompt recall of clinical goals, highlight consequences of changing or not changing practice, suggest strategies for change and encourage goal-setting and reflection on progress towards goals. Reports included remotely gathered, individualised practice data and presented achievement for relevant trial indicators in graphical and numerical forms. Bar charts ranked practices by achievement and allowed comparisons with other (anonymised) trial practices in the same CCG and wider region, with accompanying text providing positive or encouraging feedback according to whether achievement had risen or fallen since the first report. Reports also contained brief, evidence-based clinical messages, responses to common queries (e.g. concerning data validity) and action-planning templates. Practices received reports quarterly in both electronic and paper form. Reports were accompanied by computerised search tools to identify relevant patients for review and significant event audit templates to support root cause analyses (risky prescribing; anticoagulation for AF).

Educational outreach aimed to build on feedback by facilitating individual and group reflection, discussing barriers to action, sharing models of good practice, enhancing motivation and action-planning. We trained pharmacists over 2 days to deliver sessions. The 30-minute sessions were designed to fit in with existing practice meetings and were offered to but not mandatory for intervention practices. We invited all staff involved in patient and practice management to attend. We identified a key clinical contact to support practice engagement. We offered a follow-up session to review progress and refine action plans, as well as 2 days of pharmacist support for patient identification and review.

Prompts and reminders aimed to reinforce clinical messages and indicator adherence. Computerised prompts for risky prescribing were triggered during consultations and repeat prescribing on the basis of an algorithm for patient age, diagnosis, drug and duration. A one-click justification (ignore, add or stop medication) was required before users could proceed.

The prompt for AF was not operationalised in time for trial evaluation. To avoid duplication with existing quality improvement systems, we did not develop computerised prompts for diabetes or BP control. 72 We provided laminated reminders to convey key clinical information (e.g. management pathways) for BP control, anticoagulation for AF and risky prescribing. We provided pens and sticky notes containing key clinical messages to reinforce recommended practice. We developed patient-directed checklists to facilitate shared decision-making for BP control and diabetes control practices. However, we could not identify an efficient way to deliver these within routine consultations.

Identification of behaviour change techniques included within implementation packages

Each implementation package included at least 27 out of 30 potentially applicable BCTs (see Table 5), representing 15 of 16 BCT categories. Each package contained multiple unique instances of the different BCTs. Four BCTs that were intended for inclusion (‘identification of self as a role model’ and ‘verbal persuasion about capability’ in educational outreach, ‘discrepancy between current behaviour and goal’ in feedback reports and ‘anticipated regret’ in feedback reports and educational outreach) could not be confirmed in a subsequent content check.

Extent of shared and unique behaviour change technique content across implementation packages

Twenty-three BCTs were shared across all QIs (see Table 5). Twenty-seven BCTs were identified in strategies targeting risky prescribing and BP control and 30 were identified in strategies targeting anticoagulation for AF and diabetes control. Seven BCTs were unique to implementation packages largely focused on changing processes of care (risky prescribing and anticoagulation for AF contained BCTs relating to ‘goal-setting for behaviour’ and ‘monitoring of behaviour’) and five BCTs were unique to packages targeting patient outcomes (BP control and diabetes control contained BCTs relating to ‘goal-setting for outcomes’ and ‘monitoring for outcomes’). We did not operationalise ‘goal-setting for behaviour’ or ‘monitoring of behaviours’ for BP and diabetes control that focused on outcomes of behaviour.

Study design and setting

We conducted two parallel, cluster-randomised controlled trials (cRCTs) using balanced incomplete block designs. Cluster randomisation was essential as interventions were delivered at the general practice level (cluster). We maximised pragmatism in trial design and execution to ensure ‘real-world’ relevance. 74 Practices were recruited from West Yorkshire, England covering a socioeconomically diverse population of 2.2 million residents75 that is broadly typical of national demographics, with the exception of higher deprivation levels. 44 Over 300 general practices are organised within 10 CCGs.

Each trial evaluated the effect of adapted implementation packages on adherence to two of four high-impact QIs: diabetes control and risky prescribing in trial 1 and BP control and anticoagulation in AF in trial 2. We selected the four indicators based on scope for improvement in practice (guided by WP2 findings) and potential population benefit through the achievement of recommended treatment goals for all of HbA_1c, BP and cholesterol in type 2 diabetes, avoidance of risky prescribing of NSAIDs and antiplatelet drugs,76 anticoagulant prescribing for stroke prevention in AF,77,78 and achievement of recommended BP levels in patients with hypertension and others at high risk of cardiovascular events. 79 Our selection also took trial design into account. Within each trial, we assumed that any clinical effects of either implementation package would be independent of one another; thus, practices randomised to the implementation package for one indicator acted as control practices for the other implementation package and vice versa (Tables 6 and 7).

General practices were eligible if they used SystmOne, the computerised clinical system used by approximately two-thirds of West Yorkshire practices [The Phoenix Partnership, URL: www.tpp-uk.com (accessed 16 September 2019)]. We excluded practices involved in intervention development and piloting (i.e. WPs 2 and 3).

We used an opt-out approach to practice recruitment to facilitate participation and enhance generalisability. 80 We invited eligible practices to participate via recorded post and e-mail, with reminders at 2 weeks to non-responding practices. We included those which had not actively declined by 4 weeks. The use of opt-out recruitment avoided the biases associated with opt-in approaches and enhanced sample representativeness. 44,48 We obtained anonymised patient-level data81 through data extracts from SystmOne.

Procedures

We adapted the implementation package for each QI comprising a combination of A&F reports, educational outreach visits and computerised prompts with embedded BCTs, as detailed in WP3b. 66 A dedicated administrator contacted practices to offer to co-ordinate the educational outreach visits, which were also promoted in feedback reports and e-mail communications to all practices.

Trial end points

Table 8 lists all outcomes. In brief, primary outcomes at 11 months post randomisation were achievement of all recommended target levels of HbA_1c, BP and cholesterol in patients with type 2 diabetes; a composite indicator of risky prescribing; achievement of recommended BP targets for specific patient groups; and anticoagulation prescribing in eligible patients with AF.

We modified three of the indicators representing the primary outcomes from those we developed and assessed in earlier WPs:

• Risky prescribing. We modified ‘Prescribing of warfarin and a traditional oral NSAID without coprescription of a gastroprotective drug’ to ‘Prescribing of warfarin and a traditional oral NSAID.’ Combined use of warfarin and NSAIDs is contraindicated and it would not make clinical sense to add gastroprotection in this situation.

• BP control. We broadened the indicator that had focused on targets in treated hypertension to encompass BP control across a range of conditions associated with higher cardiovascular risk (e.g. CKD) as well as patients with a cardiovascular disease risk of ≥ 20%. We judged that this would potentially permit a greater population impact and assumed that the determinants of practice were unlikely to change markedly.

• Anticoagulation for AF. The indicator ‘Those patients with AF whose latest record of a congestive heart failure, hypertension, age > 75 years, diabetes mellitus and prior stroke (CHADS2) score is greater than one should be offered anti-coagulation therapy’ was superseded by one using the congestive heart failure, hypertension, age > 75 years, diabetes mellitus, stroke and vascular disease (CHA₂DS₂-VASc) score. We also dropped the specification that ‘Warfarin should be administered as the most effective thromboprophylactic agent’ to reflect emerging guidance allowing prescribing of non-vitamin K antagonist oral anticoagulants (NOACs) as an alternative. 82

Secondary outcomes included individual indicators within the composite primary outcomes, processes of care and continuous clinical outcomes such as HbA_1c, BP and cholesterol.

Sample size

We used WP2 data to inform trial sample sizes. Mean cluster size (number of eligible patients per practice), cluster size coefficient of variation, intracluster correlation coefficient and mean achievement rates were calculated for each primary indicator (Table 9).

Median effect sizes on processes and outcomes of care for a range of guideline implementation studies are around 4–9%. 83,84 Given that we were evaluating enhanced, multifaceted interventions and targeting indicators with scope for improvement, we judged an absolute difference of 15% for diabetes control, BP control and AF outcomes as realistic and relevant from a population perspective. Control group achievement rates in risky prescribing were higher and considering a potential ceiling effect, we considered a 5% difference as realistic and clinically relevant. To achieve 90% power, and allowing for an alpha error rate of 2.5% (to adjust for two outcome comparisons in each trial) and a 10% drop-out rate, we required 40 practices per arm in trial 1 (diabetes and risky prescribing) and 32 practices per arm in trial 2 (BP control and anticoagulation for AF). We therefore aimed to recruit 144 practices.

The opt-out approach resulted in 178 practices being recruited, allowing randomisation to a fifth arm: a non-intervention control group to further assess Hawthorne effects.

Randomisation

A two-stage minimisation process (incorporating a random element) was undertaken centrally by the trial statistician. First, practices were stratified by CCG and list size, and randomised to trial 1, trial 2 or non-intervention group (80 : 64 : 34). Practices from one CCG involved in a concurrent initiative targeting anticoagulation in AF were ineligible for trial 2 and allocated to either trial 1 or non-intervention. Second, practices within each trial were randomised (1 : 1) to individual trial arms, minimised by CCG, list size and pre-intervention adherence to the two targeted indicators relevant to that trial. General practices and trial personnel involved in intervention delivery were, of necessity, made aware of allocation but collection of outcomes for the primary end points was blind.

Data collection

We gathered anonymised outcome data remotely from general practices, mostly using data collected for QOF between 1 April 2015 and 31 March 2016.

We obtained data on practice characteristics from publicly available sources (Health Education England, Health and Social Care Information Centre)45,85,86 including practice list size (number of registered patients); number of GP partners and salaried GPs; practice training status; practice-level IMD, ethnic profile of practice register, achievement of QOF indicators (2014–15); patient satisfaction (proportion who would recommend the practice to others); patient-rated practice accessibility (proportion able to speak with GP or nurse within 48 hours of approach); and practice prescribing costs. Patient characteristics were extracted along with QOF achievement data and included age, sex and comorbidity (number of QOF registers on which patient appeared).

Statistical analysis

We undertook all data summaries and analyses on the intention-to-treat population, defined as all patients registered at randomised practices, regardless of intervention uptake or loss from the trial. Statistical testing was completed at a two-sided 2.5% significance level; effect sizes and 97.5% confidence intervals (CIs) were reported in each case. All comparisons were conducted using the within-trial randomised controls. Analyses utilising the non-intervention controls are reported separately.

Data completeness for analyses depended on the completeness of SystmOne medical records and could not be assessed within the trial data set. Missing Read codes could lead to patients being incorrectly included in or excluded from eligible populations. Similarly, indicator achievement may have been incorrectly specified as a result of missing data. For the primary and secondary analyses, we assumed that data were missing at random.

We compared primary outcomes between intervention and control practices using two-level binary logistic regression models, with patients (level 1) nested within registered practices (level 2). We adjusted analyses for patient-level covariates (gender, baseline age) and practice-level covariates (baseline practice list size, CCG, pre-intervention achievement against primary outcomes, overall QOF score 2014–15 and baseline proportion of patients with 0–3 comorbidities).

We analysed binary secondary outcomes for individual indicators within the composite primary outcomes and recorded processes of care using similar multilevel logistic models. We analysed continuous intermediate clinical outcomes using two-level linear models. We adjusted each model for the same covariates specified for the primary outcome analysis. Intervention effects on related wider practice behaviour (i.e. other indicators not targeted by the interventions) were assessed by QOF 2015–16 indicators mapped to trial outcomes. We assessed unintended effects on quality of care via a series of non-trial related QOF indicators in coronary heart disease, mental health, smoking and asthma (see table 1 of Willis et al. 73). We used practice-level achievement of each QOF indicator as the outcome in a linear model, adjusted for practice-level covariates (baseline practice list size, CCG, pre-intervention achievement against primary outcomes and overall QOF score 2014–15). Age and gender are patient-level covariates and were, therefore, not included in the analyses of QOF indicators.

We undertook appropriate regression diagnostics for binary and continuous outcomes to check the validity of the statistical modelling. We applied log transformations to continuous outcomes in cases where residual normality assumptions appeared to be violated.

As part of our process evaluation, we collected data to explore the fidelity of delivery, receipt, use and sustainability in both trial and process evaluation practices (described under WP5).

All analyses were planned prior to final data extraction and no interim analyses were planned or conducted. Statistical analyses were conducted in SAS^® (SAS Institute Inc., Cary, NC, USA) software version 9.4 (SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ^® indicates USA registration).

Screening and recruitment

Between February 2015 and April 2015, we screened 278 SystmOne general practices across 10 West Yorkshire CCGs (Figure 1). A total of 243 (87.41%) were eligible and invited to participate; 56 (23.05%) of these practices opted out, largely because of workload pressures, and nine were excluded for other reasons (see Report Supplementary Material 1, Tables 1–5). Typically practices opted out within 2 weeks of receiving the invitation letter (mean 12.51 days, SD 11.47 days); thus, 178 (73.25%) of eligible practices were randomised. Practice characteristics were broadly similar between those not randomised and those randomised (see Appendix 1, Table 19).

FIGURE 1.

The CONSORT flow diagram for cluster-randomised evaluation (WP4a). a, Safe haven practices are for those patients who have demonstrated violent tendencies and have been removed from usual general practice or for ex-offenders; b, one practice in the risky prescribing arm merged with a non-Action to Support Practices Implementing Research Evidence (ASPIRE) practice in advance of the fourth feedback report; however, as they received the first three feedback reports some outcome data are available and they are included in the final analyses; c, one practice in the BP control arm closed in advance of the third feedback report; however, as it received the first two feedback reports some outcome data are available and these are included in the final analyses. Adapted from Willis et al. 87 © 2020 Willis et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Adapted from Willis et al. 87 © 2020 Willis et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Forty practices were allocated to each of the diabetes control and risky prescribing arms in trial 1, 32 practices were allocated to each of the BP control and anticoagulation in AF arms in trial 2. A further 34 practices were allocated to a non-intervention control arm (see Report Supplementary Material 1, Tables 6 and 7). None of the randomised practices withdrew from any aspect of the trial. One practice in the risky prescribing arm merged with a non- Action to Support Practices Implementing Research Evidence (ASPIRE) practice following the third feedback report at 9 months; no outcome data were available after 9 months. One practice in the BP control arm closed during the study because of GP retirement. This practice received two of the four planned feedback reports; outcome data were unavailable beyond 6 months’ follow-up. Two ASPIRE practices merged during the trial, one from diabetes control and one from risky prescribing; all patients from both practices became registered at the diabetes control practice. Both practices received all four feedback reports and final outcome data were available for both practices. One further ASPIRE practice from risky prescribing merged with a non-ASPIRE practice. This practice received three of the four planned feedback reports; outcome data were unavailable beyond 9 months’ follow-up.

Baseline practice characteristics were well balanced by trial and indicator (see Appendix 1, Tables 19 and 20, and Report Supplementary Material 1, Table 8): across all 178 practices, the mean list size was 7249.76 patients (SD 4306.72 patients); the mean IMD was 30.85 (SD 13.70, falling within the top quarter of social deprivation); and mean pre-intervention adherence was 33.29% (SD 7.08%) for diabetes control, 7.91% (SD 4.07%) for risky prescribing, 65.87% (SD 6.67%) for BP control and 66.20% (SD 11.82%) for anticoagulation in AF.

Baseline characteristics for patients within the populations relevant to the four indicators, as well as all patients within the practices, were broadly similar between the trial arms (see Appendix 1, Table 21, and Report Supplementary Material 1, Tables 9–12). The data obtained relating to polypharmacy (number of repeat prescriptions per patient) were found to be incomplete and, hence, were not used in any analyses.

Primary outcomes

The primary outcome results are summarised in Table 10, with detailed summaries in Appendix 1, Table 22. In trial 1, achievement of diabetes control at 11 months for practices assigned to this implementation package was 24.19% compared with 23.74% for practices assigned to the control arm (risky prescribing); an adjusted OR of 1.03 (97.5% CI 0.89 to 1.18), thus providing no evidence of an intervention effect.

There was evidence of a significant intervention effect for the risky prescribing implementation package. The proportion of patients with a record of risky prescribing at 11 months in practices assigned to this implementation package was 4.94% compared with 5.99% for control practices (diabetes control); an adjusted OR of 0.82 (97.5% CI 0.67 to 0.99). Thus, the odds of risky prescribing for a patient in an intervention practice were 18.5% lower (i.e. better) than for a patient with the same characteristics in a control practice.

In trial 2, there was no evidence of a significant effect on primary outcomes for either implementation package. Achievement of BP control at 11 months for practices assigned to this implementation package was 53.62% compared with 52.32% for practices assigned to the control arm (anticoagulation in AF); an adjusted OR of 1.05 (97.5% CI 0.96 to 1.16). Achievement of anticoagulation in AF at 11 months for practices assigned to this implementation package was 73.20% compared with 75.18% for control practices (BP control); an adjusted OR of 0.90 (97.5% CI 0.75 to 1.09).

Secondary binary outcomes

In trial 1, secondary analyses of individual indicators forming the composite primary outcomes and recorded processes of care showed no statistically significant effects relating to the diabetes control implementation package (see Report Supplementary Material 1, Tables 13–26).

The risky prescribing implementation package resulted in significantly lower (i.e. better) levels for one individual indicator, ‘Patients aged 65 years or over prescribed aspirin and clopidogrel without coprescription of gastroprotection’. Prescribing levels were 25.32% in implementation package practices compared with 35.20% in control practices: an adjusted OR of 0.62 (97.5% CI 0.39 to 0.99). Thus, the odds of risky prescribing for a patient in an intervention practice was 37.59% lower (i.e. better) than for a patient with the same characteristics in a control practice (see Report Supplementary Material 1, Table 30). We identified no other statistically significant effects relating to other risky prescribing indicators (see Report Supplementary Material 1, Tables 27–37). In trial 2, we found no evidence of significant intervention effects on any individual indicators or processes of care (see Report Supplementary Material 1, Tables 38–51).

Secondary continuous outcomes

There were no significant intervention effects in the secondary analyses of the continuous intermediate clinical outcomes: systolic and diastolic BP (in both the diabetes control and the BP control populations); HbA_1c and total serum cholesterol (diabetes control only; see Report Supplementary Material 1, Tables 52–57). The conclusions for total serum cholesterol and HbA_1c were robust to log-transformation of the outcomes in a sensitivity analysis conducted following model diagnostic checks.

Secondary analysis of Quality and Outcomes Framework indicators

We explored intervention effects for QOF indicators to assess whether or not there were any benefits in the achievements of QOF indicator groups relevant to any of the implementation packages and whether or not there were any unintended harms of reduced achievement in QOF indicator groups not relevant to any of the packages (summary statistics; see Report Supplementary Material 1, Tables 58–61). Statistical modelling was not possible for all indicators because of violation of modelling assumptions, particularly where large proportions of practices attained 100% achievement for an indicator. For those indicators where modelling was appropriate, we found no statistically significant intervention effects (see Report Supplementary Material 1, Tables 62–75).

Intervention fidelity

Full analyses of intervention fidelity are presented under the process evaluation (WP5). In brief, all practices received A&F reports as intended (with the exception of those that closed or merged). Outreach visits were delivered to 67 (46.53%) practices, with 8 (5.56%) receiving two visits. There was similar uptake between trial arms (see Appendix 1, Tables 26 and 27). The offer of visits was declined by 77 practices, 52 (67.53%) of which did not give a reason (see Report Supplementary Material 1, Table 76). One hundred and twenty-six (87.50%) practices activated access to searches and computerised prompts (risky prescribing practices only; see Report Supplementary Material 1, Tables 102–103).

General approach and model choice

We treated the implementation packages as separate interventions, and hence did not aggregate costs and effects. We aggregated subindicators for relevant composite indicators. Since we were evaluating the implementation of NICE recommendations, we aimed to mirror evaluative tools used in the evidence syntheses underpinning NICE guidelines. This was not possible for risky prescribing since this was a new indicator although we relied heavily on previous modelling for NSAIDs.

Where possible we followed the reference case set out by NICE. 99 As such, we reported cost per incremental quality-adjusted life-year (QALY) from a health and personal social service provider perspective. We adopted a lifetime perspective and discounted costs and benefits post 1 year at the NICE preferred rate of 3.5% per annum. Where we developed or adapted models, we conducted targeted searches to identify parameter values. We developed and parameterised the models iteratively with ASPIRE team clinicians (RF, SA and DP). The decision models allowed the generation of cost per QALY estimates by translating trial effectiveness results on implementation and clinical outcomes into event risks and expected costs, survival and quality-of-life impact. We estimated costs where necessary using standard sources including NHS Reference Costs100 and the Personal Social Services Research Unit report. 101 We converted relevant historical or non-UK costs to 2017 UK prices using a health-related inflation tool. 102

We report incremental cost-effectiveness ratios (ICERs) per QALY gain in each evaluation. To aggregate cost-effectiveness within indicators (for risky prescribing and BP control), we generated weighted averages of costs and benefits based on the proportion of a general practice list eligible for each indicator. ICERs below the NICE preferred threshold of £20,000–30,000 cost per QALY were taken to indicate cost-effectiveness. We ran deterministic sensitivity analyses to test the impact of changes in parameter values and assumptions. Where possible, we also ran probabilistic sensitivity analyses to test the overall impact of parameter uncertainty on model results. We present these results as cost-effectiveness planes and cost-effectiveness acceptability curves (CEACs). 103

Intervention costing

The ASPIRE implementation package costs comprised fixed and variable components that were all expressed at practice level since all reported outcomes, namely QALY gain and health-care savings, were computed for the average practice list size of 7130 patients. The fixed cost consisted of preparing, delivering and receiving the interventions. To illustrate, outreach education involved costs relating to facilitator training, including their time and room hire (preparation costs); the time delivering the outreach visit (delivery costs); and the time of practice staff participation in outreach sessions (receipt costs). The fixed costs of the interventions are provided in Table 11 (further details available on request). Our assumptions were generally conservative.

We assumed the fixed costs were equally distributed across all four ASPIRE implementation packages. However, to retain a conservative stance in our base-case analysis, we doubled the value of this per intervention cost when computing the fixed cost per practice (n = 144) of implementing each specific intervention. We undertook sensitivity analysis on the magnitude of the risky prescribing implementation package fixed cost. The base-case fixed cost was £2439 (£175,592.36/144 × 2) per practice. We tested an optimistic scenario where the per-practice fixed cost was £1219 (£175,592.36/144).

Variable implementation package costs included additional GP consultations to review and change patients’ prescriptions, additional tests and the cost of additional medication (e.g. coprescribing gastroprotection). The variable cost element of ASPIRE was directly related to implementation package effectiveness in changing care processes; thus, no additional variable costs were incurred if GPs did not change practice on intervention receipt. Implementation package effects were measured using the trial arm differential in the proportion of eligible patients for each relevant indicator achieving recommended care or outcomes.

Risky prescribing

Our evaluation included all of the risky prescribing subindicators except for indicator nine, which covers the prescription of a diuretic in combination with an angiotensin-converting-enzyme inhibitor or angiotensin receptor blocker in those taking an oral NSAID. We could not find an acceptable evidence base on which to base parameters and it was relevant for only 3% of the current sample.

We based the risky prescribing model around a previous model of NSAID prescription used for the NICE osteoporosis guideline (CG177104) (Figure 2). We recreated and adapted this model and populated it using results reported for three of the most commonly prescribed NSAIDs (diclofenac, naproxen and ibuprofen) and aspirin. Patients receiving NICE adherent prescriptions (e.g. a proton pump inhibitor with NSAIDs where indicated) are less likely to experience negative health events such as a gastric bleed. We assumed that patients experiencing an event would stay in the post-event health state for the remainder of the model time horizon. We assumed that patients in the post-event state had their NSAID stopped and were prescribed nothing or a simple analgesic such as paracetamol. The model has 3-month cycles and a lifetime duration.

FIGURE 2.

Economic evaluation: risky prescribing model structure (WP4b). GI, gastrointestinal; HF, heart failure.

The risk of adverse events per treatment were adjusted down to reflect lower-dose prescriptions in the UK. We estimated baseline adverse event risks by applying the following weights to each NSAID treatments based on data reported in the NICE guideline (see Report Supplementary Material 1, Table 87): diclofenac, 0.459; ibuprofen, 0.248; and naproxen, 0.077. Report Supplementary Material 1, Table 88, includes the additional risks associated with risky prescribing identified with targeted literature searches. Report Supplementary Material 1, Table 89, includes the effectiveness values used and Report Supplementary Material 1, Table 90, includes the model costs. We assumed variable intervention costs comprising two GP visits (an initial consultation and one follow-up, £72 in total) and a proton pump inhibitor prescription where relevant (i.e. indicators 1–5 and 7). Given that changes in prescription may be made at other GP visits or by another means (e.g. telephone), our assumption of two GP visits is conservative.

As in the NICE guideline, the impact of adverse gastrointestinal and cardiovascular events on individuals’ quality of life was modelled via utility multipliers (Report Supplementary Material 1, Table 91) applied to age-specific baseline utility scores for the general UK population. 105 The UK population baseline scores used were 0.78 [standard error (SE) 0.26] at age 65 years and 0.73 (SE 0.27) at age 75 years. With the exception of stroke, we assumed utility would return to pre-event levels after one cycle.

We created an additional model, based on that used to inform the NICE acute kidney injury (AKI) guideline (CG169),106 to capture the risky prescribing indicator (9) covering the prescription of NSAIDs in patients with CKD (Figure 3). Transition probabilities are included in Report Supplementary Material 1, Table 92, with costs in Table 93. The variable intervention cost was assumed to be two additional GP consultations to review and change NSAID prescriptions (£72 per patient). Report Supplementary Material 1, Table 94, includes utility values, the same as used in CG169. 106 Report Supplementary Material 1, Table 95, includes the proportions used to weight the results.

FIGURE 3.

Economic evaluation: risky prescribing submodel – CKD (WP4b). RRT, renal replacement therapy. RRT are tunnel states only.

Blood pressure control

The economic evaluation broadly followed the above methods for risky prescribing. We identified but could not obtain the model107 informing the BP indicator (CG127);108 therefore, we recreated and adapted it (Figure 4 summarises the model structure). We based the modelling approach on that used for the NICE guidance for hypertension. 108 The model had a lifetime horizon and 3-monthly cycles with future costs and benefits discounted at 3.5% per annum. The cost and risks of a health event are often higher immediately post event and often decrease as time passes. To reflect this, all health states in the model have an ‘acute’ health state and a post-event health state. In all cases, if patients survive, they move automatically to the post-event health state after a period of 3 months (one cycle). The post-event health states are generally less costly and have reduced mortality risks.

FIGURE 4.

Economic evaluation: BP control model structure (WP4b). SA, stable angina; TIA, transient ischaemic attack; UA, unstable angina.

Report Supplementary Material 1, Table 96, includes the modelled effectiveness estimates and Table 97 details transition probabilities. The risks for coronary heart disease and stroke are taken from the NICE CG127 model. 108 The original model based these risk parameters on outputs from the Framingham coronary heart disease and stroke equations by gender and age groups. 109 We employed these values and simulated associated CIs using a risk calculator spreadsheet developed by the University of Edinburgh. 110 Although the Framingham equations have been generally superseded in the UK by QRISK,111 we mirrored the original modelling approach. As the risk equations do not provide more granular detail about type of coronary heart disease and stroke, we followed NICE CG127108 modelling and used percentage distributions across coronary heart disease (MI, stable angina, unstable angina) and stroke [stroke or transient ischaemic attack (TIA)] event types. 112 Although some health-event risks described may be correlated (e.g. stroke and MI), the NICE approach assumes the events are independent; we followed this assumption. Patients meeting treatment targets for hypertension have a lower risk of coronary heart disease and stroke. Intervention effectiveness was determined by mean, adjusted, systolic BP outcomes at trial end. The relative risk reduction for male and females following treatment receipt were taken from a meta-analysis. 113 To cover all indicator populations, we identified additional risk parameter values using targeted searches and risk calculators. However, we used the same background mortality (UK life tables) and BP control risk reduction values for all indicators.

We based the average cost of hypertension treatment on antihypertensive drug therapy and an annual check with the GP (see Report Supplementary Material 1, Table 98). As we had no data on changes in process of care, we explored alternative costs associated with differing levels of implementation. We took costs of cardiovascular events from the NICE guideline,108 national cost sources and published studies. As in the original evaluation, we assumed that these costs were fixed and thus not random in the probabilistic sensitivity analysis. We based the quality-of-life (utility) values (see Report Supplementary Material 1, Table 99) for the cardiovascular event health states on the NICE statins assessment. We assumed no utility decrement for receipt of hypertension treatment or experience of a TIA. Report Supplementary Material 1, Table 100, includes the proportions used to weight the results.

Diabetes

We used the United Kingdom Prospective Diabetes Study (UKPDS) outcomes model,98 which had informed the NICE diabetes guideline (NG28)114 and is a comprehensive outcomes model based on a large, UK longitudinal data set. The model produces estimates of life expectancy, quality-adjusted life expectancy and cost of therapies and complications. Confidence intervals were created using the bootstrapping technique. This individual-level model requires patient-level data on a range of demographics (e.g. age and ethnicity), risk factors (e.g. body mass index and smoking status) and existing health conditions (e.g. amputations and blindness). As the trials did not provide patient-level data, a cohort of individuals were simulated using the mean and CIs from the trial analyses. For simplicity, and in the absence of other data permitting an alternative approach, we assumed these to be normally distributed. We randomly generated data on other model input parameters (e.g. gender, smoking status and presence of AF) such that they satisfied prevalence estimates in this group (e.g. such that 3.9% of patients had existing AF; see Report Supplementary Material 1, Table 101). As correlations between these input parameters were unknown and uncontrolled for, the simulations are likely to lead to more rather than less uncertainty.

A sample of 1000 patients was simulated for each arm, the only difference in the sample being trial effectiveness data (systolic BP, low-density lipoprotein cholesterol and HbA_1c), simulated based on trial results (and their CIs). We captured first order uncertainty by simulating outcomes for each patient multiple times in loops and second order (sampling) uncertainty was captured by bootstrap resampling of the 1000 patients. One hundred loops and 1000 bootstraps were conducted. As a sensitivity analysis, we also ran the model for 1000 loops and no bootstraps, including only the mean outcome values for the two trial arms. In the absence of other data to inform dynamic parameter values, we carried forward baseline model input values into each model year. The fixed cost of the intervention at a patient level was £0.28 [(£175,592.36/178 × 2)/7130]. As the outcomes from the trial entered in the model were continuous health outcomes, it was not possible to attach to them costs associated with a change in practice (process outcome); however, we explored this using sensitivity analyses in which additional resource was expended in the trial arm.

Risky prescribing

For an average UK practice of 7130 patients, the implementation package targeting all risky prescribing indicators (except indicator 9) is more expensive but more effective than standard care (Table 12). The risky prescribing implementation package ICER is below the threshold (£20,000) used by NICE. Although the values are per practice, the ICER of £1359 is also that per patient and generally represents comparative value.

Figures 5 and 6 show the probability sensitivity analysis (PSA) results in a scatter plot and a CEAC, respectively. The spread of the simulated ICERs on the plane suggest greater uncertainty in terms of incremental QALYs than in incremental costs. A greater proportion of ICERs fall below than above the cost-effectiveness threshold line. The probability that the implementation package ICER is below £20,000 is relatively high, chiefly because of low intervention costs; hence, even at a low benefit, it is likely to represent value for money. At a QALY willingness-to-pay (WTP) threshold of £20,000, the risky prescribing intervention has a > 79% chance of being cost-effective.

FIGURE 5.

Economic evaluation: composite indicator of risky prescribing – scatterplot of simulated incremental cost and QALY (WP4b).

FIGURE 6.

Economic evaluation: composite indicator of risky prescribing – CEAC.

Results vary greatly by indicator. Figures 7–10 contrast cost-effectiveness results for indicator 4, where the trial-arm difference in prescribing is statistically significantly better in the implementation package arm; and for indicator 2, where the prescribing was better in the control arm. For indicator 4, a far greater proportion of simulated ICERs are below the cost-effectiveness threshold. Figure 11 is a CEAC run when the fixed intervention costs are adjusted to alternative optimistic (cheaper) and conservative (more expensive) scenarios. Even using a conservative costing scenario, the risky prescribing implementation package has a > 70% chance of being cost-effective.

FIGURE 7.

Economic evaluation: risky prescribing indicator 4 (prescribing of aspirin and clopidogrel in patients aged ≥ 65 years without coprescription of gastroprotection) – scatterplot of simulated incremental cost and QALY (WP4b).

FIGURE 8.

Economic evaluation: risky prescribing indicator 4 (prescribing of aspirin and clopidogrel in patients aged ≥ 65 years without coprescription of gastroprotection) – CEAC (WP4b).

FIGURE 9.

Economic evaluation: risky prescribing indicator 2 (prescribing of traditional oral NSAID in patients aged ≥ 75 years without coprescription of gastroprotection) – scatterplot of simulated incremental cost and QALY (WP4b).

FIGURE 10.

Economic evaluation: risky prescribing indicator 2 (prescribing of traditional oral NSAID in patients aged ≥ 75 years without coprescription of gastroprotection) – CEAC (WP4b).

FIGURE 11.

Economic evaluation: composite indicator of risky prescribing – CEAC for three sets of fixed intervention costs (WP4b).

Blood pressure control

The BP implementation package at practice level was more expensive (incremental cost = £42,192) but more effective (incremental QALYs = 13.00) than standard care (Table 13). The incremental values at patient level were very small (£5.92 for costs and 0.0018 for QALYs). The estimated ICER per practice and patient was £3246, which is indicative of cost-effectiveness; however, there is considerable uncertainty in the results.

Figures 12 and 13 show the PSA scatter plot and CEAC, respectively. A slightly greater proportion of ICERs fall below than above the cost-effectiveness (£20,000 per QALY) threshold line. This is reflected in the finding that the probability of the intervention being cost-effective is only just above 50%, at 51.9%.

FIGURE 12.

Economic evaluation: composite indicator of BP control – scatterplot of simulated incremental cost and QALY (WP4b).

FIGURE 13.

Economic evaluation: composite indicator of BP control – CEAC (WP4b).

Results by subindicator are largely similar with the intervention yielding positive net benefit, either dominating standard care or yielding an ICER of < £6000 per QALY gained. Two exceptions were the indicators relating to patients with a history of stroke or TIA and relating to patients with diabetes for whom standard practice (no intervention) was more cost-effective than the BP implementation package. This was as a result of greater mean reductions in BP in the control arm. In a third indicator (peripheral arterial disease) the outcomes were practically the same across arms. In the best performing indicator (9.5: BP control in patients with CKD), the PSA indicated a 63% chance of being cost-effective (Figures 14 and 15).

FIGURE 14.

Economic evaluation: BP control indicator 5 (achievement of BP < 140/90 mmHg in patients aged < 80 years with peripheral arterial disease) – scatterplot of simulated incremental cost and QALY (WP4b).

FIGURE 15.

Economic evaluation: BP control indicator 5 (achievement of BP < 140/90 mmHg in patients aged < 80 years with peripheral arterial disease) – CEAC (WP4b).

Diabetes control

Table 14 includes the mean of the individual-level summary results from the UKPDS model. The differential between trial arms in costs and benefits was negligible. Over the model’s lifetime horizon, 12.26 and 12.24 life-years were generated by the implementation package and control arms, respectively. This equated to 9.78 and 9.76 QALYs, respectively; thus, the package appeared more effective than standard practice (incremental QALYs = 0.02). The implementation package was more expensive overall (incremental cost £12.79); these incremental values yielded an ICER of £573.37. At WTP thresholds of £20,000 and £30,000 per QALY, the implementation package has only a 34% and 35% chance, respectively, of being cost-effective. The spread of ICERs from the bootstrapping is shown in Figure 16; however, the Monte Carlo Error (MCE) for the initial model (100 loops and 1000 bootstraps) was of the same magnitude as the incremental values indicating highly uncertain results. A second model run where the sample outcomes were simulated over 1000 loops (0 bootstraps) indicated incremental QALYs of –0.0007 and costs of £6.37 over a lifetime (control dominates). A further run including only mean values for the arms over 10,000 loops (0 bootstraps) generated incremental QALYs of –0.0085 and costs of £28.04.

FIGURE 16.

Economic evaluation: cost-effectiveness plane for diabetes control implementation package (WP4b).

Given these outcomes, their sensitivity to model processes and the magnitude of differences observed, we cannot be confident in the results and we cannot recommend the diabetes implementation package. As the intervention costs were negligible, and there was no reasonable argument for adjusting the effectiveness parameters input into the model, sensitivity analyses were not conducted.

Participants

Eight general practices not participating in the trials took part in the in-depth, mainly qualitative process evaluation. An independent statistician randomly assigned practices to one of four adapted implementation packages, which were the same as those used in both trials. Allocation was balanced to reflect CCG membership and practice list size. We gathered further fidelity data from all practices assigned to the implementation package in the trial.

Field work (Table 15) with the process evaluation practices was conducted by Cheryl Hunter, a social scientist researcher. Her role was to observe and collect data but not to interfere with implementation package delivery, as far as possible.

Interviews

Cheryl Hunter carried out theory-based semistructured interviews (informed by the TDF and NPT constructs) with key members of practice staff at two time points. These included clinical leads, practice managers and other staff involved in the organisation or delivery of care for the specific QI. The first interviews followed delivery of the initial educational outreach session and explored the role and responsibilities of the staff member, barriers to and facilitators of achieving the QI, responses to the intervention components and subsequent actions by the practice (35 participants; 3–6 per practice). The second interviews took place towards the end of the intervention year and revisited questions from the first interview as well as exploring changes within the practice and perceived intervention usefulness and fit (27 participants; 2–4 per practice). Interviews were audio-recorded, transcribed and anonymised.

Cheryl Hunter conducted a final group interview at all eight practices 1–3 months after delivery of the final feedback report. She asked practices to reflect on the entire year, provide feedback on intervention components and explore any further plans to conduct work around their assigned QI. Group interviews were written up in detailed anonymised field notes.

Cheryl Hunter conducted interviews with nine facilitators who delivered the educational outreach visits and outreach support. These interviews explored how outreach sessions and support were delivered in practice, whether or not the facilitators felt that these were delivered as intended and what seemed to work (or not) when delivering these components in practice.

Observations

Approximately 10 hours of observation was conducted over the year at each practice. Observations were informed by TDF and NPT constructs (e.g. attending to staff, individually and collectively, conceptualise the intervention and what barriers or enablers might contribute to implementation). Cheryl Hunter attended ASPIRE-related meetings, such as educational outreach sessions and on-site outreach support. Cheryl Hunter also observed routine practice in reception or administrative areas and attended a variety of practice meetings (including peer review, nursing team, clinical and business meetings). No clinical interactions, such as consultations, were observed. Anonymised field notes structured around the TDF and NPT constructs were created of all meetings and conversations at practices.

Documentary data

We collected related documents including protocols for the assigned QI, letter templates for patients, patient information leaflets and minutes from relevant clinical or practice meetings. E-mail correspondence and notes from telephone conversations between Cheryl Hunter and practice staff were anonymised. Practice managers were also given box files and invited to use these to store any documents relevant to the QI.

Intervention administrative data

Feedback reports were sent quarterly by post and e-mail, and copies were taken by facilitators to outreach visits. E-mail delivery receipts were recorded. For the eight practices that participated in the in-depth process evaluation, delivery, receipt and use of feedback reports, searches and significant event audit forms were tracked through observations, interviews and documentary records. Practices were invited to join a SystmOne organisational group and were reminded about it in feedback reports, by e-mail and at outreach visits. We collected data on whether or not practices joined the group (which gave automatic access to the computerised searches), but it was not possible to digitally track use of the searches. All practices were invited to participate in outreach visits by a programme administrator, who recorded data on reasons for declining outreach visits. The facilitators delivering educational outreach and support also recorded who attended outreach sessions, how many sessions were delivered, what additional pharmacist support was taken up and delivered to practices, and whether or not feedback reports were received and used. For risky prescribing only, practices were given access to computerised prompts through an organisational group (in the same manner as the computerised searches). Further insight into the delivery and receipt of the risky prescribing computerised protocols came from interviews and observations with two process evaluation practices allocated to this intervention. Laminates, pens and sticky notes summarising key clinical messages were delivered by post throughout the year (and sometimes via facilitators at outreach visits). Interviews and observations at the eight practices gave further insight into intervention receipt and use.

Post-trial survey

We aimed to estimate intervention fidelity post trial in both trial and process evaluation practices. We developed a brief e-mail survey based on NPT to explore intervention receipt, enactment and ongoing workability. The survey asked whether or not the practice received the reports, the reports were relevant to the practices, the reports were shared with colleagues in the practice, the reports were discussed within the practice, the reports were used to change how people in the practice worked and they had used the SystmOne searches. Response options were limited to ‘yes’ or ‘no’, but practices could also provide further free-text feedback if desired.

Analysis

Interview and observational data were entered into a QSR NVivo 10 project and coded using a framework approach. 54,120,121 The analytic framework included TDF constructs to compare planned with actual implementation and NPT was adopted to understand individual- and team-implementation processes, in particular how individuals and groups conceive of, engage with, and enact and reconfigure work in response to the implementation package. Data were grouped by practice and chronologically ordered. Review of coded data, chronological accounts and process models was carried out in sequential team meetings. The team comprised researchers with experience in psychology, sociology, implementation science and primary care.

Cheryl Hunter developed a logic model (see Figure 18) and process models (see Figure 19) for each intervention delivery mechanism to describe the planned process of implementation as conceived by the intervention developers. Cheryl Hunter also drew up ‘disrupted’ process models, highlighting hypothesised points where difficulties could arise to disrupt implementation as planned. Process and ‘disrupted’ process models were used to sensitise the researcher and analysis team during data collection and analytic discussions. Process models depicting implementation and chronological narrative accounts were developed for each process evaluation practice documenting intervention delivery, receipt and use. The post-trial fidelity survey coded fidelity as high, medium and low. Fidelity was considered high if practices received feedback reports, took up outreach and accessed patient-identifiable searches; medium if they received feedback and outreach or searches; and low if they received only feedback reports. Survey and outreach facilitator data from the trials were analysed descriptively and compared with the more in-depth practice case narratives, to help explain the trial findings and augment insights into implementation fidelity.

Pre-intervention context

Pre-intervention achievement was broadly comparable between trial and process evaluation practices across trials and indicators, with any variations reflecting the smaller sample of process evaluation practices (see Report Supplementary Material 1, Table 83).

The practices had varying systems and procedures in place for targeted indicators. The most advanced were generally for diabetes, for which there was a nominated lead GP and care was mainly shared between nurses and GPs. Patients were recalled for reviews at intervals according to their progress in attaining treatment goals. Care was organised similarly, if less intensively, for BP management. BP monitoring was variously carried out by patients using home monitors, machines in waiting rooms or in consultations with health-care assistants, nurses or GPs, with the latter mainly having responsibility for prescribing. Practices had delegated GP leads for cardiovascular disease or AF. Neither practice assigned to the AF package involved their nursing staff in managing this condition. Both practices were aware of anticoagulation as a priority in their CCGs and of recent changes to clinical guidelines, and had started or had plans to start systematically reviewing patients who were potentially eligible for treatment. For risky prescribing, both practices assigned to this package had nominated GP prescribing leads. In one practice, a prescribing clerk regularly reviewed repeat prescriptions and did work around reviewing costs of prescribing for medicines management. The lead GP also met with the practice manager routinely to review any audits or work required for the CCG. Risky prescribing was the only targeted indicator without specific QOF-related systems of care and goals.

Intervention delivery and receipt

All trial and process evaluation practices received feedback reports as intended (except where there had been closures or mergers). Outreach visits were delivered to 67 (46.53%) practices, with eight (5.56%) receiving two visits. Uptake was similar between trial arms. Visit timings are presented in Figure 17. Detailed summaries are in Appendix 1 (Tables 26–29).

FIGURE 17.

Cluster-randomised evaluation: distribution of delivery of outreach visits across the intervention period. Bars indicate delivery of feedback reports.

Outreach visits averaged 36 minutes (see Report Supplementary Material 1, Table 77). Risky prescribing and anticoagulation outreach meetings were all attended by someone with leadership responsibilities compared with BP control (81.82%) or diabetes control (85.00%). On average fewer practice staff attended risky prescribing meetings: three (range 1–10) compared with four (1–13) for anticoagulation, 4.5 (1–15) for diabetes control and 5.5 (2–14) for BP control (see Report Supplementary Material 1, Table 78). Key clinical topic leads were present at most visits (see Report Supplementary Material 1, Tables 79 and 80). Most practices (94.03%) that received outreach visits produced action plans (see Report Supplementary Material 1, Tables 81 and 82). During the outreach session, practices reported participating in a wide range of other quality improvement initiatives, particularly for diabetes (see Report Supplementary Material 1, Tables 83 and 84). Outreach visits were declined by 77 practices; 52 (67.53%) provided no reason and 22 declined for reasons related to workload (11 practices; 14.29%) or lack of interest (11 practices; 14.29%). One hundred and twenty-six (87.50%) practices joined the organisational groups (see Report Supplementary Material 1, Table 82; for baseline characteristics, see Tables 85–86), allowing them to access searches and, for risky prescribing, computerised prompts. Overall, fidelity receipt was high for 47 (32.64%) practices, medium for 82 (56.94%) and low for 15 (10.42%) (see Appendix 1, Table 28).

Audit and feedback

Process evaluation practices consistently viewed their QI as important and the data source as credible; however, practices typically interpreted the reports as relevant only to staff directly involved in clinical care for that indicator. This was demonstrated in the pattern of report dissemination and in how and whether or not reports were discussed as a group. Staff largely continued existing ways of working.

The feedback reports seemed motivational and to raise awareness of key clinical messages, but did not consistently encourage individual or organisational behavioural change. Required actions were generally interpreted as relevant only to one or two people, or seen as already aligned with existing tasks and organisational structures. Actions were typically assigned to one or two GPs with little existing ‘slack’ and progress depended on whether or not and how they prioritised and managed required work; subsequently, progress was often slow and sometimes non-existent.

Some staff responded negatively to continuing feedback; this occurred in diabetes control and BP control practices in which the numbers to review were high (up to 400 patients) and considerable resources were already directed towards QOF work. Feedback in this case created dissonance, as practice staff felt overwhelmed yet saw no progress. They then queried the data or the outcomes measured as a way to resolve this dissonance and protect their professional sense of self.

One practice seemed to take a whole-practice approach to involvement, facilitated by their small size (< 10 staff members). Engaging their administrative staff kept the work going even when the GPs felt too busy, and the different staff members reminded and supported each other. This collective action, with clear roles for each staff member, may explain their steady improvement throughout the study.

The searches generated lists of patients requiring action by staff. However, the searches were not particularly visible to staff, suggesting a mismatch between the administrative and managerial staff who received them and the clinicians responsible for reviewing the generated patient lists.

Educational outreach and support

Practices tended to perceive educational outreach as the start of the intervention; they received the initial reports and then awaited the proffered help. Facilitators perceived these visits to be more useful when some practice staff had reviewed and considered the reports first; our observations suggested that only a small proportion of practices did this.

Who attended outreach visits was left to the practice’s discretion. As noted earlier, practices generally had existing team arrangements for diabetes, involved everyone to some extent in BP control, and mostly had GP prescribers, with a named GP lead for prescribing. Practices also tended to fit the outreach visit into a routine meeting (as we had intended). This meant that staff awareness of the study and engagement with the educational outreach visit varied. It might have resulted in more tailored action plans if the diabetes and BP arms were given more guidance as to who to invite and include in the visits. Only practices in these two intervention arms provided any negative feedback on the outreach visits.

Outreach support seemed to create an expectation that someone else (the facilitator) would do the work for the practice. Some practices turned the outreach support down as they felt it better to do the work themselves; others accepted the offer because they lacked capacity to do the work themselves. The size of the workload was an influence, with diabetes and BP control workloads being viewed as overwhelming. Practices might have been more motivated if they had to focus on smaller target populations, or if plans had more clearly outlined individual roles and expectations. The clear link between action and outcome in the anticoagulation and risky prescribing practices facilitated engagement with action-planning and motivated staff to engage; the link between action and outcome was much less clear in the diabetes and BP control practices.

Delays between organising and delivering the outreach visits (see Figure 17) and support are likely to have differentially affected progress by QI. Practices assigned to diabetes and BP control packages typically required a number of staff to be involved in seeing patients (for tests, prescriptions and lifestyle advice), and would then require further review to establish any effect of management changes on outcomes. Arranging educational outreach sessions within the first 6 months of the trial was not feasible for many practices. If the ostensible ‘start’ of the intervention work was after the outreach visit, patients might not have been identified for action until near the end of the trial intervention period.

Computerised prompts and laminated reminders

Observations in the process evaluation practices suggested that the laminated reminders had low visibility and were unlikely to have contributed to any intervention impact. The only observed indication that the prompts influenced behaviour came from one practice, which improved the awareness of the prescribing clerk and enabled her involvement in the work. This practice was alone in enrolling an administrative member of staff in the intervention work over the full study period although others were involved in educational outreach or initial search generation.

Change as a result of the implementation package

Across the trials, achievement of the four primary outcome indicators appeared marginally better in practices that had received educational outreach meetings than in practices that had not (Table 16). Practices allocated to the diabetes control and anticoagulation for AF implementation packages that received feedback without educational meetings appeared to perform marginally worse than the control practices that had not received these packages.

There were signals from the process evaluation practices that staff had internalised the key clinical messages in risky prescribing and, to some extent, in anticoagulation for AF and felt more likely to act in accordance. However, there were also continuing barriers to acting on messages routinely. These included the relative infrequency of clinicians seeing new cases of AF, the relative complexity of managing AF, difficulties finding information in patient records and competing demands within consultations.

The anticoagulation and risky prescribing practices did systematically review all patients not achieving recommended care – although this was repeated more than once throughout the year in only one practice. The other three practices either completed a review once or were still in the process of completing it by the end of the intervention period. Some practices recognised the work as ongoing and discussed putting a further audit cycle or check on progress into their systems.

The diabetes and BP control practices tended not to engage in the systematic review of patients, unless the outreach support did substantial preparatory work for them. They struggled with capacity and felt as if they had already poured extensive resources into, and had developed particular working structures for, addressing these topics; there was a collective failure to understand how the package differed from usual ways of working. They were resistant to changing these structures or adding in extra resources. As they continued to work within structures, and using reminders, oriented towards achieving QOF – with some small adjustments that might improve QOF achievement – it seemed unlikely that the intervention would have a lasting effect in these practices.

Ethics review

WPs 1, 2 and 3 were approved by the National Research Ethics Service Committee Yorkshire and the Humber – Leeds Central (12/YH/0254). WPs 4 and 5 were approved by the UK National Research Ethics Service (14/SC/1393). All WPs were granted local NHS research governance approval.